High-expandability RNIC

Through the full-factor multi-level cache module and scheduling-prefetch mechanism, the cache miss and context switching problems of RNIC in multi-connection scenarios are solved, and high scalability and efficient packet processing are achieved, improving the performance of the data center network.

CN119003406APending Publication Date: 2024-11-22CHONGQING UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411022018.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In multi-connection scenarios, existing RNICs have significantly reduced throughput due to cache misses and context switching between host memory and RNIC, which cannot meet the high scalability requirements of data center networks.

Method used

A full-factor multi-level cache module was designed, including queue content cache module, area cache module, completion queue cache module, send queue cache module and receive queue cache module. Multi-level state machine and advanced extensible interface are used to realize efficient data reading. Fetch and cache management reduces PCIe latency and optimizes queue scheduling through the scheduling-prefetch mechanism.

Benefits of technology

It improves the high scalability of RNIC, ensures coordinated reading and writing between the host and on-chip cache, optimizes the packet processing pipeline, reduces the impact of head-of-line blockage on received packet processing, and improves overall performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003406A_ABST
    Figure CN119003406A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of high-performance computing, particularly discloses a high-expandability RNIC, and aims to solve the problem of throughput reduction caused by frequent cache miss and context switching between a host memory and the RNIC in a multi-connection scene when the connection number is increased to exceed the size of an on-chip storage. The host is enabled to cooperate with the on-chip cache, so that efficient data read-write operation is ensured; through an efficient scheduling and prefetching mechanism, efficient scheduling between QPs of the RNIC and hidden PCIe delay are achieved, and it is guaranteed that when the number of connections is increased, the performance is not reduced. By adopting the method and the device, the extensible RDMA network transmission with high QP connection number can be realized, and high bandwidth, low delay and low CPU (Central Processing Unit) occupation are ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of high performance computing technology, and in particular to a highly scalable RNIC. Background Art

[0002] In the era of rapid development of information technology, the demand for high-performance networks in the fields of high-performance computing (HPC), artificial intelligence (AI), and centralized / distributed storage has gradually emerged. According to statistics, the demand for data center network bandwidth will grow exponentially in the next few years. The demand for network bandwidth has led to the advancement of network protocols. The traditional TCP / IP protocol stack cannot meet the needs of high-speed networks due to its own multiple memory copy characteristics. RDMA (Remote Direct Memory Access) technology has been introduced in data center networks. RDMA technology allows applications to bypass the operating system kernel and communicate with remote computers in a zero-copy manner, reducing CPU load and multiple data copies, improving network transmission efficiency, and has the advantages of low latency and high bandwidth. The network interface controller that implements the RDMA protocol is called RNIC.

[0003] RDMA technology itself still has problems, especially in terms of connection scalability in data centers. Since RDMA technology was originally applied to Infiniband networks, the number of connection services is small, but in data center networks, the network scale is larger and more high-performance connections need to be provided.

[0004] Existing commercial RNIC designs include Figure 1As shown, it includes a PCIe (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard) interface, through which it interacts with the host, and a MAC (Media Access Control, media storage control) interface, through which network data is sent and received. The PCIe interface provides three functions, MMIO (Memory-Mapped Input / Output), interrupts, and DMA engines. MMIO is used by the host to read and write hardware registers. Interrupts are used for RNIC to actively initiate notifications to the host, including notifications of completion events and error handling. The DMA engine is used for the RNIC side to actively initiate memory read and write operations, including the reading and writing of context content and RDMA transmission data. MR (Memory Region) is used for address conversion, which records two parts of information; MPT (Memory Protection Table) contains the virtual address information, permission information, and key information of the registered memory area, and MTT (Memory Translation Table) specifically contains the page table information for virtual address conversion. QP (Queue Pair) content (Context) stores the configuration and status information of the QP connection, which is the key to hardware packet processing. The CQ (Completion Queue) content stores the status and configuration information of the CQ. The SQ (Send Queue) cache (cache) and the RQ (Receive Queue) cache store the WQE (Work Queue Element) in the send queue and receive queue respectively, and usually adopt a cache design. The QP Scheduler is responsible for scheduling QPs with pending transactions and arranging the sending tasks of different QPs according to the scheduling strategy. RDMA transmission includes components that implement the RDMA protocol. The requester is the party that actively initiates the request and obtains the transmission instruction from the send queue cache. The responder receives the remote request and obtains the storage instruction from the receive queue cache as needed. The command parsing module is used to parse the initialization configuration and read and write command operations sent by the upper layer.

[0005] In order to reduce costs, existing RNICs usually adopt a DRAM-free (Dynamic Random Access Memory) architecture, store context in host memory, and select limited SRAM (Static Random Access Memory) on the RNIC for data caching. In a multi-connection scenario, when the number of active connections increases beyond the on-chip storage size, frequent cache misses and context switching between host memory and RNIC will cause a significant drop in throughput, which greatly affects the overall performance of the network. It can be seen that the high scalability problem under the RNIC architecture needs to be solved urgently. Summary of the invention

[0006] The present invention provides a highly scalable RNIC, and solves the technical problem of how to design a highly scalable RNIC.

[0007] To solve the above technical problems, the present invention provides a highly scalable RNIC, which is equipped with a full-factor multi-level cache module, which is used to cache the cache areas corresponding to queue pairs, memory areas, completion queues, sending queues, and receiving queues respectively; the full-factor multi-level cache module includes a queue content cache module, an area cache module, a completion queue cache module, a sending queue completion module, and a receiving queue cache module; the queue content cache module is used to store and forward queue content context; the area cache module is used to store and forward memory area context; the sending queue completion module and the completion queue cache module are used to store work queue elements in the sending queue and the receiving queue, respectively.

[0008] Preferably, the queue content cache module includes a first-level queue content cache module, a second-level queue content cache module, and a cache bridge module; the second-level queue content cache module is used to directly interact with the upper layer for data; the first-level queue content cache module is used to provide queue content; the cache bridge module is used to realize data interaction between the second-level queue content cache module and the first-level queue content cache module.

[0009] Preferably, the first-level queue content cache module includes a first-level cache module at the sending end and a first-level cache module at the receiving end, both of which are readable and writable caches;

[0010] The secondary queue content cache module is used to cache the complete context of a single queue, and the primary queue content cache module is used to cache the queue content required by the packet processing level; the secondary queue content cache module initiates a cache maintenance request to the primary queue content cache module through the cache bridge module, and the primary queue content cache module reads the queue content through the request interface of the secondary queue content cache module and sends a notification of command execution completion; the secondary queue content cache module and the primary queue content cache module are interconnected using an advanced extensible interface to achieve efficient data reading.

[0011] Preferably, the cache line flag bits of the queue content cache module are increased with a write-back bit, a pollution bit, a sender presence bit and a receiver presence bit; the write-back bit indicates that the cache line data is being written back, and the pollution bit indicates that the cache line data is inconsistent with the memory; the sender presence bit indicates that the cache line data exists in the sender's first-level cache module, and the receiver presence bit indicates that the cache line data exists in the receiver's first-level cache module; when the second-level queue content cache module is replaced, the cache line existing in the first-level queue content cache module is not allowed to be used as a replacement object; only when the content in the first-level queue content cache module is written back to the second-level queue content cache module, the sender presence bit and the receiver presence bit are both null. When the second-level queue content cache module is valid, it can be replaced; when the software executes the locking operation, the second-level queue content cache module actively initiates an invalidation request to the first-level queue content cache module, notifying the first-level queue content cache module to write back the data. After the first-level queue content cache module completes the write-back, the second-level queue content cache module writes the data back to the memory; the commands supported by the second-level queue content cache module are the same as those of the completion queue cache module, and the second-level queue content cache module sends commands to the first-level queue content cache module through the cache bridge module; the cache bridge module is responsible for converting the blocking command initiated by the second-level queue content cache module into a non-blocking command and sending it to the first-level queue content cache module to ensure the successful execution of the command. In addition, the queue content cache module adopts a multi-level state machine design, the main state machine processes the request task, and mainly realizes the control logic; the slave state machine is mainly used to request or write back data through DMA, and mainly realizes the transmission of data; this design separates the control path from the data path, and the two do not block each other, which improves the efficiency of the overall circuit pipeline packet processing.

[0012] Preferably, the sending queue cache module, the receiving queue cache module, and the completion queue cache module are similar in structure to the secondary queue content cache module. The sending queue cache module and the receiving queue cache module are used to implement the storage of work queue elements, and the completion queue cache module is used to implement the storage of completion queue information, which is a read-only cache structure;

[0013] The memory area cache module is a read-only cache, which only needs to read the verification address and permission information, and calculate the bus address at the same time without modification; the memory area cache module is composed of a FIFO, namely a first-in first-out queue, a RAM, namely a random access memory, and a state machine; the FIFO is used to reduce the state machine blocking caused by signal handshake and improve the execution efficiency of the state machine, wherein the host-to-board request FIFO stores the host-to-board direct memory access request, the host-to-board data FIFO stores the host-to-board direct memory access data that has not been received yet, and the host-to-board raw data FIFO stores the address information of the direct memory access data, which is submitted to the state machine for processing together with the direct memory access data; the RAM is composed of a flag bit for storing cache line status information and specific data stored; the state machine uses a write port to write data, and the state machine uses a read interface to read data; the state machine is responsible for processing various requests inside and outside the cache, and the external request is divided into a blocking request and a non-blocking request, the blocking request is used for data reading, and the non-blocking request is used to execute various cache transactions.

[0014] Preferably, the memory area cache module adopts a four-way group connected structure, and the depth of each way can be dynamically configured according to parameters; MRN, i.e., the memory area cache number, is recorded as the unique identifier of the memory area cache module, and the lower N bits of MRN are mapped to the flag RAM address, and the remaining bits are stored in the cache line state as comparison flag bits, which are used for comparing cache lines of different ways within the same group, and the Nth power of 2 is the number of cache groups; the valid bits in the remaining bits indicate that the cache is valid, the existence bits indicate that the data exists on the chip, the lock bits indicate that the cache line is locked, and the suspension bits indicate that data is being read from the upper level;

[0015] The sending queue cache module, the receiving queue cache module, the completion queue cache module, and the queue content cache module adopt the same flag bit design as the memory area module.

[0016] Preferably, the highly scalable RNIC is further provided with a sending end request module, a receiving end response module, a sending scheduling module, a sending parsing module, a receiving pre-fetching module, a global ID mapping table, a sending path module, a receiving path module, a congestion control module and a flow control module;

[0017] The sending end request module is used to realize the pre-fetching and distribution of sending tasks;

[0018] The receiving end response module is used to implement pre-fetching and responding to received data packets;

[0019] The sending scheduling module is used to schedule sending transactions between multiple queues, using a bitmap structure to store queue information and a polling scheduling strategy to reduce the retrieval time of the scheduler. Every time there is a sending task in the corresponding queue, its corresponding bitmap position will be high, and the scheduler will schedule it, wait for the corresponding task to be processed, and then clear the corresponding bitmap position;

[0020] The sending parsing module is used to process the sending transaction, generate various fields in the data packet header and initiate a DMA request;

[0021] The receiving pre-fetching module is used to pre-fetch the queue content according to the received data packet;

[0022] The global ID mapping table is used to provide a global ID field for sending and receiving data packets;

[0023] The sending path module is used to assemble and send the sending data packets;

[0024] The receiving path module is used for receiving and parsing received data packets;

[0025] The congestion control module and the flow control module are used to limit the sending and receiving rates when a high data volume load cannot be processed in time.

[0026] Preferably, the sending parsing module includes an arbitration module, a DMA request module, and a header creation module. The arbitration module is responsible for receiving the sending end transaction and the receiving end transaction from the sending end request module and the receiving end response module and arbitrating the two groups of transaction interfaces, splitting the sending end transaction into raw data for generating a data packet header and discrete aggregate element data describing a data packet load, and sending them to the header creation module and the DMA request module respectively, and sending the generated data packet header information to the packetization module in the sending path module in a unified format;

[0027] The DMA request module initiates a direct memory access, i.e., a DMA request, based on the address and length information in the discrete aggregate element data and the information used to determine the remaining space in the FIFO of the subsequent packetization module;

[0028] The header creation module is used to generate a corresponding header data packet from the header information sent by the arbitration module, and then send it to the packetization module of the sending path module.

[0029] Preferably, the sending path module is provided with a packetization module, which is used to assemble data packets according to header information, and internally involves splicing of data headers and data payloads, and data payloads and data payloads, and realizes merging of multiple AXIS (Advanced Extensible Interface) data streams into one AXIS data stream through cyclic shift;

[0030] The packetization module includes a fast packet creation module, a header conversion module, a packet creation module, a merging module, and a splicing module; the fast packet creation module is used to receive the data load-free sending information from the receiving end response module to generate a corresponding data packet; the header conversion module is used to receive the header information from the sending parsing module, first perform network byte order conversion, and then assemble the data packet header according to the control field; the packet creation module is responsible for completing the splicing of the data packet header and the data packet load according to the control information to assemble into a complete data packet, and the data packet load includes the data load directly obtained through DMA and the Inline contained in the working queue element, that is, the data load is less than 512 bytes at a time; the merging module arbitrates the data packets from the fast packet creation module and the packet creation module, and sends them to the lower-level module through a set of interfaces; the splicing module is used to assemble multiple AXIS data packets into a single AXIS data packet, which is used for multi-discrete aggregation element support of the working queue element;

[0031] The receiving path module is provided with an unpacking module for completing the parsing of the data packet, extracting the header of the data packet, and generating a data packet header descriptor in a unified format; the unpacking module includes a tail reduction module, a verification module and a header reduction module; the tail reduction module is responsible for removing the padding bytes and the unchanged cyclic redundancy check field at the tail of the data packet according to the Pad bit in the basic transmission header; the verification module is responsible for completing the data packet verification and generating a data packet descriptor, and directly discarding the data packet if the verification fails; the header reduction module is responsible for removing the data packet header according to the data load offset position calculated by the verification module, retaining only the data packet load and removing the gap before the data packet load.

[0032] Preferably, the sending end request module is provided with a receiving response verification module, and the receiving end response module is provided with a receiving request verification module; the receiving response verification module adopts a dual state machine design, including a read-write state machine and a verification state machine; the read-write state machine is responsible for preparing the first-level cache data and global ID information for the received header information, and reads the queue content cache module data through the QPC first-level cache read-write interface and the SGID read-only interface from the receiving path module, and at the same time, there is a buffer inside for caching; the verification state machine is responsible for verifying the correctness of the data packet, generating the corresponding control command, and updating the context information;

[0033] The internal design of the receiving request verification module also adopts a dual-state machine design, and the internal structure is the same as that of the sending response verification module. The difference is that the receiving request verification module also needs to generate an ACK reply based on the remote request.

[0034] A highly scalable RNIC provided by the present invention adopts a scheduling-prefetching mechanism design, mainly including a sending scheduling module (QP scheduler) and a prefetching chain design, which are used to realize efficient scheduling between queues of RNIC and hide PCIe delay. The principle is: since a PCIe request requires dozens of cycles, the efficient cooperation of scheduling and prefetching can issue a prefetching request in advance. When the data is needed, the data has arrived through the PCIe interface, thereby hiding the PCIe delay; in addition, the sending scheduling module uses a bitmap structure to store queue information, and adopts a polling scheduling strategy to reduce the retrieval time of the sending scheduling module, minimize the delay caused by queue switching, and reduce the occurrence of head-of-line blocking.

[0035] Compared with the prior art, the highly scalable RNIC provided by the present invention has the following beneficial effects:

[0036] A full-factor multi-level cache module is designed to enable the host and on-chip cache to work together, and QPC (QP Context) is further split according to the module read and write requirements, ensuring efficient read and write of different modules;

[0037] The deep pipeline optimization of packet processing is realized (pipeline optimization refers to dividing the sending / receiving tasks into multiple modules, such as pre-fetch module, processing module, sending parsing module, sending path, etc., which are designed like a pipeline). According to the characteristics of the packet processing process, it is decomposed into multiple modules, separating the control plane and the data plane, and improving the packet processing efficiency;

[0038] According to the relationship between the sending scheduling module and the sending and receiving prefetching modules and the dependency of different queue contents, a scheduling-prefetching mechanism is designed to reduce the impact of head-of-line blocking on the receiving packet processing pipeline. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a diagram of the existing commercial RNIC architecture provided by the background technology of the present invention;

[0040] Figure 2 This is an architecture diagram of a highly scalable RNIC provided by an embodiment of the present invention;

[0041] Figure 3 It is a structural diagram of a full-factor multi-level cache module provided by an embodiment of the present invention;

[0042] Figure 4 It is a structural block diagram of a sending parsing module provided in an embodiment of the present invention;

[0043] Figure 5 is a structural block diagram of a packetization module provided by an embodiment of the present invention;

[0044] Figure 6 is a structural block diagram of an unpacking module provided by an embodiment of the present invention;

[0045] Figure 7 is a structural block diagram of a receiving response verification module provided by an embodiment of the present invention;

[0046] Figure 8 It is a structural block diagram of a receiving request verification module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following specifically illustrates the implementation mode of the present invention in conjunction with the accompanying drawings. The embodiments are provided for illustrative purposes only and cannot be understood as limiting the present invention. The accompanying drawings are provided for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0048] An embodiment of the present invention provides a highly scalable RNIC, whose architecture is as follows: Figure 2 As shown, in addition to the PCIe interface, MIMO, DMA engine and MAC set up in the existing RNIC, there are also a full-factor multi-level cache module, a sender request module, a receiver response module, a send scheduling module, a send parsing module, a receive prefetch module, a global ID mapping table, a send path module, a receive path module, a congestion control module and a flow control module.

[0049] The full-factor multi-level cache module is used to cache the cache areas corresponding to the queue pair (QP), memory region (MR), completion queue (CQ), send queue (SQ), and receive queue (RQ) designs respectively;

[0050] The sender request module is used to implement the pre-fetching and distribution of sending tasks;

[0051] The receiving end response module is used to implement pre-fetching and responding to received data packets;

[0052] The sending scheduling module is used to schedule sending transactions between multiple queues. It uses a bitmap structure to store queue information and adopts a polling scheduling strategy to reduce the retrieval time of the scheduler. Every time there is a sending task in the corresponding queue, its corresponding bitmap position will be high, and the scheduler will schedule it. After the corresponding task is processed, the corresponding bitmap position will be cleared;

[0053] The sending parsing module is used to process the sending transaction, generate various fields in the data packet header and initiate a DMA request;

[0054] The receiving pre-fetching module is used to pre-fetch the queue content according to the received data packet;

[0055] The global ID mapping table is used to provide the global ID field for sending and receiving data packets;

[0056] The sending path module is used to assemble and send the sending data packets;

[0057] The receiving path module is used to receive and parse the received data packets;

[0058] The congestion control module and the flow control module are used to limit the sending and receiving rates when the high data load cannot be processed in time.

[0059] like Figure 2 As shown, the full-element multi-level cache module includes a queue content cache module (QP cache), a regional cache module (MR cache), a completion queue cache module (CQ cache), a send queue completion module (SQ cache) and a receive queue cache module (RQ cache). The queue content cache module (QP cache) is used to store and forward queue content contexts. The regional cache module (MR cache) is used to store and forward MR contexts. The send queue completion module (SQ cache) and the completion queue cache module (CQ cache) are used to store WQEs (Work Queue Element) in the send queue and the receive queue, respectively.

[0060] Figure 3 It is a schematic diagram of the QPC multi-level cache provided by the present invention.

[0061] More specifically, Figure 2 , Figure 3 As shown, the queue content cache module includes a first-level queue content cache module, a second-level queue content cache module, and a cache bridge module. The second-level queue content cache module is used to directly interact with the upper layer; the first-level queue content cache module is used to provide queue content to the sending path module and the receiving path module; the cache bridge module is used to realize the data interaction between the second-level queue content cache module and the first-level queue content cache module.

[0062] More specifically, the first-level queue content cache module includes a sending-end first-level cache module and a receiving-end first-level cache module, both of which are readable and writable caches, serving the sending-end request module and the receiving-end response module respectively.

[0063] The secondary queue content cache module is used to cache the complete context of a single queue, and the primary queue content cache module is used to cache the queue content required for the packet processing level. Since the queue content cache is a multi-level cache architecture, the secondary queue content cache module needs to consider the data consistency between the data and memory and the primary queue content cache module. The secondary queue content cache module initiates a cache maintenance request to the primary queue content cache module through the cache bridge module. The primary queue content cache module reads the queue content through the request interface of the secondary queue content cache module and sends a notification of command execution completion. Since there are many queue content data and modules to be read, the secondary queue content cache module and the primary queue content cache module use AXI4 (Advanced eXtensible Interface) interconnection to achieve efficient data reading.

[0064] In addition, due to the complexity of the structure of the queue content cache module, compared with the cache line status design of the memory area cache module, the cache line flag bits of the queue content cache module have been increased to include a writeback bit (Writeback), a dirty bit (Dirty), a sender existence bit (Exist_RQST), and a receiver existence bit (Exist_RSPD) to maintain data consistency. The writeback bit indicates that the cache line data is being written back, the dirty bit indicates that the cache line data is inconsistent with the memory, the sender existence bit indicates that the cache line data exists in the sender's first-level cache module, and the receiver existence bit indicates that the cache line data exists in the receiver's first-level cache module. In order to simplify consistency maintenance, when the second-level queue content cache module is replaced, the cache line in the first-level queue content cache module is not allowed to be replaced. Only when the content in the first-level queue content cache module is written back to the second-level queue content cache module, and the sender existence bit and the receiver existence bit are both invalid, can the replacement be performed. When the software performs a locking operation, the secondary queue content cache module can actively initiate an invalidation request to the primary queue content cache module, informing the primary queue content cache module to write back the data. After the primary queue content cache module completes the write-back, the secondary queue content cache module writes the data back to the memory. The commands supported by the secondary queue content cache module are the same as those of the completion queue cache module. The secondary queue content cache module sends commands to the primary queue content cache module through the cache bridge module. In order to reduce the blocking of the execution process of the secondary queue content cache module, the cache bridge module is responsible for converting the blocking commands initiated by the secondary queue content cache module into non-blocking commands and sending them to the primary queue content cache module to ensure successful execution of the commands (the cache bridge module may need to retry the command multiple times until the execution is successful).

[0065] In addition, the queue content cache module adopts a multi-level state machine design. The master state machine processes request tasks and mainly implements control logic; the slave state machine is mainly used to request or write back data through DMA, which mainly realizes data transmission; this design separates the control path from the data path, and the two do not block each other, thereby improving the efficiency of the overall circuit pipeline packet processing.

[0066] The sending queue cache module, receiving queue cache module, and completion queue cache module are similar in structure to the secondary queue content cache module. The sending queue cache module and the receiving queue cache module implement the storage of WQE, and the completion queue cache module implements the storage of completion queue information. It is a read-only cache structure and there is no cache consistency problem.

[0067] Specifically, the memory area cache module is a read-only cache that only needs to read the verification address and permission information, and calculate the bus address at the same time without modification. The memory area cache module consists of FIFO (First Input First Output), RAM (Random Access Memory) and a state machine. FIFO is used to reduce the state machine blocking caused by signal handshake and improve the execution efficiency of the state machine. The H2C REQ (Host to Card Request) FIFO stores the H2C DMA (host to board direct memory access) request, the H2C Data (host to board data) FIFO stores the H2C DMA data that has not been received yet, and the H2C meta (host to board raw data) FIFO stores the address information of the DMA data and submits it to the state machine for processing together with the DMA data. RAM consists of a tag that stores cache line status information and specific data (Data) stored. Because the memory area cache module is a read-only cache, RAM uses SDPRAM (Simple Dual Port RAM). SDPRAM provides a set of write interfaces and a set of read interfaces. The internal state machine uses the write port to write data, and the state machine uses the read interface to read data. The state machine is responsible for processing various requests inside and outside the cache. External requests are divided into blocking requests and non-blocking requests. Blocking requests are used for data reading, and non-blocking requests are used to execute various cache transactions. In the memory area cache module, non-blocking requests are initiated by the send queue prefetch module and the command execution module (send parsing module and receive prefetch module) in the sender request module. The cache line state design and blocking and non-blocking request design will be specifically introduced below.

[0068] Specifically, the memory area cache module adopts a four-way group-connected structure, and the depth of each way can be dynamically configured according to parameters. MRN (MR Number, memory area cache number) is recorded as the unique identifier of MR, and the low N bits of MRN (2 to the power of N is the number of cache groups) are mapped to the flag RAM address, and the remaining bits are stored in the cache line state as comparison flags, which are used to compare cache lines of different ways within the same group. The valid bit (Valid) indicates that the cache is valid, the existence bit (Exist) indicates that the data exists on the chip, the lock bit (Lock) indicates that the cache line is locked, and the pending bit (Pending) indicates that data is being read from the upper level. Different from the design structure of the CPU cache, in order to ensure the consistency of data between the memory and the cache and hide PCIe delays, the present invention adds fields such as existence bits, lock bits and pending bits to achieve efficient data reading. This is also reflected in other designed caches, that is, the sending queue cache module, the receiving queue cache module, the completion queue cache module, and the queue content cache module adopt the same flag bit design as the memory area module.

[0069] Specifically, in the design of each cache module, a cache request is separated from data reading and writing, and two groups of request interfaces are provided to the outside, one group is a blocking request interface, and the other group is a non-blocking request interface. The request of the blocking interface is returned only when the execution is successful. If the execution condition is not met when the request of the non-blocking interface is executed, it will directly return failure. The two interfaces are suitable for cache operation requirements of different functions, and the request processing capability of the cache is improved to the greatest extent. When the external module accesses data, it must first initiate a cache request, obtain the corresponding cache line address, and then access the data through the data interface. The blocking interface is generally only used when the external module initiates a read. If the requested data does not exist in the cache module, the cache module will initiate a DMA request. The blocking interface will not return the address until the DMA data is stored in the cache module. The non-blocking interface is generally used for locking, unlocking, pre-fetching requests, etc. When the cache module receives a non-blocking request, it will check whether the request can be executed. If it can be executed, it will execute the request. Regardless of whether the request is executed successfully, the cache module will return the corresponding cache line flag. The requester can judge whether the request is executed successfully and make further processing based on the return value.

[0070] Figure 4 This is the structure diagram of the sending parsing module. Figure 4 As shown, the sending parsing module includes an arbitration module, a DMA request module, and a header creation module, which are used to process sending transactions, generate various fields in the header of a data packet and initiate a DMA request.

[0071] Specifically, the arbitration module is responsible for receiving the sender transactions and the receiver transactions from the sender request module and the receiver response module and arbitrating the two groups of transaction interfaces, splitting the sender transactions into raw data (HDR metadata) used to generate the data packet header and SGE (Scatter Gather Element) data describing the data packet payload, and sending them to the header creation module and the DMA request module respectively, and sending the generated data packet header information to the packetization module in the sending path module in a unified format.

[0072] The DMA request module initiates a DMA request based on the address and length information in the SGE data and the information used to determine the remaining space in the FIFO of the subsequent packet module. Due to PCIe limitations, each DMA request cannot span a 4K page (4096 bytes). If it exceeds, it needs to be split into multiple requests. The length of the first data request is the offset from the starting address to the end of the current 4K page, and subsequent DMA requests obtain 4K data each time.

[0073] The header creation module will generate data packet header information in a unified format based on the header metadata passed by the arbitration module and send it to the packetization module, which includes IP control information, data payload length, data type, etc.

[0074] like Figure 2 As shown, the sending path module includes a packaging module, a sending verification module, a sending capture module and a sending packet statistics module. The receiving path module includes an unpacking module, a receiving verification module, a receiving capture module and a receiving packet statistics module.

[0075] The packetization module is used to assemble data packets according to the header information, which involves the splicing of data header and data payload, and data payload and data payload. Through circular shift, multiple AXIS (Advanced eXtensibleInterface Stream) are merged into one AXIS data stream.

[0076] More specifically, Figure 5 As shown in the structural diagram of the packet module, the packet module consists of five parts, including a fast packet creation module, a header conversion module, a packet creation module, a merging module, and a splicing module.

[0077] The fast packet creation module is used to receive the data-free payload transmission information (such as ACK, Acknowledgement, confirmation) from the receiving end response module and generate the corresponding data packet. Because the data packet is small, the packetization process is simple and needs to be replied as soon as possible, it is packaged separately.

[0078] The header conversion module is used to receive the header information from the sending parsing module, first perform network byte order conversion, and then assemble the data packet header according to the control field.

[0079] The packet creation module is responsible for completing the splicing of the data packet header and the data packet payload according to the control information to assemble a complete data packet. The data packet payload includes the data payload directly obtained through DMA and the Inline payload (a data payload is less than 512 bytes of data) contained in WQE. For the transmission of smaller data, using the Inline method can reduce the time of a DMA request and improve the transmission efficiency.

[0080] The merging module arbitrates the data packets from the fast packet creation module and the packet creation module, and sends them to the lower-level modules through a set of interfaces. Since the ACK and read response pass through different data packet creation modules when they are sent, for a single QP (queue pair), its ACK reply to other requests after the read response needs to be guaranteed to be later than the sending of the previously unfinished read response. The RRNotify (Read Response Notify) interface is used to notify the receiving end response module of the completion of the read response sending after the merging module serializes the data packet, and the receiving end response module then generates an ACK for subsequent requests. The FIFOHandshake (FIFO handshake signal) interface is used to notify the sending parsing module of the FIFO read status, and the sending parsing module is used to calculate the remaining space in the FIFO, and initiate a DMA request based on this to avoid H2C DMA data flow blocking.

[0081] The splicing module is used to assemble multiple AXIS data packets into a single AXIS data packet for multi-SGE support of WQE, that is, a single WQE can specify multiple discrete continuous memory blocks, and solve the problem of a single sending task being split into multiple AXIS data packets because the user specifies that the sending virtual address can be an arbitrary offset and the PCIe single DMA cannot exceed 4096 bytes in size.

[0082] More specifically, the unpacking module is used to complete the parsing of the data packet, extract the header of the data packet, and generate a unified format of data packet header descriptor to facilitate subsequent modules to process, which is the reverse operation of the packaging module. Figure 6As shown in the structural diagram of the unpacking module, the unpacking module includes a tail trimming module, a verification module and a header trimming module. The tail trimming module is responsible for removing the padding bytes and ICRC (Invariant Cyclic Redundancy Check) field at the end of the data packet according to the BTH Pad (Base Transport Header Pad, Pad bit in the basic transport header). The verification module is responsible for completing the data packet verification and generating a data packet descriptor. If the verification fails, the data packet is directly discarded. The header trimming module is responsible for removing the data packet header according to the data payload offset position calculated by the verification module, retaining only the data packet payload and removing the gap before the data packet payload.

[0083] The sending verification module is responsible for checking whether there are errors in the data packet, which is mainly achieved by determining whether the ICRC is the set value; the sending capture module realizes the capture of the sending data packet, which is an auxiliary module used for debugging. For example, this module can be used to determine what data type the currently sent packet belongs to; the sending packet statistics module is used to count the data packets in the entire sending path, such as the total number of data packets sent, the number of retransmitted data packets, etc.

[0084] The receiving verification module, receiving capture module and receiving packet statistics module are similar to the corresponding sending verification module, sending capture module and sending packet statistics module, and respectively realize the received data packet verification, the received data packet capture and the statistics of various received data packets.

[0085] like Figure 2 As shown, the sending end request module includes a sending queue pre-fetching module, a sending task processing module, a receiving response verification module, and a response packet header verification module. The receiving end response module includes a receiving queue pre-fetching module, a reading response processing module, a receiving request verification module, and a receiving packet header verification module.

[0086] like Figure 7 As shown in the structural diagram of the receiving response verification module, the receiving response verification module is located in the sending end request module and is responsible for processing the response data packets generated by the remote end to the local end request, including read responses and ACK data packets. The receiving response verification module adopts a dual-state machine design, including a read-write state machine and a verification state machine. The read-write state machine is responsible for preparing the first-level cache data and global ID information for the received header information, and reads the queue content cache module data through the QPC first-level cache read-write interface and the SGID read-only interface to read data from the receiving path module. At the same time, there is a buffer inside for caching, which reduces repeated reading processes and improves the efficiency of the verification state machine;

[0087] The verification state machine is responsible for verifying the correctness of the data packet, generating the corresponding control command, and updating the context information. On the one hand, it reads the information of the read-write state machine, and on the other hand, it reads the content of the queue content cache module through the secondary cache blocking interface and the QPC secondary cache read-write interface, reads the send queue cache module data through the send queue read-only interface, reads the completion queue cache module information through the completion queue application interface, and reads the corresponding memory area data in the memory area cache module through the memory area read-only interface.

[0088] The sending queue prefetch module is used to prefetch the sending queue content by accessing the queue content cache module and the memory area cache module. The sending task processing module is used to verify the sending transaction with the scheduling information obtained by the sending scheduling module and pass the sending transaction. The response header verification module is used to verify the correctness of the data packet from the receiving path module.

[0089] like Figure 8 As shown in the structural diagram of the receiving request verification module, the receiving request verification module is located in the receiving end response module, which is responsible for processing the remote request and making corresponding responses. Its internal design also adopts a dual-state machine design, and its internal structure is similar to that of the sending response verification module. The receiving request verification module directly affects the effective data throughput of the RNIC. The structure of the receiving request verification module is similar to that of the receiving response verification module, and both adopt a two-level state machine design. The difference is that the receiving request verification module also needs to generate an ACK reply based on the remote request. Since the subsequent requests cannot be ACKed when there is an incomplete read response, the packetization module will notify the receiving request verification module to update the internal state through the read response notification interface every time it sends a completed read response, so as to ACK the subsequent processed requests.

[0090] The receiving queue pre-fetching module pre-fetches the receiving queue content by accessing the queue content cache module. The read response processing module receives the read response data packet from the receiving path and performs further parsing and processing. The receiving data packet header verification module is used to perform header verification on the load data packet of the receiving path to ensure the correctness of the load data.

[0091] The scheduling-prefetching mechanism design mainly includes the sending scheduling module (QP scheduler) and the prefetching chain design (such as Figure 3As shown in the figure, it is used to realize efficient scheduling between queues of RNIC and hide PCIe delay. The principle is: since a PCIe request takes dozens of cycles, the efficient coordination of scheduling and prefetching can issue prefetch requests in advance. When data is needed, the data has arrived through the PCIe interface, thereby hiding the PCIe delay. In addition, the sending scheduling module uses a bitmap structure to store queue information and adopts a polling scheduling strategy to reduce the retrieval time of the sending scheduling module, minimize the delay caused by queue switching, and reduce the occurrence of head-of-line blocking.

[0092] In summary, the highly scalable RNIC provided by the embodiment of the present invention has the following beneficial effects:

[0093] A full-factor multi-level cache module is designed to enable the host and on-chip cache to work together, and QPC (QP Context) is further split according to the module read and write requirements, ensuring efficient read and write of different modules;

[0094] The deep pipeline optimization of packet processing is realized (pipeline optimization refers to dividing the sending / receiving tasks into multiple modules, such as pre-fetch module, processing module, sending parsing module, sending path, etc., which are designed like a pipeline). According to the characteristics of the packet processing process, it is decomposed into multiple modules, separating the control plane and the data plane, and improving the packet processing efficiency;

[0095] According to the relationship between the sending scheduling module and the sending and receiving prefetching modules and the dependency of different queue contents, a scheduling-prefetching mechanism is designed to reduce the impact of head-of-line blocking on the receiving packet processing pipeline.

[0096] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A highly scalable RNIC, characterized by: A full-factor multi-level cache module is provided, which is used to cache the cache areas corresponding to the queue pairs, memory areas, completion queues, sending queues, and receiving queues respectively; the full-factor multi-level cache module includes a queue content cache module, an area cache module, a completion queue cache module, a sending queue completion module, and a receiving queue cache module; the queue content cache module is used to store and forward queue content context; the area cache module is used to store and forward memory area context; the sending queue completion module and the completion queue cache module are used to store the work queue elements in the sending queue and the receiving queue, respectively.

2. The highly scalable RNIC according to claim 1, characterized in that: The queue content cache module includes a first-level queue content cache module, a second-level queue content cache module, and a cache bridge module; the second-level queue content cache module is used to directly interact with the upper layer for data; the first-level queue content cache module is used to provide queue content; the cache bridge module is used to realize data interaction between the second-level queue content cache module and the first-level queue content cache module.

3. The highly scalable RNIC according to claim 2, characterized in that: The first-level queue content cache module includes a sending-end first-level cache module and a receiving-end first-level cache module, both of which are readable and writable caches; The secondary queue content cache module is used to cache the complete context of a single queue, and the primary queue content cache module is used to cache the queue content required by the packet processing level; the secondary queue content cache module initiates a cache maintenance request to the primary queue content cache module through the cache bridge module, and the primary queue content cache module reads the queue content through the request interface of the secondary queue content cache module and sends a notification of command execution completion; the secondary queue content cache module and the primary queue content cache module are interconnected using an advanced extensible interface to achieve efficient data reading.

4. The highly scalable RNIC according to claim 3, characterized in that: The cache line flag bits of the queue content cache module are increased with a write-back bit, a pollution bit, a sender presence bit, and a receiver presence bit; the write-back bit indicates that the cache line data is being written back, and the pollution bit indicates that the cache line data is inconsistent with the memory; The sending end existence bit indicates that the cache line data exists in the sending end primary cache module, and the receiving end existence bit indicates that the cache line data exists in the receiving end primary cache module; when the secondary queue content cache module is replaced, the cache line existing in the primary queue content cache module is not allowed to be replaced; replacement can only be performed when the content in the primary queue content cache module is written back to the secondary queue content cache module and both the sending end existence bit and the receiving end existence bit are invalid; when the software performs a locking operation, the secondary queue content cache module actively initiates an invalidation request to the primary queue content cache module to notify the primary queue content cache module to write back the data, and after the primary queue content cache module completes the write-back, the secondary queue content cache module writes the data back to the memory; the commands supported by the secondary queue content cache module are the same as those of the completion queue cache module, and the secondary queue content cache module sends commands to the primary queue content cache module through the cache bridge module; the cache bridge module is responsible for converting the blocking command initiated by the secondary queue content cache module into a non-blocking command and sending it to the primary queue content cache module to ensure successful execution of the command.

5. The highly scalable RNIC according to claim 4, characterized in that: The sending queue cache module, the receiving queue cache module, and the completion queue cache module are similar in structure to the secondary queue content cache module. The sending queue cache module and the receiving queue cache module are used to store work queue elements, and the completion queue cache module is used to store completion queue information, which is a read-only cache structure. The memory area cache module is a read-only cache, which only needs to read the verification address and permission information, and calculate the bus address at the same time without modification; the memory area cache module is composed of a FIFO (first-in first-out queue), a RAM (random access memory) and a state machine; the FIFO is used to reduce the state machine blockage caused by signal handshake and improve the execution efficiency of the state machine, wherein the host-to-board request FIFO stores the host-to-board direct memory access request, the host-to-board data FIFO stores the host-to-board direct memory access data that has not been received yet, and the host-to-board raw data FIFO stores the address information of the direct memory access data, which is submitted to the state machine for processing together with the direct memory access data; The RAM consists of a flag bit for storing cache line status information and specific data stored; the state machine uses a write port to write data, and the state machine uses a read interface to read data; the state machine is responsible for processing various requests inside and outside the cache, and external requests are divided into blocking requests and non-blocking requests. Blocking requests are used for data reading, and non-blocking requests are used to execute various cache transactions.

6. The highly scalable RNIC according to claim 5, characterized in that: The memory area cache module adopts a four-way group connected structure, and the depth of each way can be dynamically configured according to parameters; MRN, that is, the memory area cache number, is recorded as the unique identifier of the memory area cache module, and the lower N bits of MRN are mapped to the flag RAM address, and the remaining bits are stored in the cache line state as comparison flag bits, which are used for comparing cache lines of different ways within the same group, and the Nth power of 2 is the number of cache groups; the valid bits in the remaining bits indicate that the cache is valid, the existence bits indicate that the data exists on the chip, the lock bits indicate that the cache line is locked, and the suspension bits indicate that data is being read from the upper level; The sending queue cache module, the receiving queue cache module, the completion queue cache module, and the queue content cache module adopt the same flag bit design as the memory area module.

7. The highly scalable RNIC according to claim 6, characterized in that: The highly scalable RNIC is also provided with a sender request module, a receiver response module, a send scheduling module, a send parsing module, a receive pre-fetch module, a global ID mapping table, a send path module, a receive path module, a congestion control module and a flow control module; The sending end request module is used to realize the pre-fetching and distribution of sending tasks; The receiving end response module is used to implement pre-fetching and responding to received data packets; The sending scheduling module is used to schedule sending transactions between multiple queues; The sending parsing module is used to process the sending transaction, generate various fields in the data packet header and initiate a DMA request; The receiving pre-fetching module is used to pre-fetch the queue content according to the received data packet; The global ID mapping table is used to provide a global ID field for sending and receiving data packets; The sending path module is used to assemble and send the sending data packets; The receiving path module is used for receiving and parsing received data packets; The congestion control module and the flow control module are used to limit the sending and receiving rates when a high data volume load cannot be processed in time.

8. The highly scalable RNIC according to claim 7, characterized in that: The sending parsing module includes an arbitration module, a DMA request module, and a header creation module. The arbitration module is responsible for receiving the sending end transaction and the receiving end transaction from the sending end request module and the receiving end response module and arbitrating the two groups of transaction interfaces, splitting the sending end transaction into raw data for generating a data packet header and discrete aggregate element data describing a data packet load, and sending them to the header creation module and the DMA request module respectively, and sending the generated data packet header information to the packetization module in the sending path module in a unified format; The DMA request module initiates a direct memory access, i.e., a DMA request, based on the address and length information in the discrete aggregate element data and the information used to determine the remaining space in the FIFO of the subsequent packetization module; The header creation module is used to generate a corresponding header data packet from the header information sent by the arbitration module, and then send it to the packetization module of the sending path module.

9. The highly scalable RNIC according to claim 8, characterized in that: The sending path module is provided with a packetization module, which is used to assemble data packets according to header information, and internally involves splicing of data headers and data payloads, and data payloads and data payloads, and realizes merging of multiple AXIS (Advanced Extensible Interface) data streams into one AXIS data stream through cyclic shift; The packet encapsulation module includes a fast packet creation module, a header conversion module, a packet creation module, a merging module, and a splicing module; the fast packet creation module is used to receive the data load-free sending information from the receiving end response module and generate a corresponding data packet; the header conversion module is used to receive the header information from the sending parsing module, first perform network byte order conversion, and then assemble the data packet header according to the control field; the packet creation module is responsible for completing the splicing of the data packet header and the data packet load according to the control information to assemble into a complete data packet, and the data packet load includes the data load directly obtained through DMA and the Inline contained in the work queue element, that is, the data load is less than 512 bytes at a time; the merging module arbitrates the data packets from the fast packet creation module and the packet creation module, and sends them to the lower-level module through a group of interfaces; The splicing module is used to assemble multiple AXIS data packets into a single AXIS data packet for multiple discrete aggregation element support of work queue elements; The receiving path module is provided with an unpacking module for completing the parsing of the data packet, extracting the header of the data packet, and generating a data packet header descriptor in a unified format; the unpacking module includes a tail reduction module, a verification module and a header reduction module; the tail reduction module is responsible for removing the padding bytes and the unchanged cyclic redundancy check field at the tail of the data packet according to the Pad bit in the basic transmission header; the verification module is responsible for completing the data packet verification and generating a data packet descriptor, and directly discarding the data packet if the verification fails; the header reduction module is responsible for removing the data packet header according to the data load offset position calculated by the verification module, retaining only the data packet load and removing the gap before the data packet load.

10. The highly scalable RNIC according to claim 9, characterized in that: The sending end request module is provided with a receiving response verification module, and the receiving end response module is provided with a receiving request verification module; the receiving response verification module adopts a dual state machine design, including a read-write state machine and a verification state machine; the read-write state machine is responsible for preparing the first-level cache data and global ID information for the received header information, and reads the queue content cache module data through the QPC first-level cache read-write interface and the SGID read-only interface from the receiving path module, and at the same time there is a buffer inside for caching; the verification state machine is responsible for verifying the correctness of the data packet, generating the corresponding control command, and updating the context information; The internal design of the receiving request verification module also adopts a dual-state machine design, and the internal structure is the same as that of the sending response verification module. The difference is that the receiving request verification module also needs to generate an ACK reply based on the remote request.