Multi-node communication architecture based on RDMA

Abstract

The invention discloses a multi-node communication architecture based on RDMA (Remote Direct Memory Access), which comprises the following modules on a network card: a PCIe (Peripheral Component Interconnect Express) function module which supports PCIe x16, is downwards compatible, is used for data interaction between a master control on a local node and an RDMA function module, and supports high-speed transceiving; the MAC module is used for supporting high-speed data transceiving between the RDMA functional module and an external node by using a high-bandwidth MAC; and the RDMA function module is used for realizing a standard RoCEv2 protocol, supporting unicast and multicast, supporting reliable connection service and unreliable datagram service, supporting bilateral send / recv operation, and supporting unilateral write and read operation. The architecture can be used for a multi-core CPU heterogeneous airborne embedded supercomputing platform, realizes lossless Ethernet communication, has the characteristics of low delay, low CPU load and the like, and is used for meeting the application requirements of future airborne comprehensive processing platforms.

Description

Technical Field

[0001] The present invention relates to the field of lossless Ethernet communication, and more particularly to a multi-node communication architecture based on RDMA. Background Art

[0002] In recent years, with the rapid development of technologies such as high-performance computing, artificial intelligence, and big data analysis, the requirements for network transmission performance have become increasingly higher. The TCP / IP protocol architecture used by traditional network cards has inherent problems such as slow data processing and large network transmission delays, and is gradually unable to meet the requirements for high network transmission performance. Distributed systems are increasingly used in the aerospace field, and the scale of system networking and network data traffic are increasing, which puts forward higher and higher requirements on the security, reliability, and real-time performance of data interaction between nodes in the system.

[0003] The development of RDMA (Remote Direct Memory Access) technology has greatly reduced the cost of data transmission between nodes. RDMA technology is a technology used to solve the server data processing delay in network transmission. It supports remote direct reading and writing of off-site memory without the intervention of the CPU and operating system of both parties. Compared with traditional network transmission technology, RDMA has outstanding advantages such as zero copy, kernel bypass, high throughput, and low latency. It occupies less system resources and is suitable for large-scale parallel computer clusters. In RDMA mode, data in user applications is directly transferred to the local storage area, and data is quickly transmitted from the local system to the remote system memory through the network. The whole process has no impact on the operating system, so it does not require too much computer processing power; it eliminates the overhead of external memory copying and context switching, thereby releasing memory bandwidth and shortening CPU cycles, greatly improving the efficiency of data interaction between the two communicating parties. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-node communication architecture based on RDMA, which can be used for multi-core CPU heterogeneous airborne embedded supercomputing platforms to achieve lossless Ethernet communication, while having the characteristics of low latency and low CPU load, so as to meet the application requirements of future airborne integrated processing platforms.

[0005] The object of the present invention is achieved through the following technical solutions:

[0006] A multi-node communication architecture based on RDMA, which includes the following modules on a network card:

[0007] PCIe function module: supports PCIe x16 and is backward compatible. It is used for data interaction between the master control and RDMA function modules on the local node and supports high-speed transmission and reception.

[0008] MAC module: uses high-bandwidth MAC to support high-speed data transmission and reception between the RDMA function module and external nodes;

[0009] RDMA function module: implements the standard RoCEv2 protocol, supports unicast and multicast, supports reliable connection services and unreliable datagram services, supports bilateral send / recv operations, supports unilateral write and read operations, and the internal architecture is as follows:

[0010] The CQP_MGR module is used to control the channel. It is responsible for reading CQPE from the CQP queue and parsing CQPE into hardware internal configuration instructions to achieve QP / CQ / CEQ / AEQ queue management.

[0011] The QP_MGR module is responsible for reading the WQEs in the SQ and RQ queues to the WQE_MGR module according to the scheduling of the QP queue;

[0012] The WQE_MGR module is used to support the SQ and RQ processing channels and complete the Send-Receive, Write, and Read operations together;

[0013] The CQ_MGR module is used to receive the completion status generated by the QP queue, encapsulate the completion status into CQE, fill it into the corresponding CQ queue, and upload it to the software for processing;

[0014] The BTH module is used to implement the encapsulation and decapsulation functions of Rocev2 data frames according to the Rocev2 protocol.

[0015] In some embodiments, an AIX-MUX module is further set between a PCIe functional module and multiple RMAD functional modules, and a MAC module is set under the RMAD functional module. The AIX-MUX module implements fair polling scheduling of multiple RDMAs, thereby realizing a single master control of multiple MAC nodes on a network card.

[0016] Furthermore, the PCIe functional module also supports the PF function, using multiple PFs, corresponding to one RDMA and one MAC respectively, to achieve a one-to-one correspondence between PCIe, RDMA and MAC.

[0017] In some embodiments, a distributed aggregation module is further configured in the RDMA functional module. The RDMA functional module is connected to multiple MAC modules. The distributed aggregation module distributes the RDMA message to the specified MAC according to specific information, or obtains the corresponding RDMA message from the MAC to the PCLe functional module, thereby realizing a single master control of multiple MAC nodes on one network card.

[0018] In some embodiments, an AIX-MUX module is set between a PCIe functional module and multiple RMAD functional modules, and the AIX-MUX module implements fair polling scheduling of multiple RDMAs; a decentralized aggregation module is configured in the RDMA functional module, and the RDMA functional module is connected to multiple MAC modules. The decentralized aggregation module distributes the RDMA message to the specified MAC according to specific information, or obtains the corresponding RDMA message from the MAC to the PCLe functional module, so as to realize that one RDMA corresponds to multiple PCIe and multiple MAC, and multiple master controllers control multiple MAC nodes on one network card.

[0019] In some embodiments, an AXI_MUX module, an RDMA function module, a NIC function module, a TSN function module, a SWITCH module, and a TSN protocol processing module are configured between a group of PCIe function modules and a MAC module to implement transmission of three protocol data between a single master control and a single MAC node;

[0020] Among them, the AXI_MUX module implements the formula polling scheduling among the RDMA functional module, the NIC functional module, and the TSN functional module; the NIC functional module processes TCP and UDP related network protocols; the TSN functional module mainly implements the UDP layer protocol unloading; the SWITCH module implements fair polling scheduling and priority processing among the three functional modules of RDMA functional module, NIC functional module, and TSN functional module; the TSN protocol processing module forms a complete TSN message for the ST frame generated by the TSN functional module after passing through SWITCH and sends it to the MAC module; the TSN protocol processing module is direct to RDMA and NIC.

[0021] The beneficial effects of the present invention are:

[0022] 1. The present invention proposes a multi-node communication architecture based on RDMA, which realizes multi-master multi-MAC communication on a single network card and integrates multiple network functions such as RDMA, NIC, TSN, etc. on a single network card.

[0023] 2. Realize multi-data channel aggregation and efficient parallel processing. The present invention is based on a multi-core CPU heterogeneous embedded supercomputing platform architecture, performs lossless Ethernet data communication, and has the characteristics of low latency and low CPU load. The present invention supports centralized and efficient data processing of large computing power nodes, can simultaneously process multiple concurrent received data on a single node, fully utilize the multi-core CPU parallel computing capabilities, and greatly reduce data node processing delays.

[0024] 3. Realize single-node data distribution and distributed processing. The present invention enables a single node to distribute data to multiple channels. Each channel has an independent sending and receiving buffer area. Multiple channels can send data in parallel at the same time, which improves throughput, reduces latency, and greatly improves the efficiency of data interaction between nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of the infrastructure of a multi-node communication architecture based on RDMA.

[0026] Figure 2 This is a framework diagram of the RDMA functional modules.

[0027] Figure 3 The figure is a schematic diagram of RDMA Send / Recv operation.

[0028] Figure 4 This is a schematic diagram of the RDMA Write operation.

[0029] Figure 5 This is a schematic diagram of the RDMA Read operation.

[0030] Figure 6 This is a structural diagram of the extended architecture 1.

[0031] Figure 7 This is a schematic diagram of the structure of extended architecture 2.

[0032] Figure 8 Schematic diagram of the internal distributed aggregation module.

[0033] Fig. 9 This is a structural diagram of extended architecture 3.

[0034] Fig.10 This is a structural diagram of extended architecture 4.

[0035] Fig.11 A network connection diagram for example. DETAILED DESCRIPTION

[0036] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0037] To improve the communication efficiency between multiple nodes, this embodiment shows a multi-node communication architecture based on RDMA. Figure 1 As shown, the infrastructure on a network card includes the following modules:

[0038] PCIe function module: supports PCIe x16 and is backward compatible, used for data interaction between the master control and RDMA function modules on the local node, uses official IP, and supports high-speed transmission and reception. As an example, the PCIe module uses the QDMA IP of the Xilinx manufacturer, supports PCIe Gen3.0x16, and the bandwidth can reach up to 128Gbps.

[0039] MAC module: uses high-bandwidth MAC, supports high-speed data transmission and reception between RDMA function modules and external nodes, supports FC / PFC flow control mechanism, supports 1588 time synchronization, etc., and is encapsulated as an AXI Stream interface. Official IP or self-developed IP can be used. As an example, the MAC module uses the 10G / 25G Ethernet Subsystem IP of Xilinx manufacturer, which supports 25Gbps MAC data transmission and reception.

[0040] RDMA function module: mainly implements the standard RoCEv2 protocol, supports unicast and multicast, supports reliable connection service (RC) and unreliable datagram service (UD), supports bilateral send / recv operations, supports unilateral write and read operations, can be implemented using FPGA, and the internal architecture is as follows Figure 2 shown.

[0041] The CQP_MGR module is the control channel, responsible for reading CQPE (Control Queue Pair Element) from the CQP (Control Queue Pairs) queue and parsing CQPE into hardware internal configuration instructions, thereby realizing the management of QP (Queue Pair) / CQ (Completion Queue) / CEQ (Completion Event Queue) / AEQ (Asynchronous Event Queue) queues.

[0042] The QP_MGR module is responsible for reading the WQEs (Work Queue Elements) in the SQ (Send Queue) and RQ (Receive Queue) queues to the WQE_MGR module according to the scheduling of the QP queue.

[0043] The WQE_MGR module supports SQ and RQ processing channels to jointly complete Send-Receive, Write, and Read operations.

[0044] The CQ_MGR module receives the completion status generated by the QP queue, encapsulates the completion status into CQE (CompletionQueue Element), fills it into the corresponding CQ queue, and uploads it to the software for processing.

[0045] The AEQ queue receives exception errors generated by processing from other modules and generates AE events according to the error type; the CEQ queue receives completion events from other queues, puts the received completion events into the CEQ queue, and generates an interrupt to notify the software to process the CE event.

[0046] HMC (Host Memory Cache) is a memory area with continuous physical addresses requested by the host. It is mainly used to store QP-related structure information and implement address conversion and Host Memory read and write operations by looking up the SD (Segment Descriptor) table.

[0047] The BTH module implements the encapsulation and decapsulation functions of Rocev2 data frames according to the Rocev2 protocol, and can support ipv4 and ipv6 data frame formats through configuration.

[0048] The RDMA function module establishes multicast links through Send / Recv operations, writes data to the remote CPU through RDMA Write operations, and reads data from the remote CPU through RDMA Read operations.

[0049] 1. RDMA Send / Recv Operation

[0050] 1) Preparation before sending: The initiator CPU prepares the data and stores it in the memory; the receiver CPU prepares the memory space for receiving data and sends RQE (RQ Element, receive queue unit) to the RDMA function module;

[0051] 2) Send request: The initiator CPU sends SQE (SQ Element, send queue unit) to the RDMA function module;

[0052] 3) Get the send data: The RDMA function module at the initiator parses the location of the payload data in the main memory according to the SQE, obtains the payload and composes the Send data packet;

[0053] 4) Send data packet: send data to the remote end through the physical link;

[0054] 5) Receiving Send Packet: After receiving the message, the receiving end takes out an RQE from the RQ, performs data verification and determines the data storage location, and returns ACK or NAK to the initiator. At this time, the initiator generates a CQE, indicating that the other end has received the data.

[0055] 6) Storing data: The RDMA function module at the receiving end stores the data in the corresponding location of the main memory according to the address parsed from the RQE;

[0056] 7) The receiving end CPU obtains the data.

[0057] 2. RDMA Write operation

[0058] 1) Preparation before sending: The initiator's CPU needs to prepare the data to be written to the peer's memory in its local memory;

[0059] 2) Send request: The initiator CPU initiates a doorbell (a doorbell is a notification mechanism. When ready, write data to the doorbell address, and the RDMA hardware will know that it can take the WQE from the QP and start working. The doorbell data is QP_ID) to instruct the RDMA hardware to send a packet;

[0060] 3) Parsing WQE: After receiving the doorbell, the RDMA hardware at the initiator obtains the WQE Buffer information stored in the corresponding WQ through the ID information in it, and then calculates the address of the WQE;

[0061] 4) Assemble and send messages: RDMA hardware stores the physical address according to the calculated data (calculated according to the mapping relationship), takes the data from the memory when sending the packet, and assembles the packet to send;

[0062] 5) Receive and parse the message: After receiving the message, the RDMA hardware on the responding end will parse and verify its content. For normal RDMA data requests, ACK is returned, and for abnormal ones, NAK is returned;

[0063] 6) Write the payload into the memory: After the verification is passed and the final PA is obtained, the RDMA function module can write the payload in the message into the destination memory through the DMA engine;

[0064] 7) The receiving end CPU obtains the data.

[0065] 3. RDMA Read Operation

[0066] 1) Preparation before sending: The responding end CPU needs to prepare data in local memory;

[0067] 2) Send request: The initiator initiates a doorbell to instruct the RDMA hardware to send a packet;

[0068] 3) Assemble and send messages: The RDMA function module parses the WQE to obtain the necessary information, and then assembles the packets and sends them out. The request messages sent out do not contain payloads, and do not need to access the local buffers.

[0069] 4) Receive and parse the message: After receiving the message, the RDMA function module on the responding end first checks whether the request parameters are legal; after the verification is completed, the data is taken out from the main memory to form a response data packet;

[0070] 5) Return data packet: send the response packet back to the initiator through the physical link;

[0071] 6) Write the payload into the memory: The RDMA function module at the initiator receives the response packet and writes the payload into the user buffer pointed to by the previous WQE.

[0072] 7) The initiator CPU obtains the data.

[0073] Based on the above infrastructure, the following expansion architecture can be carried out:

[0074] Expanded architecture 1

[0075] like Figure 6 As shown in the figure, an AIX-MUX module is set between one PCIe function module and four RMAD function modules, and a MAC module is set under the RMAD function module. The AIX-MUX module implements fair polling scheduling of four RDMAs, so that a single master can control four MAC nodes on a network card. The four RDMAs correspond one to one, and the fair polling of the AXI_SWITCH module sends data to the master through the PCIe function module or obtains it from the master.

[0076] This architecture can be further expanded by enabling the PF (Physical Function) function of PCIe, using 4 PFs, corresponding to 4 RDMAs and 4 MACs respectively, to achieve a one-to-one correspondence between PCIe, RDMA and MAC, which can be used for RDMA emulation cards.

[0077] Extended architecture 2

[0078] like Figure 7 As shown in the figure, a distributed aggregation module is configured in the RDMA function module, so that 1 RDMA function module corresponds to 1 PCIe and 4 MACs, supporting a single master to control 4 MAC nodes on a network card, greatly saving resources. The RDMA function module in this architecture is slightly different from the extended architecture 1. There is only one AXI STREAM channel on the top and 4 AXISTREAM channels on the bottom. The internal distributed aggregation module is as follows Figure 8 As shown, the RDMA message can be distributed to the four-way MAC according to specific information, or the corresponding RDMA message can be obtained from the four-way MAC to the PCLe functional module. As a workaround, the decentralized aggregation module can also be an independent module connected between a single RDMA functional module and the four-way MAC.

[0079] Extended architecture 3

[0080] like Fig. 9 As shown in the figure, combined with the AXI_MUX module in the extended architecture 1 and the distributed aggregation module in the extended architecture 2, 4 master controllers control 4 MAC nodes on one network card. 1 RDMA channel corresponds to 4 PCIe channels and 4 MAC channels.

[0081] Extended architecture 4

[0082] like Fig.10 As shown, an AXI_MUX module, an RDMA function module, a NIC function module, a TSN function module, a SWITCH module, and a TSN protocol processing module are configured between a group of PCIe function modules and MAC modules to realize the transmission of three protocol data between a single master control and a single MAC node.

[0083] Among them, the AXI_MUX module implements the formula polling scheduling between the RDMA function module, the NIC function module, and the TSN function module. The NIC function module mainly processes TCP and UDP related network protocols, and implements the unloading of network protocols such as TSO and USO; the RDMA function module mainly implements the unloading of the RDMA protocol; the TSN function module mainly implements the unloading of the UDP layer protocol. The SWITCH module implements fair polling scheduling and priority processing among the three function modules of the RDMA function module, the NIC function module, and the TSN function module. Since the ST frame generated by the TSN function module has strict requirements on the sending time point, it needs to pass through the TSN protocol processing module after SWITCH to implement TSN protocols such as QaV and TAS, and then form a complete TSN message to send to the MAC module; the TSN protocol processing module is direct for RDMA and NIC.

[0084] Through multiple such combinations, multiple master controllers can control multiple MAC nodes on one network card to perform data transmission of multiple protocols.

[0085] It should be noted that the extended architectures 1 to 4 can be extended into a variety of combined architectures, including but not limited to these architectures, all of which are within the scope of the present invention.

[0086] by Fig.11 The network structure shown takes RDMA Write as an example and combines the extended architecture 2 to illustrate the functions implemented by the present invention.

[0087] Step 1: The local node uses UD multicast to establish CM links with four external nodes and agree on the QP numbers to be used respectively.

[0088] Step 2: The four external nodes prepare the data they need to send to the local node and store them in their respective memories;

[0089] Step 3: The four-way external node software notifies the CQP queue processing by writing the doorbell register. The CQP_MGR module is responsible for reading CQPEs from the CQP queue and parsing the CQPE into hardware internal configuration instructions. The software writes registers cqphigh and cqplow in sequence to inform the CQP_MGR module to start creating and enabling the CQP queue.

[0090] Step 4: After the CQP queue in the four-way remote RDMA function module is successfully created, when the software has configuration instructions to be sent, the configuration instructions are filled into the CQP queue in the format of CQPE, and then the CQP queue is notified to process by writing the doorbell register. CQP reads the CQPEs in the CQP queue base address through the DMA interface, parses them into configuration commands one by one, and configures them to other modules.

[0091] Step 5: After receiving the doorbell, the four-way remote RDMA hardware obtains the QPN and WQE index through the information in it, and then calculates the WQE address based on the WQE Buffer information recorded in it.

[0092] Step 6: The four-way remote RDMA hardware SQ channel reads the data payload of the message and the corresponding packet information info from the host memory to the lower-level packet processing module, assembles the packet according to the RoCEv2 frame format, and sends it out through the physical link.

[0093] Step 7: The local node receives the RDMA Write data frames sent by the four external nodes at the same time, and parses and verifies their contents in their respective RDMA function modules. ACK is returned for normal RDMA data, and NAK is returned for abnormal RDMA data.

[0094] Step 8: The local node receives four 25 Gbps data frames and writes them into the memory in turn through PCIe through fair polling.

[0095] Step 9: The local node host reads the data in the memory and performs efficient parallel processing. It can process the data written by four external nodes at the same time.

[0096] It is understandable that those skilled in the art can make equivalent substitutions or changes based on the technical solution and inventive concept of the present invention, and all these changes or substitutions should fall within the protection scope of the claims attached to the present invention.

Claims

1. A multi-node communication architecture based on RDMA, characterized in that The following modules are included on a network card: PCIe function module: supports PCIe x16 and is backward compatible. It is used for data interaction between the master control and RDMA function modules on the local node and supports high-speed transmission and reception. MAC module: uses high-bandwidth MAC to support high-speed data transmission and reception between the RDMA function module and external nodes; RDMA function module: implements the standard RoCEv2 protocol, supports unicast and multicast, supports reliable connection services and unreliable datagram services, supports bilateral send / recv operations, supports unilateral write and read operations, and the internal architecture is as follows: The CQP_MGR module is used to control the channel. It is responsible for reading CQPE from the CQP queue and parsing CQPE into hardware internal configuration instructions to achieve QP / CQ / CEQ / AEQ queue management. The QP_MGR module is responsible for reading the WQEs in the SQ and RQ queues to the WQE_MGR module according to the scheduling of the QP queue; The WQE_MGR module is used to support the SQ and RQ processing channels and complete the Send-Receive, Write, and Read operations together; The CQ_MGR module is used to receive the completion status generated by the QP queue, encapsulate the completion status into CQE, fill it into the corresponding CQ queue, and upload it to the software for processing; The BTH module is used to implement the encapsulation and decapsulation functions of Rocev2 data frames according to the Rocev2 protocol.

2. According to claim 1, a multi-node communication architecture based on RDMA is characterized in that: An AIX-MUX module is set between a PCIe function module and multiple RMAD function modules, and a MAC module is set under the RMAD function module. The AIX-MUX module implements fair polling scheduling of multiple RDMAs, so that a single master controls multiple MAC nodes on a network card.

3. According to claim 2, a multi-node communication architecture based on RDMA is characterized in that: The PCIe functional module also supports the PF function, using multiple PFs, corresponding to one RDMA and one MAC respectively, to achieve a one-to-one correspondence between PCIe, RDMA and MAC.

4. According to claim 1, a multi-node communication architecture based on RDMA is characterized in that: A distributed aggregation module is configured in the RDMA function module. The RDMA function module is connected to multiple MAC modules. The distributed aggregation module distributes the RDMA message to the specified MAC according to specific information, or obtains the corresponding RDMA message from the MAC to the PCLe function module, so as to realize a single master control of multiple MAC nodes on one network card.

5. The multi-node communication architecture based on RDMA according to claim 1, characterized in that: An AIX-MUX module is set between a PCIe functional module and multiple RMAD functional modules, and the AIX-MUX module implements fair polling scheduling for multiple RDMAs; a distributed aggregation module is configured in the RDMA functional module, and the RDMA functional module is connected to multiple MAC modules. The distributed aggregation module distributes RDMA messages to the specified MAC according to specific information, or obtains the corresponding RDMA message from the MAC to the PCLe functional module, so that one RDMA corresponds to multiple PCIe and multiple MACs, and multiple master controllers control multiple MAC nodes on one network card.

6. The multi-node communication architecture based on RDMA according to claim 1, characterized in that: An AXI_MUX module, an RDMA function module, a NIC function module, a TSN function module, a SWITCH module, and a TSN protocol processing module are configured between a group of PCIe function modules and a MAC module to realize the transmission of three protocol data between a single master control and a single MAC node; Among them, the AXI_MUX module implements the formula polling scheduling among the RDMA functional module, the NIC functional module, and the TSN functional module; the NIC functional module processes TCP and UDP related network protocols; the TSN functional module mainly implements the UDP layer protocol unloading; the SWITCH module implements fair polling scheduling and priority processing among the three functional modules of RDMA functional module, NIC functional module, and TSN functional module; the TSN protocol processing module forms a complete TSN message for the ST frame generated by the TSN functional module after passing through SWITCH and sends it to the MAC module; the TSN protocol processing module is direct to RDMA and NIC.

Images