ROCE communication and transmission method based on non-HOST main memory

Abstract

The invention discloses a ROCE communication and transmission method based on a non-HOST main memory, and the method comprises the steps: employing an FPGA as a ROCE network card, enabling the FPGA network card to be connected with a CPU through a PCIe bus, and externally expanding four groups of DDRs through a BANK resource in the FPGA network card; and meanwhile, four paths of GbE ports are instantiated by using GbE MAC / PHY resources in the FPGA network card and are used for RDMA communication, and the CPU runs the RDMA. Compared with a traditional design method, the method can support a multi-path 100GbE data distribution scene, solves the problem that the maximum bandwidth upper limit is insufficient, expands the DDR space of a CPU and the number of memory channels, constructs a multi-channel DDR scene of simultaneous reading and writing, and improves the efficiency of concurrent execution.

Description

Technical Field

[0001] The present invention relates to the technical field of high-speed data communication, and in particular to a ROCE communication and transmission method based on non-HOST main memory. Background Art

[0002] ROCE is a network protocol defined in the InfiniBand Trade Association standard that allows the use of RDMA over Ethernet. ROCE v2, as the most scalable and high-performance transmission protocol in RDMA, is being widely used in various processing scenarios. It uses Kernel Bypass and Zero Copy technology to provide low latency features, while reducing CPU usage, reducing memory bandwidth bottlenecks, and providing high bandwidth utilization. The traditional ROCE implementation architecture is as follows Figure 1 As shown in the figure, in the traditional ROCE architecture, the CPU sends WQE to the ROCE network card. After receiving the control instruction, the network card parses the transmission type, data address, transmission length, access permission information, and obtains the main memory data mounted on the CPU side through the address conversion table to perform unilateral and bilateral communication. This design architecture ROCE network card moves and transmits data for the Host main memory, which has very high versatility.

[0003] However, these measures currently adopted contain at least the following three problems: 1) High cost: For application scenarios where data is distributed after being received, especially one-to-many communication, the received data does not need to be processed by the CPU, but needs to be distributed as quickly as possible to maximize the use of the underlying transmission bandwidth resources. Distribution after entering the CPU will waste a lot of CPU and PCIE bandwidth. 2) Low efficiency; this is because considering the impact of localization, the PCIE channel between the ROCE network card and the CPU is limited. If it is GEN3.0, according to PCIe3.0×16 calculation, the maximum theoretical bandwidth is 128Gbps, which can only meet the transmission bandwidth requirements of 1-way 100GbE, far from meeting the transmission requirements of multiple-way 100GbE; 3) Poor adaptability: For current domestically produced CPUs, including FT2000 / 4, FTD2000 / 8 and FTD3000, there are only two groups of DDR controllers, and the number of memory channels cannot be expanded, resulting in a single-point bottleneck when performing concurrent read and write operations, and it cannot adapt well to high-performance scenarios of concurrent transmission and reading and writing. Summary of the invention

[0004] In view of the above three problems, the purpose of the present invention is to propose a ROCE communication and transmission method based on non-HOST main memory. By adopting an FPGA as a ROCE network card and expanding 4 groups of 4GB of DDR, the traditional ROCE communication and transmission method can overcome the problem of not being able to adapt well to high-performance scenarios of concurrent transmission and reading and writing. The FPGA network card uses 100GbE MAC / PHY resources to instantiate 4 100GbE ports, which can support multi-channel 100GbE data distribution scenarios and solve the problem of insufficient maximum bandwidth limit.

[0005] This is achieved through the following technical solutions: A ROCE communication and transmission method based on non-HOST main memory adopts an FPGA as a ROCE network card, the FPGA network card and the CPU are interconnected by a PCIe bus, and four groups of DDR are expanded through the BANK resources inside the FPGA network card; at the same time, the FPGA network card uses GbE MAC / PHY resources to instantiate four GbE ports for RDMA communication, and the CPU runs RDMA, wherein the data communication process of the method includes the following steps: S1, the request end software adds a WQE task to the SQ queue according to the application trigger, and then the request end of the FPGA network card parses the WQE task to obtain the virtual address of the source data and the virtual address of the destination data, the data length, the key and the operation code; by using the operation code, the request end of the FPGA network card queries the MTT table according to the virtual address of the source data and compares the storage domain address with the FPGA The conversion between DDR addresses obtains the physical address of the data stored in the FPGA network card's external DDR; the data to be sent is then copied to the FPGA network card for storage through DMA operation, and the data is encapsulated according to the protocol to obtain the corresponding data packet; S2, the request end of the FPGA network card sends the data packet to the GbE After the MAC / PHY sending interface, the data packet is sent to the responding end of the FPGA network card through the sending interface and parsed to obtain the virtual address, data length and key where the application data and the destination data are stored, and then the MR address conversion table is queried according to the virtual address and data length where the destination data is stored to obtain the corresponding physical address, and then the application data is written to the physical address for storage, and the application data is decrypted by the key; S3, after the responding end of the FPGA network card receives the data packet sent by the requesting end of the FPGA network card, it replies with an ACK message to the requesting end of the FPGA network card. After receiving the ACK message, the requesting end of the FPGA network card sends the ACK message to the message parsing module of the RDMA for parsing and obtaining the parsing result. The requesting end of the FPGA network card identifies the ACK based on the parsing result, and then adds the completion indication element corresponding to the WQE task to the CQ queue. When the requesting end software polls the completion indication element, it will obtain the task completion information and inform the application of the result of this transmission.

[0006] Aiming at the limitations of traditional ROCE communication and transmission methods, the present invention proposes a ROCE communication and transmission method based on non-HOST main memory, which can support multi-channel 100GbE data distribution scenarios, solve the problem of insufficient maximum bandwidth upper limit, expand the DDR space and memory channel number of the CPU, construct a multi-channel DDR scenario of simultaneous reading and writing, and improve the efficiency of concurrent execution.

[0007] Preferably, in step S1, the MTT table is used to manage the data structure of the memory page in the RDMA transmission, and provides a mapping mechanism from the virtual address to the physical address. By querying the MTT table, the conversion between the storage domain address and the FPGA DDR address can be performed quickly and efficiently to obtain the physical address of the data stored in the FPGA network card external DDR.

[0008] Preferably, in step S1, when the storage domain address and the FPGA DDR address are converted, the address mapping between the FPGA network card external DDR space and the CPU bus domain is completed by allocating BAR space by the CPU in the PCIe scanning phase; wherein the CPU completes the access control of the FPGA network card external DDR in the PCIe bus space by directly accessing the mapping address of the FPGA network card in the storage space. By expanding 4 groups of 4GB of DDR by the FPGA network card, the DDR space and memory channel number of the CPU are expanded, and a multi-channel DDR scenario of simultaneous reading and writing is constructed, which improves the efficiency of concurrent execution.

[0009] Preferably, when converting the storage domain address and the FPGA DDR address, a Host main bridge needs to be set between the storage domain address and the PCIe bus domain, wherein the Host main bridge is used to map the address space of the CPU bus domain to the address space of the PCIe bus, realize the communication between the CPU and the PCIe bus, and is responsible for the data transmission between the CPU and the PCIe bus. The communication between the CPU and the PCIe bus is completed through the Host main bridge, realizing the CPU's access to the PCIe bus address space.

[0010] Preferably, when allocating BAR space, based on the PCIe BAR size of the CPU, the FPGA network card can expand the DDR space by using a window switching mechanism; when DDR is 16G, the FPGA network card uses a window switching mechanism to divide the 16GB DDR into 64 256MB BAR subspaces; wherein, the 64-bit PCIe address mapped by the Host main bridge is used to take the lower 34 bits, of which 0 to 27 bits are used to determine the lower address of the DDR, and 28 to 33 bits are used to indicate the subspace currently accessed. Through the window switching mechanism, the CPU can access the entire DDR space of the FPGA network card.

[0011] Preferably, in step S2, the MR address translation table is used to perform memory management in RDMA and record the address translation relationship of the memory; after the responding end of the FPGA network card obtains the virtual address where the destination data is stored, the MR address translation table is used to obtain the corresponding physical address. The MR address translation table can accurately and efficiently complete the correct mapping of the virtual address to the physical address, thereby realizing process isolation and memory protection.

[0012] Compared with the prior art, the present invention has the following beneficial effects: The technical solution of the present invention adopts a high-performance FPGA as a ROCE network card, expands 4 groups of DDR with a size of 4GB, and uses 100GbE MAC / PHY resources inside the FPGA network card to instantiate 4 100GbE ports, which can maximize the use of underlying transmission bandwidth resources, improve system throughput bandwidth, reduce PCIe data movement operations, and effectively reduce transmission delay. It is suitable for real-time sensitive transmission scenarios. At the same time, it meets the demand for communication bandwidth in multi-channel parallel transmission scenarios, expands the DDR space and memory channel number of the CPU, constructs a multi-channel DDR scenario of simultaneous reading and writing, and improves the efficiency of concurrent execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The schematic diagram of the architecture design of a ROCE communication and transmission method based on non-HOST main memory; Figure 2 A schematic diagram of address conversion between a memory domain and a PCIe bus domain in a ROCE communication and transmission method based on non-HOST main memory; Figure 3 A data interaction diagram of a ROCE communication and transmission method based on non-HOST main memory; Figure 4 The figure is a flow chart of RDMA data communication of ROCE communication and transmission method based on non-HOST main memory. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0015] like Figure 1The figure shows an architectural design diagram of a ROCE communication and transmission method based on non-HOST main memory. A high-performance FPGA is used as a ROCE network card. The FPGA network card and the CPU are interconnected using PCIe3.0×16. Through the BANK resources inside the FPGA network card, 4 groups of DDR are expanded. Each group of DDR is 4GB in size and has a frequency of 2666MT / s. At the same time, the FPGA network card uses 100GbE MAC / PHY resources instantiate multiple 100GbE ROCE communication ports, including Port0, Port1, Port2 and Port3, for RDMA communication, and the CPU runs RDMA; RDMA is a network communication technology that allows the memory of a computer system to directly access the memory of other computer systems without the involvement of the CPU. This communication technology can reduce communication latency and CPU load, and improve the efficiency of network communication; the BANK resources inside the FPGA network card are obtained by dividing the input and output pins of the FPGA into several groups. The interface standard of each group is determined by its interface voltage VCCO, and each group has only one interface voltage VCCO and can be different. This division can make the FPGA easy to manage and adapt to multiple electrical standards, improving the flexibility and applicability of the FPGA; 100GbE MAC / PHY refers to the media access MAC controller and physical layer chip used for 100G Ethernet communication, and 100GbE MAC / PHY mainly includes a MAC controller and a physical layer chip.

[0016] like Figure 4 As shown, it is an RDMA data communication flow chart of the ROCE communication and transmission method based on non-HOST main memory. During RDMA data communication, a WQE task is added to the SQ queue through software driver. After the request end of the FPGA network card parses the WQE task and obtains data information, it encapsulates the data information according to the protocol and obtains the corresponding data packet; the request end of the FPGA network card sends the data packet to the responder of the FPGA network card and parses it. After obtaining the corresponding data information, the data is written into the physical address for storage; after receiving the data packet sent by the request end of the FPGA network card, the responder of the FPGA network card replies an ACK message to the request end of the FPGA network card and parses the ACK message. The request end of the FPGA network card informs the application of the parsing result.

[0017] The data communication process of the method specifically includes the following steps: S1. The request end of the FPGA network card adds a WQE task to the SQ queue according to the application trigger, and identifies the virtual address of the data stored in the DDR and the virtual address of the destination data, data length, key and operation code in the WQE task; then the WQE task is taken out from the SQ queue through the request end of the FPGA network card and the WQE task is parsed to obtain the virtual address of the source data and the virtual address of the destination data, data length, key and operation code; by using the operation code, the request end of the FPGA network card queries the MTT table and converts the storage domain address and the FPGA DDR address according to the virtual address of the source data, and the CPU directly accesses the mapping address of the FPGA network card in the storage space to complete the access control of the FPGA network card external DDR in the PCIe bus space, and obtains the physical address of the data stored in the FPGA network card external DDR; then the data to be sent is copied to the inside of the FPGA network card for storage through the DMA operation, and the data is encapsulated according to the protocol to obtain the corresponding data packet. Among them, DDR space refers to the storage space provided by the double-rate synchronous dynamic random access memory, which is used to store data. By performing two data transmissions in one clock cycle, the data transmission rate is improved; SQ queue refers to the sending queue, which is used to store data packets to be sent. In RDMA data communication, each data channel has a pair of queues at both ends, including the SQ queue; WQE task refers to the work queue element, which is mainly used to describe work requests in RDMA data communication; DMA operation refers to direct memory access operation, which is a data transmission method in computer systems and can improve the efficiency of data transmission.

[0018] like Figure 2 As shown, it is a schematic diagram of address conversion between the memory domain and the PCIe bus domain in a ROCE communication and transmission method based on non-HOST main memory. When the memory domain address is converted with the FPGA DDR address, the CPU directly accesses the mapping address of the FPGA network card in the memory domain to complete the access control of the external DDR of the FPGA network card in the PCIe bus domain, that is, the CPU bus domain address - memory domain address - PCIe bus domain address.

[0019] The CPU first directly accesses the mapping address of the FPGA network card in the memory domain, and then sets the Host main bridge between the storage domain address and the PCIe bus domain to obtain the space address of the FPGA network card external DDR in the PCIe bus domain. Then the CPU controls the access to the FPGA network card external DDR. The memory domain address includes: the mapping address of the BAR space in the PCIe bus, the mapping address of the FPGA network card and the Host main memory address space. The PCIe bus domain is divided into 16GB of FPGA DDR space and BAR space of other devices. The BAR space refers to the address space mapped by the base address register in the PCIe bus domain. Each PCIe bus domain has multiple BAR spaces for communication between the PCIe bus and the system software.

[0020] like Figure 3 The figure shows a data interaction diagram of a ROCE communication and transmission method based on non-HOST main memory. When the storage domain address and the FPGA DDR address are converted, data interaction is also performed. Through the input channel, management channel, message channel and data channel, the input and analysis of external data are completed and the corresponding events are generated.

[0021] External data enters the RDMA subsystem through the input channel. After the external data is processed by the protocol in the RDMA subsystem, the corresponding raw data is obtained. The raw data will be stored in the external DDR on the FPGA network card through the AXi-MM data channel. At the same time, the address, length and other information of the data currently stored in the DDR will be notified to the CPU through the AXI-Lite management channel. At the same time, considering that some external input data carries slightly larger data such as instructions and control tables, and this part needs to be passed to the CPU for further processing, an AXI-Stream message channel is added. After the RDMA subsystem extracts the instruction and control table data, it is transmitted to the CPU through the AXI-Stream message channel. The CPU obtains the instruction and control table data through the driver interface and parses it, and guides the application to generate corresponding events based on the parsing results. Among them, the AXi-MM data channel, the AXI-Stream message channel and the AXI-Lite management channel are all a data transmission method in the AXi bus. The AXi-MM data channel is mainly used for high-performance and large-volume data transmission. The AXI-Stream message channel has no address stage when performing data transmission, which is suitable for streaming data transmission. The AXI-Lite management channel is mainly used for small-batch data transmission.

[0022] In this embodiment, the address mapping between the external DDR space of the FPGA network card and the CPU bus domain needs to be completed by the CPU in the PCIe scanning phase by allocating the BAR space; when allocating the BAR space, based on the PCIe BAR size of the CPU, the external DDR space of the FPGA network card is realized by adopting the window switching mechanism, and the 16GB DDR is divided into 64 256MB BAR subspaces, and the 64-bit PCIe address mapped by the Host main bridge is used to take the lower 34 bits, of which 0~27 bits are used to determine the low address of the DDR, and 28~33 bits are used to indicate the subspace currently accessed. Expanding the DDR space and memory channel number of the CPU, building a multi-channel DDR scenario of simultaneous reading and writing, improving the efficiency of concurrent execution, and completing the address mapping of the CPU bus domain enables the CPU to access the external DDR of the FPGA network card in the PCIe bus space; at the same time, through the window switching mechanism, the CPU can access the entire DDR space of the FPGA network card.

[0023] In this embodiment, the MTT table is used to manage the data structure of the memory page in the RDMA transmission, and provides a mapping mechanism from the virtual address to the physical address. By querying the MTT table, the conversion between the storage domain address and the FPGA DDR address can be performed quickly and efficiently to obtain the physical address of the data stored in the FPGA network card external expansion DDR.

[0024] In this embodiment, when the storage domain address and the FPGA DDR address are converted, a Host main bridge needs to be set between the storage domain address and the PCIe bus domain. The Host main bridge is used to map the address space of the CPU bus domain to the address space of the PCIe bus, realize the communication between the CPU and the PCIe bus, and is responsible for the data transmission between the CPU and the PCIe bus. Among them, the non-HOST main memory in the present invention means that the HOST is not used as the main memory, the CPU cannot directly access it, and it has different functions and roles in the system from the Host main bridge.

[0025] S2. After the request end of the FPGA network card sends the data packet to the 100GbE MAC / PHY sending interface, the data packet is sent to the responder end of the FPGA network card through the sending interface and parsed to obtain the virtual address, data length and key where the application data and the destination data are stored, and then the MR address conversion table is queried according to the virtual address and data length where the destination data is stored, and the virtual address where the destination data is stored is accurately mapped to the physical address through the MR address conversion table to obtain the corresponding physical address, and then the application data is written to the physical address for storage, and the application data is decrypted by the key. Among them, 100GbEMAC / PHY refers to the media access MAC controller and physical layer chip used for 100G Ethernet communication, and 100GbE MAC / PHY mainly includes MAC controller and physical layer chip, MAC controller is responsible for sending data to physical layer chip, physical layer chip is responsible for speed negotiation and conversion of digital signal to analog signal, and outputs the signal to the network cable.

[0026] In this embodiment, the MR address translation table is used to manage memory in RDMA and record the address translation relationship of the memory; after the responding end of the FPGA network card obtains the virtual address where the destination data is stored, it uses the MR address translation table to obtain the corresponding physical address. The MR address translation table can accurately and efficiently complete the correct mapping of virtual addresses to physical addresses, thus achieving process isolation and memory protection.

[0027] S3, after the responding end of the FPGA network card receives the data packet sent by the requesting end of the FPGA network card, it replies with an ACK message to the requesting end of the FPGA network card. After receiving the ACK message, the requesting end of the FPGA network card sends the ACK message to the message parsing module of the RDMA. The ACK message header contains multiple fields, including: source port number, destination port number, sequence number and confirmation number, etc. The message parsing module of the RDMA will parse these fields one by one and obtain the parsing results. Then the message parsing module of the RDMA will return the parsing results to the requesting end of the FPGA network card. The requesting end of the FPGA network card identifies the ACK based on the parsing results, and then adds the completion mark element corresponding to the WQE task to the CQ queue. When the requesting end software polls the completion mark element, it will obtain the task completion information and inform the application of the result of this transmission. Among them, the CQ queue refers to the completion queue, which is used to store the notification of the completion of the sending and receiving operations. A CQ queue is required at both ends of each data channel.

[0028] In this embodiment, the ACK message is a confirmation message, which is sent by the responding end of the FPGA network card to the requesting end of the FPGA network card, and is used to confirm that the requesting end of the FPGA network card has successfully received the data packet, prevent data loss or resending, and ensure reliable data transmission.

[0029] In this embodiment, the RDMA message parsing module is used to receive the ACK message sent by the requesting end of the FPGA network card, and parse the fields in the ACK message one by one, extract useful data and information, and perform further analysis and processing to ensure the order of data transmission and the integrity and correctness of the data.

[0030] In summary, the present invention adopts a high-performance FPGA as a ROCE network card, expands 4 groups of DDR with a size of 4GB, and uses 100GbE MAC / PHY resources inside the FPGA network card to instantiate 4 100GbE ports, which can maximize the use of underlying transmission bandwidth resources, improve system throughput bandwidth, reduce PCIe data movement operations, and effectively reduce transmission delay. It is suitable for real-time sensitive transmission scenarios. At the same time, it meets the demand for communication bandwidth in multi-channel parallel transmission scenarios, expands the DDR space and memory channel number of the CPU, constructs a multi-channel DDR scenario of simultaneous reading and writing, and improves the efficiency of concurrent execution. It has significant progress.

[0031] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A ROCE communication and transmission method based on non-HOST main memory, characterized in that: An FPGA is used as a ROCE network card. The FPGA network card and the CPU are interconnected by a PCIe bus. Four groups of DDR are expanded through the BANK resources inside the FPGA network card. At the same time, the GbE MAC / PHY resources inside the FPGA network card are used to instantiate four GbE ports for RDMA communication. The CPU runs RDMA. The data communication process of the method includes the following steps: S1. The request end software adds a WQE task to the SQ queue according to the application trigger, and then the request end of the FPGA network card parses the WQE task to obtain the virtual address where the source data is stored and the virtual address where the destination data is stored, the data length, the key and the operation code; by using the operation code, the request end of the FPGA network card queries the MTT table and converts the storage domain address and the FPGA DDR address according to the virtual address where the source data is stored, and obtains the physical address where the data is stored in the external DDR of the FPGA network card; then the data to be sent is copied to the internal FPGA network card for storage through DMA operation, and the data is encapsulated according to the protocol to obtain the corresponding data packet; S2. After the request end of the FPGA network card sends the data packet to the GbE MAC / PHY sending interface, the data packet is sent to the responder of the FPGA network card through the sending interface and parsed to obtain the virtual address, data length and key where the application data and the destination data are stored, and then the MR address conversion table is queried according to the virtual address and data length where the destination data is stored to obtain the corresponding physical address, and then the application data is written to the physical address for storage, and the application data is decrypted by the key; S3. After the responding end of the FPGA network card receives the data packet sent by the requesting end of the FPGA network card, it replies with an ACK message to the requesting end of the FPGA network card. After receiving the ACK message, the requesting end of the FPGA network card sends the ACK message to the message parsing module of the RDMA for parsing and processing to obtain the parsing result. The requesting end of the FPGA network card identifies the ACK based on the parsing result, and then adds the completion indication element corresponding to the WQE task to the CQ queue. When the requesting end software polls the completion indication element, it will obtain the task completion information and inform the application of the result of this transmission.

2. According to claim 1, a ROCE communication and transmission method based on non-HOST main memory is characterized in that: In step S1, the MTT table is used to manage the data structure of the memory page in the RDMA transmission and provide a mapping mechanism from the virtual address to the physical address.

3. The ROCE communication and transmission method based on non-HOST main memory according to claim 1 is characterized in that: In step S1, when the storage domain address and the FPGA DDR address are converted, the address mapping between the FPGA network card external expansion DDR space and the CPU bus domain is completed by allocating BAR space by the CPU in the PCIe scanning phase; wherein, the CPU completes the access control of the FPGA network card external expansion DDR in the PCIe bus space by directly accessing the mapping address of the FPGA network card in the storage space.

4. A ROCE communication and transmission method based on non-HOST main memory according to claim 3, characterized in that: When converting the storage domain address and the FPGA DDR address, a Host bridge is set between the storage domain address and the PCIe bus domain. The Host bridge is used to map the address space of the CPU bus domain to the address space of the PCIe bus to achieve communication between the CPU and the PCIe bus.

5. The ROCE communication and transmission method based on non-HOST main memory according to claim 3 is characterized in that: When allocating BAR space, based on the PCIe BAR size of the CPU, a window switching mechanism is used to implement external DDR space expansion of the FPGA network card.

6. A ROCE communication and transmission method based on non-HOST main memory according to claim 5, characterized in that: DDR uses 16G, and the window switching mechanism is used in the FPGA network card to divide the 16GB DDR into 64 256MB BAR subspaces; among them, the 64-bit PCIe address mapped by the Host main bridge is used to take the lower 34 bits, of which 0~27 bits are used to determine the low-order address of DDR, and 28~33 bits are used to indicate the subspace currently accessed.

7. The ROCE communication and transmission method based on non-HOST main memory according to claim 1 is characterized in that: In step S2, the MR address translation table is used to perform memory management in RDMA and record the address translation relationship of the memory; after the responding end of the FPGA network card obtains the virtual address where the destination data is stored, it uses the MR address translation table to obtain the corresponding physical address.

Images