High bandwidth low latency data processing method based on ROCEv2

Abstract

The application discloses a high-bandwidth low-delay data processing method based on ROCEv2, relates to the technical field of data processing, and is realized based on a ROCEv2 protocol and comprises a host server and a chip. The host server comprises a mapping table of a system memory and a virtual address for each queue pair. The chip comprises a ROCEv2 protocol processing interface, an address resource FIFO, a channel manager, a DMA engine and multi-channel service data packets. The application uses less hardware resources of system devices, thereby reducing hardware cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a high-bandwidth and low-latency data processing method based on ROCEv2. BACKGROUND

[0002] In various high-speed data acquisition and processing systems, the requirements for transmission bandwidth performance and low-latency response are increasingly high, and the scalability of system devices is also increasingly high. Using network technology for distributed processing and storage of high-speed data signals is obviously a very suitable means.

[0003] The ROCEv2 technology offloads data transmission by a dedicated DMA engine without the participation of the operating system, and its performance is greatly improved compared with traditional TCP / UDP network technology, and ROCEv2 can utilize existing network routing equipment. Compared with the dedicated InfiniBand device, the cost of the ROCEv2 technology is much lower. Although there are the above benefits, ROCEv2 is a relatively complex technology that requires more software and hardware support services.

[0004] Usually, when ROCEv2 does two-end DMA interaction, data needs to be written into DDR cache for two-end address space mapping management, and the data needs to pass through the cache throughput. At this time, whether it is a remote RDMA READ or a local RDMA WRITE, a large-capacity high-speed data cache is required. In a high-speed data acquisition and processing system, this high-speed data cache area is prone to become a bottleneck of system performance.

[0005] In the face of multi-service high-bandwidth concurrency, multiple QPs (Queue Pair) are usually established to manage the RDMA transmission context, which further complicates the solution and makes it difficult to reconcile the contradiction between high efficiency and low cost.

[0006] Especially in FPGA / ASIC solutions, the FPGA / ASIC chip must be externally connected with a large-capacity high-performance DDR chip. This is very disadvantageous to single-board design, hardware cost control, and solution response delay. SUMMARY

[0007] To solve the problems in the prior art, the purpose of the present application is to provide a high-bandwidth and low-latency data processing method based on ROCEv2, which uses fewer hardware resources of system devices and reduces hardware costs.

[0008] In order to achieve the above object, the technical scheme adopted by the present application is: a high-bandwidth and low-delay data processing method based on ROCEv2, comprising a host server and a chip, the host server comprising a mapping table of a system memory and a virtual address for each queue pair, and the chip comprising an ROCEv2 protocol processing interface, an address resource FIFO, a channel manager, a DMA engine and a multi-channel service data packet; the method comprises the following steps:

[0009] Step 1: the ROCEv2 protocol processing interface establishes a connection with the ROCEv2 network card of the host server, the host server maps discrete memory resources into the DMA virtual address space of the queue pair, and sends the discrete memory resources to the ROCEv2 protocol processing interface of the chip;

[0010] Step 2: after the ROCEv2 protocol processing interface receives the sending packet of the host server, the discrete memory resources are parsed, and entry information is constructed to describe the virtual address and length information, and after the construction, the entry is written into the address resource FIFO;

[0011] Step 3: the address resource FIFO stores the entry, and after receiving the entry application of the channel manager, an entry is taken out from the address resource FIFO and given to the channel manager to complete an address application operation;

[0012] Step 4: after the channel manager applies for the entry, the virtual address and length information are loaded into the register circuit and the DMA context is maintained for the use of the DMA engine, when a DMA length transmission is completed or an interrupt is required, a completion entry is generated, and the address information and the completion data length information are sent to the ROCEv2 protocol interface;

[0013] Step 5: the DMA engine switches to the DMA context information of the corresponding channel manager according to the channel information of the multi-channel service data packet, encapsulates the virtual address, length information and data frame through the ROCEv2 protocol processing interface, and sends them to the host server;

[0014] Step 6: the host server parses the virtual address and length information, queries the mapping table to obtain the actual address information, and pushes the actual address and length information to the application program.

[0015] As a further improvement of the present application, in step 1, the host server also manages the state and recycling of the discrete memory resources.

[0016] As a further improvement of the present application, in step 4, when the channel manager detects a DMA length completion, or the service data packet carries an interrupt flag, the channel manager ends the current DMA after the DMA engine packetizing; meanwhile the channel manager applies for a new entry to the address resource FIFO, and the DMA context is updated.

[0017] As a further improvement of the present application, the address resource FIFO in the FPGA is mapped to the resources of the same queue pair, so that different service channels run using one queue pair; the data packets of different channel services are assigned different virtual addresses for access by the channel manager.

[0018] As a further improvement of the present application, the chip is an FPGA chip or an ASIC chip.

[0019] The present application has the following beneficial effects:

[0020] The present application is based on the ROCEv2 protocol, and proposes a high-bandwidth low-latency data processing method based on ROCEv2, which can efficiently utilize the inherent advantage of high bandwidth of ROCEv2, avoid large data packet caching; does not depend on external cache, and can meet the demand of high bandwidth using on-chip cache of FPGA / ASIC, service data packets can be transmitted and interrupted in host in time at any time, reduce response delay, improve system response speed, and use less hardware resources of system equipment, which can reduce hardware cost. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 Fig. 1 is a method block diagram for implementing the chip internally in the embodiment of the present application;

[0022] Figure 2 Fig. 2 is a structure schematic diagram of the entry in the embodiment of the present application;

[0023] Figure 3 Fig. 3 is a structure schematic diagram of the completed entry in the embodiment of the present application;

[0024] Figure 4 Fig. 4 is a work flow diagram of the channel manager in the embodiment of the present application;

[0025] Figure 5 Fig. 5 is a network packet interaction schematic diagram in the embodiment of the present application. DETAILED DESCRIPTION

[0026] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0027] EMBODIMENT

[0028] As shown in Fig. 1, a high-bandwidth low-latency data processing method based on ROCEv2, Figure 1 Figure 1 ​In the embodiment, in addition to the green color being an existing general technology module in the industry, the rest is the invention content of the embodiment, and the host server side is a server with an ROCEv2 network card. The main modules of the FPGA / ASIC chip are an ROCEv2 protocol processing interface module, an address resource FIFO, a channel manager, a DMA engine, and a multi-channel service data packet, allowing the data packet to carry an interrupt flag setting, so as to obtain a low-delay response of the host; the method of the embodiment specifically includes:

[0029] The ROCEv2 protocol processing interface realizes the ROCEv2 standard protocol interface function, which is realized on the FPGA / ASIC chip; it is responsible for the establishment of a connection, the management of a Queue Pair, and the encapsulation and analysis of a ROCEv2 data packet. After a Queue Pair is established (QP is a standard protocol, which refers to a link), the host server side has a mapping table of a system memory and a virtual address for each QP, and the address resources are maintained. The host server maps discrete memory resources into the DMA virtual address space of the QP, and the SEND operation of these address resources sends the discrete memory resources to the FPGA side. At the same time, the state and recycling of these resources are also managed.

[0030] As shown in Figure 2 , the content of the SEND packet includes an identifier and a valid bit flag, which is beneficial to the use of circuit logic analysis, realizes high efficiency, and avoids delay.

[0031] The discrete memory resources of the host server side are parsed, and entry information is constructed to describe the virtual address and the length. After the construction, the entry is written into the address resource FIFO.

[0032] The address resource FIFO stores the entry, and after receiving the entry application of the channel manager, an entry is taken out from the address resource FIFO to the channel manager to complete an address application operation.

[0033] As shown in Figure 4 , the channel manager is responsible for entry application, and after the entry is applied, the address and length information are loaded into the register circuit and the DMA context is started to be maintained for the use of the DMA engine. When the channel management module detects a DMA length completion or a service data packet carrying an interrupt flag, the channel manager ends the current DMA after the DMA engine packs, and at the same time, the channel manager applies a new address entry to the address resource FIFO, and the context of the DMA is updated. At the same time, a completion entry is generated, as shown in Figure 3 , the address information and the completion data length information are sent to the ROCEv2 protocol processing interface.

[0034] Each service if need interrupt host, only need to carry interrupt flag in data packet, can be generated by channel manager complete entry to host, response delay is very small.

[0035] Address resource FIFO and channel manager are realized by FPGA circuit resources, which can jump at each clock cycle. Therefore, the application resource allocation and DMA context switching are very fast, which can be completed in several clock cycles, i.e. tens of NS. Therefore, in the scene of multi-channel and multi-service type switching, the service data packet does not need to wait for a long time in the cache due to address resource management, and the FPGA on-chip cache can be used without additional FPGA off-chip cache resources. Simplify the hardware design, reduce the device cost, and greatly improve the device performance.

[0036] The DMA engine switches to the DMA context information of the corresponding channel manager according to the channel information of the multi-channel service data packet, encapsulates the virtual address and length information and the data frame into an Infiniband data packet, and sends it to the ROCEv2 protocol processing interface.

[0037] The ROCEv2 encapsulates the Infiniband data packet with address and length into the RDMA WRITE operation of the ROCEv2 and sends it to the host server side.

[0038] An address resource FIFO uses the same QP resource. Instead of a separate QP for each channel, this can save QP resources. In the RDMA WRITE operation in a QP, the data packets of different channel services are assigned different virtual addresses by the channel manager, as shown in the network packet interaction diagram. Figure 5

[0039] The ROCEv2 protocol processing interface encapsulates the completion entry into the SEND operation and sends it to the host server side. In the scene of small service data packet and frequent interruption, multiple return entries are aggregated into a SEND operation. Avoid frequent SEND operations to occupy too much link resource.

[0040] After the host server side receives the complete SEND, the virtual address and completion data length information are parsed, the actual address information is obtained by querying the mapping table, and the actual address and length information are pushed to the application program. The user application program can directly obtain data from the memory to realize zero copy. The service data packet with interrupt flag can realize low delay response.

[0041] This embodiment fully utilizes the technical advantages of ROCEv2, and the digital signal processing device can utilize the network advantage for distributed processing; the high bandwidth scene does not depend on DDR cache, simplifies the single board design, reduces the device cost, and realizes the low delay response of the service.​

[0042] The above embodiments only express the specific implementation of the present application, which is described in more detail and in more detail, but it cannot be understood as a limitation on the scope of the patent of the present application. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application.

Claims

1. A high-bandwidth low-latency data processing method based on ROCEv2, characterized in that, The chip includes a ROCEv2 protocol processing interface, an address resource FIFO, a channel manager, a DMA engine and a multi-channel service data packet. Step 1, the ROCEv2 protocol processing interface establishes a connection with the ROCEv2 network card of the host server, the host server maps discrete memory resources into the DMA virtual address space of the queue pair, and sends the discrete memory resources to the ROCEv2 protocol processing interface of the chip; Step 2, after the ROCEv2 protocol processing interface receives the sending packet of the host server, the discrete memory resources are parsed, and entry information is constructed to describe the virtual address and length information, and after the construction is completed, the entry is written into the address resource FIFO; Step 3, the address resource FIFO stores the entry, receives the entry application of the channel manager, takes out an entry from the address resource FIFO for the channel manager, and completes an address application operation; Step 4, after the channel manager applies for the entry, the virtual address and length information are loaded into the register circuit and the DMA context is maintained for the DMA engine, when a DMA length transmission is completed, or an interrupt is required, a completion entry is generated, and the address information and completion data length information are sent to the ROCEv2 protocol interface; Step 5, the DMA engine switches to the DMA context information of the corresponding channel manager according to the channel information of the multi-channel service data packet, encapsulates the virtual address, length information and data frame through the ROCEv2 protocol processing interface, and sends them to the host server; Step 6, the host server parses the virtual address and length information, queries the mapping table to obtain the actual address information, and pushes the actual address and length information to the application program.

2. The ROCEv2 based high bandwidth low latency data processing method according to claim 1, wherein, In step 1, the host server also manages the state and recycling of the discrete memory resources. 3.The ROCEv2-based high-bandwidth and low-latency data processing method of claim 1, wherein, In step 4, when the channel manager detects that a DMA length is completed, or the service data packet carries an interrupt flag, the channel manager ends the current DMA after packetizing by the DMA engine; meanwhile, the channel manager applies for a new entry from the address resource FIFO, and the DMA context is updated.

4. The ROCEv2 based high bandwidth low latency data processing method of claim 1, wherein, The address resource FIFO in the FPGA is mapped to the resources of the same queue pair, so that different service channels are realized by using a queue pair; the data packets of different channel services are assigned different virtual addresses by the channel manager.

5. The ROCEv2 based high bandwidth low latency data processing method according to any of claims 1-4, characterized in that, The chip is an FPGA chip or an ASIC chip.

Images