Packet processing method, device and system
By storing entries such as MTT, QPC, and CQC in the memory of the message processing device, the problems of PCIe bandwidth occupation and latency when RDMA devices process messages are solved, achieving efficient CPU offloading and reduced latency.
Patent Information
- Application Number
- CN202211701573.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-29
- Publication Date
- 2026-02-10
- Estimated Expiration
- 2042-12-29
AI Technical Summary
When existing RDMA devices process messages, the entries are stored in the host-side memory, which increases PCIe bandwidth usage and command processing latency.
Entries such as MTT, QPC, and CQC are stored in the first memory of the message processing device and read from the first memory only when they are missing in the cache, reducing the need to read from the host memory via PCIe.
This achieves CPU offloading of the target host while reducing PCIe bandwidth usage and command processing latency.
Smart Images

Figure CN116016570B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a message processing method, apparatus and system. Background Technology
[0002] With the development of computer technology, a technology has emerged that uses RDMA devices to process messages. These RDMA devices can be applied to DPUs, which, as data processing units, are primarily used to offload CPU tasks.
[0003] Currently, in the process of RDMA devices processing messages, it is usually necessary to use certain pre-stored table entries to obtain relevant data and use the determined data to perform message processing. For example, QPC table entries can be used to obtain context information to determine the ownership of the message command, or PBL table entries can be used to obtain the physical address corresponding to the access command to realize memory access.
[0004] However, in current RDMA device message command processing techniques, the aforementioned entries are typically stored in the host-side memory. Therefore, when using these entries to execute message command processing, the RDMA device needs to first read the corresponding entries from the host memory via PCIe. Consequently, current RDMA device message command processing techniques consume PCIe bandwidth and increase command processing latency. Summary of the Invention
[0005] Therefore, it is necessary to provide a message processing method, apparatus, and system that can reduce command processing latency in response to the above-mentioned technical problems.
[0006] The first aspect of the present invention provides a message processing apparatus that communicates with a target host device via a PCIe bus. The message processing apparatus includes a processor, an RDMA device, and a first memory.
[0007] The processor is used to communicate with the target host in the target host device and to establish MTT and QPC in the first memory;
[0008] The first memory is used to store the MTT and QPC;
[0009] The RDMA device is used to receive and parse command requests for sending messages, determine the QPC address of the command request based on the parsing result, and if it is determined based on the QPC address that the QPC of the command request does not exist in the cache of the RDMA device, then the QPC is read from the first memory based on the QPC address to obtain the base address of the SQ.
[0010] The RDMA device is further configured to read and parse the SQ WQE from the second memory in the target host device based on the base address of the SQ, to obtain the type of the message to be sent, the virtual address of the payload, and the index address of the MTT. When the type of the message to be sent is a send message, if it is determined from the index address that the MTT requested by the command is not in the cache of the RDMA device, the MTT is read from the first memory to obtain the base address of the PBL, the PBL is obtained based on the base address of the PBL, the physical address of the payload is obtained based on the virtual address of the payload and the PBL, the payload is read from the physical address, a message is generated based on the payload, and the message is sent.
[0011] Preferably, the processor is further configured to establish a CQC in the first memory;
[0012] The first memory is also used to store the CQC;
[0013] The RDMA device is also used to receive and parse ACK messages, determine the CQC address of the command request and generate a CQE based on the parsing result, and if it is determined based on the CQC address that the CQC of the command request does not exist in the cache of the RDMA device, then the CQC is read from the first memory based on the CQC address to obtain the address of the CQE, and the CQE is written to the address of the CQE.
[0014] A second aspect of the present invention provides a message processing method applied to a message processing apparatus, the message processing apparatus communicating with a target host device via a PCIe bus, the message processing apparatus including a processor, an RDMA device, and a first memory, wherein the method includes:
[0015] The processor communicates with the target host in the target host device and establishes MTT and QPC in the first memory;
[0016] The first memory stores the MTT and QPC;
[0017] The RDMA device receives and parses the command request of the transmission message, determines the QPC address of the command request based on the parsing result, and if it is determined based on the QPC address that the QPC of the command request does not exist in the cache of the RDMA device, then the QPC is read from the first memory based on the QPC address to obtain the base address of the SQ.
[0018] The RDMA device further reads and parses the SQ WQE from the second memory in the target host device based on the base address of the SQ to obtain the type of the message to be sent, the virtual address of the payload, and the index address of the MTT. When the type of the message to be sent is a send message, if it is determined from the index address that the MTT requested by the command is not in the cache of the RDMA device, the MTT is read from the first memory to obtain the base address of the PBL, the PBL is obtained based on the base address of the PBL, the physical address of the payload is obtained based on the virtual address of the payload and the PBL, the payload is read from the physical address, a message is generated based on the payload, and the message is sent.
[0019] Preferably, the method further includes:
[0020] The processor establishes a CQC in the first memory;
[0021] The first memory stores the CQC;
[0022] The RDMA device receives and parses the ACK message, determines the CQC address of the command request and generates a CQE based on the parsing result. If the CQC of the command request is not in the cache of the RDMA device based on the CQC address, the CQC is read from the first memory based on the CQC address to obtain the address of the CQE, and the CQE is written to the address of the CQE.
[0023] A third aspect of the present invention provides a message processing apparatus that communicates with a target host device via a PCIe bus. The message processing apparatus includes a processor, an RDMA device, and a first memory.
[0024] The processor is used to communicate with the target host in the target host device and to establish QPC and MTT in the first memory;
[0025] The first memory is used to store the QPC and the MTT;
[0026] The RDMA device is used to receive and parse messages to obtain the message type, payload, and QPC address of the message. When the message type is a send message, if it is determined from the QPC address that the QPC of the message does not exist in the cache of the RDMA device, the QPC is read from the first memory to obtain the base address of the RQ.
[0027] The RDMA device is further configured to read and parse the RQ WQE from the second memory in the target host device based on the base address of the RQ, obtain the virtual address of the payload and the index address of the MTT, and if it is determined from the index address that the MTT of the packet does not exist in the cache of the RDMA device, then read the MTT from the first memory, obtain the base address of the PBL, obtain the PBL of the packet based on the base address of the PBL, obtain the physical address of the payload based on the virtual address of the payload and the PBL, and write the payload to the physical address.
[0028] Preferably,
[0029] The processor is also used to communicate with the target host in the target host device and to establish CQC in the first memory;
[0030] The RDMA device is also used to receive and parse ACK messages, determine the address of CQC based on the parsing result and generate CQE. If it is determined based on the address of the CQC that the CQC of the message is not in the cache of the RDMA device, then the CQC is read from the first memory based on the address of the CQC to obtain the address of the CQE, and the CQE is written into the second memory based on the address of the CQE.
[0031] A fourth aspect of the present invention provides a message processing method, wherein the message processing device communicates with a target host device via a PCIe bus, and the message processing device includes a processor, an RDMA device, and a first memory, wherein...
[0032] The processor is used to communicate with the target host in the target host device and to establish QPC and MTT in the first memory;
[0033] The RDMA device is used to receive and parse messages to obtain the message type, payload, and QPC address of the message. When the message type is a send message, if it is determined from the QPC address that the QPC of the message does not exist in the cache of the RDMA device, the QPC is read from the first memory to obtain the base address of the RQ.
[0034] The RDMA device is further configured to read and parse the RQ WQE from the second memory in the target host device based on the base address of the RQ, obtain the virtual address of the payload and the index address of the MTT, and if it is determined from the index address that the MTT of the packet does not exist in the cache of the RDMA device, then read the MTT from the first memory, obtain the base address of the PBL, obtain the PBL of the packet based on the base address of the PBL, obtain the physical address of the payload based on the virtual address of the payload and the PBL, and write the payload to the physical address.
[0035] Preferably, the method further includes:
[0036] The processor communicates with the target host in the target host device and establishes a CQC in the first memory;
[0037] The RDMA device receives and parses the ACK message, determines the address of the CQC based on the parsing result, and generates a CQE. If the CQC of the message is not in the cache of the RDMA device based on the address of the CQC, the CQC is read from the first memory based on the address of the CQC to obtain the address of the CQE, and the CQE is written into the second memory based on the address of the CQE.
[0038] A fifth aspect of the present invention provides a message processing system, including a target host device and a message processing device as described above.
[0039] A sixth aspect of the present invention provides a message processing system, including a target host device and the message processing device as described above.
[0040] Beneficial effects: By storing the MTT, QPC, and CQC related to MR registration in the first memory of the message processing device, when the RDMA device processes a message and the corresponding MTT, QPC, and CQC are not in the cache, they only need to be read from the first memory, eliminating the need to retrieve them from the host memory via PCIe.
[0041] By reading the corresponding table entries, the CPU services of the target host can be offloaded, while the PCIE0 bandwidth usage can be reduced, and the latency of command processing can be reduced. Attached Figure Description
[0042] Figure 1 This is a schematic diagram of the structure of a message processing device in one embodiment;
[0043] Figure 2 This is a schematic diagram of the structure of a specific message processing device in one embodiment;
[0044] 5 Figure 3 This is a flowchart illustrating a message processing method in one embodiment;
[0045] Figure 4 This is a flowchart illustrating a message processing method in one embodiment. Detailed Implementation
[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0047] Embodiment 1 of the present invention provides a message processing device, such as Figure 1 As shown, the message processing device 2 communicates with the target host device 1 via the PCIe bus. The target host device is located on the server side, and the message...
[0048] The processing device is located on the device side. The message processing device 2 includes a processor 21, a first memory 22, and an RDMA device 23. The processor 21 communicates with the target host 11 in the target host device 1 and establishes a QPC and MTT in the first memory 22, which stores the QPC and MTT. The RDMA device receives and parses command requests for sending messages, determines the QPC address of the command request based on the parsing result, and if the QPC address indicates that the command request does not exist in the cache of the RDMA device 23, then the QPC is read from the first memory 22 based on the QPC address to obtain the base address of the SQ. The RDMA device is also used to read and parse the SQ from the second memory 12 in the target host device based on the base address. WQE obtains the type of the message to be sent, the virtual address of the payload, and the index address of the MTT. When the type of the message to be sent is a send message, if it is determined from the index address that the MTT requested by the command is not in the cache of the RDMA device, then the MTT table is read from the first memory to obtain the base address of the PBL, the PBL is obtained according to the base address of the PBL, the physical address of the payload is obtained according to the virtual address of the payload and the PBL, the payload is read from the physical address, a message is generated according to the payload, and the message is sent.
[0049] Specifically, the target host device 1 includes a target host 11 and a second memory 12. The target host 11 is the CPU on the server side, and the second memory 12 can be DDR. The message processing device can be a DPU (Data Processing Unit), where the processor 21 is a CPU or other processor core, and the first memory 22 can be DDR.
[0050] In this system, the RDMA device 23 communicates with the target host 11 via Queue Pair (QP). A QP contains a Send Queue (SQ) and a Receive Queue (RQ). When a user needs to send a request, it encapsulates the request into a Work Queue Element (WQE) and sends it to the SQ. According to the IB protocol, a QP is a virtual interface between hardware and software. A QP is a queue structure that sequentially stores tasks (WQEs) issued by the software to the hardware. A QPC (Queue PairContext) is used to store QP-related attributes.
[0051] When registering a memory region, you need to specify the access permissions for that memory and store the access permission information in the Memory Protection Tables (MPT) for user permission verification.
[0052] Since DMA can only access based on physical addresses, it is necessary to save a virtual memory-to-physical memory mapping table (PBL) of the target memory region. This mapping table is stored in the Memory Translation Table (MTT).
[0053] Payload: The part of the data that contains information. Usually, when transmitting data, in order to make data transmission more reliable, the original data is transmitted in batches, and certain auxiliary information is added to the beginning and end of each batch of data, thus forming the basic transmission unit in the transmission channel.
[0054] like Figure 2As shown, the target host (i.e., the host-side CPU) in the target host device establishes communication with the processor (device-side CPU) in the message processing device. The target host performs memory registration in the second memory (host-side memory). The device-side CPU is used to establish the MTT and MPT corresponding to the memory registration in the first memory (i.e., device-side memory). The RDMA device establishes connections with both the host-side CPU and the device-side CPU. The host-side CPU establishes QPC and CQC in the device-side memory. It should be noted that QPC is used, but is not limited to, storing the PI / CI pointers and base addresses of SQ / RQ, and CQC is used, but is not limited to, storing the PI / CI pointers and base addresses of CQ. The target host writes SQ WQE into the host-side memory. The host-side CPU sends a doorbell to the RDMA device, notifying it to prepare to send a message. The RDMA device receives and parses the command request. Based on the parsing result, the RDMA device obtains the QPC address corresponding to the command request and determines whether the QPC is in the RDMA device's cache. If the QPC is in the RDMA device's cache, it is read from the cache. If not, the RDMA device retrieves the QPC from its memory based on the address. Reading the QPC yields the SQ base address and PI / CI pointer. The RDMA device also reads the SQ WQE from the secondary memory based on the SQ base address and PI / CI pointer. The RDMA device parses the read SQ WQE to obtain the message type, payload virtual address, MPT index address, and MTT index address of the message to be sent. When the type of the message to be sent is a send message, the index address of the MPT is used to determine whether the MPT requested by the command is in the cache of the RDMA device. If it is not in the cache of the RDMA device, the MPT is read from the memory on the device side. The access permission is determined based on the MPT. If the access permission is granted, if the index address determines that the MTT requested by the command is not in the cache of the RDMA device, the MTT is read from the memory on the device side to obtain the base address of the PBL. The PBL is obtained based on the base address of the PBL. The physical address of the payload is obtained based on the virtual address of the payload and the PBL. The payload is read from the physical address to generate the message to be sent and then send the message.
[0055] Specifically, the RDMA device parses the command request to obtain the corresponding QPN (Queue Pair Number) and function ID. Based on the QPN and function ID, it determines the QPC address corresponding to the command request. The RDMA device then determines whether the QPC is in its cache. If it is, the RDMA device reads the QPC directly from the cache; otherwise, it reads it from the first memory. After reading the QPC, it obtains the base address and PI / CI pointer of the SQ. Based on the SQ's base address and PI / CI pointer, it reads the SQWQE from the second memory. The SQWQE carries the SGE, which contains the virtual address and key of the payload, as well as the type of the message to be sent. The RDMA device then parses the SQ... WQE obtains the type of the message to be sent, the virtual address of the payload, and the key. The key contains data used to verify whether the user has access rights and the index address of the MPT. When the type of the message to be sent is a send message, it determines whether the MPT of the request command is in the cache of the RDMA device based on the index address of the MPT. If it is not in the cache of the RDMA device, it reads the MPT from the first memory, obtains the user permission data in the MPT, and determines whether the user has access rights by comparing the obtained permission data with the access rights data in the key. If the user has access rights, it determines whether the MTT is in the cache of the RDMA device based on the index address of the MTT. If it is not in the cache of the RDMA device, it reads the MTT from the first memory, obtains the base address of the PBL based on the MTT, obtains the PBL based on the base address of the PBL, obtains the physical address of the payload based on the base address of the payload and the PBL, reads the payload from the physical address of the payload, generates a message based on the payload, and sends the message.
[0056] In one specific embodiment, the processor is further configured to establish a CQC in the first memory, and the first memory is further configured to store the CQC. The RDMA device is further configured to receive an ACK message sent by the message receiver (i.e., the peer), parse the ACK message, determine the CQC address of the command request and generate a CQE based on the parsing result, and if it is determined that the CQC of the command request is not in the cache of the RDMA device based on the address of the CQC, then the CQC is read from the first memory based on the address of the CQC to obtain the address of the CQE, and the CQE is written to the address of the CQE (Completion Queue Element).
[0057] According to the IB protocol, when the peer receives a send message, it will return an ACK message to the local peer to indicate whether the send message has been completed.
[0058] The message processing device in this embodiment of the invention stores the MTT, MPT, QPC, and CQC related to MR registration in the first memory of the message processing device. When the RDMA device processes a message, if the corresponding MTT, MPT, QPC, and CQC are not in the cache, they only need to be read from the first memory. The corresponding entries can be read from the host memory without going through PCIe. This achieves the offloading of CPU services of the target host while reducing the occupation of PCIe bandwidth, thereby reducing the latency of command processing.
[0059] Based on Embodiment 1 of the present invention, Embodiment 2 of the present invention provides a message processing method, such as... Figure 3 As shown, the method includes the following steps:
[0060] S11. The processor is used to communicate with the target host in the target host device and to establish MTT and QPC in the first memory;
[0061] S12. The RDMA device receives and parses the command request, determines the QPC address of the command request based on the parsing result, and if it is determined based on the QPC address that the QPC to be used does not exist in the RDMA cache, then it reads the QPC table from the first memory based on the QPC address to obtain the base address of the send queue. The RDMA device reads and parses SQWQE from the base address in the second memory of the target host device to obtain the type of the message to be sent, the virtual address of the payload, and the index address of the MPT. When the type of the message to be sent is a send message, if it is determined based on the index address that the MPT table to be used is not in the cache of the RDMA device, then it reads the MPT table from the second memory to obtain the base address of the PBL, obtains the PBL table based on the base address of the PBL, obtains the physical address of the payload based on the virtual address of the payload and the PBL table, reads the payload from the physical address, forms a send message based on the payload, and sends the message.
[0062] In one specific embodiment, the processor also communicates with the target host in the target host device and establishes a CQC table in the first memory. The RDMA device also receives and parses CQE, determines the address of CQC based on the parsing result, and if it is determined based on the address of the CQC that the CQC to be used is not in the cache of the RDMA device, then the CQC table is read from the address of the CQC in the first memory to obtain the address of the CQE, and the CQE is written into the second memory based on the address of the CQE.
[0063] The message processing method in this embodiment of the invention stores the MTT, MPT, QPC, and CQC related to MR registration in the first memory of the message processing device. When the RDMA device processes a message, if the corresponding MTT, MPT, QPC, and CQC are not in the RDMA device's cache, they only need to be read from the first memory. The corresponding entries can be read from the host memory without going through PCIe, thereby achieving the offloading of CPU services of the target host while reducing the occupation of PCIe bandwidth and reducing the latency of command processing.
[0064] Embodiment 3 of the present invention provides a message processing device, such as Figure 1 As shown, the message processing device communicates with the target host device via a PCIe bus. The message processing device includes a processor, an RDMA device, and a first memory. The processor communicates with the target host in the target host device and establishes a QPC and MTT in the first memory. The RDMA device receives and parses messages to obtain the message type, payload, and QPC address. When the message type is a send message, if the QPC is not present in the RDMA device's cache based on the QPC address, the QPC is read from the first memory to obtain the base address of the RQ. The RDMA device is also used to read and parse the RQ from a second memory in the target host device based on the base address of the RQ. WQE obtains the virtual address of the payload and the index address of the MTT. If it is determined from the index address that the MTT of the packet does not exist in the cache of the RDMA device, the MTT is read from the first memory to obtain the base address of the PBL. The PBL of the packet is obtained from the base address of the PBL. The physical address of the payload is obtained from the virtual address of the payload and the PBL. The payload is written to the physical address.
[0065] The following is for reference Figure 2The following explanation is provided. The target host (i.e., the host-side CPU) in the target host device communicates with the processor (device-side CPU) in the message processing device. The host-side CPU performs MR registration in the second memory (i.e., host-side memory), while the device-side CPU establishes MPT and MTT in the first memory (device-side memory). The host-side CPU writes RQ WQE into the device-side memory. The RDMA device receives messages sent by the peer RDMA device and parses the received messages to obtain the message payload, message type, and QPN. The QPN provides the message's QPC address. When the message type is a send message, the QPC address is used to determine if the message's QPC is in the RDMA device's cache. If not, the QPC is retrieved from the device-side memory, and its contents are read to obtain the SEG. The SEG includes the index addresses of the MTT and MPT, as well as a key. The key contains data used for identity verification. The MPT index address is used to determine if the message's MPT is in the RDMA device's cache. If the MPT is not found in the DMA device's cache, it is read from the device's memory. The contents of the MPT are read, and the user's access rights are determined based on the identity verification data in the MPT and the identity verification data in the key. If the user has access rights, the MTT index address is used to determine whether the MTT of the packet is in the RDMA device's cache. If the MTT does not exist, it is read from the device's memory, and the contents of the MTT are read to obtain the base address of the PBL. The PBL is obtained based on the base address of the PBL, and the physical address of the payload is obtained based on the base address of the PBL and the virtual address of the payload. The payload is then written to the physical address of the payload.
[0066] In one specific implementation, the RDMA device receives and parses the ACK packet, determines the address of the CQC based on the parsing result, and generates a CQE. If the CQC of the packet is not in the cache of the RDMA device based on the address of the CQC, the CQC is read from the first memory based on the address of the CQC to obtain the address of the CQE, and the CQE is written into the second memory based on the address of the CQE.
[0067] Based on Embodiment 3 of the present invention, Embodiment 4 of the present invention provides a message processing method, such as... Figure 4 As shown, the method includes the following steps:
[0068] S21. The processor is used to communicate with the target host in the target host device and to establish QPC and MTT in the first memory;
[0069] S22. The RDMA device is used to receive and parse packets to obtain the packet type, payload, and QPC address of the packet. When the packet type is a send packet, if it is determined from the QPC address that the QPC of the packet does not exist in the cache of the RDMA device, the QPC is read from the first memory to obtain the base address of the RQ. The RDMA device is also used to read and parse the RQ WQE from the second memory in the target host device based on the base address of the RQ to obtain the virtual address of the payload and the index address of the MTT. If it is determined from the index address that the MTT of the packet does not exist in the cache of the RDMA device, the MTT is read from the first memory to obtain the base address of the PBL. The PBL of the packet is obtained based on the base address of the PBL. The physical address of the payload is obtained based on the virtual address of the payload and the PBL. The payload is written to the physical address.
[0070] In one specific embodiment, the processor communicates with the target host in the target host device and establishes a CQC in the first memory;
[0071] The first memory stores the CQC;
[0072] The RDMA device receives and parses the ACK, determines the address of the CQC based on the parsing result, and generates a CQE. If the CQC of the packet is not in the cache of the RDMA device based on the address of the CQC, the CQC is read from the first memory based on the address of the CQC to obtain the base address of the CQ. The CQC table is read from the first memory based on the base address of the CQ to obtain the base address of the CQE. The CQE is written into the second memory based on the base address of the CQE.
[0073] The message processing method in this embodiment of the invention stores the MTT, MPT, QPC, and CQC related to MR registration in the first memory of the message processing device. When the RDMA device processes a message, if the corresponding MTT, MPT, QPC, and CQC are not in the cache, they only need to be read from the first memory. The corresponding entries can be read from the host memory without going through PCIe, thereby achieving the offloading of CPU services of the target host while reducing the occupation of PCIe bandwidth and reducing the latency of command processing.
[0074] Based on Embodiment 1 of the present invention, Embodiment 5 of the present invention provides a message processing system, which includes: a target host device and a message processing device as described in Embodiment 1.
[0075] The target host device is the same as the target host device in Embodiment 1, and therefore will not be described again here.
[0076] Based on Embodiment 3 of the present invention, Embodiment 6 of the present invention provides a message processing system, which includes: a target host device and a message processing device as described in Embodiment 3.
[0077] The target host device is the same as the target host device in Embodiment 3, and therefore will not be described again here.
[0078] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0079] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A message processing apparatus, characterized in that, The message processing device communicates with the target host device via a PCIe bus. The message processing device includes a processor, an RDMA device, and a first memory. The processor is used to communicate with the target host in the target host device and to establish MTT and QPC in the first memory; The first memory is used to store the MTT and QPC; The RDMA device is used to receive and parse command requests for sending messages, determine the QPC address of the command request based on the parsing result, and if it is determined based on the QPC address that the QPC of the command request does not exist in the cache of the RDMA device, then the QPC is read from the first memory based on the QPC address to obtain the base address of the SQ. The RDMA device is further configured to read and parse the SQ WQE from the second memory in the target host device based on the base address of the SQ, to obtain the type of the message to be sent, the virtual address of the payload, and the index address of the MTT. When the type of the message to be sent is a send message, if it is determined from the index address that the MTT requested by the command is not in the cache of the RDMA device, the MTT is read from the first memory to obtain the base address of the PBL, the PBL is obtained based on the base address of the PBL, the physical address of the payload is obtained based on the virtual address of the payload and the PBL, the payload is read from the physical address, a message is generated based on the payload, and the message is sent.
2. The apparatus according to claim 1, characterized in that: The processor is also used to establish CQC in the first memory; The first memory is also used to store the CQC; The RDMA device is also used to receive and parse ACK messages, determine the CQC address of the command request and generate a CQE based on the parsing result, and if it is determined based on the CQC address that the CQC of the command request does not exist in the cache of the RDMA device, then the CQC is read from the first memory based on the CQC address to obtain the address of the CQE, and the CQE is written to the address of the CQE.
3. A message processing method, characterized in that, Applied to a message processing apparatus that communicates with a target host device via a PCIe bus, the message processing apparatus includes a processor, an RDMA device, and a first memory, wherein the method includes: The processor communicates with the target host in the target host device and establishes MTT and QPC in the first memory; The first memory stores the MTT and QPC; The RDMA device receives and parses the command request of the transmission message, determines the QPC address of the command request based on the parsing result, and if it is determined based on the QPC address that the QPC of the command request does not exist in the cache of the RDMA device, then the QPC is read from the first memory based on the QPC address to obtain the base address of the SQ. The RDMA device further reads and parses the SQ WQE from the second memory in the target host device based on the base address of the SQ to obtain the type of the message to be sent, the virtual address of the payload, and the index address of the MTT. When the type of the message to be sent is a send message, if it is determined from the index address that the MTT requested by the command is not in the cache of the RDMA device, the MTT is read from the first memory to obtain the base address of the PBL, the PBL is obtained based on the base address of the PBL, the physical address of the payload is obtained based on the virtual address of the payload and the PBL, the payload is read from the physical address, a message is generated based on the payload, and the message is sent.
4. The method according to claim 3, characterized in that, The method further includes: The processor establishes a CQC in the first memory; The first memory stores the CQC; The RDMA device receives and parses the ACK message, determines the CQC address of the command request and generates a CQE based on the parsing result. If the CQC of the command request is not in the cache of the RDMA device based on the CQC address, the CQC is read from the first memory based on the CQC address to obtain the address of the CQE, and the CQE is written to the address of the CQE.
5. A message processing apparatus, characterized in that, The message processing device communicates with the target host device via a PCIe bus. The message processing device includes a processor, an RDMA device, and a first memory. The processor is used to communicate with the target host in the target host device and to establish QPC and MTT in the first memory; The first memory is used to store the QPC and the MTT; The RDMA device is used to receive and parse messages to obtain the message type, payload, and QPC address of the message. When the message type is a send message, if it is determined from the QPC address that the QPC of the message does not exist in the cache of the RDMA device, the QPC is read from the first memory to obtain the base address of the RQ. The RDMA device is further configured to read and parse the RQ WQE from the second memory in the target host device based on the base address of the RQ, obtain the virtual address of the payload and the index address of the MTT, and if it is determined from the index address that the MTT of the packet does not exist in the cache of the RDMA device, then read the MTT from the first memory, obtain the base address of the PBL, obtain the PBL of the packet based on the base address of the PBL, obtain the physical address of the payload based on the virtual address of the payload and the PBL, and write the payload to the physical address.
6. The apparatus according to claim 5, characterized in that: The processor is also used to communicate with the target host in the target host device and to establish CQC in the first memory; The RDMA device is also used to receive and parse ACK messages, determine the address of CQC based on the parsing result and generate CQE. If it is determined based on the address of the CQC that the CQC of the message is not in the cache of the RDMA device, then the CQC is read from the first memory based on the address of the CQC to obtain the address of the CQE, and the CQE is written into the second memory based on the address of the CQE.
7. A message processing method, characterized in that, The message processing device communicates with the target host device via a PCIe bus. The message processing device includes a processor, an RDMA device, and a first memory. The processor is used to communicate with the target host in the target host device and to establish QPC and MTT in the first memory; The RDMA device is used to receive and parse messages to obtain the message type, payload, and QPC address of the message. When the message type is a send message, if it is determined from the QPC address that the QPC of the message does not exist in the cache of the RDMA device, the QPC is read from the first memory to obtain the base address of the RQ. The RDMA device is further configured to read and parse the RQ WQE from the second memory in the target host device based on the base address of the RQ, obtain the virtual address of the payload and the index address of the MTT, and if it is determined from the index address that the MTT of the packet does not exist in the cache of the RDMA device, then read the MTT from the first memory, obtain the base address of the PBL, obtain the PBL of the packet based on the base address of the PBL, obtain the physical address of the payload based on the virtual address of the payload and the PBL, and write the payload to the physical address.
8. The method according to claim 7, characterized in that, The method further includes: The processor communicates with the target host in the target host device and establishes a CQC in the first memory; The RDMA device receives and parses the ACK message, determines the address of the CQC based on the parsing result, and generates a CQE. If the CQC of the message is not in the cache of the RDMA device based on the address of the CQC, the CQC is read from the first memory based on the address of the CQC to obtain the address of the CQE, and the CQE is written into the second memory based on the address of the CQE.
9. A message processing system, characterized in that, It includes the target host device and the message processing device as described in any one of claims 1-2.
10. A message processing system, characterized in that, It includes the target host device and the message processing device as described in any one of claims 5-6.
Citation Information
Patent Citations
Communication method for reducing high-speed bus delay
CN109491809A
Method and computer equipment for processing remote direct memory access request
CN111221758A