A memory translation table application method and system for RDMA

Abstract

This invention provides a method and system for applying a memory translation table in RDMA. The method is applied to a network card connected to DDR, the network card including an out-of-order cache module, a DDR control module, and a RAM control module. The method includes the following steps: receiving a lookup request; the out-of-order cache module determining the packet type of the lookup request; if the packet type is a first packet, obtaining the virtual address of the lookup request, and matching the corresponding physical address from the memory address translation table stored in the DDR based on the DDR control module, and storing the obtained correspondence between the virtual address and the physical address in the cached address translation table of the network card's RAM memory; if the packet type is an intermediate packet or a last packet, obtaining the virtual address of the lookup request, and matching the corresponding physical address from the cached address translation table stored in the RAM memory based on the RAM control module. This solution ensures the comprehensiveness of the memory address translation table through DDR and improves the matching processing efficiency through RAM memory.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to an RDMA memory translation table application method and system. Background Art

[0002] Remote direct memory access (RDMA) was created to solve the delay of server-side data processing in network transmission. RDMA transfers data directly to the computer's storage area through the network, quickly moving data from one system to the remote system memory without any impact on the operating system. This does not require much computer processing power, eliminates the overhead of external memory copying and context switching, and thus frees up memory bandwidth and CPU cycles to improve application system performance.

[0003] The sender of the RDMA operation will access the memory address space of the receiver through the virtual address (VA), but the receiver cannot directly locate the memory address space of the receiver through the VA, so the memory translation table (MTT) is needed to convert the VA into the physical address (PA) of the receiver's memory. Then the receiver can access the memory space requested by the sender through the PA. A set of corresponding VA and PA is called an entry information of the MTT.

[0004] There are two main ways to implement MTT: one is to implement it through the internal storage resources of the chip. The advantage of this implementation method is that the speed from VA to PA is very fast and the delay is low. The disadvantage is that the internal storage resources of the chip are limited and cannot store a large amount of MTT table entry information; the second is to implement it by allocating the double-rate synchronous dynamic random access memory (DDR) space of the chip peripheral. The advantage of this implementation method is that the storage space of DDR is large and a large amount of MTT table entry information can be stored, but the disadvantage is that the access rate of DDR is slow and the delay of accessing DDR will be greater than directly accessing the internal storage resources of the chip. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method for applying a memory translation table of RDMA to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of the present invention provides a method for applying a memory translation table of RDMA, the method being applied to a network card connected to a DDR, the network card comprising a disordered cache module, a DDR control module and a RAM control module, the method comprising the following steps:

[0007] The network card receives a table lookup request, and the out-of-order cache module determines the message type of the table lookup request;

[0008] If the message type is the first packet, obtain the virtual address of the table lookup request, and match the corresponding physical address from the memory address translation table stored in the DDR based on the DDR control module, and store the obtained correspondence between the virtual address and the physical address in the cache address translation table of the network card RAM memory;

[0009] If the message type is an intermediate packet or a tail packet, the virtual address of the table lookup request is obtained, and the corresponding physical address is matched from the cache address translation table stored in the RAM memory based on the RAM control module.

[0010] The above scheme is adopted. This scheme first stores the cache address translation table in the DDR. When the message type is the first packet, the corresponding physical address is matched from the DDR and stored in the cache address translation table of the network card RAM memory. If the message type is an intermediate packet or a tail packet, it means that the first packet of the message has been processed. Therefore, the physical address is matched from the cache address translation table first. This scheme cooperates with the DDR and the RAM memory to store the correspondence between the virtual address and the physical address of the table lookup request in the network card RAM memory. When it is matched again, it can be processed quickly. On the one hand, the comprehensiveness of the memory address translation table is guaranteed by the DDR, and on the other hand, the matching processing efficiency is improved by the RAM memory.

[0011] In some embodiments of the present invention, in the step of storing the acquired correspondence between the virtual address and the physical address in the cache address translation table of the network card RAM memory, the RAM control module determines whether there is remaining storage space in the RAM memory. If so, the acquired correspondence between the virtual address and the physical address is stored in the cache address translation table of the network card RAM memory. If not, the correspondence between the virtual address and the physical address is not added to the RAM memory.

[0012] In some embodiments of the present invention, in the step of obtaining the virtual address of the table lookup request and matching the corresponding physical address from the cache address translation table stored in the RAM memory based on the RAM control module, if the corresponding physical address is not matched from the cache address translation table stored in the RAM memory, then the corresponding physical address is matched from the memory address translation table stored in the DDR based on the DDR control module.

[0013] In some embodiments of the present invention, the message type of the table lookup request includes a single table lookup request and a non-single table lookup request, wherein the message type of the non-single table lookup request includes a first packet, a middle packet, and a last packet. The steps of the method also include, if the message type is a single table lookup request, obtaining a virtual address of the single table lookup request, and matching the corresponding physical address from a memory address translation table stored in the DDR based on the DDR control module.

[0014] like Figure 3 As shown, in some embodiments of the present invention, the network card further includes an address calculation module, and the steps of the method include: the address calculation module obtains the corresponding memory storage unit based on the acquired physical address.

[0015] In some embodiments of the present invention, the method comprises the following steps: the RAM control module performs memory management on the RAM memory at preset management time intervals, and the memory management step comprises deleting the correspondence between the virtual address and the physical address stored in the RAM memory.

[0016] In some embodiments of the present invention, the step of deleting the correspondence between the virtual address and the physical address stored in the RAM memory includes obtaining the last matching time of each correspondence between the virtual address and the physical address stored in the RAM memory, calculating a first time interval between the current time and the last matching time, and if the first time interval is greater than a preset first threshold time, deleting the corresponding correspondence between the virtual address and the physical address from the RAM memory.

[0017] In some embodiments of the present invention, the step of deleting the correspondence between the virtual address and the physical address stored in the RAM memory also includes obtaining the correspondence between the virtual address and the physical address that was last matched as a tail packet in the RAM memory, and obtaining the last matching time of the tail packet, calculating a second time interval between the current time and the last matching time of the tail packet, and if the second time interval is greater than a preset second threshold time, deleting the corresponding correspondence between the virtual address and the physical address from the RAM memory.

[0018] In some implementations of the present invention, in the step of deleting the correspondence between the virtual address and the physical address stored in the RAM memory, the second threshold time is smaller than the first threshold time.

[0019] The second aspect of the present invention also provides an RDMA memory translation table application system, the system includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, when the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0020] The third aspect of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps implemented by the aforementioned RDMA memory translation table application method.

[0021] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention can be specifically pointed out and obtained in the specification and the accompanying drawings.

[0022] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0024] Figure 1 A schematic diagram of an implementation of a method for applying a memory translation table of RDMA of the present invention;

[0025] Figure 2 A schematic diagram of an implementation architecture of the RDMA memory translation table application method of the present invention;

[0026] Figure 3 It is a schematic diagram of another implementation architecture of the RDMA memory translation table application method of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0028] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0029] The steps of the present invention specifically include:

[0030] like Figure 1 and 2 As shown, the present invention proposes an RDMA memory translation table application method, the method is applied to a network card connected to a DDR, the network card includes a disorder cache module, a DDR control module and a RAM control module, and the steps of the method include:

[0031] In the specific implementation process, the network card is a device with storage capacity and processing capacity. Specifically, the network card is provided with an intelligent chip, and the intelligent chip can be a field programmable gate array chip (Field Programmable Gate Array, FPGA) and a system on chip (System on Chip, SoC) and the like.

[0032] In the specific implementation process, in RDMA technology, the control instructions sent by the CPU to the network card are collectively called WR (WorkRequest). The issued instructions will be stored in WQ (Work Queue), and the stored instructions are called WQE (WorkQueue Element). The corresponding hardware will automatically receive instructions from WQ. According to the type of stored instructions, WQ is divided into two categories: SQ (Send Queue) and RQ (Receive Queue), which together constitute the smallest unit of communication in RDMA, QP (QueuePairs).

[0033] Step S100, the network card receives a table lookup request, and the out-of-order cache module determines the message type of the table lookup request;

[0034] In a specific implementation process, the table lookup request is a table lookup request of a data packet, and the network card receives the data packet and obtains the table lookup request in the data packet.

[0035] Step S200, if the message type is the first packet, obtain the virtual address of the table lookup request, and match the corresponding physical address from the memory address translation table stored in the DDR based on the DDR control module, and store the obtained correspondence between the virtual address and the physical address in the cache address translation table of the network card RAM memory;

[0036] In the specific implementation process, the sender of the RDMA operation will access the memory address space of the receiver through the virtual address (VA), but the receiver cannot directly locate the memory address space of the receiver through the VA. Therefore, a memory translation table (MTT) is required to convert the VA into the physical address (PA) of the receiver's memory.

[0037] Step S300: If the message type is an intermediate packet or a tail packet, obtain the virtual address of the table lookup request, and match the corresponding physical address from the cache address translation table stored in the RAM memory based on the RAM control module.

[0038] In the specific implementation process, if the type is an intermediate packet or a tail packet, it will first determine whether there is a cached table lookup request for the same QP, that is, the first packet of the same QP has been processed. If so, the DDR access is skipped and the result is matched directly through the internal storage resources; if there is no matching result, the result is searched through the DDR.

[0039] In the specific implementation process, in RDMA communication, the first packet is usually used to establish a connection and initialize communication, and contains the necessary information required to establish a connection, such as the identifier of the target queue pair (Queue Pair), the request response (Acknowledge Request) flag, and the packet sequence number (Packet Sequence Number). The sender of the first packet needs to copy the data from the user space to the Socket Buffer in the kernel space, add a data packet header for encapsulation, and then send it to the network after a series of network protocol processing. The receiver needs to copy the received data packet from the network card buffer to the Socket Buffer, parse the data after processing by the network protocol, and copy it to the application buffer in the user space.

[0040] In RDMA communication, the middle packet is usually used to transmit the actual data payload, which contains the data to be sent or received by the application. Compared with the first packet, the processing of the middle packet is relatively simple because they do not need to establish a connection or initialize. The sender copies the data from the user space to the Socket Buffer in the kernel space and sends the data directly to the target memory address through the RDMA protocol. The receiver can also directly transfer data from the network card buffer to the application buffer in the user space through the RDMA protocol without passing through the Socket Buffer in the kernel space.

[0041] In RDMA communication, the tail packet is usually used to indicate the end of data transmission or to perform some cleanup work. For example, when an RDMA connection is closed, a tail packet needs to be sent to indicate the termination of the connection. The tail packet may contain some status information or instructions for cleanup work. Compared with the first and middle packets, the processing of the tail packet is also relatively simple.

[0042] The above scheme is adopted. This scheme first stores the cache address translation table in the DDR. When the message type is the first packet, the corresponding physical address is matched from the DDR and stored in the cache address translation table of the network card RAM memory. If the message type is an intermediate packet or a tail packet, it means that the first packet of the message has been processed. Therefore, the physical address is matched from the cache address translation table first. This scheme cooperates with the DDR and the RAM memory to store the correspondence between the virtual address and the physical address of the table lookup request in the network card RAM memory. When it is matched again, it can be processed quickly. On the one hand, the comprehensiveness of the memory address translation table is guaranteed by the DDR, and on the other hand, the matching processing efficiency is improved by the RAM memory.

[0043] In some embodiments of the present invention, in the step of storing the acquired correspondence between the virtual address and the physical address in the cache address translation table of the network card RAM memory, the RAM control module determines whether there is remaining storage space in the RAM memory. If so, the acquired correspondence between the virtual address and the physical address is stored in the cache address translation table of the network card RAM memory. If not, the correspondence between the virtual address and the physical address is not added to the RAM memory.

[0044] Adopting the above scheme, this scheme first determines whether there is remaining storage space in the RAM memory through the RAM control module. If there is no remaining storage space, the correspondence between the obtained virtual address and the physical address is stored in the cache address translation table of the network card RAM memory. The first packet is stored in the high-speed RAM memory to ensure the processing efficiency of the subsequent middle and tail packets.

[0045] In some embodiments of the present invention, in the step of obtaining the virtual address of the table lookup request and matching the corresponding physical address from the cache address translation table stored in the RAM memory based on the RAM control module, if the corresponding physical address is not matched from the cache address translation table stored in the RAM memory, then the corresponding physical address is matched from the memory address translation table stored in the DDR based on the DDR control module.

[0046] In the specific implementation process, if the corresponding physical address is not matched from the cache address translation table stored in the RAM memory, the corresponding physical address is matched from the memory address translation table stored in the DDR. The DDR stores a comprehensive memory address translation table to ensure the matching success rate.

[0047] In some embodiments of the present invention, the message type of the table lookup request includes a single table lookup request and a non-single table lookup request, wherein the message type of the non-single table lookup request includes a first packet, a middle packet, and a last packet. The steps of the method also include, if the message type is a single table lookup request, obtaining a virtual address of the single table lookup request, and matching the corresponding physical address from a memory address translation table stored in the DDR based on the DDR control module.

[0048] In a specific implementation process, a single table lookup request in RDMA (Remote Direct Memory Access) communication is a specific operation request used to find a specific data item in the remote memory.

[0049] With the above solution, if it is a single table lookup request, the correspondence between the same virtual address and the physical address is only used once. Therefore, there is no need to save the correspondence between the virtual address and the physical address in the single table lookup request into the RAM memory, thereby improving the utilization of the RAM memory.

[0050] In some implementations of the present invention, the network card further includes an address calculation module, and the method comprises the following steps: the address calculation module obtains a corresponding memory storage unit based on the acquired physical address.

[0051] In the specific implementation process, the main function of the DDR control module is to control the reading of DDR, process the read DDR data, complete the bit width conversion, and store the converted data in FIFO, waiting for the address calculation module to use it.

[0052] In some embodiments of the present invention, the method comprises the following steps: the RAM control module performs memory management on the RAM memory at preset management time intervals, and the memory management step comprises deleting the correspondence between the virtual address and the physical address stored in the RAM memory.

[0053] In the specific implementation process, the RAM control module performs memory management on the RAM memory at preset management time intervals, regularly manages the content stored in the RAM memory, deletes expired storage items, and ensures efficient use of the RAM memory.

[0054] In some embodiments of the present invention, the step of deleting the correspondence between the virtual address and the physical address stored in the RAM memory includes obtaining the last matching time of each correspondence between the virtual address and the physical address stored in the RAM memory, calculating a first time interval between the current time and the last matching time, and if the first time interval is greater than a preset first threshold time, deleting the corresponding correspondence between the virtual address and the physical address from the RAM memory.

[0055] By adopting the above scheme, for the correspondence between the virtual address and the physical address of each item stored in the RAM memory, if it has not been used for a long time, it means that the corresponding data transmission has ended or been interrupted, then the corresponding correspondence between the virtual address and the physical address is deleted from the RAM memory to ensure efficient use of the RAM memory.

[0056] In some embodiments of the present invention, the step of deleting the correspondence between the virtual address and the physical address stored in the RAM memory also includes obtaining the correspondence between the virtual address and the physical address that was last matched as a tail packet in the RAM memory, and obtaining the last matching time of the tail packet, calculating a second time interval between the current time and the last matching time of the tail packet, and if the second time interval is greater than a preset second threshold time, deleting the corresponding correspondence between the virtual address and the physical address from the RAM memory.

[0057] By adopting the above scheme, for the correspondence between the virtual address and the physical address that was last matched as the tail packet stored in the RAM memory, if it has not been used for a long time, it means that the corresponding data transfer has ended, then the corresponding correspondence between the virtual address and the physical address is deleted from the RAM memory to ensure efficient use of the RAM memory.

[0058] In some implementations of the present invention, in the step of deleting the correspondence between the virtual address and the physical address stored in the RAM memory, the second threshold time is smaller than the first threshold time.

[0059] By adopting the above scheme, the correspondence between the virtual address and the physical address of the last packet matched as the tail packet indicates that the data transmission corresponding to the match has a high probability of being completed. Therefore, the second threshold time is smaller than the first threshold time to further ensure efficient use of the RAM memory.

[0060] The present invention provides a method of combining the internal storage resources of a chip with an external DDR to realize MTT. A large amount of MTT table item information is stored in the DDR, and the table item information of the large data packet operation is stored in the storage resources inside the chip. When the operation of the large data packet is completed, it will be released for use by other table item information. In this way, for the table item information requested by the large data packet, the subsequent low-latency VA to PA conversion can be realized at the cost of a read and write delay from the DDR to the storage resources (RAM) inside the chip.

[0061] The beneficial effects of this program include:

[0062] 1. This solution uses DDR to increase the space used to store MTT entries from MB to GB, which is about a 1,000-fold increase.

[0063] 2. This solution can reduce latency by reducing access to DDR by writing table entries in DDR into the internal storage resources of the chip.

[0064] An embodiment of the present invention also provides an RDMA memory translation table application system, the system includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, when the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0065] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps implemented by the aforementioned RDMA memory translation table application method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0066] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0067] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0068] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0069] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for applying a memory translation table of RDMA, characterized in that: The method is applied to a network card connected to a DDR, the network card comprising a random order cache module, a DDR control module and a RAM control module, and the steps of the method include: The network card receives a table lookup request, and the out-of-order cache module determines the message type of the table lookup request; If the message type is the first packet, obtain the virtual address of the table lookup request, and match the corresponding physical address from the memory address translation table stored in the DDR based on the DDR control module, and store the obtained correspondence between the virtual address and the physical address in the cache address translation table of the network card RAM memory; If the message type is an intermediate packet or a tail packet, the virtual address of the table lookup request is obtained, and the corresponding physical address is matched from the cache address translation table stored in the RAM memory based on the RAM control module.

2. The RDMA memory translation table application method according to claim 1, characterized in that: In the step of storing the obtained correspondence between the virtual address and the physical address in the cache address translation table of the network card RAM memory, the RAM control module determines whether the RAM memory has remaining storage space. If so, the obtained correspondence between the virtual address and the physical address is stored in the cache address translation table of the network card RAM memory. If not, the correspondence between the virtual address and the physical address is not added to the RAM memory.

3. The RDMA memory translation table application method according to claim 1, characterized in that: In the step of obtaining the virtual address of the table lookup request and matching the corresponding physical address from the cache address translation table stored in the RAM memory based on the RAM control module, if the corresponding physical address is not matched from the cache address translation table stored in the RAM memory, then the corresponding physical address is matched from the memory address translation table stored in the DDR based on the DDR control module.

4. The RDMA memory translation table application method according to claim 1, characterized in that: The message types of the table lookup request include a single table lookup request and a non-single table lookup request, wherein the message types of the non-single table lookup request include a first packet, a middle packet and a tail packet. The method further includes, if the message type is a single table lookup request, obtaining a virtual address of the single table lookup request, and matching a corresponding physical address from a memory address translation table stored in the DDR based on the DDR control module.

5. The RDMA memory translation table application method according to claim 1, characterized in that: The network card further includes an address calculation module, and the method comprises the following steps: the address calculation module obtains a corresponding memory storage unit based on the acquired physical address.

6. The RDMA memory translation table application method according to any one of claims 1 to 5, characterized in that: The method comprises the following steps: the RAM control module performs memory management on the RAM memory at every preset management time, and the memory management step comprises deleting the correspondence between the virtual address and the physical address stored in the RAM memory.

7. The RDMA memory translation table application method according to claim 6, characterized in that: The step of deleting the correspondence between the virtual address and the physical address stored in the RAM memory includes obtaining the last matching time of each correspondence between the virtual address and the physical address stored in the RAM memory, calculating a first time interval between the current time and the last matching time, and if the first time interval is greater than a preset first threshold time, deleting the corresponding correspondence between the virtual address and the physical address from the RAM memory.

8. The RDMA memory translation table application method according to claim 7, characterized in that: The step of deleting the correspondence between the virtual address and the physical address stored in the RAM memory also includes obtaining the correspondence between the virtual address and the physical address that was last matched as a tail packet in the RAM memory, and obtaining the last matching time of the tail packet, calculating a second time interval between the current time and the last matching time of the tail packet, and if the second time interval is greater than a preset second threshold time, deleting the corresponding correspondence between the virtual address and the physical address from the RAM memory.

9. The RDMA memory translation table application method according to claim 8, characterized in that: In the step of deleting the correspondence between the virtual address and the physical address stored in the RAM memory, the second threshold time is smaller than the first threshold time.

10. A memory translation table application system for RDMA, characterized in that: The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions. The processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method as described in any one of claims 1 to 9.

Images