A memory address translation table compression method in RDMA ROCE

By dividing the physical address space into multiple TB segments and establishing TB segment tables, the problem of excessive memory translation table occupies in the RDMA ROCE network card is solved, and 50% memory savings and higher RDMA memory region support is achieved.

CN117851286BActive Publication Date: 2024-11-29CHINA TELECOM CLOUD TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311708482.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-13
Publication Date
2024-11-29
Estimated Expiration
2043-12-13

AI Technical Summary

Technical Problem

The size of the memory translation table in the RDMA ROCE network card has become a key issue in system resource usage. Especially when creating a large number of RDMA memory regions, the memory usage cannot be effectively compressed, resulting in waste of resources.

Method used

By dividing the physical address space into multiple 1TB TB segments and establishing a TB segment table, encode the 32b memory translation table entry into a 4b TB segment index and a 28b TB segment offset, the address translation table is compressed, saving 50 % continuous physical memory footprint.

Benefits of technology

It effectively reduces the physical memory usage of memory translation table, supports the creation of more RDMA memory regions, and improves system resource utilization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117851286B_ABST
    Figure CN117851286B_ABST
Patent Text Reader

Abstract

The present invention discloses a memory address translation table compression method in RDMA ROCE, which comprises the following specific steps: establishing an RDMA ROCE network card address translation architecture; a translation process from a virtual address to a PA; compressing a memory translation table entry, wherein the memory translation table entry is an 8-byte entry; establishing an address translation table; a physical address compression scheme based on a TB segment table; dividing the physical address space into a plurality of TB segments of a size of 1TB; establishing a TB segment table, and writing the base address of the TB segment into the table; dividing a 32-byte memory translation table entry into two segments; changing a physical frame number in the memory translation table entry; shifting the physical frame number left by 12 bits and adding the offset in the virtual address to obtain a final PA address. The present invention utilizes a memory address translation table compression method in RDMA ROCE. The method adopted is to divide the system RAM into a maximum of 16 1TB segments, and establish a TB segment table. The 32-bit memory translation table entry is encoded into a 4-bit TB segment index and a 28-bit TB segment offset. During address translation, the TB segment index is used to index the TB segment table to obtain a 1TB base address, and the TB segment offset is added to obtain a physical frame number, thereby saving 50% of continuous physical memory occupancy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of address translation, and in particular to a method for compressing a memory address translation table in an RDMA ROCE. Background Art

[0002] The RDMA memory translation table is used to translate the IOvirtualaddress sent by the ROCE NIC during DMA into a PA address. It serves the same purpose as the page table on the CPU side. Each time an RDMA memoryregion (MR) is created, a contiguous area of ​​the MMT table is occupied. When a large number of MRs are created, the MMT table itself will be very large. Furthermore, it requires contiguous physical memory and must be reserved. It cannot be reused by other programs. As a result, the size of the memory translation table becomes a key issue in terms of system resource usage, making it difficult to compress the memory translation table in the RDMA ROCE NIC to reduce the contiguous physical memory usage. Summary of the invention

[0003] The object of the present invention is to provide a method for compressing a memory address translation table in an RDMA ROCE to solve the problems raised in the above background technology.

[0004] To achieve the above object, the present invention provides the following technical solution: a method for compressing a memory address translation table in an RDMA ROCE, comprising the following specific steps:

[0005] S1: Establish RDMA ROCE network card address translation architecture;

[0006] S2: translation process from virtual address to PA;

[0007] S3: Compressed memory translation table entry. A memory translation table entry is an 8-byte entry.

[0008] S4: Create an address translation table;

[0009] S5: Physical address compression scheme based on TB segment table;

[0010] S5.1: Divide the physical address space into multiple 1TB segments;

[0011] S5.2: Create a TB segment table and write the base address of the TB segment into this table;

[0012] S5.3: Divide the 32-byte memory translation table entry into two segments.

[0013] S5.4: Change the physical frame number in the memory translation table entry.

[0014] Preferably, the RDMA ROCE network card address translation architecture established in S1 is that when the ROCE network card accesses the memory, the first thing it gets is the IOvirtual address, which is a user-defined virtual address (virtualaddress), which is translated into a physicaladdress (PA) through a memory protection table (memory protection table) and a memorytranslation table (memory translation table). The memoryprotection table mainly checks the local key and the remote key to see whether they have access rights, and the memorytranslation table contains the final PA.

[0015] Preferably, the translation process from the virtual address to the PA in S2 is that when RDMA accesses the memory, the memory protection table is first indexed with the local key (local key) or the remote key (remote key) to obtain the Mkey entry, which includes a base address pointing to the memory translation table. The base address points to a series of physically continuous memory translation table entries, each entry including an 8-byte physical frame number (physical frame number). Then, the virtual frame number (virtual frame number) of the virtual address is taken as the index to index the memory translation table starting with the base to obtain a specific physical frame number. The physical frame number is shifted left by 12 bits and added to the lower 12 bits of the virtual address, i.e., the page offset, to obtain the final PA.

[0016] Preferably, the translation process from virtual address to PA in S2 includes:

[0017] Use the local key / remote key to index the memory protection table to get the Mkey entry, and then get the memory translation tablebase.

[0018] Use the virtual frame number in the virtual address to get the memory translation table entry based on the base index memory translation table;

[0019] Use the TB segment index in the memory translation tableentry to index the TB segment table to get the TB segment base.

[0020] Shift the obtained TB segment base left by 28 bits and add it to the offset in the memory translation table entry to get the physical frame number.

[0021] The physical frame number is shifted left by 12 bits and added to the offset in the virtual address to get the final PA address.

[0022] Preferably, the compressed memory translation table entry in S3 contains a 64-bit physical address, which is a physical frame number, the lower 12 bits of which are unused. The memory translation table is indexed using an offset, requiring that the entire memory translation table be a contiguous block of physical memory.

[0023] Preferably, the address translation table established in S4 is an address translation table entry occupying 8 bytes. When the translated virtual address range is large, the address translation table itself will occupy a large memory. Translating 1TB of virtualaddress space requires (1TB / 4KB)x8B=2GB of memory space.

[0024] Preferably, the physical address space is divided into multiple TB segments of 1 TB in S5.1. The physical address space is large, but the space actually occupied by the RAM is small, and a maximum of 16 TB segments are used to describe the entire physical memory.

[0025] Preferably, a TB segment table is established in S5.2, and the base address of the TB segment is written into this table, so that the size of each entry is 1 byte (1 TB alignment). Since there are 16 table entries, the size of this table is 16 bytes, which is stored as an on-chip register. Accessing this table does not incur additional overhead.

[0026] Preferably, S5.3 divides the 32-bit memory translation table entry into two segments: the first segment is a 4-bit TB segment index, which is used to index the TB segment table (up to 16 entries); the second segment is a 28-bit TB segment offset, which represents the offset within 1TB in units of 4K pages (12 bits). 28+12=40 bits represents the offset within 1TB (40 bits).

[0027] Preferably, the step of changing the physical frame number in the memory translation table entry in S5.4 is to retrieve the memory translation table entry, index the TBsegment table with the TB segment index therein, obtain the TB Segment base (1B), shift the TB segment base left by 28 bits, and add the TB segment offset (28b) in the memory translation table entry to obtain the final physical frame number, where the TBsegment table is only the size of two registers and is in hardware.

[0028] Technical effects and advantages of the present invention:

[0029] The present invention utilizes a memory address translation table compression method in RDMA ROCE, which adopts a method of dividing the system RAM into a maximum of 16 1TB segments, and establishing a TB segment table, encoding a 32b memory translation table entry into a 4b TB segment index and a 28b TB segment offset, and using the TB segment index to index the TB segment table to obtain a 1TB base address during address translation, and then adding the TB segment offset to obtain a physical frame number, thereby saving 50% of continuous physical memory occupancy. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flowchart of the present invention.

[0031] Figure 2 This is a diagram of the RDMA ROCE memory address translation table compression system of the present invention.

[0032] Figure 3 This is a diagram of the MMT table system of the present invention.

[0033] Figure 4 This is a system diagram of the MMT table compression process of the present invention.

[0034] Figure 5 This is a diagram of the translation system from virtual address to PA of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] The present invention provides Figure 1-5 A memory address translation table compression method in an RDMA ROCE is shown, comprising the following specific steps:

[0037] S1: Establish RDMA ROCE network card address translation architecture;

[0038] S2: translation process from virtual address to PA;

[0039] S3: Compressed memory translation table entry. A memory translation table entry is an 8-byte entry.

[0040] S4: Create an address translation table;

[0041] S5: Physical address compression scheme based on TB segment table;

[0042] S5.1: Divide the physical address space into multiple 1TB segments;

[0043] S5.2: Create a TB segment table and write the base address of the TB segment into this table;

[0044] S5.3: Divide the 32-byte memory translation table entry into two segments.

[0045] S5.4: Change the physical frame number in the memory translation table entry.

[0046] RDMA ROCE A compression method for the memory translation table in the network card. It can compress a 64-bit physical address into a 32-bit physical address, thereby saving 50% of continuous physical memory usage. The method used is to divide the system RAM into up to 16 1TB segments, and set up a TB segment table. The 32-bit memory translation table entry is encoded as a 4-bit TB segment index and a 28-bit TB segment offset. When translating addresses, the TB segment index is used to index the TB segment table to get the 1TB base address, and then the TB segment offset is added to get the physical frame number. RDMA (Remote Direct Memory Access) is a technology for directly accessing remote host memory in a computer network without going through the operating system's protocol stack, and can achieve high-performance, low-latency data transmission. RDMA over Converged Ethernet (RoCE) is an Ethernet-based RDMA technology that allows data transmission over Ethernet using the RDMA protocol. RoCE provides a communication protocol on the Ethernet data link layer, which enables RDMA to be implemented on traditional Ethernet. function, thus overcoming the limitations of Ethernet in terms of low latency and high bandwidth requirements. RoCE uses a remote direct memory access protocol similar to InfiniBand to achieve direct access to remote host memory through data packets, thereby avoiding data copying and processing, and improving the efficiency of data transmission. RoCE technology is usually used in scenarios such as data centers and cloud computing. It can provide very high bandwidth and low latency data transmission, and is suitable for high-performance computing, storage and network interconnection. RoCE can support a variety of applications, such as distributed storage systems, network acceleration in virtualized environments, machine learning training, etc. RDMA overConverged Ethernet (RoCE) is a technology that uses Ethernet to achieve high-performance, low-latency data transmission. It uses a remote direct memory access protocol similar to InfiniBand and is suitable for scenarios such as data centers and cloud computing.

[0047] Furthermore, the RDMA ROCE network card address translation architecture established in S1 is that when the ROCE network card accesses the memory, it first obtains the IOvirtual address, which is a user-defined virtual address (virtual address), which is translated into physicaladdress (PA) by the memory protection table (memory protection table) and the memory translation table (memory translation table). The memory protection table mainly checks the local key and remote key to see if they have access rights. The memory translation table contains the final PA. The contents of the storage unit can be taken out or stored at will as needed, and the access speed is independent of the location of the storage unit. This type of memory will lose its storage content when power is off, so it is mainly used to store programs used for a short time. According to the working principle of the storage unit, the random access memory is divided into static random access memory and dynamic random access memory.

[0048] Furthermore, the translation process from virtual address to PA in S2 is that when RDMA accesses memory, the localkey (local key) or remote key (remote key) is first used to index the memory protection table to obtain Mkeyentry, which contains a base address pointing to the memory translation table. This base address points to a series of physically consecutive memory translation table entries. Each entry contains an 8-byte physicalframe number (physical frame number) page frame number. Then, the virtualframe number (virtual frame number) of the virtual address is taken as the index to index the memory translation table starting with the base to obtain a specific physical frame number. This physical frame number is shifted left by 12 bits and added to the lower 12 of the virtual address, that is, the page offset, to obtain the final PA.

[0049] Furthermore, the translation process from virtual address to PA in S2 includes:

[0050] Use the local key / remote key to index the memory protection table to get the Mkey entry, and then get the memory translation tablebase.

[0051] Use the virtual frame number in the virtual address to get the memory translation table entry based on the base index memory translation table;

[0052] Use the TB segment index in the memory translation tableentry to index the TB segmenttable to get the TB segment base.

[0053] Shift the obtained TB segment base left by 28 bits and add it to the offset in the memory translation table entry to get the physical frame number.

[0054] The physical frame number is shifted left by 12 bits and then added to the offset in the virtual address to get the final PA address.

[0055] memory translation table: Memory translation table, used by the RDMA network card to translate the virtual address into PA.

[0056] Furthermore, the compressed memory translation table entry in S3 contains a 64-bit physical address, which is a physical frame number. Its lower 12 bits are not used. The offset is used to index the memory translation table, requiring the entire memory translation table to be a continuous piece of physical memory. The address translation table created in S4 is an address translation table entry that occupies 8 bytes. When the range of the translated virtualaddress is large, the address translation table itself will occupy a large amount of memory. Translating 1TB of virtual address space requires (1TB / 4KB)x8B=2GB of memory space.

[0057] In particular, S5.1 divides the physical address space into multiple 1TB segments. The physical address space is large, but the actual RAM space is small. We make a reasonable assumption here that the RAM space of the entire system does not exceed 16 TB, so we can use a maximum of 16 TB segments to describe the entire physical memory;

[0058] In S5.2, a TB segment table is created and the base address of the TB segment is written into the table. The size of each entry is 1 byte (1 TB alignment). Since there are 16 table entries, the size of the table is 16 bytes and stored as an on-chip register. Accessing the table does not incur additional overhead.

[0059] S5.3 divides the 32-byte memory translation table entry into two parts. The first part is the 4-byte TBsegment index, which is used to index the TB segment table (up to 16 entries). The second part is the 28-byte TBsegment offset, which represents the offset within 1TB in units of 4K pages (12 bytes). 28 + 12 = 40 bytes represents the offset within 1TB (40 bytes).

[0060] The physical frame number in S5.4 is changed by obtaining the memory translation table entry, indexing the TB segment table with the TB segment index, obtaining the TB Segment base (1B), shifting it left 28 bits, and adding the TB segment offset (28b) in the memory translation table entry to get the final physical frame number. The TB segment table is only two registers in size and is in hardware.

[0061] Example

[0062] Implementation of memory translation table compression in ROCE network cards, such as Figure 2 As shown;

[0063] The software and hardware framework of the RDMA ROCE network card: The software is divided into kernel-mode drivers and user-mode drivers. The control plane commands (such as register MR) are first sent from the user state to the kernel state, and then the kernel-mode driver programs and configures the hardware table. The data plane commands (such as post send) are directly sent from the user state to the hardware through QP (queue pair = send queue + receivequeue) without passing through the kernel state. The hardware is divided into a programmable table part and an execution engine part. The former accepts the configuration of the kernel driver, and the latter actively reads the WQE request sent by the software in the QP, parses the WQE, and translates the virtualaddress therein to execute DMA to generate the final message and send it out.

[0064] Control plane flow of address translation table compression: When registering a memory region (MR) in user mode, the RDMA kernel driver is entered. At this time, the kernel driver first divides the physical address space where the RAM is located into 16 1TB segments according to the system RAM memory layout, and then writes their base addresses into the TB segment table. Then, the memory translation table is programmed, and a free memory translation table entry is selected. The address of the physical page corresponding to each virtual page is written into the memory translation table entry. At this time, address translation table compression is adopted, that is, instead of writing directly into the PA, the address of the physical page is divided into a 4-bit TB segment index and a 28-bit TBsegment offset, which are combined into a 32-bit value and written into the memory translation table. Finally, the memory protection table is programmed, and a free MKey entry is selected. The starting position of the free memory translation table entry selected in the previous step is written into the MKey entry. The index position of the selected MKey entry in the memory protection table is the local key returned by the final registration of the MR.

[0065] Data plane flow using the compressed address translation table: After the upper-layer application calls ibv_post_send, the user-mode driver puts a send WQE in the QP, which represents a send request and contains several key information<localvirtual address, local key, length> , which means sending data from a local virtual memory to the remote end. The hardware engine of the ROCE network card reads the WQE in the QP and parses it. Ultimately, the DMA engine is used to move the data from the memory to the message buffer to generate a message that can be transmitted on the network. However, before the DMA moves the data, the virtual address in the WQE needs to be translated into PA. At this time, the address translation engine uses the local key to index the memory protection table to obtain the MKey entry and then the base address of the memory translation table. The virtual frame number obtained by shifting the virtual address right by 12 bits is then used to index the memory translation table to obtain the memory translation table entry. The TB segment index and TB segment offset are then used to index the TB segment table to obtain the base address of the TB segment where the translated physical page is located. The combination of TB segment base (40b-47b) + TB segment offset (12b-39b) + page offset (0b-11b) gives the PA physical address obtained by translating the virtual address.

[0066] The current page table uses a traditional 4-level page table, and each pfe entry occupies 64b. This is based on a very common scenario, but does not take into account that most existing servers will not use 64b of physical address space. The method proposed in this patent can compress half of the physical memory occupied by the page table, and on the basis of the same physical memory occupancy, it can support 200% of the original MR number.

[0067] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A memory address translation table compression method in RDMA ROCE, It is characterized in that The specific steps include: S1: Establish RDMA ROCE network card address translation architecture; S2: translation process from virtual address to PA; S3: Compressed memory translation table entry. A memory translation table entry is an 8-byte entry. S4: Create an address translation table; S5: Physical address compression scheme based on TB segment table; S5.1: Divide the physical address space into multiple 1TB segments; S5.2: Create a TB segment table and write the base address of the TB segment into this table; S5.3: Divide the 32-byte memory translation table entry into two segments. S5.4: Change the physical frame number in the memory translation table entry. The translation process from virtual address to PA in S2 includes: Use the local key to index the memory protection table to get the Mkey entry, and then get the memory translation table base; Use the virtual frame number in the virtual address to obtain the memory translation table entry based on the base index. Use the TB segment index in the memory translation table entry to index the TB segment table to obtain the TB segment base. Shift the TB segment base left by 28 bits and add it to the offset in the memory translation table entry to get the physical frame number. The physical frame number is shifted left by 12 bits and then added to the offset in the virtual address to get the final PA address. S5.3 divides the 32-byte memory translation table entry into two segments: the first segment is the 4-byte TBsegment index, which is used to index the TB segment table. The second segment is the 28-byte TB segment offset, which represents the offset in 4K pages within 1TB. 28 + 12 = 40 bits represent the offset within 1TB. The method for changing the physical frame number in the memory translation table entry in S5.4 is to retrieve the memory translation table entry, index the TB segment table with the TB segment index therein, obtain the TB Segment base, shift it left 28 bits, and add the TB segment offset in the memory translation table entry to get the final physical frame number. The TB segment table is only the size of two registers and is in hardware.

2. According to claim 1, a memory address translation table compression method in RDMA ROCE, It is characterized in that The RDMA ROCE network card address translation architecture established in S1 is ROCE. When the network card accesses the memory, it first obtains the IOvirtual address, which is a user-defined virtual address. It is translated into a physical address through the memory protection table and the memory translation table. The memory protection table verifies the local key and the remote key to see if they have access rights. The memory translation table contains the final PA.

3. According to the method for compressing a memory address translation table in RDMA ROCE according to claim 1, It is characterized in that The translation process from virtual address to PA in S2 is that when RDMA accesses memory, the local key is first used to index the memory protection table to obtain the Mkey entry, which contains a base address pointing to the memory translation table. This base address points to a series of physically consecutive memory translation table entries, each of which contains an 8-byte physical frame number. The virtual frame number of the virtual address is then used as the index to index the memory translation table starting with the base to obtain a specific physical frame number. The physical frame number is shifted left by 12 bits and added to the lower 12 bits of the virtualaddress, i.e., the page offset, to obtain the final PA.

4. The method for compressing a memory address translation table in an RDMA ROCE according to claim 1, It is characterized in that The compressed memory translation table entry in S3 contains a 64-bit physical address, which is a physical frame number. The lower 12 bits are unused. The memory translation table is indexed using an offset, which requires that the entire memory translation table be a contiguous block of physical memory.

5. The method for compressing a memory address translation table in an RDMA ROCE according to claim 1, It is characterized in that The address translation table established in S4 is an address translation table entry that occupies 8 bytes. When the translated virtual address range is large, the address translation table itself will occupy a large memory. Translating 1TB of virtual address space requires 1TBx8B=2GB of memory space.

6. The method for compressing a memory address translation table in an RDMA ROCE according to claim 1, It is characterized in that In S5.1, the physical address space is divided into multiple TB segments of 1 TB in size. The physical address space is large, but the space actually occupied by the RAM is small. A maximum of 16 TB segments are used to describe the entire physical memory.

7. The method for compressing a memory address translation table in an RDMA ROCE according to claim 1, It is characterized in that In S5.2, a TB segment table is established and the base address of the TB segment is written into the table. The size of each entry is 1 byte. Since there are 16 table entries, the size of the table is 16 bytes. It is stored as an on-chip register and accessing the table does not generate additional overhead.

Citation Information

Patent Citations

  • Apparatus, system, and method for solid-state storage as cache for high-capacity, non-volatile storage

    CN101622606A

  • Memory protection method and system and network interface controller

    CN103488588A