A low-latency multi-channel RDMA data transfer method and system

Abstract

This disclosure relates to a low-latency multi-channel RDMA data transmission method and system. Based on preset memory requirements, a contiguous physical memory region is reserved, and corresponding contiguous physical memory blocks are allocated to RDMA transmission channel queue pairs during the RDMA initialization phase. The initial physical addresses of the contiguous physical memory blocks are mapped to obtain initial virtual addresses. During the RDMA memory registration phase, permission description items are generated based on each RDMA transmission channel queue pair, the initial physical address, and the initial virtual address. All permission description items are written to a field-programmable gate array (FPGA), and the work requests are quickly converted into work queue item requests based on the work requests issued by the application software program and a preset address mapping relationship. The FPGA performs permission verification processing on the work queue item requests, and after confirming that the permission verification of the work queue item requests is successful, data transmission processing is performed based on the work queue item requests. This effectively meets the requirements for low-power and high-performance real-time data transmission.

Description

Technical Field

[0001] The embodiments in this specification belong to the field of high-performance network communication technology, and in particular relate to a low-latency multi-channel RDMA data transmission method and system. Background Technology

[0002] Remote Direct Memory Access (RDMA) technology allows network devices to directly access host memory without the involvement of the Central Processing Unit (CPU), thereby improving data transfer efficiency. Currently, in traditional RDMA implementations, before issuing work queue entries and executing data transfer, it is typically necessary to query the Memory Translation Table (MTT) to translate virtual addresses into physical addresses and perform permission verification using the Memory Protection Table (MPT).

[0003] However, for resource-constrained devices such as edge computing and embedded systems, memory translation tables and memory protection tables require a significant amount of host memory. When network interface card (NIC) resources are limited, frequently migrating memory translation tables and memory protection tables to the NIC can easily lead to increased resource consumption and latency. Furthermore, in concurrent transmission scenarios with multi-queue pairs (multi-QP), it can also lead to a decrease in cache hit rate and a reduction in overall throughput, thereby exacerbating address translation latency. In addition, in high-throughput, continuous data transmission scenarios, frequent memory management can also bring additional overhead and memory fragmentation, thus affecting data transmission performance. Summary of the Invention

[0004] The embodiments of this disclosure present a low-latency multi-channel RDMA data transmission method and system.

[0005] In a first aspect of this disclosure, a low-latency multi-channel RDMA data transfer method is provided. The method includes a processor reserving a contiguous physical memory region based on preset memory requirements, and during the RDMA initialization phase, allocating corresponding contiguous physical memory blocks to multiple RDMA transfer channel queue pairs based on the contiguous physical memory region. The method further includes the processor mapping the initial physical addresses of each contiguous physical memory block to obtain initial virtual addresses, and during the RDMA memory registration phase, generating permission description entries based on each RDMA transfer channel queue pair, the corresponding initial physical address, and the initial virtual address. The method also includes the processor writing all permission description entries to a field-programmable gate array (FPGA), and converting the work request into a work queue item request based on the work request issued by the application software program and a preset address mapping formula. Furthermore, the method includes the FPGA performing permission verification processing on the work queue item request based on all permission description entries, and after determining that the permission verification of the work queue item request has passed, performing data transfer processing based on the work queue item request.

[0006] In a second aspect of this disclosure, a low-latency multi-channel RDMA data transmission system is provided. The system includes a memory allocation module configured to have a processor reserve contiguous physical memory regions based on preset memory requirements, and during the RDMA initialization phase, allocate corresponding contiguous physical memory blocks to multiple RDMA transmission channel queue pairs based on the contiguous physical memory regions. The system also includes a permission description module configured to have the processor map the initial physical addresses of each contiguous physical memory block to obtain initial virtual addresses, and during the RDMA memory registration phase, generate permission descriptions based on each RDMA transmission channel queue pair, the corresponding initial physical address, and the initial virtual address. The system further includes a request translation module configured to have the processor write all permission descriptions to a field-programmable gate array (FPGA), and convert work requests into work queue item requests based on work requests issued by application software programs and preset address mapping relationships. Furthermore, the system includes a permission verification module configured to have the FPGA perform permission verification processing on the work queue item requests based on all permission descriptions, and after determining that the permission verification of the work queue item request has passed, perform data transmission processing based on the work queue item request.

[0007] In a third aspect of this disclosure, a machine-readable storage medium is provided. The machine-readable storage medium stores machine-executable instructions, wherein when the machine-executable instructions are executed on a computer and a field-programmable gate array (FPGA), the computer and the FPGA perform to implement the method provided according to a first aspect of this disclosure.

[0008] In a fourth aspect of this disclosure, an RDMA hardware device is provided. The electronic device includes a processor, a field-programmable gate array (FPGA), and a memory associated with the processor and the FPGA, the memory storing program instructions that, when read and executed by the processor and the FPGA, are executed to implement the method provided according to a first aspect of this disclosure.

[0009] It should be understood that the description in the Summary of the Invention section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 A flowchart illustrating a prior art RDMA data transmission process based on some embodiments of this disclosure is shown;

[0012] Figure 2 A flowchart illustrating a low-latency multichannel RDMA data transmission method according to some embodiments of this disclosure is shown;

[0013] Figure 3 A schematic diagram of the structure of a permission description item according to some embodiments of this disclosure is shown;

[0014] Figure 4 A schematic diagram illustrating the storage of a permission description item according to some embodiments of this disclosure is shown;

[0015] Figure 5 This diagram illustrates a multichannel RDMA transport channel queue pair resource allocation according to some embodiments of the present disclosure;

[0016] Figure 6 A flowchart illustrating a low-latency multichannel RDMA data transmission process according to some embodiments of this disclosure is shown;

[0017] Figure 7 A block diagram of a low-latency multi-channel RDMA data transmission system according to some embodiments of the present disclosure is shown; and

[0018] Figure 8 A block diagram of an RDMA hardware device according to some embodiments of the present disclosure is shown. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0020] The terms “comprising” and “having”, and any variations thereof, in this specification, claims, and the foregoing drawings are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. Depending on the context, the word “if” as it applies herein may be interpreted as “when”, “when”, “in response to determination”, or “in response to detection”.

[0021] As mentioned above, in traditional RDMA implementations, before issuing work queue entries and executing data transfer, it is typically necessary to translate virtual addresses into physical addresses by querying the Memory Translation Table (MTT) and to perform permission verification using the Memory Protection Table (MPT). The MTT records the mapping relationship between virtual addresses and corresponding physical addresses, generally using 4KB pages. Setting larger page sizes (e.g., 2MB) cannot guarantee the availability of the required memory space and also poses a system security risk. The MPT records the permission information for the memory block at the requested physical address, including the available size of the memory block, the length of the starting virtual address, read / write permission descriptions, and the location of the corresponding MTT storage table.

[0022] However, for resource-constrained devices such as edge computing and embedded systems, traditional RDMA technology has significant shortcomings in terms of resource consumption, address translation latency, multi-queue concurrency performance, and dynamic management overhead, making it difficult to operate efficiently. Specifically, these shortcomings are as follows:

[0023] 1. Resource overhead and performance bottlenecks of memory translation tables and memory protection tables: 1) Because the memory translation table needs to maintain the mapping from virtual addresses to physical addresses, and the memory protection table needs to store memory block permission information, it consumes a large amount of host memory; 2) When network card resources are limited, the memory translation table and memory protection table need to be frequently migrated from host memory to the network card, resulting in increased resource consumption and latency; 3) Before each data transmission, the memory translation table needs to be queried to convert the virtual address to the physical address, and the permission is verified through the memory protection table, introducing a significant latency.

[0024] 2. Memory registration and management overhead: 1) Memory registration and deregistration in traditional RDMA technology are dynamic processes, and frequent operations will increase additional overhead and memory fragmentation; 2) Although traditional RDMA technology uses dynamic management to improve flexibility, frequent memory management will significantly affect performance in high-throughput and continuous data transfer scenarios.

[0025] 3. Performance bottlenecks in multi-queue concurrent transmission scenarios: 1) When multiple queues are transmitted concurrently, random access to the memory translation table can easily lead to cache invalidation, which in turn exacerbates address translation latency; 2) When multiple queues access the memory translation table at the same time, it may also cause resource contention, which in turn reduces the overall throughput.

[0026] Please see Figure 1 The diagram illustrates a prior art RDMA data transmission flowchart of some embodiments of this disclosure. (See also...) Figure 1 As shown, the hardware devices used in the RDMA data transmission process 100 based on existing technology mainly include processors, memory, and network card hardware (of course, it also includes front-end sensors and other structures, but is not limited to these). Among them, the memory is used to store multi-queue management (i.e., QP queue management), memory translation table (i.e., MTT table), and memory protection table (i.e., MPT table); the network card hardware is used to store a portion of the memory translation table (i.e., MTT mini-cache) and a portion of the memory protection table (i.e., MPT mini-cache) determined based on historical data transmission records, in order to maximize the buffer hit rate.

[0027] Understandably, during the RDMA memory registration phase, the processor initiates a request to the memory to build the memory translation table and memory protection table. This involves constructing the memory translation table and memory protection table in memory, and then feeding this information back to the processor. However, because the memory stores the memory translation table and memory protection table with fine granularity (e.g., using 4KB page sizes for address paging), it can easily lead to large storage requirements and long processing times when data transfer demands are high. Therefore, it is common practice to cache a portion of the memory translation table and memory protection table in the network interface card (NIC) hardware, based on the processor's historical data transfer records, to maximize buffer hit rate.

[0028] Subsequently, during RDMA data transfer, the processor first issues an RDMA request to the network interface card (NIC) hardware. When the partial memory translation table in the NIC hardware contains the virtual address corresponding to the RDMA request, and the partial memory translation table contains the memory block permission information corresponding to the RDMA request, it indicates that the RDMA request has been buffered and hit in the NIC hardware. At this point, the NIC hardware can directly perform address translation processing on the virtual address corresponding to the RDMA request based on the partial memory translation table, and perform permission verification on the RDMA request based on the partial memory protection table. After the address translation processing and permission verification are successful, the NIC hardware completes the data transfer and feeds back the data to the processor. Data transmission is complete. When a portion of the memory translation table in the network interface card (NIC) hardware does not contain the virtual address corresponding to the RDMA request, and / or a portion of the memory translation table does not contain the memory block permission information corresponding to the RDMA request, it indicates that the RDMA request has not been cached in the NIC hardware. In this case, the processor needs to perform address translation processing on the virtual address corresponding to the RDMA request based on the complete memory translation table, and perform permission verification on the RDMA request based on the complete memory protection table. After the address translation processing and permission verification are successful, the memory will return the query result to the NIC hardware, and the NIC hardware will then complete the data transfer and report the data transmission completion to the processor.

[0029] It can be seen that existing optimization methods for traditional RDMA technology, such as memory pre-registration, mainly focus on the data buffer itself and have not yet completely solved the problems of address translation latency, multi-queue concurrency performance and dynamic management overhead. Moreover, for resource-constrained devices such as edge computing and embedded systems, it is still difficult to meet the real-time requirements of low power consumption and high performance.

[0030] To address this, embodiments of this disclosure propose a low-latency multi-channel RDMA data transmission method. The method includes a processor reserving a contiguous physical memory region based on preset memory requirements, and during the RDMA initialization phase, allocating corresponding contiguous physical memory blocks to multiple RDMA transmission channel queue pairs based on the contiguous physical memory region. The method further includes a processor mapping the initial physical addresses of each contiguous physical memory block to obtain initial virtual addresses, and during the RDMA memory registration phase, generating permission description entries based on each RDMA transmission channel queue pair, the corresponding initial physical address, and the initial virtual address. The method also includes a processor writing all permission description entries to a field-programmable gate array (FPGA), and converting work requests into work queue item requests based on work requests issued by the application software program and a preset address mapping formula. Furthermore, the method includes a FPGA performing permission verification processing on the work queue item requests based on all permission description entries, and after determining that the permission verification of the work queue item request has passed, performing data transmission processing based on the work queue item request.

[0031] In this way, the address mapping of the work request can be completed quickly during each data transmission based on the preset address mapping relationship. The field-programmable gate array is used to perform weight verification and data transmission processing based on the stored permission description items. It can be widely used in resource-constrained scenarios such as edge computing and embedded systems, and effectively meets the requirements of low power consumption and high performance real-time data transmission.

[0032] It should be noted that, compared with the prior art, the embodiments of this disclosure replace the network card hardware with a field-programmable gate array (FPGA), which allows data interaction to be unaffected by insufficient PCIe resources of the processor (especially in resource-constrained scenarios such as edge computing and embedded systems). Moreover, the processor can directly read or send data from the FPGA without affecting its speed and performance. Secondly, the FPGA is more customizable and has lower modification costs than the network card hardware. It can also store permission descriptions for verification, thereby effectively meeting the low-power and high-performance real-time data transmission requirements of resource-constrained scenarios such as edge computing and embedded systems.

[0033] Please see Figure 2 The flowchart illustrates a low-latency multi-channel RDMA data transfer method according to some embodiments of the present disclosure. Method 200 can be applied to an RDMA hardware device primarily composed of a processor, memory, and a field-programmable gate array (FPGA), and can be executed jointly by the processor and the FPGA. Furthermore, the RDMA hardware device can be a requesting end device, a responding end device, or both of the requesting and responding end devices for RDMA data transfer, and the data transfer types it performs can be RDMA read operations, RDMA write operations, RDMA transmit operations, and RDMA receive operations.

[0034] like Figure 2 As shown in block 202, method 200 allows the processor to reserve a contiguous physical memory region based on preset memory requirements, and during the RDMA initialization phase, allocate corresponding contiguous physical memory blocks to multiple RDMA transfer channel queues based on the contiguous physical memory region. Here, when the operating system running by the processor is in the startup or initialization phase, multiple contiguous physical memory regions can be reserved in memory based on preset memory requirements through kernel startup parameters or device tree configuration, etc., to provide a physically contiguous memory buffer for subsequent RDMA data transfer; then, after the operating system runs, when the RDMA driver running by the processor is in the startup or initialization phase, the RDMA driver can obtain the reserved contiguous physical memory region by calling the operating system interface, and allocate multiple available contiguous physical memory blocks to multiple RDMA transfer channel queues based on the contiguous physical memory region, and the contiguous physical memory blocks allocated to each RDMA transfer channel queue can meet the corresponding memory size requirements. Understandably, each RDMA transmission channel queue can be established by the application software program according to the RDMA connection requirements. By building static memory allocation logic for dynamic management in the RDMA driver, physical memory blocks can be quickly allocated and reclaimed. This not only ensures the efficient use of memory resources and provides stable and continuous physical memory resources for each of the multiple queues of RDMA communication, but also effectively avoids performance degradation caused by memory fragmentation.

[0035] In some implementations, when the processor reserves a contiguous physical memory region based on a preset memory requirement, the processor reserves the contiguous physical memory region in memory based on the preset memory requirement. Here, the preset memory requirement can be understood as the actual service transmission requirement during RDMA data transmission, such as including the size of a single memory block to be partitioned from memory and the total number of memory blocks, and the size of the contiguous physical memory region reserved in memory meets the preset memory requirement.

[0036] Subsequently, the processor can perform paging processing on contiguous physical memory regions based on preset paging requirements. This can be understood as follows: by setting up a unified memory management unit in the RDMA driver kernel mode region run by the processor, the processor can perform paging processing on contiguous physical memory regions based on preset paging requirements, such as paging contiguous physical memory in 4KB sizes, thereby ensuring that the physical address of the physical memory contained in each page is aligned with 4KB. Furthermore, the unified memory management unit can allocate multiple available contiguous physical memory blocks to multiple RDMA transfer channel queues based on contiguous physical memory regions. During the allocation process, the allocated contiguous physical memory blocks can also be marked as occupied to prevent duplicate allocation or access conflicts.

[0037] It should be noted that, in the embodiments of this disclosure, when the operating system running on the processor is in the startup or initialization phase, multiple contiguous physical memory regions can be reserved in the DDR memory of the field-programmable gate array (FPGA) based on preset memory requirements. This is to provide a physical address contiguous memory buffer for subsequent RDMA data transfer. Here, the DDR memory of the FPGA needs to reserve contiguous physical memory regions for RDMA data transfer and design corresponding configuration registers in advance during the design phase. After the operating system running on the processor identifies the existence of the FPGA through PCIe bus enumeration or other methods, it can allocate a host physical address space for the FPGA and map this host physical address space to the virtual address space of the RDMA driver kernel mode area. In this way, the processor can access the DDR memory and configuration registers of the FPGA by reading and writing to this virtual address space. It is understood that the difference between reserving contiguous physical memory regions in memory and reserving contiguous physical memory regions in the DDR memory of the FPGA is that the former can be directly accessed by the RDMA driver through virtual addresses, while the latter requires the use of address mapping relationships because the virtual address and physical address of the memory are assigned the same value.

[0038] In some implementations, the low-latency multi-channel RDMA data transmission method 200 further includes the processor determining whether any RDMA transmission channel queue pair has been destroyed. Here, the application software running on the processor can request the destruction of a certain RDMA transmission channel queue pair according to actual needs. At this time, the RDMA driver can be used to check whether the work request corresponding to the RDMA transmission channel queue pair has been processed and simultaneously notify the field-programmable gate array (FPGA) to stop receiving new work queue item requests corresponding to the RDMA transmission channel queue pair and complete the work queue item request being processed, through register writing or other means. Then, after determining that the work request corresponding to the RDMA transmission channel queue pair has been processed and the FPGA has completed the work queue item request being processed, the RDMA driver can determine that the RDMA transmission channel queue pair has been destroyed; otherwise, it can determine that the RDMA transmission channel queue pair has not yet been destroyed.

[0039] Subsequently, in response to the determination that any RDMA transfer channel queue pair has been destroyed, the processor reclaims the contiguous physical memory blocks allocated to the RDMA transfer channel queue pair into the contiguous physical memory region. Here, the RDMA driver, in response to the determination that any RDMA transfer channel queue pair has been destroyed, can mark the corresponding permission description entry in the FPGA as invalid (or modify the remote key and local key in the permission description entry to zero) through PCIe bus write operations, etc., to ensure that even if delayed network packets arrive subsequently, the FPGA's permission verification will fail. Furthermore, the aforementioned unified memory management unit can also release the contiguous physical memory blocks allocated to the RDMA transfer channel queue pair by calling the release interface (or mark the corresponding contiguous physical memory blocks as free) and reclaim them into the contiguous physical memory region. In this way, multiple small blocks of physical memory can be dynamically merged into longer contiguous physical address blocks, thereby improving the efficiency and continuity of subsequent memory allocation.

[0040] In block 204, method 200 allows the processor to map the initial physical addresses of each contiguous physical memory block to obtain initial virtual addresses. During the RDMA memory registration phase, permission description entries are generated based on each RDMA transfer channel queue pair, the corresponding initial physical address, and the initial virtual address. Here, a memory usage management unit is set up in the user-mode region of the RDMA driver run by the processor to map the initial physical addresses of each contiguous physical memory block based on a preset mapping mechanism between physical memory block addresses and virtual addresses, thereby obtaining the corresponding initial virtual addresses. It is understood that the preset mapping mechanism between physical memory block addresses and virtual addresses can be provided by the aforementioned unified memory management unit. This preset mapping mechanism uses a fixed linear mapping method to ensure that the physical address of each physical memory block in each contiguous physical memory block has a corresponding virtual address, and that each virtual address increments with the allocation order of the corresponding physical memory block, thereby avoiding complex address translation processes and improving system operating efficiency.

[0041] In one example, the initial physical address of each contiguous physical memory block can be understood as the physical address of each allocated physical memory block. For instance, taking a contiguous physical memory block with a physical address range of 0x800000000-0x840000000 and a size of 1GB as an example, and managing 1GB in 4KB increments, the initial physical address could include the physical address of the first physical memory block (0x800000000), the physical address of the second physical memory block (0x800000000+4096), and so on, up to the physical address of the last physical memory block (0x84000000-4096). Correspondingly, the initial virtual address of each contiguous physical memory block can be understood as the virtual address corresponding to each allocated physical memory block. For example, it could include the virtual address of the first physical memory block (VA), the virtual address of the second physical memory block (VA+4096), and so on, up to the virtual address of the last physical memory block (VA+(1GB-4KB)).

[0042] It should be noted that the memory usage management unit can also apply for data cache space from the aforementioned unified memory management unit according to the transmission requirements of each RDMA transmission channel queue pair. The unified memory management unit then allocates multiple available contiguous physical memory blocks to the RDMA transmission channel queue pair based on the contiguous physical memory region. After multiple available contiguous physical memory blocks have been allocated to the RDMA transmission channel queue pair, the memory usage management unit uses a preset mapping mechanism between physical memory block addresses and virtual addresses to map the initial physical addresses of the multiple available contiguous physical memory blocks allocated to the RDMA transmission channel queue pair to obtain the corresponding initial virtual addresses.

[0043] Subsequently, when the RDMA driver running by the processor is in the memory registration phase, the memory usage management unit can generate permission description entries based on each RDMA transmission channel queue pair, the corresponding initial physical address, and the initial virtual address. Each RDMA transmission channel queue pair has a corresponding permission description entry. Here, the permission description entry is typically 32 bytes (32B) and is used to describe the access permissions to the memory regions corresponding to each RDMA transmission channel queue pair. This includes, for example, the local key, remote key, virtual base address, physical base address, available memory size, and memory permission description. The local key, remote key, and memory permission description information can be preset manually. The virtual base address can be the first physical address in the initial physical address set, the physical base address can be the first virtual address in the initial virtual address set, and the available memory size can be the memory size corresponding to the initial physical address.

[0044] Please see Figure 3 The diagram illustrates the structure of a permission description item according to some embodiments of the present disclosure. For example...Figure 3 As shown, the permission description item 300 may include the protection domain number, local key, remote key, memory region number, virtual base address, physical base address, available memory size, and memory permission description. Each permission description information has a corresponding bit width and size.

[0045] It should be noted that when the RDMA driver running on the processor is in the memory registration phase, the RDMA driver also needs to perform software authentication of the local device in advance using a local key. For example, taking an RDMA write operation as an example, the requesting device for RDMA data transfer can use the local key to authenticate the accessibility of the local physical address (i.e., determine whether data can be read); and the responding device for RDMA data transfer can use the local key to authenticate the accessibility of the local physical address (i.e., determine whether data can be written). Subsequent RDMA data write processing can only proceed when the requesting device determines that it can read data and the responding device determines that it can write data.

[0046] In block 206, method 200 allows the processor to write all permission descriptions to the field-programmable gate array (FPGA) and convert the work requests into work queue item requests based on the work requests issued by the application software program and a preset address mapping relationship. Here, by setting a permission storage unit in the cache area of ​​the FPGA (such as Block RAM), the aforementioned memory usage unit writes all permission descriptions to the permission storage unit, and the permission storage unit uniformly manages the memory access permissions of each RDMA transmission channel queue pair. This ensures that when the FPGA processes RDMA requests, it can perform fast and secure access control based on the permission descriptions corresponding to the RDMA transmission channel queue pair, avoiding unauthorized access or unauthorized operations, thereby improving system security and reliability. It is understandable that the permission storage unit can set independent storage space for each permission description item (e.g., the size can be 32B) to minimize resource consumption (e.g., the actual occupied size in bytes = 32B * the maximum number of open channels N of RDMA), and can perform storage processing according to the specified arrangement order. Its design method meets the requirements of resource efficiency and performance optimization in the hardware design of field-programmable gate arrays, avoids complex permission verification logic at the software layer, and only performs simple linear permission verification in the field-programmable gate array, thereby improving the overall response speed and security of the system.

[0047] Please see Figure 4 The diagram illustrates the storage of a permission description item according to some embodiments of the present disclosure. Figure 4As shown, the permission storage unit stores N permission description items 400, each permission description item has a corresponding storage space (32B), and each permission description item is stored in ascending order of its number.

[0048] In addition, you may refer to Figure 5 The diagram illustrates a multi-channel RDMA transport channel queue pair resource allocation according to some embodiments of the present disclosure. Figure 5 As shown, the multi-channel RDMA transmission channel queue pair resource allocation process 500 includes allocating multiple available contiguous physical memory blocks (i.e., contiguous physical block A, contiguous physical block B, and contiguous physical block C) to each RDMA transmission channel queue pair (i.e., queue pair QPA, queue pair QPB, and queue pair QPC) used for memory request within a contiguous physical memory region reserved in memory; then, through a preset mapping mechanism between physical memory block addresses and virtual addresses, the initial physical addresses of each contiguous physical memory block are mapped to obtain the corresponding initial virtual addresses (i.e., contiguous virtual memory A, contiguous virtual memory B, and contiguous virtual memory C); then, based on each RDMA transmission channel queue pair, the corresponding initial physical address, and the initial virtual address, permission description items (i.e., permission description item A, permission description item B, and permission description item C) are generated. The continuous physical block between continuous physical block B and continuous physical block C can be understood as follows: after the corresponding continuous physical block C is allocated to the queue pair QPC, the RDMA transmission channel queue pair corresponding to the continuous physical block is destroyed based on data transmission requirements. At this time, the continuous physical block is reclaimed to the continuous physical memory area reserved in memory.

[0049] As can be seen, the embodiments of this disclosure determine the corresponding permission description items for each RDMA transmission channel queue, which can effectively ensure that even if there are software vulnerabilities, a channel cannot access the memory of other channels or applications without authorization, thus achieving efficient hardware-level memory protection, and the authentication rules are simple and efficient. In addition, compared with the memory protection table in the traditional RDMA technology implementation, changing the permission management from single memory block permissions to QP queue pairs consumes less resources and is simpler to maintain, making it more suitable for application in resource-constrained scenarios such as edge computing and embedded systems.

[0050] Subsequently, the application software program can issue RDMA request instructions (i.e., work requests) based on virtual addresses through the InfiniBand Verbs interface. The memory usage management unit set in the user-mode region of the RDMA driver running the processor then uses a preset address mapping relationship to convert the work request into a work queue item request based on a physical address. Here, a work request can be understood as the application software program obtaining a virtual address of a specified size from the available memory requested by the corresponding RDMA transfer channel queue pair, and filling the virtual address and the corresponding address size into the RDMA request instruction structure corresponding to the RDMA transfer channel queue pair. In addition to the filled virtual address and the corresponding address size, the work request can also include at least the corresponding RDMA transfer channel queue pair number (i.e., the unique identifier of the work request), the type of data transfer to be performed (such as RDMA read operation, RDMA write operation, RDMA send operation, or RDMA receive operation), the source physical address (or remote virtual address), and key parameters (such as local key or remote key).

[0051] In one example, taking an RDMA write operation as an example, after obtaining the permission description item corresponding to the current RDMA transmission channel queue pair with reference to the above embodiment, the responding end device for RDMA data writing can send the virtual base address and remote key information in the permission description item to the requesting end device for RDMA data writing through the application software program. At this time, the requesting end device can determine the target virtual address of a specified size in the available memory requested by the RDMA transmission channel queue pair based on the virtual base address, and fill the target virtual address, the corresponding address size and the remote key into the RDMA request instruction structure through the application software program to obtain the work request of the current RDMA transmission channel queue pair. Then, the application software program can send the generated work request to the responding end device through the InfiniBand Verbs interface.

[0052] It should be noted that when the data transmission type is an RDMA read operation, the key parameter in the work request is the remote key; when the data transmission type is an RDMA send operation or an RDMA receive operation, the key parameter in the work request is the local key. This enables unilateral operations to prevent remote unauthorized access and bilateral operations to regulate local access, achieving low latency, high throughput, and high security goals even in resource-constrained scenarios.

[0053] In some implementations, when converting a work request into a work queue item request based on a work request issued by an application program and a preset address mapping formula, the target virtual address can be identified from the work request. Then, the target virtual address, the virtual address base address corresponding to the RDMA transmission channel queue where the work request resides, and the physical address base address are substituted into the preset address mapping formula to obtain the target physical address. Here, the physical address base address corresponding to the RDMA transmission channel queue where the work request resides can be understood as the physical address of the first physical memory block in the contiguous physical memory blocks allocated by the RDMA transmission channel queue where the work request resides; the virtual address base address can be understood as the virtual address corresponding to the physical address of the first physical memory block in the contiguous physical memory blocks allocated by the RDMA transmission channel queue where the work request resides.

[0054] It is understandable that the preset address mapping relationship can be found in the following:

[0055]

[0056] In the above formula, For the target physical address, The target virtual address (which can be understood as the base address of the virtual address). The base address of the virtual address. This is the physical address base address.

[0057] Next, based on the work request and the target physical address, a work queue item request is obtained. Here, the work queue item request can be understood as containing all information in the work request except for the target virtual address and the target address information. This other information may include, for example, the corresponding RDMA transport channel queue pair number (i.e., the unique identifier of the work request), the type of data transfer performed (such as RDMA read operation, RDMA write operation, RDMA transmit operation, or RDMA receive operation), the source physical address (or remote virtual address), and key parameters (such as local key or remote key).

[0058] In this way, complex page table lookups can be simplified to a subtraction and an addition operation. This allows the software driver to directly construct a work queue item request based on a physical address from a user-space work request based on a virtual address using the aforementioned preset address mapping relationship when submitting a request. This means that the work queue item request processed by the subsequent hardware directly contains the physical address, completely eliminating the need for memory translation table lookups.

[0059] In some implementations, after converting a work request into a work queue item request based on a work request issued by an application program and a preset address mapping formula, the processor can further determine whether the number of target virtual addresses in the work request is greater than a preset threshold. This preset threshold can be set to 1, for example. In response to determining that the number of target virtual addresses in the work request is greater than the preset threshold, a doorbell triggering operation is performed based on the number of target virtual addresses. Here, when the memory usage management unit set in the user-mode region of the RDMA driver running the processor identifies that the number of target virtual addresses in the work request is greater than 1, it substitutes each target virtual address, the virtual address base address corresponding to the RDMA transmission channel queue where the work request is located, and the physical address base address into the preset address mapping formula to obtain the corresponding target physical address. Based on the work request and each target physical address, the corresponding work queue item request is obtained; that is, each target virtual address has a corresponding work queue item request.

[0060] Understandably, when there are multiple work queue item requests, the RDMA driver running on the processor can read the current value of the FPGA doorbell register, obtain an updated value based on the number of work queue item requests and the current value (such as summing the current value and the number of work queue item requests), and write the updated value to the FPGA doorbell register. When the FPGA recognizes that the newly written value to the doorbell register is different from the current value, it indicates that the necessary conditions for hardware operation have been triggered. At this time, the DMA engine of the FPGA can be triggered to read the multiple work queue item requests from memory.

[0061] Furthermore, when a work queue item request exists, the RDMA driver running on the processor can read the current value of the FPGA doorbell register, directly update the current value (such as incrementing the current value by one), and write the updated value to the FPGA doorbell register. When the FPGA recognizes that the newly written value in the doorbell register is different from the current value, it indicates that the necessary conditions for hardware operation have been triggered. At this time, the DMA engine of the FPGA can be triggered to read the work queue item request from memory.

[0062] This batch address delivery mechanism can significantly reduce the number of interactions between the FPGA and the IB Verbs software. Since the FPGA's DMA engine can handle multiple work queue requests simultaneously, it can perform read and write operations in parallel through a pipeline mechanism, which not only reduces idle cycles and further improves bandwidth utilization, but also achieves sub-microsecond end-to-end latency while ensuring high throughput transmission performance.

[0063] In block 208, method 200 allows the field-programmable gate array (FPGA) to perform permission verification on the work queue item request based on all permission descriptions. After determining that the permission verification of the work queue item request has passed, data transmission processing is performed based on the work queue item request. Here, after the FPGA obtains the work queue item request, it can retrieve the request from the queue through the permission storage unit set in the FPGA's buffer area (such as Block RAM), and perform permission verification on the request based on all permission descriptions. It is understood that the data type for permission verification processing can be, for example, key parameters, permission descriptions, and a target virtual address. When the key parameters, permission descriptions, and target virtual address all pass verification, the permission verification of the work queue item request is determined to be successful; otherwise, the permission verification of the work queue item request is determined to be unsuccessful.

[0064] Subsequently, upon successful authorization verification of the work queue item request, the DMA engine of the field-programmable gate array (FPGA) can perform data transmission processing based on the target physical address in the work queue item request. This includes, for example, reading data from the source physical address or writing received network data to the target physical address. Taking an RDMA write operation as an example, after successful authorization verification of the work queue item request, the responding device for RDMA data writing can write the received network data read from the source physical address by the requesting device to the target physical address. After completing the network data writing, it can also generate a response packet and send it back to the requesting device for RDMA data writing.

[0065] In this way, the frequent address translation overhead in traditional RDMA technology can be avoided, significantly improving the data transmission efficiency of RDMA to achieve high-bandwidth and low-latency high-performance network transmission. Furthermore, since each RDMA transmission channel queue has a corresponding permission description, hardware-level address range verification can quickly determine the access legitimacy, thus avoiding the accumulation of latency caused by permission checks.

[0066] In some implementations, when the field-programmable gate array (FPGA) performs permission verification on a work queue item request based on all permission descriptions, if the work request is in RDMA read / write mode, the FPGA can identify the RDMA transport channel queue pair number, remote key, permission description, and target virtual address from the work queue item request. Here, RDMA read / write mode can be understood as either RDMA read mode or RDMA write mode, which can be determined by querying the wr.opcode field in the work queue item request. For example, if the wr.opcode field is IBV_WR_RDMA_WRITE, the work request can be determined to be in RDMA write mode; or, if the wr.opcode field is IBV_WR_RDMA_READ, the work request can be determined to be in RDMA read mode.

[0067] Next, the target permission description item corresponding to the RDMA transmission channel queue pair number can be selected from all permission description items. Based on the target permission description item, it can be determined whether the remote key matches, whether the permission description matches, and whether the target virtual address is within the address range. Here, determining whether the remote key matches can be understood as judging whether the remote key in the work queue item request is consistent with the remote key in the target permission description item; determining whether the permission description matches can be understood as judging whether the memory permission description in the target permission description item contains the data transfer type in the work queue item request; and determining whether the target virtual address is within the address range can be understood as judging whether the target virtual address is within the address range determined by the virtual base address and the available memory size in the target permission description item.

[0068] Subsequently, in response to remote key matching, permission description matching, and the target virtual address being within the address range, it can be determined that the permission verification of the work queue item request has passed; otherwise, it will be determined that the permission verification of the work queue item request has failed. Of course, embodiments of this disclosure can also perform verification processing on other permission information, but these will not be elaborated upon here.

[0069] Alternatively, when the field-programmable gate array (FPGA) performs permission verification on the work queue item request based on all permission descriptions, it can also identify the RDMA transmission channel queue pair number, local key, permission description, and target virtual address from the work queue item request if the work request is determined to be in RDMA receive / transmit mode. Here, the RDMA receive / transmit mode can be determined by querying the wr.opcode field in the work queue item request. For example, when the wr.opcode field is IBV_WR_SEND, the work request can be determined to be in RDMA receive / transmit mode. In this case, the RDMA hardware device is the sender of the RDMA data transmission (i.e., the requesting device). Furthermore, it can be determined by querying other request interfaces that the corresponding RDMA data transmission work request is the receiver in RDMA receive / transmit mode (i.e., the responding device).

[0070] Next, the target permission description item corresponding to the RDMA transmission channel queue pair number can be selected from all permission description items. Based on the target permission description item, it is determined whether the local key matches, whether the permission description matches, and whether the target virtual address is within the address range. In response to the local key matching, permission description matching, and the target virtual address being within the address range, the permission verification of the work queue item request is determined to be successful. The process can be referred to the above, and will not be elaborated here. It should be noted that the purpose of verifying the local key here is to prevent the requested address from being released prematurely or being in an abnormal state.

[0071] In some implementations, the low-latency multi-channel RDMA data transfer method 200 further includes a field-programmable gate array (FPGA) generating an RDMA error response network packet corresponding to the work queue item request in response to the determination that the permission verification of the work queue item request has failed; and the FPGA feeding back the RDMA error response network packet to the processor, which then reclaims the contiguous physical memory blocks allocated to the corresponding RDMA transmission channel queue pair into a contiguous physical memory region. For example, after the FPGA of the responding device determines that the permission verification of the work queue item request has failed, it can generate an RDMA error response network packet corresponding to the work queue item request and feed the RDMA error response network packet back to the FPGA of the requesting device for processing; at this time, after reading the error status, the FPGA of the requesting device reports the error code to the application software via registers, and the application software requests the destruction of the corresponding RDMA transmission channel queue pair, thereby reclaiming the contiguous physical memory blocks allocated to the corresponding RDMA transmission channel queue pair into a contiguous physical memory region.

[0072] Please see Figure 6 A flowchart illustrating a low-latency multichannel RDMA data transmission process according to some embodiments of the present disclosure is shown.Figure 6 As shown, the low-latency multi-channel RDMA data transmission process 600 mainly utilizes a processor, memory, and a field-programmable gate array (FPGA) for RDMA data transmission. The memory is used to store multi-queue management (i.e., QP queue management) and a memory allocator. The memory allocator can be understood as the static memory allocation logic for dynamic management built in the RDMA driver mentioned in the above embodiment, such as controlling the unified memory management unit, memory usage unit, and permission storage unit to execute corresponding means respectively. The FPGA is used to store all permission description items (i.e., PD queues).

[0073] Understandably, during the operating system startup or initialization phase, the processor reserves a contiguous physical memory region in memory based on preset memory requirements, and then the memory feeds back the reserved contiguous physical memory region to the processor. Next, during the RDMA driver startup or initialization phase, the processor allocates corresponding contiguous physical memory blocks to multiple RDMA transfer channel queues based on the contiguous physical memory region, and maps the initial physical addresses of each contiguous physical memory block to obtain the initial virtual address. Then, during the RDMA memory registration phase, the processor, based on each RDMA transfer channel queue pair, the corresponding initial physical address, and the initial... The virtual address generates permission descriptions and writes all permission descriptions to the field-programmable gate array (FPGA). Then, during RDMA data transfer, the processor converts the work request (WR) issued by the application software program into a work queue item request (WQE) based on the preset address mapping relationship, and feeds the work queue item request back to the FPGA. Next, the FPGA authenticates the work queue item request based on the permission descriptions, and after the authentication is successful, it directly initiates a DMA operation based on the target physical address in the work queue item request and feeds back the data transfer completion to the processor.

[0074] Figure 7 A block diagram of a low-latency multi-channel RDMA data transmission system according to some embodiments of the present disclosure is shown. The various embodiments in this specification are described in a progressive manner, with reference to each other for similar or identical parts. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments. Figure 7As shown, the low-latency multi-channel RDMA data transmission system 700 may include at least a memory allocation module 702, configured to have the processor reserve contiguous physical memory regions based on preset memory requirements, and allocate corresponding contiguous physical memory blocks to multiple RDMA transmission channel queue pairs based on the contiguous physical memory regions during the RDMA initialization phase. The low-latency multi-channel RDMA data transmission system 700 also includes a permission description module 704, configured to have the processor map the initial physical addresses of each contiguous physical memory block to obtain initial virtual addresses, and generate permission description items based on each RDMA transmission channel queue pair, the corresponding initial physical address, and the initial virtual address during the RDMA memory registration phase. The low-latency multi-channel RDMA data transmission system 700 also includes a request translation module 706, configured to have the processor write all permission description items to a field-programmable gate array (FPGA), and convert work requests into work queue item requests based on work requests issued by the application software program and preset address mapping relationships. In addition, the low-latency multi-channel RDMA data transmission system 700 also includes an authorization verification module 708, which is configured to perform authorization verification processing on work queue item requests based on all authorization description items by a field-programmable gate array, and perform data transmission processing based on work queue item requests after determining that the authorization verification of the work queue item request has passed.

[0075] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer and a field-programmable gate array, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0076] Figure 8 A block diagram of an RDMA hardware device that can implement several embodiments of the present disclosure is shown. Figure 8 As shown, the RDMA hardware device 800 includes a processor 801, which can perform various appropriate actions and processes according to computer program instructions stored in memory 802. Memory 802 can also store various programs and data required for the operation of the RDMA hardware device 800. The processor 801, memory 802, and field-programmable gate array 803 are interconnected via a PCIe bus 804.

[0077] The various processes and handling described above, such as method 200, can be executed jointly by processor 801 and field-programmable gate array 803. For example, in some embodiments, method 200 can be implemented as a software program tangibly contained in a machine-readable medium. In some embodiments, part or all of the software program can be loaded and / or installed onto RDMA hardware device 800 via memory 802.

[0078] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to the processor and field-programmable gate array (FPGA) of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor and FPGA, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, partially on a remote machine as a standalone software package, or entirely on a remote machine or server.

[0079] This disclosure can be a method, apparatus, system, and / or program product. The program product may include a machine-readable storage medium on which machine-readable program instructions for performing various aspects of this disclosure are loaded. The machine-readable program instructions described herein can be downloaded from the machine-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the machine-readable program instructions from the network and forwards them to the machine-readable storage medium in the respective computing / processing device.

[0080] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A low-latency multi-channel RDMA data transmission method, characterized in that, include: The processor reserves a contiguous physical memory region based on preset memory requirements, and during the RDMA initialization phase, allocates corresponding contiguous physical memory blocks to multiple RDMA transmission channel queues based on the contiguous physical memory region. The processor maps the initial physical addresses of each of the contiguous physical memory blocks to obtain initial virtual addresses, and during the RDMA memory registration phase, it generates permission description items based on each of the RDMA transmission channel queue pairs, the corresponding initial physical addresses, and the initial virtual addresses. The processor writes all the permission description items to the field-programmable gate array and converts the work request into a work queue item request based on the work request issued by the application software program and the preset address mapping relationship. as well as The field-programmable gate array performs permission verification processing on the work queue item request based on all the permission description items, and after determining that the permission verification of the work queue item request is successful, it performs data transmission processing based on the work queue item request.

2. The method according to claim 1, characterized in that, The process of reserving a contiguous physical memory region by the processor based on preset memory requirements includes: The processor reserves a contiguous physical memory region in memory based on preset memory requirements; and The processor performs paging processing on the contiguous physical memory region based on preset paging requirements.

3. The method according to claim 1, characterized in that, The method further includes: The processor determines whether any of the RDMA transport channel queue pairs should be destroyed: and In response to determining that any of the RDMA transport channel queue pairs is destroyed, the processor reclaims the contiguous physical memory blocks allocated to the RDMA transport channel queue pairs into the contiguous physical memory region.

4. The method according to claim 1, characterized in that, The process of converting a work request into a work queue item request based on a work request issued by the application software program and a preset address mapping relationship includes: Based on the work request issued by the application software program, the target virtual address is identified from the work request; Substituting the target virtual address, the virtual address base address corresponding to the RDMA transmission channel queue where the work request is located, and the physical address base address into a preset address mapping formula, the target physical address is obtained; and Based on the work request and the target physical address, a work queue item request is obtained.

5. The method according to claim 4, characterized in that, After converting the work request into a work queue item request based on the work request issued by the application software program and the preset address mapping relationship, the method further includes: The processor determines whether the number of target virtual addresses in the job request is greater than a preset threshold; and In response to determining that the number of target virtual addresses in the work request is greater than the preset number threshold, a doorbell trigger operation is performed based on the number of target virtual addresses.

6. The method according to claim 4, characterized in that, The permission verification process performed by the field-programmable gate array on the work queue item request based on all the permission description items includes: When the work request is in RDMA read / write mode, the field-programmable gate array (FPGA) identifies the RDMA transmission channel queue pair number, remote key, permission description, and target virtual address from the work queue item request; Filter out the target permission description item that corresponds to the RDMA transmission channel queue pair number from all the permission description items, and determine whether the remote key matches, whether the permission description matches, and whether the target virtual address is within the address range based on the target permission description item; In response to the remote key matching, the permission description matching, and the target virtual address being within the address range, it is determined that the permission verification for the work queue item request has passed; or When the work request is in RDMA receive / transmit mode, the field-programmable gate array (FPGA) identifies the RDMA transmission channel queue pair number, local key, permission description, and target virtual address from the work queue item request; Filter out the target permission description item that corresponds to the RDMA transmission channel queue pair number from all the permission description items, and determine whether the local key matches, whether the permission description matches, and whether the target virtual address is within the address range based on the target permission description item; In response to the local key matching, the permission description matching, and the target virtual address being within the address range, it is determined that the permission verification for the work queue item request has passed.

7. The method according to claim 1 or 6, characterized in that, The method further includes: In response to the determination that the authorization verification for the work queue item request has failed, the field-programmable gate array (FPGA) generates an RDMA error response network packet corresponding to the work queue item request; and The field-programmable gate array (FPGA) feeds back the RDMA error response network packet to the processor, and the processor reclaims the contiguous physical memory block allocated to the corresponding RDMA transmission channel queue pair into the contiguous physical memory region.

8. A low-latency multi-channel RDMA data transmission system, characterized in that, include: The memory allocation module is configured to reserve a contiguous physical memory region by the processor based on preset memory requirements, and during the RDMA initialization phase, allocate corresponding contiguous physical memory blocks to multiple RDMA transmission channel queues based on the contiguous physical memory region. The permission description module is configured to have the processor map the initial physical addresses of each of the contiguous physical memory blocks to obtain initial virtual addresses, and generate permission description items based on each of the RDMA transmission channel queue pairs, the corresponding initial physical addresses, and the initial virtual addresses during the RDMA memory registration phase. The request conversion module is configured to have the processor write all the permission description items to the field-programmable gate array, and convert the work request into a work queue item request based on the work request issued by the application software program and the preset address mapping relationship. as well as The permission verification module is configured to perform permission verification processing on the work queue item request based on all the permission description items by the field-programmable gate array, and perform data transmission processing based on the work queue item request after determining that the permission verification of the work queue item request has passed.

9. A computer-readable storage medium having a computer program stored thereon, the computer-readable storage medium storing instructions that, when executed on a computer and a field-programmable gate array (FPGA), cause the computer and the FPGA to perform the steps of the method as claimed in any one of claims 1-7.

10. An RDMA hardware device, characterized in that, include: Processors, field-programmable gate arrays, and A memory associated with the processor and the field-programmable gate array (FPGA), the memory being used to store program instructions that, when read and executed by the processor and the FPGA, perform the steps of the method as described in any one of claims 1-7.

Images