Data transmission method and system based on shared buffer pool

Abstract

The embodiment of the invention relates to the technical field of communication, and discloses a data transmission method and system based on a shared buffer pool, and the data transmission method based on the shared buffer pool comprises the steps: constructing the shared buffer pool with a physical staggered mapping structure, and enabling inline data and an SGE descriptor to exist in the same physical memory area, a mode selection domain is set to store a transmission mode of application data, the transmission mode comprises an inline mode and an SGE mode, and a logical address domain is set to store a logical address, pointing to a shared buffer pool, of a work queue element, so that a physical address can be calculated by utilizing the logical address; the physical address is used for reading the application data in the shared buffer pool, memory copy delay caused by switching of different transmission modes can be eliminated, and then the utilization rate of memory resources is increased.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a data transmission method and system based on a shared buffer pool. Background Technology

[0002] RDMA (Remote Direct Memory Access) is a high-performance network communication technology that allows a computer to directly access the memory of a remote host without the involvement of the remote CPU.

[0003] The consumer organizes WQEs and sends them to the Work queue. The hardware retrieves these WQEs, executes them, and then organizes the completion instruction status and some necessary information into CQEs and writes them into the completion queue. The consumer retrieves CQEs from the completion queue through interrupts or continuous polling and adjusts the strategy for sending WQEs based on the information in the CQEs.

[0004] To improve the performance of small data packet transmission, RDMA technology introduces inline data transfer technology, which embeds data directly into work queue elements (WQEs). However, inline transfer and Scatter / Gather List (SGE) transfer require different memory regions. Inline data transfer is stored in reserved space within the WQE, while SGE data transfer is stored in a separate application buffer, with the WQE only containing SGE descriptors pointing to this buffer.

[0005] Typically, different conditions require switching between different modes. When a strategy decides to switch the data transfer mode from one mode to another (e.g., data originally planned to be inlined needs to be switched to SGE due to exceeding limits), data needs to be copied from one memory area to another. Therefore, time-consuming memory copying is required, resulting in significant latency and consequently, insufficient utilization of memory resources. Summary of the Invention

[0006] This application provides a data transmission method and system based on a shared buffer pool to improve the utilization of memory resources.

[0007] The embodiments of this application provide the following technical solutions: In a first aspect, embodiments of this application provide a data transmission method based on a shared buffer pool, applied to an RDMA network card, the method comprising: Retrieve work queue elements, where each work queue element includes a mode selection field and a logical address field. The mode selection field is used to store the transmission mode of the application data, which includes inline mode or SGE mode. The logical address field is used to store the logical address of the work queue element pointing to the shared buffer pool. When executing a work queue element, the physical address is calculated based on the application data transmission mode and logical address; Based on the physical address, the application data corresponding to the physical address is read from and transferred from the shared buffer pool. The shared buffer pool consists of multiple memory blocks with contiguous physical addresses, and each memory block includes an inline data segment and an SGE descriptor segment.

[0008] In some embodiments, Logical addresses include inline data logical addresses; Calculate the physical address based on the application data transmission mode and logical address, including: If the transmission mode is inline mode, the physical address is calculated based on the first mapping rule, which includes: Calculate the block number of the memory block based on the inline data logical address and the byte length of the memory block, where the block number of the memory block = (inline data logical address / the byte length of the memory block) rounded down. The physical address is calculated based on the block number of the memory block and the base address of the shared buffer pool. The physical address is calculated as: physical address = base address of the shared buffer pool + block number of the memory block * length of a memory block in bytes.

[0009] In some embodiments, The logical address includes the logical address of the SGE descriptor; Calculate the physical address based on the application data transmission mode and logical address, including: If the transmission mode is SGE mode, the physical address is calculated based on the second mapping rule, which includes: Calculate the SGE descriptor number based on the SGE descriptor logical address and the SGE descriptor's byte length. The SGE descriptor number is the floor value of (SGE descriptor logical address / SGE descriptor's byte length). The physical address is calculated based on the SGE descriptor number and the base address of the shared buffer pool. The physical address is calculated as follows: physical address = base address of shared buffer pool + SGE descriptor number * length of one memory block in bytes + length of one inline data in bytes.

[0010] In some embodiments, Based on the physical address, read and transfer the application data corresponding to the physical address from the shared buffer pool, including: If the application data corresponding to the physical address is located in the inline data segment, then the inline data in the inline data segment is read and transmitted directly. If the application data corresponding to the physical address is located in the SGE descriptor segment, then the SGE descriptor in the SGE descriptor segment is obtained, and the application data corresponding to the SGE descriptor is read. The SGE descriptor points to the physical address of the inline data, making the application data corresponding to the SGE descriptor inline data.

[0011] In some embodiments, RDMA network cards include on-chip cache; The method also includes: Based on the physical address, query the on-chip cache. If the physical address matches the on-chip cache, read and transfer the application data corresponding to the physical address from the on-chip cache. If the physical address does not hit the on-chip cache, the application data corresponding to the physical address is read from the shared buffer pool and transferred.

[0012] In some embodiments, The method also includes: Retrieve application data from the shared buffer pool; Write application data to the on-chip cache; Update the data in the on-chip cache using either the Least Recently Used algorithm or a random replacement algorithm.

[0013] In some embodiments, The method also includes: Real-time monitoring of application data transmission patterns; If the transfer mode is continuous inline mode, a DMA request is sent to the shared buffer pool in advance to write the memory block corresponding to the next logical address corresponding to the current logical address into the on-chip cache. or, If the transfer mode switches from SGE mode to inline mode, a DMA request is sent to the shared buffer pool in advance to write the memory block corresponding to the next logical address corresponding to the current logical address to the on-chip cache.

[0014] In some embodiments, The shared buffer pool consists of N memory blocks, each of which is of equal length; The inline data segment is used to store inline data, and the SGE descriptor segment is used to store SGE descriptors. The length of the inline data segment in each memory block is equal, and the length of each inline data segment is equal to the length of a cache line in bytes of the processor. The length of the SGE descriptor segment in each memory block is equal, and the length of each SGE descriptor segment is equal to the byte length of an SGE descriptor.

[0015] Secondly, embodiments of this application provide a data transmission method based on a shared buffer pool, applied to a host, the method comprising: Construct a shared buffer pool, which consists of multiple memory blocks with contiguous physical addresses. Each memory block includes an inline data segment and an SGE descriptor segment. Retrieve application data and write it to the shared buffer pool.

[0016] In some embodiments, Inline data segments correspond to inline data logical addresses, and SGE descriptor segments correspond to SGE descriptor logical addresses. Retrieve application data and write it to the shared buffer pool, including: The inline data logical address is obtained through the first allocation function, and the application data is written to the inline data segment corresponding to the inline data logical address. The SGE descriptor logical address is obtained through the second allocation function, and the SGE descriptor corresponding to the application data is written into the SGE descriptor segment corresponding to the SGE descriptor logical address. The SGE descriptor includes the physical address and data length of the application data.

[0017] Thirdly, embodiments of this application provide a data transmission system based on a shared buffer pool, the system comprising: a host and an RDMA network card, wherein, The host includes: The driver module is used to construct a shared buffer pool, which includes multiple memory blocks with contiguous physical addresses, each memory block including an inline data segment and an SGE descriptor segment; and to retrieve application data and write the application data into the shared buffer pool. RDMA network cards include: The address management unit is used to obtain work queue elements. Each work queue element includes a mode selection field and a logical address field. The mode selection field stores the transmission mode of the application data, which includes inline mode or SGE mode. The logical address field stores the logical address of the work queue element pointing to the shared buffer pool. When executing a work queue element, the physical address is calculated based on the transmission mode of the application data and the logical address. The DMA engine is used to read and transfer application data corresponding to a physical address from a shared buffer pool based on the physical address.

[0018] In some embodiments, RDMA network interface cards also include: a prefetch engine, a connection address management unit, and a DMA engine; The prefetch engine includes: On-chip cache is used to cache application data; The cache controller, connected to the DMA engine, is used to query the on-chip cache based on the physical address. If the physical address hits the on-chip cache, the application data corresponding to the physical address is read from the on-chip cache; if the physical address does not hit the on-chip cache, a DMA request is initiated to the DMA engine to read the application data corresponding to the physical address from the shared buffer pool through the DMA engine. The prefetcher is used to monitor the transfer mode of application data in real time. If the transfer mode is continuous inline mode, it sends a DMA request to the DMA engine to write the memory block corresponding to the next logical address corresponding to the current logical address to the on-chip cache. Alternatively, if the transfer mode is switched from SGE mode to inline mode, it sends a DMA request to the DMA engine to write the memory block corresponding to the next logical address corresponding to the current logical address to the on-chip cache.

[0019] Fourthly, embodiments of this application provide a non-volatile computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the aforementioned data transmission method based on a shared buffer pool.

[0020] The beneficial effects of this application embodiment are as follows: Unlike existing technologies, this application embodiment provides a data transmission method based on a shared buffer pool, applied to an RDMA network card. The method includes: obtaining a work queue element, wherein the work queue element includes a mode selection field and a logical address field, wherein the mode selection field is used to store the transmission mode of application data, the transmission mode including inline mode or SGE mode, and the logical address field is used to store the logical address of the work queue element pointing to the shared buffer pool; when executing the work queue element, calculating the physical address according to the transmission mode of the application data and the logical address; and reading and transmitting the application data corresponding to the physical address from the shared buffer pool according to the physical address, wherein the shared buffer pool includes multiple memory blocks with contiguous physical addresses, each memory block including an inline data segment and an SGE descriptor segment.

[0021] By constructing a shared buffer pool with a physically interleaved mapping structure, inline data and SGE descriptors reside in the same physical memory region. A mode selection field is set to store the transfer mode of application data, including inline mode and SGE mode. Furthermore, a logical address field is set to store the logical address of the work queue element pointing to the shared buffer pool. This allows the logical address to be used to calculate the physical address, and the physical address to be used to read application data in the shared buffer pool. This eliminates the memory copy latency caused by switching between different transfer modes, thereby improving the utilization of memory resources. Attached Figure Description

[0022] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0023] Figure 1 This is a schematic diagram of the processing flow of a work queue element provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating how inline data and SGE descriptors are cached independently, as provided in an embodiment of this application. Figure 3 This is a schematic diagram of a data transmission system based on a shared buffer pool provided in an embodiment of this application; Figure 4 This is a flowchart illustrating a data transmission method based on a shared buffer pool provided in an embodiment of this application; Figure 5 This is a schematic diagram of a shared buffer pool provided in an embodiment of this application; Figure 6 This is a schematic diagram illustrating a dynamic remapping of a driver and physical memory provided in an embodiment of this application; Figure 7 yes Figure 4 A detailed flowchart of step S402 in the process; Figure 8 This is a flowchart illustrating another data transmission method based on a shared buffer pool provided in an embodiment of this application; Figure 9 This is a schematic diagram illustrating the process of an RDMA network card acquiring data from a shared buffer pool, provided in an embodiment of this application. Figure 10 This is a schematic diagram of the workflow of intelligent cache prediction provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of a host provided in an embodiment of this application; Figure 12 This is a schematic diagram of the structure of an RDMA network card provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0025] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0026] Before providing a detailed description of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows: (1) Work Queue Element (WQE) refers to the core operation instruction carrier in the Remote Direct Memory Access (RDMA) communication architecture, and is also the smallest unit for RNIC (RDMA network card) to execute data transmission. A work queue element is a piece of structured memory data containing all the key information required for RDMA operation (such as operation type, data address, length, target identifier, etc.). It is submitted by the application / driver to the queue pair (QP) or physical shared transmit queue (SSQ) for RDMA hardware to read and execute directly without CPU intervention. It is the basis for realizing zero-copy, low-latency transmission in RDMA.

[0027] (2) The Completion Queue Element (CQE) is the operation result feedback carrier in the RDMA (Remote Direct Memory Access) communication architecture, and also the core basis for the consumer to obtain the RDMA operation execution status. The Completion Queue Element (CQE) is a piece of structured memory data generated by the hardware module of the RNIC (RDMA network card) after the completion work queue element (WQE) is executed. It contains key feedback such as whether the operation was successful, transmission details, and error information, and is written into the Completion Queue (CQ) for the consumer to read through interrupts or polling, ultimately realizing the RDMA communication closed loop of "software initiates request → hardware executes → software obtains result".

[0028] Please see Figure 1 , Figure 1 This is a schematic diagram of the processing flow of a work queue element provided in an embodiment of this application.

[0029] like Figure 1As shown, the consumer organizes work queue elements (WQE) and sends them to the work queue. The RDMA network card hardware retrieves these WQEs, executes them, and then organizes the status of the completion instruction and some necessary information into completion queue elements (CQE) and writes them to the completion queue. The consumer retrieves completion queue elements (CQE) from the completion queue through interrupts or continuous polling, and adjusts the strategy for sending work queue elements (WQE) based on the information in the completion queue elements (CQE).

[0030] It is understandable that the consumer is the software entity that initiates RDMA operations and processes operation completion feedback, while the hardware is the collection of core hardware modules in the RDMA network card, which is the hardware carrier that executes RDMA data transmission and generates completion status.

[0031] To improve the performance of small data packet transmission, RDMA technology introduces inline data transfer technology, which embeds data directly into work queue elements (WQEs). However, inline transfer and traditional SGE transfer require the use of different memory regions, specifically: For Inline Data technology, data is stored in reserved space within WQE; while for Scatter / Gather List (SGE) data transfer technology, data is stored in a separate application buffer, and WQE only stores SGE descriptors pointing to this buffer.

[0032] As can be seen, for inline data transmission and SGE data transmission, the data is stored in different spaces, which means that inline data and SGE data need to be set up with independent dedicated caches and managed through dedicated circuits.

[0033] This approach has the following problems: (1) Resource solidification: The length of the cache partition is fixed and cannot adapt to the dynamic changes in traffic patterns, which may lead to the waste of resources such as one cache overflowing while another cache is idle.

[0034] (2) High hardware complexity: Independent management logic and circuits need to be designed for the two cache areas, which increases chip area and design complexity.

[0035] (3) Copy overhead: When the strategy decides to change the data from one transmission mode to another (such as data originally planned to be inline need to be converted to SGE due to exceeding the limit), a time-consuming memory copy is required.

[0036] Please refer to the following: Figure 2 , Figure 2 This is a schematic diagram illustrating how inline data and SGE descriptors are cached independently, as provided in an embodiment of this application.

[0037] like Figure 2 As shown, the inline data dedicated cache and the SGE descriptor dedicated cache are set up separately. When the policy decides to change the data transfer mode from one mode to another (such as data that was originally planned to be inline need to be converted to SGE due to exceeding the limit), the data needs to be copied from one memory area to another. Therefore, a time-consuming memory copy is required, resulting in a large delay and thus insufficient utilization of memory resources.

[0038] Based on this, embodiments of this application provide a data transmission method and system based on a shared buffer pool to improve the utilization of memory resources.

[0039] Please see Figure 3 , Figure 3 This is a schematic diagram of a data transmission system based on a shared buffer pool provided in an embodiment of this application.

[0040] like Figure 3 As shown, the data transmission system 300 based on a shared buffer pool includes: a host 100 and an RDMA network card 200.

[0041] The host 100 includes: application module 110, driver module 120, and shared buffer pool 130.

[0042] Specifically, the application module 110 is connected to the driver module 120 and is used to send data transmission requests to the driver module 120.

[0043] In this embodiment, the application module 110 includes an application program, which is a software program running in the user mode of the operating system. The user can submit a data transmission request through the application program, which is used to read and write application data of the application program to the shared buffer pool 130.

[0044] Specifically, the driver module 120 is connected to the shared buffer pool 130 and is used to write application data to the shared buffer pool 130 or read application data from the shared buffer pool 130.

[0045] In this embodiment, the driver module 120 includes a driver, which is a kernel-space driver and is a low-level software component running in the kernel mode of the operating system on the host 100.

[0046] The driver is used to build the shared buffer pool 130, and to obtain application data sent by the application and write the application data into the shared buffer pool.

[0047] Specifically, the shared buffer pool 130 is located in the host memory of the host and is a segment of physical memory with contiguous addresses in the host memory. The shared buffer pool 130 includes multiple memory blocks with contiguous physical addresses. Each memory block has an equal and fixed length and includes an inline data segment and an SGE descriptor segment.

[0048] The RDMA network card 200 includes: an address management unit 210, a DMA engine 220, and a prefetch engine 230. The prefetch engine 230 includes a prefetcher 231, an on-chip cache 232, and a cache controller 233.

[0049] Specifically, the Address Management Unit (AMU) 210, located inside the RDMA network card 200, is a key module for achieving zero copy.

[0050] In this embodiment, the address management unit 210 is connected to the driver module 120 and is used to obtain the work queue element (WQE) in the send queue. The work queue element includes a mode selection field and a logical address field. The mode selection field is used to store the transmission mode of the application data, which includes inline mode or SGE mode. The logical address field is used to store the logical address of the work queue element pointing to the shared buffer pool. When executing the work queue element, the physical address is calculated according to the transmission mode of the application data and the logical address.

[0051] For example, the address management unit 210 receives a work queue element issued by the driver module 120. The work queue element includes a logical address and a mode flag. The address management unit 210 determines the data transmission mode through the mode flag and dynamically calculates the physical address corresponding to the logical address based on the logical address in the work queue element through hardware circuitry.

[0052] For example, the mode flag is represented by two bits. The bit values ​​corresponding to inline mode and SGE mode are 01 and 00, respectively. That is, when the bit value of the mode flag is 01, it indicates that the data transmission mode is inline mode; when the bit value of the mode flag is 00, it indicates that the data transmission mode is SGE mode.

[0053] Understandably, when switching transfer modes, there is no need to move data; simply modifying the mode flag in the mode selection field of the work queue element is sufficient to switch the transfer mode of application data. The address management unit 210 can automatically interpret the same physical memory as inline data or an SGE descriptor, thereby eliminating the memory copy latency caused by switching between different transfer modes and achieving zero copy.

[0054] In this embodiment of the application, an address management unit (AMU) is added to the RDMA network card to realize dynamic mapping from logical addresses to physical addresses of the shared buffer pool. The address management unit is implemented by adders, multipliers and shifters, which is much smaller in chip area than the complex state machine and arbitration logic that manage two independent caches, thus reducing chip complexity. Furthermore, the calculation of the address management unit can be completed within one clock cycle without increasing the critical path latency of the RDMA network card.

[0055] Specifically, DMA engine 220 connects to shared buffer pool 130 and is used to obtain application data in shared buffer pool 130 through DMA operations.

[0056] Specifically, the prefetch engine 230 connects the address management unit 210 and the DMA engine 220 to predict the next possible memory block to be accessed based on the mode flag bit of the work queue element, and write the data of the next possible memory block to the on-chip cache 232 of the prefetch engine 230 in advance.

[0057] Specifically, the prefetcher 231, connected to the DMA engine 220, is used to monitor the transfer mode of application data in real time. If the transfer mode is continuous inline mode, a DMA request is sent to the DMA engine 220 to write the memory block corresponding to the next logical address corresponding to the current logical address to the on-chip cache 232. Alternatively, if the transfer mode is switched from SGE mode to inline mode, a DMA request is sent to the DMA engine to write the memory block corresponding to the next logical address corresponding to the current logical address to the on-chip cache 232.

[0058] Specifically, the on-chip cache 232 is connected to the DMA engine 220 and is used to cache application data in the shared buffer pool 130.

[0059] In this embodiment, the on-chip cache 232 has a small capacity, for example, several KB, but is extremely fast. The cache line structure of the on-chip cache 232 is aligned with the memory blocks of the shared buffer pool 130; that is, the byte length of a cache line in the on-chip cache 232 is equal to the byte length of a memory block in the shared buffer pool. For example, if the byte length of a memory block in the shared buffer pool is 80 bytes, then the byte length of a cache line in the on-chip cache 232 is also set to 80 bytes. Each cache line corresponds to a unique cache line tag, which records the physical address of a memory block in the shared buffer pool corresponding to the cache line.

[0060] Specifically, the on-chip cache 232 is a static random access memory (SRAM), enhanced dynamic random access memory (eDRAM), or magnetoresistive random access memory (MRAM). Preferably, in this embodiment, the on-chip cache 232 is a static random access memory (SRAM).

[0061] Specifically, the cache controller 233 is connected to the DMA engine 220 and the on-chip cache 232. It is used to query the on-chip cache 232 according to the physical address. If the physical address hits the on-chip cache 232, the application data corresponding to the physical address is read from the on-chip cache 232. If the physical address does not hit the on-chip cache 232, a DMA request is initiated to the DMA engine 220 to read the application data corresponding to the physical address from the shared buffer pool 130 through the DMA engine 220.

[0062] Please see Figure 4 , Figure 4 This is a flowchart illustrating a data transmission method based on a shared buffer pool provided in an embodiment of this application.

[0063] The data transfer method based on a shared buffer pool is applied to a host, and the execution entity of the data transfer method based on a shared buffer pool is at least one processor of the host.

[0064] like Figure 4 As shown, the flow of this data transmission method based on a shared buffer pool includes the following steps S401-S402: Step S401: Build a shared buffer pool.

[0065] Specifically, the host driver requests a segment of physical memory with contiguous addresses and uses this segment of physical memory as a shared buffer pool. The shared buffer pool consists of multiple memory blocks with contiguous physical addresses. Each memory block has an equal and fixed length and includes an inline data segment and an SGE descriptor segment.

[0066] Please see Figure 5 , Figure 5 This is a schematic diagram of a shared buffer pool provided in an embodiment of this application.

[0067] like Figure 5 As shown, the shared buffer pool includes N memory blocks, i.e., physical blocks. The N memory blocks are memory block 0 to memory block N-1. The physical addresses of the N memory blocks are contiguous, and each memory block includes an inline data segment and an SGE descriptor segment. For example, memory block 0 includes inline data segment 0 and SGE descriptor segment 0, ..., memory block N-1 includes inline data segment N-1 and SGE descriptor segment N-1.

[0068] In this embodiment, although the overall length of a memory block remains constant, the lengths of the inline data segment and the SGE descriptor segment of the memory block can be adjusted according to specific needs. Furthermore, each memory block has contiguous physical addresses, and its internal physical storage space provides interleaved mapping for the inline data segment and the SGE descriptor segment.

[0069] It is understandable that, for drivers, the inline data segment and the SGE descriptor segment correspond to independent memory regions: an inline data area and an SGE descriptor area.

[0070] Assuming each memory block is 80 bytes long, Table 1 below is a schematic diagram of the physical layout of a memory block provided in an embodiment of this application:

[0071] Table 1 As shown in Table 1, the first 64 bytes of the memory block are used as the inline data segment, and the last 16 bytes are used as the SGE descriptor segment.

[0072] In this embodiment, the length of the inline data segment is equal to the length of a cache line of the host processor. For example, if the length of a cache line is 64 bytes, then the length of the inline data segment is also 64 bytes to ensure the efficiency of data reading and writing.

[0073] In this embodiment, the length of the SGE descriptor segment is greater than or equal to the length of one SGE descriptor, ensuring that the SGE descriptor segment can accommodate at least one standard SGE descriptor. For example, if an SGE descriptor includes a 64-bit address + 32-bit length + 32-bit key, then the length of one SGE descriptor is 16 bytes (128 bits). In this case, the length of the SGE descriptor segment is greater than or equal to 16 bytes. Preferably, the length of the SGE descriptor segment in this embodiment is 16 bytes, just enough to accommodate one SGE descriptor.

[0074] In this embodiment of the application, by constructing a shared buffer pool with physically interleaved mapping, the memory utilization of the host can be improved.

[0075] Please see Figure 6 , Figure 6 This is a schematic diagram of dynamic remapping between a driver and physical memory provided in an embodiment of this application.

[0076] like Figure 6 As shown, from the driver's perspective, the driver still believes that it is managing two separate logical areas, namely the inline data area and the SGE descriptor area, and does not know that these two logical addresses will eventually point to the same physical memory.

[0077] Step S402: Obtain application data and write the application data to the shared buffer pool.

[0078] Specifically, the host application writes application data to the shared buffer pool through the driver.

[0079] Please refer to the following: Figure 7 , Figure 7 yes Figure 4 A detailed flowchart of step S402 in the process.

[0080] like Figure 7 As shown, step S402: obtaining application data and writing the application data to the shared buffer pool includes the following steps S421-S422: Step S421: Obtain the inline data logical address through the first allocation function, and write the application data into the inline data segment corresponding to the inline data logical address.

[0081] Step S422: Obtain the logical address of the SGE descriptor through the second allocation function, and write the SGE descriptor corresponding to the application data into the SGE descriptor segment corresponding to the logical address of the SGE descriptor. The SGE descriptor includes the physical address and data length of the application data.

[0082] In this embodiment of the application, the shared buffer pool needs to be initialized before it is applied. Specifically, the initialization includes the following steps: (1) When the driver starts, it requests a block of physical memory with contiguous physical addresses from the host operating system. For example, it requests 1MB of physical memory with contiguous physical addresses.

[0083] (2) The driver formats the memory region into N 80-byte memory blocks.

[0084] (3) The driver creates two logical memory allocators and uses two allocation functions to obtain the logical address of the inline data segment and the logical address of the SGE descriptor segment.

[0085] In this embodiment, the two logical memory allocators are an inline data segment allocator and an SGE descriptor segment allocator. It can be understood that the inline data segment allocator and the SGE descriptor segment allocator are essentially two software modules. Specifically, the inline data segment allocator includes a first free list, a first allocation function, and a first deallocation function; the SGE descriptor segment allocator includes a second free list, a second allocation function, and a second deallocation function.

[0086] The first free list stores the logical addresses of all unallocated inline data segments.

[0087] The processing logic of the first allocation function includes: (1) Retrieve the logical address of the inline data segment from the first free list; (2) Mark the status of the logical address of the inline data segment as allocated to avoid duplicate allocation; (3) Return the logical address of the inline data segment to the driver; The processing logic of the first release function includes: The system receives the inline data segment logical address returned by the driver, marks the status of the returned inline data segment logical address as idle, and adds the inline data segment logical address in the idle state back to the first free list.

[0088] In this embodiment, the first allocation function includes the alloc_inline_chunk() function, which is used to obtain the inline data logical address and the driver writes the application data to the inline data segment corresponding to the inline data logical address.

[0089] The second free list stores the logical addresses of all unallocated SGE descriptor segments.

[0090] The processing logic of the second allocation function includes: (1) Retrieve the logical address of the SGE descriptor segment from the second free list; (2) Mark the state of the logical address of the SGE descriptor segment as allocated to avoid duplicate allocation; (3) Return the logical address of the SGE descriptor segment to the driver; The processing logic of the second release function includes: The system receives the SGE descriptor segment logical address returned by the driver, marks the returned SGE descriptor segment logical address as idle, and adds the idle SGE descriptor segment logical address back to the second idle list.

[0091] In this embodiment, the second allocation function includes the alloc_sge_chunk() function, which is used to obtain the logical address of the SGE descriptor, and the driver writes the application data into the SGE descriptor segment corresponding to the logical address of the SGE descriptor.

[0092] After the host driver builds a shared buffer pool and writes application data into the shared buffer pool, the RDMA network card can use the shared buffer pool for data transmission.

[0093] In this embodiment, since the shared buffer pool is shared by all transmission modes, the shared buffer pool can be dynamically allocated according to traffic, thereby improving resource utilization.

[0094] Please see Figure 8 , Figure 8 This is a flowchart illustrating another data transmission method based on a shared buffer pool provided in an embodiment of this application.

[0095] like Figure 8 As shown, the flow of this data transmission method based on a shared buffer pool includes the following steps S801-S803: Step S801: Obtain work queue elements.

[0096] The Work Queue Element (WQE) includes a mode selection field and a logical address field. The mode selection field stores the transmission mode of the application data, which can be either inline mode or SGE mode. The logical address field stores the logical address of the work queue element pointing to the shared buffer pool.

[0097] Specifically, the address management unit retrieves the work queue element (WQE) from the send queue of the work queue.

[0098] Step S802: When executing a work queue element, calculate the physical address based on the application data transmission mode and logical address.

[0099] Specifically, the address management unit retrieves work queue elements from the work queue, reads the mode flag bits in the work queue elements to determine the transmission mode of the application data, and reads the logical address in the work queue elements to calculate the physical address based on the transmission mode of the application data and the logical address.

[0100] In the embodiments of this application, the application data transmission modes include inline mode and SGE mode. Under different transmission modes, the addressing method of the address management unit is different.

[0101] (1) When the application data transmission mode is inline mode, the logical address includes the inline data logical address. In this case, the physical address is calculated based on the first mapping rule, wherein the first mapping rule includes: Calculate the block number of the memory block based on the inline data logical address and the byte length of the memory block, where the block number of the memory block = (inline data logical address / the byte length of the memory block) rounded down. The physical address is calculated based on the block number of the memory block and the base address of the shared buffer pool. The physical address is calculated as: physical address = base address of the shared buffer pool + block number of the memory block * length of a memory block in bytes.

[0102] For example: Suppose a memory block has a length of 80 bytes, then the first mapping rule is: Phys_Addr = Base_Addr + (Virt_Inline / 80) * 80 Where Phys_Addr is the physical address, Base_Addr is the base address of the shared buffer pool, and Virt_Inline is the logical address of the inline data. (Virt_Inline / 80) It is a rounded-down value of (inline data logical address / length of a memory block in bytes), representing the largest integer not greater than (Virt_Inline / 80), used to identify the block number of the memory block.

[0103] It is understandable that each memory block corresponds to a unique block number. Multiplying the block number by the length of the memory block in bytes yields the starting physical address of that block. Since the starting position of each memory block is an inline data segment, the starting physical address of that block corresponds exactly to the starting position of the inline data segment. Adding the starting position of the inline data segment to the base address of the shared buffer pool gives the physical address corresponding to the logical address of the inline data.

[0104] (2) When the application data transmission mode is SGE mode, the logical address includes the SGE descriptor logical address. In this case, the physical address is calculated based on the second mapping rule, which includes: Calculate the SGE descriptor number based on the SGE descriptor logical address and the SGE descriptor's byte length. The SGE descriptor number is the floor value of (SGE descriptor logical address / SGE descriptor's byte length). The physical address is calculated based on the SGE descriptor number and the base address of the shared buffer pool. The physical address is calculated as follows: physical address = base address of shared buffer pool + SGE descriptor number * length of one memory block in bytes + length of one inline data in bytes.

[0105] For example: Suppose a memory block has a length of 80 bytes, an inline data block has a length of 64 bytes, and an SGE descriptor has a length of 16 bytes, then the second mapping rule is: Phys_Addr = Base_Addr + (Virt_SGE / 16) * 80 + 64 Where Phys_Addr is the physical address, Base_Addr is the base address of the shared buffer pool, and Virt_SGE is the logical address of the SGE descriptor. (Virt_SGE / 16) It is a rounded-down value of (logical address of SGE descriptor / byte length of an SGE descriptor), representing the largest integer not greater than (Virt_SGE / 16), used to identify the number of the SGE descriptor.

[0106] Understandably, each memory block includes an SGE descriptor, and each SGE descriptor corresponds to a unique number. By calculating the SGE descriptor number and multiplying it by the byte length of a memory block, we can obtain the starting physical address of the memory block corresponding to that SGE descriptor. Since the starting location of a memory block is an inline data segment, and the byte length of an inline data segment is 64 bytes, adding 64 bytes to the starting physical address of the memory block gives us the starting physical address of the SGE descriptor, which is the physical address corresponding to the logical address of the SGE descriptor.

[0107] For example, the process of calculating the physical address corresponding to the logical address of the SGE descriptor includes the following steps (1)-(4): Assuming the base address of the shared buffer pool is Base_Addr = 0x10000, and the driver calls the second allocation function alloc_sge_chunk() to obtain the logical address of the SGE descriptor Virt_SGE = 0x20030, and the application data transfer mode is SGE mode, then: Step (1): Calculate the number of the SGE descriptor.

[0108] Specifically, (Virt_SGE / 16) = 0x20030 / 0x10 = 0x2003.

[0109] Step (2): Multiply the SGE descriptor number by the length of a memory block in bytes.

[0110] Specifically, 0x2003 * 80 = 0x2003 * 0x50 = 0x1000F0.

[0111] Step (3): Add the length of an inline data segment in bytes.

[0112] Specifically, 0x1000F0 + 64 = 0x1000F0 + 0x40 = 0x100130 Step (4): Based on the base address of the shared buffer pool, obtain the physical address corresponding to the logical address of the SGE descriptor.

[0113] Specifically, the physical address corresponding to the logical address of the SGE descriptor is Phys_Addr = 0x10000 + 0x100130 = 0x110130.

[0114] Understandably, after obtaining the physical address 0x110130 corresponding to the logical address of the SGE descriptor, the hardware will access the 16 bytes of content at physical address 0x110130 and interpret it as an SGE descriptor.

[0115] Step S803: Based on the physical address, read and transmit the application data corresponding to the physical address from the shared buffer pool.

[0116] Specifically, based on the physical address, the application data corresponding to the physical address is read from and transferred from the shared buffer pool, including: If the application data corresponding to the physical address is located in the inline data segment, then the inline data in the inline data segment is read and transmitted directly. If the application data corresponding to the physical address is located in the SGE descriptor segment, then the SGE descriptor in the SGE descriptor segment is obtained, and the application data corresponding to the SGE descriptor is read. The SGE descriptor points to the physical address of the inline data, making the application data corresponding to the SGE descriptor inline data.

[0117] The following explains in detail how to switch the application data transmission mode: 1. Initial state Assuming the initial state is that the application data transmission mode is SGE mode, then: (1) The driver writes the application data Data_A into the inline data segment (bytes 0-63 of physical block N) corresponding to the inline data logical address obtained by the function alloc_inline_chunk().

[0118] (2) The driver constructs an SGE descriptor SGE_A, the contents of which are: address: address 0 of block N, length: 64, key: keyA.

[0119] (3) The driver writes SGE_A to the SGE descriptor segment (64-79 bytes of physical block N) corresponding to the logical address of the SGE descriptor obtained by the function alloc_sge_chunk().

[0120] (4) The driver submits a work queue element (WQE) with the mode flag set to SGE mode. The RDMA network card reads and sends Data_A normally.

[0121] 2. Switching Operation The driver does not need to move Data_A; it only needs to resubmit a new WQE. (1) Generate the logical address where Data_A was previously stored using the function alloc_inline_chunk().

[0122] (2) Modify the mode flag to inline mode.

[0123] (3) When the RDMA network card receives a new WQE, the address management unit (AMU) of the RDMA network card parses the new WQE, identifies the mode flag bit in the WQE, and determines the transmission mode of the application data. Specifically, based on the mode flag bit, it determines that the transmission mode of the application data is inline mode, and calculates the physical address using the formula: Phys_Addr=Base_Addr + (Virt_Inline / 80) * 80 By calculating the physical address corresponding to the inline data logical address, this physical address is the 0-byte address pointing to physical block N.

[0124] (4) The contents of physical block N from 0 to 63 are directly extracted as inline data and sent, while the contents of 64 to 79 are completely ignored.

[0125] Understandably, the same physical data Data_A is interpreted differently by the hardware under different transmission modes. When switching from SGE mode to inline mode, no data copy occurs. Only a mode flag bit in WQE is changed, thus achieving zero copy and eliminating the memory copy latency caused by switching between different transmission modes, thereby improving the utilization of memory resources.

[0126] The following section uses specific application scenarios to illustrate how to switch application data transmission modes: 1. Scenario 1: Dynamic Load Balancing and Congestion Avoidance In data center networks, link congestion occurs dynamically. A path can go from idle to congested in an instant.

[0127] Processing procedure: (1) Initial state: The external application is preparing to send a medium-sized data packet (e.g., 300 bytes). To achieve higher throughput, based on the packet size and the current good network conditions, the decision is made to send in SGE mode. The data has been placed into the shared buffer pool in SGE mode. (PCIe transaction count in SGE mode: 1 read WQE + 1 read SGE descriptor + 1 read data).

[0128] (2) Sudden situation: Just as the packet is about to be sent, the network monitoring unit detects a sudden congestion on the target path, resulting in an increase in packet loss rate.

[0129] (3) Intelligent switching: To avoid triggering more DMA operations and retransmissions on congested links (which would exacerbate congestion), the policy engine intervenes immediately, driving the transmission mode of the unsent data packet to switch from SGE to inline data mode. (PCIE transaction count in inline mode: 1 read WQE + 1 read data).

[0130] (4) Zero-copy switching: The driver does not need to copy data from the SGE buffer to the WQE, but only needs to modify the mode flag in the WQE. The RDMA network card then sends the data as inline data.

[0131] As can be seen, by using the above method, fewer PCIe transactions are generated in the inline mode during congestion, resulting in less impact on congested links. This application can avoid packet loss and retransmission, thus improving the overall stability of the network.

[0132] 2. Scenario 2: Retransmission Optimization If a data packet is lost during transmission, the hardware needs to initiate a retransmission.

[0133] Processing procedure: (1) First transmission: A data packet (e.g., 200 bytes) is sent using SGE mode, but is lost on the link.

[0134] (2) Retransmission decision: Existing technology requires DMA read operation to be performed again for retransmission. However, at this time, the original data may still be in the buffer of the sending end.

[0135] (3) Intelligent retransmission: In order to complete the retransmission as quickly as possible and with the least overhead, the sending end strategy engine decides to use the inline mode for retransmission packets. This allows for fast retransmission directly from the cache without having to access the potentially distant main memory to perform DMA again.

[0136] (4) Zero-copy retransmission: The driver finds the original memory block in the cache, modifies the mode flag of WQE to inline mode, and re-enqueues WQE.

[0137] As can be seen, this application can significantly reduce retransmission delay through the above methods, which is crucial for applications such as financial transactions that have zero tolerance for latency jitter.

[0138] 3. Scenario 3: Pipeline-style data processing A data stream needs to be processed sequentially by multiple processing units.

[0139] Processing procedure: (1) Phase 1: Remote processing: A large data task is sent to a remote GPU for processing. This phase uses SGE mode to efficiently transfer large amounts of data.

[0140] (2) Phase 2: Complete the result feedback: After the remote GPU processing is completed, a small result summary is generated.

[0141] (3) Mode switching: In order to send this small result summary back at high speed, the remote server no longer requests a new buffer, but directly reuses a memory block that was previously used to receive big data, writes the result summary into the memory block, and switches the transmission mode of the memory block from SGE mode to inline mode.

[0142] (4) Zero-copy sending: Quickly send back the result summary in inline mode.

[0143] As can be seen, by using the above method, this application can avoid the overhead of allocating new memory and copying data, achieve seamless connection from processing to transmission, and further reduce end-to-end latency.

[0144] In this embodiment, a zero-copy mode switching mechanism that only needs to change the mode flag bit during mode switching, without the need for data transfer, eliminates the memory copy latency caused by mode switching. For scenarios requiring high-frequency and dynamic transmission, it can reduce data latency and improve transmission efficiency.

[0145] Please see Figure 9 , Figure 9 This is a schematic diagram of a process for an RDMA network card to obtain data from a shared buffer pool, provided in an embodiment of this application.

[0146] like Figure 9 As shown, the process of the RDMA network card obtaining data from the shared buffer pool includes the following steps S901-S907: Step S901: The host writes the application data to the shared buffer pool.

[0147] Step S902: The host generates work queue elements.

[0148] Step S903: The RDMA network card obtains the work queue element sent by the host.

[0149] Step S904: The address management unit of the RDMA network card obtains the mode flag and logical address from the work queue element.

[0150] Step S905: If the mode flag is inline mode, calculate the physical address of the inline data segment based on the first mapping rule.

[0151] Step S906: If the mode flag is SGE mode, calculate the physical address of the SGE descriptor segment based on the second mapping rule.

[0152] Step S907: The RDMA network card performs a DMA operation, retrieves application data from the shared buffer pool in the physical address, and sends it.

[0153] It can be seen that when implementing zero-copy mode switching, the RDMA network card needs to initiate an additional direct memory access operation (DMA operation) to read application data from the shared buffer pool, which requires additional PCIe read transaction overhead.

[0154] Therefore, further, this application embodiment also eliminates the DMA latency caused by RDMA network card accessing the shared buffer pool by integrating intelligent caching and prefetching mechanisms.

[0155] For details, please refer to [link / reference]. Figure 3 ,like Figure 3 As shown, a prefetch engine 230 is configured in the RDMA network card 200. The prefetch engine 230 includes an on-chip cache 232 for caching application data in a shared buffer pool.

[0156] Specifically, the RDMA network card 200 is used to obtain application data from the shared buffer pool and write the application data to the on-chip cache 232.

[0157] Understandably, due to the limited capacity of on-chip caches, they are only used to cache all data from a single operation or data that is about to be read. Furthermore, the data in the on-chip cache needs to be updated using either the Least Recently Used (LRU) algorithm or a random replacement algorithm.

[0158] Specifically, after the RDMA network card calculates the physical address, the method also includes: Based on the physical address, query the on-chip cache. If the physical address matches the on-chip cache, read and transfer the application data corresponding to the physical address from the on-chip cache. If the physical address does not hit the on-chip cache, the application data corresponding to the physical address is read from the shared buffer pool and transferred.

[0159] In some embodiments, the method further includes: Real-time monitoring of application data transmission patterns; If the transfer mode is continuous inline mode, a DMA request is sent to the shared buffer pool in advance to write the memory block corresponding to the next logical address corresponding to the current logical address into the on-chip cache. or, If the transfer mode switches from SGE mode to inline mode, a DMA request is sent to the shared buffer pool in advance to write the memory block corresponding to the next logical address corresponding to the current logical address to the on-chip cache.

[0160] Please see Figure 10 , Figure 10 This is a schematic diagram of the workflow of intelligent cache prediction provided in an embodiment of this application.

[0161] like Figure 10 As shown, the workflow of this intelligent cache prediction includes the following steps S1001-S1013: Step S1001: Obtain work queue elements.

[0162] Step S1002: The address management unit calculates the physical address.

[0163] Specifically, the address management unit calculates the physical address based on the mode flag and logical address in the work queue element.

[0164] Step S1003: On-chip cache hit.

[0165] Specifically, determining whether a physical address hits the on-chip cache includes: checking the cache line label of the on-chip cache; if there is a cache line label in the on-chip cache that is equal to the physical address, then the on-chip cache is determined to have a cache hit (CacheHit). If a cache line tag in the on-chip cache does not exist that is equal to the physical address, then an on-chip cache miss is determined.

[0166] Step S1004: The prefetcher continuously monitors the address stream and transmission mode of the address management unit; Specifically, the prefetcher continuously monitors the address flow of the address management unit. When the prefetcher observes consecutive inline mode accesses, it predicts that the memory block corresponding to the next logical address may also be accessed, and then initiates a DMA read in advance to prefetch the memory block corresponding to the next logical address into the on-chip cache. Also, when the prefetcher observes that the transfer mode switches from SGE mode to inline mode, it predicts that the memory block may be accessed again (such as in a retransmission scenario), and then prefetches the memory block into the on-chip cache in advance.

[0167] Step S1005: Initiate a DMA read request to the shared buffer pool; Step S1006: Is the triggering condition met? Specifically, determine whether the application data transmission mode is continuous inline mode, or whether the transmission mode has switched from SGE mode to inline mode. If so, the triggering condition is met, and the process proceeds to step S1011: Initiate predictive prefetch DMA to the shared buffer pool.

[0168] Specifically, determining whether the application data transmission mode is a continuous inline mode includes: The judgment is made based on the address stream data of the address management unit. For example, if 80 bytes of space are used as logical addresses for inline data, and the logical addresses in the address stream increase by 80 bytes three or more times consecutively, then the transmission mode is determined to be continuous inline mode.

[0169] In this embodiment of the application, the prefetch engine has a built-in policy algorithm that determines whether the triggering conditions are met based on the address stream data of the address management unit. The triggering conditions include: (1) Sequential access prediction: When consecutive access to physical blocks N, N+1, N+2... is detected, it is predicted that N+3 will be accessed soon. If the triggering condition is met, a prefetch DMA operation is initiated to obtain the data of physical block N+3 from the shared buffer pool and write the data to the on-chip cache.

[0170] In this embodiment, the number of consecutive logical addresses in the address stream depends on the maximum read payload size (max_read_payload) of the PCIe. The maximum read payload size refers to the maximum number of data blocks (bytes) that a PCIe device can read from the target device in a single read transaction. For example, if max_read_payload_size = 512 bytes and the physical block length is 80 bytes, merging consecutive DMA read operations into one operation can achieve optimal performance. For instance, if the number of consecutive read operations is 5, 512 - 5 * 80 = 112 bytes, leaving 112 bytes, which can be prefetched once (80 bytes). If the number of consecutive read operations is 4, 512 - 4 * 80 = 192 bytes, allowing for two more prefetches.

[0171] (2) Mode switching prediction: When the mode flag bit of the same logical address is detected to switch from 00 (SGE mode) to 01 (inline mode), it is predicted that the data corresponding to the logical address will be accessed again (e.g., retransmission), and prefetching is triggered immediately.

[0172] Step S1007: Read data directly from the on-chip cache; Specifically, if the on-chip cache hits, the cache controller directly retrieves the data corresponding to the physical address from the on-chip cache, so that the cache controller does not need to initiate PCIe DMA operation through the DMA engine.

[0173] By directly retrieving the data corresponding to the physical address from the on-chip cache, the DMA latency caused by the RDMA network card accessing the shared buffer pool can be eliminated, thereby improving data transmission efficiency.

[0174] Step S1008: Write the data to the on-chip cache; Specifically, if the on-chip cache misses, the cache controller needs to initiate a PCIe DMA operation to the shared buffer pool through the DMA engine to read the application data in the shared buffer pool. At the same time as reading the application data, the application data is written to the on-chip cache.

[0175] Understandably, after obtaining application data from the shared buffer pool, it can be used to assemble data packets for transmission, or the application data can be written to the on-chip cache.

[0176] Step S1009: Organize and send the data packet; Step S1010: Data transmission completed; Step S1011: Initiate predictive prefetch DMA to the shared buffer pool; Specifically, data is retrieved from the shared buffer pool via DMA operations and then written to the on-chip cache.

[0177] Step S1012: After data retrieval, it is automatically written to the on-chip cache; Step S1013: Continue monitoring.

[0178] The prefetching strategy of this application will be explained below in the context of specific application scenarios: 1. Scenario 1: Performance in a retransmission scenario (1) First transmission: Data packet A is transmitted in SGE mode. After transmission, its memory block is cached in the on-chip cache.

[0179] (2) Assuming packet loss: Packet loss occurs in the network.

[0180] (3) Zero-overhead retransmission: If the driver wants to retransmit data packet A, it determines to use inline mode, and then only modifies the mode flag when submitting WQE.

[0181] (4) Ultimate performance path: A. The address management unit calculates the physical address PA_A of data packet A.

[0182] B. The cache controller queries the on-chip cache and finds a PA_A hit (because it was cached after the first transmission).

[0183] C. RDMA network cards do not trigger PCIe DMA transactions at all, but read data directly from the ultra-low latency on-chip cache and send it in inline mode.

[0184] As can be seen, in the retransmission scenario, the retransmission time only includes the buffer access time (nanosecond level), which is far lower than the overhead of traditional memory copy + PCIe DMA or even secondary PCIe DMA, thus reducing latency.

[0185] 2. Scenario 2: High-performance computing (collaborative computing) In computational fluid dynamics simulations, a computation node needs to send its boundary data to neighboring nodes. The data volume is large and the memory is distributed widely (non-contiguous). The computation is iterative, and the same memory region is read, written, and sent multiple times.

[0186] In existing technical solutions, each time scattered data is sent, the CPU needs to be involved to obtain the data, or multiple RDMA operations need to be initiated, resulting in high CPU overhead and latency.

[0187] The implementation steps of the solution in this application embodiment include: (1) Initial settings: The compute nodes register the frequently transmitted, distributed data regions into the shared buffer pool. The driver pre-constructs an array of SGE descriptors for this data and places it into the shared buffer pool.

[0188] (2) Initial preheating and sending: The driver uses extended mode. In extended mode, the WQE inline portion contains opcodes that point to an external SGE array.

[0189] A. RDMA network card handles WQE: It reads the SGE array through DMA operations, then reads all scattered memory blocks through DMA operations based on the SGE array, and finally sends the data. The CPU does not participate in the entire process.

[0190] B. All these read, scattered memory blocks are cached in the on-chip cache of the RDMA network card.

[0191] (3) Subsequent iterations: A. In the next iteration, the same updated batch of data needs to be sent again.

[0192] B. The driver directly uses inline mode and submits a logical address pointing to the memory block.

[0193] C. The address management unit of the RDMA network card calculates the physical address and then checks the cache, and all addresses are hit.

[0194] D. RDMA network cards do not need to initiate any PCIe DMA operations; they can directly read the latest data from their on-chip buffer and send it.

[0195] In this scenario, although the solution in this embodiment incurs a one-time DMA overhead during the initial transmission, it effectively warms up the on-chip cache. Subsequent transmissions achieve true zero-copy functionality without requiring DMA transfers, resulting in extremely low and stable latency. Furthermore, it completely frees up the CPU to focus on computation itself.

[0196] In this embodiment, based on zero-copy mode switching, DMA operations are further avoided through cache hits. The intelligent prefetch mechanism effectively hides DMA access latency, making the data transmission process smoother and achieving the ultimate performance of zero-copy + zero DMA.

[0197] In this embodiment, a data transmission method based on a shared buffer pool is provided and applied to an RDMA network card. The method includes: obtaining a work queue element, wherein the work queue element includes a mode selection field and a logical address field, wherein the mode selection field is used to store the transmission mode of the application data, the transmission mode includes inline mode or SGE mode, and the logical address field is used to store the logical address of the work queue element pointing to the shared buffer pool; when executing the work queue element, calculating the physical address according to the transmission mode of the application data and the logical address; and reading and transmitting the application data corresponding to the physical address from the shared buffer pool according to the physical address, wherein the shared buffer pool includes multiple memory blocks with contiguous physical addresses, and each memory block includes an inline data segment and an SGE descriptor segment.

[0198] By constructing a shared buffer pool with a physically interleaved mapping structure, inline data and SGE descriptors reside in the same physical memory region. A mode selection field is set to store the transfer mode of application data, including inline mode and SGE mode. Furthermore, a logical address field is set to store the logical address of the work queue element pointing to the shared buffer pool. This allows the logical address to be used to calculate the physical address, and the physical address to be used to read application data in the shared buffer pool. This eliminates the memory copy latency caused by switching between different transfer modes, thereby improving the utilization of memory resources.

[0199] Please refer to the following: Figure 11 , Figure 11 This is a schematic diagram of the structure of a host provided in an embodiment of this application.

[0200] like Figure 11 As shown, the host 100 includes one or more processors 101 and a memory 102. Wherein, Figure 11 Take a processor 101 as an example.

[0201] Processor 101 and memory 102 can be connected via a bus or other means. Figure 11 Taking the example of a connection between China and Israel via a bus.

[0202] The processor 101 is used to provide computing and control capabilities to control the host 100 to perform corresponding tasks, such as controlling the host 100 to perform the data transmission method based on the shared buffer pool in any of the above method embodiments.

[0203] Processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0204] Memory 102, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the data transmission method based on a shared buffer pool in the embodiments of this application. Processor 101 can implement the data transmission method based on a shared buffer pool in any of the above method embodiments by running the non-transitory software programs, instructions, and modules stored in memory 102. Specifically, memory 102 may include volatile memory (VM), such as random access memory (RAM); memory 102 may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), or other non-transitory solid-state storage devices; memory 102 may also include combinations of the above types of memory.

[0205] Memory 102 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 102 may optionally include memory remotely located relative to processor 101, and these remote memories may be connected to processor 101 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0206] One or more modules are stored in memory 102. When executed by one or more processors 101, they perform the data transfer method based on a shared buffer pool in any of the above method embodiments, for example, the method described above. Figure 4 The steps shown.

[0207] Please refer to the following: Figure 12 , Figure 12 This is a schematic diagram of the structure of an RDMA network card provided in an embodiment of this application.

[0208] like Figure 12 As shown, the RDMA network card 200 includes one or more processors 201 and a memory 202. Wherein, Figure 12 Take a processor 201 as an example.

[0209] The processor 201 and the memory 202 can be connected via a bus or other means. Figure 12 Taking the example of a connection between China and Israel via a bus.

[0210] The processor 201 is used to provide computing and control capabilities to control the RDMA network card 200 to perform corresponding tasks, such as controlling the RDMA network card 200 to perform the data transmission method based on the shared buffer pool in any of the above method embodiments.

[0211] Processor 201 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0212] Memory 202, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the data transmission method based on a shared buffer pool in the embodiments of this application. Processor 201 can implement the data transmission method based on a shared buffer pool in any of the above method embodiments by running the non-transitory software programs, instructions, and modules stored in memory 202. Specifically, memory 202 may include volatile memory (VM), such as random access memory (RAM); memory 202 may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), or other non-transitory solid-state storage devices; memory 202 may also include combinations of the above types of memory.

[0213] Memory 202 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 202 may optionally include memory remotely located relative to processor 201, and these remote memories may be connected to processor 201 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0214] One or more modules are stored in memory 202. When executed by one or more processors 201, they perform the data transfer method based on a shared buffer pool in any of the above method embodiments, for example, the method described above. Figure 8 The steps shown.

[0215] This application also provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors. For example, the one or more processors can execute the data transfer method based on a shared buffer pool in any of the above method embodiments, such as performing the steps described above.

[0216] The apparatus or device embodiments described above are merely illustrative. The unit modules described as separate components may or may not be physically separate, and the components shown as module units may or may not be physical units; that is, they may be located in one place or distributed across multiple network module units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0217] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general-purpose hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions for a computer device (which may be a personal computer, server, or network device, etc.) to execute the various embodiments or some parts of the embodiments.

[0218] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above. For the sake of brevity, they are not provided in detail; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data transmission method based on a shared buffer pool, characterized in that, Applied to RDMA network cards, the method includes: Obtain a work queue element, wherein the work queue element includes a mode selection field and a logical address field, wherein the mode selection field is used to store the transmission mode of the application data, the transmission mode includes inline mode or SGE mode, and the logical address field is used to store the logical address of the work queue element pointing to the shared buffer pool. When executing the work queue element, the physical address is calculated based on the transmission mode of the application data and the logical address; Based on the physical address, the application data corresponding to the physical address is read from and transmitted from the shared buffer pool, wherein the shared buffer pool includes multiple memory blocks with consecutive physical addresses, and each memory block includes an inline data segment and an SGE descriptor segment.

2. The method according to claim 1, characterized in that, The logical address includes inline data logical addresses; The step of calculating the physical address based on the transmission mode of the application data and the logical address includes: If the transmission mode is inline mode, the physical address is calculated based on the first mapping rule, wherein the first mapping rule includes: The block number of the memory block is calculated based on the inline data logical address and the byte length of the memory block, wherein the block number of the memory block is the floor value of (the inline data logical address / the byte length of the memory block); The physical address is calculated based on the block number of the memory block and the base address of the shared buffer pool, wherein the physical address = the base address of the shared buffer pool + the block number of the memory block * the byte length of the memory block.

3. The method according to claim 1, characterized in that, The logical address includes the logical address of the SGE descriptor; The step of calculating the physical address based on the transmission mode of the application data and the logical address includes: If the transmission mode is SGE mode, then the physical address is calculated based on the second mapping rule, wherein the second mapping rule includes: The number of the SGE descriptor is calculated based on the logical address of the SGE descriptor and the byte length of the SGE descriptor, wherein the number of the SGE descriptor is the floor value of (the logical address of the SGE descriptor / the byte length of the SGE descriptor). The physical address is calculated based on the SGE descriptor number and the base address of the shared buffer pool, wherein the physical address = the base address of the shared buffer pool + the SGE descriptor number * the length of one memory block in bytes + the length of one inline data in bytes.

4. The method according to claim 1, characterized in that, The step of reading and transmitting application data corresponding to the physical address from the shared buffer pool based on the physical address includes: If the application data corresponding to the physical address is located in the inline data segment, then the inline data in the inline data segment is directly read and transmitted. If the application data corresponding to the physical address is located in the SGE descriptor segment, then the SGE descriptor in the SGE descriptor segment is obtained, and the application data corresponding to the SGE descriptor is read. The SGE descriptor points to the physical address of the inline data, so that the application data corresponding to the SGE descriptor is the inline data.

5. The method according to claim 1, characterized in that, The RDMA network card includes an on-chip cache; The method further includes: Based on the physical address, query the on-chip cache. If the physical address matches the on-chip cache, read and transmit the application data corresponding to the physical address from the on-chip cache. If the physical address does not match the on-chip cache, the application data corresponding to the physical address is read from the shared buffer pool and transmitted.

6. The method according to claim 5, characterized in that, The method further includes: Retrieve application data from the shared buffer pool; Write the application data into the on-chip cache; The on-chip cache is updated using either the Least Recently Used algorithm or a random replacement algorithm.

7. The method according to claim 5, characterized in that, The method further includes: Real-time monitoring of the application data transmission mode; If the transmission mode is a continuous inline mode, a DMA request is sent to the shared buffer pool in advance to write the memory block corresponding to the next logical address corresponding to the current logical address into the on-chip cache. or, If the transmission mode switches from SGE mode to inline mode, a DMA request is sent to the shared buffer pool in advance to write the memory block corresponding to the next logical address corresponding to the current logical address into the on-chip cache.

8. The method according to any one of claims 1-7, characterized in that, The shared buffer pool includes N memory blocks, each of which has an equal length; The inline data segment is used to store inline data, and the SGE descriptor segment is used to store SGE descriptors. The length of the inline data segment in each memory block is equal, and the length of each inline data segment is equal to the length of a cache line in bytes of the processor. The length of the SGE descriptor segment in each memory block is equal, and the length of each SGE descriptor segment is equal to the byte length of an SGE descriptor.

9. A data transmission method based on a shared buffer pool, characterized in that, Applied to a host, the method includes: Construct a shared buffer pool, wherein the shared buffer pool comprises multiple memory blocks with contiguous physical addresses, and each memory block comprises an inline data segment and an SGE descriptor segment; Obtain application data and write the application data to the shared buffer pool.

10. The method according to claim 9, characterized in that, The inline data segment corresponds to the inline data logical address, and the SGE descriptor segment corresponds to the SGE descriptor logical address. The step of acquiring application data and writing the application data to the shared buffer pool includes: The inline data logical address is obtained through the first allocation function, and the application data is written into the inline data segment corresponding to the inline data logical address. The SGE descriptor logical address is obtained through the second allocation function, and the SGE descriptor corresponding to the application data is written into the SGE descriptor segment corresponding to the SGE descriptor logical address. The SGE descriptor includes the physical address and data length of the application data.

11. A data transmission system based on a shared buffer pool, characterized in that, The system includes: a host and an RDMA network card, wherein, The host includes: A driver module is used to construct a shared buffer pool, wherein the shared buffer pool includes multiple memory blocks with contiguous physical addresses, each memory block including an inline data segment and an SGE descriptor segment; and to acquire application data and write the application data into the shared buffer pool. The RDMA network card includes: The address management unit is used to obtain work queue elements, wherein the work queue element includes a mode selection field and a logical address field, wherein the mode selection field is used to store the transmission mode of the application data, the transmission mode includes inline mode or SGE mode, and the logical address field is used to store the logical address of the work queue element pointing to the shared buffer pool; when executing the work queue element, the physical address is calculated according to the transmission mode of the application data and the logical address. The DMA engine is used to read and transfer application data corresponding to the physical address from the shared buffer pool based on the physical address.

12. The system according to claim 11, characterized in that, The RDMA network interface card further includes: a prefetch engine, which connects the address management unit and the DMA engine; The prefetch engine includes: On-chip cache is used to cache the application data; A cache controller, connected to the DMA engine, is used to query the on-chip cache based on the physical address. If the physical address hits the on-chip cache, the application data corresponding to the physical address is read from the on-chip cache. If the physical address does not hit the on-chip cache, a DMA request is initiated to the DMA engine to read the application data corresponding to the physical address from the shared buffer pool through the DMA engine. A prefetcher is used to monitor the transmission mode of the application data in real time. If the transmission mode is a continuous inline mode, a DMA request is sent to the DMA engine to write the memory block corresponding to the next logical address corresponding to the current logical address into the on-chip cache. Alternatively, if the transmission mode switches from SGE mode to inline mode, a DMA request is sent to the DMA engine to write the memory block corresponding to the next logical address corresponding to the current logical address into the on-chip cache.

Images