RDMA Master-Slave Memory Information Exchange Method

Abstract

This invention discloses an RDMA master-slave memory information exchange method, relating to the field of computer network technology. The method includes updating memory information using a combination of single-end operations: data RDMA-WRITE and data RDMA-READ. Specifically, both the master and slave ends manage memory through free memory information indexes and occupied memory information indexes. The free memory information index points to free, usable memory; the occupied memory information index points to memory occupied by user data, where the data is waiting to be transmitted or used by the user. This invention reduces the CPU load.

Description

Technical Field

[0001] This invention relates to the field of computer network technology, and in particular to an RDMA master-slave memory information exchange method. Background Technology

[0002] Remote Direct Memory Access (RDMA) is a direct memory access technology that transfers data directly from the memory of one computer to the memory of another without the intervention of the operating systems of both computers. This allows for high-throughput, low-latency network communication, making it particularly suitable for use in large-scale parallel computer clusters. The counterpart to RDMA is traditional local DMA. RDMA operations require the user to issue tasks from the WQ queue and retrieve task completion information from the CQ queue.

[0003] RDMA has two types of operations: two-way operation and one-way operation, both involving the requesting and responding ends. Two-way operation requires the participation of both ends of the RDMA process; primarily, the sending end fills its WQ queue with SEND operation instructions, and the receiving end fills its WQ queue with RECV operation instructions. In other words, the CPUs at both the sending and receiving ends must participate in the instruction issuance action; the message command is RDMA-SEND. One-way operation is true RDMA because only one end's CPU participates, while the other end's CPU is completely unaware. It mainly includes two types of operations: RDMA-WRITE and RDMA-READ.

[0004] RDMA only defines the process of copying data from one end of memory to the other end of memory. However, in actual engineering implementation, there is often another problem: the memory information of the two ends needs to be constantly exchanged and updated to maintain the continuous operation of the underlying RDMA. The requesting end and the responding end need to constantly exchange information about free / used memory at the higher layer of the RDMA protocol so that RDMA knows which memory space to operate on next in order to maintain the continuous operation of RDMA. Otherwise, the requesting end of RDMA will not know which memory address of the other end to operate on. It can be seen that it includes three levels of operation: (1) requesting end DMA; (2) two ends RDMA; (3) higher layer memory information exchange.

[0005] Memory information exchange, because it operates outside the RDMA protocol, can be achieved through any means necessary to exchange memory information. Common methods include exchanging via TCP / IP and exchanging via two-way operations of the RDMA protocol (RDMA-SEND and RDMA-RECV), with the two-way operation being more widely used.

[0006] Both RDMA-WRITE and RDMA-READ involve memory information exchange between the master and slave ends. The responding end needs to send memory information to the requesting end, which then initiates an RDMA operation. After the RDMA operation is complete, the memory information is sent back to the responding end. Because of this memory information exchange, both the master and slave CPUs need to be involved, sending and receiving RDMA-SEND messages. Therefore, in high-bandwidth transmission scenarios, a large amount of memory information needs to be exchanged per second, and the exchange operations are very frequent. Both CPUs need to allocate significant performance resources to handle this memory information exchange, resulting in a substantial impact on CPU performance. The mechanism for memory information exchange using TCP / IP is similar.

[0007] In summary, under the traditional approach, both the master and slave CPUs need to be involved in the exchange of memory information because the messages used are RDMA-SEND operations, which are two-way operations. The master needs to send the WQE of the SEND operation to the SQ queue, and the slave needs to send the WQE of the RECV operation to the RQ queue. When the RDMA bandwidth is large, the memory space exchange between the master and slave becomes very frequent, resulting in significant CPU resource consumption. Summary of the Invention

[0008] To address the problems existing in the prior art, the purpose of this invention is to provide an RDMA master-slave memory information exchange method, which reduces the CPU burden.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is: an RDMA master-slave memory information exchange method, comprising:

[0010] The system employs single-end operations of RDMA-WRITE and RDMA-READ to update memory information. Specifically, both the master and slave ends manage memory through free memory information indexes and occupied memory information indexes. The free memory information index points to free, usable memory, while the occupied memory information index points to memory occupied by user data, where the data is waiting to be transmitted or used by the user.

[0011] As a further improvement of the present invention, the single-ended operation of data RDMA-WRITE combined with updating memory information specifically includes the following steps:

[0012] (1) The free memory information index of the response end is placed in the fixed address space of the system memory of the response end. The request end initiates an RDMA-READ operation to query the memory information of the response end. When the request end completes the RDMA operation, the occupied memory information index of the response end is updated through the RDMA-WRITE operation. The CPU of the response end queries to obtain the latest memory status.

[0013] (2) The requesting end queries the memory free status of the responding end and initiates a data RDMA operation;

[0014] (3) After the data RDMA operation is completed, refresh the memory usage information index of the response end;

[0015] (4) The response end periodically queries the memory occupancy information index of a fixed address in the system memory to obtain the latest memory occupancy information.

[0016] As a further improvement of the present invention, in (2), the requesting end queries the free state of the memory of the responding end as follows:

[0017] The requesting CPU sends the RDMA-READ operation work queue element WQE to the send queue SQ, and queries the responding end's free memory status at a fixed address. The underlying network interface card (NIC) parses the WQE from the SQ queue and obtains the RDMA-READ packet from the requesting end's system memory. The NIC then sends the RDMA-READ message to the responding end. After parsing the RDMA-READ message, the responding end's underlying hardware encapsulates the free memory information index stored at the fixed address in the responding end's system memory and sends it to the requesting end as an acknowledgment message (ACK). The requesting end's NIC stores the RDMA-READ return information carried by the ACK at a specified address in the system memory and generates a completion queue (CQ) to notify the CPU.

[0018] As a further improvement to the present invention, (3) is specifically as follows:

[0019] The requesting CPU sends the WQE of the RDMA-WRITE operation to the SQ queue; the underlying network interface card (NIC) parses the WQE from the SQ queue and retrieves the RDMA-WRITE body from system memory; the underlying NIC sends the RDMA-WRITE to the responding end; after parsing the RDMA-WRITE message, the responding end's underlying NIC refreshes the message content to the responding end's occupied memory information index space, which has a fixed address in system memory, and simultaneously sends an ACK message to the requesting end; the requesting end's underlying NIC converts the ACK into a CQ to notify the CPU.

[0020] As a further improvement to the present invention, (4) is specifically as follows:

[0021] The responding CPU queries the system's fixed address space for the index of occupied memory information; the responding CPU obtains the index of occupied system memory information and identifies the data in the valid memory space; the responding CPU sends the address of the valid data space to the application; the application notifies the CPU that the memory space can be released after processing the data; the responding CPU refreshes the system's free memory information index and updates the free memory information for subsequent queries by the requesting CPU.

[0022] As a further improvement of the present invention, the single-ended operation of data RDMA-WRITE combined with updating memory information specifically includes the following steps:

[0023] ① The requesting CPU obtains the memory index of the responding end through RDMA-READ operation, retrieves free memory from the local free memory information index, and then initiates a data RDMA read operation;

[0024] ② After the data is sent to the requesting end, the free memory information index of the responding end is updated again through the RDMA-WRITE operation;

[0025] ③ The CPU on the responding end obtains available memory from the free memory resource pool and performs local DMA again, while the CPU on the requesting end notifies the application to use the data.

[0026] As a further improvement of the present invention, in ①, the requesting CPU obtains the memory index of the responding end's occupied information through RDMA-READ operation as follows:

[0027] The requesting CPU places the work queue element WQE corresponding to RDMA-READ into the send queue SQ and retrieves the memory usage information index from the fixed address of the responding end. After parsing the WQE task, the underlying network interface card (NIC) of the requesting end encapsulates the RDMA-READ message body stored in the fixed memory address space of the requesting end and transmits it to the responding end. The underlying NIC of the responding end encapsulates the data of the memory usage information index stored in the local fixed address space and sends it to the requesting end through the acknowledgment character ACK message corresponding to RDMA-READ. Upon receiving the ACK corresponding to RDMA-READ, the underlying NIC of the requesting end places the content data into the local memory at the specified address and simultaneously generates a completion queue CQ to notify the CPU that it has retrieved the memory usage information index of the responding end.

[0028] As a further improvement of the present invention, ② is specifically as follows:

[0029] The requesting CPU places the WQE corresponding to RDMA-WRITE into the SQ queue and issues a task to update the free memory information index of the responding end. After parsing the WQE task, the underlying network card of the requesting end encapsulates the RDMA-WRITE message stored in the fixed memory address space of the requesting end and transmits it to the responding end. After receiving the RDMA-WRITE, the underlying network card of the responding end updates its content data to the corresponding address space in its local memory, completes the update of the free memory information index, and sends an ACK message to the requesting end. When the underlying network card of the requesting end receives the ACK corresponding to RDMA-WRITE, it generates a CQ to notify the CPU that the update task has been completed.

[0030] The beneficial effects of this invention are:

[0031] This invention does not use the two-end operation RDMA-SEND, but instead uses the one-end operations RDMA-WRITE and RDMA-READ to update memory information, which can reduce the CPU load. Attached Figure Description

[0032] Figure 1 This is a flowchart of the data RDMA-WRITE process in an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram illustrating the principle of data RDMA-WRITE in an embodiment of the present invention;

[0034] Figure 3 This is a flowchart of data RDMA-READ in an embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram illustrating the principle of data RDMA-READ in an embodiment of the present invention. Detailed Implementation

[0036] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0037] Example

[0038] A new RDMA master-slave memory information exchange method eliminates the need for the two-way RDMA-SEND operation. Instead, it uses the one-way RDMA-WRITE and RDMA-READ operations to update memory information. Both the master and slave ends manage memory through free memory information indices and occupied memory information indices. The free memory information index points to free, usable memory; the occupied memory information index points to memory occupied by user data, where the data is waiting to be transmitted or used by the user.

[0039] like Figure 1 and Figure 2 As shown, the RDMA-WRITE data specifically includes:

[0040] Since the responding end no longer actively informs the responding end of its memory status via RDMA-SEND, the requesting end needs to initiate an RDMA-READ operation to query the responding end's memory information. The responding end's free memory information index (including but not limited to information such as whether memory is free) is stored in a fixed address space within the responding end's system memory, so the requesting end always queries this fixed memory address. After the requesting end's RDMA operation is complete, it no longer informs the responding end which memory addresses are in use via RDMA-SEND; instead, it updates the responding end's occupied memory information index via RDMA-WRITE, and the responding end's CPU queries to obtain the latest memory status.

[0041] Operations belonging to local DMA:

[0042] (1) The requesting CPU obtains free memory from the space memory resource pool.

[0043] (2) The requesting CPU allocates free memory resources to the underlying DMA hardware.

[0044] (3) Write data to free memory using the underlying DMA hardware.

[0045] (4) After the data is written, the underlying DMA hardware notifies the CPU.

[0046] The requesting end queries the responding end's memory free status, which is a requesting end RDMA query operation:

[0047] (5) The requesting CPU sends the WQE of the RDMA-READ operation to the SQ queue in order to query the free memory status of the responding end from the fixed address (RDMA query).

[0048] (6) The underlying network card parses WQE from the SQ queue and obtains the RDMA-READ message in the memory of the requesting end system. Here, the body of the RDMA-READ message can be stored at a fixed address in the memory of the requesting end (RDMA query).

[0049] (7) The underlying network card sends the RDMA-READ message to the responding end (RDMA query).

[0050] (8) After the underlying hardware of the responding end parses the RDMA-READ message, it encapsulates the free memory information index stored at a fixed address in the system memory of the responding end and sends it to the requesting end with an ACK message (RDMA query).

[0051] (9) After the requesting end's underlying network card stores the RDMA-READ return information carried by the ACK into the specified address of the system memory, it generates a CQ to notify the CPU (RDMA query).

[0052] At this point, the requesting CPU already knows the memory status of both the master and slave ends, and the next step is to initiate the data RDMA operation:

[0053] (5') The requesting CPU sends the WQE containing local memory information and response memory information to the SQ queue (data RDMA).

[0054] (6') The underlying network card retrieves data from local memory and assembles data packets (data RDMA).

[0055] (7') The requesting network card sends the data packet to the responding network card. The responding network card parses the data and stores it in the corresponding memory space (data RDMA).

[0056] (8') The responding end replies with an ACK message. After receiving the ACK, the requesting end fills the CQ queue (data RDMA).

[0057] (9') The requesting CPU receives CQ indicating that the data RDMA is complete (data RDMA).

[0058] After the data RDMA operation is completed, the memory usage information index of the responding end needs to be refreshed so that the CPU of the responding end knows which memory spaces it has just transferred from the requesting end:

[0059] (5) The requesting CPU sends the WQE of the RDMA-WRITE operation to the SQ queue (RDMA update).

[0060] (6) The underlying network card parses WQE from the SQ queue and obtains the RDMA-WRITE message body in the system memory. The RDMA-WRITE message body can be stored at a fixed address in memory (RDMA update).

[0061] (7) The underlying network card sends RDMA-WRITE to the responding end (RDMA update).

[0062] (8) After the underlying network card of the responding end parses the RDMA-WRITE message, it refreshes the message content to the memory information index space occupied by the responding end, which is located at a fixed address in the system memory. At the same time, it sends an ACK message to the requesting end (RDMA update).

[0063] (9) The requesting end's underlying network card will convert ACK to generate CQ to notify the CPU (RDMA update).

[0064] For the responding end, it periodically queries the memory usage index at fixed addresses in the system memory to obtain the latest memory usage information and determine which spaces are available for data use.

[0065] (A) The CPU at the response end queries the memory usage information index in the fixed address space of the system memory.

[0066] (B) The CPU at the response end obtains the system memory usage information index and identifies which memory spaces have valid data.

[0067] (C) The responding end sends the address of the valid data space to the application.

[0068] (D) The application processes data.

[0069] (E) Once the application has finished processing the data, it notifies the CPU that the memory space can be released.

[0070] (A') The responding CPU refreshes the system's free memory information index and updates the free memory information for subsequent queries by the requesting end.

[0071] Thus, a complete data RDMA process was achieved without using the two-end operation RDMA-SEND.

[0072] like Figure 3 and Figure 4 As shown, data RDMA-READ specifically includes:

[0073] The data RDMA-READ operation can also be performed without using the traditional two-way RDMA-SEND operation. The entire process can be completed by simply using the separate RDMA-READ and RDMA-WRITE operations to update the memory information.

[0074] The DMA operation on the response end is largely the same as in traditional schemes. That is, regardless of the mechanism used for information exchange between the master and slave ends, it does not affect the local DMA operation on the response end.

[0075] (1) The CPU at the response end obtains free memory from the space memory resource pool.

[0076] (2) The CPU at the response end allocates free memory resources to the underlying DMA hardware.

[0077] (3) Write data to free memory using the underlying DMA hardware.

[0078] (4) After the data is written, the underlying DMA hardware notifies the CPU.

[0079] (5) The memory occupied information index is located in the fixed address space of the system memory on the response end. After the CPU on the response end receives the DMA completion indication, it updates the memory occupied information index so that the requesting end can query it.

[0080] (A) The requesting end obtains free memory from the local free memory information index.

[0081] The requesting CPU obtains the memory index of the responding end through the RDMA-READ operation, thus knowing which data will be transferred on the RDMA channel next:

[0082] (B) The requesting CPU puts the WQE corresponding to RDMA-READ into the SQ queue. Its purpose is to obtain the memory occupied information index from the fixed address of the responding end (RDMA query).

[0083] (C) After the requesting end's underlying network card parses the WQE task, it encapsulates the RDMA-READ message body stored in the fixed memory address space of the requesting end and transmits it to the responding end (RDMA query).

[0084] (D) The underlying network card of the responding end encapsulates the data of the memory occupied information index stored in the local fixed address space and sends it to the requesting end through the ACK message corresponding to RDMA-READ (RDMA query).

[0085] (E) The requesting end's underlying network card receives the ACK corresponding to RDMA-READ, puts the content data into the local memory at the specified address, and generates a CQ to notify the CPU that it has queried the memory information index of the responding end (RDMA query).

[0086] At this point, the requesting end knows its local free memory address and the data memory address of the other end, and thus initiates a data RDMA read operation.

[0087] (B') The requesting CPU sends the WQE containing local memory information and response memory information to the SQ queue (data RDMA).

[0088] (C') After the requesting end's underlying network card parses WQE, it initiates an RDMA-READ operation to the responding end (data RDMA).

[0089] (D') After the underlying network card of the responding end retrieves the data from the corresponding local memory address, it encapsulates the message and transmits the data to the requesting end using an ACK message. After the underlying hardware of the requesting end parses the message, the data is placed in the local free memory at the specified address, and the CQ queue is filled at the same time (data RDMA).

[0090] (E') When the requesting CPU receives CQ, it indicates that the data RDMA is complete (data RDMA).

[0091] After the data has been sent to the requesting end, indicating that the data RDMA is complete, the free memory information index on the responding end is updated again via the RDMA-WRITE operation:

[0092] (B) The requesting CPU puts the WQE corresponding to RDMA-WRITE into the SQ queue. Its purpose is to issue a task to update the free memory information index of the responding end (RDMA update).

[0093] (C) After the requesting end's underlying network card parses the WQE task, it encapsulates the RDMA-WRITE message stored in the requesting end's fixed memory address space and transmits it to the responding end. The message body indicates which memory address spaces on the responding end can be released (RDMA update).

[0094] (D) After receiving RDMA-WRITE, the underlying network card of the responding end updates its content data to the corresponding address space of local memory, completing the update of the free memory information index. At the same time, it sends an ACK message to the requesting end (RDMA update).

[0095] (E”) The requesting end's underlying network card receives the ACK corresponding to RDMA-WRITE, generates a CQ to notify the CPU, indicating that the update task has been completed (RDMA update).

[0096] After the above operations are completed, the free memory information index of the responding end has been updated, and the CPU of the responding end can obtain available memory from the free memory resource pool to perform local DMA again.

[0097] The requesting CPU knows that new data has arrived after step (E') and can notify the application to use the data.

[0098] (F) After the requesting CPU knows the available data memory space, it notifies the user program to use the data.

[0099] (G) The requesting user program retrieves data from the valid data memory space and processes it.

[0100] (H) After the requesting user program finishes processing the data, it notifies the CPU that the memory space can be released.

[0101] (I) Once the application has finished processing the data, the CPU returns the memory to the idle resource pool for the next operation.

[0102] Thus, a complete data RDMA-READ process was achieved without using the two-end operation RDMA-SEND.

[0103] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method for exchanging memory information in RDMA master and slave, characterized in that, Comprise: Single-end operation with data RDMA-WRITE and data RDMA-READ is used to update memory information, specifically, both master and slave ends manage memory through free memory information index and occupied memory information index, wherein the free memory information index points to free available memory; the occupied memory information index points to memory occupied by user data, and the data stored in the memory is waiting for transmission or user use; Single-end operation with data RDMA-WRITE is used to update memory information, specifically comprising the following steps: (1) Place the free memory information index of the response end in the fixed address space of the system memory of the response end, and the request end initiates RDMA-READ operation to query the memory information of the response end, and after the RDMA operation of the request end is completed, update the occupied memory information index of the response end through RDMA-WRITE operation, and the CPU of the response end queries to obtain the latest memory state; (2) The request end queries the free state of the memory of the response end and initiates data RDMA operation; (3) After the data RDMA operation is completed, refresh the occupied memory information index of the response end; (4) The response end regularly queries the occupied memory information index in the fixed address of the system memory to obtain the latest occupied information of the memory; Single-end operation with data RDMA-WRITE is used to update memory information, specifically comprising the following steps: ① The CPU of the request end obtains the occupied information memory index of the response end through RDMA-READ operation, and obtains free memory from the local free memory information index, and then initiates data RDMA read operation; ② After the data is sent to the request end, the free memory information index of the response end is updated again through RDMA-WRITE operation; ③ The CPU of the response end obtains available memory from the free memory resource pool and performs local DMA again, and the CPU of the request end notifies the application program to use the data.

2. The method of claim 1, wherein, In (2), the request end queries the free state of the memory of the response end as follows: The CPU of the request end issues a work queue element WQE of RDMA-READ operation to a sending queue SQ to query the free memory state of the response end at a fixed address; the bottom card parses the WQE from the SQ queue to obtain the RDMA-READ message in the system memory of the request end; the bottom card sends the RDMA-READ message to the response end; after the bottom hardware of the response end parses the RDMA-READ message, the free memory information index stored in the fixed address of the system memory of the response end is packaged and sent to the request end with an acknowledgement character ACK message; the bottom card of the request end stores the RDMA-READ return information carried by the ACK in the specified address of the system memory, and generates a completion queue CQ to notify the CPU.

3. The method of claim 2, wherein the RDMA master-slave memory information exchange method is characterized by, The (3) is specifically as follows: The CPU of the request end issues a WQE of RDMA-WRITE operation to the SQ queue; the bottom card parses the WQE from the SQ queue to obtain the RDMA-WRITE text in the system memory; the bottom card sends the RDMA-WRITE to the response end; The bottom card of the response end parses the RDMA-WRITE message, and flushes the message content to the response end occupied memory information index space in the system memory, and sends an ACK message to the request end; the bottom card of the request end converts the ACK to generate a CQ to notify the CPU.

4. The method of claim 3, wherein the RDMA master-slave memory information exchange method is characterized by, The (4) is specifically as follows: The response end CPU queries the occupied memory information index in the system memory fixed address space; the response end CPU obtains the system occupied memory information index, identifies the data of the effective memory space; the response end sends the address of the effective data space to the application program; the application program notifies the CPU after processing the data, and the memory space can be released; the response end CPU flushes the system free memory information index, updates the free memory information, and provides the request end for subsequent query.

5. The method of claim 1, wherein the RDMA master-slave memory information exchange method is characterized by, In ①, the request end CPU obtains the occupied information memory index of the response end through the RDMA-READ operation, and the specific process is as follows: The request end CPU puts the WQE corresponding to the RDMA-READ into the sending queue SQ, and obtains the occupied memory information index from the fixed address of the response end; the bottom card of the response end parses the WQE task, encapsulates the RDMA-READ message text stored in the fixed memory address space of the request end, and transmits it to the response end; the bottom card of the response end encapsulates the data of the occupied memory information index stored in the local fixed address space, and sends it to the request end through the RDMA-READ corresponding ACK message; the bottom card of the request end receives the RDMA-READ corresponding ACK, and puts the content data into the local memory at the specified address, and generates the completion queue CQ to notify the CPU that the occupied memory information index of the response end has been queried.

6. The method of claim 5, wherein, The (2) is specifically as follows: The request end CPU puts the WQE corresponding to the RDMA-WRITE into the SQ queue, and issues a task to update the free memory information index of the response end; the bottom card of the request end parses the WQE task, encapsulates the RDMA-WRITE message stored in the fixed memory address space of the request end, and transmits it to the response end; the bottom card of the response end receives the RDMA-WRITE, updates the content data to the corresponding address space of the local memory, completes the update of the free memory information index, and sends an ACK message to the request end; the bottom card of the request end receives the RDMA-WRITE corresponding ACK, generates a CQ to notify the CPU, indicating that the update task has been completed.

Images