Uninterrupted virtual remote direct memory access

Abstract

The invention relates to uninterrupted virtual remote direct memory access. A system and method may facilitate software-based transmission control protocol (TCP)-based remote direct memory access (RDMA) communications. And the system establishes TCP connection with the second virtual machine through the RDMA simulation software layer in the first virtual machine. In response to establishing the TCP connection, the system identifies, by the RDMA emulation software layer in the first virtual machine, a TCP socket associated with the TCP connection based on socket information received from the second virtual machine. In response to an RDMA sending operation initiated by the first virtual machine, the system performs the following operations through the RDMA simulation software layer in the first virtual machine: attaching a protocol header associated with the RDMA simulation software layer to the front of an RDMA sending frame; and transmitting the pre-attached RDMA transmission frame via the TCP socket. And the system executes the RDMA sending operation through the TCP connection.

Description

Background Art Technical Field

[0001] Some current computing systems can be deployed on private clouds, but may require specific hardware with remote direct memory access (RDMA) capabilities, such as RDMA over Converged Ethernet (RoCE) network interface cards (NICs). However, RoCE NICs may not be used in most cloud-based systems. In addition, some current systems may run based on the User Datagram Protocol (UDP) and support a small number of vendor-specific hypervisors. Therefore, some current computing systems may be limited to hardware-based designs of UDP-based RoCE NICs and a limited set of hypervisors. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1 An environment for facilitating software-based Transmission Control Protocol (TCP)-based RDMA communications according to an aspect of the present application is illustrated, the environment including a computing system having two virtual machines.

[0003] Figure 2 A diagram illustrating a communication flow between two virtual machines having an RDMA emulation software layer for creating a communication channel according to an aspect of the present application is presented.

[0004] Figure 3 A diagram illustrating a communication flow between two virtual machines having an RDMA emulation software layer for performing an RDMA send operation according to an aspect of the present application is presented.

[0005] Figure 4 A diagram illustrating a communication flow between two virtual machines having an RDMA emulation software layer for performing an RDMA write operation according to an aspect of the present application is presented.

[0006] Figure 5 A diagram illustrating a communication flow between two virtual machines having an RDMA emulation software layer for performing an RDMA read operation according to an aspect of the present application is presented.

[0007] Figure 6 A diagram illustrating a communication flow between two virtual machines having an RDMA emulation software layer for closing a communication channel according to an aspect of the present application is presented.

[0008] Fig. 7A A flow chart illustrating a method of facilitating software-based TCP-based RDMA communications according to an aspect of the present application is presented.

[0009] Figure 7B A flow chart illustrating a method of facilitating software-based TCP-based RDMA communications according to an aspect of the present application is presented.

[0010] Figure 8 A computer system that facilitates software-based TCP-based RDMA communications according to an aspect of the present application is illustrated.

[0011] In the drawings, like reference numerals refer to like drawing elements. DETAILED DESCRIPTION

[0012] Aspects of the present application provide a software layer that simulates the RDMA protocol and promotes RDMA communication based on TCP. As described above, some current computing systems (e.g., virtualized non-stop (vNS) systems) have been deployed on private clouds, which may require specific hardware components with RDMA capabilities (e.g., RDMA (RoCE) NICs based on converged Ethernet). Standard cloud products may not support RoCE NICs. In addition, some current computing systems run based on UDP and support the use of a small amount of vendor-specific hypervisors. Modifying a specific operating system (OS) (e.g., non-stop OS) to run on other hypervisors may require a lot of cost and time (in years). In addition, some current computing systems may be limited to using orchestration tools dedicated to private clouds. Implementing cloud native orchestration tools for other cloud environments may also require a lot of cost and time (also in years).

[0013] Various aspects of the present application provide a system and method for addressing the above-mentioned limitations. The described aspects provide a software stack that can be deployed on both private and public clouds, i.e., performing RDMA operations over TCP connections without the use of RoCE NICs and without being limited to any specific hypervisor. The software stack may include a software layer (e.g., Non-stop Virtual Remote Direct Memory Access (NVRDMA)) that emulates the TCP-based RDMA protocol, thereby eliminating the RoCE NIC requirement. The RDMA emulation software layer may further allow the use of any hypervisor that uses a generic Ethernet NIC, thereby eliminating the limitation of vendor-specific hypervisors. The following is about Figure 1 A detailed description of the RDMA emulation software layer in the architecture is provided, and the following Figures 2 to 6 A diagram is provided for establishing and closing a communication channel and performing RDMA operations.

[0014] The terms "virtual RDMA (VRDMA)" and "virtual RDMA NIC (VRDMA NIC)" are used interchangeably in the present disclosure and refer to an RDMA emulation software layer. In the described aspects, the RDMA emulation software layer may include a first software layer on a first virtual machine and a second software layer on a second virtual machine. "Nonstop VRDMA (NVRDMA)" is an example of virtual RDMA.

[0015] The term "RDMA-capable hardware component" may refer to hardware or networking components that can provide (eg, UDP-based) RDMA communications. An example of an RDMA-capable hardware component may be an RDMA over Converged Ethernet (RoCE) NIC.

[0016] The term "connection" refers to a communication channel that can be established between two computing entities, including a virtual machine and a physical machine. A connection or communication channel can be established between hardware components or software layers of the computing entities.

[0017] An environment for facilitating software-based TCP-based RDMA communications

[0018] Figure 1 An environment 100 for facilitating software-based TCP-based RDMA communication according to one aspect of the present application is illustrated, the environment including a computing system with two virtual machines. The environment 100 may include two virtual machines (VMs): a processor VM 110; and a cluster input / output (I / O) module (CLIM) VM 150. The processor VM 110 may include: a first operating system (OS) 120 having dedicated cores (e.g., 121, 122, and 123); and a side processing unit (SPU) 130 having dedicated cores (e.g., 131, 132, and 133), the side processing unit including a second operating system, for example, a FreeBSD side OS 140. The OS 120 may include the following layers, such as: a library for handling connection services (e.g., an uninterruptible connection service library (NCSL) 124); an open structure enterprise distribution (OFED) framework 126; and a virtual RDMA unit (e.g., a virtual RDMA NIC 128) including a software layer. FreeBSD 140 may include: a TCP / Internet Protocol (IP) layer 142; and an Ethernet NIC driver 144 for communicating with an Ethernet NIC 148 via a hypervisor 146. CLIM VM 150 may include: a library for handling connection services (e.g., NCSL 152); an OFED framework 154; a virtual RDMA unit including a software layer (e.g., virtual RDMA NIC 156); a TCP / IP layer 158; and an Ethernet NIC driver 160 for communicating with an Ethernet NIC 164 via a hypervisor 162.

[0019] The OS 120 and the FreeBSD side OS 140 of the SPU 130 may operate side by side within the confines of the processor VM 110 using separate resources (e.g., dedicated cores for each OS). The OS 120 and the FreeBSD side OS 140 may communicate with each other using producer-consumer queues located in the shared memory interface 112 (e.g., via communications 170 and 172). The VRDMA NIC event queue may be used by the OS 120 to send events / commands from the OS 120 kernel to the SPU 130, and the SPU event queue may be used by the SPU 130 to send notifications and command responses from the SPU 130 to the OS 120 kernel. The OS 120 and the FreeBSD side OS 140 may also communicate using inter-processor interrupts (IPIs) 174 between the OS 120 cores (121-123) and the SPU cores (131-133). The SPU cores (131-133) may provide an offloaded TCP layer to improve VRDMA NIC performance. IPI 174 may be used to cause consumers on both OS 120 and FreeBSD side OS 140 to process queued events.

[0020] Processor VM 110 and CLIM VM 150 may communicate with each other via one or more fabric switching networks (e.g., via X fabric switching network 104 via communications 180 and 182, or via Y fabric switching network 106 via communications 184 and 186) via their respective Ethernet NICs 148 and 164. Networks 104 and 106 may be redundant fabrics, as the VRDMA software layer may require each virtual machine to have two Ethernet ports to support fault tolerance. The ports may be provided by a single dual-port Ethernet NIC or two single-port Ethernet NICs. Providing the link activity status of the Ethernet port to the VRDMA software layer may allow the VRDMA software layer to represent the link activity status as a VRDMA NIC port status.

[0021] Although in Figure 11, but the environment 100 may include other components related to the processor VM 110, such as a boot loader for the FreeBSD side OS 140 for starting the OS 120 and the SPU 130. The FreeBSD Side OS 140 may be started first, and the OS 120 may be started subsequently. The environment 100 may also include coordinated error handling and recovery logic to respond to emergency events that may be encountered by either OS. The environment 100 may also include tools for collecting and analyzing merged core dumps that contain states from two operating systems (OS) and supportability features of both OSs (e.g., event logging and performance monitoring). In addition, the environment 100 may include support for providing key-value initialization parameters to the processor VM 110 across multiple hypervisors. The CLIM VM 150 does not require a side OS because it can run a specific Linux OS.

[0022] Additionally, the two virtual machines (processor VM 110 and CLIM VM 150) may be deployed and managed using multi-cloud orchestration tool 102. Tool 102 may be generalized to interface with many multi-cloud orchestrator frameworks widely available among hyperscalers, which may eliminate the requirement to write tools natively for multiple cloud providers.

[0023] In some aspects, processor VM 110 may include an uninterruptible processor VM, OS 120 may include an uninterruptible OS, CLIM VM 150 may include an uninterruptible CLIM VM, and virtual RDMA NIC 128 / 156 may include an RDMA software emulation layer or an uninterruptible virtual RDMA (NVRDMA) NIC.

[0024] Establishing a communication channel using software-based RDMA over TCP

[0025] Figure 2 A diagram 200 is presented that illustrates a communication flow between two virtual machines (a first VM 202 and a second VM 204) having an RDMA emulation software layer for creating a communication channel according to one aspect of the present application. In some aspects, the terms "client" and "server" are used to describe the initiator and responder of a particular sequence of operations. For example, in diagram 200, the first VM 202 can be considered a client, and the second VM 204 can be considered a server. Although Figures 2 to 6The communication in is depicted as being initiated by the first VM 202 (where the first VM 202 acts as a client and the second VM 204 acts as a server), but the communication may occur in the reverse direction (i.e., where the second VM 204 acts as a client and the first VM 202 acts as a server). In addition, the first virtual machine and the second virtual machine may be located on the same or different computing systems or entities.

[0026] exist Figures 2 to 6 , bold lines and text may indicate communications between the virtual RDMA software layers of the two virtual machines. Non-bold lines and text may indicate communications through RDMA-capable hardware components (e.g., communications tunneled through the UDP protocol using the RoCE line-level protocol). These same communications in the non-bold lines may also occur when virtual RDMA is used instead of RDMA-capable hardware components. Thus, the communications in bold text correspond to various aspects of the description of the present application. Additionally, in Figures 3 to 6 In the example, virtual machines 302 / 304, 402 / 404, 502 / 504 and 602 / 604 may respectively depict Figure 2 Non-limiting example of virtual machines 202 / 204 in.

[0027] As part of establishing a communication channel, hardware components may communicate via unreliable datagram (UD) queue pairs (QPs) using, for example, InfiniBand (IB) connection management (CM) messages, as depicted by the three IB CM messages in diagram 200: a request (REQ) message 214; a reply (REP) message 244; and an RTU message 254.

[0028] In the various aspects described, to establish a communication channel between QPs using the RDMA emulation software layer, the same three IB CM messages (shown in non-bold) are sent, but additional VRDMA layer messages (shown in bold) are also sent and woven between the line-level protocols. Instead of using the hardware line-level protocol (e.g., via the RoCE NIC), the VRDMA layer can use its own line-level protocol operating over, for example, UDP port 18402. In addition to sending the UD QP's messages, the VRDMA layer can also establish a TCP connection between the QPs. The software layers above the VRDMA layer may still be unaware of the TCP connection. The VRDMA layer can use a standard three-way handshake as well as two additional protocol messages (both shown in bold) to establish the TCP connection.

[0029] During operation, the VRDMA on the first VM 202 may listen on TCP port 18402 and start a thread to block executing accept(), while the server on the second VM 204 may call rdma_listen() to listen for incoming connection requests (not shown). The client on the first VM 202 may call rdma_connect() to connect to the QP (operation 208), which causes a REQ 214 message to be sent. The REQ 214 may include a local QP ID to be used for subsequent communications, and may be sent as a frame 210 prepended with a VRDMA protocol header 212. The VRDMA of the second VM 204 may receive the REQ 214 and notify the server, which may call rdma_accept() to accept the connection (operation 216). As a result of calling rdma_accept(), the second VM 204 may modify the state of the QP (operation 216), for example, by moving the QP to a "ready to receive" state using a modify_qp() verb. The VRDMA on the second VM 204 may issue a connect() and block (operation 218) to establish a TCP connection with the first VM 202 via a standard three-way handshake: the VRDMA of the second VM 204 may transmit a SYN 220 message to the VRDMA of the first VM 202; the VRDMA of the first VM 202 may then transmit a SYNACK 222 message to the VRDMA of the second VM 204; and the VRDMA of the second VM 204 may send an ACK 226 message back to the VRDMA of the first VM 202.

[0030] Upon receiving the ACK 226, the VRDMA of the first VM 202 may determine that the previously called accept() returns the socket (i.e., the VRDMA of the first VM 202 may identify the stream socket associated with the TCP connection), but the QP associated with the socket is unknown (operation 228). Upon receiving the SYNACK 222 by the VRDMA of the second VM 204, the previously called connect() may return (operation 224). Additionally, the VRDMA of the second VM 204 may transmit the first protocol message by sending the remote QP ID of the new TCP socket via the TCP socket (operation 230 of sending a QP_ID request (REQ) 232). Upon receiving the QP_ID information (of the QP_ID REQ 232), the VRDMA of the first VM 202 may associate the socket with the identifier of the QP (operation 234), and may also transmit the second protocol message by sending an acknowledgment (QP_ID ACK 236) of the received QP_ID information (232). These two protocol messages allow the software layer (ie, the VRDMA of both virtual machines 202 and 204) to associate the stream sockets on each side of the connection with their associated QPs.

[0031] The second VM 204 may complete the rdma_accept() process previously started in operation 216 (operation 238) and transmit a REP 244 message, which may be sent as a frame 240 preceded by a VRDMA protocol header 242. In response to the REP 244 message, the VRDMA of the first VM 202 may send its final message (operation 246) as an RTU message 254, which may be sent as a frame 250 preceded by a VRDMA protocol header 252. Finally, the VRDMA of the first VM 202 may notify the first VM 202 that the connection is accepted (operation 256), and the VRDMA of the second VM 204 may notify the second VM 204 that the connection is complete (operation 258).

[0032] In diagram 200, communications 240 and 236 are depicted as interleaved because they are not interdependent. Although communications 232 and 236 occur sequentially after three-way handshake communications 220, 222, and 226 to establish a TCP connection, communications 240 and 250 may occur before or after communications 232 and 236.

[0033] Perform RDMA operations using software-based RDMA over TCP

[0034] Figure 3 A diagram 300 is presented illustrating a communication flow between two virtual machines (302 and 304) having an RDMA emulation software layer for performing an RDMA send operation in accordance with an aspect of the present application.

[0035] When performing an IB send operation using an RDMA-capable NIC (i.e., a RoCE NIC), the client may call the post_send() verb, which formats a work queue element (WQE) describing the send operation and typically writes directly to the RoCE NIC hardware to trigger the RoCE NIC to process the WQE. As part of the RoCE line-level protocol, the hardware may transmit data associated with the send operation over Ethernet using IB send frames tunneled over UDP. The RoCE line-level protocol may provide the option of cumulatively acknowledging the entire sequence of send frames with a single ACK frame (e.g., Figure 3 The transfer of data from the transmit frame to the receive message buffer can be performed entirely in hardware.

[0036] When using TCP-based VRDMA as the underlying data transfer protocol, VRDMA can use a much larger frame size (e.g., as a byte stream) than when using UDP-based RoCE to transmit data. Therefore, although multiple frames can be used, VRDMA may not require multiple frames to transmit the send data. In addition, the transfer of data from the send frame to the receive message buffer can be performed in software by VRDMA. VRDMA's simulated line-level protocol can allow the software layer to piggyback on the ACK sent by the client and the server. That is, the RDMA emulation software on VMs 302 and 304 can allow additional information to be piggybacked on messages that have been passed back and forth between VMs 302 and 304, thereby obtaining more efficient communication and performance, i.e., performance optimization. In contrast, the RoCE line-level protocol does not allow such performance optimization.

[0037] During operation, a client on the first VM 302 may call a post_send() verb (operation 302), and in response, the VRDMA on the first VM 302 may issue a sendmsg() (operation 304) using the QP stream socket that was previously connected to the first VM 302. Figure 2314. The VRDMA of the first VM 302 may prepend a first VRDMA protocol header 312 to an RDMA send frame (send_1 / frame_1 314), and may optionally postpend VRDMA extension data 316 to the RDMA send frame. The first VRDMA protocol header 312 may be associated with an RDMA emulation software layer, and the VRDMA extension data 316 may include additional data for optimization in subsequent communications. The prepended (and optionally postpended) RDMA send frame (e.g., a byte stream or "encapsulated" frame 310) may be transmitted to the second VM 304 via a stream socket. The second VM 304 may receive the encapsulated frame 310 (i.e., received into a socket buffer by the underlying Ethernet hardware), and the VRDMA of the second VM 304 may transfer the data associated with send_1 / frame_1 314 (i.e., from the socket buffer) to a receive message buffer associated with the second VM 304 (operation 318). The first VM 302 may transmit additional RDMA send frames that are also prepended and optionally postpended, for example: the VRDMA of the first VM 302 may prepend a VRDMA protocol header 322 to an RDMA send frame (send_1 / frame_2 324) and may optionally postpend VRDMA extension data 326 to the RDMA send frame 324; and the VRDMA of the first VM 302 may prepend a VRDMA protocol header 332 to an RDMA send frame (send_1 / frame_N 334) and may optionally postpend VRDMA extension data 336 to the RDMA send frame 334. These additional prepended and optionally postpended RDMA send frames may be transmitted as byte streams or “encapsulated” frames 320 and 330 and received by the second VM 304, and the associated data may be transferred by the VRDMA of the second VM 304 to a receive message buffer associated with the second VM 304 (e.g., operations 328 and 338).

[0038] The second VM 304 may send a cumulative acknowledgment (send_1ACK 358) of all send_1 frames (e.g., frames 314, 324, and 334) (operation 340). The VRDMA of the second VM 304 may piggyback or add to the cumulative acknowledgment 358 by prepending the second VRDMA protocol header 352, send_2 / frame_X 354, and VRDMA extension data 356 (operation 342). The second VRDMA protocol header 352 may indicate the size of the frame 354, the extension data 356, and the cumulative acknowledgment 358. In some aspects, the extension data may not be included, in which case the second VRDMA protocol header 352 may indicate the size of the frame 354 and the cumulative acknowledgment 358. In other aspects, the second VM 304 may not transmit the send frame 354, in which case the VRDMA of the VM 304 may send the cumulative acknowledgment 358 alone with the prepended second protocol header 352 (indicating the size of the cumulative acknowledgment 358).

[0039] The VRDMA of the second VM 304 may issue a sendmsg() using the QP stream socket (operation 342), and may transmit the pre-appended cumulative acknowledgment 358 (e.g., a byte stream or "encapsulation" frame 350) to the first VM 302 via the TCP socket. Upon receiving the cumulative acknowledgment 358, the VRDMA of the first VM 302 may notify the first VM 302 that the send has been completed (operation 360). Additionally, in the event that send_2 / frame_X 354 is included in the encapsulation frame 350, the VRDMA of the first VM 302 may transfer the data associated with the send frame 354 to a receive message buffer associated with the first VM 302 (operation 362), and may also notify the first VM 302 that the receive has been completed (operation 364).

[0040] Figure 4 A diagram 400 is presented that illustrates a communication flow between two virtual machines (402 and 404) having an RDMA emulation software layer for performing an RDMA write operation according to an aspect of the present application. Diagram 400 may be similar to diagram 300, with one difference being that the destination location of the data in the send frame of diagram 300 is not known or defined by the sender, whereas the destination location of the data in the RDMA write frame of diagram 400 is identified by the sender.

[0041] During operation, a client on the first VM 402 may call a post_send() verb (operation 402), and in response, the VRDMA on the first VM 402 may issue a sendmsg() (operation 404) using the QP TCP socket that was previously connected to the first VM 402. Figure 2418. The VRDMA of the first VM 402 may prepend a second VRDMA protocol header 412 to the RDMA write frame (RDMA write frame_1 414), and may optionally postpend VRDMA extension data 416 to the RDMA write frame_1 414. The second VRDMA protocol header 412 may be associated with an RDMA emulation software layer, and the VRDMA extension data 416 may include additional data for optimization in subsequent communications. The prepended (and optionally postpended) RDMA write frame (e.g., a byte stream or “encapsulated” frame 410) may be transmitted to the second VM 404 via a TCP socket. The second VM 404 may receive the encapsulated frame 410 and transfer the data associated with the RDMA write frame_1 414 to an RDMA write target buffer associated with the second VM 404 (operation 418). The first VM 402 may transmit additional RDMA write frames that are also prepended and optionally postpended, e.g., the VRDMA of the first VM 402 may prepend a VRDMA protocol header 422 for RDMA write frame_2 424 and may optionally postpend VRDMA extension data 426 for RDMA write frame_2 424; and the VRDMA of the first VM 402 may prepend a VRDMA protocol header 432 for RDMA write frame_N 434 and may optionally postpend VRDMA extension data 436 for RDMA write frame_N 434. These additional prepended and optionally postpended RDMA write frames may be transmitted as byte streams or “encapsulated” frames 420 and 430 and received by the second VM 404, and the associated data may be transferred by the VRDMA of VM 404 to an RDMA write target buffer associated with the second VM 404 (e.g., operations 428 and 438).

[0042] The second VM 404 may send a cumulative acknowledgement (RDMA Write ACK 458) of all RDMA write frames (e.g., frames 414, 424, and 434) (operation 440). The VRDMA of the second VM 404 may piggyback or add to the cumulative acknowledgement 458 by prepending a third VRDMA protocol header 452 (operation 442). The VRDMA of the second VM 404 may also prepend another frame (e.g., frame_X 454) to the RDMA Write ACK 458 and may also prepend VRDMA extension data (not shown). The third VRDMA protocol header 452 may indicate the size of frame_X 454 (if prepended), the extension data (if prepended), and the RDMA Write ACK 458. Figure 3300, the second VM 404 may not transmit frame_X 454, in which case the VRDMA of VM 404 may send the cumulative acknowledgment 458 alone with the third protocol header 452 prepended (indicating the size of the cumulative acknowledgment 458).

[0043] The pre-appended cumulative acknowledgment 458 (e.g., a byte stream or "encapsulated" frame 450) may be transmitted to the first VM 402 via a TCP socket. Upon receiving the cumulative acknowledgment 458, the VRDMA of the first VM 402 may notify the first VM 402 that the RDMA write has completed (operation 460). Additionally, in the event that frame_X 454 is included in the encapsulated frame 450, the VRDMA of the first VM 402 may process frame_X 454 (which may correspond to any RDMA operation) (operation 462).

[0044] Figure 5 A diagram 500 is presented that illustrates the communication flow between two virtual machines (502 and 504) having an RDMA emulation software layer for performing an RDMA read operation according to an aspect of the present application. Diagram 500 may be similar to diagrams 300 and 400, but includes some communication in the reverse direction based on the depiction of the first VM 502 as the initiating entity.

[0045] As described above, when performing an IB RDMA read operation using an RDMA-capable NIC (i.e., a RoCE NIC), the client can call the post_send() verb, which formats a work queue element (WQE) describing the RDMA read operation and typically writes directly to the RoCE NIC hardware to trigger the RoCE NIC to process the WQE. The client's hardware can send the RDMA read request to the server's hardware, and the server can return the requested data using an IB RDMA read reply frame transmitted over a UDP tunnel as part of the RoCE line-level protocol. Similar to the send and RDMA write operations described above, the RoCE line-level protocol can provide the option of cumulatively confirming the entire sequence of send frames with a single ACK frame. The transfer of data from the RDMA read reply frame to a local buffer on the client can be performed entirely in hardware.

[0046] When using VRDMA over TCP (rather than using RoCE over UDP) as the underlying data transfer protocol, VRDMA can use a much larger frame size. Therefore, although multiple frames can be used, VRDMA may not require multiple frames to transmit RDMA read reply data (such as Figure 5504). In addition, the transfer of data from the RDMA read reply frame to the local buffer can be performed in software via VRDMA. The RDMA emulation software on VMs 502 and 504 can allow additional information to be piggybacked onto messages (such as RDMA read reply frames) that are already being passed back and forth between VMs 502 and 504. As described above, this performance optimization cannot be achieved using the RoCE line-level protocol.

[0047] During operation, a client on the first VM 502 may call a post_send() verb (operation 502), and in response, the VRDMA on the first VM 502 may issue a sendmsg() (operation 504) using the QP TCP socket that was previously connected to the first VM 502. Figure 2 514. The VRDMA of the first VM 502 may prepend a second VRDMA protocol header 512 to an RDMA read request frame (RDMA read request 514). The second VRDMA protocol header 512 may be associated with an RDMA emulation software layer and may indicate the size of the RDMA read request 514. The prepended RDMA read request (e.g., byte stream or "encapsulation" frame 510) may be transmitted to the second VM 504 via a TCP socket. The second VM 504 may obtain and return the requested read data (operation 516). The VRDMA of the second VM 504 may issue a sendmsg() using a QP TCP socket (operation 518), and may prepend a third VRDMA protocol header 522 to an RDMA read reply frame (RDMA read reply frame_524), and may optionally append VRDMA extension data 526 to the RDMA read reply frame. The pre-appended (and optionally post-appended) RDMA read reply frame (e.g., byte stream or "encapsulation" frame 520) may be transmitted via a TCP socket to the first VM 502. After receiving the RDMA read reply frame_1 524, the first VM 502 may transfer the data associated with the RDMA read reply frame_1 524 to a local buffer associated with the first VM 502 (operation 528).

[0048] The second VM 504 may transmit additional RDMA read reply frames, which are also prepended and optionally postpended, for example: the VRDMA of the second VM 504 may prepend a VRDMA protocol header 532 for RDMA read reply frame_2 534, and may optionally postpend VRDMA extension data 536 for RDMA read reply frame_2 534; and the VRDMA of the second VM 504 may prepend a VRDMA protocol header 542 for RDMA read reply frame_N 544, and may optionally postpend VRDMA extension data 546 for RDMA read reply frame_N 544. These additional prepended and optionally postpended RDMA read reply frames may be transmitted as byte streams or “encapsulated” frames 530 and 540 and received by the first VM 502, and the associated data may be transferred by the VRDMA of the first VM 502 to a local buffer associated with the first VM 502 (e.g., operations 538 and 548). Upon receiving the RDMA read reply frames 524, 534, and 544, the VRDMA of the first VM 502 may notify the first VM 502 that the RDMA read has been completed (operation 550).

[0049] Closing the communication channel using software-based RDMA over TCP

[0050] Figure 6 A diagram 600 illustrating a communication flow between two virtual machines ( 602 and 604 ) having an RDMA emulation software layer for closing a communication channel in accordance with an aspect of the present application is presented.

[0051] When releasing a connection between reliable connections (RC) QPs using a RoCE NIC, the hardware may exchange a pair of IB CM messages (e.g., a disconnect request (DREQ) 614 message and a disconnect response (DREP) 634 message) over a UD QP, which may be tunneled over UDP as part of the RoCE line-level protocol.

[0052] When releasing the connection between RC QPs using VRDMA over TCP, VRDMA may transmit the same pair of messages, but the VRDMA software layer may use its own protocol operating over UDP port 18402. Thus, in addition to the messages sent over the UD QPs, the VRDMA software layer may exchange a pair of TCP reset segments, which may abruptly release or close the TCP connection. The VRDMA software layer may release the TCP connection between QPs, and software layers above the VRDMA level may remain unaware of the released TCP connection.

[0053] During operation, a client on the first VM 602 may call a rdma_disconnect() or rdma_destroy_id() verb (operation 602) and issue a DREQ 614, which may be sent as a frame 610 preceded by a VRDMA protocol header 612. In response to operation 602, the VRDMA on the first VM 602 may set the socket delay to a value of "0", call shutdown() and close(), and issue a first reset (RST) 608 message (operation 604). Additionally, in response to operation 602, the second VM 604 (e.g., an RDMA library) may receive the DREQ 614 and modify the state of the QP (operation 620), for example, by moving the QP to an "error" state using a modify_qp() verb.

[0054] In response to operation 620, the VRDMA on the second VM 604 may set the socket delay to a value of "0", call shutdown() and close(), and issue a second reset (RST) 624 message (operation 622). In addition, in response to operation 620, the VRDMA of the second VM 604 may reply to the DREQ 614 by sending a DREP 634 (operation 626), which may be sent as a frame 630 preceded by a VRDMA protocol header 632. The first VM 602 may receive the DREP 634 (operation 622), and the VRDMA associated with the first VM 602 may notify the first VM 602 that the connection is closed (operation 624). In addition, the VRDMA associated with the second VM 604 may notify the second VM 604 that the connection is closed (operation 620).

[0055] Improved Example

[0056] The various aspects described (including the RDMA emulation software layer) can be packetized by a byte stream transmission protocol and implement performance optimizations, such as minimizing memory copies and avoiding penalties incurred in soft RoCE due to attempts to create reliable communications based on UDP. The various aspects described can also utilize additional cores to mitigate the increased overhead of emulating RDMA using TCP / IP and Ethernet. In addition, the various aspects described can use a low-latency shared memory interface between the first OS of the processor VM and the unloaded TCP / IP stack. Additionally, if the system is deployed on a server with a RoCE adapter, the various aspects of the present application can retain the ability to utilize the RDMA protocol implemented in hardware.

[0057] The described aspects may also include multi-cloud orchestration tools (e.g., Figure 1Tools 102 depicted and described). In addition to eliminating the requirement to write tools natively for multiple cloud providers, multi-cloud orchestration tools can also enforce anti-affinity rules for VMs and storage to enable fault-tolerant system deployments, for example, by deploying multiple processor VMs and paired CLIM VMs in different servers, or by deploying virtual drive pairs on different storage pools to leverage storage mirroring features implemented by the first OS of the processor VM. The multi-cloud orchestration tool can also support deployments based on: single root input / output virtualization (SR-IOV); peripheral component interconnect (PCI) pass-through; and fully virtualized system fabric interfaces. In addition to the improvements described herein, there can be other improvements.

[0058] Method for facilitating software-based TCP-based RDMA communications

[0059] Fig. 7A A flowchart 700 is presented that illustrates a method for facilitating software-based TCP-based RDMA communications according to an aspect of the present application. During operation, a system establishes a transmission control protocol (TCP) connection with a second virtual machine through a remote direct memory access (RDMA) emulation software layer in a first virtual machine (operation 702). In response to establishing the TCP connection, the system identifies, through the RDMA emulation software layer in the first virtual machine, a TCP socket associated with the TCP connection based on socket information received from the second virtual machine (operation 704). If the first virtual machine did not initiate an RDMA send operation (decision 706), the operation Figure 7B Operation 736 continues.

[0060] If the first virtual machine initiates an RDMA send operation (decision 706), the system prepends a first protocol header associated with the RDMA emulation software layer to the RDMA send frame through the RDMA emulation software layer in the first virtual machine (operation 708). The system transmits the prepended RDMA send frame through the TCP socket through the RDMA emulation software layer in the first virtual machine (operation 710). The system performs the RDMA send operation through the TCP connection (operation 712). Figure 7B Continue at A.

[0061] Figure 7BA flowchart 720 is presented according to an aspect of the present application, the flowchart illustrating a method for facilitating software-based TCP-based RDMA communication. During operation, the system transmits an RDMA send frame to a second virtual machine, which causes data associated with the RDMA send frame to be transferred to a receive message buffer associated with the second virtual machine (operation 722). The system notifies the second virtual machine that the reception of the data in the RDMA send frame has been completed (operation 724). The system receives a cumulative acknowledgment of the RDMA send frame and any additional RDMA send frames associated with the RDMA send operation from the second virtual machine through the first virtual machine, the cumulative acknowledgment being preceded by a second protocol header associated with the RDMA emulation software layer, and the second protocol header indicating the size of the cumulative acknowledgment (operation 726). The system notifies the first virtual machine that the RDMA send operation has been completed (operation 728). The cumulative acknowledgment is further preceded by a second RDMA send frame and additional data, and the second protocol header indicates the size of the second RDMA send frame, the additional data, and the cumulative acknowledgment (operation 730).

[0062] The system transmits data associated with the second RDMA send frame to a receive message buffer associated with the first virtual machine through the first virtual machine (operation 732). The system notifies the first virtual machine that reception of the data in the second RDMA send frame has been completed (operation 734). Therefore, as a result of the TCP connection and the RDMA emulation software layer established in the two virtual machines, the system uses the RDMA emulation software layer to perform one or more of an RDMA send operation, an RDMA write operation, and an RDMA read operation through the TCP connection without using a dedicated NIC with RDMA capabilities (operation 736). The operation returns.

[0063] Computer system for facilitating software-based TCP-based RDMA communications

[0064] Figure 8A computer system 800 is illustrated that facilitates software-based TCP-based RDMA communication according to an aspect of the present application. The computer system 800 may include a first virtual machine 802 and a second virtual machine 852. The first virtual machine 802 may include: a pair of operating systems (not shown); a processor or core 812 dedicated to each OS; a shared memory 814, via which the pair of operating systems can communicate; a hardware component 816 that can execute a line-level protocol; and an RDMA emulation software unit 820, which may include a unit for performing the methods described herein. For example, the RDMA simulation software unit 820 may include: a communication unit 822, which is used to transmit / receive requests, messages, data, frames, pre-attached frames, post-attached frames or encapsulated frames; a TCP management unit 824, which is used to establish, manage or close a TCP connection; a frame encapsulation unit 826, which is used to pre-attach and post-attach information (such as the pre-attached protocol header and the post-attached additional data) to a frame or an RDMA frame; and an RDMA operation management unit 828, which is used to handle at least RDMA send operations, RDMA write operations and RDMA read operations.

[0065] The second virtual machine 852 may include: a hardware component 866 that can execute a line-level protocol; and an RDMA simulation software unit 870, which may include units 872, 874, 876 and 878 for executing the methods described herein and related to units 822-828 of the RDMA simulation software unit 820.

[0066] First virtual machine 802 and second virtual machine 852 can communicate over network 890 via Ethernet NIC 804 associated with first virtual machine 802 and Ethernet NIC 854 associated with second virtual machine 854, respectively. Network 890 can be any fabric switched network over which communications can occur via Ethernet NICs.

[0067] Although not depicted, each of the first virtual machine 802 and the second virtual machine 804 can include volatile memory (e.g., RAM) that serves as managed memory and can be used to store one or more memory pools. In addition, each of the first virtual machine 802 and the second virtual machine 804 can be coupled to a peripheral input / output (I / O) user device (e.g., a display device, a keyboard, and a pointing device) (not shown). Each of the first virtual machine 802 and the second virtual machine 804 can also include a storage device (not shown) that stores data.

[0068] Elements of computer system 800 may also represent non-transitory computer-readable storage media storing instructions that, when executed by a computer, cause the computer to perform the methods described herein (e.g., as performed by first and second virtual machines 802 and 804, including by RDMA emulation software units 820 and 870). Computer system 800 may include, for example, Figure 8 Fewer or more units than shown.

[0069] Although the described embodiments depict that the use of an RDMA emulation software layer eliminates the requirement for an RDMA-capable hardware component such as a RoCE NIC, in some aspects a RoCE NIC may also be associated with each of the first and second virtual machines 802 and 804. The RoCE NIC, when included, may operate in parallel with the RDMA emulation software layer described herein.

[0070] Aspects and variations of the present application

[0071] In general, the disclosed aspects provide a method, a computer system, and a non-transitory computer-readable storage medium for facilitating software-based TCP-based RDMA communication. In one aspect, the system establishes a transmission control protocol (TCP) connection with a second virtual machine through a remote direct memory access (RDMA) emulation software layer in a first virtual machine. In response to establishing the TCP connection, the system identifies a TCP socket associated with the TCP connection through the RDMA emulation software layer in the first virtual machine based on socket information received from the second virtual machine. In response to the first virtual machine initiating an RDMA send operation, the system performs the following operations: prepending a first protocol header associated with the RDMA emulation software layer to an RDMA send frame through the RDMA emulation software layer in the first virtual machine; and transmitting the prepended RDMA send frame through the RDMA emulation software layer in the first virtual machine via a TCP socket. The system performs an RDMA send operation through a TCP connection.

[0072] In a variation on this aspect, the RDMA emulation software layer includes: a first software layer in the first virtual machine; and a second software layer in the second virtual machine.

[0073] In a further variation, the first protocol header indicates the size of the RDMA send frame. The system transmits the RDMA send frame to the second virtual machine, which causes data associated with the RDMA send frame to be transferred to a receive message buffer associated with the second virtual machine. The system receives a cumulative acknowledgment of the RDMA send frame and any additional RDMA send frames associated with the RDMA send operation from the second virtual machine through the first virtual machine. The cumulative acknowledgment is preceded by a second protocol header associated with the RDMA emulation software layer, and the second protocol header indicates the size of the cumulative acknowledgment. The system notifies the first virtual machine that the RDMA send operation has been completed.

[0074] In a further variation, the cumulative acknowledgment is further preceded by a second RDMA send frame and additional data, and the second protocol header indicates the size of the second RDMA send frame, the additional data, and the cumulative acknowledgment. The system transmits the data associated with the second RDMA send frame to a receive message buffer associated with the first virtual machine through the first virtual machine. The system notifies the first virtual machine that reception of the data in the second RDMA send frame has been completed.

[0075] In a further variation, in response to the first virtual machine initiating an RDMA write operation, the system performs the following operations: appending a second protocol header associated with the RDMA emulation software layer to the RDMA write frame through the first software layer; and transmitting the pre-appended RDMA write frame through the TCP socket through the first software layer, so that the data associated with the RDMA write frame is transferred to the RDMA write target buffer associated with the second virtual machine. The system receives a cumulative acknowledgment of the RDMA write frame and any additional RDMA write frames associated with the RDMA write operation from the second virtual machine through the first virtual machine. The cumulative acknowledgment is pre-appended with a third protocol header associated with the RDMA emulation software layer, and the third protocol header indicates the size of the cumulative acknowledgment. The system notifies the first virtual machine that the RDMA write operation has been completed.

[0076] Further variations include one or more of the following: the RDMA write frame is followed by additional data, and the second protocol header preceded indicates the size of the RDMA write frame and the additional data; and the cumulative confirmation is further preceded by a second RDMA write frame and the additional data, and the third protocol header indicates the size of the second RDMA write frame, the additional data, and the cumulative confirmation. The system transmits data associated with the second RDMA write frame to an RDMA write target buffer associated with the first virtual machine through the first virtual machine. The system notifies the first virtual machine that the RDMA write operation has been completed.

[0077] In a further variation, in response to the first virtual machine initiating an RDMA read operation, the system performs the following operations: appending a second protocol header associated with the RDMA emulation software layer to the RDMA read request through the first software layer; and transmitting the appended RDMA read request through the TCP socket through the first software layer; in response to the transmission, the second virtual machine obtains the data requested in the RDMA read request in an RDMA read reply frame and returns the data, the RDMA read reply frame is prepended with a third protocol header associated with the RDMA emulation software layer, and the third protocol header indicates the size of the RDMA read reply frame. The system receives the prepended RDMA read reply frame through the first virtual machine. The system transmits the data obtained in the RDMA read reply frame to a local buffer associated with the first virtual machine through the first virtual machine. The system notifies the first virtual machine that the RDMA read operation has been completed.

[0078] In a further variation, the RDMA read reply frame is followed by additional data, and the third protocol header indicates the size of the RDMA read reply frame and the additional data.

[0079] In a further variation, the system establishes a TCP connection in response to a first message transmitted by the first virtual machine for establishing a communication channel with the second virtual machine. The system transmits a request to close the communication channel with the second virtual machine through the first virtual machine. In response to the transmitted request to close the communication channel, the system transmits a first reset message to close the TCP connection of the first virtual machine to the second software layer through the first software layer, the transmitted request causes the second virtual machine to issue a response to the request, and further causes the second software layer to transmit a second reset message to close the TCP connection of the second virtual machine to the first software layer. The system receives a response to the request through the first virtual machine. The system notifies the first virtual machine and the second virtual machine that the communication channel is closed.

[0080] In a further variation, identifying a TCP socket for transmitting data associated with a TCP connection includes: receiving, by a first software layer from a second software layer, an identifier of a queue pair corresponding to the TCP socket; associating, by the first software layer, the TCP socket with the identifier of the queue pair; transmitting, by the first software layer to the second software layer, a confirmation of the identifier; and after exchanging a reply confirming the first message and a confirmation of the reply, notifying the first virtual machine and the second virtual machine that a communication channel has been successfully established.

[0081] In a further variation, the system performs one or more of an RDMA send operation, an RDMA write operation, and an RDMA read operation over a TCP connection without using a dedicated RDMA-capable network interface card.

[0082] In another aspect, a computer system includes a first virtual machine and a second virtual machine. The first virtual machine includes: a first operating system, the first operating system is associated with a first software layer that emulates a remote direct memory access (RDMA) protocol; and a second operating system, the second operating system implements a transmission control protocol (TCP) and communicates with a first Ethernet NIC associated with the second operating system via a first hypervisor. The second virtual machine includes: a second software layer, the second software layer emulates the RDMA protocol, the second virtual machine implements TCP and communicates with a second Ethernet NIC associated with the second virtual machine via a second hypervisor.

[0083] In a variation on this aspect, the first operating system and the second operating system communicate via a shared memory interface and inter-processor interrupts.

[0084] In a further variation, the first virtual machine and the second virtual machine communicate via one or more fabric switched networks using the first Ethernet NIC and the second Ethernet NIC.

[0085] In a further variation, a multi-cloud orchestration tool deploys and manages the first virtual machine and the second virtual machine.

[0086] In a further variation, the first virtual machine includes a virtual machine in a massively parallel processing architecture (MPP) (e.g., an uninterruptible processor virtual machine), the first operating system includes an operating system in the MPP architecture (e.g., an uninterruptible OS), the second virtual machine includes a cluster input / output module virtual machine in the MPP architecture (e.g., an uninterruptible CLIM VM), and the first software layer and the second software layer include an RDMA emulation software layer. In a further variation, the first software layer and the second software layer are deployed in the first virtual machine and the second virtual machine based on at least one of the following: single root input / output virtualization; peripheral component interconnect pass-through; and a fully virtualized system structure interface.

[0087] In yet another aspect, a non-transitory computer-readable storage medium stores instructions that, when executed by a computer, cause the computer to perform the above method, including Figures 2 to 6 , Fig. 7A and Figure 7B The method described.

[0088] The foregoing description is presented to enable any person skilled in the art to make and use the various aspects and examples, and is provided in the context of a specific application and its requirements. Various modifications to the disclosed aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects and applications without departing from the spirit and scope of the present disclosure. Therefore, the various aspects described herein are not limited to the aspects shown, but are intended to conform to the maximum scope consistent with the principles and features disclosed herein.

[0089] In addition, the foregoing descriptions of various aspects have been presented for the purpose of illustration and description only. These descriptions are not intended to be exhaustive or to limit the various aspects described herein to the disclosed forms. Accordingly, many modifications and variations will be apparent to those of ordinary skill in the art. In addition, the above disclosure is not intended to limit the various aspects described herein. The scope of the various aspects described herein is defined by the appended claims.

Claims

1. A computer-implemented method comprising: Establishing a transmission control protocol (TCP) connection with the second virtual machine by a remote direct memory access (RDMA) emulation software layer in the first virtual machine; In response to establishing the TCP connection, identifying, by the RDMA emulation software layer in the first virtual machine, a TCP socket associated with the TCP connection based on the socket information received from the second virtual machine; In response to the first virtual machine initiating an RDMA sending operation, the following operations are performed: The RDMA simulation software layer in the first virtual machine adds a first protocol header associated with the RDMA simulation software layer before sending a frame for RDMA; as well as The RDMA emulation software layer in the first virtual machine transmits the RDMA send frame attached via the TCP socket; as well as The RDMA send operation is performed through the TCP connection.

2. The method of claim 1, wherein the RDMA simulation software layer comprises: a first software layer in the first virtual machine; as well as A second software layer in the second virtual machine.

3. The method according to claim 2, The first protocol header indicates the size of the RDMA send frame, wherein: The method further comprises: transmitting the RDMA send frame to the second virtual machine, which causes data associated with the RDMA send frame to be transferred to a receive message buffer associated with the second virtual machine; receiving, by the first virtual machine from the second virtual machine, a cumulative acknowledgment of the RDMA Send frame and any additional RDMA Send frames associated with the RDMA Send operation, The cumulative acknowledgment is preceded by a second protocol header associated with the RDMA emulation software layer, and the second protocol header indicates a size of the cumulative acknowledgment; and The first virtual machine is notified that the RDMA send operation has been completed.

4. The method according to claim 3, The cumulative acknowledgement is further preceded by a second RDMA send frame and additional data, The second protocol header indicates the size of the second RDMA send frame, the additional data, and the cumulative acknowledgement, and in, The method further comprises: transmitting, by the first virtual machine, data associated with the second RDMA transmit frame to a receive message buffer associated with the first virtual machine; and The first virtual machine is notified that reception of the data in the second RDMA send frame has been completed.

5. The method of claim 2, further comprising: In response to the first virtual machine initiating an RDMA write operation, the following operations are performed: The first software layer appends a second protocol header associated with the RDMA emulation software layer before writing a frame for RDMA; as well as transmitting, by the first software layer via the TCP socket, the prepended RDMA write frame, which causes data associated with the RDMA write frame to be transferred to an RDMA write target buffer associated with the second virtual machine; receiving, by the first virtual machine from the second virtual machine, a cumulative acknowledgement of the RDMA write frame and any additional RDMA write frames associated with the RDMA write operation, The cumulative confirmation is preceded by a third protocol header associated with the RDMA emulation software layer, and the third protocol header indicates a size of the cumulative confirmation; and The first virtual machine is notified that the RDMA write operation has been completed.

6. The method of claim 5, further comprising one or more of the following: The RDMA write frame is followed by additional data, and the second protocol header previously appended indicates the size of the RDMA write frame and the additional data; and The cumulative confirmation is further preceded by a second RDMA write frame and additional data, the third protocol header indicates the size of the second RDMA write frame, the additional data and the cumulative confirmation, and in, The method further comprises: transferring, by the first virtual machine, data associated with the second RDMA write frame to an RDMA write target buffer associated with the first virtual machine; and The first virtual machine is notified that the RDMA write operation has been completed.

7. The method of claim 2, further comprising: In response to the first virtual machine initiating an RDMA read operation, the following operations are performed: The first software layer adds a second protocol header associated with the RDMA emulation software layer to the RDMA read request; Transmitting, by the first software layer via the TCP socket, the previously attached RDMA read request; In response to the transmission, the second virtual machine obtains the data requested in the RDMA read request in an RDMA read reply frame and returns the data, wherein the RDMA read reply frame is preceded by a third protocol header associated with the RDMA emulation software layer, and the third protocol header indicates a size of the RDMA read reply frame; The first virtual machine receives the previously attached RDMA read reply frame; The first virtual machine transmits the data obtained in the RDMA read reply frame to a local buffer associated with the first virtual machine; as well as The first virtual machine is notified that the RDMA read operation has been completed.

8. The method of claim 7, further comprising: The RDMA read reply frame is followed by additional data, and The third protocol header indicates the size of the RDMA read reply frame and the additional data.

9. The method of claim 2, further comprising: establishing the TCP connection in response to the first virtual machine transmitting a first message for establishing a communication channel with the second virtual machine; The first virtual machine transmits a request to close a communication channel with the second virtual machine; In response to the transmitted request to close the communication channel, the first software layer transmits a first reset message to close the TCP connection of the first virtual machine to the second software layer, The transmitted request causes the second virtual machine to issue a response to the request, and further causes the second software layer to transmit a second reset message to the first software layer to close the TCP connection of the second virtual machine; Receiving, by the first virtual machine, the response to the request; as well as The first virtual machine and the second virtual machine are notified that the communication channel is closed.

10. The method of claim 9, wherein: Identifying the TCP socket for transmitting data associated with the TCP connection comprises: receiving, by the first software layer from the second software layer, an identifier of a queue pair corresponding to the TCP socket; associating, by the first software layer, the TCP socket with an identifier of the queue pair; transmitting, by the first software layer to the second software layer, an acknowledgement of the identifier; and After exchanging replies confirming the first message and confirmations of the replies, the first virtual machine and the second virtual machine are notified of the successful establishment of the communication channel.

11. The method of claim 1, further comprising: One or more of the RDMA send operation, the RDMA write operation, and the RDMA read operation are performed over the TCP connection without using a dedicated network interface card with RDMA capabilities.

12. A computer system comprising: A first virtual machine, wherein the first virtual machine comprises: a first operating system associated with a first software layer that emulates a remote direct memory access (RDMA) protocol; and a second operating system that implements a Transmission Control Protocol (TCP) and communicates with a first Ethernet NIC associated with the second operating system via the first hypervisor; and A second virtual machine, wherein the second virtual machine comprises: a second software layer, wherein the second software layer emulates the RDMA protocol, The second virtual machine implements TCP and communicates with a second Ethernet NIC associated with the second virtual machine via a second hypervisor.

13. The computer system of claim 12, The first operating system and the second operating system communicate via a shared memory interface and inter-processor interrupts.

14. The computer system of claim 12, The first virtual machine and the second virtual machine communicate via one or more fabric switched networks using the first Ethernet NIC and the second Ethernet NIC.

15. The computer system of claim 12, in, The multi-cloud orchestration tool deploys and manages the first virtual machine and the second virtual machine.

16. The computer system of claim 12, The first virtual machine comprises a virtual machine in a massively parallel processing (MPP) architecture, The first operating system includes an operating system in the MPP architecture, The second virtual machine includes a cluster input / output module virtual machine in the MPP architecture, and The first software layer and the second software layer include an RDMA emulation software layer.

17. The computer system of claim 12, The first software layer and the second software layer are deployed in the first virtual machine and the second virtual machine based on at least one of the following: Single root input / output virtualization; Peripheral component interconnect pass-through; and Fully virtualized system architecture interface.

18. A non-transitory computer-readable storage medium storing instructions which, when executed by a computer, cause the computer to perform a method comprising: Establishing a transmission control protocol (TCP) connection with the second virtual machine by a remote direct memory access (RDMA) emulation software layer in the first virtual machine; In response to establishing the TCP connection, identifying, by the RDMA emulation software layer in the first virtual machine, a TCP socket associated with the connection based on the socket information received from the second virtual machine; as well as In response to the first virtual machine initiating an RDMA write operation, the following operations are performed: The RDMA emulation software layer in the first virtual machine appends a protocol header associated with the RDMA emulation software layer before writing a frame for RDMA; as well as transmitting, by the RDMA emulation software layer in the first virtual machine via the TCP socket, the prepended RDMA write frame, which causes data associated with the RDMA write frame to be transferred to an RDMA write target buffer associated with the second virtual machine; as well as receiving, by the first virtual machine from the second virtual machine, a cumulative acknowledgement of the RDMA write frame and any additional RDMA write frames associated with the RDMA write operation, The cumulative acknowledgement is preceded by a protocol header associated with the RDMA emulation software layer, and the protocol header indicates a size of the cumulative acknowledgement.

19. The non-transitory computer readable storage medium of claim 18, The RDMA write frame is followed by additional data, and the protocol header previously appended indicates the size of the RDMA write frame and the additional data; and The cumulative acknowledgement is further preceded by a second RDMA write frame and additional data, The protocol header indicates the size of the second RDMA write frame, the additional data, and the cumulative acknowledgement, and in, The method further comprises: transferring, by the first virtual machine, data associated with the second RDMA write frame to an RDMA write target buffer associated with the first virtual machine; and The first virtual machine is notified that the RDMA write operation has been completed.

20. The non-transitory computer readable storage medium of claim 18, The RDMA emulation software layer includes a first software layer in the first virtual machine, and the RDMA emulation software layer further includes a second software layer in the second virtual machine, in, Identifying the TCP socket associated with the TCP connection comprises: Receiving, by the RDMA emulation software layer in the first virtual machine from the RDMA second software layer, an identifier of a queue pair corresponding to the TCP socket; and associating, by the first software layer, the TCP socket with an identifier of the queue pair; transmitting, by the first software layer to the second software layer, an acknowledgement of the identifier; and After exchanging replies confirming the first message and confirmations of the replies, the first virtual machine and the second virtual machine are notified of the successful establishment of the communication channel.

Images