Untrusted rdma nic in secure computing environment

Abstract

The present disclosure relates to untrusted RDMA NICs in a secure computing environment. A computing device includes a bus interface and a central processing unit (CPU). The bus interface communicates with an untrusted remote direct memory access (RDMA) network adapter over a peripheral bus. The CPU runs a confidential virtual machine (CVM) and enables the CVM to perform RDMA transactions over a network through the untrusted RDMA network adapter by mapping one or more RDMA memory regions (MRs) to shared memory accessible by the untrusted RDMA network adapter.

Description

Technical Field

[0001] This disclosure relates generally to secure computing, and more particularly to methods and systems for performing Remote Direct Memory Access (RDMA) communication in a secure computing environment. Background Technology

[0002] Confidential computing refers to the use of advanced technologies to protect data as it is processed, ensuring that sensitive information remains secure even in untrusted environments. For example, this protection can be achieved using hardware-based trusted execution environments (TEEs). A TEE provides an isolated area for executing code and disposing of data, thus preventing unauthorized access. A confidential virtual machine (CVM) is a technology that enables confidential computing; it is a dedicated virtual machine designed specifically for running in a secure environment. CVMs use TEEs to provide strong data privacy and integrity guarantees, ensuring that workloads can be executed securely without exposing sensitive data to potential threats, even those from cloud providers or system administrators. Summary of the Invention

[0003] The embodiments described herein provide a computing device including a bus interface and a central processing unit (CPU). The bus interface is used to communicate with an untrusted remote direct memory access (RDMA) network adapter via a peripheral bus. The CPU is used to run a confidential virtual machine (CVM) and enables the CVM to perform RDMA transactions through the network adapter by mapping one or more RDMA memory regions (MRs) to shared memory accessible by the untrusted RDMA network adapter.

[0004] In some embodiments, the CPU also runs an untrusted hypervisor, and the CVM is used to communicate with an untrusted RDMA network adapter through this untrusted hypervisor. In one disclosed embodiment, the untrusted RDMA network adapter uses one or more queue pairs (QPs) to exchange packets with the CVM, and the CPU maps one or more QPs to shared memory.

[0005] In one exemplary embodiment, the CVM is used to send data packets by writing the encrypted payload of the data packet to a MR mapped to shared memory and publishing a work queue element (WQE) indicating a data packet to be sent on a queue pair (QP) associated with the MR. In one exemplary embodiment, the CVM receives data packets by reading a completion queue element (CQE) indicating a received data packet from the queue pair (QP) associated with the MR mapped to shared memory and reading the encrypted payload of the data packet from the MR.

[0006] In some embodiments, the CVM authorizes an untrusted RDMA network adapter to access a designated region in the shared memory of an external device coupled to the peripheral bus. In one exemplary embodiment, the external device is a graphics processing unit (GPU).

[0007] In one embodiment, the CVM is used to implement a key-value (KV) repository comprising encrypted key-value (KV) pairs in shared memory. In an exemplary embodiment, when a given MR is mapped to shared memory, the CVM should copy the existing content of the MR from private memory to shared memory unless an instruction is provided that the existing content can be discarded. In one embodiment, a user application in the CVM pre-allocates shared memory, and the CVM registers the allocated shared memory as an MR for use by RDMA.

[0008] Furthermore, according to the embodiments described herein, a computing method is also provided, comprising: running a confidential virtual machine (CVM) in a central processing unit (CPU). By mapping one or more remote direct memory access (RDMA) memory regions (MR) to shared memory accessible to an untrusted RDMA network adapter, the CVM is enabled to perform RDMA transactions over a network via the untrusted RDMA network adapter.

[0009] This disclosure will be more fully understood by referring to the accompanying drawings and the following detailed description of the embodiments. Attached Figure Description

[0010] Figure 1 The block diagram, according to the embodiments described herein, schematically illustrates a computing device that performs secure communication via an untrusted remote direct memory access (RDMA) network interface controller (NIC); Figure 2 The flowchart, based on the embodiments described herein, schematically illustrates a method for securely transmitting data from a CVM using an untrusted RDMA NIC; Figure 3The block diagrams, according to embodiments described herein, schematically illustrate a system including a computing device and a graphics processing unit (GPU) for secure communication via an untrusted RDMANIC; and Figure 4 The block diagram, based on the embodiments described herein, schematically illustrates a computing system employing secure communication via an untrusted RDMANIC. Detailed Implementation

[0011] Overview Traditional CVMs typically do not trust any external devices, including network adapters and even the host hypervisor. Therefore, all application memory in a CVM is usually mapped as private memory, inaccessible to external devices. Consequently, external devices cannot use Direct Memory Access (DMA) to read from or write data to CVM memory. As a result, traditional CVMs cannot communicate using RDMA or any other user-space networking protocols. Integrating RDMA with CVMs is particularly difficult because the entire RDMA protocol stack is typically implemented in hardware.

[0012] The embodiments described herein provide improved secure computing techniques that enable CVMs to use RDMA for secure communication via untrusted RDMANICs and / or untrusted hypervisors.

[0013] In the disclosed embodiments, the central processing unit (CPU) runs a CVM that accesses an untrusted RDMA NIC via a peripheral bus. The CVM runs kernel-space code (e.g., a driver) that interacts with the NIC. Specifically, the kernel maps various data structures used for communication between the CVM and the NIC (e.g., transmit and receive queue pairs (QPs), memory regions (MRs), and doorbells) to shared memory accessible to the NIC.

[0014] Based on this memory mapping, user-space applications running in CVM can communicate efficiently using RDMA transactions without compromising confidentiality. Since shared memory is untrusted, user-space applications are expected to encrypt the data written to it.

[0015] In one embodiment, the CVM sends a data packet by: (i) writing the encrypted payload of the data packet to a MapReduce (MR) mapped to shared memory; and (ii) publishing a Work Queue Element (WQE) on a queue pair (QP) associated with the MR, the WQE indicating the data packet to be sent. In response, the RDMA NIC reads the WQE, retrieves the data packet payload from the shared memory, generates the data packet header, and sends the data packet to the network.

[0016] In one embodiment, when the RDMA NIC receives a data packet, the NIC (i) writes the encrypted data packet payload to a MapReduce (MR) mapped to shared memory, and then (ii) publishes a Completion Queue Element (CQE) on a Queue Point (QP) associated with the MR, the CQE indicating the received data packet. The Completion Virtual Machine (CVM) reads the CQE and, in response, reads the data packet payload from the MR.

[0017] This document describes various implementations and use cases of the disclosed technology. In one exemplary embodiment, CVM implements a key-value store (KVS) in shared memory using one-sided RDMA operations (e.g., read and write operations). In another exemplary embodiment, CVM authorizes an untrusted RDMA NIC to access a designated area in the shared memory of an external device (e.g., a graphics processing unit (GPU)).

[0018] The methods and systems described in this paper enable computing devices to combine the confidentiality of confidential computing with the high communication performance of RDMA. Alternative solutions may involve a separate process of encrypting PCIe traffic, such as using the TEE Device Interface Security Protocol (TDISP). However, such solutions require multiple encryption and decryption operations for each transaction. In contrast, the techniques disclosed in this paper minimize the number of encryption and decryption operations, thereby reducing latency and power consumption.

[0019] System Description Figure 1 This is a block diagram schematically illustrating a computing device 20 that communicates securely via an untrusted RDMA network adapter (RDMA NIC 28 in this example) according to embodiments described herein. The computing device 20 may include, for example, a server in a data center or high-performance computing (HPC) cluster, or any other suitable computing device.

[0020] Computing device 20 is connected to NIC 28 via a peripheral bus, which in this example is a high-speed peripheral component interconnect (PCIe) bus 32. Alternatively, any other suitable peripheral bus may be used, such as an Extensible Interface (AXI), Compute Fast Link (CXL), Nvlink, or Nvlink Chip-to-Chip Link (Nvlink-C2C). NIC 28 provides computing device 20 with the ability to send and receive RDMA traffic over network 36. Network 36 may include, for example, an InfiniBand™ or Ethernet network.

[0021] In this example, computing device 20 includes a central processing unit (CPU) 24 and a bus interface (I / F) 40. The CPU 24 performs various computing tasks of the computing device. The interface 40 connects the CPU 24 to the PCIe bus 32.

[0022] CPU 24 runs hypervisor 44, which allocates computing resources to one or more confidential virtual machines (CVMs) 48. For clarity, only a single CVM 48 is shown in the figure. CVM 48 can access two types of memory: private memory 52 and shared memory 56.

[0023] Private memory 52 is part of the CVM user space and is inaccessible to external devices such as NIC 28 or hypervisor 44. Therefore, applications running in CVM 48 can write plaintext (unencrypted) data to private memory.

[0024] On the other hand, shared memory 56 is accessible to external devices (such as NIC 28 or hypervisor 44). At least a portion of shared memory 56 resides in the CVM kernel space. The CVM kernel code is expected to write encrypted data to shared memory. In various embodiments, shared memory 56 may be mapped to user address space and / or kernel address space.

[0025] The figure depicts private memory 52 and shared memory 56 as part of CPU 24. However, typically, private memory 52 and / or shared memory 56 can be physically located anywhere suitable, such as outside CPU 24, in NIC 28, etc. In some embodiments, one of the most significant bits (MSB) of the physical address is dedicated to marking whether each address is shared or private.

[0026] In one embodiment, CVM 48 communicates via network 36 by performing RDMA transactions using RDMA NIC 28. The techniques described herein enable CVM 48 to maintain the confidentiality of communication traffic even if both NIC 28 and hypervisor 44 are considered untrusted.

[0027] Confidential communication via untrusted RDMA NIC Figure 2 This is a flowchart illustrating, according to an embodiment described herein, a method for confidentially sending data from a CVM 48 using RDMA via an untrusted RDMA NIC 28. The method begins when a user application running in the user space of the CVM 48 needs to send data in the form of data packets to a peer endpoint.

[0028] In QP creation operation 60, kernel space code in CVM 48 creates a QP to establish an RDMA connection with a peer. The kernel space code connects the QP to the peer, for example, using an RDMA connection manager (CM). The kernel space code maps the QP to shared memory 56.

[0029] In memory registration operation 64, the kernel space code registers a memory region (MR) associated with the QP. The kernel space code also maps the MR to shared memory 56.

[0030] In payload storage operation 68, the user application writes the encrypted payload of the data packet to the MR. In some embodiments, the user application first writes the data to be sent (in plaintext) to private memory 52, then encrypts the data and writes the encrypted data to the MR in shared memory 56.

[0031] In WQE publish operation 72, the user application publishes a "SEND" WQE on the QP. Since both the QP and MR are mapped to shared memory 56, the RDMA NIC 28 can access them.

[0032] In packet construction operation 76, NIC 28 constructs an RDMA packet containing (i) a header and (ii) an encrypted payload. To construct the packet, NIC 28 generates the packet header and reads the encrypted payload from the MR in shared memory 56. The packet header is typically unencrypted. In packet transmission operation 80, NIC 28 sends the packet to the peer via network 36.

[0033] Figure 2 The flowchart shown is merely an example flow, chosen purely for clarity of concept. In alternative embodiments, the disclosed technique can be implemented using any other suitable flow. For example, in one embodiment, packet reception may include the following operations: 1. Receive data packets containing encrypted payloads via RDMA NIC 28. The data carried by the packets will be sent to a specific user application in CVM 48.

[0034] 2. NIC 28 writes the encrypted payload to the MR mapped to shared memory 56.

[0035] 3. NIC 28 publishes a Completion Queue Element (CQE) on the QP associated with the MR, which indicates the packets that have been received.

[0036] 4. User applications in CVM 48 read CQE.

[0037] 5. In response to CQE, the user application reads the encrypted packet payload from the MR in shared memory 56.

[0038] 6. The user application decrypts the payload to reconstruct the plaintext data. For example, the user application can store the plaintext data in private memory 52.

[0039] Typically, when a MapReduce (MR) is mapped to a memory range [addr1..addr2], the existing contents in [addr1..addr2] should be copied from private memory 52 to shared memory 56. In some embodiments, the user application can instruct the kernel code (e.g., using a flag) that the existing contents in [addr1..addr2] can be discarded. This mechanism allows CVM to avoid unnecessary copying operations. Alternatively, the application can directly allocate shared memory, thus requiring no additional action to map the MR to shared memory.

[0040] Use cases involving one-sided RDMA operations In some embodiments, certain RDMA transactions supported by the disclosed technology are one-sided operations, such as read and write. By using one-sided operations, CVM 48 enables one or more RDMA initiators to access encrypted data in shared memory 56.

[0041] One example use case is a Key-Value Repository (KVS), which shares plaintext KVS server storage among multiple clients. In this context, the term "client" refers to any suitable entity, such as those using KVS and trusting each other. In one embodiment, a client may negotiate an encryption key (not to be confused with the key that is part of a key-value pair) and then use one-sided RDMA read and write transactions to write the encrypted key-value pair to the KVS storage (in shared memory 56). Authentication data residing alongside the key-value pair can help identify potentially corrupted data that may occur when two clients simultaneously write to the same key.

[0042] GPU-Direct use case "GPU direct" refers to a term encompassing various technologies where I / O devices (e.g., network adapters or storage devices) bypass the CPU and directly read or write to the GPU's memory. For example, in a system including a VM, RDMA NIC, and GPU, the VM can register a region of GPU memory with the RDMA NIC and then perform RDMA read / write operations in that memory region without the involvement of the CPU's memory subsystem. In some embodiments described herein, the "GPU direct" scheme can be implemented securely, where the CVM runs in the CPU and the GPU-CVM (GCVM) runs in the GPU.

[0043] Figure 3 The block diagram illustrates a system 90 according to an alternative embodiment described herein, wherein a computing device 20 and a GPU 94 communicate securely via an untrusted RDMA NIC 28. Figure 3 The computing device 20 in Figure 1 The computing device 20 is similar. The GPU 94 includes a bus interface (I / F) 98 for connecting to the PCIe bus 32 and shared GPU memory 106. The GPU runs GCVM 102.

[0044] In one embodiment, CVM 48 (located in CPU 24) authorizes an untrusted RDMA NIC 28 to access a designated region in the shared memory 106 of GPU 94. After authorization, GCVM 102 can perform confidential RDMA transactions using the memory region in question via NIC 28 without the involvement of CPU 24. In one exemplary embodiment, CVM 48 designates the memory region based on a range of bus address ranges (BARs) addresses in the memory space of PCIe bus 32.

[0045] More generally, the above mechanism is not limited to GPUs but can also be used with other external devices coupled to the peripheral bus, such as storage devices. In such an embodiment, CVM authorizes an untrusted RDMA network adapter to access a designated region in the shared memory of an external device coupled to the peripheral bus. Once authorized, the external device can perform confidential RDMA transactions using the assigned memory region through the untrusted RDMA network adapter without CPU involvement.

[0046] like Figure 1 and Figure 3 As shown, the configurations of computing devices 20 and 90 and their components (e.g., CPU 24 and GPU 94) are merely illustrative and intended for conceptual clarity. Any other suitable configuration may be used in other embodiments.

[0047] In various embodiments, computing devices 20 and 90 and their components may be implemented using suitable software, suitable hardware (such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs)), or a combination of hardware and software.

[0048] In some embodiments, certain elements of computing devices 20 and 90, such as CPU 24 and / or GPU 94, are implemented using one or more general-purpose processors that are software-programmed to perform the techniques described herein. For example, the software may be downloaded to the processor electronically via a network, or alternatively or additionally provided and / or stored on a non-transitory tangible medium, such as magnetic, optical, or electronic memory.

[0049] Example System Use Cases Figure 4 This is a block diagram schematically illustrating a computing system 1000, such as a data center or high-performance computing (HPC) cluster, according to embodiments described herein, which communicates confidentially via an untrusted RDMA NIC. According to at least one embodiment, system 1000 includes multiple subsystems, such as multiple mutually coupled processing devices, multiple network devices, and multiple networks. The computing system 1000 is designed with multiple integrated circuits (referred to as processing devices), each of which may include one or more CPUs and GPUs, thereby forming a robust and flexible architecture.

[0050] Various processing devices are interconnected via NVLink or other high-speed interconnects to enable high-speed communication between subsystems, and are also connected via NICs or DPUs to ensure efficient data transfer across computing system 1000 and with one or more external networks 1030, 1036. In this example, system 1000 includes a packet switch 1048 that connects NIC / DPU 1028 to network 1030 and a packet switch 1050 that connects NIC / DPU 1032 to network 1036.

[0051] Seamless data exchange and parallel processing are enabled through NVLink coupling of the processing device, thereby improving overall computing performance. The processing device connects to multiple networks via one or more network interface cards (NICs) or DPUs, enabling the system to handle complex multi-network tasks with high bandwidth and low latency. This configuration is ideal for demanding applications requiring significant processing power, such as artificial intelligence (AI), machine learning (ML), and data-intensive computing, while ensuring robust connectivity and scalability across diverse networking environments. The integrated circuits of the computing system 1000 may include one or more CPUs and one or more GPUs.

[0052] Figure 4 An example architecture of a multi-GPU architecture is also demonstrated. As illustrated, computing system 1000 includes processing device 1002 with a multi-GPU architecture. Specifically, processing device 1002 may be a system-on-a-chip and includes multiple subsystems such as CPU 1006, GPU 1008, and GPU 1010. CPU 1006 may be coupled to GPU 1008 via die-to-die (D2D) or chip-to-chip (C2C) interconnects 1012 (such as ground reference signaling interconnects (GRS interconnects)). CPU 1006 may be coupled to GPU 1010 via D2D or C2C interconnects 1014. CPU 1006 may also be coupled to GPU 1008 and GPU 1010 via PCIe interconnects.

[0053] The CPU 1006 can be coupled to one or more NICs or DPUs, which in turn are coupled to one or more networks. For example, as Figure 4 As illustrated, CPU 1006 is coupled to a first NIC / DPU 1026, which is coupled to network 1030. CPU 1006 is also coupled to a second NIC / DPU 1028, which is coupled to network 1030 via switch 1048. For example, NIC / DPU 1026 and NIC / DPU 1028 can be coupled to network 1030 via Ethernet (ETH), NVLINK, or InfiniBand (IB) connections.

[0054] The computing system 1000 also includes a processing device 1004 with a multi-GPU architecture. Specifically, the processing device 1004 includes multiple subsystems, including a CPU 1016, a GPU 1018, and a GPU 1020. The CPU 1016 can be coupled to the GPU 1018 via a D2D or C2C interconnect 1022. The CPU 1016 can be coupled to the GPU 1020 via a D2D or C2C interconnect 1024. The CPU 1016 can also be coupled to the GPU 1018 and GPU 1020 via a PCIe interconnect. The CPU 1016 can be coupled to one or more NICs or DPUs, which in turn are coupled to one or more networks. For example, as... Figure 4As illustrated, CPU 1016 is coupled to a first NIC / DPU 1032, which is coupled to network 1036. CPU 1016 is also coupled to a second NIC / DPU 1034, which is coupled to network 1036 via switch 1050. NIC / DPU 1032 and NIC / DPU 1034 can be coupled to network 1036 via Ethernet (ETH), NVLINK, or InfiniBand (IB) connections.

[0055] In at least one embodiment, processing device 1002 and processing device 1004 can communicate with each other via NIC / DPU 1038 (such as via PCIe interconnect). Processing device 1002 and processing device 1004 can also communicate with each other via high-bandwidth communication interconnect 1040 (such as NVLink interconnect or other high-speed interconnect).

[0056] Figure 4 The packet switches in the system 1000 may include, for example, Nvidia Quantum-2 switches. The NICs / DPUs in the system 1000 may include, for example, Nvidia Bluefield DPUs. In various embodiments, any NIC / DPU of system 1000 may include an untrusted RDMA NIC. Any processing device of system 1000, such as devices 1002 and 1004, may employ the disclosed secure computing techniques.

[0057] While the embodiments described herein are primarily geared towards CVMs using RDMA communication, the methods and systems described herein can also be used with other user-space networking protocols. Non-limiting examples include Ultra Ethernet, Google Falcon, and user-space protocols built on frameworks such as DPDK or Google Snap.

[0058] Therefore, it should be understood that the embodiments described above are merely examples, and the present invention is not limited to the specific content shown and described above. The scope of the present invention includes not only combinations and sub-combinations of the various features described above, but also various variations and modifications that can be conceived by those skilled in the art after reading the foregoing specification and that are not disclosed in the prior art. Documents incorporated herein by reference shall be considered an integral part of this application; however, if any definition of a term in these incorporated documents conflicts with the express or implied definitions in this specification, only the definitions in this specification shall be considered.

Claims

1. A computing device, comprising: Bus interface for communicating with untrusted remote direct memory access (RDMA) network adapters via peripheral bus; as well as Central Processing Unit (CPU) is used for: Running a confidential virtual machine (CVM); as well as By mapping one or more RDMA memory regions (MRs) to shared memory accessible by the untrusted RDMA network adapter, the CVM is able to perform RDMA transactions on the network through the untrusted RDMA network adapter.

2. The computing device according to claim 1, wherein, The CPU is also used to run an untrusted hypervisor, wherein the CVM is used to communicate with the untrusted RDMA network adapter through the untrusted hypervisor.

3. The computing device according to claim 1, wherein, The untrusted RDMA network adapter uses one or more queue pairs of QPs to exchange data packets with the CVM, and the CPU is used to map the one or more QPs to the shared memory.

4. The computing device according to claim 1, wherein, The CVM is used to send data packets in the following manner: Write the encrypted payload of the data packet to the MR mapped to the shared memory; and A work queue element WQE indicating the data packet to be sent is published on the queue pair QP associated with the MR.

5. The computing device according to claim 1, wherein, The CVM is used to receive data packets in the following manner: Read the completion queue element CQE indicating the received data packet from the queue pair QP associated with the MR mapped to the shared memory; and The encrypted payload of the data packet is read from the MR.

6. The computing device according to claim 1, wherein, The CVM is used to authorize the untrusted RDMA network adapter to access a designated area in the shared memory of an external device coupled to the peripheral bus.

7. The computing device according to claim 6, wherein, The external device is a graphics processing unit (GPU).

8. The computing device according to claim 1, wherein, The CVM is used to implement a KV repository containing encrypted key-value pairs in the shared memory.

9. The computing device according to claim 1, wherein, When mapping a given MR to the shared memory, the CVM is used to copy the existing content of the MR from the private memory to the shared memory, unless an instruction is provided indicating that the existing content can be discarded.

10. The computing system according to claim 1, wherein, The user application in the CVM is used to pre-allocate shared memory, and the CVM is used to register the allocated shared memory as MR for use in RDMA.

11. A calculation method, comprising: The confidential virtual machine (CVM) runs in the central processing unit (CPU). as well as By mapping one or more Remote Direct Memory Access (RDMA) memory regions (MRs) to shared memory accessible by an untrusted RDMA network adapter, the CVM is enabled to perform RDMA transactions on the network via the untrusted RDMA network adapter.

12. The method of claim 11, further comprising: An untrusted hypervisor runs in the CPU, and the untrusted hypervisor communicates between the CVM and the untrusted RDMA network adapter.

13. The method according to claim 11, wherein, The untrusted RDMA network adapter uses one or more queue pairs (QPs) to exchange data packets with the CVM, and the CVM is able to perform RDMA transactions by mapping the one or more QPs to the shared memory.

14. The method of claim 11, further comprising sending data packets by the CVM in the following manner: Write the encrypted payload of the data packet to the MR mapped to the shared memory; and A work queue element WQE indicating the data packet to be sent is published on the queue pair QP associated with the MR.

15. The method of claim 11, further comprising: Data packets are received by the CVM in the following manner: Read the completion queue element CQE indicating the received data packet from the queue pair QP associated with the MR mapped to the shared memory; and The encrypted payload of the data packet is read from the MR.

16. The method of claim 11, further comprising: The CVM authorizes the untrusted RDMA network adapter to access a designated area in the shared memory of the external device.

17. The method of claim 16, wherein the external device is a graphics processing unit (GPU).

18. The method of claim 11, further comprising: The CVM implements a KV repository containing encrypted key-value pairs in the shared memory.

19. The method of claim 11, further comprising: When mapping a given MR to the shared memory, the CVM copies the existing content of the MR from the private memory to the shared memory, unless an instruction is provided indicating that the existing content can be discarded.

20. The method of claim 11, further comprising: Shared memory is pre-allocated by the user application in the CVM, and the CVM registers the allocated shared memory as MR for use in RDMA.

Images