High-performance heterogeneous trusted execution environment implementation method and system

Abstract

The present invention provides a high-performance heterogeneous trusted execution environment implementation method and system, including: constructing an XpuTEE monitor to manage all XpuEnclaves; the XpuEnclave is composed of at least one CEnclave and a plurality of XEnclaves; the CEnclave is an isolated environment on a CPU; the XEnclave is an isolated environment on a GPU; the CEnclave and the XEnclave belonging to the same XpuEnclave realize fast communication through an XpuEnclave channel; the XpuTEE monitor can directly access the CEnclave running on the CPU, and the XpuTEE monitor uses MMIO to access the XEnclave on the GPU.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular, to a method and system for implementing a high-performance heterogeneous trusted execution environment, and more particularly, to a software-based high-performance heterogeneous trusted execution environment. Background Art

[0002] After several years of development, artificial intelligence technology has been widely used in many fields, such as autonomous driving, face recognition, and medical imaging. The massive amount of training data and high-value artificial intelligence models have made data privacy issues a focus of attention. Due to their extensive resource requirements, modern artificial intelligence applications are usually deployed in high-performance data centers, such as where ChatGPT is trained, using more than 10K NVIDIA A100 GPUs. The complexity of the software stack and potential vulnerabilities (such as the Linux kernel with more than 360 million lines of code and more than 3K vulnerabilities), as well as potential malicious employees, threaten the data privacy of artificial intelligence applications. On the one hand, the platform administrator of the data center itself may have malicious behavior. On the other hand, the existing platform usually has a very deep software stack, which brings a large number of software vulnerabilities and attack surfaces. Once the cloud platform software system is hijacked, the security of the machine learning application cannot be guaranteed. Even if the training is carried out in the machine within the enterprise, the training process may be damaged by internal malicious employees or attackers. Therefore, how to ensure the security of machine learning applications and prevent them from being attacked by the hijacked operating system has become a problem that has received widespread attention from academia and industry.

[0003] Trusted Execution Environment (TEE) is widely used to protect applications from attacks by untrusted system software. TEE ensures that any component outside the TEE cannot access or modify the data flow and control flow of components inside the TEE. There are already many ways to build TEE, and almost all CPU manufacturers, including Intel, AMD, and ARM, have introduced their hardware TEE extensions. However, modern AI applications rely heavily on heterogeneous accelerators, such as GPUs, NPUs, etc., which are widely used to accelerate the calculation of AI applications. However, these accelerators cannot be directly protected by the TEE on the CPU side. Therefore, the security improvement of the TEE on the CPU side for AI applications is very limited.

[0004] Therefore, in order to protect the data security of AI applications, it is necessary to build a trusted execution environment on the AI ​​accelerator to ensure that malicious operating systems cannot access the accelerator's data or tamper with the accelerator's computing process. GPU is a common accelerator for AI applications, and building a GPU TEE is an effective way to protect AI applications.

[0005] The GPU is connected to the CPU via PCIe and has its own device memory. The GPU memory has a higher bandwidth to match the high parallelism of GPU threads. The GPU also supports virtual memory, whose page table is stored in the GPU memory by default and initialized by the GPU driver. For DMA, the copy engine (or DMA engine) can copy data between the GPU memory and other memories. At the same time, the command processing unit (Command Processor) obtains commands submitted by the user space or kernel space driver from the command queue (GPU channel) and distributes these commands to various internal engines. Finally, the GPU exposes some MMIO registers and some device memory specified by the PCIe base address register (BAR) to the CPU. When performing operations on the GPU, the software must submit commands through the GPU channel. The channel has a buffer, which is collectively referred to as the channel buffer here. The GPU has two channels with different privileges: privileged channels and non-privileged channels. Non-privileged channels cannot execute privileged methods. Any attempt to run privileged operations from such a channel will cause an interrupt and call the software driver. When creating a channel, the driver creates the corresponding structure in the GPU memory. For GPU memory areas that are not exposed by the PCIe BAR, the software needs to send commands through the GPU channel to let the DMA engine copy data to these areas. The CUDA driver, as a user-space driver for applications, can directly access a small portion of the GPU channel state through MMIO.

[0006] NVIDIA H100 GPU contains many security features that can limit untrusted software's access to GPU content. There is an on-chip security processor inside the GPU, which supports multiple types and levels of encryption and provides hardware-protected memory areas. The confidential computing function of NVIDIA H100 GPU needs to be used in conjunction with confidential virtual machines. When a confidential virtual machine is protected by the TEE on the CPU side, a confidential virtual machine can be associated with several H100 GPUs in secure computing mode, or several MIG computing instances. After that, the GPU will establish a secure channel mechanism with the confidential virtual machine on the CPU. The data transmission between the GPU and the CPU will be encrypted and decrypted by hardware. At the same time, the relevant hardware can ensure that untrusted virtual machine monitors cannot directly access the GPU in secure computing mode. NVIDIA H100 GPU allows users to verify that they are communicating with an NVIDIA GPU with confidential computing enabled and verify the security status of the GPU.

[0007] Currently, the internal details of its GPU have not been made public, and its specific GPU security protection mechanism cannot be obtained from existing public information.

[0008] NVIDIA's confidential computing technology has strict usage requirements and cannot be used on existing hardware. First, the H100 GPU is required, but there are a large number of existing GPUs in existing data centers. These GPUs cannot directly benefit from the confidential computing mechanism applied in the H100 GPU. At the same time, this technology requires the support of confidential virtual machines. Only a small number of CPUs can support the existing confidential virtual machines. Most of the existing CPUs cannot support NVIDIA's confidential computing technology. Therefore, its usage scenarios are relatively limited and require additional hardware replacement costs.

[0009] Some existing academic work usually modifies the CPU or GPU hardware to support GPU TEE. Some such work has already been done.

[0010] Graviton is a work done by Microsoft Research in collaboration with the University of Lisbon and was presented at the 2018 OSDI conference. The main purpose of the Graviton system is to protect AI programs running on the GPU through software and hardware solutions. The system assumes that existing TEE technologies such as Intel SGX have been used on the CPU to protect applications on the CPU. Graviton further uses hardware-layer encryption technology to ensure that AI applications can be run safely on the GPU. Specifically, the Graviton system ensures the privacy of code and data on the GPU by building an encrypted channel. Inside the CPU, there is a module called the Command Processor, which is responsible for receiving GPU operation instructions initiated by the CPU. The Graviton system has expanded the Command Processor by adding a series of trusted channel management instructions for the creation, destruction, and channel attribute configuration of trusted channels. When the GPU is needed for AI computing, the application can first obtain an encrypted channel key from the instruction processing unit in the GPU through remote verification. After that, each time an instruction is submitted to the GPU or data is uploaded, the key is used to protect the privacy and integrity of the instruction and data. This ensures that malicious system software cannot destroy the code and data privacy of AI applications in the GPU. After the ciphertext instructions and data are passed to the GPU, the instruction processing unit in the GPU will be able to decrypt these instructions and data inside the GPU and directly use the plaintext instructions and data for calculations. The core is to use encryption during the CPU-GPU transmission process to ensure that malicious operating systems without channel keys cannot steal instructions and data running on the GPU.

[0011] HIX is a system that supports GPU TEE by modifying CPU hardware. It was published at the 2019 ASPLOS conference. The HIX system attempts to directly extend Intel's SGX technology and further build a GPU-based TEE by protecting the GPU driver in a dedicated GPU enclave. Because this work does not require modifying the GPU hardware, it can theoretically support existing GPU devices. The enclave abstraction built by Intel SGX technology itself cannot directly use the GPU. To solve this problem, the HIX system protects the GPU driver by building a dedicated GPU enclave. The GPU enclave can directly access the GPU securely through MMIO, while the general enclave can run artificial intelligence tasks on the GPU by securely communicating with the GPU enclave across enclaves. Specifically, the HIX system will add an EGCREATE instruction to create a GPU enclave, and extend SGX's security check on memory to allocate the GPU's management memory exclusively to the GPU enclave, thereby ensuring that only the GPU driver in the GPU enclave can directly operate the GPU. In the process of copying data to the GPU, HIX enables the user program enclave and the GPU to share a key. After the user program encrypts the data using the shared key and transmits it to the GPU, the GPU can obtain the plaintext data using the shared key through the decrypted GPU kernel.

[0012] Because customized modifications are required and there is currently no hardware available on the market, it is difficult to deploy in an actual production environment, and existing infrastructure platforms cannot directly benefit from it.

[0013] Existing academic work does not require hardware-level modifications and uses software to support GPU TEE, but this work can only be applied to integrated graphics cards on ARM platforms.

[0014] StrongBox is a research work published by Southern University of Science and Technology at the CCS conference in 2022. This work uses the existing hardware features to build a GPU TEE to support confidential computing, which is mainly aimed at integrated graphics cards existing in ARM terminals. Unlike independent graphics cards that require PCIe connection, the integrated graphics card of the ARM platform shares a unified memory with the CPU. When performing confidential computing on the Arm GPU, StrongBox provides an isolated execution environment by ensuring exclusive access to the GPU. This work also utilizes the memory protection mechanism provided by TrustZone and the two-stage address translation provided by the virtualization extension to implement a fine-grained memory protection strategy, dividing the memory into non-secure memory and protected secure memory without hardware modification. This work requires running a trusted monitor in EL3 as a trusted base. The monitor will perform key operations such as memory permission configuration. The user program needs to encrypt and transmit the data to the monitor. The monitor decrypts the data set and interacts with the GPU through MMIO on behalf of the user program to start the computing task. This work used an existing Arm Mali GPU to test a prototype system, and in several benchmarks, the performance overhead was less than 20% compared to programs not protected by GPU TEE.

[0015] This work cannot be run on the x86 platform and cannot support independent graphics cards. However, in current data centers and cloud platforms, independent graphics cards are used for training. The computing power of integrated graphics cards is very limited and it is difficult to use them for training models with large computing requirements. Summary of the invention

[0016] In view of the defects in the prior art, the object of the present invention is to provide a high-performance heterogeneous trusted execution environment implementation method and system.

[0017] A method for implementing a high-performance heterogeneous trusted execution environment provided by the present invention includes:

[0018] Build an XpuTEE monitor to manage all XpuEnclaves;

[0019] The XpuEnclave is composed of at least one CEnclave and several XEnclaves; the CEnclave is an isolated environment on the CPU; the XEnclave is an isolated environment on the GPU;

[0020] CEnclave and XEnclave belonging to the same XpuEnclave achieve fast communication through the XpuEnclave channel;

[0021] The XpuTEE monitor can directly access the CEnclave running on the CPU, and the XpuTEE monitor accesses the XEnclave on the GPU through MMIO.

[0022] Preferably, an independent execution environment is created for the XpuTEE monitor to prevent the operating system from modifying the XpuTEE monitor.

[0023] Preferably, the XpuTEE monitor includes: a memory permission control module, a PCIe protection module, a channel management module and an Enclave authentication module;

[0024] The memory permission management module is responsible for controlling memory mapping and configuring memory permissions;

[0025] The PCIe protection module captures all modifications to the PCIe MMIO configuration and protects the integrity of the PCIe MMIO settings;

[0026] The channel management module constructs an XpuEnclave channel for fast communication between CEnclave and XEnclave;

[0027] The Enclave authentication module generates information for remote authentication, enabling users to remotely authenticate content running in the enclave.

[0028] Preferably, the memory permission management module includes: constructing enclave memory in the operating system and device memory to prevent untrusted CPU software or GPU tasks from accessing protected memory, and managing the memory mapping relationship on the CPU and GPU by exclusively controlling the CPU MMU, IOMMU and GPU MMU;

[0029] For the CPU MMU, the extended page table is used to control the permissions of physical memory. For the GPU MMU, it manages the conversion of the GPU's virtual address to the physical address and enforces isolation of different XEnclaves on the GPU. It also manages IOMMU mapping to prevent untrusted devices from accessing the CEnclave.

[0030] Preferably, the memory permission control module includes: CPU memory mapping, GPU memory mapping and IOMMU memory mapping, and constructs CEnclave and XEnclave through memory mapping;

[0031] The CPU memory mapping includes: controlling the memory access rights of the CPU through exclusive MMU configuration; maintaining different page tables for XpuEnclave and untrusted software programs, in which kernel code pages, page table pages and IOMMU page table pages are mapped as write protection; in addition, PCIe configuration memory and preset GPU MMIO areas are mapped as read-only, and the GPU driver running in the kernel space needs to configure the GPU through the MMIO registers on the GPU and the GPU memory exposed to the CPU through PCIe. The memory permission management module maps these areas used to configure the GPU as read-only in the page table of the system software, so the GPU driver in the kernel space must explicitly use the interface provided by XpuTEE to call the memory permission management module to operate the GPU; each GPU channel has a user-accessible GPU MMIO area, the memory permission management module maps this area to the page table of the corresponding application, so that the user space GPU can use the GPU channel to send instructions and create computing tasks on the GPU when running; the memory permission management module configures a preset page table for the XpuTEE monitor, which can access all physical addresses. In order to save the data and code of the XpuTEE monitor when running, the memory permission management will reserve a preset size of address space in the memory to protect the XpuTEE monitor itself. This physical memory area is not mapped in any other page table;

[0032] The GPU memory mapping includes: the GPU as an independent graphics card supports a virtual memory mechanism, and uses a page table stored in the GPU memory to convert the GPU virtual address into the GPU physical address; the GPU page table page is located in the GPUMMIO memory area exposed by the PCIe BAR; in order to exclusively control the GPU mapping, the memory permission management module maps the GPU MMIO area in the page table of the kernel space driver as read-only by controlling the memory mapping, and at the same time, it maps the buffer of the privileged GPU channel as read-only in the page table of the driver, and in addition, the message queue for the GPU firmware RPC is also mapped as read-only; after adopting such a mapping mode, the kernel space GPU driver must call the memory permission management module in the XpuTEE monitor to modify the GPU page table, and the memory permission management module can check all GPU memory mapping modifications at this time, and force the XEnclave's memory not to be mapped to untrusted tasks;

[0033] The IOMMU mapping includes: the memory permission management module exclusively controls the IOMMU; when the IOMMU is enabled, any access of the device to the system memory or other device memory must go through the IOMMU, which converts IOVA to IOPA; the IOMMU uses the root table and context table to map the device to the IOMMU page table, and the XpuTEE monitor maps the pages of the root table and context table, the IOMMU page table pages, and the configuration registers of the IOMMU as non-writable except for the page table of the XpuTEE monitor; therefore, the kernel must call the XpuTEE monitor to configure the IOMMU;

[0034] The CEnclave construction includes: CEnclave is a CPU-side TEE used to protect applications coordinated with XEnclave; XpuTEE ensures that the memory of a CEnclave can only be accessed by software in the corresponding XpuEnclave;

[0035] The XEnclave construction includes: XEnclave is created by CEnclave in the same XpuEnclave and cannot be tampered with by untrusted GPU tasks. Each GPU channel can set the page table it uses, so XpuTEE isolates the address space of the GPU channel as XEnclave, and data transmission between XEnclave and untrusted processes on the CPU is prohibited.

[0036] A high-performance heterogeneous trusted execution environment implementation system provided by the present invention includes:

[0037] Build an XpuTEE monitor to manage all XpuEnclaves;

[0038] The XpuEnclave is composed of at least one CEnclave and several XEnclaves; the CEnclave is an isolated environment on the CPU; the XEnclave is an isolated environment on the GPU;

[0039] CEnclave and XEnclave belonging to the same XpuEnclave achieve fast communication through the XpuEnclave channel;

[0040] The XpuTEE monitor can directly access the CEnclave running on the CPU, and the XpuTEE monitor accesses the XEnclave on the GPU through MMIO.

[0041] Preferably, an independent execution environment is created for the XpuTEE monitor to prevent the operating system from modifying the XpuTEE monitor.

[0042] Preferably, the XpuTEE monitor includes: a memory permission control module, a PCIe protection module, a channel management module and an Enclave authentication module;

[0043] The memory permission management module is responsible for controlling memory mapping and configuring memory permissions;

[0044] The PCIe protection module captures all modifications to the PCIe MMIO configuration and protects the integrity of the PCIe MMIO settings;

[0045] The channel management module constructs an XpuEnclave channel for fast communication between CEnclave and XEnclave;

[0046] The Enclave authentication module generates information for remote authentication, enabling users to remotely authenticate content running in the enclave.

[0047] Preferably, the memory permission management module includes: constructing enclave memory in the operating system and device memory to prevent untrusted CPU software or GPU tasks from accessing protected memory, and managing the memory mapping relationship on the CPU and GPU by exclusively controlling the CPU MMU, IOMMU and GPU MMU;

[0048] For the CPU MMU, the extended page table is used to control the permissions of physical memory. For the GPU MMU, it manages the conversion of the GPU's virtual address to the physical address and enforces isolation of different XEnclaves on the GPU. It also manages IOMMU mapping to prevent untrusted devices from accessing the CEnclave.

[0049] Preferably, the memory permission control module includes: CPU memory mapping, GPU memory mapping and IOMMU memory mapping, and constructs CEnclave and XEnclave through memory mapping;

[0050] The CPU memory mapping includes: controlling the memory access rights of the CPU through exclusive MMU configuration; maintaining different page tables for XpuEnclave and untrusted software programs, in which kernel code pages, page table pages and IOMMU page table pages are mapped as write protection; in addition, PCIe configuration memory and preset GPU MMIO areas are mapped as read-only, and the GPU driver running in the kernel space needs to configure the GPU through the MMIO registers on the GPU and the GPU memory exposed to the CPU through PCIe. The memory permission management module maps these areas used to configure the GPU as read-only in the page table of the system software, so the GPU driver in the kernel space must explicitly use the interface provided by XpuTEE to call the memory permission management module to operate the GPU; each GPU channel has a user-accessible GPU MMIO area, the memory permission management module maps this area to the page table of the corresponding application, so that the user space GPU can use the GPU channel to send instructions and create computing tasks on the GPU when running; the memory permission management module configures a preset page table for the XpuTEE monitor, which can access all physical addresses. In order to save the data and code of the XpuTEE monitor when running, the memory permission management will reserve a preset size of address space in the memory to protect the XpuTEE monitor itself. This physical memory area is not mapped in any other page table;

[0051] The GPU memory mapping includes: the GPU as an independent graphics card supports a virtual memory mechanism, and uses a page table stored in the GPU memory to convert the GPU virtual address into the GPU physical address; the GPU page table page is located in the GPUMMIO memory area exposed by the PCIe BAR; in order to exclusively control the GPU mapping, the memory permission management module maps the GPU MMIO area in the page table of the kernel space driver as read-only by controlling the memory mapping, and at the same time, it maps the buffer of the privileged GPU channel as read-only in the page table of the driver, and in addition, the message queue for the GPU firmware RPC is also mapped as read-only; after adopting such a mapping mode, the kernel space GPU driver must call the memory permission management module in the XpuTEE monitor to modify the GPU page table, and the memory permission management module can check all GPU memory mapping modifications at this time, and force the XEnclave's memory not to be mapped to untrusted tasks;

[0052] The IOMMU mapping includes: the memory permission management module exclusively controls the IOMMU; when the IOMMU is enabled, any access of the device to the system memory or other device memory must go through the IOMMU, which converts IOVA to IOPA; the IOMMU uses the root table and context table to map the device to the IOMMU page table, and the XpuTEE monitor maps the pages of the root table and context table, the IOMMU page table pages, and the configuration registers of the IOMMU as non-writable except for the page table of the XpuTEE monitor; therefore, the kernel must call the XpuTEE monitor to configure the IOMMU;

[0053] The CEnclave construction includes: CEnclave is a CPU-side TEE used to protect applications coordinated with XEnclave; XpuTEE ensures that the memory of a CEnclave can only be accessed by software in the corresponding XpuEnclave;

[0054] The XEnclave construction includes: XEnclave is created by CEnclave in the same XpuEnclave and cannot be tampered with by untrusted GPU tasks. Each GPU channel can set the page table it uses, so XpuTEE isolates the address space of the GPU channel as XEnclave, and data transmission between XEnclave and untrusted processes on the CPU is prohibited.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] Security: This system can protect sensitive data and code on both the CPU and GPU. When other untrusted applications try to steal protected process data, the protected program data cannot be accessed by them by ensuring that there is no memory mapping of the protected program in the untrusted application. Similarly, on the GPU, the protected GPU memory is isolated from other tasks on the GPU through the GPU's own page table, so that other GPU tasks cannot directly access the protected GPU memory. At the same time, even if the operating system is malicious, this system will hand over key operations on the GPU and CPU to a trusted micro-monitor, so that malicious operating systems cannot steal data in protected CPU programs and GPU tasks.

[0057] Compatibility with CPU devices: XpuTEE builds a trusted execution environment through software design and existing hardware, so there is no need for special hardware modifications to the CPU, and it does not rely on special CPU hardware extensions (such as Intel SGX, AMD-SEV, etc.). Therefore, most CPUs can use XpuTEE as long as they support ordinary page table isolation and virtualization support.

[0058] Compatibility with GPU devices: XpuTEE builds a trusted execution environment through software design and existing hardware. XpuTEE does not require special hardware modifications to the GPU and does not rely on special GPU hardware extensions (such as NVIDIA H100's confidential computing extensions). Therefore, most GPUs without integrated graphics cards can use XpuTEE.

[0059] Performance: XpuTEE builds a unified trusted execution environment through page table isolation. The secure computing tasks of the GPU can directly access the secure computing tasks on the CPU side. Therefore, the secure computing tasks of the GPU can directly read the secure memory on the CPU side without an encrypted channel, thereby saving the time of cryptographic operations and achieving efficient data transmission and communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0061] Figure 1 This is an abstract diagram of XpuTEE.

[0062] Figure 2 This is a diagram that provides an overview of the XpuTEE architecture.

[0063] Figure 3 This is a schematic diagram of the main modules of the XpuTEE monitor.

[0064] Figure 4 This is a diagram of XpuTEE memory mapping. DETAILED DESCRIPTION

[0065] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0066] Example 1

[0067] The present invention provides a high-performance heterogeneous trusted execution environment implementation method and system, which is a heterogeneous TEE abstraction of XpuEnclave, which spans the CPU and XPU. XpuEnclave includes a CEnclave for protecting CPU-side logic and multiple XEnclaves for protecting XPU tasks; the data and code in the XpuEnclave are protected and cannot be directly accessed by other applications and operating systems. The high-performance heterogeneous trusted execution environment XpuTEE builds CEnclave and XEnclave by building a micro monitor and isolates them from untrusted components. Multiple XEnclaves can exist on the GPU at the same time to share hardware resources; the effective and secure communication method proposed in the present invention can significantly reduce the performance overhead between CEnclave and XEnclave without the need for additional memory copies during communication.

[0068] In the application scenario of the present invention, the hardware and firmware of the CPU and GPU, including: CPU virtualization extensions and GPU firmware, are correctly implemented and trusted, and AI applications protected by TEE should not deliberately expose sensitive data. An attacker may hijack the underlying operating system and kernel space GPU driver to steal data protected in the TEE, and the attacker can even launch a TEE to attack other TEEs. Like most existing TEEs, the present invention does not consider denial of service (DoS) attacks or side channel attacks, such as cache-based time side channel attacks, power-based side channel attacks, etc. XpuTEE does not focus on physical memory attacks, such as cold-boot attacks and bus eavesdropping attacks, which can be defended by other methods, such as adding physical sensors to the device.

[0069] Previous work built TEEs separately on each accelerator, but did not propose a unified abstraction to manage and represent secure heterogeneous computing tasks. To address this limitation, the present invention introduces an abstract XpuEnclave (Cross-PU Enclave) for managing confidential computing tasks in heterogeneous platforms. XpuEnclave consists of at least one CEnclave and several XEnclaves. CEnclave is an isolated environment on the CPU, while XEnclave is an isolated environment on the accelerator, and multiple tasks on the accelerator can run in the same XEnclave. For each XpuEnclave, there should be a protected area in the system memory and each involved accelerator memory. The protected areas belonging to the same XpuEnclave in different memories constitute the enclave memory of the XpuEnclave, and the enclave memory of the XpuEnclave cannot be accessed by components outside the XpuEnclave. If the computing task is located in the XpuEnclave, no matter which accelerator the task runs on, then XpuTEE will not restrict its memory access to the enclave memory. According to the specific needs of the upper layer application, CEnclave can still isolate different XEnclaves within an XpuEnclave. Each XpuEnclave should contain at least one CEnclave. The CEnclave and XEnclave running in the same XpuEnclave are referred to as related in the present invention, such as Figure 1 shown.

[0070] The present invention is mainly aimed at general computing tasks of independent graphics cards, such as artificial intelligence and high-performance computing tasks. XpuTEE ensures that untrusted GPU tasks and CPU software cannot access the enclave memory of XpuEnclave. At the same time, it supports fast communication between CEnclave and XEnclave in the same XpuEnclave. XpuTEE does not introduce additional CPU-GPU memory copy or encryption operations for internal communication of XpuEnclave, so it has a small performance overhead. The key component of XpuTEE is a trusted XpuTEE monitor (XpuTEE monitor, XM), which intercepts all security-related operations in the system. The construction of XM can be based on memory and permission isolation or virtualization in the same permission level, without the need for special hardware modification. Therefore, XpuTEE has high compatibility and can be deployed on existing machines in data centers.

[0071] XpuTEE protects the entire application as a process-level CEnclave. Multiple GPU tasks from the same process can be started in different XEnclaves or in the same XEnclave. A CEnclave can create multiple XEnclaves belonging to the same XpuEnclave, and the XpuEnclave channel is used for fast communication between CEnclave and XEnclave, such as Figure 2 As shown;

[0072] The key functions of the system are all supported by an XpuTEE monitor (XM), which is the core of the entire system. In order to build this XpuTEE monitor, first of all, an independent execution environment must be created for this monitor to prevent the operating system from arbitrarily modifying the XpuTEE monitor. At the same time, the interior of XM is divided into several functional modules according to the different functions it is responsible for.

[0073] XM can be understood as the manager of all XpuEnclaves. It is because of XM that XpuEnclave can be supported. Some tasks that require privileged operations are performed by XM (because only XM is trusted, and the operating system is not trusted).

[0074] After XM is successfully built, it can support XpuEnclave. A protected user-mode application can use the programming interface to create an XpuEnclave. An XpuEnclave is divided into at least one CEnclave, which runs on the CPU logic; and several XEnclaves, which run on the GPU logic. The CEnclave and XEnclave belonging to the same XpuEnclave can access each other's memory. The efficient data access between the two is called the XpuEnclave channel. Ordinary GPU tasks cannot access XEnclave or CEnclave. Ordinary CPU applications cannot access XEnclave and CEnclave. Untrusted operating systems cannot directly access CEnclave or XEnclave. In addition to the XpuEnclave itself, only XM can access these XpuEnclaves. Because XM runs on the CPU, it can directly access the CEnclave running on the CPU. Because XEnclave runs on the GPU, XM needs to use MMIO to access the GPU.

[0075] For an XpuEnclave created by a process, only the XpuEnclave itself and XM are trusted. Each process can create its own XpuEnclave, and different XpuEnclaves are not trusted by each other, so they cannot access each other.

[0076] XpuEnclave is constructed and protected by XM. It configures memory mapping on the CPU and GPU and captures necessary system events such as privilege level transitions and process context switches. The GPU driver in kernel space does not directly access application data on the CPU or GPU. To protect XpuEnclave, XM inserts GPU access without porting all the logic of the GPU driver. According to previous research work, directly porting the GPU driver to the TCB significantly increases the chances of being exploited by attackers.

[0077] XM consists of four main functional modules, such as Figure 3 As shown:

[0078] Memory permission management module: This module constructs enclave memory in system and device memory to prevent untrusted CPU software or GPU tasks from accessing protected memory. This module manages the memory mapping relationship on the CPU and GPU by exercising exclusive control over the CPU MMU, IOMMU, and GPU MMU. For the CPU MMU, the extended page table (EPT) is used to control the permissions of physical memory. For the GPU MMU, it manages the conversion of the GPU's virtual address to the physical address and enforces isolation of different XEnclaves on the GPU. In addition, it also specifically manages IOMMU mappings to prevent untrusted devices from accessing the CEnclave.

[0079] PCIe Protection Module: This module captures all modifications to the PCIe MMIO configuration and protects the integrity of the PCIe MMIO setup.

[0080] Channel management module: This module builds XpuEnclave channels for secure and efficient communication, and reduces performance overhead by creating efficient communication methods between CEnclave and XEnclave.

[0081] Enclave Attestation Module: This module generates information for remote attestation, enabling users to remotely attest to the content running in the enclave.

[0082] Specifically, the memory permission management module includes:

[0083] The memory permission management module in XM is responsible for controlling memory mapping and configuring memory permissions. Therefore, this module can create enclave memory in system and GPU memory and provide necessary protection for the secure memory used by XM. This module controls three types of memory mapping: CPU memory mapping, GPU memory mapping, and IOMMU memory mapping, and builds CEnclave and XEnclave through memory mapping, such as Figure 4 As shown;

[0084] The CPU memory mapping includes: the memory permission management module controls the memory access rights of the CPU through the exclusive MMU configuration. It maintains different page tables for XpuEnclave and untrusted software programs. In all page tables, kernel code pages, page table pages and IOMMU page table pages are mapped as write protection. In addition, PCIe configuration memory and most GPU MMIO areas are mapped as read-only. The GPU driver running in the kernel space needs to configure the GPU through the MMIO registers on the GPU and the GPU memory exposed to the CPU through PCIe. The memory permission management module maps these areas used to configure the GPU as read-only in the page table of the system software. Therefore, the GPU driver in the kernel space must explicitly use the interface provided by XpuTEE to call the memory permission management module to operate the GPU. Each GPU channel has a user-accessible GPU MMIO area. The memory permission management module maps this area to the page table of the corresponding application, so the user space GPU can use the GPU channel to send instructions and create computing tasks on the GPU when it is running. The memory permission management module configures a special page table for XM, which can access all physical addresses. In order to save the data and code of XM runtime, the memory permission management will reserve 200MB of address space in the memory to protect XM itself. This physical memory area is not mapped in any other page table.

[0085] The GPU memory mapping includes: In order to prevent untrusted software from maliciously modifying the GPU memory mapping, XpuTEE must deprive them of the ability to configure the GPU MMU. As an independent graphics card, the GPU supports a virtual memory mechanism and uses a page table stored in the GPU memory to convert the GPU's virtual address to the GPU's physical address. The GPU page table page is located in the GPU MMIO memory area exposed by the PCIe BAR. Currently, there are three ways for software to modify the GPU page table: 1) directly modify the GPU page table page in the MMIO area; 2) send commands through a privileged GPU channel; or 3) send RPC to the GPU firmware. Non-privileged GPU channels (used by user space drivers in applications) cannot modify the GPU page table. In order to exclusively control the GPU mapping, the memory permission management module maps the GPU MMIO area in the kernel space driver's page table as read-only by controlling the memory mapping. At the same time, it maps the privileged GPU channel's buffer as read-only in the driver's page table. In addition, the message queue used for the GPU firmware RPC is also mapped as read-only. After adopting this mapping mode, the kernel space GPU driver must call the memory permission management module in XM to modify the GPU page table. The memory permission management module can now check all GPU memory mapping modifications and force XEnclave's memory not to be mapped to untrusted tasks.

[0086] The IOMMU mapping includes: the memory permission management module exclusively controls the IOMMU to further ensure the security of CPU-GPU communication. When the IOMMU is enabled, any access of the device to the system memory or other device memory must go through the IOMMU, which converts IOVA to IOPA. The IOMMU uses the root table and context table to map the device to the IOMMU page table, and the XM maps the pages of the root table and context table, the IOMMU page table pages, and the IOMMU configuration registers as non-writable except for the XM page table. Therefore, the kernel (including the GPU driver) must call the XM to configure the IOMMU. With this setting, the XM manages the IOMMU mapping, effectively limiting any modifications to the IOMMU mapping from untrusted software.

[0087] The CEnclave construction includes: CEnclave is a CPU-side TEE used to protect applications that coordinate with XEnclave. XpuTEE ensures that the memory of a CEnclave can only be accessed by software in the corresponding XpuEnclave.

[0088] All memory of CEnclave is not mapped in all untrusted code tables. XM can track all context switches and user-kernel transitions to capture CEnclave entry and exit. Only XM can modify privileged registers such as CR3. Binary checks can be performed to ensure that no other privileged register modification instructions such as CR3 appear in the system software. The system software must explicitly call XM's memory permission management module to modify CR3 and other privileged registers. The memory permission management module also intercepts all system calls, interrupts, and exceptions to capture the transitions between CEnclave and untrusted kernels. XpuTEE modifies the code of the corresponding processing function to enable the software to call XM directly at the entry and exit points. In addition, all kernel code pages are mapped as non-writable to prevent attackers from tampering with these codes. The page table is also mapped as read-only, so that the kernel cannot maliciously map new code pages and bypass the system call hook. Using the above method, XM can safely capture all user-kernel transitions.

[0089] Since XpuTEE protects the entire process as a CEnclave in user space, system calls should be supported in the protected process. When a system call occurs, the kernel still needs to access the CEnclave's memory to get the parameters. In XpuTEE, the kernel cannot directly access the CEnclave's memory. Therefore, it must call the XM's memory permission management module to copy the contents of the enclave to a buffer in the kernel. However, some system call parameters may be sensitive, such as the buffer passed to the write system call to write the contents to a file. In order to protect these sensitive parameters, two common methods can be used: 1) rely on XM (understanding the semantics of the system call) to transparently encrypt the parameters, or 2) the application itself encrypts the sensitive content before passing the parameters to the operating system kernel. XpuTEE can choose to let the application encrypt the parameters, reducing the complexity of XM.

[0090] XpuTEE manages device access to system memory through IOMMU. In addition to GPUs, some devices can also access system memory using DMA. When a device accesses system memory, the IOMMU hardware uses the IOMMU page table associated with the device for IOVA address translation. To prevent untrusted devices from arbitrarily accessing system memory, XM limits the kernel's ability to configure IOMMU. XM ensures that the memory mapping of IO devices is configured correctly and prevents untrusted devices from contacting CEnclave.

[0091] XpuTEE needs to control GPU access to system memory. On the one hand, it must ensure that untrusted applications cannot access CEnclave memory, and on the other hand, CEnclave needs to selectively share memory pages with XEnclaves in the same XpuEnclave. There are two ways for the GPU to access system memory: 1) map a system memory area to the GPU's virtual address and allow the GPU task to access the area; and 2) send a request to the GPU DMA engine to access the system memory. For the former method, XM controls all GPU memory mappings, and if the kernel attempts to map CEnclave memory to an untrusted GPU task, the operation will be intercepted and rejected. For the DMA method, untrusted applications can only send DMA requests through non-privileged GPU channels, but the GPU hardware enforces that DMA through non-privileged channels must use the GPU's virtual address. So XM's memory permission management module can also prevent such malicious DMA operations from the GPU by controlling the GPU page table and enforcing that CEnclave's memory is never mapped to untrusted GPU tasks. Although kernel-mode drivers can still call privileged GPU channels to send DMA requests with physical addresses, GPU drivers must call XM to access privileged GPU channels. XM can then check all pushed commands and prohibit invalid DMA operations.

[0092] The XEnclave construction includes: XEnclave is created by CEnclave in the same XpuEnclave and cannot be tampered with by untrusted GPU tasks. Each GPU channel can set the page table it uses, so XpuTEE isolates the address space of the GPU channel as XEnclave, and data transmission between XEnclave and untrusted processes on the CPU is prohibited.

[0093] As mentioned earlier, XM's memory permission management module exclusively controls memory mapping on the GPU. XpuTEE allows the GPU driver to allocate physical and virtual address ranges, but the driver must call XM to build the mapping between them. When CEnclave calls the driver to create a GPU channel, XM records the GPU page table pointer for the channel. For each 4KB page on the GPU, XM records the owner of the page. If the driver attempts to map a page belonging to an XpuEnclave to a GPU channel that is not allocated by the XpuEnclave, this behavior is considered malicious and is rejected.

[0094] For different XpuEnclaves, since GPU memory isolation is already enforced by XM, XpuTEE can support concurrent xpuenclaves. XpuTEE can support concurrent XEnclaves, in which XpuTEE does not enforce isolation of internal XEnclaves and allows applications to create shared GPU memory for them. This is because complex computing workloads usually consist of multiple GPU channels, and different channels may share part or all of the address space. CEnclave is allowed to set different page tables for different objects, create shared shared memory areas, or use the same page tables for all corresponding XEnclaves. It is important to ensure that the GPU does not fetch commands from the channel buffers of unrelated CPUs. There are registers on the GPU to store the addresses of the buffers of these channels, and they are automatically saved and stored by the GPU hardware during context switches. Because these addresses are still GPU virtual addresses, untrusted tasks cannot fetch commands from the buffers of CEnclaves.

[0095] While the OS kernel and kernel-space GPU drivers cannot modify GPU MMIO registers or memory directly, XpuTEE allows direct reading from GPU MMIO registers or memory to reduce runtime overhead since the GPU driver may poll on some GPU MMIO registers. Since the readable GPU MMIO region exposed to software only contains control registers and structures, their contents do not expose sensitive data of the application.

[0096] Specifically, the PCIe protection module includes:

[0097] Usually, only a small part of the GPU memory area used for GPU configuration is exposed to software access through MMIO. XpuTEE should prevent malicious modification of the MMIO configuration to expose more GPU memory operations. The base address and length of the PCIe MMIO area are determined by the BAR in the PCIe device configuration space. The BAR is initialized by the BIOS, but the kernel still has the authority to change the MMIO configuration by changing the BAR. In order to ensure the correctness of PCIe message routing, XpuTEE captures all modifications to the PCIe configuration. In modern x86 CPUs, the PCIe configuration can be modified in two ways: one is through the I / O port and the other is through a special memory mapping area. Using the first method, the PCIe configuration can be accessed through the I / O ports 0xCF8 and 0xCFC. The second method is also called the Enhanced Configuration Access Mechanism (ECAM), which reserves a total of 256MB of continuous physical space for the configuration space of all PCIe devices, 4KB for each device. This space is mapped to physical memory, and the processor converts loads or stores in this physical memory area into accesses to the PCIe configuration. XM's PCIe protection module checks the system software code to ensure that there are no instructions to directly read and write special I / O. At the same time, the ECAM area is mapped as read-only in the system's page table. Therefore, the PCIe configuration cannot be directly modified. The software must explicitly call XM's PCIe protection module to complete the configuration, and this module will ensure that the GPU's MMIO configuration will not be modified.

[0098] Specifically, the channel management module includes:

[0099] XpuTEE provides an XpuEnclave channel based on the GPU channel for CEnclave-XEnclave interaction in the same XpuEnclave without the need for additional encryption operations and memory copies. Through the XpuEnclave channel, the CEnclave can send commands to the GPU, copy data to the GPU memory, and start GPU tasks without other intermediate steps. The XpuEnclave channel is built based on the GPU channel. When the GPU channel is created by the CEnclave, the channel buffer is located in the protected memory area of ​​the CEnclave and is only mapped to the page table of the current CEnclave. Each GPU channel has an area in the GPU memory that is mapped to the user-mode application so that the user-mode program can observe the status of instruction processing. The channel management module ensures that this area is only mapped to the page table of the CEnclave. Other untrusted software running on the CPU, including the kernel, other CEnclaves, or unprotected processes, cannot directly access the channel buffer or the user-accessible area in the CEnclave. At this time, the protected GPU channel can only be accessed by the corresponding CEnclave, and such a channel is called an XpuEnclave channel. Based on the XpuEnclave channel, a CEnclave can securely interact with the corresponding XEnclave, and sensitive plaintext data can be copied directly from or to the XEnclave through such a channel. The XpuEnclave channel eliminates unnecessary encryption operations and memory copies, thereby achieving higher communication performance.

[0100] Specifically, the Enclave authentication module includes:

[0101] XpuTEE supports remote attestation with the help of a Trusted Platform Module (TPM). XM's Enclave attestation module exclusively controls the TPM by restricting access to its MMIO registers. XM can generate a key pair for attestation and have the TPM hold a public attestation key. The private attestation key is stored in XM's protected memory. The GPU code is compiled as part of the application binary, and at runtime it also goes through the JIT compiler in the userspace driver and is then transferred to the GPU. When the binary is loaded and a CEnclave is created, XM can traverse the pages of the binary and generate measurements, which can contain a hash of the page and a summary of the memory layout information. The information of the measurement can be signed by XM's private attestation key, and the summary, signature, and TPM's quote (used to prove the state of the TPM itself and the boot system) are used as the XpuEnclave's quote and used for remote attestation of the user. Because the GPU code is part of the binary, the quote also contains information about the GPU task, and the quote can be used to verify the initial state and identity of the XpuEnclave, similar to HyperEnclave and Intel SGX.

[0102] More specifically, the XpuEnclave creation process includes:

[0103] Each XpuEnclave contains at least one CEnclave. One CEnclave may create multiple XEnclaves. The creation process of XpuEnclave is as follows:

[0104] Step 1: A startup tool application is started and runs normally. The tool program calls the enclave_execve system call, which notifies the operating system to start creating a CEnclave. The parameters of the system call specify the binary file used to create the CEnclave.

[0105] Step 2: The operating system loads the binary file specified by the enclave_execve system call. Unlike the general execve system call, this system call causes the operating system to load the entire binary file into memory.

[0106] Step 3: The operating system calls xm_enclave_create, which is an XM call and directly enters XM.

[0107] Step 4: XM allocates internal data structures for the new XEnclave;

[0108] Step 5: XM now constructs a new EPT for CEnclave and moves the process's pages to protected pages;

[0109] Step 6: XM generates a quote for remote authentication and then returns the call;

[0110] Step 7: The operating system returns the process, and after returning, the process runs in the CEnclave;

[0111] Step 8: The process can use a special XM call xm_get_enclave_status to verify that it is running in an XpuEnclave. If the operating system has not called xm_enclave_create before, the XM will inform the process at this time;

[0112] Step 9: CEnclave creates a GPU channel. The creation operation is intercepted and recorded by XM. The address space of the channel is isolated as an XEnclave.

[0113] Step 10: CEnclave creates a GPU task by sending commands in the GPU channel, and then the created task is executed using the page table of that GPU channel.

[0114] Destruction process when XpuEnclave ends normally: When the process running in CEnclave ends, the entire XpuEnclave is considered to be destroyed, and operations such as cleaning data and reclaiming memory are required;

[0115] Step 1: CEnclave ends the main program logic and actively clears its sensitive data in XpuEnclave;

[0116] Step 2: When CEnclave exits, the OS uses XM to call xm_enclave_destroy, which directly enters XM.

[0117] Step 3: XM returns the CEnclave’s pages to the OS and marks the GPU memory as available for other applications.

[0118] Destruction process when XpuEnclave ends abnormally: For the unexpected termination of XpuEnclave, the CPU page can be easily cleared and returned by XM, but the GPU page may not be cleared by XpuEnclave. In this case, the GPU page of XpuEnclave is still retained by XM. The unexpected termination of XpuEnclave should not occur frequently, otherwise it should be regarded as the occurrence of malicious behavior. The destruction process at this time is as follows:

[0119] Step 1: XM clears all CEnclave contents and recycles the pages in CEnclave;

[0120] Step 2: XM notifies a trusted service running in CEnclave through a small shared buffer between XM and the service;

[0121] Step 3: The trusted service monitors whether the GPU is idle. When the GPU is idle, it allocates as much memory as possible and writes zeros to all GPU memory, clears resident data and sensitive data, and releases memory.

[0122] Step 4: The cleared memory is reclaimed by XM, allowing it to be allocated to other applications.

[0123] Specifically, the process of building the XpuTEE monitor includes:

[0124] The present invention can be implemented on an x86 platform, and XM is isolated in the same privilege level. The method for constructing XM is similar to the Nested Kernel work published at the ASPLOS conference in 2015. The WP bit in the register CR0 is used. When the bit is 0, the operating system can directly write to the memory page mapped as read-only in the page table, and when the bit is 1, the operating system cannot write to the memory page mapped as read-only in the page table. Only when running in XM, the CR0.WP bit will be set to 0, and XM will use a separate page table and adopt an independent stack. For pages that need to prevent the operating system from writing (such as page table pages), they will be mapped as read-only, so that the general operating system cannot write because WP is 1, and when switching to XM, because WP is 0, XM can directly write to the location. For the location that needs to prevent the operating system from reading and writing, the mapping in the page table of the general operating system and the user program will be removed, and only the mapping will be performed in the page table of XM, so that the memory page can be accessed after switching to XM.

[0125] XM has dedicated entry and exit functions. The operating system needs to call the entry or exit function to enter XM. The entry function needs to set CR0.WP to 0, switch to the XM page table, and switch to the XM stack; the exit function needs to set CR0.WP to 1, switch to the general kernel page table, and switch to the ordinary kernel stack. In order to ensure that only the entry and exit functions and XM can modify the CR0, CR3, CR4, and MSR_EFER registers, a binary scan method is required to ensure that the code of the untrusted ordinary kernel part does not contain such instructions.

[0126] XM provides special interfaces for other codes to call. Untrusted kernels can use these interfaces to notify XM, and XM will perform certain operations after security checks.

[0127] Unlike TEEs that only support the CPU, XpuTEE can support a unified abstraction to support the collaboration of trusted execution environments on different accelerators and multiple tasks on the same accelerator.

[0128] XpuTEE does not require specific hardware modifications, but only requires common mechanisms in existing CPUs and GPUs to be implemented, so it is compatible with existing CPU and GPU platforms.

[0129] XpuTEE's CPU and GPU interaction process does not require encryption operations and additional memory copies, and the internal communication overhead is smaller, so higher performance can be achieved.

[0130] In this embodiment, it can also be implemented based on the virtualization mechanism of the x86 platform. When virtualization is used for implementation, XM can be run in VMX-root mode, and other programs can be run in VMX-non-root mode. At the same time, EPT is used to protect memory pages to ensure that untrusted software programs do not access protected memory. Other software can use vmcall instructions to call the services provided by XM. Sensitive instructions and registers can be configured in VMCS so that sensitive instructions and registers will sink to XM in VMX-root mode when executed.

[0131] In this embodiment, the present invention can also be implemented based on the ARM platform, and the overall implementation concept is similar to that of the x86 platform. Because ARM can also support virtualization and secondary page tables, it can also be constructed using a similar method.

[0132] The present invention also provides a high-performance heterogeneous trusted execution environment implementation system, which can be implemented by executing the process steps of the high-performance heterogeneous trusted execution environment implementation method, that is, those skilled in the art can understand the high-performance heterogeneous trusted execution environment implementation method as a preferred implementation method of the high-performance heterogeneous trusted execution environment implementation system.

[0133] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0134] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for implementing a high-performance heterogeneous trusted execution environment, characterized in that: include: Build an XpuTEE monitor to manage all XpuEnclaves; The XpuEnclave is composed of at least one CEnclave and several XEnclaves; the CEnclave is an isolated environment on the CPU; the XEnclave is an isolated environment on the GPU; CEnclave and XEnclave belonging to the same XpuEnclave achieve fast communication through the XpuEnclave channel; The XpuTEE monitor can directly access the CEnclave running on the CPU, and the XpuTEE monitor accesses the XEnclave on the GPU through MMIO; The XpuTEE monitor includes: a memory permission management module, a PCIe protection module, a channel management module and an Enclave authentication module; The memory permission management module is responsible for controlling memory mapping and configuring memory permissions; The PCIe protection module captures all modifications to the PCIe MMIO configuration and protects the integrity of the PCIe MMIO settings; The channel management module constructs an XpuEnclave channel for fast communication between CEnclave and XEnclave; The Enclave authentication module generates information for remote authentication, enabling a user to remotely authenticate content running in the enclave; The memory permission management module includes: constructing enclave memory in the operating system and device memory to prevent untrusted CPU software or GPU tasks from accessing protected memory, and managing the memory mapping relationship on the CPU and GPU by exclusively controlling the CPU MMU, IOMMU and GPU MMU; For the CPU MMU, the extended page table is used to control the permissions of physical memory. For the GPU MMU, it manages the conversion of the GPU's virtual address to the physical address and enforces isolation of different XEnclaves on the GPU. It also manages IOMMU mapping to prevent untrusted devices from accessing the CEnclave.

2. The method for implementing a high-performance heterogeneous trusted execution environment according to claim 1, characterized in that: An independent execution environment is created for the XpuTEE monitor to prevent the operating system from modifying the XpuTEE monitor.

3. The method for implementing a high-performance heterogeneous trusted execution environment according to claim 1, characterized in that: The memory permission management module includes: CPU memory mapping, GPU memory mapping and IOMMU memory mapping, and constructs CEnclave and XEnclave through memory mapping; The CPU memory mapping includes: controlling the memory access rights of the CPU through exclusive MMU configuration; maintaining different page tables for XpuEnclave and untrusted software programs, in which kernel code pages, page table pages and IOMMU page table pages are mapped as write protection; in addition, PCIe configuration memory and preset GPU MMIO areas are mapped as read-only, and the GPU driver running in the kernel space needs to configure the GPU through the MMIO registers on the GPU and the GPU memory exposed to the CPU through PCIe. The memory permission management module maps these areas used to configure the GPU as read-only in the page table of the system software, so the GPU driver in the kernel space must explicitly use the interface provided by XpuTEE to call the memory permission management module to operate the GPU; each GPU channel has a user-accessible GPU MMIO area, the memory permission management module maps this area to the page table of the corresponding application, so the user space GPU uses the GPU channel to send instructions and create computing tasks on the GPU when running; the memory permission management module configures a preset page table for the XpuTEE monitor, which can access all physical addresses. In order to save the data and code of the XpuTEE monitor during runtime, the memory permission management will reserve a preset size of address space in the memory to protect the XpuTEE monitor itself. This physical memory area is not mapped in any other page table; The GPU memory mapping includes: the GPU as an independent graphics card supports a virtual memory mechanism, and uses a page table stored in the GPU memory to convert the GPU's virtual address into a GPU's physical address; the GPU page table page is located in the GPU MMIO memory area exposed by the PCIe BAR; in order to exclusively control the GPU mapping, the memory permission management module maps the GPU MMIO area in the page table of the kernel space driver as read-only by controlling the memory mapping, and at the same time, it maps the buffer of the privileged GPU channel in the page table of the driver as read-only, and in addition, the message queue for the GPU firmware RPC is also mapped as read-only; after adopting such a mapping mode, the kernel space GPU driver must call the memory permission management module in the XpuTEE monitor to modify the GPU page table, and the memory permission management module can now check all GPU memory mapping modifications and force the XEnclave's memory not to be mapped to untrusted tasks; The IOMMU mapping includes: the memory permission management module exclusively controls the IOMMU; when the IOMMU is enabled, any access of the device to the system memory or other device memory must go through the IOMMU, which converts IOVA to IOPA; the IOMMU uses the roottable and context table to map the device to the IOMMU page table, and the XpuTEE monitor maps the pages of the root table and contexttable, the IOMMU page table pages, and the configuration registers of the IOMMU as non-writable except for the page table of the XpuTEE monitor; therefore, the kernel must call the XpuTEE monitor to configure the IOMMU; The CEnclave construction includes: CEnclave is a CPU-side TEE used to protect applications coordinated with XEnclave; XpuTEE ensures that the memory of a CEnclave can only be accessed by software in the corresponding XpuEnclave; The XEnclave construction includes: XEnclave is created by CEnclave in the same XpuEnclave and cannot be tampered with by untrusted GPU tasks. Each GPU channel sets the page table it uses, so XpuTEE isolates the address space of the GPU channel as XEnclave, and data transmission between XEnclave and untrusted processes on the CPU is prohibited.

4. A high-performance heterogeneous trusted execution environment implementation system, characterized in that: include: Build an XpuTEE monitor to manage all XpuEnclaves; The XpuEnclave is composed of at least one CEnclave and several XEnclaves; the CEnclave is an isolated environment on the CPU; the XEnclave is an isolated environment on the GPU; CEnclave and XEnclave belonging to the same XpuEnclave achieve fast communication through the XpuEnclave channel; The XpuTEE monitor can directly access the CEnclave running on the CPU, and the XpuTEE monitor accesses the XEnclave on the GPU through MMIO; The XpuTEE monitor includes: a memory permission management module, a PCIe protection module, a channel management module and an Enclave authentication module; The memory permission management module is responsible for controlling memory mapping and configuring memory permissions; The PCIe protection module captures all modifications to the PCIe MMIO configuration and protects the integrity of the PCIe MMIO settings; The channel management module constructs an XpuEnclave channel for fast communication between CEnclave and XEnclave; The Enclave authentication module generates information for remote authentication, enabling a user to remotely authenticate content running in the enclave; The memory permission management module includes: constructing enclave memory in the operating system and device memory to prevent untrusted CPU software or GPU tasks from accessing protected memory, and managing the memory mapping relationship on the CPU and GPU by exclusively controlling the CPU MMU, IOMMU and GPU MMU; For the CPU MMU, the extended page table is used to control the permissions of physical memory. For the GPU MMU, it manages the conversion of the GPU's virtual address to the physical address and enforces isolation of different XEnclaves on the GPU. It also manages IOMMU mapping to prevent untrusted devices from accessing the CEnclave.

5. The high-performance heterogeneous trusted execution environment implementation system according to claim 4, characterized in that: An independent execution environment is created for the XpuTEE monitor to prevent the operating system from modifying the XpuTEE monitor.

6. The high-performance heterogeneous trusted execution environment implementation system according to claim 4, characterized in that: The memory permission management module includes: CPU memory mapping, GPU memory mapping and IOMMU memory mapping, and constructs CEnclave and XEnclave through memory mapping; The CPU memory mapping includes: controlling the memory access rights of the CPU through exclusive MMU configuration; maintaining different page tables for XpuEnclave and untrusted software programs, in which kernel code pages, page table pages and IOMMU page table pages are mapped as write protection; in addition, PCIe configuration memory and preset GPU MMIO areas are mapped as read-only, and the GPU driver running in the kernel space needs to configure the GPU through the MMIO registers on the GPU and the GPU memory exposed to the CPU through PCIe. The memory permission management module maps these areas used to configure the GPU as read-only in the page table of the system software, so the GPU driver in the kernel space must explicitly use the interface provided by XpuTEE to call the memory permission management module to operate the GPU; each GPU channel has a user-accessible GPU MMIO area, the memory permission management module maps this area to the page table of the corresponding application, so the user space GPU uses the GPU channel to send instructions and create computing tasks on the GPU when running; the memory permission management module configures a preset page table for the XpuTEE monitor, which can access all physical addresses. In order to save the data and code of the XpuTEE monitor during runtime, the memory permission management will reserve a preset size of address space in the memory to protect the XpuTEE monitor itself. This physical memory area is not mapped in any other page table; The GPU memory mapping includes: the GPU as an independent graphics card supports a virtual memory mechanism, and uses a page table stored in the GPU memory to convert the GPU's virtual address into a GPU's physical address; the GPU page table page is located in the GPU MMIO memory area exposed by the PCIe BAR; in order to exclusively control the GPU mapping, the memory permission management module maps the GPU MMIO area in the page table of the kernel space driver as read-only by controlling the memory mapping, and at the same time, it maps the buffer of the privileged GPU channel in the page table of the driver as read-only, and in addition, the message queue for the GPU firmware RPC is also mapped as read-only; after adopting such a mapping mode, the kernel space GPU driver must call the memory permission management module in the XpuTEE monitor to modify the GPU page table, and the memory permission management module can now check all GPU memory mapping modifications and force the XEnclave's memory not to be mapped to untrusted tasks; The IOMMU mapping includes: the memory permission management module exclusively controls the IOMMU; when the IOMMU is enabled, any access of the device to the system memory or other device memory must go through the IOMMU, which converts IOVA to IOPA; the IOMMU uses the roottable and context table to map the device to the IOMMU page table, and the XpuTEE monitor maps the pages of the root table and contexttable, the IOMMU page table pages, and the configuration registers of the IOMMU as non-writable except for the page table of the XpuTEE monitor; therefore, the kernel must call the XpuTEE monitor to configure the IOMMU; The CEnclave construction includes: CEnclave is a CPU-side TEE used to protect applications coordinated with XEnclave; XpuTEE ensures that the memory of a CEnclave can only be accessed by software in the corresponding XpuEnclave; The XEnclave construction includes: XEnclave is created by CEnclave in the same XpuEnclave and cannot be tampered with by untrusted GPU tasks. Each GPU channel sets the page table it uses, so XpuTEE isolates the address space of the GPU channel as XEnclave, and data transmission between XEnclave and untrusted processes on the CPU is prohibited.

Images