Peripherals with resource isolation

Abstract

A peripheral device package for use in a host computing device has a plurality of computing elements and a plurality of resources shared by the plurality of computing elements. A data structure is stored in a hidden memory of the peripheral device package. The data structure maintains metadata related to ownership of the resources of the peripheral device package by a plurality of user runtime processes of the host computing device that use the computing elements. At least one of the user runtime processes is a secure user runtime process. The peripheral device package has a command processor configured to use the data structure to implement isolation of resources used by the secure user runtime processes.

Description

Background Art

[0001] Peripherals such as Graphics Processing Units, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits, Cryptographic Accelerators, Video Processing Units, Regular Expression Accelerators, Data Compression Accelerators, Artificial Intelligence Accelerators, etc. are becoming more and more widely used to supplement the Central Processing Unit (CPU). Peripherals that include computer hardware that performs functions more efficiently than software running on a general-purpose CPU are often very useful units that are separate from the CPU and act as hardware accelerators. Typically, these types of peripherals improve the execution of specific types of algorithms by allowing greater concurrency.

[0002] There is an increasing need to work with sensitive code and / or sensitive data and maintain security and privacy. Often, large amounts of sensitive code and / or data will be processed using resource-intensive algorithms, and in this case, peripherals are an option to improve efficiency. However, in the case of using peripherals, additional challenges are introduced regarding the security and privacy of sensitive code and / or data, as both the host and the peripheral may be vulnerable to attack by malicious parties.

[0003] The embodiments described below are not limited to implementations that solve any or all disadvantages of known peripheral devices. Summary of the invention

[0004] A simplified summary of the present disclosure is presented below in order to provide the reader with a basic understanding. This summary is neither intended to identify key features or essential features of the claimed subject matter nor to be used to limit the scope of the claimed subject matter. Its sole purpose is to present a selection of concepts disclosed herein in a simplified form as a prelude to a more detailed description that is presented later.

[0005] In various examples, there is a peripheral device package for use in a host computing device. The peripheral device package has a plurality of computing elements and a plurality of resources shared by the plurality of computing elements. A data structure is stored in a hidden memory of the peripheral device package. The data structure maintains metadata related to ownership of resources of the peripheral device package by a plurality of user runtime processes of the host computing device using the computing elements. At least one of the user runtime processes is a secure user runtime process. The peripheral device package has a command processor configured to implement isolation of resources used by the secure user runtime processes using the data structure.

[0006] In various examples, there is a computer-implemented secure user runtime process having instructions for execution on a peripheral device enclosure of an untrusted host computing device. The secure user runtime process includes sending commands to a command processor of the peripheral device enclosure via the untrusted host, the commands using data structures in a hidden memory of the peripheral device enclosure to implement isolation of resources of the peripheral device enclosure used by the secure user runtime process.

[0007] Many of the attendant features will be better understood as they become better appreciated by reference to the following detailed description considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present description will be better understood from the following detailed description read in conjunction with the accompanying drawings, in which:

[0009] Figure 1 is a schematic diagram of a data center including an untrusted host computing device having a trusted peripheral device;

[0010] Figure 2 is a schematic diagram of an untrusted host computing device, a trusted peripheral device, and two trusted user applications and a user runtime process;

[0011] Figure 3 is a schematic cross-sectional view through a peripheral device package;

[0012] Figure 4 is a schematic diagram of a data structure known as a protected resource metadata table;

[0013] Figure 5 is a flowchart of the method of operation at the user runtime process;

[0014] Figure 6 is a flow chart of a method of operation at a command processor according to a create channel command;

[0015] Figure 7 is a flow chart of a method of operation at a command processor in accordance with a channel page directory entry command;

[0016] Figure 8 is a flow chart of a method of operation at a command processor according to a channel page table entry command;

[0017] Fig. 9 is a flow chart of an operation method at a command processor according to a channel destroy command;

[0018] Fig.10 is a flowchart of a method of operation at a command processor when it is detected that a user runtime process is not responding;

[0019] Fig.11 is a flow chart of a method of operation at a command processor according to a channel measurement command;

[0020] Fig.12 is a schematic diagram of a central processing unit of a host computing device, a graphics processing unit connected to the host computing device, and showing software and hardware components;

[0021] Fig.13 is a schematic diagram of the host memory and graphics processing unit memory space;

[0022] Fig.14 It is a schematic diagram of channel-level address space management;

[0023] Fig.15 It is a schematic diagram of command submission to the peripheral device package;

[0024] Fig.16 is a schematic diagram of the boot channel between the driver and the command processor;

[0025] Fig.17 is a timing diagram of the secure memory copy protocol;

[0026] Fig.18 is a graph of Cifar-10 performance; and

[0027] Fig.19 It is a graph of Black-Scholes execution time.

[0028] In the drawings, like reference numerals are used to indicate like parts. DETAILED DESCRIPTION

[0029] The detailed description provided below in conjunction with the accompanying drawings is intended as a description of this example, rather than being intended to represent the only form of constructing or utilizing this example. This description sets forth the functions of the example and the sequence of operations for constructing and operating the example. However, the same or equivalent functions and sequences may be implemented by different examples.

[0030] In various examples described herein, peripheral device encapsulation is used together with a host computing device. The host computing device is untrustworthy and may be malicious. The host computing device has multiple user runtime processes, which use the resources of the peripheral device encapsulation, such as computing elements, buses, memory pages or other resources, to more efficiently calculate tasks, such as image processing tasks, encryption tasks or other types of resource-intensive calculations. In order to effectively use the resources of the peripheral device encapsulation, the host device can control which user runtime processes use which resources. However, this may cause potential security vulnerabilities when the host device is untrustworthy, because the host device can reallocate or destroy the resources of the peripheral devices that support sensitive code and / or data.

[0031] One approach to providing security for secure user runtime processes using peripheral devices is to statically partition resources between the secure user runtime process and other user runtime processes on the host. However, this results in underutilization of resources on the peripheral devices because the partitioning is static and cannot be changed dynamically based on specific needs at a specific time. When using static partitioning, resources are not shared between user runtime processes.

[0032] In at least some embodiments described herein, isolation of resources used by a secure user runtime process is provided while allowing dynamic partitioning of resources. This approach has been found to have significant benefits in increasing resource utilization on peripheral devices while providing security for sensitive code and / or data of secure user runtime processes utilizing the peripheral devices.

[0033] A device driver in a host computing device is capable of performing dynamic allocation of resources of a peripheral device. The device driver makes resource allocation decisions across the entire resource pool. However, dynamic resource allocation requests made by the device driver to the peripheral device are audited by a command processor at the peripheral device. The command processor will implement the resource partitioning request only if the request meets certain requirements. The specific requirements are checked by checking a data structure stored in a hidden memory at the peripheral device. The hidden memory is a memory that is invisible to the host computing device. The data structure maintains ownership data related to which user runtime processes own which resources of the peripheral device at the current time. By checking the data structure and only implementing resource partitioning requests that pass the check, resource isolation can be implemented for user runtime processes to be securely executed on the peripheral device. The user runtime process receives a response from the peripheral device and can authenticate the response. In this way, even if the host is untrustworthy and may be malicious, a secure user runtime process at the host computing device can utilize resources on a trusted peripheral device.

[0034] Figure 11 is a schematic diagram of a data center 104 including a plurality of untrusted host computing devices 102. Individual ones of the untrusted host computing devices 102 are connected to a trusted peripheral device 100. The untrusted host computing devices 102 communicate with each other using a communication link (not shown for clarity). Figure 1 102 and 104). The data center 104 is assumed to be used to perform resource-intensive computing tasks 110 using sensitive code 106 and sensitive data 108. The data center is able to distribute resource-intensive computing tasks among various user runtime processes at an untrusted host 102. The trusted peripherals 100 are able to perform computations in an efficient manner using parallelization, and are able to perform computations in a secure manner even if the host 102 is untrusted and even if other unsecure processes are executing simultaneously in the data center. It is expected that in a few years, most of the computing cycles in the public cloud will be contributed by peripherals such as accelerators.

[0035] Figure 1 A data center scenario is shown. However, the untrusted host 102 and trusted peripheral device 100 may also be used in a standalone scenario or in other types of deployments.

[0036] Figure 2 2 is a schematic diagram of a host computing device 202 including a host memory 204 and a device driver 206. The host computing device 202 includes other components of a conventional computer, such as a central processing unit (CPU) and a CPU. Figure 2 208 and User 2 210 are software installed on the host computing device 202. Application 1 has an associated user runtime process 212, and Application 2 has an associated user runtime process 214. In practice, there are two or more applications, each with an associated user runtime process, and for clarity, this example is simplified to the case of two applications.

[0037] In some examples, the host computing device 202 has a trusted execution environment (TEE) that includes trusted hardware in the CPU of the host computing device 202. The TEE is used to protect sensitive code and data from system administrators and attackers who may exploit kernel vulnerabilities and take control of the entire software stack, including the operating system and any hypervisors. Existing TEEs are limited to the CPU and cannot be used in applications that offload computations to peripherals. This limitation results in an unfavorable tradeoff between security and performance.

[0038] The present technology attempts to add trusted execution environment support to the peripheral device 200 . Figure 2A trusted peripheral device 200 is shown connected to a host 202. In some cases, the peripheral device is referred to as a peripheral device package to indicate that the components of the peripheral device are supported on a silicon interposer, as described below with reference to Figure 3 Described in more detail. Peripherals include multiple resources, such as computing elements 216, memory pages, peripheral component interconnect (PCI) buses, and other resources. A resource is something that will be used by one or more of the user runtime processes.

[0039] Peripheral device 200 has memory shared by computing element 216. The memory includes hidden memory 228 and optional protected memory 220 and optional unprotected memory 222. Hidden memory is memory that is not visible to host 202. Protected memory 220 is memory that is visible to the host but not accessible to host 202. Unprotected memory 222 is visible and accessible to host 202.

[0040] The peripheral device optionally has a memory controller 218 for controlling the use of the shared memory 218 , and has a command processor 224 that receives commands from the host 202 and dispatches the commands onto the computing element 216 .

[0041] The peripheral device has a root of trust 226, which is a cryptographic key (called an endorsement key) integrated into the peripheral device's hardware during manufacturing, and a cryptographic engine for asymmetric key generation and signing.

[0042] like Figure 2 As shown, the device driver is connected to the command processor 224 of the peripheral device 200 via a communication link. A communication link also exists between the device driver 206 and the memory controller 218.

[0043] The device driver in the host manages the device resources of the peripheral device, such as the memory of the peripheral device, and has visibility to the peripheral device in previous solutions. As a result, it is a challenge to design a trusted execution environment in the peripheral device when the host operating system is not trusted. In addition, high-throughput peripherals achieve high performance by integrating a large number of computing elements and using high-bandwidth memory to meet their huge bandwidth requirements. Any major changes to the computing elements or memory subsystems will result in unacceptably large overhead. For example, due to sensitivity to address translation delays, implementing memory isolation by checking during address translation will cause the accelerator to be severely underutilized. Similarly, providing memory integrity and confidentiality via an encryption engine will significantly increase memory bandwidth utilization.

[0044] like Figure 2As shown, peripheral device 200 is used with host computing device 202. Peripheral device 200 includes multiple computing elements 216 and multiple resources shared by the multiple computing elements. For example, the resources are pages of shared memory, individual computing elements and peripheral components interconnect resources. Data structure ( Figure 2 The data structure (not shown) is stored in the hidden memory 228 of the peripheral device 200, and the data structure maintains metadata related to the ownership of the resources of the peripheral device 200 by multiple user runtime processes 212, 214 of the host computing device 202 using the computing element 216, at least one of the user runtime processes 212, 214 is a secure user runtime process. The command processor 224 is configured to use the data structure to implement isolation of resources used by the secure user runtime process. In this way, the resources of the peripheral device can be dynamically allocated by sending commands from the user runtime process to the command processor. However, the command processor is able to implement isolation of appropriate resources when necessary to provide security by checking using the data structure.

[0045] In various examples, the TEE on the peripheral package takes the form of a secure context, including a collection of peripheral resources (memory, command queues, registers, etc.) and all other peripheral contexts that are cryptographically bound to a public / private key pair and isolated from untrusted software on the host (including drivers). Once a secure context is created, only the user application / runtime that possesses the corresponding private key can access its resources. As long as the key is protected from adversary attacks (e.g., hosted in the CPU TEE), the adversary cannot access the address space of the context. Two additional primitives are also supported: measurements for generating remotely verifiable profiles of the state of the context and the platform, and secure memory allocation and deallocation for enabling device drivers to dynamically allocate and release device memory without compromising security.

[0046] Strong security is achieved by redefining the interface between the device driver in the host and the hardware of the peripheral device. The driver is blocked from directly accessing security-sensitive resources such as page directories, page tables, and other pages containing sensitive code and data. Instead, the driver routes resource allocation requests through the peripheral device's command processor. The command processor tracks ownership of resources using data structures in its hidden memory and ensures that an adversary cannot access resources owned by a victim context. The command processor also ensures that resources are properly initialized when allocated to a security context and cleaned up when destroyed to prevent attacks that exploit improper initialization.

[0047] The peripheral has low hardware complexity, low performance overhead, and cryptographic agility. There are no changes to the peripheral compute element or peripheral memory subsystem, and most changes are limited to peripheral components, such as the peripheral command processor. There are no restrictions on the instruction set available in the TEE of the peripheral. The CUDA (trademark) runtime can use the primitives described in this article to build secure versions of higher-level APIs, such as memory copy, kernel launch, and CUDA streams, which in turn can be used to build applications with end-to-end confidentiality and integrity.

[0048] Secure user runtime processes (such as Figure 2 212) has instructions for execution on the peripheral device 200 of the untrusted host computing device 202, the secure user runtime process 212 includes:

[0049] Commands are sent to the command processor 224 of the peripheral device 200 via the untrusted host 202, and these commands use data structures in the hidden memory 228 of the peripheral device 200 to implement isolation of the resources of the peripheral device 200 used by the secure user runtime process 212. In this way, the user runtime process can operate on the untrusted host and still isolate the resources it uses on the peripheral device.

[0050] Figure 3 is a diagram showing the silicon interposer Figure 2 Schematic diagram of a peripheral device, the silicon interposer supports multiple computing elements 316, supports a memory controller 318 and supports a shared memory 324. In this case, the peripheral device is a package because it is formed on silicon. The root of trust 226 is Figure 3 , but in some examples is burned into the device's electronic fuses during the manufacturing process, or is integrated into the hardware of the device package in any permanent manner. The root of trust 226, together with a certificate authority that communicates with the host computing device over a communications network, enables remote attestation to establish trust between a security context and a remote challenger. The context includes multiple channels on the peripheral computing device, where a channel is a plurality of resources on the peripheral computing device that are allocated to a specific user runtime process. More details about channels will be given later in this document. The security context includes one or more secure channels used by a secure user runtime process on the peripheral device package. The secure channel includes one or more resources, such as page tables, computing elements used by the secure user runtime process and isolated from other resources on the peripheral device package.

[0051] When the peripheral device package is initialized, it generates a new attestation key pair (including a public key and a private key) and stores the private key securely in a secure random access memory (SRAM) within the command processor. The peripheral device package signs the public portion of the attestation key with the endorsement key and makes the signed public portion of the attestation key available to the device driver 206 in the host 202. The host 202 sends the signed public portion of the attestation key to the certificate authority. The certificate authority verifies the signature using a private endorsement key repository provided by the peripheral device package manufacturer and generates a signed attestation key certificate. The certificate is sent to the device driver 206 and stored. The certificate is used by the device driver 206 during security context creation to prove to the challenger that the peripheral device package holds and protects the private portion of the attestation key.

[0052] In the hidden memory of the peripheral device is Figure 4 An example of a data structure 400 is schematically shown in FIG. The data structure 400 is shown as a table, but may be in other formats, such as an array or a list, which has the same functionality as a table. In some examples, the data structure is referred to as a peripheral resource metadata table (PRM). The data structure stores ownership data for resources of a peripheral device. Figure 4 In the example of , there is a row 408 for each resource of the peripheral device. Column 402 stores a number identifying the user process that currently owns the resource. Ownership in this context means that the user process has been assigned the ability to use the resource. Column 404 stores a binary number indicating whether the resource is owned. Column 406 stores a binary number indicating whether the resource is locked. A locked resource is a resource that can only be unmapped with the user runtime process authorization. An unlocked resource has the potential to be used by a different user runtime process and can be unmapped without the user runtime process authorization. The other columns of the data structure store information about any other characteristics of the resource tracked by the peripheral device. Figure 4 The PRM shown in is only an example, and other forms of PRM are possible. Another example of a PRM is given later in this document.

[0053] Figure 5Flowchart of a method of operation in the case of an untrusted host and a trusted peripheral device. A user runtime process requests 500 a security context. The device driver receives the request and makes 502 a resource allocation decision for allocating device memory (such as page directories and tables) required by context management. The device driver then sends 504 a secure channel creation command to the command processor of the peripheral device. The secure channel creation command is sent to the command processor over a non-secure channel, and the command includes a public key preset by the user runtime process. More details about the secure channel creation protocol and how the public key is used will be explained later in this document. The command processor continues to check the PRM to see if the resource allocation decision of the device driver is appropriate for the secure channel. If the PRM check fails, the process fails 512. If the PRM check succeeds, a secure channel is established 514.

[0054] Resources on the peripheral device are allocated after context creation, either explicitly for data or implicitly for code. The user runtime process follows the protocol described later in this document to securely transfer 516 sensitive code and data to the peripheral device. The sensitive code and sensitive data are executed in a secure channel on the peripheral device.

[0055] The execution results on the secure channel are returned to the user runtime process 518 in a secure manner.

[0056] Thus, the command processor receives resource management commands from the user runtime process via a driver of the peripheral device package, the driver being in the host computing device. The command processor receives the command after the specification of the resources in the command by the driver (that is, the driver makes a resource allocation decision across the entire resource pool and specifies the result of the resource allocation in the command). The command processor is configured to return a response to the received command for authentication by the user runtime process (such as by sending a signed digest).

[0057] Note that the non-secure channel is used to send commands for channel and resource management to the command processor, and for returning responses from the command processor to the user runtime process. Some but not all commands sent over the non-secure channel contain a signed digest (keyed MAC) from the user runtime process as part of the command. Some but not all responses returned by the command processor over the non-secure channel contain a signed digest (keyed MAC) of the command processor as part of the response. The signed digest enables inspection of communications over the non-secure channel.

[0058] The command processor on the peripheral package is extended with commands for the creation, management, and destruction of secure channels on the peripheral package, and these commands are part of the command processor's application programming interface (API). The commands for the creation, management, and destruction of secure channels use PRM data structures in hidden memory on the peripheral device, as now referenced Figures 6 to 11 When a user runtime process wants to securely offload and execute sensitive code and data on a peripheral device package, it can create a suitable secure channel and then follow the protocol for secure memory copy and kernel launch, which is described later in this document.

[0059] In at least some examples of the technology, the API of the command processor is extended to include the following commands. These commands will be described in detail later in this document.

[0060]

[0061]

[0062] A secure channel is created using the command CH_CREATE (described in detail below) and specifying the public key UKpub and specifying a channel identifier (∈[1...N]). N is the number of channels supported by the device. Upon receiving a request, the command processor generates a symmetric key called a channel encryption key (CEK) that is intended to be used to encrypt and sign commands issued to the channel. The CEK and UKpub for each channel are stored in an area of ​​the device memory accessible only to the command processor. CH_CREATE can be used to create multiple channels associated with the same security context by passing the same UKpub, in which case all such channels will use the same CEK.

[0063] After generating the CEK, the command processor establishes a session by securely transferring the CEK to the user runtime process. The command processor encrypts the CEK using UKpub and generates a response including a reference that contains the encrypted CEK and a hash of UKpub. The reference also contains the channel identifier and all security-critical platform-specific attributes, such as the firmware version, and is signed by the AK. The device driver passes a response including the reference and the AK certificate (obtained during initialization) to the user runtime process. The user runtime process authenticates the response by (i) verifying the AK certificate, (ii) verifying the reference using the public AK embedded in the certificate, and (iii) checking that the public key in the reference matches UKpub. The user runtime process can then decrypt the CEK and use it to encrypt and sign all commands sent to the GPU.

[0064] For its part, the command processor uses the CEK to authenticate and decrypt all commands received over the secure channel. This ensures that only users with the CEK can perform tasks that access the context's address space. The protocol also uses a per-channel freshness value (nonce) (derived in monotonically increasing order from the initially agreed-upon seed) to protect commands from message drop, replay, and reordering attacks. This ensures that the commands generated by the GPU runtime are delivered to the command processor in the correct order without tampering.

[0065] Figure 6 600 is a flow chart of a method CH_CREATE for creating a secure channel at a command processor. The command takes as a parameter the page directory address (pgd_address) for the newly created secure channel child. The command processor receives 600 the CH_CREATE command and checks 602 (by looking at a data structure in hidden memory) whether the page directory is allocated on a page in a protected area of ​​a shared memory on a peripheral device package and checks that the page is in an "idle" state (the "idle" state indicates that the page has not been used). The previous constraint ensures that after the channel is created, the driver does not bypass the command processor and does not read and write the page directory directly on the memory mapped input output register (MMIO).

[0066] If the check 602 succeeds, the page transitions 606 to the "mapped" state (as opposed to "free"), and the owner_id attribute of the page containing the page directory is updated to the identifier of the channel being created. If a secure channel is being created (using a public key), the page will transition to the "locked" state. The command processor then updates 608 the address of the page directory in the channel descriptor, and clears 610 the contents of the page storing the page directory to prevent an attacker from injecting stale translations. If the channel descriptor or page directory is already locked or mapped to an existing channel, then CH_CREATE fails 604.

[0067] Figure 7 700 . The command processor receives a channel page directory entry command CH_PDE. The command will unmap the existing page table (if any) and map a new page table at the specified index (pg_index) in the page directory of the channel child. The entry in the page directory can reference a small or large page table.

[0068] Before unmapping, the command checks 702 whether the page table is in the "unlocked" state, or the page table is in the "locked" state and the pgt_entrycnt attribute is zero. In either case, the command processor decrements 706 a counter denoted as refcnt. If ref cnt decreases to zero 708, the page transitions 710 to the "free" state. The entry is then cleared 712 in the page directory.

[0069] Before mapping a new page table, the command checks 714 (by checking a data structure in hidden memory) whether the page table is allocated on a page in the protected area and is in the "free" state. If the check is successful, the page transitions 720 to the "mapped" state. In addition, if the channel is secure, the page is transitioned to the "locked" state by setting the lock bit. However, if the pages have already been "mapped" (not selected at check 714), the command checks 716 whether the channel that owns the page (current owner_id) and the channel to which the page table is mapped belong to the same context by comparing the corresponding public key hashes. If the hashes match, the reference count of the page is incremented 722. This allows physical page tables and therefore physical pages to be shared between channels (as long as they share the same context); this is necessary to support features such as CUDA (trademark) streams. If any of the checks are successful, the command creates 724 a new entry in the page directory and clears the contents of the page storing the page table. If the page table is mapped to a channel associated with a different context, the command fails 718.

[0070] Figure 8 is a flow chart of a method of operation at a command processor when the command processor receives a 800 Channel Page Table Entry CH_PTE command. CH_PTE removes any existing mappings from the page table and creates a new mapping (specified by an array called Page Table Entries (PTEs)) starting at a given virtual address pgt_index and spanning a specified range (size). Each PTE entry can reference a small or large page (specified by a bit in the PTE).

[0071] Before removing an existing page table entry, the command checks 802 (by looking at a data structure in hidden memory) whether the physical page is in the "unlocked" state. In order to remove the mapping for a "locked" page, the command requires an explicit authorization 804 on the command in the form of a message authentication code (MAC), which is generated using the channel encryption key (CEK) of the secure session between the user runtime process and the command processor. This prevents a malicious driver from unmapping physical pages assigned to a secure channel and then remapping them to another channel. If the check succeeds, the page transitions 808 to the "free" state and the entry is removed 810. If the explicit authorization fails 804, the command fails 806.

[0072] Instead, before creating a new mapping, the command checks 812 whether the page is "free". In addition, if the request is a secure allocation request 816, the command checks 818 whether the page is located in a protected area (for sensitive code and data). Pages allocated on behalf of a secure allocation will be locked and included as a summary in the signature summary. Pages allocated on behalf of an unsecure allocation do not need to be locked or included as a summary in the signature summary. If the check is successful, the page transitions 822 to the "mapped" state, and if the page is being mapped 824 to a secure channel, the page transitions 826 to "locked". Note that CH_PTE also allows pages in unprotected areas to be mapped to secure channels; these pages can be accessed through MMIO and are used for objects required for synchronization by the storage driver, such as fence buffers. Upon success, the command increments 828 the pgt_entrycnt of the page. If any page being mapped has already been mapped, the command fails 814.

[0073] A response is generated 830 only for secure allocations. The mapping is compactly encoded using the starting virtual address and the number of pages allocated in the protected area. The response also contains a channel identifier and a key MAC using the channel CEK. The response is sent 830 to the user runtime process via the device driver. In the case where the object spans multiple page tables, the driver collects responses from multiple CH_PTE commands. The driver returns the response to the user runtime process. The user runtime process that has access to the CEK can detect whether the response has been tampered with or replayed, and verify that the entire object has been allocated in the protected area and the correct channel.

[0074] When the command processor receives the CH_DESTROY command, use Fig. 9This command frees the memory allocated to the channel by walking 908 the page directory, finding the physical pages owned by the channel, and clearing 910 their information from the data structures in the hidden memory. It then unmaps 912 the physical pages of the page directory and channel descriptors, decrements 914 the ref cnt for the page of the page table, and transitions 918 the page whose re ref is reduced to 0 to "free" (see Fig. 9 Check 916 in).

[0075] For a secure channel 902, the command requires explicit authorization 904 in the form of a MAC of the CEK on the command, otherwise the command fails 906. However, in some cases, the user runtime no longer responds (e.g. Fig.10 The kernel 1000 detects this), and the channel resources must be cleared. For such a situation ( Fig. 9 ), the driver will issue CH_DESTROY 1002 without authorization. In this case, the command processor browses the channel's page directory and page table to find the physical pages mapped specifically to the channel's address space and clears their contents to prevent information leakage.

[0076] The command processor is also extended with the command CH_MEASURE to generate a verifiable artifact that summarizes the contents of the secure channel. The artifact can be used to prove to the challenger that the channel exists on the peripheral device hardware in a state that can guarantee channel isolation. When the command processor receives 1100 the CH_MEASURE command, it receives the range of virtual pages that should be included in the measurement as a parameter of the command. The command processor generates 1102 a measurement that contains a summary of the contents of the pages in the range, a starting virtual address, and a MAC using a CEK. The summary is sent 1104 to the challenger, who can check whether the summary is as the challenger expects. In some examples, the user runtime process uses the measurement to verify whether the kernel has been copied to the address space of the secure channel without tampering.

[0077] In various examples now described in detail, the peripheral device package is a graphics processing unit GPU. However, these examples are applicable to other types of peripheral devices. A non-exclusive list of examples of suitable peripheral devices is: graphics processing unit, field programmable gate array (FPGA), application specific integrated circuit, cryptographic accelerator, video processing unit, regular expression accelerator, data compression accelerator, artificial intelligence accelerator.

[0078] To aid understanding of the technology, the architecture of an example graphics processing unit, its software stack, and the associated CUDA programming model are now described, focusing on the parts of the architecture that are important in terms of security.

[0079] Fig.12 A CPU 1200 of a host computing device is shown. The host computing device has a software stack that includes a hypervisor 1202, a host operating system 1204, a peripheral device driver (in this example, a GPU driver 1206), a user runtime process 1208, and a user application 1210. The host computing device is connected to a GPU 1214 via a peripheral component interconnect bus 1212. The GPU has a software stack that includes firmware 1216 and code and data 1218. The hardware of the GPU includes an internal bus 1220, a peripheral component interconnect (PCI) control engine 1232, a command processor 1230, a copy engine 1222, a memory controller 1226, a memory 1224, and a compute engine 1228. The compute engine 1228 includes multiple graph processing cluster (GPC) secondary caches. Each GPC includes multiple streaming multiprocessors (SMs).

[0080] User space applications use an API provided by a user space GPU runtime (e.g., CUDA runtime) to program the GPU execution unit with a piece of code called a kernel and transfer data between the host and device memory. The GPU runtime converts each API call into a set of GPU commands for configuring the device and controlling kernel launches and data transfers. The device driver consists of a user space component that acts as a wrapper for the kernel space component. The kernel space GPU driver is responsible for submitting commands to the GPU via the PCI bus and for managing device memory.

[0081] The GPU interfaces with the host CPU 1200 via a PCI control engine 1232, which is connected to the rest of the GPU components via an internal bus 1220. Notable components are the command processor 1230, the compute 1228 and copy engines 1222 (DMA), and the memory system, including a memory controller 1226 and memory chips 1224. The GPU engine may be controlled by a microcontroller executing firmware 1216, which is loaded at device boot time.

[0082] The PCI control engine 1232 consists of the following: (a) a PCI controller that receives incoming and outgoing PCI transactions, and (b) a main control engine that exposes a set of memory mapped IO (MMIO) registers accessed by the host CPU to enable and disable the GPU engine. The command processor 1230 (also known as the channel engine) receives commands submitted by the device driver as a set of command queues (called channels) and forwards them to the corresponding engine when they are idle. The channel is configured through a set of memory locations (called the channel control area), which is mapped on the MMIO and serviced by the command processor 1230.

[0083] The compute engine 1228 includes a set of graphics processing clusters (GPCs) and a shared L2 cache. Each GPC includes multiple streaming multiprocessors (SMs) for running GPU kernels. Each SM includes multiple cores and a private memory hierarchy, including a read-only cache, an L1 cache, and application management memory. The GPU kernel specifies the number of threads to be created (organized in blocks and grids). However, the scheduling unit on each SM is a warp. Threads belonging to the same warp share cache and application management memory.

[0084] Execution on the GPU is context-based. A context is a collection of resources and state (memory, data, etc.) required to execute a kernel (a piece of code). Resources are allocated to a context to run a computational task and are released when the context is destroyed. Each context has its own address space. The GPU uses channels to isolate the address space of a context from other contexts. Channels are the only way to submit commands to the GPU. Therefore, each GPU context is assigned at least one GPU channel.

[0085] Fig.13 The GPU memory and host memory are shown. The host memory 1302 includes memory for user space 1304 and kernel space memory 1306. The user space memory 1304 includes data 1310, GPU kernel objects 1308. The kernel space memory 1306 includes direct memory access (DMA) buffers 1312, command buffers 1314, and ring buffers 1316.

[0086] The GPU memory includes data 1310 , GPU kernel objects 1308 , channel descriptors 1328 , page directories 1326 , page tables 1324 , channel engine controllers 1322 , compression engine controllers 1320 , and miscellaneous buffers 1318 .

[0087] Fig.14 Channel level address space management is shown. Channel control area 1400 has a pointer to a channel descriptor 1402. The channel descriptor has a pointer to a page directory 1404. The page directory points to one or more page tables 1406. The page table entries point to memory locations 1408.

[0088] To create a channel, the device driver allocates a channel descriptor 1402 and a two-level page table 1406 in the device memory. The two-level page table includes a page directory (PGD) 1404 and multiple leaf page tables (PGT) 1406. The driver writes the channel descriptor 1402 address to the channel control area and writes the page directory address into the channel descriptor 1402. The page directory 1404 includes an entry pointing to the leaf page table 1406, and the leaf page table 1406 contains a virtual to physical mapping. The page table supports small pages (4K) and large pages (128K), which is indicated by a bit in the page table entry. The device driver updates all these data structures through the PCI bus.

[0089] Once a channel is created, the device driver allocates device memory for: (a) storing the channel's internal state and compute engines on the channel switch; (b) fence buffers for synchronization between the host CPU and the GPU, and (c) interrupt buffers for transmitting interrupts generated by the GPU engine to the host.

[0090] The command processor 1230 is responsible for taking commands submitted by the software stack and relaying them to the appropriate GPU engine. Fig.15 The data structures created for command submission are shown, including a ring buffer 1506 and a memory-mapped user command buffer 1508. The driver allocates the command buffer 1508 and the ring buffer 1506 in the kernel space. The command buffer 1508 is memory-mapped to the user runtime memory space. The runtime (command pusher 1500) pushes command groups to the command buffer 1508, updates the channel's ring buffer 1506 with the size and offset of each group, and then updates the PUT pointer on the MMIO. When the PUT register 1504 is updated, the command processor 1230 retrieves the command group 1510, 1512 from the buffer and updates the GET register 1502 to notify the runtime that the command has been received.

[0091] To help understand the technique, the main stages of assigning GPU cores to the GPU are now described.

[0092] An application that wishes to use the GPU first creates a context, such as a CUDA (trademark) context. During context creation, the runtime allocates DMA buffers for data transfers between host memory and device memory, such as Fig.13As shown. Subsequently, the application loads one or more CUDA (trademark) modules into the context. For each kernel defined in the module, the runtime creates a corresponding kernel object on the GPU by allocating device memory for: (a) the kernel's code, (b) constant memory used by the kernel, and (c) local memory used by each thread associated with the kernel. The runtime then copies the code and constant memory to device memory via DMA.

[0093] The application then uses the memory allocation API to allocate device memory for storing the kernel's input and output. The memory allocation is serviced by the driver, which will update the page directory and page table accordingly.

[0094] When the application issues a host-to-device copy, the runtime pushes a command group to the context's channel to pass the virtual addresses of the source and destination to the copy engine. Once the copy engine is configured, it converts the source and destination addresses to physical addresses and initiates a DMA transfer.

[0095] When the application executes a kernel, the runtime pushes a command group that includes the kernel's context, the base address of the code segment, the entry program counter, the grid configuration, and the kernel's environment, including the stack and parameter values.

[0096] The GPU can be used to execute multiple kernels from multiple host processes using techniques such as preemptive multitasking, spatial multitasking, simultaneous execution, multiprocess services, or virtualization. In this case, it is the host (driver)'s responsibility to isolate the kernels using channel abstractions and virtual memory. Even in devices that partition resources in hardware, system software is still responsible for assigning virtual devices to virtual machines.

[0097] Consider a powerful adversary who controls the entire system software (including the operating system, device drivers, and hypervisor) and has physical access to all server hardware including the GPU. Obviously, such an adversary can read and tamper with the code or data of any victim process. The attacker can also access or tamper with user data in DMA buffers or commands submitted to the GPU by the victim application. This allows the adversary to control the address of the kernel being executed and the parameters passed to the kernel through properties. The adversary can also access device memory directly through MMIO or map the user's GPU context memory space to a channel controlled by the adversary. In a multitasking GPU, a malicious kernel can be dispatched to the GPU and thus access memory belonging to the victim's context. These attacks are possible even in virtualized environments (e.g., even if the device supports SR-IOV) because the mapping between VMs and virtual devices is also controlled by the hypervisor.

[0098] An adversary with physical access to the server can launch snooping attacks on the host memory bus and the peripheral component interconnect bus. In the example described in this article, the TEE is used to host user applications and the GPU runtime. Unlike the untrusted host memory, the on-package GPU memory is trusted because the GPU core is attached to the memory using a silicon interposer or pass-through silicon technology, which makes it difficult for an attacker to snoop.

[0099] Even under this threat model, the examples described in this article achieve confidentiality and integrity for applications using GPUs. The examples enable secure transmission of code and data to and from the GPU by a trusted user application (e.g., running in a TEE such as a CPU isolate or local machine), and the adversary cannot observe or tamper with the secure transmission. Computations on the GPU will continue without interference from the adversary.

[0100] In existing GPUs, the responsibility for managing device resources (such as memory) lies with the device driver. For example, when allocating memory for an application object, the driver determines the virtual address of the allocated object, then determines the physical page to map to the virtual page, and finally updates the virtual-physical mapping in the channel's page table (via MMIO). This mechanism creates a large attack vector. A compromised driver can easily violate channel-level isolation, for example, by mapping the victim's page into the address space of a malicious channel.

[0101] One way to prevent such attacks and achieve isolation is to statically partition resources between channels. However, this will result in underutilization of resources and does not allow for low-cost resource sharing between channels, which is used to implement features such as streaming. Instead of static partitioning, the examples described herein provide isolation of security contexts by imposing strict ownership guidelines on resources in hardware while allowing the driver to dynamically partition resources. Specifically, hardware changes are made to the GPU to enforce the following invariants, which imply isolation.

[0102] Invariant 4.1 A physical page mapped to a secure channel associated with a security context (ie, a public key) cannot be mapped to a channel associated with a different context.

[0103] Invariant 4.2 Physical pages mapped to a secure channel containing sensitive information may not be unmapped without authorization.

[0104] Invariant 4.3 All physical pages mapped into secure channels containing sensitive information cannot be accessed through MMIO by untrusted software on the host CPU.

[0105] Objects (code and data) allocated in a security context and the address space management structures of the security context (ie, channel descriptors, page directories, and page tables) are considered sensitive. The hardware extensions used to enforce these invariants are now described.

[0106] The first extension is to divide the device memory into three areas: unprotected, protected, and hidden, each with different access permissions.

[0107] The unprotected region is an area in memory that is visible and accessible from the host (via the peripheral component interconnect base address register). The driver can use this region to allocate channels that do not require security, and non-sensitive objects accessed through MMIO (for example, synchronization and interrupt buffers). This region can also be accessed from the compute and copy engines.

[0108] The protected area is visible to the host, but not accessible from the host. In other words, the driver can allocate objects in this area (by creating page mappings), but cannot access this area directly through MMIO. Therefore, this area can only be accessed by the GPU engine.

[0109] The hidden region is not visible or accessible to the host CPU or graphics engine on the GPU. Pages in this region are not accessible through the peripheral component interconnect and are not mapped into the virtual address space of any channel. This region is reserved for use by the command processor to maintain metadata such as ownership status of protected memory pages and per-channel encryption keys.

[0110] Memory regions are implemented using range checks on every MMIO access and on commands that update the address translation in the Peripheral Component Interconnect Control Engine and Command Processor respectively. The size of these regions is configured by the untrusted host software during initialization. The size does not affect security, only usability, for example, an administrator can prevent the creation of a security context by allocating a very small protected region.

[0111] To prevent malicious drivers from violating Invariants 4.1 and 4.2, the tasks of allocating and deallocating virtual and physical memory are decoupled from the tasks of managing device memory-resident address translation data structures (i.e., page directories and page tables), which are then delegated to the GPU's command processor. Drivers are able to decide where in virtual and physical memory objects will reside. However, drivers are required to use references Fig.10Requests to update the page directory and page table are routed through a command processor using commands described above. In the command processor, the implementation of the API enforces invariants by tracking ownership of physical pages in protected regions in a data structure called "protected resource metadata (PRM)". PRM is now described using an example where the resource is a memory page. Note that this example can be extended to situations where other types of resources are used.

[0112] The PRM is a data structure located in hidden memory, making it invisible to the host. In one example, it is indexed using the physical address of the memory page. Pages are tracked at a granularity of small pages (i.e., 4KB). The PRM maintains the following attributes for each physical page.

[0113] The property owner_id is the channel that owns the page, represented by its identifier.

[0114] The attribute status∈{FREE, MAPPED} indicates whether the page is free or mapped to a channel. FREE is the default value.

[0115] The attribute ref count keeps track of the number of channels that the physical page has been mapped to.

[0116] The attribute lock∈{UNLOCKED, LOCKED} indicates whether the page requires explicit authorization to be unmapped.

[0117] The attribute pgd_index is an index into the page directory that points to the page table containing the mapping for the current page. Using this attribute, the command processor can reconstruct the virtual address of the physical page. In this sense, the PRM acts as a reverse page table for protected regions.

[0118] The attribute pgt_entrycnt is a 2-byte value that tracks the number of page table entries allocated in the page table. Using this attribute, the command processor can know if the locked page table is empty and therefore possibly unmapped.

[0119] Attribute "pgt_type". Attribute pgt_type tracks whether the page belongs to a page table of large pages or small pages. When a page directory entry is updated with a page table address, this attribute will be set for the page of the page table. When a page table entry is updated with an allocated object, this attribute will be read in order to know how many secure page entries need to be updated. For example, when updating a large page table, the update will cover 128KB of address space and therefore 32 PMM entries will be updated.

[0120] In one example, assuming that each PRM entry requires 72 bits, the total size of the PRM for a GPU with 3GB of physical memory is 6MB, which is approximately 0.2% of the total memory.

[0121] The introduction of a command-based API for address space management raises the following question: How can the driver send commands for managing the address space of a secure channel without access to the channel-specific CEK? This problem can be solved by enabling the driver to use a separate channel (which we call the boot channel) (see Fig.16 1602) to route address space management commands of all other channels. The driver creates and configures the boot channel on MMIO 1601 and allocates its own address space management structure in the unprotected area. To ensure that the drive does not use the boot channel to violate the isolation of other (secure) channels, the command processor prohibits the boot channel 1602 from issuing commands to the replication and compute engines because such commands can be used to access sensitive states. To ensure that the drive does not use the boot channel to violate the isolation of other (secure) channels, the command processor only allows the execution of resource management commands (such as address space management commands or other resource management commands) to ensure that other commands executed in the command processor cannot access sensitive states.

[0122] refer to Fig.16 , the boot channel 1602 is an instance of a class of channels created by the driver 1600 for configuration purposes. These channels are called configuration channels. The command processor identifies the configuration channel as follows. The logic of the command processor that intercepts the MMIO 1601 write to the channel descriptor 1604 attribute in the channel control area is expanded. The command processor 224 can access the page table 1608, the page directory 1606, and the channel descriptor 1604. If the address to which the attribute is being written is in an unprotected area, the corresponding channel is marked as a configuration channel in an array stored in the SRAM of the command processor 224. The command processor 224 uses the array to determine whether it should forward the command from the channel to a replica or a computing engine. For the configuration channel, the command processor is allowed to execute only resource management commands (such as address space management commands) to ensure that other commands available in the command processor cannot access sensitive states. The array is also used during CH_CREATE to ensure that the channel descriptor of the secure channel is allocated in the protected area.

[0123] When a command fails, the command processor writes an error to an SRAM register accessible by the device driver via MMIO. This allows the device driver to take necessary actions to ensure the consistency of the address space of the channel between the command processor and the device driver.

[0124] In addition to isolating the secure channel, hardware support is provided to enforce Invariant 4.3, namely, that all sensitive code and data are allocated in protected areas. The CH_PTE command generates a response that contains a new mapping added to the PTE. The mapping is compactly encoded using the starting virtual address and the number of pages allocated in the protected area. The response also contains a channel identifier and a key MAC using the channel CEK. In the case where an object spans multiple page tables, the driver collects responses from multiple CH_PTE commands. The driver returns the response to the user runtime process. The user runtime process that has access to the CEK is able to detect if the response has been tampered with or replayed, and verify that the entire object has been allocated in the protected area and the correct channel.

[0125] The command processor enforces Invariant 4.2 by including as parameters for each CH_PTE command the starting virtual address and size of the object signed by the CEK. The command processor authenticates the parameters and then checks whether the unmapped page range is within the range specified by the parameters. If the check succeeds, the command processor decrements the ref count and transitions the page to the "free" state if the ref count is reduced to 0. Note that the hardware does not clear the contents of any page on allocation and deallocation; the user runtime process will clear the page.

[0126] A set of primitives is now described that use a secure context to transfer sensitive kernel and data to peripheral devices and boot kernels while guaranteeing confidentiality and integrity.

[0127] The peripheral runtime supports a new primitive (cudaSecureMemcpy) for securely copying code and data from trusted host memory (i.e., enclave memory) to device memory and vice versa. Fig.17 The protocol used by the primitives is shown.

[0128] After the secure context is created, the user runtime process will initiate DMA to copy the kernel performing authenticated decryption and its constant memory 1708 to context memory at a peripheral such as the GPU 1706. The kernel is copied in plain text (because it is well known). The user runtime process sends a measurement command 1710 to the command processor to ensure that the kernel is copied correctly by comparing 1714 the digest included in the measurement 1712 with the digest of the kernel calculated in the host TEE memory.

[0129] Upon receiving a request for a secure memory copy, the user process runtime encrypts 1716 the data to be copied using the new symmetric key in the CPU TEE and copies the encrypted data to the untrusted memory in the host.

[0130] The user runtime process uses secure memory allocation to allocate device memory in a protected area for the encrypted data and initiates DMA to transfer the encrypted data to that memory 1718. The command group that initiates the DMA is issued using a secure context and therefore uses the CEK for encryption and integrity protection.

[0131] The peripheral device runtime uses the kernel to authenticate 1722 and decrypt the data on the peripheral device. To do this, it issues a command group to start the kernel, passing the data address, data encryption key and expected MAC as kernel parameters.

[0132] The peripheral device authenticates the encrypted data and generates a MAC that is checked 1724 against the expected MAC, and finally decrypts 1726 the data in the device memory, overwriting the encrypted data in the process.

[0133] A property of secure memory replicas is crypto-agility. Since the primitives are implemented entirely in software, the runtime can support a variety of encryption and authentication schemes without requiring hardware changes.

[0134] Secure Kernel Launch (cudaSecureKernelLaunch) utilizes secure copies to transfer the given kernel code and constant memory to peripherals, and then issues a command group to launch the kernel to pass parameter values. The command processor is modified to disable preemption of kernels submitted through a secure channel. The combination of secure copy and no preemption ensures that the kernel completes without information leakage or tampering. When preemption is supported, the command processor saves the context of the preempted kernel in hidden memory and resets registers and application-managed memory. Subsequently, another kernel begins execution. To switch back, the command processor reads and restores the state from hidden memory.

[0135] CUDA(TM) streams are primitives for overlapping host and peripheral computations and input / output (I / O) transfers. Each stream is allocated a separate channel, and each channel shares the same address space to enable concurrency and asynchronous submission of independent tasks. By allowing channels in the same context to share pages, the techniques described in this article naturally support secure streams (cudaSecureStreamCreate).

[0136] Empirical results are now described that demonstrate the effectiveness of the techniques described in this article. In the example, the well-known Caffe framework is used to train a neural network using the techniques described in this article on the well-known Cifar-10 dataset, which consists of 60,000 32×32 images of 10 categories. The neural network consists of 11 layers: 3 layers of convolution, pooling, rectified linear unit nonlinearity (RELU), followed by local contrast normalization and a linear classifier. 10,000 training iterations are run (batch size is 200), and the neural network is tested every 250 iterations (using 10,000 images). For testing, two different batch sizes (200 and 400) are evaluated. The neural network has the same accuracy whether trained using the techniques described in this article or using conventional techniques.

[0137] Fig.18 (Left) shows the impact of the technique on execution time at three execution stages (i.e., initialization, testing, and training), with a batch size of 200 for training and a batch size of 400 for testing. Fig.18 In the figure, the vertical line filled area represents the baseline, the dashed line filled area represents isolation, the cross-hatched area represents command authorization, and the unfilled area represents secure copy. For training, the execution time of 25 iterations averaged over all iterations is reported. The overhead is broken down into three intervals: isolation (i.e., using command-based APIs), command encryption / authentication, and secure memory replication. Each of these stages is reduced by 90%, 56%, and 39%, respectively. Overhead accounts for 40% of the entire execution process, as training takes more than 93% of the time.

[0138] Overhead during initialization is due to security context and module creation to enforce isolation (11% of overhead), secure copies of models and data for initial testing (41% of overhead), and the last testing phase due to command group authentication which accounts for 48% of the overhead.

[0139] The breakdown of the test and training overhead shows that command group encryption / authentication accounts for 58% and 78% of the overhead, respectively. This is because this workload executes a large number of relatively short kernels (one per batch and layer). Analysis of the time spent on kernel launches shows that approximately 80% of the overhead is due to simulated authenticated decryption. In particular, each launch incurs a 9μsec latency, and is 0.7μsec for encryption at runtime and 2.8μsec for decryption in the emulator. The overhead could be significantly reduced if the command processor had access to a hardware cryptographic engine.

[0140] Pipelining in Caffe is implemented to overlap host-side authenticated encryption of a batch with the execution of previous batches on the GPU. Fig.18(Right) shows the impact of pipelined safety copies. The vertical line filled area represents batch 200, the cross-hatched area represents batch 200opt, and the unfilled area represents batch 400opt. It is found that the indirect overhead decreases as the batch size increases. This is because there is more room for overlap.

[0141] The well-known Black-Scholes process is run with 10 batches, 4 million options per batch, and 2500 iterations per batch. Fig.19 As shown, (where vertical filled areas represent executions, dashed filled areas represent allocations, and unfilled areas represent copies), the first batch incurs higher overhead (44%) due to the cost of secure module loading and secure memory copying. Subsequent batches have less overhead (35% on average) since no modules are loaded. Unlike Cifar-10, command authentication is not a consideration for Black-Scholes because it executes one long-running kernel per batch.

[0142] As an alternative or in addition to other examples described herein, examples include any combination of the following:

[0143] Clause 1. A peripheral device package for use in a host computing device, the peripheral device package comprising:

[0144] multiple computing elements;

[0145] Multiple resources, shared by multiple computing elements;

[0146] a data structure stored in a hidden memory of the peripheral device package, the data structure maintaining metadata relating to ownership of resources of the peripheral device package by a plurality of user runtime processes of a host computing device using the computing element, at least one of the user runtime processes being a secure user runtime process;

[0147] A command processor is configured to use a data structure to enforce isolation of resources used by the secure user runtime process.

[0148] Clause 2. A peripheral device package according to clause 1, wherein a command processor receives a command from a user runtime process via a driver of the peripheral device package, the driver being in a host computing device, and wherein the command processor receives the command after the driver specifies a resource in the command; and wherein the command processor is configured to return a response to the received command to be authenticated by the user runtime process.

[0149] Clause 3. A peripheral device package according to Clause 1, comprising:

[0150] One or more silicon interposers that support the computing elements, memory, and command processors; and

[0151] A root of trust, integrated into the silicon of the device package, is suitable for generating attestation keys for verification using a trusted certificate authority to enable the user runtime process to trust the peripheral device via an untrusted host if the user runtime process is trusted.

[0152] Clause 4. A peripheral device package according to clause 1, wherein the command processor is configured to use the data structure to ensure trusted execution of the secure user runtime process on the peripheral device by: when a request is received to create a secure channel on the peripheral device package for use by the secure user runtime process, checking the data structure to find out whether one or more resources associated with the request are protected from direct access by the host and are free, and if the check fails, the request fails, and if the check succeeds, the request is implemented and the data structure is updated.

[0153] Clause 5. A peripheral device package according to clause 1, wherein the command processor is configured to use the data structure to ensure trusted execution of a secure user runtime process on the peripheral device by: when a channel page directory entry command is received, only if a check of the data structure succeeds, unmap an existing page table at a specified index in a page directory for a specified channel and map a new page table.

[0154] Clause 6. A peripheral device package according to clause 5, wherein the check includes: before unmapping, ensuring that existing page tables are unlocked or are all locked and authorized by a user signature; and wherein the check includes: before mapping a new page table, checking whether the new page table is allocated on a page in a protected area of ​​memory that is not directly accessible to the host, and checking whether the new page table is free.

[0155] Clause 7. A peripheral device package according to clause 1, wherein the command processor is configured to use the data structure to ensure trusted execution of a secure user runtime process on the peripheral device by: when a channel page table entry command is received, and only if a check of the data structure succeeds, removing an existing mapping from a specified page table entry and creating a new mapping based on a specified array starting at a specified virtual address and spanning a specified range.

[0156] Clause 8. A peripheral device package according to clause 7, wherein the checking comprises: checking whether the physical page is in an unlocked state before removing the existing page table entry.

[0157] Clause 9. A peripheral device package according to clause 8, wherein the check includes: if the existing mapping to be removed is for a locked page, checking the existence of a message authentication code MAC through the channel page table entry command, wherein the message authentication code MAC is generated using a channel encryption key, and the channel encryption key is formed from a trust root integrated in the silicon of the peripheral device package.

[0158] Clause 10. A peripheral device package according to clause 1, wherein the command processor is configured to use the data structure to ensure trusted execution of a secure user runtime process on the peripheral device by: when a channel destroy command is received, freeing memory allocated to the channel by browsing a page directory to find physical pages owned by the specified channel and clearing information of the physical pages from the data structure.

[0159] Clause 11. A peripheral device package according to clause 10, wherein if the channel to be destroyed is secure, the command is executed only if authorization including a message authentication code MAC is received with the command, the message authentication code MAC using the channel encryption key of the channel to be destroyed.

[0160] Clause 12. A peripheral device package according to clause 10, wherein the command processor is further configured to: unmap physical pages of the page directory and channel descriptor of the channel to be destroyed, decrement the counter for the page table, and transition any page having a counter reduced to zero to an idle state.

[0161] Clause 13. The peripheral device package of clause 1, wherein the command processor is configured such that: when a measurement command is received, the command processor computes a verifiable artifact summarizing the contents of the secure channel.

[0162] Clause 14. The peripheral device package of clause 13, wherein the verifiable artifact comprises: a message authentication code, a starting virtual address, and a digest of the contents of pages in a specified range.

[0163] Clause 15. A computer-implemented secure user runtime process having instructions for execution on a peripheral device enclosure of an untrusted host computing device, the secure user runtime process comprising:

[0164] Commands are sent to a command processor of the peripheral device encapsulation via the untrusted host, the commands using data structures in the hidden memory of the peripheral device encapsulation to enforce isolation of resources of the peripheral device encapsulation used by the secure user runtime process.

[0165] Clause 16. The computer-implemented secure user runtime process of clause 15, comprising: in response to sending a command to a control processor, receiving a response and authenticating the response to ensure that an untrusted host has not tampered with the sent command.

[0166] Clause 17. A computer-implemented user runtime process according to clause 15, comprising: a protocol for securely copying sensitive information from secure memory on a host computing device to a peripheral device enclosure, the protocol comprising copying a kernel that performs authenticated decryption to the peripheral device enclosure and receiving a digest from a command processor in response, and checking whether the kernel has been correctly copied by comparing the received digest with a digest of the kernel computed by the host computing device.

[0167] Clause 18. A user runtime process executed by a computer according to Clause 15, comprising: securely copying code and constant memory of a designated kernel to a peripheral device package, and issuing a command set to launch the kernel.

[0168] Clause 19. A computer-implemented user runtime process according to Clause 15, comprising: sending commands to an untrusted host in encrypted form via a secure session between the user runtime process and a command processor.

[0169] Clause 20. A peripheral device package for use in a host computing device, the peripheral device package comprising:

[0170] a plurality of computing elements for execution of a plurality of user runtime processes of a host computing device, wherein at least one of the user runtime processes is secure;

[0171] Multiple resources, shared by multiple computing elements;

[0172] A command processor is configured to receive a command from a user runtime process via a driver in a host computing device, and wherein the command processor receives the command after the driver specifies a resource in the command; and wherein the command processor is configured to return a response to the received command for authentication by the user runtime process.

[0173] The term "computer" or "computing-based device" is used herein to refer to any device having processing capabilities to enable it to execute instructions. Those skilled in the art will appreciate that such processing capabilities are incorporated into many different devices, and thus the terms "computer" and "computing-based device" include personal computers (PCs), servers, mobile phones (including smartphones), tablet computers, set-top boxes, media players, game consoles, personal digital assistants, wearable computers, and many other devices, respectively.

[0174] In some examples, the methods described herein are performed by software in a machine-readable form on a tangible storage medium, for example in the form of a computer program including computer program code means adapted to perform all of the operations of one or more methods described herein when the program is run on a computer, and wherein the computer program may be embodied on a computer-readable medium. The software is suitable for execution on a parallel processor or a serial processor, such that the method operations may be performed in any suitable order or simultaneously.

[0175] This recognizes that software is a valuable, separately tradable commodity. It is intended to encompass software that runs on or controls "dumb" or standard hardware to perform a desired function. It is also intended to cover software that "describes" or defines the configuration of hardware, such as HDL (Hardware Description Language) software, which is used to design silicon chips or configure general-purpose programmable chips to perform a desired function.

[0176] Those skilled in the art will recognize that the storage device for storing program instructions is optionally distributed across a network. For example, a remote computer can store an example of a process described as software. A local or terminal computer can access the remote computer and download part or all of the software to run the program. Alternatively, the local computer can download software as needed, or execute some software instructions at a local terminal, and execute some software instructions at a remote computer (or computer network). Those skilled in the art will also recognize that, by utilizing conventional techniques known to those skilled in the art, all or part of the software instructions can be executed by a dedicated circuit (such as a digital signal processor (DSP), a programmable logic array, etc.).

[0177] As will be apparent to one skilled in the art, any range or device value given herein may be extended or altered without losing the effect sought.

[0178] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0179] It will be understood that the benefits and advantages described above may relate to one embodiment or may relate to several embodiments. The embodiments are not limited to embodiments that solve any or all of the problems described, or embodiments that have any or all of the benefits and advantages described. It will also be understood that reference to "an" item refers to one or more of these items.

[0180] The operations of the methods described herein may be performed in any suitable order, or simultaneously where appropriate. In addition, individual blocks may be removed from any method without departing from the scope of the subject matter described herein. Aspects of any of the above examples may be combined with aspects of any other example described to form other examples without losing the effect sought.

[0181] The term "comprising" is used herein to mean including the identified method blocks or elements, but such blocks or elements do not comprise an exclusive list and the method or apparatus may contain additional blocks or elements.

[0182] The term "subset" is used herein to refer to a subset in place, such that a subset of a set does not include all elements of the set (ie, at least one element of the set is missing from the subset).

[0183] It will be understood that the above description is given by way of example only, and that various modifications may be made by those skilled in the art. The above description, examples, and data provide a complete description of the structure and use of the exemplary embodiments. Although various embodiments have been described above with a certain degree of particularity or with reference to one or more individual embodiments, those skilled in the art may make various changes to the disclosed embodiments without departing from the scope of this specification.

Claims

1. A peripheral device package for use in a host computing device, the peripheral device package comprising: multiple computing elements; a plurality of resources shared by the plurality of computing elements; a data structure stored in a hidden memory of the peripheral device package, the data structure maintaining metadata relating to ownership of resources of the plurality of resources of the peripheral device package by a plurality of user runtime processes of the host computing device that use computing elements of the plurality of computing elements, at least one of the user runtime processes being a secure user runtime process, wherein the hidden memory is not mapped into a virtual address space of the host computing device; A command processor is configured to determine resource allocation approval based on an inspection of data structure metadata to implement isolation of the resources used by the secure user runtime process.

2. A peripheral device package according to claim 1, wherein the command processor receives a command from the user runtime process via a driver of the peripheral device package, the driver being in the host computing device, and wherein the command processor receives the command after the driver makes a designation of resources in the command; and wherein the command processor is configured to return a response to the received command for authentication by the user runtime process.

3. The peripheral device package according to claim 1, comprising: one or more silicon interposers supporting the computing elements, memory, and the command processor; as well as A root of trust is integrated into the silicon of the device package, the root of trust being suitable for generating an attestation key for verification using a trusted certificate authority to enable the user runtime process to trust the peripheral device via the untrusted host if the user runtime process is trusted.

4. A peripheral device package according to claim 1, wherein the command processor is configured to: use the data structure to ensure trusted execution of the secure user runtime process on the peripheral device by: when a request is received to create a secure channel on the peripheral device package for use by the secure user runtime process, check the data structure to find out whether one or more resources associated with the request are protected from direct access by the host and are free, and if the check fails, fail the request, and if the check succeeds, implement the request and update the data structure.

5. A peripheral device package according to claim 1, wherein the command processor is configured to: use the data structure to ensure trusted execution of the secure user runtime process on the peripheral device by: when a channel page directory entry command is received, unmap an existing page table and map a new page table at a specified index in the page directory of the specified channel only if the check of the data structure succeeds.

6. The peripheral device package of claim 5, wherein the checking comprises: Before unmapping, ensure that the existing page table is unlocked or has authorization signed by the user; and The checking includes: before mapping the new page table, checking whether the new page table is allocated on a page in a protected area of ​​the memory that the host cannot directly access, and checking whether the new page table is free.

7. A peripheral device package according to claim 1, wherein the command processor is configured to: use the data structure to ensure trusted execution of the secure user runtime process on the peripheral device by: when a channel page table entry command is received, only if the check of the data structure succeeds, remove the existing mapping from the specified page table entry and create a new mapping based on the specified array starting from the specified virtual address and spanning the specified range.

8. The peripheral device package of claim 7, wherein the checking comprises: Before removing an existing page table entry, check if the physical page is in unlocked state.

9. The peripheral device package of claim 8, wherein the checking comprises: If the existing mapping to be removed is for a locked page, the channel page table entry command checks for the presence of a message authentication code MAC generated using a channel encryption key formed from a root of trust integrated in the silicon of the peripheral device package.

10. The peripheral device package of claim 1 , wherein the command processor is configured to: use the data structure to ensure trusted execution of the secure user runtime process on the peripheral device by: when a channel destroy command is received, freeing memory allocated to the channel by browsing a page directory to find a physical page owned by the specified channel and clearing information of the physical page from the data structure.

11. The peripheral device package of claim 10, wherein if the channel to be destroyed is secure, the command is executed only if authorization including a message authentication code MAC is received with the command, the message authentication code MAC using a channel encryption key of the channel to be destroyed.

12. The peripheral device package of claim 10, wherein the command processor is further configured to: unmap physical pages of the page directory and channel descriptor of the channel to be destroyed, decrement a counter for a page table, and transition any page having a counter that has decreased to zero to an idle state.

13. The peripheral device package of claim 1, wherein the command processor is configured such that, when a measurement command is received, the command processor computes a verifiable artifact summarizing the contents of the secure channel.

14. The peripheral device package of claim 13, wherein the verifiable artifact comprises: The message authentication code, the starting virtual address, and a digest of the contents of the pages in the specified range.

15. A method for a computer-implemented secure user runtime process having instructions for execution on a peripheral device package of an untrusted host computing device, said method for said computer-implemented secure user runtime process comprising: Sending commands to a command processor of the peripheral device package via the untrusted host computing device, the commands using a data structure in a hidden memory of the peripheral device package, including metadata stored in the data structure, to implement isolation of resources of the peripheral device package used by the secure user runtime process by performing a check on the metadata to determine whether a resource allocation decision is approved, wherein the hidden memory is not mapped into a virtual address space of the untrusted host computing device.

16. The computer-implemented secure user runtime process of claim 15, comprising: In response to sending a command to the command processor, a response is received and authenticated to ensure that the untrusted host has not tampered with the sent command.

17. The computer-implemented secure user runtime process of claim 15, comprising a protocol for securely copying sensitive information from secure memory on the host computing device to the peripheral device enclosure, the protocol comprising: A kernel performing authenticated decryption is copied into the peripheral device package and a digest is received from the command processor in response and checking that the kernel has been correctly copied by comparing the received digest with a digest of the kernel computed by the host computing device.

18. The computer-implemented secure user runtime process of claim 15, comprising: The code and constant memory of a specified kernel are securely copied to the peripheral package, and a command set is issued to start the kernel.

19. The computer-implemented secure user runtime process of claim 15, comprising: The command is sent in encrypted form to the untrusted host via a secure session between the user runtime process and the command processor.

20. A host computing device for use with a peripheral device package, the host computing device comprising: A memory, the memory comprising: a plurality of computing elements for execution of a plurality of user runtime processes of the host computing device, at least one of the user runtime processes being secure and having a corresponding computing element stored in a hidden memory that is not mapped into a virtual address space of the host computing device; a plurality of resources shared by the plurality of computing elements; a command processor configured to receive a command from the user runtime process via a driver in the host computing device, and wherein the command processor receives the command after a resource is specified in the command by the driver; wherein the command processor uses the metadata from the hidden memory to implement isolation of the resources by performing a check on the metadata to determine whether a resource allocation decision is approved, and wherein the command processor is configured to return a response to the received command for authentication by the user runtime process.

Images