Apparatus and Method for Protecting Consumer Data in a Public Cloud Environment

Abstract

Methods, systems, computer-readable media, and apparatuses are provided for securing a cloud environment, in which a public cloud service provider can remove its code from a trusted computing base (TCB) of its cloud service consumers. The method for securing a cloud environment keeps a virtual machine monitor (VMM), devices, firmware, and physical adversaries (where a rogue administrator / technician attempts to directly access cloud host hardware) outside of a consumer's virtual machine (VM) TCB. Only the consumer who owns the secure VM can modify the VM or access the content of the VM as determined by the consumer.

Description

Technical Field

[0001] The embodiments relate to the security of public clouds and, in particular, enable consumers of public cloud services to ensure that the consumer's process execution in the cloud and the consumer's private data are protected from access and modification by others, including the public cloud service provider. Background Art

[0002] The term "cloud computing" is used to describe network-based computing (typically over the Internet). According to Wikipedia, "Cloud computing provides shared processing resources and data on demand to computers and other devices. Cloud computing is a model for enabling ubiquitous, on-demand access to a shared pool of configurable computing resources (such as networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort. Cloud computing and storage solutions provide users and enterprises with various capabilities for storing and processing their data in third-party data centers. Cloud computing relies on resource sharing to achieve coherence and economies of scale, similar to public utilities (such as the electric grid) on the network." (Source: Wikipedia, https: / / en.wikipedia.org / wiki / Cloud_computing, accessed on August 11, 2016, citation omitted).

[0003] The current availability of high-capacity networks, low-cost computers, and storage devices, as well as the widespread adoption of hardware virtualization, service-oriented architectures, and autonomic and utility computing, have led to growth in cloud computing. Companies can scale up by requesting additional resources from a cloud service provider when computing needs increase and then scale down again when demand decreases.

[0004] Cloud computing provides resources as a service. "Cloud computing providers supply their'services' according to different models, and the three standard models according to the National Institute of Standards and Technology (NIST) are Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). These models provide increasing levels of abstraction; they are thus often depicted as layers in a stack, where the infrastructure forms the bottom layer; the platform as a service forms the middle layer; and the software as a service forms the top layer. These layers can be implemented independently of each other. For example, SaaS implemented on a physical machine (bare metal) can be provided without using the underlying PaaS or IaaS layers; and conversely, a program can be run on IaaS and accessed directly without wrapping it as SaaS." (Source: Wikipedia, https: / / en.wikipedia.org / wiki / Cloud_computing, accessed on August 11, 2016, citation omitted).

[0005] The NIST definition of cloud computing defines the service models as follows:

[0006] Software as a Service (SaaS). The ability provided to the consumer is to use the provider's applications running on a cloud infrastructure. The applications are accessible from various client devices through a thin client interface such as a web browser (e.g., web-based email) or a program interface. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even the individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0007] Platform as a Service (PaaS). The ability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications that are created using programming languages, libraries, services, and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but has control over the deployed applications and possibly has configuration settings for the application hosting environment.

[0008] Infrastructure as a Service (IaaS). The ability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, and deployed applications; and possibly has limited control over the selection of networking components (e.g., host firewalls). (Source: Wikipedia, https: / / en.wikipedia.org / wiki / Cloud_computing, accessed on August 11, 2016, citation omitted).

[0009] One enabling technology for cloud computing is virtualization. "Virtualization software separates physical computing devices into one or more 'virtual' devices, each of which can be easily used and managed to perform computing tasks. Hardware virtualization is the virtualization of a computer into a complete hardware platform, some logical abstractions of its component parts, or simply the functionality required to run various operating systems. Virtualization hides the physical characteristics of the computing platform from the user and instead presents another abstract computing platform", commonly referred to as a "virtual machine". (Source: Wikipedia, https: / / en.wikipedia.org / wiki / Hardware_virtualization, accessed on August 11, 2016, citation omitted). The software that controls virtualization is called a "hypervisor" or "virtual machine monitor". Provisioning and executing the hypervisor / virtual machine monitor to create virtual machines on behalf of consumers is an example of a service provided by a public cloud service provider. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram showing a typical virtual machine environment.

[0011] Figure 2 is a block diagram showing a virtual machine environment according to an embodiment of the present invention.

[0012] Figure 3 is a block diagram of a cloud service environment according to an embodiment of the present invention.

[0013] Figure 4 is a diagram showing an apparatus that can be used to implement an embodiment of the present invention.

[0014] Figure 5 is a flowchart of a method performed by a consumer of a cloud service according to an embodiment of the present invention.

[0015] Figure 6 is a flowchart of a method performed by a cloud service provider according to an embodiment of the present invention.

[0016] Figure 7 is a diagram showing the components of a consumer domain image according to an embodiment of the present invention.

[0017] Figure 8 is a diagram showing a physical data address according to an embodiment of the present invention.

[0018] Figure 9 is a diagram showing a virtual-to-physical memory mapping according to an embodiment of the present invention.

[0019] Figure 10is a diagram that shows another virtual-to-physical memory mapping according to an embodiment of the present invention.

[0020] Figure 11 is a diagram that shows the initial steps performed by a cloud service provider for providing a domain image to a consumer according to an embodiment of the present invention.

[0021] Figure 12 is a diagram that shows the messages between a consumer and a cloud service provider for providing a domain image to the consumer according to an embodiment of the present invention.

[0022] Figure 13 is a diagram that shows a consumer providing an encrypted domain image according to an embodiment of the present invention.

[0023] Figure 14 is a diagram that shows the messages between components of a cloud service environment for encrypting a domain image and establishing a key domain according to an embodiment of the present invention.

[0024] Figure 15 is a diagram that shows the messages between components of a cloud service environment for loading a consumer's encrypted domain image into the memory of a server capable of having a key domain according to an embodiment of the present invention.

[0025] Figure 16 is a diagram that shows the initialization of a key domain according to an embodiment of the present invention.

[0026] Figure 17 is a flowchart of an operation method of a CPU of a server capable of having a key domain in performing a "create key domain" operation according to an embodiment of the present invention.

[0027] Figure 18 is a diagram that shows the verification of a domain image according to an embodiment of the present invention.

[0028] Figure 19 is a diagram that shows the messages between components of a cloud service environment for verifying a domain image according to an embodiment of the present invention.

[0029] Figure 20 is a flowchart of an operation method of a CPU of a server capable of having a key domain in performing a "hash key domain" operation according to an embodiment of the present invention.

[0030] Figure 21 is a diagram that shows the switching between key domains according to an embodiment of the present invention.

[0031] Figure 22is a diagram that shows messages between components of a cloud service environment when executed within a key domain, according to an embodiment of the present invention.

[0032] Figure 23 is a flowchart of an operation method of a CPU of a server capable of having a key domain in performing a "switch key domain" operation, according to an embodiment of the present invention.

[0033] Figure 24 is a flowchart of an operation method of a CPU of a server capable of having a key domain in performing a paging structure walk in response to a page miss, according to an embodiment of the present invention.

[0034] Figure 25 is a diagram that shows the growth of a domain image, according to an embodiment of the present invention.

[0035] Figure 26 is a diagram that shows messages between components of a cloud-based environment for growing a domain manager (VMMlet), according to an embodiment of the present invention.

[0036] Figure 27 is a diagram that shows messages between components of a cloud service provider environment for running a domain manager (VMMlet) to request more memory pages from a memory manager, according to an embodiment of the present invention.

[0037] Figure 28 is a diagram that shows messages between components of a cloud service environment showing a request for additional memory pages when scheduling a VM on a single CPU, according to an embodiment of the present invention.

[0038] Figure 29 is a diagram that shows a running domain manager (VMMlet), according to an embodiment of the present invention.

[0039] Figure 30 is a diagram that shows a plurality of virtual machines within a key domain managed by a domain manager (VMMlet) and a second key domain managed by another domain manager (OSlet), according to an embodiment of the present invention.

[0040] Figure 31A is a diagram that shows the determination of a full row location and a slot from a physical memory address, according to an embodiment of the present invention.

[0041] Figure 31B is a diagram that shows a data row stored in a data memory address space and an integrity value stored in a full data address space.

[0042] Figure 32 is a diagram that illustrates a system that can be used to implement an embodiment of the present invention.

[0043] Figure 33 is a diagram that illustrates a system that can be used to implement an embodiment of the present invention.

[0044] Figure 34 is a diagram that illustrates a system that can be used to implement an embodiment of the present invention. Detailed Description

[0045] In a known public cloud service environment, the hypervisor of the public cloud service provider controls: the instantiation of virtual machines on behalf of consumers, the execution of the virtual machines to provide services to consumers, and the access by consumers to the resources of the provider via the consumer virtual machines. Due to this high level of control provided by the public cloud service provider, consumer data within the virtual machines is vulnerable to access and / or compromise by adversaries who can access the resources of the public cloud service provider. For example, consumer data within the virtual machines can be accessed, disclosed, and / or destroyed by a compromised hypervisor, a rogue administrator of the resources of the public cloud service provider, and / or a malicious person with physical access to the resources of the public cloud service provider. In response to government guarantees for consumer private data, the private data of consumers can be accessible even to the public cloud service provider.

[0046] A secure public cloud environment is provided herein, where the public cloud service provider can remove its code and data from the trusted computing base (TCB) of its cloud service consumers. The secure public cloud environment keeps the hypervisor (VMM), devices, firmware, and physical adversaries (where a rogue administrator / technician attempts to directly access the cloud host hardware) outside the TCB of the consumer's virtual machine (VM).

[0047] By using the techniques described herein, consumers of public cloud services can control the cloud environment in which their data is processed. The consumer can provide code to execute on the servers of the cloud service provider for establishing a secure environment for the consumer, which is referred to herein as a key domain. The consumer can provide all the code to execute within the server software stack of the cloud service provider, including privileged code such as hypervisor (VMM) code, operating system code, application code, and the like.

[0048] The consumer creates a password-secured code / data image (which is referred to herein as the consumer domain image or domain image), which can only be decoded and executed by the server hardware of the cloud service provider by using the encrypted key domain key provided by the consumer. The consumer can securely exchange the password-type (multiple) key domain keys, which are used to create the consumer's secure password domain image, which exclusively utilizes the server hardware of the cloud service provider on which it will be executed. The cloud service provider software, administrators, and technicians outside the consumer's password-secured environment / key domain are not provided with the consumer's encrypted key domain key and have no ability to view, augment, or modify the content of the consumer domain image (whether the domain image is in transit, during storage, or during execution).

[0049] The consumer can password-encrypt the domain image and only provide the encrypted domain image to the server of the cloud service provider. In addition, the consumer can password-bind the domain image to a specific memory location / address on the server of the cloud service provider on which the domain image will be executed. This password binding will ensure that the domain image is executed within the consumer's secure environment / key domain at the specified memory location.

[0050] The consumer can additionally enable the domain image to communicate with the cloud service provider software and / or other domains outside the consumer's secure environment / key domain through the specified shared memory area. The consumer can additionally enable the domain image to interact with the device, perform input / output (I / O), and communicate via exchanging messages, audio, video, etc.

[0051] The techniques described herein also enable a public cloud service provider to verify the consumer's code, thereby providing the consumer's secure environment / key domain on the server of the cloud service provider. In particular, the cloud service provider can verify through hardware that the part of the domain image supplied by the consumer that includes privileged code is as expected. For example, the cloud service provider can verify that the virtual machine monitor (VMM) code supplied by the consumer, which would typically be provided by the cloud service provider, is the same VMM code expected by the cloud service provider. This verification can be performed by the hardware of the cloud service provider, even though the cloud service provider's software has never been provided with the unencrypted domain image or the unencrypted key domain key.

[0052] In addition, the cloud service provider server can correctly enter and start executing the consumer's domain image, where the hardware checks that the consumer and the cloud service provider agree on how the domain image is initially executed and the correct state of the hardware (CPU) when entering the cryptographic domain image. The cloud service provider hardware can securely switch from one consumer's domain image to another consumer's domain image. In addition, the cloud service provider hardware can securely switch from one consumer's domain image to the cloud service provider's software (such as a memory manager). Additionally, the cloud service provider hardware can cryptographically verify the integrity of the domain image to detect tampering / modification by exchanging a cryptographic integrity check value or a message authentication code. The cloud service provider can also prevent replay of the consumer domain image content by allowing writing / configuring of the integrity check value, but blocking reading of the integrity check value outside the hardware.

[0053] Only consumers with a secure environment / key domain can modify the domain image or access the content of the domain image. The lack of control over the consumer domain image makes it technically infeasible for a public cloud service provider to access consumer data within the consumer's secure environment, even in the face of a legal warrant from the government to do so. This functionality makes the public cloud as secure as a private cloud placed under an internal lock and key.

[0054] Referring now to FIG. 1, a block diagram is shown that illustrates the components of a typical virtual machine environment 100. A typical implementation of the virtual machine environment provided in the cloud service provider's server is shown. Running on the server hardware 110 is a virtual machine monitor (VMM) layer 120. In the typical virtual machine environment 100 shown, the VMM layer 120 is computer software or firmware that creates and runs virtual machines (VMs), such as VM1 130 1 , VM2 130 2 , and VM3 130 3 . Each of the VMs VM1 130 1 , VM2 130 2 , and VM3 130 3 is shown as a separate block in Figure 1 , which represents different VMs all under the control of the common VMM layer 120. The VMM layer 120 provides access to server resources, such as the server hardware 110, to the VMs controlled by the VMM.

[0055] The VMM layer 120 uses data structures, such as the VM control structure (VMCS) 124 and the extended page table (EPT) 126, to control the execution of the VM. The VMCS is a data structure that exists once in memory for each VM while it is being managed by the VMM. In the case of each change in the execution context between different VMs, the VMCS is restored for the current VM, thus defining the state of the virtual processor of the VM. The extended page table (EPT) is used to launch the virtual processor of the VM with the privilege of being an "unrestricted guest".

[0056] The software or firmware of the VMM layer 120 is provided by the cloud service provider and is part of the trusted computing base (TCB) for each VM. According to Wikipedia, "The trusted computing base (TCB) of a computer system is the collection of all hardware, firmware, and / or software components that are critical to its security in the sense that a vulnerability or weakness in the TCB could compromise the security properties of the entire system. Conversely, parts of the computer system outside the TCB must not be able to misbehave to the extent that they would leak any more privileges than are granted to them... Modern operating systems strive to reduce the size of the TCB so that a thorough examination of its code base (by means of manual or computer-assisted software auditing or program verification) becomes feasible." (See Wikipedia, https: / / en.wikipedia.org / wiki / Trusted_computing_base, accessed on August 9, 2016).

[0057] In Figure 1 the normal virtual machine environment 100, the VMM 122 provided by the cloud service provider is in the TCB of each of VM VM1130 1 、VM2 130 2 、and VM3 130 3 . Including the VMM 122 in the TCB prevents a particular VM, such as VM1 130 1 from seeing, measuring, or trusting the VMM 122 that controls that particular VM. The cloud service provider can change the VMM 122 at any time without the knowledge of the owner of VM VM1130 1 . Moreover, there is no cryptographic separation between VMs. If the VMM has been compromised, a degraded VM can access the private data in a second VM via the compromised VMM, which is, however, trusted by the second VM.

[0058] For consumers to receive trustworthy guarantees for the process / VM that controls the consumers, the most well-known techniques use hardware to measure the software / firmware running on a remote machine in the cloud (in this case, the VMM 122), and prove back to the consumers that the software / firmware running on the remote machine in the cloud is the software / firmware version expected by the consumers. In the case where the VMM of a public cloud service provider is included in the consumer's TCB, the consumer has no way to independently assess the trustworthiness proof made by the public cloud service provider.

[0059] Figure 2 FIG. 4 is a block diagram of a virtual machine environment 200 according to an embodiment of the present invention. In this environment, the concepts of key domains and domain managers are introduced. A key domain is a cryptographically separated portion of the memory, where access to the data stored in the memory locations belonging to the key domain requires using the associated key domain key to decrypt the data. The domain manager can use the key domain to cryptographically separate data belonging to different owners; in a cloud service environment, the domain manager can use the key domain to cryptographically separate data belonging to different consumers of cloud services, such as bank services.

[0060] For example, in Figure 2 the virtualization environment 200, key domains KD1 250 1 and KD2 250 2 are used to separate data belonging to different virtual machines VM1 230 1 and VM2 230 2 . The data belonging to each of the virtual machines VM1 230 1 and VM2 230 2 may include, for example, consumer secrets (such as bank account numbers, social security numbers, etc.) belonging to each of the virtual machines VM1 230 1 and VM2 230 2 . As another example, the data belonging to each of the virtual machines VM1 230 1 and VM2 230 2 may include computer code (which is also referred to as a code image or simply an image), and the computer code will be executed to protect the secrets of each corresponding virtual machine within the environment of the cloud service provider.

[0061] The corresponding domain managers (VMMlets 222 1 and 222 2 ) act on behalf of their corresponding host owners VM1 230 1 and VM2 230 2 of the virtual machine monitor (VMM, such as Figure 1a similar role to that of the VMM 122). The domain manager (VMMlet) provides VMM functionality within the VM, rather than as a completely separate VMM layer as shown in Figure 1 . The domain manager (VMMlet) is privileged code that has the ability to create, exit, and resume VM execution. These privileges can be referred to as "vmxroot" functionality and include the ability to execute commands such as: virtual machine control structure (VMCS) save / restore, general-purpose register (GPR) save / restore, and / or vmexit / vmresume. Additionally, the domain manager (VMMlet) controls critical resources such as the interrupt descriptor table (IDT), advanced programmable interrupt controller (APIC) instructions, and paging data structures such as page tables and extended page tables (EPT). In some embodiments, the domain manager (VMMlet) portion may only include data structures that control the VM (such as the VMCS), its associated data structures, and the EPT associated with the VM.

[0062] The domain manager (VMMlet) restricts access by its host VM to a corresponding cryptographically separated portion of memory - called the key domain. The contents of each physical memory location belonging to the key domain are encrypted by hardware using a common key domain key. When the hardware writes data to a memory location belonging to the key domain, the data is encrypted using the key domain key; when the hardware reads data from a memory location belonging to the key domain, the data is decrypted using the key domain key.

[0063] In one embodiment, the key domain key is created by the consumer that owns the key domain and is provided directly and securely to the server hardware of the cloud service provider. In other embodiments, the consumer may transform a key provided by another entity (such as the server of the cloud service provider) into another key for encrypting memory locations belonging to the key domain. In still other embodiments, different keys may be used to encrypt IP blocks (sets of memory locations) belonging to the key domain; for example, different keys may be used to encrypt the following IP blocks: the IP blocks that contain code from the consumer VM image of the key used to encrypt other consumer secrets. To simplify the description of the embodiments herein, this application describes the contents of each physical memory location belonging to the key domain encrypted by the key domain key created by the consumer that owns the key domain, although other embodiments are also within the scope of the invention.

[0064] If the content of a physical memory location belonging to a key domain is decrypted using an incorrect key domain key, the resulting plaintext will be corrupted. Additionally, if the memory is integrity-protected and the content of a physical memory location belonging to a key domain is decrypted using an incorrect key domain key, the resulting plaintext will not meet the integrity criteria for the physical memory location belonging to the key domain. Although the scope of the present invention does not require that memory locations belonging to a key domain be integrity-protected, memory integrity protection can be used to enhance the security of the techniques described herein.

[0065] In one embodiment, a key domain is defined by using unused physical address bits (or other metadata passed through the cache). For example, since there are likely to be fewer physical memory locations installed in the system than can be addressed using a 64-bit physical memory address, the unused most significant address bits can be used to select between different key domains. Two different key domain addresses can alias to the same physical memory location. However, when data from that physical memory location is read into the cache, the cache independently holds the key domain address with full address resolution (e.g., including the full 64-bit physical memory address). The key domain address uniquely identified when considering the unused physical address bits of the full 64-bit physical memory address determines the key domain to which the physical memory location belongs. By identifying the key domain to which a physical memory location belongs, the key domain key that can be used to decrypt the content of that physical memory location is also identified.

[0066] The memory manager can select between different address values that alias to the same physical memory location; i.e., the memory manager can select between different key domains based on address aliasing. In one embodiment, an integrity check value (ICV, such as a keyed hash message authentication code (HMAC)) is calculated using the key domain key created by the owner (consumer) of the key domain. The memory manager can access an integrity check value table (or an authorized portion thereof) to determine whether the correct key domain key was used to access the data. If an incorrect key domain key is used to decrypt the data, the resulting plaintext will be corrupted and will not match the corresponding integrity check value in the integrity check value table.

[0067] In one embodiment, when data is read into a cache line, the data is compressed to provide space for an integrity check value and / or a key domain identifier / selector (i.e., unused address bits are embedded into the cache line). When writing to memory, the key domain identifier / selector can also be included in the compressed data. When reading memory for a compressed data line, the actual unused address bits indicating the key domain are compared with the key domain identifier / selector value embedded in the compressed data cache. If the key domain values match, the data is decompressed and forwarded to the cache. Compression is an integrity optimization that avoids the need to consult an integrity check value table whenever data is accessed in memory. Additionally, compressing the key domain into the cache line alleviates the need for some caches to include the key domain identifier as metadata. Although some embodiments of the present invention may compress data written to a cache line or memory, data compression is not required to implement the present invention.

[0068] If the key domain values do not match when the actual unused address bits indicating the key domain are compared with the key domain identifier / selector value embedded in the compressed data cache, a determination is made as to which key domain is currently authorized. If the address used to read memory corresponds to the current key domain, the data is cleared (i.e., the data bits are set to zero), and a cache eviction of the old key domain address is performed. (Although two key domain addresses alias to the same physical memory location, the cache holds the key domain addresses independently with full address resolution).

[0069] Referring again to Figure 2 , VMI 230 1 and VM2 230 2 each are shown as having their own domain manager (VMMlet) 222 1 and 222 2 . The domain manager VMMlet1 222 1 is shown inside VMI 230 1 , and the domain manager VMMlet2 222 2 is shown inside VM2 230 2 , to indicate that the code for each respective domain manager (VMMlet) is included within the code for the respective VM. When a consumer requests a service that requires virtualization, a code image implementing the functionality of the domain manager (VMMlet) is provided by the cloud service provider to the consumer. The domain manager (VMMlet) image provided by the cloud service provider is incorporated into the consumer's domain (VM) image.

[0070] Owning VM1 230 1Consumers can measure and verify the domain manager (VMMlet) 222 1 code, and then incorporate the VMMlet1 222 1 into the consumer's domain (VM1 230 1 ). By arranging for the consumer's VM to control the entire software stack of the consumer VM image, including the domain manager (VMMlet), the consumer can measure, verify, and trust the image used to instantiate the domain manager (VMMlet) running within the consumer's VM. Ultimately, the consumer creates a memory location - related domain launch image (including the domain manager image) based on physical addresses, encrypts the domain launch image using the consumer's own key domain key, and provides the encrypted domain launch image to the cloud service provider server, which will launch the domain launch image.

[0071] In one embodiment, the consumer creates the encrypted domain launch image in a proven SGX (Intel® Software Guard Extension) enclave on the cloud service provider's server. In this embodiment, the domain launch image is encrypted using the key domain key within the enclave, and the encrypted domain launch image (and any associated ICV value) is written to memory outside the enclave.

[0072] When the cloud service provider receives the encrypted domain launch image (including the domain manager image) from the consumer, the cloud service provider can measure, verify, and trust that the domain launch image encrypted by the consumer contains the same domain manager image provided to the consumer. In one embodiment, the cloud service provider's server hardware provides a mechanism for measuring the domain manager portion of the encrypted domain launch image by the consumer (creating its hash), so that the cloud service provider can then prove that the domain manager image included in the encrypted domain launch image by the consumer is the same as the domain manager image supplied by the cloud service provider (and thus trusted by the cloud service provider). In one embodiment, the hash function for measuring the domain manager image is location - related, such that the domain manager image must be loaded into the correct memory location in the cloud service provider server's memory to be properly decrypted. For example, even if the contents of two different memory locations are the same (e.g., all zeros), only the domain manager image loaded into the correct memory location will produce the expected location - related hash result. The nature of the location - related hash verification function provides the following security advantage: that an adversary attempting to change the behavior of the domain manager image cannot rearrange the encrypted portion of the domain manager image in memory.

[0073] In this collaborative model, the domain manager image is verified by both the consumer and the cloud service provider. The consumer can trust the domain manager image provided by the public cloud service provider and trust that the cloud service provider's hardware will enforce the security and confidentiality of the consumer's virtual machines (VMs). This verification is important for the security of the VMs because the domain manager (VMMlet) has full vmxroot privileges, including the ability to execute commands such as virtual machine control structure (VMCS) save / restore, general-purpose register (GPR) save / restore, and / or vmexit / vmresume. Additionally, the interrupt descriptor table (IDT), advanced programmable interrupt controller (APIC) instructions, and paging data structures (such as page tables and / or extended page tables (ETP)) are encrypted in the key domain. In some embodiments, the domain manager image only includes VM control structures such as VMCS and associated data such as EPT, which control the behavior of the consumer's VMs, but does not include code or data for VMX root operations, which may reside outside the consumer's key domain.

[0074] This collaborative model enables the consumer to trust the privileged software provided by the cloud service provider by moving the measurement and verification to the consumer. The consumer can ensure the security of the consumer's own workloads in the cloud, which is guaranteed by the cloud service provider's server hardware. The cloud service provider can then re-verify that the correct domain manager image is being used. This model greatly simplifies the hardware requirements for providing a truly secure public cloud foundation. No changes are required to the operating system (OS) portion of the virtual machine (VM). Most of the implementation complexity is contained in the design of the domain manager (VMMlet), which is software that can be easily patched, updated, measured, and attested. In one implementation, hardware instructions are used to create key domains, switch between key domains, and verify the contents of the key domains, where the verification is by computing the hash value of the contents of the memory locations corresponding to the key domains and comparing the hash value with the expected hash value for the valid contents of the key domains.

[0075] Referring again to Figure 2 , the processor (which is included in hardware 210) switches between VMs 230 1 and 230 2 and their respective key domains KD1 250 1 and KD2 250 2 in response to commands issued by the memory manager 240, by using the SwitchKD (Switch Key Domain) instruction. Switching from one key domain to another (e.g., from key domain KD2 250 2 to KD1 250 1The result of ) is that the control of specific physical memory aliasing is passed to the VM (230 1 authorized to access the current key domain KD1 250. 1 ) Different hardware key domains accessed via key domain keys prevent the leakage of consumer private data across VMs and even by adversaries who can access the external physical memory manager 240. The key domain identifier / selector (e.g., a part of the physical address) keeps the VM memory areas separated in the cache. In one embodiment, instead of a key domain switch instruction, the VMX root vmlaunch / vmresume instruction will switch the key domain to the key domain of the VMCS identified by the key domain identifier in the address provided by the vmptrld instruction, and the vmptrld instruction loads the pointer to the current VMCS from the address specified in the vmptrld instruction. The vmexit will then switch back to the VMX root key domain or the shared memory area.

[0076] In one embodiment, a portion 212s of the memory 212 is shared and used for communication across the key domain cryptographic boundary. In other words, the shared memory is not encrypted and can be used to transfer information between VMs that would otherwise only be able to access memory locations belonging to the key domain that each specific VM is authorized for. The shared memory is shown to have a physical address with one bit disabled, which is described herein as the "k-bit". The k-bit is used to determine whether the current key domain is used to restrict VM access to memory locations belonging to a key domain (such as one of key domains KD1 250 1 or KD2 250 2 ), or to allow the sharing of unencrypted information across key domains in the shared memory 212s. The k-bit indicates to the CPU whether the key domain indicated in the physical address should be set to the shared key domain (plaintext / !k) or to the currently active key domain (encrypted).

[0077] The above embodiments have been described with respect to a domain manager (VMMlet) for managing virtual machines, although the present invention is not so limited. A similar key domain model can be used to support processes or containers; although there is no corresponding VMM, OS kernel (or microkernel) for similar purposes. Each process or container image in each key domain will have a cooperative OS kernel component (referred to herein as a domain manager or OSlet), which is measured by the cloud service provider. The domain manager (OSlet) responds to memory manager commands, interrupts, scheduling, resource management, etc. in a manner similar to the domain manager (VMMlet).

[0078] Now refer to Figure 3, a block diagram of a cloud service environment according to an embodiment of the present invention is shown. As Figure 3 shown, the network 300 can be used to allow consumers to request services, including virtualization services, from a public cloud service provider. As can be seen, the network 300 can correspond to any type of communication network and can include many different types of computing devices interconnected via a given network, such as the Internet 320.

[0079] The cloud storage device 310 can be provided as part of a data center that includes various computing devices, storage devices, and so on. As an example, the cloud storage device 310 can be a storage device that includes multiple storage components, such as disk, optical, or semiconductor-based storage devices. The cloud storage device 310 can act as a repository for, for example, the master copies of various applications, including a virtual machine monitor (VMM) application that instantiates virtual machines to provide services in response to consumer requests. In Figure 1 the embodiment shown, the master copy of the VMM application is stored in the form of a VMM image 312. The VMM image 312 is a software image that contains a software stack designed to provide a virtual machine platform in the form of a virtual machine monitor (VMM).

[0080] Thus, as further seen in Figure 3 the same location, for example as part of the same data center, one or more public cloud service provider servers, such as the public cloud provider server 315 1 and 315 2 can be coupled to the cloud storage device 310. In various embodiments, the public cloud service provider servers can be used to service consumer service requests, including virtualization requests. For example, each public cloud service provider server can host one or more virtual machines on behalf of a consumer. In Figure 3 the example shown, the public cloud provider server 315 1 hosts two virtual machines VM1 340 1 and VM2 340 2 . Similarly, the public cloud provider server 315 2 hosts two virtual machines VM1 340 3 and VM2 340 4 .

[0081] As Figure 3 shown, there can be various consumer devices, such as the cloud service consumer device 330 1 and 330 2Such a cloud service consumer device can be a personal device of a given user, such as a smart phone, a tablet computer, a desktop computer, and so on. Alternatively, the cloud service consumer device can be a server of an organization that consumes cloud services. Additionally, a cloud service consumer device can be emulated via software. In other words, an emulator or simulator can emulate the hardware of the cloud provider in software so that a consumer can run an emulator of the cloud provider's hardware on the consumer's device.

[0082] Cloud service consumer device 330 1 and 330 2 each provide a corresponding cloud service consumer 331 1 and 331 2 as well as a corresponding VM image 332 1 and 332 2 The cloud service consumer 331 1 and 331 2 can be, for example, a client component of a cloud service application for requesting cloud services. Cloud service consumers, such as cloud service consumer 331 1 and 331 2 are referred to herein as "consumers". The VM image 332 1 and 332 2 can be stored in a storage device (not shown) coupled to the corresponding cloud service consumer device 330 1 and 330 2 . These VM images are provided by the consumers to the cloud service provider and are used to create secure VMs, such as VM1 340 1 , which runs on the cloud provider's server 315 1 .

[0083] When a secure VM has been established on the cloud service provider's server according to the techniques described herein, the consumer can then use the VM, with the consumer's secret key, to create additional VMs on behalf of the consumer. Thus, once a consumer VM can be securely established in the cloud service provider's cloud, the VM can then perform Figure 3 all the operations of the consumer device, including creating additional secure VMs.

[0084] Similarly, a consumer can use multiple cloud service providers to establish secure VMs, and these secure VMs can securely interact via a secure communication channel using the consumer's secret key.

[0085] Figure 4is a diagram that illustrates an apparatus according to an embodiment of the present invention. An apparatus 400 for protecting a public cloud environment according to an embodiment is illustrated. The apparatus 400 can include any computing device and / or data platform, such as a laptop computer, a personal digital assistant (PDA), a media content player, an imaging device, a mobile Internet device (MID), any smart device (such as a wireless smart phone, a smart tablet device, a smart TV), a computer server, etc., or a combination thereof. Additionally, the apparatus 10 can include any platform having the following: computing functionality (such as a personal digital assistant / PDA, a laptop computer, a smart tablet device), communication functionality (such as a wireless smart phone), imaging functionality, media playback functionality (such as a smart TV / TV), etc., or a combination thereof (such as a mobile Internet device / MID).

[0086] The illustrated apparatus 400 includes a memory 412, which can be external to the processor 411 (such as an external memory), and / or can be coupled to the processor 411 via, for example, a memory bus. Additionally, the memory 412 can be implemented as a main memory. The memory 412 can include, for example, volatile memory, non-volatile memory, etc., or a combination thereof. For example, the memory 412 can include dynamic random access memory (DRAM) configured as one or more memory modules, such as, for example, dual in-line memory modules (DIMMs), small outline DIMMs (SODIMMs), etc., read-only memory (ROM) (such as programmable read-only memory (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.), phase change memory (PCM), etc., or a combination thereof.

[0087] The memory 412 can include an array of memory cells arranged in rows and columns, which are partitioned into independently addressable storage locations. Thus, accessing the memory 412 can involve using an address for the storage location, such as, for example: a row address that identifies the row containing the memory location to be stored, and a column address that identifies the column containing the memory location to be stored. Additionally, a device inside the apparatus 400 and / or a device outside the apparatus 400 can perform the access to the memory 412. The access to the memory 412 can involve, for example, direct memory access (DMA).

[0088] The memory 412 can be protected by using encryption and integrity checking. In one embodiment, an encryption technique known as a tweakable block cipher is used. A tweakable block cipher accepts a second input called a tweak, along with the plaintext or ciphertext input to be encrypted. The tweak, along with the key, selects the permutation computed by the cipher. For example, the tweak function can use the physical memory address as a tweak to the block cipher to bind the unencrypted data to the physical memory address. The tweak function 445 can include, for example, an XTS (XOR-Encrypt-XOR) / XEX-based tweaked codebook mode that utilizes a ciphertext stealing algorithm, the Liskov, Rivest, and Wagner (LRW) algorithm, etc., or a combination thereof.

[0089] Regarding the integrity of the memory 412, in one embodiment, the hardware capabilities of memory encryption with integrity are used, which are described in U.S. Patent 9,213,653 B2, "Memory Integrity," hereinafter referred to as the Total Memory Encryption engine with Integrity or TMEi. In another embodiment, memory encryption with integrity is provided by a Memory Encryption Engine (MEE), as described in U.S. Patent 8,819,455, "Parallelized Counter TreeWalk for Low Overhead Memory Replay Protection." However, the present invention is not limited to these implementations, as any cryptographic mechanism that provides memory encryption via a memory location-related ("tweaked") cipher can be used. Additionally, any memory integrity mechanism can be used to enhance the security provided solely by encryption, although a memory integrity mechanism is not required for the implementation of the present invention.

[0090] The processor 411 can include any type of processor, such as, for example, a microprocessor, an embedded processor, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a vision processing unit (VPU), a network processor, a device for executing code to implement the techniques described herein, etc., or a combination thereof. The processor 411 can include one or more cores, such as, for example, core 416 and core 418. Cores 416, 418 can include single-threaded cores, multi-threaded cores where each core includes more than one hardware thread context (or "logical processor"), etc., or a combination thereof. Cores 416, 418 can include instruction decoders for identifying and / or decoding instructions (which, for example, come from an instruction register), activating appropriate circuitry to execute the instructions, verifying that the instruction stream (e.g., the opcode, etc.) will compute, etc., or a combination thereof.

[0091] For example, cores 416, 418 may execute one or more instructions, such as read instructions, write instructions, erase instructions, move instructions, arithmetic instructions, control instructions, etc., or combinations thereof. Cores 416, 418 may, for example, execute one or more instructions to move data (such as program data, operation codes, operands, etc.) between registers (not shown) and memory 412, read data from memory 412, write data to memory 412, perform arithmetic operations (such as addition, subtraction, bitwise operations, comparison, etc.) by using the data, perform control operations (such as branching, etc.) associated with the data, etc., or combinations thereof. Instructions may include any code representation, such as, for example, binary code, octal code, and / or hexadecimal code (such as machine language), symbolic code (such as assembly language), decimal code, alphanumeric code, higher-level programming language code, etc., or combinations thereof. Thus, for example, hexadecimal code may be used to represent the operation codes (such as opcodes) of the x86 instruction set, including: the byte value "00" for an addition operation, the byte value "8B" for a move operation, the byte value "FF" for an increment / decrement operation, etc.

[0092] Processor 411 may include internal storage, such as, for example, including one or more levels of processor cache. The processor cache may not be encrypted and / or may share the same die on the same chip with processor 411. Additionally, the processor cache may be integrated onto one or more of cores 416, 418. The illustrated processor 411 includes cache 413, which may store data (such as instructions, operands, program data, etc.) utilized by one or more components of processor 411. Cache 413 may include any type of cache, such as, for example: instruction cache, data cache, single-level cache, multi-level cache, shared cache, strictly inclusive cache, exclusive cache, etc., or combinations thereof. For example, cache 413 may include an intermediate-level cache, such as a second-level (L2), third-level (L3), fourth-level (L4) or other level of cache, last-level cache (LLC), etc., or combinations thereof. Cores 416, 418 may check whether data is located in cache 413 to execute one or more instructions and / or other data (such as program data, etc.), where a cache miss may cause data to be transferred from memory 412 into a fixed-size block (such as a cache line) of cache 413.

[0093] Each core 416, 418 can be coupled to a respective memory, for example, via a respective memory controller such as memory controller 417, to a shared memory via a shared memory controller, to a respective memory via a shared memory controller, and so on, or a combination thereof. Additionally, a shared cache can be coupled to the shared memory controller, multiple caches can be coupled to multiple respective memory controllers, and so on, and combinations thereof. For example, memory controller 417 can be shared between cores 416, 418, can be coupled to cache 413 (e.g., a shared multi-level cache), and can couple cores 416, 418 to memory 412 (e.g., a shared DRAM). Memory controller 417 can be coupled to memory 412 (e.g., external memory, DRAM, etc.).

[0094] Processor 411 further includes a memory encryption engine 415. The illustrated memory encryption engine 415 includes an encryptor 441 that can encrypt unencrypted data. The unencrypted data can include, for example, plaintext data, cleartext data, and so on, or a combination thereof. The plaintext data can be encoded in a special format (e.g., Hypertext Transfer Markup Language (HTML), Rich Text Format (RTF), etc.) and read by an appropriate program (e.g., a word processor, a text editor, etc.) without decryption. The plaintext data can include pre-encrypted data, such as plaintext data that will be encrypted before transmission and / or storage. Additionally, the plaintext data can include decrypted data, such as data as a result of decryption on the received and / or retrieved data.

[0095] Additionally, the plaintext data can include data encoded in any format, such as audio / video data (e.g., Moving Picture Experts Group (MPEG) data, etc.), image data (e.g., Joint Photographic Experts Group (JPEG) data, etc.), financial data (e.g., Automated Teller Machine (ATM) transaction data, etc.), and so on, or a combination thereof. The plaintext data can include program data, such as at least a part of a program, an operating system (OS), an application, a virtual machine (e.g., hypervisor (VMM) code, etc.), and so on, or a combination thereof. The plaintext data can further include, for example, instructions that include an opcode, operands, and so on, or a combination thereof.

[0096] The unencrypted data can include multiple bits. The multiple bits can include one or more bits represented in any code (such as bytes, etc.), and the any code representation can be, for example, binary code, octal code, hexadecimal code, symbolic code, decimal code, alphanumeric code, high-level programming language code, etc., or a combination thereof. For example, a memory reference instruction can include one bit for an opcode, one bit for an address, etc., and the bits of the memory reference instruction can be represented in hexadecimal code (such as machine language), in symbolic code (such as assembly language), etc., or a combination thereof. Additionally, the multiple bits can be converted to and / or from binary code, where the binary code can be executed by cores 416, 418, can be classified at memory 412, can be retrieved from memory 412, etc., or a combination thereof.

[0097] The encryptor 441 can include any type of cipher for generating ciphertext data, such as, for example, a block cipher in any desired mode of operation. The block cipher can include a fixed block size, where the block cipher can be repeatedly implemented to encrypt data larger than the block size. For example, the block cipher can include the Advanced Encryption Standard (AES) in the Propagating Cipher Block Chaining (PCBC) mode of operation. Additionally, the block cipher can include an expandable block size.

[0098] In one example, the block cipher is Threefish, which can be implemented to obtain an expandable block size of any length (such as 256 bits, 512 bits, 1024 bits, etc.). For example, Threefish can utilize a tweak (such as 128 bits), which can include a memory address and / or location, and a key, where the key can be the same width as the block. Threefish can utilize multiple rounds (such as 72) to encrypt 256-bit and 1024-bit blocks, utilize multiple rounds (such as 80) for 1024-bit blocks, etc. Threefish can utilize the function MIX, which includes an addition operation, a rotation operation by a constant, and an exclusive or (XOR) operation. For example, after each set of MIX functions (such as 2, 4, or 8, corresponding to the block size), the words can be permuted. Sub-keys can be injected into the system, for example, every number of rounds (such as 4), where the sub-keys can be generated by parts of the key, the tweak, and a counter value. At the end, additional words can be given to the key and the tweak (such as the XOR of all other words).

[0099] The illustrated memory encryption engine 415 further includes a decryptor 442 that can decrypt ciphertext data to generate unencrypted data. The decryptor 442 can include the inverse of the encryptor 441. For example, the decryptor 442 can include the inverse of AES-PCBC. Additionally, the decryptor 442 can include the inverse of Threefish. For example, the subkeys can be applied in the reverse order, where each round includes an inverse word permutation, followed by an inverse MIX function. Thus, when unencrypted data is to be stored in the memory 412 (e.g., a write instruction), the unencrypted data (e.g., plaintext data) can be provided as an input to the encryptor 441 to generate an unreadable copy of the unencrypted data (e.g., ciphertext data), where the decryptor 442 can be implemented to decrypt the ciphertext data when the ciphertext data is to be retrieved from the memory 412 (e.g., a read instruction) and generate unencrypted data.

[0100] The memory encryption engine 415 can include a cache line monitor to identify cache lines corresponding to free address aliases that alias with addresses from a plurality of addresses, and flush the identified cache lines. The memory encryption engine 415 can further include an integrity check value selector 443 to determine an integrity check value to apply to unencrypted and / or encrypted data lines (e.g., aliased by at least one of the plurality of addresses). The memory encryption engine 415 can further include a memory initializer to write to a location in the memory without first reading the data previously stored at that location in the memory. The memory encryption engine 415 can include an allocator to assign / bind the flushed cache lines to data line physical addresses.

[0101] The memory encryption engine 415 can further include a cache line interpreter to determine, for each cache line, a data physical memory address, as Figure 31A illustrated, which includes: a data line byte; a data line physical address that includes a full line slot selector and a full line index; and a key domain selector formed from unused address bits of the data physical memory address. The full line index identifies a full line address location in the memory, and the full line slot selector identifies a full line slot in the full line address, where the full line slot value is stored and used to determine whether an address alias is valid.

[0102] The memory encryption engine 415 may further include an aliasing manager for determining the data row physical address for multiple cache lines identifying aliased addresses, where the aliased addresses alias to a single memory location. The memory encryption engine 415 may include an integrity check value calculator for setting the key domain selector of a cache line with a valid integrity value, the valid integrity value being used to label the cache line as a currently valid address alias. The memory encryption engine 415 may include: a data retriever for reading an encrypted data row from the data row physical address of the data physical memory address for the cache line; and a decryptor 428 for decrypting the encrypted data row. The decrypted data row may identify the data row physical address, the full row index, and the full row slot selector (e.g., as illustrated in Figure 31A ). The memory encryption engine 415 may include: a slot value interpreter for reading the full row slot value stored in the full row slot; and a comparator (e.g., integrity validator 444) for confirming a match between the key domain selector at the data physical memory address for the decrypted data (e.g., data row) and the full row slot value. The integrity validator 444 may determine a mismatch / match between the plaintext of the integrity value (e.g., a copy stored in the integrity check row) and the plaintext of the data row (e.g., a copied portion of the data row), which indicates an error degradation or validity of the integrity value and / or the data row. The integrity validator 444 may further compare the hash value of the data with the expected hash value of the data.

[0103] The memory encryption engine 415 and / or the aliasing manager may store aliasing bits (e.g., full row slot selectors, full row indexes, key domain selectors, and / or valid integrity values, or some combination thereof) in a separate location (e.g., aliasing bit cache lines and / or aliasing bit memory locations), separate from the data row bytes, and the memory encryption engine 415, the data retriever, and / or the aliasing manager may retrieve the aliasing bits and compare them with a request (e.g., a request for data identified by the corresponding data row physical address) to ensure that a particular access control policy matches. In the case where the comparison of the aliasing bits with the request fails (e.g., no match results), the memory encryption engine 415 and / or the aliasing manager may report (e.g., issue a warning) the no-match condition as one or more of an error or a fault.

[0104] The memory encryption engine 415 data retriever (or cores 416, 418) may read an encrypted data row from a data row physical address of a physical memory address of data for the at least one cache line of the plurality of cache lines. The decryptor 442 may decrypt the encrypted data row, wherein the decrypted data row identifies a data row physical address, a full row index, and a full row slot selector for the decrypted data row. A comparator (e.g., integrity verifier 444) may identify no match between the stored full row slot value and a key domain selector for the data physical memory address of the decrypted data row, and the memory encryption engine 415 and / or the comparator in response to the no match identification may cause the memory encryption engine 415 or its components to flush the cache line and report the no match condition as one or more of an error or a fault.

[0105] The memory encryption engine 415 may additionally include an integrity value embedder for embedding data row bytes for each cache line having a valid integrity value for a data physical memory address. The memory encryption engine 415 may also include a compressor to compress the data row bytes embedded with the valid integrity value. The encryptor 441 may encrypt the compressed data row bytes embedded with the valid integrity value. The memory encryption engine 415 may additionally include a data row writer for writing to a location in memory identified by a data row physical address: the valid integrity value for a key domain selector, the data row physical address, and the encrypted and compressed data row bytes embedded with the valid integrity value.

[0106] The memory encryption engine 415 and / or the compressor may determine that the data row bytes of a particular cache line are not compressible, and instead of attempting to embed aliased bits (e.g., full row slot selector, full row index, key domain selector, and / or valid integrity value, or some combination thereof) into the data row having the data row bytes, may store the valid integrity value separately (e.g., in a separate location, such as another cache line and / or memory location).

[0107] When retrieving the illustrated ciphertext (e.g., read operation) discussed herein from the memory 412, the ciphertext can be decrypted to generate unencrypted data. The illustrated memory encryption engine 415 can further include a tweak function 445 that uses the physical memory address as a tweak to a block cipher to bind the unencrypted data to the physical memory address. The tweak function 445 can include, for example, a tweaked codebook mode based on XTS (XOR-Encrypt-XOR) / XEX that uses a ciphertext stealing algorithm, the Liskov, Rivest, and Wagner (LRW) algorithm, etc., or a combination thereof. The tweak function 445 can, for example, expand the original physical memory address, take the XOR (exclusive or) of the address with the unencrypted data, and run the result through the encryptor 441 with a key to bind the unencrypted data to the address.

[0108] The illustrated memory encryption engine 415 can further include a decoder 447 that decodes the unencrypted data and identifies one or more instructions. For example, when an entire data row (e.g., a 64-byte cache line) is retrieved from the memory 102 and decrypted, the unencrypted data without degradation (e.g., valid plaintext) can contain an opcode. Thus, when the decoder 447 decodes the plaintext data, the decoder 447 can identify the instruction set, an opcode such as, for example, the x86 instruction set, etc.

[0109] The illustrated memory encryption engine 415 can further include a key / tweak value selector 448 that selects a key from a plurality of keys (e.g., a key domain) and / or a tweak from a plurality of tweaks (e.g., a tweak domain) for a physical location in the memory 412. For example, the illustrated memory encryption engine 415 can include a function detector that determines when a function (e.g., a program, middleware, an operating system, firmware, a virtual machine, a VMM, an operating system (OS) kernel, etc.) or a part of a function (e.g., a part of a program, etc.) is first launched or first given access to a physical location in the memory 412. When the function (and / or its part) is given access, the key / tweak value selector 448 can, in response, select a key and / or a tweak (e.g., a key from a key domain, a different key from the same key domain, a different key from a different key domain, a tweak from a tweak domain, a different tweak from the same tweak domain, a different tweak from a different tweak domain, etc.) for the physical location in the memory.

[0110] The key / fine-tuning value selector 448 can select a key based on a value that is determined according to a bit of the physical memory address for a data row, such as an unused address bit. The key domain for a particular physical memory location can be defined by multiple unused address bits that are selected to determine the value. For example, a particular physical memory location may once belong to a particular key domain, where the unused address bits can be used to define the key domain (e.g., a key domain including 16 keys for a single physical memory location that utilizes four unused address bits). Thus, based on the domain mapped to by the location, a physical memory location can use different keys at different time points. The key / fine-tuning value selector 448 can derive a key, for example, by encrypting a value (e.g., 0001, 0010, etc.) using a secret master key that can be protected by the device 400 (e.g., in a trusted execution environment). Additionally, the key / fine-tuning value selector 448 can derive a key, for example, by retrieving a key from an array of protected keys using the value as a pointer to the array.

[0111] In addition, the key / fine-tuning value selector 448 can select a fine-tuning by setting a bit of the physical memory address that will be used by the fine-tuning function 445 as fine-tuning. In this regard, the fine-tuning for the XTS mode will include the unused address bits and the used address bits of the physical memory address. Thus, when the key / fine-tuning value selector 448 selects / changes the unused address bits, different ciphertexts will result from different addresses used for fine-tuning (even though the same physical memory location is actually involved).

[0112] The illustrated memory encryption engine 415 also includes logic 449 that can utilize components of the processor 410, such as, for example, cores 416, 418, encryptor 441, decryptor 442, etc., to maintain (e.g., ensure, verify, test, etc.) the security and integrity of the memory 412.

[0113] When components (e.g., internal or external devices, accelerators, etc.) access the memory using an address that can be related to a particular key domain or aliasing and fine-tuning, memory degradation caused by these components can be detected. These devices can use the current and correct address for accessing the memory. Similarly and conversely, software that degrades the memory of such devices can also be detected when an incorrect or non-current address is used.

[0114] Although not shown in Figure 4is illustrated, but apparatus 400 may include other elements on a chip having a processor 411. For example, processor 411 may include input / output (IO) control logic integrated with a memory encryption engine 415. Additionally, apparatus 400 may include, for example, an IO module, sometimes also referred to as the south bridge of a chipset, which acts as a host device and may communicate with, for example, the following: front / back image sensors (such as two-dimensional cameras, three-dimensional cameras, etc.), microphones, displays (such as screens), motion sensors (such as accelerometers, gyroscopes, etc.), mass storage devices (such as hard disk drives / HDDs, optical discs, flash memories, etc.), network interfaces for providing a variety of communication functionalities (such as cellular phones, WiFi, WiMax, Global Positioning System (GPS), spread spectrum (such as 900 MHz), other radio frequency (RF), etc.). Processor 14 and the IO module may be implemented, for example, as a system on a chip (SoC).

[0115] Additionally, although the examples have shown separate components for illustrative purposes, it should be understood that one or more of the components of apparatus 400 may be combined, may reside in the same and / or different physical and / or virtual locations, etc., or combinations thereof. For example, logic 449 may include one or more of the components of memory encryption engine 415 to perform its corresponding functionality, and the logic 449 may reside in the same or different location as cores 416, 418, memory 412, etc., or combinations thereof. Additionally, one or more components of memory encryption engine 415 may be implemented with computer program code, such as a software value selector, which may interface with one or more components of memory encryption engine 415 implemented with logic hardware.

[0116] Some of the functionality provided by apparatus 400 may be delivered by a system on a chip (SoC) IP block on the memory / DRAM (dynamic random access memory) side of a processor cache, such that the functionality can be used by software running on a host processor (such as a central processing unit / CPU) core, as well as on other IP blocks and accelerators, such as general-purpose graphics processing units (GPGPUs) and integrated graphics (such as Intel® Processor Graphics).

[0117] The illustrated apparatus 400 uses unused physical address bits (and / or other metadata passed through the cache) to manipulate a cryptographic memory integrity value pointer (e.g., to implement one or more access control policies), such that a software memory allocation routine can control the assignment of the pointer (e.g., "allocate memory" and "free"). The apparatus 400 can generally use the unused address bits as a key domain. For example, there may be less external physical memory installed in the system than can actually be addressed by a 64-bit physical memory address, so the most significant address bits can be used to select between different "key domains" because the cache can still deliver these addresses fully resolved in terms of the physical memory address to the apparatus 400. The illustrated apparatus 400 can use 5-level paging for virtual memory, and 64-bit addressing to allow a software memory allocator / manager (e.g., Figure 2 the memory manager 240) to select between different address values that alias to the same physical memory location. The software memory allocator / manager can control the integrity value table (or an authorized portion thereof) to determine which aliases are currently valid, such that software use of an invalid alias / address can then raise a fault in hardware, which can be reported to a software monitor in response to a memory violation.

[0118] Figure 5 is a flowchart of a method performed by a consumer of a cloud service, according to an embodiment of the present invention. In the "Request service from cloud service provider" box 502, the consumer requests a service from the cloud service provider. For example, the request can be for a virtualization service, or the request can be to perform a transaction, for which the cloud service provider will set up a virtual machine or other process to perform the transaction.

[0119] The cloud service provider identifies a server or group of servers that can have a key domain to service the consumer's request. In the "Receive domain manager image and memory location-related address information from cloud service provider" box 504, the consumer receives from the cloud service provider a domain manager image and memory location-related address information, which is also referred to herein as fix-up variable information. The memory location-related address information specifically identifies the physical location in the memory of the server(s) that service the consumer's request. This memory location-related address information can include, for the server(s) that service the consumer's request: the physical address of a page in memory, the physical address for a page table, control register information (e.g., CR3 value), interrupt descriptor table register information, and so on. The domain manager image can contain the page table structure(s) that map the linear / virtual address of the domain manager image to the physical address at which the domain manager image will reside in the memory of the cloud service provider's server.

[0120] Control then passes from the "Receive domain manager image and memory location related address information from cloud service provider" box 504 to the "Measure domain manager image" box 506, where the consumer measures the domain manager image to ensure that the domain manager image has not been compromised. The consumer can verify the domain manager image by using known whitelist techniques, such as calculating the hash of the domain manager image and comparing the hash value with the master hash value for the master domain manager image (which is known to be undegraded); the source code can be inspected and recompiled into a matching image; the government certification of the image can be verified; it can be confirmed that the image is consistent with open source software, etc. If the image will not leak consumer data, the image is considered trustworthy. For example, if all communications are secured by using the consumer's secret key, and files / memory pages are encrypted and integrity verified when saved and / or restored to / from the storage device, the image can be considered trustworthy.

[0121] From the "Measure domain manager image" box 506, control passes to the "Verified " decision point 508. If the domain manager image is not verified, control passes to the "Error" box 522, where the consumer disposes of the situation where the cloud provider's domain manager image has not been verified. In such a situation, the consumer can choose not to use the services of that particular public cloud service provider.

[0122] If the domain manager image is verified at the "Verified " decision point 508, control passes to the "Create domain launch image" box 510. At box 510, the consumer creates a domain launch image, which will be executed on the cloud service provider's server to "launch" the key domain. Launching the key domain can include, for example, creating a key domain such that the hardware uses the key domain key to encrypt data stored in the memory locations belonging to the key domain, and storing data (such as code that will be executed to initially establish the key domain) in the memory locations belonging to the key domain.

[0123] In one embodiment, the consumer uses the memory location - related address information provided by the cloud service provider in box 404, "Receive domain manager image and memory location - related address information from cloud service provider", to modify the provider - supplied domain manager image as part of the code to be executed to launch the key domain. For example, the consumer can modify the page table of the domain manager image such that the physical addresses in the page table are updated (trimmed) taking into account the physical memory address where the domain manager image will be located. Once the paging structure is updated, all linear / virtual addresses used by the code, data, and programs of the domain manager image will map to the correct corresponding physical memory addresses on the cloud service provider's server. In one embodiment, the consumer encrypts the trimmed domain manager image using the consumer's key domain key and creates an integrity check value (ICV) for the encrypted trimmed domain manager image using the consumer's key domain key.

[0124] In one embodiment, the consumer creates a domain launch image that includes the encrypted trimmed domain manager image for distribution to the cloud service provider's server. The consumer also includes the secret key in the domain launch image for use in paging, migration, attestation, communication, and other functions provided by the execution of a domain process (such as a VM, OS, etc.). When the domain launch image is encrypted, the corresponding page table structure included within the domain launch image is also encrypted.

[0125] Since the domain launch image is encrypted (and integrity - checked) using a memory location - related "fine - tuned" cipher, an adversary cannot move parts of the domain launch image around in memory. The page table maps the programs and data of the domain launch image to the correct physical memory addresses on the cloud service provider's server, so the program behavior cannot be maliciously altered considering that the domain launch image is cryptographically bound to the correct physical memory location. In other words, if the domain launch image is not loaded into the correct physical memory location on the cloud service provider's server, the domain launch image cannot be correctly decrypted. Additionally, the integrity check value can detect any attempt to modify either the domain launch image content and / or the location in memory where the domain launch image is loaded.

[0126] Control passes from box 510, "Create domain launch image", to box 512, "Verify the certificate of the server / group capable of having a key domain and obtain the public key of the server / group capable of having a key domain".

[0127] In box 512, the consumer verifies the certificate of the identified cloud service provider server / group and obtains the public key of the identified server / group capable of having a key domain.

[0128] Control is passed from box 512 to the "Exchange key domain key with (a) server(s) verified to be key domain capable" box 514. The consumer exchanges a key domain key with the (a) server(s) verified to be key domain capable in box 512. One aspect of the key domain key exchange is that the key domain key is provided directly by the consumer to the hardware of the server(s) capable of having a key domain only in encrypted form (such as Figure 4 the memory encryption engine 415). Since the software of the server(s) capable of having a key domain does not receive the key domain key, the software of the server(s) capable of having a key domain cannot decrypt the content of the key domain without requesting the hardware to perform decryption. In one embodiment, the consumer uses the public key of the server / group obtained in box 512 to encrypt the consumer's key domain key and then provides the encrypted key domain key to the hardware of the server(s) capable of having a key domain.

[0129] In another embodiment, the key domain key can be negotiated between the consumer and the server hardware. The key domain key can be generated directly using the hardware (such as microcode, firmware, CSME, SMM), where the server hardware can provide its unique (or group) identifier and public key / CERT, and then the Diffie Hellman key exchange (or RSA) can complete the key domain key exchange with the consumer. This embodiment requires the consumer to be online to perform the key exchange when the domain image is launched.

[0130] This key exchange enables the virtual machine running on the verified key domain capable server to access the domain launch image data encrypted with the consumer's key domain key without exposing the key domain key itself. The encrypted message is passed through the software stack of the cloud service provider on the key domain capable server. The key domain capable server hardware provides a cryptographic endpoint for commands. For example, the consumer can use the public key of the server to encrypt the key it uses for the key domain and send the encrypted key domain key to the cloud service provider. The cloud service provider can then issue instructions on the key domain capable server hardware, such as a "Create Key Domain (CreatKD)" instruction, to create a new key domain. Additionally, the provider can also use the same "Create Key Domain (CreateKD)" instruction to recreate the key domain, for example if the VM has been suspended and will be resumed.

[0131] Control passes from the "Exchange key domain key with (a) server(s) verified to have a key domain" box 514 to the "Encrypt the boot image including the domain manager image for the server(s) with a key domain for which the key domain key has been exchanged" box 516. Once the key domain key is established (or before, as it is the consumer's key), the consumer uses the key domain key to encrypt the domain boot image including the domain manager image for the particular server(s) with which the consumer has exchanged the key domain key. Given the address information related to memory location provided by the cloud service provider as trim variable information, the consumer encrypts the domain boot image. In one embodiment, an encryption technique known as a tweakable block cipher is used. A tweakable block cipher accepts a second input called a tweak, along with the plaintext or ciphertext input to be encrypted. The tweak, along with the key, selects the permutation computed by the cipher. In encrypting the consumer's domain boot image, the physical memory address of the server with a key domain is used as the tweak, such that the resulting encrypted boot image is memory location related. The encrypted boot image is described as memory location related because the encrypted boot image must be loaded into the correct physical memory address of the cloud service provider's server before it can be correctly decrypted.

[0132] In one embodiment, the domain boot image is encrypted by using an XEX-based tweaked codebook mode with ciphertext stealing (XTS). The consumer encrypts the domain boot image in a memory location related XTS mode by using the page address tweak and the key domain key. The correct physical address where the domain boot image is to be loaded is included in the XTS tweak for each encrypted block. Other tweakable ciphers, such as Liskov, Rivest, and Wagner (LRW) or counter mode ciphers, may also be used in other embodiments.

[0133] The consumer may also compute an integrity check value (ICV, such as a keyed hash message authentication code (HMAC)) for the domain image by using the key domain key. In one embodiment, the integrity check value is also memory location related such that the address / memory location of the corresponding data row in memory is considered in verifying the integrity of the data. In a situation where the consumer knows the address location of the ICV table on the server corresponding to the consumer's encrypted boot image, the consumer may include the ICV value in the encrypted boot image. The ICV value table may also be encrypted by using a tweak that indicates the correct server memory address for the ICV table, by using the key domain key. The cloud service provider server will then load the ICV portion of the encrypted boot image into the correct slot of the ICV table at those same server memory addresses for the ICV table.

[0134] From the "Encrypt the boot image including the domain manager image for a server capable of having a key domain that has exchanged a key domain key" box 516, control passes to the "Establish a key domain using a server capable of having a key domain" box 518. In box 518, the consumer sends a request to create a key domain to a server capable of having a key domain. The request may include an encrypted key domain key that is used as an input value for the "Create Key Domain (CreatKD)" instruction to be executed by the processor of the server capable of having a key domain. The key domain selector / identifier to be used is a local determination made by the memory manager of the cloud service provider, as the memory manager of the cloud service provider needs to manage a limited key domain namespace. The consumer does not need to know the key domain selector / identifier, and the value of the key domain selector / identifier can be changed by the cloud service provider to avoid local conflicts. The actual key domain key provides security for the consumer's VM image, while the key domain selector / identifier tells the hardware of the cloud service provider's server in which slot / register the key domain key is currently locally stored.

[0135] From box 518, control passes to the "Send the encrypted domain boot image to a server(s) capable of having a key domain" box 520. The encrypted domain boot image is sent to the cloud service provider, and the software stack of the cloud service provider on the server capable of having a key domain loads the domain boot image into memory at the correct physical memory address (i.e., into the k-bit off (i.e., unencrypted) area of memory).

[0136] Figure 6 Is a flowchart of a method performed by a cloud service provider according to an embodiment of the present invention. Control begins at the "Provide a domain manager image to the consumer in response to the consumer's request for a service" box 602. The consumer's request may specifically be for a virtualization service, or the consumer's request may be to perform a transaction that the cloud service provider will perform for the consumer via a virtual machine or other process.

[0137] Control proceeds from the "Provide the domain manager image to the consumer in response to the consumer's request for a service" box 602 to the "Allocate space for the domain manager image and provide memory location-related address information to the requesting consumer" box 604. In this box, the cloud service provider allocates space in memory for the domain manager image and notifies the requesting consumer of the memory location-related address information for the allocated memory space. The memory location-related address information can in particular include the physical address of a page in memory, the physical address for a page table, control register information, interrupt descriptor table register information, and so on. The memory location-related address information can also include the expected entry point. As an alternative embodiment, the cloud service provider can create a trimmed domain image that the consumer can re-verify as correct.

[0138] As mentioned in the "Request a service from the cloud service provider" box 502 referenced above Figure 5 the cloud service provider can identify a group of servers that can provide key domain capabilities. For example, each server in the server group can use the same key, which is referred to as a group key, such as a group public verification key for direct anonymous attestation / enhanced privacy identifier (DAA / EPID). DAA is a digital signature algorithm that supports anonymity. Unlike traditional digital signature algorithms in which each entity has a unique public verification key and a unique private signature key, DAA provides a common group public verification key associated with many (typically millions) unique private signature keys. DAA was created such that a device can prove to an external party what kind of device it is (and optionally what software is running on the device) without providing the device's identity, i.e., prove that the device is a trusted member of the group without revealing which member. EPID enhances DAA by providing the following additional utility: given a signature created by a private key, the key can be revoked even if the key itself remains unknown.

[0139] From box 604, control proceeds to the "Exchange the key domain key with the consumer" box 606, where a server that can have a key domain obtains the key domain key from the consumer. The key domain key is provided by the consumer as an encrypted key, where the consumer's key domain key has been encrypted using the public key of the server that can have a key domain. In one embodiment, the memory manager of the server that can have a key domain causes the encrypted key domain key to be written to a slot / register of the server that can have a key domain, and a memory encryption engine (such as Figure 4 the memory encryption engine 415) reads the encrypted key domain key from the slot / register and decrypts the key domain key by using the private key of the server that can have a key domain.

[0140] From block 606, control proceeds to block 608, "Load the domain launch image into the allocated space in memory in response to the consumer supplying a domain launch image." When a consumer supplies a VM workload to a server capable of having a key domain, the consumer supplies a domain launch image that is encrypted using the consumer's key domain key. The server capable of having a key domain loads the domain launch image into the physical memory space allocated in block 604. The domain launch image is installed in the physical memory on the cloud service provider's server at the physical memory location that is transmitted to the consumer via memory location-related address information. A shared, unencrypted memory location can be made available (e.g., by the cloud service provider using a portion of the physical address, such as k bits) for initially loading the encrypted launch image in memory.

[0141] Since multiple servers can share the same public key, identifying memory location-related address information may require resolving memory conflicts among multiple servers. In one embodiment, memory location conflicts among multiple servers in a group are resolved because the image located in the server's memory that is related to the domain launch is the consumer's domain launch image, which can be transient. That is, the domain launch image is used to launch the consumer's larger domain (VM) image, which can be paged anywhere in the memory selected by the cloud service provider. After the consumer's larger domain image has been launched, the portion of the encrypted image that is related to the location can be removed from the memory (where the function of launching the consumer's larger domain image has been performed). Thus, memory usage can be managed by the cloud service provider, which makes space for the location-related launch image, uses the location-related launch image to launch the remainder of the domain image into variable memory, and then releases the space occupied by the domain launch image (e.g., makes space for different domain launch images of different key domains that happen to overlap those same memory locations).

[0142] The consumer's domain image can be launched in multiple phases under software control. The first phase is to execute the domain launch image, which is encrypted by the consumer based on memory location-related address information. The second phase is to launch the remainder of the consumer domain image, which is not required to be loaded into a specific physical memory location on the cloud service provider's server.

[0143] From block 608, control proceeds to the "Create and Initialize Key Domain in Memory" block 610. In one embodiment, a server capable of having a key domain receives a request from a consumer to create a key domain. The request may include an encrypted key domain key, which may be used as an input value for a "Create Key Domain (CreatKD)" instruction to be executed by the server capable of having a key domain. The CreatKD instruction may also initialize a new key domain by silencing the processor core, flushing the cache and translation lookaside buffer (TLB) of the old key domain, and initializing the memory encryption engine with the new key for the key domain. Initializing the memory encryption engine with the new key domain key may include writing the key domain key to a memory slot / register accessible by the memory encryption engine hardware. Alternatively, these initialization functions may be performed via a separate "Initialize Key Domain ((InitKD)" instruction.

[0144] From block 610, control proceeds to the "Measure Domain Boot Image" block 612. The cloud service provider verifies that the expected domain manager image is present within the consumer's encrypted domain boot image. This verification ensures that privileged code, such as the VMX root components and data structures, is included in the consumer's encrypted domain boot image.

[0145] In one embodiment, the memory manager uses a Hash Key Domain (HashKD) instruction to verify that the pages of the domain boot image contain the provider's domain manager (VMMlet) image. A "secure hash" function, such as the Secure Hash Algorithm 3 (SHA3) defined by the National Institute of Standards and Technology (NIST), is used to compute a hash value for the provider domain manager image within the encrypted domain boot image. The secure hash algorithm transforms data by using a hash function, which may be an algorithm including bitwise operations, modular addition, and compression functions. The hash function then produces a fixed-size string that looks nothing like the original input string. These algorithms are designed to be one-way functions, meaning that once the original input data has been transformed into a hash value, it is practically impossible to transform the hash value back into the original input data.

[0146] By constructing the domain manager image from the consumer's encrypted domain boot image in local memory, the cloud service provider can verify that the domain manager image is present within the consumer's domain boot image. The cloud service provider can then perform the same verification function (i.e., the hash function) that the HashKD instruction uses on the contents of the local memory location for the constructed domain manager image. If the verification function (hash) value of the contents of the local memory location matches the result of the HashKD instruction, the cloud service provider can ensure that the provider's domain manager image is correctly incorporated as part of the consumer's encrypted domain boot image.

[0147] In one embodiment, the HashKD instruction can provide a hash value for a cache line, or in another embodiment, the HashKD instruction can provide a hash value for up to one page of memory at a time.

[0148] In one embodiment, the HashKD instruction only provides a hash value such that the consumer secrets in the domain launch image are not revealed to the cloud service provider. The consumer's secrets can be in the VMX non-root part of the domain launch image, for example as part of the operating system running above the domain manager (VMMlet). Providing only the hash value as a result of the HashKD instruction enables the cloud service provider to verify only the provider part (domain manager image part) of the encrypted domain launch image. Verifying the provider part independently of the part of the encrypted domain launch image modified by the consumer (including the consumer secrets) prevents the disclosure of the consumer's secrets to the cloud service provider.

[0149] From block 612, control proceeds to the "verified " decision point 614. If the domain launch image measurement is not verified, control proceeds to the "error" block 626, where the cloud service provider can report the verification failure to the consumer. If the image measurement is verified at the "verified " decision point 614, control proceeds to the "execute the consumer's domain launch image and verify the entry point" block 616.

[0150] At the "execute the consumer's domain launch image and verify the entry point" block 616, the stack of the server with the key domain available will execute the consumer's domain launch image at the expected entry point (as provided to the consumer via memory location related address ("trim" variable) information). The memory manager VM loads the consumer-encrypted domain launch image into an encrypted memory page (where k bits are disabled). A new key domain is initiated.

[0151] In one embodiment, the processor of the server with the key domain available executes a "Switch Key Domain" (SwitchKD) instruction, providing as input a destination key domain identifier / selector, an entry point address, and control register information. Additionally, in one embodiment, a keyed hash message authentication code (HMAC) calculated by the consumer (e.g., by using the key domain key or its derivative) is used to verify that the entry point address and control register information are correct.

[0152] Before launching an image in the execution domain, a server in the key domain can turn off interrupts. In one embodiment, the first instruction executed after switching the key domain is a special class of ENDBRANCH instructions that identify the expected entry point for the key domain switch. After the ENDBRANCHKD instruction, the destination domain manager (VMMlet) code verifies that the VMM is in protected mode. The destination domain manager (VMMlet) code also verifies that the control registers and interrupt descriptor table registers, etc. are correct. The destination domain manager (VMMlet) code then restarts the interrupt and resumes execution from the saved state.

[0153] In one embodiment, the SwitchKD (Switch Key Domain) instruction is implemented by using the HMAC function to verify the consumer's domain launch image. This implementation is the preferred embodiment of SwitchKD (Switch Key Domain) because it is the most flexible. The consumer can use a secret established using the server hardware, such as the key domain key or its derivative, to compute an HMAC (e.g., SHA3 HMAC) on the expected processor state for entering the key domain (e.g., verifying the processor's registers for the instruction pointer, stack pointer, CR0, CR3, CR4, IDTR, GDTR, LDTR, any MSR that can affect VM security, etc.). The HMAC implementation of the SwitchKD (Switch Key Domain) instruction can be dynamically established by the consumer's domain image, and multiple entry points can be supported by computing multiple HMACs, one for each unique valid entry point into the consumer's domain image. This flexibility of dynamically defining new entry points by using HMAC allows the server to start with the original encrypted domain launch image, execute the original encrypted domain launch image at a fixed initial entry point, and then internally (from within the key domain) copy the domain launch image to a newly dynamically assigned memory location (according to the provider's memory management policy), as well as the new entry point location established for the newly dynamically assigned memory location. Now, the original domain launch image, and the static memory location to which the original domain launch image is cryptographically bound, can then be released by the cloud service provider, leaving only the dynamically re-assigned VM image in memory, at the location dynamically defined by the provider's memory manager software. In this way, even if multiple initial launch images for different consumers happen to overlap in memory, they can be loaded sequentially, transferred to dynamic memory locations, and the memory locations of the domain launch images are freed up for the next consumer's domain launch image, etc., where each executed domain image recomputes the HMAC by using the consumer key domain key for the new entry point when creating each dynamic image.

[0154] Alternatively, when creating a new key domain (CreateKD), the entry point values (instruction pointer register, stack pointer register, control register, interrupt descriptor table register, etc.) can be established by the consumer using a server capable of having a key domain and verified by the cloud service provider.

[0155] When a server capable of having a key domain executes the domain launch image, the page table is referenced by the control register of the processor (i.e., CR3), which specifies the physical address of the root for the page table structure. When switching into the key domain, the control register must be set to the correct value. In one embodiment, the SwitchKD instruction includes a key hash parameter, such as SHA3 HMAC. The key hash parameter is used to ensure that when the domain launch image is executed, the correct page table structure within the image is used by the processor of the cloud service provider's server (and thus all memory mappings are correct). The key hash parameter is used to confirm that when entering the domain launch image, the server processor state of the cloud service provider is correct, as the processor will verify the key hash parameter (HMAC) against the processor control register state, instruction pointer, stack pointer, etc. of the cloud service provider's server.

[0156] From the "Execute the consumer's domain launch image and verify the entry point" box 616, control continues to the "Load the remainder of the consumer's domain image into memory" box 618. A server capable of having a key domain loads the remainder of the consumer's domain image into memory. The remainder of the consumer's domain image can include, for example Figure 25 the remainder of the domain image 2532, including the operating system(s), application(s), script(s), or other code.

[0157] From the "Load the remainder of the consumer's domain image into memory" box 618, control then continues to the "Verify additional pages of the consumer domain image by using the secret key included in the domain launch image" box 620. The running and verified domain image can now verify additional pages of the consumer's domain image by using the secret key from the domain launch image. For example, the domain launch image can include secret keys for paging, migration, attestation, communication, and other functions.

[0158] From the "Additional page for verifying the consumer's domain image by using the secret key included in the domain launch image" box 620, control proceeds to the "Perform secure operations within the consumer key domain" box 624. Once the consumer domain (VM) image has been properly executed and the corresponding key domain has been switched, the domain manager can complete loading the operating system and request additional resources (memory pages, I / O resources, etc.) from the cloud service provider's memory manager. Save and restore memory operations (involving, for example, VM control structures, control registers, etc.) stay within the key domain and are directly executed by the memory encryption engine hardware and are not exposed to the cloud service provider. Since the domain manager image originates as the cloud service provider's software, once verified, the executing domain manager will obey the memory manager commands and cooperate with other domain managers. Additionally, like a normal VMM, the domain manager will protect the server's hardware and resources from the less privileged code of the consumer domain, such as the operating system, applications, etc.

[0159] Figure 7 is a diagram that shows the components of a consumer domain image (e.g., a consumer VM image) according to one embodiment of the present invention. The consumer domain image 710 includes a static provider-supplied domain manager portion 712 and a dynamic consumer-supplied portion 714. In one embodiment, the static provider-supplied domain manager portion 712 corresponds to a domain manager (VMMlet), which is privileged code that instantiates and manages the consumer virtual machine. The static provider-supplied domain manager portion 712 can also issue commands to the hardware of the cloud provider service to create a key domain, thereby providing the consumer's encrypted key domain key to be used to encrypt memory locations belonging to the newly created key domain. The static provider-supplied domain manager portion 712 can also issue commands to the hardware of the cloud provider service to switch to a different key domain, thereby providing the consumer's encrypted key domain key to be used to control the key domain to be switched to. The virtual machine managed by the domain manager (VMMlet) can then be made to operate within the currently active key domain. The privileged code given by the domain manager (VMMlet) can be measured and verified by the consumer, enabling the consumer to trust the privileged code given by the domain manager (VMMlet) as part of its trusted computing base.

[0160] To establish a consumer domain image 710 in the memory of a cloud provider server, the consumer creates an encrypted domain launch image that is executed in the memory of the cloud provider server. The domain launch image can contain only the basic code needed to do the following: (1) cause the cloud service provider server hardware to create a new key domain or switch to an existing key domain within the memory of the cloud service provider server, and (2) cause some baseline code to operate within that key domain. For example, the domain launch image can create a new virtual machine or cause an existing virtual machine to access data within a memory location of the key domain established by the portion of the code provided in (1).

[0161] The domain launch image is created by the consumer because it will appear in a specified memory location in the memory of the cloud service provider server. For example, by using a memory location - related password to cryptographically bind the domain launch image to the specified memory location in the memory of the cloud service provider server, the consumer can encrypt the domain launch image using the consumer's key domain key. Once the encrypted domain launch image is loaded into the memory location specified by the memory location - related password, the existing encrypted domain launch image can then bootstrap to dynamically load additional domain image code (such as the portion 714 code dynamically supplied by the consumer) into the consumer's domain image 710. In one embodiment, the portion 714 dynamically supplied by the consumer corresponds to the less privileged code of the consumer domain image, such as the operating system, applications, and so on.

[0162] In one embodiment, the consumer's encrypted domain launch image includes at least code privileged by a domain manager (VMMlet). In at least one embodiment, the consumer's encrypted domain launch image also includes some consumer - supplied code.

[0163] Since the domain launch image is encrypted by the consumer in the consumer's own environment using the consumer's key domain key, the encrypted static portion 712 that is executed can be described as being made to operate "outside" the key domain. Since only the consumer knows the key domain key, the cloud service provider cannot create, add code to, or modify the consumer's encrypted domain launch image without degrading the consumer's encrypted domain launch image.

[0164] Once the code included in the consumer's domain launch image begins to execute on behalf of the consumer within the key domain, the executed consumer's domain launch image code can take over and extend the consumer's domain image 710. Extending the consumer's domain image 710 includes, for example, dynamically adding new code (such as the dynamically supplied portion 714 of the consumer) to the consumer's domain image 710. Using a protocol determined by the consumer (e.g., the consumer's domain image 710 can be extended only after verifying the new extended code segment), new code can be added to the consumer's domain image 710 from within the key domain, and / or modifications can be made to the consumer's domain image 710.

[0165] When the consumer's domain image 710 is written to memory from within the key domain, the data from those memory write operations is encrypted and fine-tuned by the memory encryption engine using the memory address. Read and write operations performed from within the key domain are thus location-dependent because they are created from code executed within the key domain. Such operations can be described as being made "inside the key domain" by the memory encryption engine. In other words, cloud service provider software executed outside the key domain cannot modify or rearrange this dynamically created portion of the consumer domain image.

[0166] In one embodiment, a consumer domain image that has been dynamically extended can be converted into a static version of the consumer domain image. For example, when the execution of a virtual machine instantiated from the consumer domain image has been suspended and will be resumed, the conversion from a dynamic to a static consumer domain image can be performed. A copy of the dynamic consumer domain image can be captured when the virtual machine is suspended, the copy of the dynamic consumer domain image can be flushed to memory, and the ciphertext bound to the address from memory can be saved. The consumer can recompute any integrity check values associated with the memory address and recreate the consumer domain image to incorporate those integrity check values. When the virtual machine is to be resumed, the recreated consumer domain image can be relaunched as a static consumer domain image.

[0167] As referenced Figure 5 and 6 As described, the encrypted domain launch image created by the consumer includes a consumer domain manager (VMMlet) image, which is a modified version of the domain manager (VMMlet) image supplied by the cloud service provider. The provider-supplied domain manager (VMMlet) image is modified to incorporate memory location-related address information for the specified server of the cloud service provider. The consumer domain manager image is statically bound to the specified server and the memory address of the specified server, meaning that the consumer domain manager image must be installed and executed at the specified memory address of the specified server in order to function properly.

[0168] The cloud service provider executes the encrypted domain launch image of the consumer (including the consumer's domain manager (VMMlet) image), which causes the initial static domain manager image to be installed at the specified static memory address of the specified server. The initial static domain manager image is executed on the cloud service provider's server as the consumer domain manager (VMMlet). The consumer domain manager (VMMlet) manages the virtual machine on behalf of the consumer by causing the code of the consumer's VM image to be loaded into memory and executed as the consumer domain (VM). The consumer domain (VM) performs operations on the data in the server's memory through the server's memory encryption engine. Since the content of the consumer domain (VM) image changes dynamically, the memory footprint for the consumer domain (VM) image grows and shrinks dynamically.

[0169] Figure 8 is a diagram that shows the data physical address 870 according to one embodiment of the present invention. The data physical address 870 can be used to determine the key or fine-tuning discussed above.

[0170] As described above, the key domain can be defined by using the unused physical address bits 874 (also referred to as the alias bits 874) of the data physical address 870 (or alternatively, by caching other metadata passed). For example, since there are likely to be fewer physical memories installed in the system compared to what can be addressed by using a 64-bit physical memory address, the unused most significant address bits 874 can be used to select between different "key domains". As described above, the term "key domain" refers to a set of memory locations encrypted with a common key domain key. The unused bits of the data physical address 870 can be used to determine, for example, which key and / or fine-tuning will be used when encrypting and / or decrypting memory for a physical memory address. Based on the unused address / alias bits 874, different keys can be selected for the same data physical address 870. For example, the encryption technique XTS (XEX-based tweaked codebook mode with ciphertext stealing) can use the unused address / alias bits 874 for fine-tuning for the same physical memory location, where different address aliasing can result in different ciphertexts even if the data is the same.

[0171] The remaining bits 876 of the data physical address are used to identify the physical memory address of the location in the memory where the data is stored. Although two key domain addresses can alias to the same external memory location, when data from the physical memory location is read into the cache, the cache holds the key domain address independently with full address resolution (e.g., including the full 64-bit physical memory address).

[0172] Different keys can be selected based on unused address bits (e.g., XTS can use alias bits for fine - tuning for the same physical memory location), where different address aliasing can result in different ciphertexts even if the data is the same.

[0173] Since unused address bits 874 alias to the same physical address in the memory for the key domain when there are unused address bits due to unfilled memory, the key domain selector can be set to the value of the unused address bit. Alternatively, if the data in the physical address in the memory is to be shared (i.e., not limited to a specific key domain), the key domain selector can be set to zero.

[0174] In one embodiment, the "k - bit" field 872 represents a bit of the data physical address 870, in which case, it is the uppermost bit of the data physical address 870. The k - bit can be set in the page table or extended page table by the domain manager (VMMlet) or by the virtual machine (VM) to indicate whether the data resulting from a memory access should be encrypted using the corresponding key domain key. When k - bit = 0, the k - bit is said to be disabled and the data resulting from the memory access is not encrypted by the key domain key (although it is possible that the data can be encrypted using a shared key). When k - bit = 1, the k - bit is said to be enabled, and the result of the memory access is encrypted using the key domain key. The k - bit field 872 can also be used to specify a memory range that is shared and does not require key domain encryption. In an alternative embodiment, the k - bit can be additional metadata associated with a cache line and be carried by the cache rather than a component of the data physical address.

[0175] In a scenario where the system has sufficient installed memory such that all address bits of the data physical address 870 are used (except for one k - bit 872), when the k - bit is true / enabled, the key domain address consumes the physical range of the total filled memory (corresponding to the physical address bits of the key domain). When the k - bit is off / disabled, the key domain selector bit 874 references all memory ranges, but as plaintext (or shared), such that all filled memory is addressable as shared memory.

[0176] Figure 9is a diagram that shows a virtual-to-physical memory mapping according to an embodiment of the present invention. Many computer systems today use virtual memory systems to manage memory and allocate memory to the various processes running within the system. Virtual memory allows each process running on the system to operate as if it has control over the full range of addresses provided by the system. The operating system (OS) maps the virtual address space for each process to the actual physical address space for the system. The mapping from physical addresses to virtual addresses is typically implemented by using page tables.

[0177] The term "address space" is used herein to mean a set of addresses in memory corresponding to a given process or virtual machine (VM), and an "address space identifier (ASID)" can be any number, code, or other notation that identifies one or more address spaces with which the ASID is associated.

[0178] Figure 9 represents a case where there is no aliasing; that is, sufficient memory is available in the system such that the key domain selector address bit 974 together with the page address 976 and the cache line selector 978 are used to select the actual physical memory location referenced by the data line physical address 975. Here, each individual key domain will be in a non-overlapping range of the physical memory 920.

[0179] Figure 9 shows a virtual address to physical address mapping according to an embodiment of the present invention. The physical address 924 within the physical page 822 in the physical memory 920 can be addressed by using the virtual address 900. As shown, the virtual address 900 includes multiple fields to index a multi-level paging structure 960 to access the physical address 924, which addresses a specific physical page 922 within the physical memory 920. Note that the multi-level paging structure 960 is only one example of a multi-level paging structure for accessing physical memory locations. Although the multi-level paging structure 960 is described with reference to a 64-bit virtual address, different page table structures can be used for 32-bit virtual addresses, physical address extension (PAE) extended mode addresses, or other types of virtual addresses.

[0180] In virtual address 900, an offset field 902 (such as bits 0-11 of a 64-bit address) is used to address a physical address 924 within a physical page 922 of physical memory 920 (as indicated by pointer 903). A page table entry field 904 (referred to as a "table", such as bits 12-20 of a 64-bit address) addresses a page table entry 932 in a page table 930 (as indicated by pointer 962c). A page directory entry 906 (referred to as a "directory", such as bits 21-29 of a 64-bit address) addresses a page directory entry 942 in a page directory 640 (as indicated by pointer 962b). A page directory pointer 909 (referred to as a "PDP", such as bits 30-38 of a 64-bit address) addresses a page directory pointer entry 952 in a page directory pointer table (PDPT) 950 (as indicated by pointer 962a). The base address of the OS paging structure 960 can be accessed by using a control register, such as pointer 961 in CR3. In this way, a 64-bit linear address can be used to implement a multi-level paging structure to access a physical address.

[0181] Figure 9 Also shown is a component of a data physical address 970 corresponding to the physical address 924 of the physical page 922 of physical memory 920. A "K-bit" field 972 represents a bit of the data physical address 970, in this case, the uppermost bit of the data physical address 970. The k-bit can be set in the page table or an extended page table by a domain manager (VMMlet) or by a virtual machine (VM) to indicate whether data generated by a memory access should be encrypted using a corresponding key domain key. When the k-bit = 0, the k-bit is said to be disabled and data generated by a memory access is not encrypted by a key domain key (although it is possible that the data can be encrypted using a shared key). When the k-bit = 1, the k-bit is said to be enabled, and the result of a memory access is encrypted using a key domain key. The k-bit field 772 can also be used to specify a memory range that is shared and does not require key domain encryption. In an alternative embodiment, the k-bit can be additional metadata associated with a cache line and be carried by the cache rather than a component of the data physical address.

[0182] In the data physical address 970, "unused address bits": The "key domain selector" field 974 can represent a set of unused address bits used to distinguish between key domains. If the unused address bits of two data physical addresses have different values, then they alias to the same physical address in the memory. The "page address" field 976 represents the address of the physical page 922 in the physical memory 920. The "cache line selector" field 978 represents the cache line within the page referenced by the "page address" field 976. Together, the "physical address" field 976 and the "cache line selector" field 978 constitute the "data line physical address" field 975, which represents the actual physical location in the physical memory 920. The "cache line byte" field 979 contains the number of bytes in the cache line.

[0183] Now refer to Figure 10 , which shows another virtual address to physical address mapping according to an embodiment of the present invention. As Figure 10 shown, the aliased guest physical address 1014 within the aliased guest physical page 1012 in the aliased physical memory 1010 can be addressed by using the virtual address 1000. As shown, the virtual address 1000 includes multiple fields to index a multi-level paging structure 1060 to access the aliased guest physical address 1014, which addresses a specific page 1022 within the physical memory location 1020. Note that the multi-level paging structure 1060 is only one example of a multi-level paging structure for accessing physical memory locations. Although the multi-level paging structure 1060 is described with reference to a 64-bit virtual address, different page table structures can be used for 32-bit virtual addresses, physical address extension (PAE) extended mode addresses, or other types of virtual addresses.

[0184] The aliased physical memory 1010 also includes an aliased guest physical page 1016, which represents a second range of the aliased guest physical memory 610 that is aliased to the same physical memory location 1022.

[0185] In virtual address 1000, an offset field 1002 (such as bits 0 - 11 of a 64 - bit address) is used to address an aliased guest physical address 1014 within an aliased guest page 1012 of an aliased physical memory 1010 (as indicated by pointer 1003). A page - table - entry field 1004 (referred to as a "table", such as bits 12 - 20 of a 64 - bit address) addresses a page - table entry 1032 in a page table 1030 (as indicated by pointer 1062c). A page - directory - entry 1006 (referred to as a "directory", such as bits 21 - 29 of a 64 - bit address) addresses a page - directory entry 1042 in a page directory 640 (as indicated by pointer 1062b). A page - directory - pointer 1008 (referred to as a "PDP", such as bits 30 - 38 of a 64 - bit address) addresses a page - directory - pointer entry 1052 in a page - directory - pointer table (PDPT) 1050 (as indicated by pointer 1062a). The base address of the OS paging structure 1060 can be accessed by using a control register, such as pointer 1061 in CR3. In this way, a 64 - bit linear address can be used to implement a multi - level paging structure to access a physical address.

[0186] Figure 11 is a diagram showing the initial steps, performed by a cloud service provider according to an embodiment of the present invention, for providing a domain image to a consumer.

[0187] In Figure 11 the example shown, a memory manager 1140 of a cloud service provider server including hardware 1110 allocates space 1114 in a memory 1112 for a domain image 1122 and notifies a requesting consumer of memory - location - related address ("trim variable") information. This memory - location - related address ("trim variable") information can in particular include the physical address of a page in the memory (such as the physical addresses of the pages making up space 1114), the physical address for a page table, control - register information, interrupt - descriptor - table - register information, and so on. As an alternative embodiment, the cloud service provider can create a trimmed domain image that the consumer can re - verify as correct. In particular, the part of the domain image that needs to be changed is the physical - memory - page address in the page table as Figure 9 shown in, page - table entry 832. Page - table entry 932 points to physical page 922 in physical memory 920. A domain image can be considered a series of pages (e.g., each 4K bytes), where each page is given a physical - page address (its location in the memory). Image verification then includes checking that, given the content of the pages making up the domain image, the virtual - to - physical mapping through the page table is correct.

[0188] Figure 12is a diagram that shows a message for providing a domain manager image, such as a VMMlet image Figure 11 of Figure 11 to a consumer between a consumer 1201 and a memory manager 1240 of a cloud service provider.

[0189] In response to a consumer's request for a service, software (i.e., the memory manager 1240) of a server of the cloud service provider is configured to provide a domain manager image, such as a VMMlet image Figure 11 of Figure 11 to the consumer. The memory manager 1240 also sends address information related to the memory location of the domain manager image, also referred to herein as trimming variable information, to the consumer. The consumer verifies that the domain manager image is valid or uses a third party to verify that the domain manager image is valid.

[0190] After determining that the domain manager (VMMlet) image is valid, as described with reference to Figure 11 the consumer modifies the verified domain manager image using the address information related to the memory location that identifies the memory location provided by the cloud service provider as trimming variable information. The verified domain manager image is provided by the cloud service provider to create a domain launch image, thereby launching the domain manager (VMMlet). Alternatively, the domain manager image can be "trimmed" by the cloud service provider so that the domain manager image is ready to run in the allocated memory location.

[0191] In one embodiment, the consumer can also add the consumer's own components, such as the consumer's secret key for secure communication, to the domain launch image. Having a method for secure communication allows the consumer's basic domain launch image to securely retrieve the rest of the consumer domain (VM) image from the consumer using the consumer's secret key. The consumer can also include the consumer's own operating system, applications, etc. in the domain launch image.

[0192] Finally, when the consumer's domain launch image includes any consumer-supplied components, the consumer encrypts the domain launch image. "Trimming" the domain manager (VMMlet) image and creating an encrypted domain launch image are further described with reference to Figure 13 .

[0193] Figure 13is a diagram that shows messages for encrypting a domain launch image and establishing a key domain among components of a cloud service environment according to an embodiment of the present invention. As described above, consumer 1301 modifies the verified domain manager image provided by the cloud service provider to create a domain launch image for the domain manager (VMMlet). The domain launch image is then encrypted by using the memory location-related "fine-tuned" cipher and the consumer's key domain key.

[0194] Consumer 1301 can also compute an integrity check value (ICV, e.g., a keyed hash message authentication code (HMAC) value) for the encrypted domain launch image by using the key domain key. The ICV can be computed as a location-related value and used to verify the location and the content of the associated memory location for the encrypted domain launch image.

[0195] Consumer 1301 requests the cloud service provider memory manager 1340 to identify a server in the cloud service provider's network that provides key domain management functionality. The cloud service provider memory manager 1340 obtains the server certificate for the server capable of having a key domain (in this example, from the server with CPU 1311) and provides the server certificate to consumer 1301. Consumer 1301 verifies that the service certificate is signed by an authority that attests that the identified server provides key domain management functionality.

[0196] Consumer 1301 encrypts the consumer's key domain key using the public key of the key domain-capable server of the cloud service provider corresponding to the certificate of the server capable of having a key domain. Consumer 1301 sends the encrypted key domain key, the encrypted launch image, and (optionally) an integrity check value (ICV) to the cloud service provider memory manager 1340, which provides a "Create Key Domain (CreateKD)" command to the key domain-capable server. In one embodiment, the cloud service provider memory manager 1340 identifies a key domain address selector to be used for the new key domain and provides the key address domain selector to the CPU 1311 of the key domain-capable server. The CPU 1311 of the key domain-capable server creates and initializes the key domain. Initializing the key domain may include flushing the cache of any prior key domain (identified by the prior key domain address selector) and flushing the translation lookaside buffer that caches the address mapping for the prior key domain. As an alternative to performing the initialization function as part of the "Create Key Domain" instruction, the CPU 1311 of the key domain-capable server may execute an "Initialize Key Domain (InitKD)" instruction to flush the cache and the translation lookaside buffer. The CPU 1311 of the key domain-capable server also provides the encrypted key domain key and the key domain address selector identifying the new key domain to the memory encryption engine 1315 (shown in Figure 13 as the total memory encryption engine with integrity, labeled as TMEi 1315), although alternative embodiments will use a memory encryption engine (MEE)).

[0197] Figure 14 is a diagram that shows a consumer providing an encrypted launch image for a domain manager (VMMlet) according to an embodiment of the present invention. As described above with reference to Figure 5 , 12 and 13, the consumer encrypts the domain launch image by using the memory location-related address information provided by the cloud service provider and the key domain key. In one embodiment, the consumer encrypts the domain launch image in a memory location-related XTS mode by using page address micro-adjustment and the key domain key.

[0198] In Figure 14 , consumer 1410 sends the trimmed VMMlet 1462 (which is Figure 10The modified version of the original VMMlet 1022 of the provider is sent as part of the encrypted domain (VM) boot image 1460 to the memory manager 1440 of the cloud service provider. The memory manager 1440 of the cloud service provider loads the trimmed VMMlet image 1462 of the encrypted VM boot image 1460 into a previously allocated memory space 1414 that has been reserved (as Figure 10 space 1014) in the shared memory 1412s. Since the shared memory 1412 is an unencrypted memory page (with k bits disabled), the memory manager 1440 needs to ensure that the encrypted VM boot image is fully loaded into the physical memory 1412s and does not remain resident in the cache. Writing the encrypted VM boot image 1460 to the physical memory 1412s can be achieved by flushing the encrypted VM boot image 1460 from the cache (e.g., by using the CLFLUSH instruction), or by using cache - bypass / write - through / non - temporal memory access. These techniques for the write operation ensure that the consumer's encrypted image data is directly written through the memory encryption engine of the hardware 1410 and into the memory 1412s (and does not remain resident in the cache).

[0199] Figure 15 is a diagram that shows the messages for loading an encrypted domain image of a consumer 1501 into the memory 1512 of a server capable of having a key domain among the components of a cloud service environment according to an embodiment of the present invention. As described above with respect to Figure 11 the software of the cloud service provider, such as the memory manager 1540 of a server capable of having a key domain, loads the consumer - encrypted domain boot image into an unencrypted memory page (with k bits disabled) of the memory 1512. The software of the cloud service provider (i.e., the memory manager 1540) also writes the ICV for the encrypted domain image to the ICV table. The ICV table can be a protected range of the memory 1512 that is managed and protected by the memory encryption engine (TMEi) engine 1515. Write operations to the memory addresses of this range by software components that are not part of the memory manager 1540 can be intercepted by the memory encryption engine (TMEi) 1515, which can similarly configure the ICV values in the memory 1512.

[0200] Similarly, the memory encryption engine (TMEi) 1515 can prevent software from reading the ICV values from the protected memory range to prevent malware from replaying the ICV values. Only the memory encryption engine (TMEi) 1515 can read the ICV values once they have been established. Preventing software from replaying the ICV values prevents the replaying of dynamic domain image content (such as the remainder of the consumer domain image that is subsequently provided to the domain launch image). Static image content (such as the domain launch image) can be replayed because the ICV values for the static image content are provided by the consumer.

[0201] The ICV itself provides an integrity check of the data and the data location (address), and the ICV is encrypted by a key domain key or its derivative that is used to encrypt the data row that the ICV is checking. (For example, HMAC uses a secret key as done by Galois / Counter Mode (GCM) and IPHash).

[0202] In one embodiment, a partial copy of the diffused cache line data is XTS encrypted using the key domain key to compute the secure ICV. In one embodiment, the ICV table entries are encrypted using the same key domain key as the memory location used for the integrity check. Encrypting both the ICV and the data in the memory location protected by the same key domain key cryptographically ensures that the ICV belongs to the same key domain as the data they protect.

[0203] An address selector for the key domain is also provided in the unencrypted memory (where the k bits are disabled). When the ICV computed by the consumer using the consumer's key domain key is written to the ICV table, the address also indicates the location in the ICV table that contains the ICV for the key domain. (In other words, for each data row written to the memory from the consumer's encrypted domain launch image, there is a corresponding integrity check value written to the ICV table for that data row).

[0204] No key domain is used to write the consumer's encrypted domain launch image data to the memory 1512 (since the consumer's domain launch image data has already been encrypted by the consumer). In other words, the memory manager 1540 can write the consumer's encrypted domain launch image to a shared, unencrypted location in the memory 1512 without any additional encryption. (The memory manager 1540 loads the consumer's encrypted image into the memory 1512 such that when memory encryption is turned on using the consumer's key (setting the k bits to 1 or "enabled"), the memory encryption engine (TMEi) 1515 will decrypt the consumer image properly when reading the consumer image from the memory 1512).

[0205] The CPU 1511 of a server capable of having a key domain obtains an address selector for the key domain from an unencrypted location in the memory 1512 and provides the address selector for the key domain to the memory encryption engine (TMEi) 1515. The memory encryption engine (TMEi) 1515 then writes the encrypted launch image to the memory 1512 of the server capable of having a key domain. Similarly, the CPU 1511 of a server capable of having a key domain obtains an address that indicates a location in the ICV table that contains the ICV for the key domain. The CPU 1511 provides the location of the ICV table that contains the ICV for the key domain to the memory encryption engine (TMEi) 1515. The memory encryption engine (TMEi) 1515 then updates the ICV table in the memory 1512 with the ICV for the key domain. The memory manager 1540 flushes these values from the cache 1513 (e.g., by issuing a command to execute the CLFLUSH instruction) or uses cache - bypassed / write - through / non - transient memory accesses to ensure that the ICV data is written directly to the memory 1512.

[0206] Updating the integrity check value (ICV) for the key domain in the memory of a server capable of having a key domain is a write - only operation such that the software of the cloud service provider (memory manager 1540) cannot read the ICV for the key domain. This write - only operation prevents replay of the dynamic image data. Only the domain launch image ICV can be replayed in place (the consumer knows where because the consumer created the encrypted launch image and the ICV). This functionality allows the provider to suspend, store, and later resume the VM, reusing the consumer's domain launch image without exposing additional ICVs, such as those dynamically created when an application executing within the current key domain updates the memory, to the software of the cloud service provider (not even the memory manager 1540).

[0207] Figure 16 is a diagram that shows the initialization of a key domain according to an embodiment of the present invention. By issuing an "Initialize Key Domain (InitKD)" command to the domain manager (VMMlet) 1622, the memory manager 1640 of a server capable of having a key domain can initialize a new key domain 1650. The InitKD command causes the CPU 1611 of the server capable of having a key domain to execute the InitKD instruction, which silences the core, flushes the cache of the old key domain, flushes all translation lookaside buffers and address space identifiers (ASIDs) that contain the old key domain mapping, and initializes the memory encryption engine (TMEi) of the server capable of having a key domain with the new key domain key for the key domain address selector.

[0208] In one embodiment, initializing the key domain is one of the actions performed by the "Create Key Domain (CreateKD)" instruction. References hereinafter to the "Create Key Domain (CreateKD)" instruction may refer to the CreateKD instruction, which creates a key domain not only based on the server's public-key encrypted key domain key, but also initializes a new domain by silencing the cores, flushing the cache of the old key domain, flushing all translation lookaside buffers and address space identifiers (ASIDs) containing the old key domain mapping, and initializing the memory encryption engine (TMEi) of the server capable of having a key domain with the new key domain key for the key domain address selector.

[0209] Figure 17 FIG. is a flowchart of an operating method of a CPU of a server capable of having a key domain in performing the "Create Key Domain" operation according to an embodiment of the present invention. In block 1710 of "Receiving the 'Create Key Domain' command and the encrypted key domain key", the server CPU capable of having a key domain receives the "Create Key Domain" command, along with the input parameters KD Id, local key domain identifier (key domain address selector), and the encrypted key, the encrypted key domain key. Control proceeds to block 1720 of "Decrypting the encrypted key domain key by using the server's private key and decrypting the optional configuration policy", where the encrypted key domain key is decrypted by using the server's private key, a secret key unknown / unexposed to the cloud service provider. Optionally, the configuration policy may also be decrypted, again by using the server's private key, or alternatively, the hash value of the policy data may be decrypted by using the server's private key. Control proceeds to decision point 1730 of "The decrypted key domain key and the policy are valid". Examples of policy data that can be evaluated include the amount of memory the expected server has installed, the encryption algorithm the server should use, the number of socketed CPUs, whether hardware debugging is allowed, and so on. This policy data is compared by the hardware against the current configuration of the server to ensure that: before using the consumer's key domain key, the configuration of the server is valid as expected by the consumer. If the decrypted key domain key and the configuration policy are not valid, control proceeds to block 1740 of "Returning an error", where the CPU returns an error in response to the "Create Key Domain" command. In "The decrypted key domain key and the policy are valid"

[0210] In "The decrypted key domain key and the policy are valid" ”At decision point 1730, if the decrypted key domain key and the configuration policy are valid, control proceeds to the “Establish New Key Domain” box 1750. In establishing a new key domain, the CPU of the server capable of having a key domain prevents other CPUs from using the key domain identifier, or otherwise verifies that other CPUs are not currently using the key domain identifier, flushes the cache for the key domain identifier, flushes all translation lookaside buffer address space identifiers for the key domain identifier, and sets the current key domain identifier and the key domain key in the memory encryption engine. Control then proceeds to the “Assign ASID Tag and Resume for New Key Domain Identifier” box 1760, where a new address space identifier (ASID) tag is assigned for the new key domain identifier, and the process that issued the “Create Key Domain” command is resumed. Additionally, if previously silenced, all processors are restarted.

[0211] Figure 18 is a diagram that illustrates the verification of a domain image according to an embodiment of the present invention. Using a Hash Key Domain (HashKD) instruction, the verification hash function 1846 of the memory manager 1840 of the server capable of having a key domain verifies that the domain image (e.g., VMMlet 1822) for the key domain 1850 is correct. Some values in the domain image will be values for machine-specific variables / trimmings (such as physical addresses used in page tables). The hash values for these machine-specific variables can be virtually reconstructed (substituting the current address values) by the cloud service provider. When the resulting HashKD hash value matches the expected value of the hash for the domain manager (VMMlet) provider / static part of the image, both the consumer and the cloud service provider agree that the domain launch image is correct. Some other image locations may contain secrets of the consumer (keys / codes / data, e.g., in the OS part of the domain image). These locations can be hashed, but the hash value will not reveal the memory plaintext (and thus the secrets) to the cloud service provider. The hash value can have a minimum granularity, such as no less than a cache line or no less than a page of memory (e.g., 4KB).

[0212] When the secure VM first executes, the secure VM can turn off the HashKD functionality into its key domain, because HashKD can be used at initialization to read through the key domain, providing the cloud service provider with visibility that the domain manager (VMMlet) is properly supplied by the consumer. Otherwise, the HashKD functionality may not be needed.

[0213] Figure 19is a diagram that shows messages for verifying a domain image among components in a cloud service environment according to an embodiment of the present invention. In a first message, a cloud service provider software, such as a memory manager 1940 of a server capable of having a key domain, requests a hash key domain (HashKD) function to be performed on a memory location where an encrypted boot image for a consumer of a domain manager (VMMlet) is installed. The CPU 1911 executes a hash key domain (HashKD) instruction, thereby providing, via the cache 1913, a current address selector that identifies the key domain to be hashed to a memory encryption engine (TMEi) 1915. The memory encryption engine (TMEi) 1915 reads an encrypted data row from the memory location where the encrypted boot image is installed, and the memory encryption engine (TMEi) 1915 decrypts the data row by using the key for the key domain identified by the address selector. The memory encryption engine (TMEi) 1915 sends the decrypted data to the cache 1913 to tag the decrypted data with the address and the key domain address selector. The CPU 1911 of the server capable of having a key domain creates a hash value for the decrypted data, stores the resulting hash value in a register of the CPU 1911 or a memory location of the memory 1912, and the software of the cloud provider (i.e., the memory manager 1940) verifies that the hash value matches the expected hash value for the original domain image provided to the consumer.

[0214] Figure 20 is a flowchart of an operation method of a CPU of a server capable of having a key domain in performing a "hash key domain" operation. At a "Receive hash key domain command" box 2010, the CPU of the server capable of having a key domain receives a hash key domain command, along with input parameters: a key domain identifier and a physical address. Control proceeds to a "Key domain identifier and address valid " decision point 2020, where the CPU of the server capable of having a key domain determines whether the key domain identifier and the physical address are valid. To make this determination, the CPU of the server capable of having a key domain can verify that the physical address points to a populated memory location and that there is a page table mapping and a read permission for the physical address. The CPU of the server capable of having a key domain can also verify that the key domain identifier has a corresponding key domain key installed in a memory encryption engine (TMEi) for the key domain identifier. If the key domain identifier and the physical address are not valid, control proceeds to a "Return error" box 2030, where an error is returned to the issuer of the hash key domain command. If the key domain identifier and the physical address are "Key domain identifier and address valid If it is valid at decision point 2020, the control proceeds to the "Set key domain identifier in the physical address, set k bits, read the content of the memory location at the physical address" box 2040. The unused bits of the physical address are set to the key domain identifier, and the k-bit value is set to 1 to indicate that the encrypted data will be read from the memory location at the physical address, and the content of the memory location at the physical address is read by the memory encryption engine (TMEi) using the key domain key for the key domain identified by the key domain identifier. When reading the content of the memory location at the physical address, the memory encryption engine (TMEi) decrypts the content using the key domain key. The memory encryption engine (TMEi) places the decrypted content of the memory location at the physical address into the cache, and the CPU of the server capable of having the key domain calculates a hash value, such as a SHA2 / 3 hash value, by hashing the decrypted content in the cache. The control proceeds to the "Return the hash value for the memory content" box 2050, where the hash value is returned to the issuer of the HashKD command. The issuer of the HashKD instruction can then determine whether to switch to the verified key domain.

[0215] Figure 21 is a diagram that shows the switching between key domains according to an embodiment of the present invention. The switching to a new key domain is initiated by the memory manager 2140 of the server capable of having the key domain to switch from one key domain to another. The switching to a new key domain can be initiated, for example, in response to a new consumer request for a service received by the server capable of having the key domain. Having previously provided the domain manager image to the consumer, the memory manager 2140 obtains a trimmed version of the domain manager image (e.g., VMMlet 2122), which contains memory location-related address information, such as the entry point address 2123 provided by the consumer via the unencrypted (k-bit disabled) memory 2112. The memory manager 2140 issues a "Switch Key Domain (SwitchKD)" command to the CPU of the hardware 2110, which causes the domain manager image (VMMlet) 2122 to start executing at the entry point address 2123 in the memory of the server capable of having the key domain, thereby establishing the key domain 2150.

[0216] In one embodiment, before entering a new key domain, the consumer computes an HMAC value for the expected processor state. The HMAC value for the expected processor state includes the expected value for the instruction pointer; the stack pointer; control registers (such as control registers 0, 3, and 4); and special descriptor table registers, such as the GDTR, LDTR, any MSRs that can be correct for the proper execution of the encrypted domain that contains the domain manager image, and the IDTR. For example, this HMAC value will ensure proper execution within the key domain (i.e., interrupts are turned off), or otherwise ensure that no new key domain is entered and no key domain switch occurs.

[0217] The execution of the domain manager (VMMlet) within the new key domain can be achieved by using the "Switch Key Domain (SwitchKD)" instruction. This CPU instruction determines an HMAC value by using the key domain key to verify the current processor state of the processor upon entry. For example, a hash function can be computed based on the same information as used to compute the HMAC value for the expected processor state, including the instruction pointer, stack pointer, control registers, and special descriptor table registers. If the computed HMAC value for the expected processor state does not match the hash value of the CPU's current processor state upon entry, the "Switch Key Domain" instruction will fail. The key domain selector will remain unchanged, and the execution of the domain manager (VMMlet) will not switch to the new key domain.

[0218] In another embodiment, a control flow transfer into a new key domain terminates on an "End Branch Key Domain (ENDBRANCHKD)" instruction. The only way to change the control flow from another key domain to a new key domain is to enter the new key domain at an entry point where the next instruction to be executed is the "End Branch Key Domain" instruction. This requirement for changing key domains ensures to the consumer that the control flow transfer is through the expected entry point.

[0219] After the correct execution of the "Switch Key Domain (SwitchKD)" instruction, the domain manager (VMMlet) is now measured and runs correctly within the key domain. All other functionality is provided by software to load the rest of the consumer's domain (VM) image, to perform secure storage, communication, paging (e.g., for page out, the consumer's domain (VM) image needs to go through the domain manager (VMMlet)), migration, input / output (I / O), and so on.

[0220] Figure 22is a diagram that shows messages between components of a cloud service environment when executed within a key domain, according to an embodiment of the present invention. In one embodiment, before entering a new key domain, the consumer calculates a hash message authentication code (HMAC) value for the expected processor state. The HMAC value for the expected processor state includes the expected value for the instruction pointer; the stack pointer; control registers (such as control registers 0, 3, and 4); and special descriptor table registers, such as the GDTR, LDTR, any relevant MSRs, and the IDTR. For example, the HMAC value will ensure that the processor state is such that proper execution of the domain manager (VMMlet) occurs within the key domain (i.e., interrupts are turned off).

[0221] In the next communication, cloud service provider software, such as the memory manager 2240 on a server capable of having key domains, issues a "switch KD" command to cause the CPU 2211 of the server capable of having key domains to switch key domains. The CPU 2211 sets the key domain selector and checks that: the expected processor state HMAC value matches the HMAC value calculated for the current CPU state. If the HMAC values match, the key domain is switched, the instruction pipeline is flushed, the translation lookaside buffer (TLB) address space identifier (ASID) tags are changed for the key domain identifier, and the domain manager (VMMlet) executes in the new key domain. When executing in the new key domain in which k bits are enabled, the memory locations of the key domain are accessed by using an address set to the key domain selector value. When executing in the new key domain, the memory encryption engine (TMEi) 2215 may read and write key domain-encrypted data from / to the memory 2212 and check and / or update the integrity check value (ICV) for the encrypted data. If the ICV value is consistent with the encrypted data, the encrypted data is decrypted into the cache 2213 for the address and key domain selector.

[0222] Figure 23 is a flowchart of an operation method of a CPU of a server capable of having key domains in performing a "switch key domain" operation. At the "Receive switch key domain command, with input parameters: key domain identifier, CPU state, and expected HMAC value" box 2310, the CPU receives the "switch key domain" command. The input parameters of the "switch key domain" command include the key domain identifier to switch to, the expected CPU state to switch to the new key domain, and the expected HMAC value for the processor state. Control proceeds to "The expected HMAC value matches the CPU state "Decision point 2320. At decision point 2320, the following determination is made: whether the current CPU state and / or the proposed CPU state designated as the parameter for the SwitchKD (Switch Key Domain) instruction matches the expected HMAC value. Some CPU state information, such as the instruction pointer, is set to the state of the parameter by the SwitchKD (Switch Key Domain) instruction. If the HMAC also matches the instruction pointer parameter, then SwitchKD (Switch Key Domain) will correspondingly set the instruction pointer register in the CPU to resume in the new key domain starting at that instruction location. Alternatively, all CPU state values can be parameters for SwitchKD (Switch Key Domain), meaning that SwitchKD (Switch Key Domain) will fill all register states into the input parameter if the HMAC matches the proposed input state. At decision point 2320, if the expected HMAC value for the processor state does not match the HMAC value calculated for the current CPU state or the proposed CPU state designated as the parameter for the SwitchKD (Switch Key Domain) instruction, then the key domain is not switched and an error is returned to the issuer of the "Switch Key Domain" command in the "Return Error (Do Not Switch Key Domain)" box 2330.

[0223] "At the decision point 2320 where the "expected HMAC value matches the CPU state" At decision point 2320, if the expected HMAC value for the processor state matches the HMAC value calculated for the current CPU state, then control proceeds to the "Switch to New Key Domain" box 2340. At the "Switch to New Key Domain" box 2340, the CPU pipeline is flushed, and the address space identifier (ASID) tag for the translation lookaside buffer (TLB) is set to the new key domain identifier or the TLB is flushed. The current key domain is set to the key domain identifier, and the CPU registers are set to match the CPU state input parameter values. Control then proceeds to the "Branch to Execute Instruction at Location Indicated by Instruction Pointer of CPU State" box 2350. At box 2350, the CPU of the server capable of having a key domain branches to execute an instruction at the location indicated by the instruction pointer of the CPU state, where the instruction pointer of the CPU state is provided as the input parameter for the SwitchKD (Switch Key Domain) instruction. Upon completion of the execution of the SwitchKD (Switch Key Domain) instruction, the domain manager (VMMlet) operates within the new key domain.

[0224] Figure 24Flowchart of an operation method of a CPU of a server capable of having a key domain in performing paging structure walk in response to a page miss. The control starts at the "When there is a page miss, the processor walks the paging structure" box 2410. When a page miss is encountered (where the page that the CPU attempts to read or write is not found in the translation lookaside buffer), the CPU of the server capable of having a key domain starts walking the paging structure (such as the OS paging structure 860 described with reference to Figure 9 or the OS paging structure 960 described with reference to Figure 9 ). For example, the CPU of the server capable of having a key domain can start by reading the control register 3 (CR3) for a pointer to the base address of the paging structure. The control then continues to the "Paging structure misconfiguration " decision point 2420, where the CPU determines whether the paging structure is configured as expected. For example, the CPU determines whether a page fault for the operating system has occurred, or a VMExit for the domain manager (VMMlet) has occurred. These faults are still within the same key domain as the domain (VM) image that caused the fault. If the paging structure is not configured as expected, the control continues to the "Hard fault, the CPU reports an error" box 2430, where the CPU causes a hard fault and reports an error to the process that encountered the page miss.

[0225] At the "Paging structure misconfiguration " decision point 2420, if the paging structure is properly configured, the control continues to the "Determine the ASID tag assigned to the current key domain" box 2440. The address space identifier (ASID) tag assigned to the current key domain is determined, and the control continues to the "The K bit is set " decision point 2450. If the K bit of the ASID tag is not set, the control continues to the "Fill the TLB with the address as-is with the K bit off" box 2460. At box 2460, the CPU of the server capable of having a key domain causes the translation lookaside buffer (TLB) to be filled with the physical address in fact. The physical address is as-is so that data can be directly read from the unencrypted memory without using the key domain key.

[0226] At the "The K bit is set " decision point 2450, if the k bit of the ASID tag is set, the control continues to the "Get the current key domain and replace the upper physical address bits with the key domain identifier" box 2470. The current key domain is determined according to the internal processor state set by the "Switch Key Domain (SwitchKD)" instruction, and the "Switch Key Domain (SwitchKD)" instruction sets the current key domain to the key domain identifier of the new key domain, as described with reference toFigure 23 As described. The upper bits in the physical address are replaced with a key domain identifier / selector for the current key domain, where k bits (which are the uppermost bits in one embodiment) are enabled. Control then proceeds to the "Set translation lookaside buffer using the address and ASID tag" box 2480, where the translation lookaside buffer is set using the physical address for the current key domain (including the key domain selector and the enabled k bits, or k bits = 1) and the ASID tag.

[0227] Figure 25 is a diagram that shows the growth of a domain manager (VMMlet) according to an embodiment of the present invention. Growth of the domain manager (VMMlet) may be required, for example, to include additional memory for loading the remainder of the consumer's VM image after the consumer's domain launch image has been loaded and the domain manager (VMMlet) is executing. Once the secure domain manager (VMMlet) 2522 with the consumer's secret key 2523 is running in the key domain 2550, the consumer can securely transfer the remainder of the consumer's VM image 2532 to the domain manager (VMMlet) 2522. The remainder of the consumer's VM image 2532 may include, for example, an (optional) operating system, an (optional) application, a script, or other code.

[0228] By using the consumer's secret key 2523 for the original encrypted domain launch image given to the cloud service provider, a secure communication channel between the consumer and the domain manager (VMMlet) 2522 can be enabled by a transport layer security / secure sockets layer (TLS / SSL) connection with the consumer's network. In other words, if the original encrypted domain launch image has an operating system and the consumer's secret key, and the operating system has an OpenSSL stack above the domain manager (VMMlet), then the OpenSSL software stack can be executed to retrieve the remainder of the consumer's VM image 2532 from the consumer's network.

[0229] The operating system running above the domain manager (VMMlet) 2522 may support full volume storage encryption such that the operating system can securely page in encrypted pages, files, etc. from the k-bit off (shared) channel, where the memory manager 2540 acts as an intermediary. Once the original encrypted domain launch image is loaded into memory and executing, the domain manager (VMMlet) 2522 can allow other software, such as the operating system, to page in additional information from the consumer using any desired security method.

[0230] The addition of memory pages for the remainder of the VM image 2532 for the consumer can cause the domain manager (VMMlet) 2522 to require additional memory allocation. In one embodiment, the domain manager (VMMlet) 2522 can grow by requesting more memory from the memory manager 2540. The memory manager 2540 can allocate additional memory to the domain manager (VMMlet) 2522, as shown by the "allocate memory" action 2501. This additional memory enables the consumer to perform write-only operations, such as non-temporary move (MOVNT) operations (uncached write-combining operations for writing to memory without first reading the memory), to write additional pages in the domain / VM workload image from the consumer or a third party authorized by the consumer. For example, the consumer can provide the remainder of the VM image 2532 to the domain (VMMlet) 2522 via a secure connection, including the operating system(s), application(s), script, or other code.

[0231] Figure 26 is a diagram that shows messages for growing a domain manager (VMMlet) among components of a cloud service environment according to one embodiment of the present invention. The consumer 2601 sends the remainder of the VM image to cloud service provider software, such as the memory manager 2640 of a server capable of having a key domain. In one embodiment, the remainder of the VM image is transferred from the consumer to the running domain manager (VMMlet) via a Transport Layer Security (TLS) / Secure Sockets Layer (SSL) communication session, which is a TLS stack from the consumer using a consumer secret key (such as Figure 25 the key 2523) to the running domain image.

[0232] As referred to above Figure 25 the consumer's secret key is included as part of the encrypted domain launch image given to the cloud service provider by the consumer. At the time point represented by Figure 26 the consumer's VM is running securely and is self-sufficient, running any software provided by the consumer on top of the VMMlet (similar to an operating system running on top of a VMM). Data packets are sent from the consumer 2601 to the running domain manager (VMMlet) via the memory manager 2640, through shared unencrypted memory (i.e., memory 2612 in which k bits are disabled). These data packets can include a stream of software-encrypted data that can be decrypted and verified by the consumer software running within the consumer's VM on top of the running domain manager (VMMlet).

[0233] Cloud provider software can send data for the remainder of the VM image on behalf of the consumer via the CPU 2611 and the Trusted Memory Encryption engine (TMEi) 2615, through the shared unencrypted memory (i.e., the memory 2612 where k bits are disabled). The data for the remainder of the VM image is shown flowing from the CPU 2611 through the Trusted Memory Encryption engine (TMEi) 2615 to the memory 2612, as illustrated by the two "Write Data" actions. When the data for the remainder of the VM image is provided to the cloud service provider, the cloud service provider software can cause the CPU 2611 to execute a "Switch Key Domain (SwitchKD)" instruction to switch to the key domain for the consumer's running domain manager (VMMlet). Alternatively, the CPU 2611 can provide control to the consumer's running domain manager (VMMlet) running on another thread or another CPU. These actions by the CPU 2611 are illustrated by Figure 26 the "SwitchKD (or KD running on another CPU / thread)" action.

[0234] The running domain manager (VMMlet) copies data from the unencrypted memory (including the remainder of the VM image) to the encrypted memory that is part of the consumer VM's key domain. As shown by the "Read Data" action, the Trusted Memory Encryption engine (TMEi) 2615 reads data from the unencrypted memory. In the "Read Data for address with shared!k KD selector" action, the domain manager (VMMlet) running on the CPU 2611 reads data from the unencrypted memory location identified by the key domain address selector, which is provided in the unencrypted memory along with the data.

[0235] The running domain manager (VMMlet) can process the data, decrypt the data in software, perform integrity checks, etc. For example, the running domain manager (VMMlet) can use the key domain address selector provided in the unencrypted memory to request the Trusted Memory Encryption engine (TMEi) 2615 to write the encrypted data to a memory address. As shown in Figure 26 the "Memory access with k bits enabled has address set to KD selector; use MOVNT instruction on first write to new memory address" action, the CPU 2611 writes the encrypted data and the associated integrity check value to the address in the memory 2612 specified in the key domain identifier / selector (where k bits are enabled, indicating that the data is encrypted with the key domain key before being written to the memory 2612).

[0236] Writing data to a memory address establishes the "owner" of the memory location. When switching between key domains, the owner of the memory location changes from the owner of one key domain to the owner of another key domain. When the key domain changes, the corresponding key domain key used to encrypt the data stored in the memory locations belonging to the key domain changes accordingly.

[0237] When reading data from a memory location belonging to a key domain, the "current" key domain key is used. After a "switch key domain" instruction, a new "current" key domain key must be established. As described above, the "current" key domain key is established when writing data to a memory location, thereby establishing a new owner of the memory location. When reading data from a memory location, the read operation will use the "current" key domain key. If data is read from a memory location before the owner of the new key domain has written to it, the read operation will use the current key domain key, which has not yet been changed to reflect the new owner of the key domain. The read operation will be unsuccessful because the current integrity check value for the memory location will belong to the previous key domain. The integrity check will fail because the reader cannot read data belonging to another key domain.

[0238] To mitigate this problem, when a new key domain is established, the owner of the new key domain writes new data to a memory location within the key domain without attempting to read the memory location first. On the write operation, a new integrity check value (ICV) will be calculated for the new key domain; thus, the owner of the new key domain will now own the memory content (and will be able to read and write to the memory location without integrity failures).

[0239] In one embodiment, the MOVNT instruction is used to perform the first write operation to a new memory address. The memory encryption engine (TMEi) writes the encrypted data and ICV to the memory, thereby completing the process of copying data from the unencrypted memory to the encrypted memory that is part of the key domain of the consumer VM.

[0240] The MOVNT instruction is a write-combining operation, which means that the MOVNT instruction does not require a read operation to fetch the memory content because the current content is not needed. The MOVNT instruction can bypass the cache and write directly to the memory. As an alternative to using the MOVNT instruction, the running domain manager (VMMlet) can use an uncacheable write operation to copy data from the unencrypted memory to the encrypted memory that is part of the key domain of the consumer VM. By writing to a memory address without first reading from that memory address, a new integrity check value (ICV) is created (via the write for ownership).

[0241] Once the full consumer domain (VM) image is installed, the domain (VM) will act as a normal VM by using a secret key to establish secure communication with the consumer, third parties authorized by the consumer, and other authorized VMs. Secure storage is achieved by encrypting the full volume and / or files in the file system using the consumer secret key. Secure communication is achieved via IPSec / TLS and the consumer secret key. Authentication is achieved by using the consumer's secret key (PKI, etc.). Secure migration of the domain (VM) between servers in the cloud service provider's infrastructure can be achieved as follows: use the consumer's secret key to encrypt the VM image pages (and compute integrity check values for those VM image pages). The VM image pages can then be sent to other servers in the cloud service provider's infrastructure, along with the consumer's domain manager (VMMlet) pages, for securely migrating the consumer's VM image from one server to another.

[0242] Figure 27is a diagram that shows a message for a running domain manager (VMMlet) to request more memory pages from a memory manager software of a cloud service provider among components in a cloud service provider environment. In this environment, multiple CPUs share memory 2712 simultaneously. Cloud provider software (such as a memory manager) 2740 runs on a first CPU1 of a server that can have a key domain. A virtual machine 2730 runs on a second CPU2 of a server that can have a key domain. VM 2730 requests additional memory, as shown in the "Request more memory" action between VM 2730 and memory manager 2740. (In fact, the operating system, which is part of VM 2730 running on top of the VMMlet, may require more memory. The operating system can cause a VMExit from VM 2730, thereby invoking the host VMMlet, which then requests more memory from the cloud provider's memory manager 2740.) The domain manager (VMMlet) running on CPU2 writes a request on behalf of the consumer's VM 2730 via a shared unencrypted memory (where k bits are disabled), as shown by the "!k request message" action between VM2730 and the memory encryption engine (TMEi) 2715. The memory encryption engine (TMEi) 2715 passes the memory request to the shared memory 2712 without processing the request because the k bits are disabled. The memory manager 2740 on CPU1 reads the request for additional memory written by VM 2730 on CPU2, as indicated by the dashed line from memory 2712 to the memory manager 2740 on CPU1. The domain image (i.e., VM 2730) running on CPU2 waits for a response, such as an interrupt (IPI), as shown in the "Wait for response, such as interrupt (IPI)" action of VM 2730. When the cloud service provider's memory manager software 2740 provides an idle memory location, the memory manager 2740 on CPU1 writes response data to the shared memory 2712 (where k bits are disabled), as shown by the dashed line from the memory manager 2740 on CPU1 to the shared memory 2712. VM 2730 on CPU2 reads the response data from the shared memory 2712 (where k bits are disabled), as shown in the "Read response data" action between the shared memory 2712 and the memory encryption engine (TMEi) 2715. The memory encryption engine (TMEi) 2715 passes the response data until VM 2730 on CPU2, as shown by the "!k response message" action from TMEi 2715 to VM 2730 on CPU2. VM 2730 updates the page table in the key domain of the VM, as shown by the "Update page table in the key domain of the VM" action between VM 2730 and the memory encryption engine (TMEi) 2715.The Memory Encryption Engine (TMEi) 2715 writes the encrypted data and the integrity check value to the memory 2712 (with k bits enabled), as shown by the "Write Encrypted Data and ICV" action between the Memory Encryption Engine (TMEi) 2715 and the shared memory 2712. The domain manager (VMMlet) of the hosted VM 2730 causes the CPU 2 to execute the MOVNT instruction to write data to newly allocated memory in the key domain of the VM, as indicated by the "MOVNT to Newly Allocated Memory in the Key Domain of the VM" action between the VM 2730 and the Memory Encryption Engine (TMEi) 2715. In response, the Memory Encryption Engine (TMEi) 2715 writes the encrypted data and the ICV to the newly allocated encrypted memory.

[0243] Figure 28 Is a diagram that shows the messages between the components of a cloud service environment that shows a request for an additional memory page when VMs are scheduled on a single CPU. The memory manager 2840 of the cloud service provider decides on the scheduling scheme regarding which VMs are currently executing. The domain manager (VMMlet) running on the CPU / core of the cloud service provider receives timer events and generates times for other VMs based on the memory manager command queue (a shared memory area with k bits disabled). The "Switch Key Domain (SwitchKD)" operation is used to switch to another domain manager (VMMlet).

[0244] Reference Figure 28, the VM 2830 prepares a message to request additional memory from the memory manager 2840 in the "Prepare message for memory manager" action, and places the message in the cache 2813 in the "!k request in cache" action between the VM 2830 and the memory encryption engine (TMEi) 2815 to be read into the unencrypted (k-bit disabled) memory 2812. The memory encryption engine (TMEi) 2815 writes the request data into the memory 2812 (where k bits are disabled) in the "Write request data" action between the memory encryption engine (TMEi) 2815 and the memory 2812. The VM 2830 saves the processor state in the "Save processor state" action and places the saved VM processor state in the cache 2813 in the "Save state in VM's KD k-bit enabled". The saved state of the VM's processor is written into the key domain encrypted (k-bit enabled) memory of the VM. The memory encryption engine (TMEi) 2815 writes the saved VM processor state as encrypted data and ICV into the key domain encrypted (k-bit enabled) memory 2812 of the VM in the "Write encrypted data and ICV" action between the memory encryption engine (TMEi) 2815 and the memory 2812. Saving the processor state of the VM enables the domain manager (VMMlet) to later resume the execution of the VM by using the saved processor state. In one embodiment, the domain manager (VMMlet) clears the registers after the processor state has been saved, so that the secrets of the VM are not available after switching to another key domain.

[0245] To enable the memory manager 2840 to allocate additional memory for the VM 2830, the key domain is switched from the VM2830 key domain to the memory manager 2840 key domain. In the "SwitchKD to provider" action between the VM 2830 and the memory manager 2840, the VM 2830 sends a "Switch key domain" instruction to the cloud service provider. The memory manager 2840 starts restoring the processor state associated with the memory manager 2840 key domain in the "Restore processor state" action. In the "Read encrypted data and ICV" action between the memory 2812 and the memory encryption engine (TMEi) 2815, the encrypted data and integrity check value for the memory manager 2830 key domain are read from the memory 2812. The memory encryption engine (TMEi) 2815 decrypts the data for the current key domain and sends the decrypted data to the cache (assuming the corresponding ICV value is correct). The memory manager 2840 restores the processor state from the memory manager 2840 key domain in the "Restore state from memory manager's KD" action between the memory encryption engine (TMEi) 2815 and the memory manager 2840.

[0246] In the "Read Request Data" operation between the memory 2812 and the Memory Encryption Engine (TMEi) 2815, the Memory Encryption Engine (TMEi) 2815 reads additional memory request data from the k-bit disabled command queue of the memory 2812 into the cache 2813. In the "!k Data Request in Cache" operation between the Memory Encryption Engine (TMEi) 2815 and the Memory Manager 2840, the Memory Manager 2840 reads the additional memory data request stored in the cache 2813.

[0247] In the "!k Provide Free Memory Location" operation between the Memory Manager 2840 and the Memory Encryption Engine (TMEi) 2815, the Memory Manager 2840 sends a message via the unencrypted (k-bit disabled) memory command queue to provide a free memory location to the Memory Encryption Engine (TMEi) 2815. In the "Write Response Data" operation between the Memory Encryption Engine (TMEi) 2815 and the memory 2812, the Memory Encryption Engine (TMEi) 2815 writes response data to the memory 2812, including the address of the free memory location allocated for the VM 2830. In response to the completion of the allocation of additional memory in response to a request from the VM 2830, the Memory Management Engine 2840 saves the current processor state in the key domain of the controller in the "Save State in the KD of the Memory Manager" operation. In the "Write Encrypted Data and ICV" operation between the Memory Encryption Engine (TMEi) 2815 and the memory 2812, the Memory Encryption Engine (TMEi) 2815 writes the encrypted data (the saved processor state) and the integrity check value to the key domain of the Memory Manager 2840 in the memory 2812. The Memory Manager 2840 then performs the "Switch Key Domain (SwitchKD)" operation to switch back to the key domain of the VM.

[0248] In response to switching to the VM 2830 key domain, the VM 2830 starts restoring the saved processor state of the VM2830 in the "Restore Processor State" operation. The Memory Encryption Engine (TMEi) 2815 reads the encrypted data (including the processor state) and the integrity check value for the VM 2830 key domain in the "Read Encrypted Data and ICV" operation between the Memory Encryption Engine (TMEi) 2815 and the memory 2812. In the "Restore State from the VM's KD" operation between the Memory Encryption Engine (TMEi) 2815 and the VM 2830, the VM 2830 restores the saved processor state from its VM 2830 key domain.

[0249] When the VM 2830 was executed earlier before switching to the memory manager 2840 key domain, the VM 2830 had requested additional memory. In the "Read Response Data" operation between the memory encryption engine (TMEi) 2815 and the memory 2812, the memory encryption engine (TMEi) 2815 reads the response data for the additional memory request and provides the response data in the cache 2813 for the VM 2830. The VM 2830 updates the page table in the VM 2830 key domain in the "Update Page Table in VM's KD" between the VM 2830 and the memory encryption engine (TMEi) 2815 to reflect the newly allocated memory location. In the "Write Encrypted Data and ICV" operation between the memory encryption engine (TMEi) 2815 and the memory 2812, the memory encryption engine (TMEi) 2815 writes the encrypted data (the updated page table) and the integrity check value to the memory 2812.

[0250] To establish ownership of the newly allocated memory, in the "MOVNT to Newly Allocated Memory in VM's KD" operation between the VM 2830 and the memory encryption engine (TMEi) 2815, the VM 2830 then performs a MOVNT operation (or other write operation that does not read the content of the memory location before writing to the memory location) on the newly allocated memory in the VM's key domain. The MOVNT operation establishes the VM 2830 as the owner of the newly allocated memory. In the "Write Encrypted Data and ICV" operation between the memory encryption engine (TMEi) 2815 and the memory 2812, the memory encryption engine (TMEi) 2815 writes the encrypted data and the ICV to the memory 2812. As part of this write operation, the memory encryption engine (TMEi) 2815 calculates a new integrity check value for the newly allocated memory in the VM 2830 key domain. The new integrity check value will ensure that the VM 2830 key domain key can be used to decrypt the content of the newly allocated memory.

[0251] Figure 29 It is a diagram that shows the running domain manager (VMMlet) 2922. Before running the domain manager (VMMlet) 2922, the memory manager 2940 of the server that can have a key domain verifies the hash of the processor state before executing the domain launch image for the domain manager (VMMlet). Once the processor state is verified, the domain launch image is executed to run the domain manager (VMMlet).

[0252] The memory manager 2940 issues commands to the running domain manager (VMMlet) 2922 via the unencrypted (k-bit disabled) memory 2912. Similarly, the hardware 2910 of a server capable of having key domains issues direct memory access (DMA) requests to the running domain manager (VMMlet) via the unencrypted (k-bit disabled) memory 2912. In response to receiving these commands or DMA requests, the domain manager (VMMlet) 2922 interacts with the server hardware 2910 capable of having key domains to set and / or access register values, process interrupts, perform VM entry and exit, and so on.

[0253] The memory manager 2940 determines a scheduling scheme regarding which VM is currently executing; in Figure 29 the currently executing VM is VM2 2930, and the associated key domain is key domain 2950. The "Switch Key Domain (SwitchKD)" operation is used to switch to another domain (VM).

[0254] Once the dynamic portion of the VM image (i.e., the remaining portion of the VM image not included in the domain launch image) is loaded, dynamic entry points can be created locally within the VM. For example, in response to a "Switch Key Domain (SwitchKD)" instruction, a new keyed hash message authentication code (HMAC) can be computed based on the key domain key.

[0255] Interrupt and VM exit instructions are delivered to the current domain manager (VMMlet) running on the CPU / core of a server capable of having key domains. The running domain manager (VMMlet) determines whether the interrupt / asynchronous event is intended for the current running domain manager (VMMlet) or for another domain manager (VMMlet). If the interrupt / asynchronous event is intended for another domain manager (VMMlet), the domain manager (VMMlet) will schedule the correct domain manager (VMMlet) or notify the memory manager.

[0256] Regarding resource management, paging is implemented through software rather than via the hardware paging mechanism. The domain manager (VMMlet) encrypts the pages (including integrity metadata) in software (e.g., by using Intel® Advanced Encryption Standard New Instructions (AESNI) for accelerating AES encryption), updates the page tables and extended page tables, and sends the encrypted pages via the k-bit disabled memory for storage or migration.

[0257] For input / output operations, direct assignment or virtualized device models can be used. The k-bit specified unencrypted memory regions for DMA and memory-mapped input / output (MMIO) are not encrypted. Although direct assignment of DMA is possible, the MMIO / PCIe device space must be k-bit disabled (unencrypted) memory. The processor must ensure that key domain transactions are only permitted to dynamic random access memory (DRAM) and not to the device space.

[0258] Figure 30 is a diagram that shows multiple virtual machines within a key domain managed by a domain manager (VMMlet) and a second key domain managed by another type of domain manager (OSlet).

[0259] Since the domain manager (VMMlet) is a full-functional VMM, the domain manager (VMMlet) can host multiple guest operating systems (OS) / VMs within its key domain. VM2 3033 and VM3 3034 are shown as running within the key domain KD1 3050 of the domain manager (VMMlet) 3022 2 and process 3031 is shown as running within the key domain KD2 3050 of the OSlet 3060 1 As is the case when switching between domain managers (VMMlets), the memory manager 3040 issues a "Switch Key Domain (SwitchKD)" command to switch between domain types; that is, the SwitchKD command is issued to switch between the domain manager (VMMlet) and the domain manager (OSlet).

[0260] The consumer wants to be ensured that the public cloud service provider cannot access its workload, even when served by a government that issues approvals to do so. In the case of having the features provided by the secure public cloud environment described herein, the cloud service provider, on-site administrator, or technician cannot access the secure VM data, even if the VMM itself is rewritten (since the consumer can measure its entire TCB).

[0261] The above embodiments have been described with respect to a domain manager (VMMlet) for managing virtual machines, although the present invention is not so limited. The same model can support containers; although there is no corresponding VMM, the OS kernel is equivalent. Each container image within each key domain will have a collaborative kernel component (which is referred to herein as a domain manager (OSlet)) that is measured by the provider. The domain manager (OSlet) responds to memory manager commands, interrupts, scheduling, resource management, etc. in a manner similar to the domain manager (VMMlet).

[0262] Figure 31Aand 31B Illustrated is a hardware function as a memory encryption engine to determine a full row location and a slot based on a physical memory address. Unused address bits are passed through the cache, but they are not used because they correspond to unfilled physical memory. The unused bits are used to encode key domain (KD) selector information in the address. Different keys can be selected based on the unused address bits related to memory location encryption of the data row and the corresponding integrity check value.

[0263] According to an embodiment, the physical memory address 3100 can be used to determine the key or fine-tuning discussed above, and / or the integrity check row 3112 and slots 3114 (3114a - 3114h) for the integrity value associated with the data row 3116 (3116a - 3116h). The physical memory address 3100 can include multiple address bits that can be partitioned into multiple segments. The segments of the data physical memory address 3100 can be identified as the data row byte 3102, the data row physical address 3104 (the actual location in the data memory) including the full row slot selector 3110 and the full row index 3108 (e.g., the offset for the integrity check row 3112), and the unused address bits 3106 (e.g., alias bits) that alias to the same physical memory. The unused address bits 3106 are passed through the cache but are not used because unfilled external memory can be used to encode the alias information in the data physical memory address 3100. Thus, different keys can be selected based on the unused address bits. For example, the encryption technique XTS (tweakable codebook mode based on XEX (XOR-Encrypt-XOR) with ciphertext stealing) can use the alias bits for fine-tuning for the same physical memory location, where different address aliasing can result in different ciphertexts even if the data is the same.

[0264] A memory selector (e.g., Figure 4 the memory encryption engine 415) can read the data row physical address 3104 and identify the corresponding full row address 3112 and full row slots (e.g., 3114a - 3114h) to verify the validity of the data row byte 3102 and / or the integrity check value stored in the full row slots (e.g., 3114a - 3114h). The value in the alias bits 3106 can be stored in the full row slots (e.g., 3114a - 3114h) as ciphertext, which is used to decrypt the data row byte, and / or compared with the value read from the full row slots (e.g., 3114a - 3114h) identified by the alias bits 3106 to verify the data row byte.

[0265] Obviously, not all bits of the memory can be addressable because, for example, the actual memory deployed in a computing platform may be significantly smaller than the maximum amount of possible memory for which the maximum amount of address space is provided. For example, not all 64 bits (of a 64-bit system) of physical memory are addressable (e.g., occupied by sufficient DIMMs). Thus, the otherwise unused bits of physical memory address 3100 can be used to determine, for example, which key and / or tweak will be used when encrypting and / or decrypting the memory for a particular data row.

[0266] The key domain and / or tweak domain for physical memory address 3100 can be of any size. In the illustrated example, the value selector can use the unused address bit 3106 to derive the key and / or tweak for the same physical memory address 3100. For example, a software value selector can select from among 16 keys (and / or 16 tweaks) defined by the four most significant bits of the unused address bit 3106. In one example, setting the first bit to zero (0000) or to one (0001) can be used to derive key tweak (e.g., encrypt with 1 if the bit is set to 1, encrypt with 0 if set to 0), or obtain a tweak (e.g., use 1 in the address for tweak if the bit is set to 1, use 0 in the address for tweak if set to 0, etc.). Thus, different keys and / or tweaks can be used. In this case, when decrypting data with the wrong key and / or wrong tweak, the first integrity check will fail, and / or when checking the integrity value against the inappropriately decrypted data, the second integrity check will fail.

[0267] Additionally, integrity check rows and / or slots can be determined and selected based on physical memory address 3100. For example, a full row selector can select an integrity check row from the full row index section 3108 of physical memory address 3100, and / or a slot selector can select a slot from the full row slot selector section 3110 of physical memory address 3100.

[0268] As Figure 31B shown, data rows can be stored in a data memory address space, and integrity values (e.g., ICV, copy, etc.) can be stored in an integrity data address space. For example, data rows in the data memory address space start at address 0, and integrity check rows in the integrity data memory address space start 1000 cache lines away from the data row at address 1000.

[0269] While in embodiments various strategies can be implemented to map between each data row and each integrity check row (and / or each of its slots), using the data row address can be an efficient way to determine and select the appropriate integrity check row and appropriate slot. For example, it may not be necessary to have any lookup tables to determine the appropriate integrity check row and / or slot for an integrity value. In this regard, the value defined by the middle bit of each of data rows 3116a - 3116h can be mapped to integrity check row 3112, as indicated by the arrows from data rows 3116a - 3116h with addresses 0 - 7 to integrity check row 3112; and the value defined by the least significant bit of each of data rows 3116a - 3116h can be mapped to the appropriate slots 3114a - 3114h, which will accommodate the specific integrity value for each of data rows 3116a - 3116h, as indicated by the position of the arrows.

[0270] Generally, the selection of the appropriate integrity check row and appropriate slot can be based on a function, such as (D - Dstart) / 8 + Istart, where the starting address Dstart of the data memory region is subtracted from the address D of the data row to be accessed, where Istart is the start of the integrity value memory address space, and where the integer division by 8 can be performed by shifting the address offset 3 bits to the right (or selecting the top bits minus the first 3 bits). Additionally, once the appropriate integrity check row is fetched, the offset for the appropriate slot can be determined by (D - Dstart)%8, where the modulo operation can select the least significant 3 bits of the address. It should be understood that while 3 bits can be used to select from 8 slots on an integrity check row, the integrity check rows can be of different sizes (e.g., half the size), such that 4 bits can be used to select from 16 slots per integrity check row to save on integrity value overhead, and so on.

[0271] The middle bit and / or the least significant bit can also be used as an index into an array of assigned locations stored in privileged / secure memory locations to identify the mapping. There can also be an implicit mapping, where the first slot 3114a of the first integrity check row 3112 can be automatically selected for data row 3116a with address 0, the second slot 3114b of the first integrity check row 3112 can be automatically selected for data row 3116b with address 1, and so on. Any function, mapping, and / or assignment can be used such that data rows 3116a - 3116h with addresses 0 - 7 can be mapped anywhere in the integrity data address space, can be mapped anywhere within integrity check row 3112, and so on.

[0272] Implement a secure public cloud environment without incurring additional performance overhead beyond that of the memory encryption engine. In one embodiment, the memory encryption engine is provided as a Memory Encryption Engine (MEE), as described in U.S. Patent No. 8,819,455, "Parallelized Counter Tree Walk for Low Overhead Memory Replay Protection". In another embodiment, a memory encryption engine with integrity is provided, as described in U.S. Patent 9,213,653, "Memory Integrity". In one implementation of a secure public cloud environment with a Total Memory Encryption with Integrity (TMEi) engine, the TMEi engine operates with only a 3% overhead. Ultimately, only minimal hardware changes are used to secure the public cloud environment, which employs a memory encryption engine, such as the TMEi engine, and pushes most of the complexity to the software (specifically the VMM). These features allow for simple verification of the VMM and a fast time to market for the hardware supporting the secure public cloud environment functionality.

[0273] Figure 32 is a diagram that illustrates a system according to one embodiment. As can be seen, system 3200 can be a smart phone or other wireless communicator or any other IoT device. The baseband processor 3205 is configured to perform various signal processing on communication signals that will be transmitted from or received by the system. Further, the baseband processor 3205 is coupled to an application processor 3210, which can be the main CPU of a system for executing the OS and other system software in addition to user applications, such as many well-known social media and multimedia apps (applications). The application processor 3210 can additionally be configured to perform various other computing operations for the device.

[0274] Further, the application processor 3210 can be coupled to a user interface / display 3220, such as a touch screen display. Additionally, the application processor 3210 can be coupled to a memory system, including non-volatile memory, i.e., flash memory 3230, and system memory, i.e., DRAM 3235. In some embodiments, the flash memory 3230 can include a secure section 3232, where keys, other secrets, and other sensitive information can be stored and operated on. One or more of these storage devices can store information for providing the secure public cloud described herein. As also seen, the application processor 3210 is also coupled to a capture device 3245, such as one or more image capture devices, which can record video and / or still images.

[0275] Still referring to Figure 32, a Universal Integrated Circuit Card (UICC) 3240 includes a subscriber identity module which, in some embodiments, includes a secure storage device 3242 to store secure identity information. The system 3200 may further include a secure processor 3250 which may implement a Trusted Execution Environment (TEE), and the secure processor 3250 may be coupled to an application processor 3210. Additionally, the application processor 3210 may implement a secure operating mode, such as Intel® Software Guard Extensions (SGX) for a given instruction set architecture, and circuitry for hosting a Trusted Execution Environment (TEE). The secure processor 3250 and / or the application processor 3210 may be configured to participate in operations to support providing a secure public cloud as described herein. A plurality of sensors 3225, including one or more multi-axis accelerometers, may be coupled to the application processor 3210 to enable input of various sensed information, such as motion and other environmental information. Additionally, one or more authentication devices 3295 may be used to receive, for example, user biometric input for use in authentication operations.

[0276] As further illustrated, a Near Field Communication (NFC) contactless interface 3260 is provided which communicates in the NFC near field via an NFC antenna 3265. While separate antennas are shown in Figure 4 , it is understood that in some implementations, one antenna or a different set of antennas may be provided to enable implementation of various types of wireless functionality. A Power Management Integrated Circuit (PMIC) 3215 is coupled to the application processor 3210 to perform platform-level power management. To this end, the PMIC 3215 may issue power management requests to the application processor 3210 to enter certain low-power states as desired. Additionally, based on platform constraints, the PMIC 3215 may also control the power levels of other components of the system 3200.

[0277] In order to enable communication to be transmitted and received, for example, in one or more IoT networks, various circuits can be coupled between the baseband processor 3205 and the antenna 3290. In particular, there can be a radio frequency (RF) transceiver 3270 and a wireless local area network (WLAN) transceiver 3275. Generally, the RF transceiver 3270 can be used to receive and transmit wireless data and calls according to a given wireless communication protocol, such as a 3G or 4G wireless communication protocol, such as according to code division multiple access (CDMA), global system for mobile communications (GSM), long term evolution (LTE), or other protocols. Additionally, there can be a GPS sensor 3280, where location information is provided to the security processor 3250, which can be used in certain security operations. Other wireless communications can also be provided, such as receiving or transmitting radio signals, such as AM / FM and other signals. Additionally, via the WLAN transceiver 3275, local wireless communication can also be achieved, such as according to the Bluetooth™ or IEEE 802.11 standards.

[0278] Now referring to Figure 33 , a block diagram of a system according to another embodiment of the present invention is shown. As Figure 32 shown, the multi-processor system 3300 can be implemented as a point-to-point interconnect system, such as a server system capable of having a key domain. The system 3300 includes a first processor 3370 and a second processor 3380 coupled via a point-to-point interconnect 3350. As Figure 5 shown, each of the processors 3370 and 3380 can be a multi-core processor, such as a SoC, including first and second processor cores (i.e., processor cores 3374a and 3374b, and processor cores 3384a and 3384b), although potentially many more cores can be present in the processors. Additionally, each of the processors 3370 and 3380 can include security engines 3375 and 3385 to perform security common cloud operations as described herein.

[0279] Still referring to Figure 33 , the first processor 3370 further includes a memory controller hub (MCH) 3372 and point-to-point (P-P) interfaces 3376 and 3378. Similarly, the second processor 3380 includes an MCH 3382 and P-P interfaces 3386 and 3388. As Figure 33 shown, the MCHs 3372 and 3382 couple the processors to the respective memories, i.e., memories 3332 and 3334, which can be portions of the main memory (e.g., DRAM) locally attached to the respective processors. The first processor 3370 and the second processor 3380 can be coupled to the chipset 3390 via P-P interconnections 3352 and 3354, respectively. As Figure 33As shown, chipset 3390 includes P-P interfaces 3394 and 3398.

[0280] In addition, chipset 3390 includes interface 3392 to couple chipset 3390 to high-performance graphics engine 3338 via P-P interconnect 3339. Further, chipset 3390 may be coupled to first bus 3316 via interface 3396. As Figure 33 shown, various input / output (I / O) devices 3314 may be coupled to first bus 3316, along with bus bridge 3318 that couples first bus 3316 to second bus 3320. Various devices may be coupled to second bus 3320, including, for example, keyboard / mouse 3322, communication device 3326, and data storage unit 3328, such as a non-volatile storage device or other mass storage device. As seen, data storage unit 3328 may include code 3330, which in one embodiment includes code for performing the secure public cloud operations described herein. Also as seen, data storage unit 3328 further includes trusted storage device 3329 for storing sensitive information to be protected. In addition, audio I / O 3324 may be coupled to second bus 3320.

[0281] Embodiments may be used in an environment where IoT devices may include wearable devices or other small form factor IoT devices, such as actuators and / or sensors. Now refer to Figure 34, shown is a block diagram of module 3400 according to another embodiment. In one particular implementation, module 3400 can be an Intel® Curie™ module, which includes multiple components adapted within a single small module. Module 3400 can be configured to participate in the secure public cloud operations described herein. As can be seen, module 3400 includes a core 3410 (of course, in other embodiments, there can be more than one core). Such a core can be a relatively low-complexity in-order core, such as based on the Intel Architecture® Quark™ design. In some embodiments, core 3410 can implement a trusted execution environment. Core 3410 is coupled to various components, including a sensor hub 3420, which can be configured to interact with multiple sensors 3480, such as one or more biometric, motion environment, or other sensors. Along with non-volatile storage 3440, there is a power delivery circuit 3430. In an embodiment, the circuit can include a rechargeable battery and a recharging circuit, which can receive charging power wirelessly in one embodiment. There can be one or more input / output (IO) interfaces 3450, such as one or more interfaces compatible with one or more of the USB / SPI / I2C / GPIO protocols. Additionally, there is a wireless transceiver 3490, which can be Bluetooth™ Low Energy or other short-range wireless transceiver, to enable wireless communication as described herein. It is understood that in different implementations, IoT modules can take many other forms, which have a small form factor, low power requirements, a limited instruction set, a relatively slow computational throughput, or any of the above compared to a typical general-purpose CPU or GPU.

[0282] The following examples pertain to additional embodiments.

[0283] In Example 1, an apparatus for protecting consumer data in a public cloud environment includes: a processor; a memory coupled to the processor; and memory encryption logic coupled to the processor and the memory, wherein the processor is configured to execute instructions to create a key domain in the memory, wherein the instructions include an encrypted key domain key, the key domain includes multiple memory locations of the memory, execute the instructions to decrypt the encrypted key domain key to generate a key domain key for the key domain, and provide the key domain key to the memory encryption logic, and the memory encryption logic is configured to decrypt the first data at the first memory location of the key domain by using the key domain key when reading the first data from the first memory location of the key domain, and the memory encryption logic is configured to encrypt the second data at the second memory location by using the key domain key when writing the second data to the second memory location.

[0284] In Example 2, the instruction further includes a key domain identifier for the key domain, the processor is configured to set unused bits of a physical address to the key domain identifier, the physical address identifies a memory location among the multiple memory locations of the key domain, the processor is configured to set one bit of the physical address to a value indicating the encrypted content of the memory location at the physical address to be read by using the key domain key, the memory encryption logic is configured to read the encrypted content from the memory location at the physical address by using the key domain key to generate decrypted key domain content, and the processor is configured to execute the decrypted key domain content.

[0285] In Example 3, the decrypted key domain content includes a domain image, the domain image includes code to be executed when the key domain is the current key domain, and a command for executing an instruction to create a key domain is issued by code external to the domain image.

[0286] In Example 4, the processor of Example 3 is further configured to execute a second instruction to verify the encrypted key domain content while maintaining the confidentiality of the key domain key and the decrypted key domain content, wherein a command for executing the second instruction is issued by code external to the key domain image.

[0287] In Example 5, the second instruction includes a key domain identifier for the key domain, and the processor is configured to hash the decrypted key domain content to generate a hash result.

[0288] In Example 6, the processor of Example 5 is further configured to: execute a third instruction to switch from a first key domain to a second key domain, wherein executing the third instruction causes the processor to execute code at an authorized memory location in a memory belonging to the second key domain, and the content of the authorized memory location is subsequently encrypted or decrypted by using a second key domain key for the second key domain.

[0289] In Example 7, maintaining the confidentiality of the key domain key includes providing the key domain key to the memory encryption logic as an encrypted key domain key such that software executed on the processor cannot access the key domain key, and maintaining the confidentiality of the decrypted key domain content includes providing the encrypted key domain content to the memory encryption logic as an encrypted key domain content such that software executed on the processor cannot access the decrypted key domain content.

[0290] Note that the above processor can be implemented by using various means. In an example, the processor includes a system-on-chip (SoC) incorporated in a touch-enabled device of a user equipment. In another example, a system includes a display and a memory, and includes the processor of one or more of the above examples.

[0291] In Example 8, a method for protecting consumer data in a public cloud environment includes: providing a domain manager image to the consumer in response to a request from the consumer for a service; allocating a portion of a memory for the domain manager image and providing the consumer with address information related to memory location for this portion of the memory; loading the encrypted domain launch image provided by the consumer into this portion of the memory; sending a command to a processor to create a key domain, where the command includes an encrypted key domain key, the key domain includes a plurality of locations of the memory, and each memory location of the key domain is encrypted by the key domain key for the key domain; sending a second command to the processor to cryptographically verify that the encrypted domain launch image includes the domain manager image; and if the processor verifies that the encrypted domain launch image includes the domain manager image, then execute the domain launch image resulting from decrypting the encrypted domain launch image.

[0292] In Example 9, decrypting the encrypted domain launch image to produce a domain launch image includes comparing the content of the domain launch image with an integrity check value provided by the consumer for the encrypted domain launch image, and if the content of the domain launch image does not match the integrity check value, then abort the execution of the domain launch image.

[0293] In Example 10, the processor of Example 8 further verifies, in response to the second command, that an entry point address of the memory portion into which the encrypted domain launch image is loaded matches the specified entry point address in the address information related to memory location.

[0294] In Example 11, the method of Example 8 further includes: sending a third command to the processor to switch from a first key domain to a second key domain, where switching to the second key domain causes the processor to execute code at an authorized memory location in the memory belonging to the second key domain, and the content of the authorized memory location is subsequently encrypted or decrypted using a second key domain key for the second key domain.

[0295] In Example 12, before switching to the second key domain, the processor determines whether the current processor state matches an expected processor state for the second key domain, and if the current processor state matches the expected processor state, then switches to the second key domain.

[0296] In Example 13, an expected processor state for the second key domain is provided to the processor as a hash value computed by a secure hash function of the expected processor state for the second key domain, and the hash value is encrypted using the second key domain key.

[0297] In Example 14, the method of Example 8 further includes: receiving, from the consumer, the remainder of the domain image, wherein the encrypted domain launch image includes a domain manager portion of the domain image; decrypting or validating at least one page of the remainder of the domain image by using a secret key included in the encrypted domain launch image; and executing the remainder of the domain image.

[0298] In Example 15, the method of Example 8 further includes: decrypting the encrypted key domain key to produce a key domain key; performing read / write operations on corresponding data in corresponding memory locations of the key domain by using the key domain key; and determining an integrity check value for the corresponding data in the corresponding memory locations of the key domain when performing the read / write operations.

[0299] In Example 16, an apparatus for protecting consumer data in a public cloud environment includes: means for providing a domain manager image to the consumer in response to a consumer request for a service; means for allocating a portion of memory for the domain manager image and providing the consumer with address information related to the location of that portion of memory; means for loading the encrypted domain launch image provided by the consumer into that portion of memory; means for sending a command to a processor to create a key domain, wherein the command includes an encrypted key domain key, the key domain includes a plurality of locations of the memory, and each memory location of the key domain is encrypted by the key domain key for the key domain; means for sending a second command to the processor to cryptographically verify that the encrypted domain launch image includes the domain manager image; and means for executing the domain launch image produced by decrypting the encrypted domain launch image if the processor verifies that the encrypted domain launch image includes the domain manager image.

[0300] In Example 17, decrypting the encrypted domain launch image to produce a domain launch image includes comparing the content of the domain launch image with an integrity check value provided by the consumer for the encrypted domain launch image, and if the content of the domain launch image does not match the integrity check value, aborting the execution of the domain launch image.

[0301] In Example 18, the processor of Example 16 or 17 further verifies, in response to a second command, that an entry point address of a memory portion into which the encrypted domain launch image is loaded matches a specified entry point address in the memory location-related address information.

[0302] In Example 19, the apparatus of claim 16 further includes: means for sending a third command to the processor to switch from a first key domain to a second key domain, wherein switching to the second key domain causes the processor to execute code at an authorized memory location in a memory belonging to the second key domain, and wherein the content of the authorized memory location is subsequently encrypted or decrypted using a second key domain key for the second key domain.

[0303] In Example 20, before switching to the second key domain, the processor determines whether a current processor state matches an expected processor state for the second key domain, and switches to the second key domain if the current processor state matches the expected processor state.

[0304] In Example 21, the expected processor state for the second key domain is provided to the processor as a hash value computed by a secure hash function of the expected processor state for the second key domain, and the hash value is encrypted using the second key domain key.

[0305] In Example 22, the apparatus of Example 16 further includes: means for receiving the remainder of the domain image from the consumer, wherein the encrypted domain launch image includes a domain manager portion of the domain image; means for decrypting or verifying at least one page of the remainder of the domain image using a secret key included in the encrypted domain launch image; and means for executing the remainder of the domain image.

[0306] In Example 23, the apparatus of Example 16 further includes: means for decrypting the encrypted key domain key to produce a key domain key; means for performing read / write operations on corresponding data in corresponding memory locations of the key domain using the key domain key; and means for determining an integrity check value for the corresponding data in the corresponding memory locations of the key domain when performing the read / write operations.

[0307] In Example 24, a computer-readable medium includes computer-readable instructions that, when executed, are used to implement the method as described in any one of Examples 8-15.

[0308] In Example 25, an apparatus includes means for performing the method as described in any one of Examples 8-15.

[0309] In Example 26, an apparatus for protecting consumer data in a public cloud environment includes: a processor; a memory coupled to the processor; and memory encryption logic coupled to the processor and the memory, wherein the processor is configured to execute instructions to create a key domain in the memory, wherein the instructions include an encrypted key domain key, the key domain includes a plurality of memory locations of the memory, execute instructions to decrypt the encrypted key domain key to generate the key domain key and provide the key domain key to the memory encryption logic, and when writing corresponding data to a corresponding memory location, the memory encryption logic encrypts the corresponding data of the corresponding memory location of the key domain by using the key domain key.

[0310] In Example 27, the processor is further configured to execute second instructions to verify the key domain by hashing the contents of the plurality of memory locations to generate a hash result and comparing the hash result with an expected hash result, wherein when the hash result matches the expected hash result, the key domain is verified.

[0311] In Example 28, the processor is further configured to execute third instructions to switch from a first key domain to a second key domain, wherein a first memory location in the memory belongs to both the first key domain and the second key domain, when the first key domain is the current key domain, encrypt or decrypt the contents of the first memory location by using a first key domain key for the first key domain, and when the second key domain is the current key domain, encrypt or decrypt the contents of the first memory location by using a second key domain key for the second key domain.

[0312] Note that the above processor can be implemented by using various means. In an example, the processor includes a system-on-chip (SoC) incorporated in a user equipment touch-enabled device. In another example, a system includes a display and a memory, and includes a processor of one or more of the above examples.

[0313] In Example 29, a method for protecting consumer data in a public cloud environment includes: receiving domain manager image and memory location-related address information in response to a request for a service from a cloud service provider; verifying the domain manager image; identifying and encrypting a key domain key, where the key domain key will be used to encrypt data stored in the key domain, and the key domain includes multiple memory locations of the memory of a server capable of having a key domain; using the key domain key and the memory location-related address information to encrypt the domain launch image such that the encrypted domain launch image is cryptographically bound to at least one memory location of the key domain; sending the encrypted domain launch image and the encrypted key domain key to the server capable of having a key domain, where a processor of the server capable of having a key domain executes instructions to create the key domain.

[0314] In Example 30, the memory location-related address information includes an entry point address in the memory of the server capable of having a key domain; and the encrypted domain launch image is cryptographically bound to start execution at the entry point address.

[0315] In Example 31, the encrypted launch image includes the domain manager image; and the memory location-related address information includes a page table structure that maps the virtual addresses of a domain manager created by executing the domain manager image to the physical memory locations of the key domain.

[0316] In Example 32, the memory location-related address information includes an expected processor state that existed before the processor of the server capable of having a key domain could switch to the key domain; and if the expected processor state matches the current processor state of the processor, the processor switches to the key domain.

[0317] In Example 33, the method of Example 4 further includes modifying the domain manager image to include a secret key for functions provided by the rest of the consumer domain image, where the first part of the consumer domain image and the modified domain image are included in the encrypted domain launch image.

[0318] In Example 34, encrypting the key domain key includes encrypting the key domain key using the public key of the server capable of having a key domain, and the command includes the encrypted key domain key.

[0319] In Example 35, performing the instructions to create the key domain includes decrypting the encrypted key domain key by using the private key of the server capable of having a key domain, and providing the key domain key to a memory encryption engine associated with the processor and memory of the server capable of having a key domain, and when reading first data from a first memory location of the key domain, the memory encryption engine decrypts the first data stored in the first memory location by using the key domain key.

[0320] In Example 36, when second data is written to a second memory location of the key domain, the memory encryption engine encrypts the second data by using the key domain key.

[0321] In Example 37, the method of Example 29 further includes using the key domain key to calculate an integrity check value for at least one memory location of the key domain; and sending the integrity check value to the server capable of having a key domain.

[0322] In Example 38, the method of Example 29 further includes sending a message to a domain manager executing outside the key domain via a shared area of the memory of the server capable of having a key domain.

[0323] In Example 39, a method for protecting consumer data in a public cloud environment includes: providing a domain manager image to the consumer in response to a request by the consumer for a service; allocating a portion of memory for the domain manager image and providing the consumer with address information related to the memory location of the portion of memory; loading the encrypted domain launch image provided by the consumer into the portion of memory; executing the encrypted domain launch image; and sending a command to a processor coupled to the memory to create a key domain, where the command includes an encrypted key domain key, the key domain includes a plurality of locations of the memory, and each memory location of the key domain will be encrypted by the key domain key.

[0324] In Example 40, the method of Example 39 further includes: sending a command to the processor to verify that the encrypted domain launch image includes the domain manager image and the entry point address for the portion of memory into which the encrypted domain launch image is loaded matches a specified entry point address specified in the memory location related address information; and executing the domain manager image if the processor verifies that the encrypted domain launch image includes the domain manager image and the entry point address matches the specified entry point address.

[0325] In Example 41, the method of Example 39 further includes sending a command to the processor to switch from the key domain to a second key domain, wherein the processor executes instructions to determine whether the current processor state of the processor matches an expected processor state, and switches to the second key domain if the current processor state matches the expected processor state.

[0326] In Example 42, the method of Example 39 further includes sending a command to the processor to switch from the key domain to a memory manager key domain, wherein the processor executes instructions to determine whether the current processor state of the processor matches an expected processor state, and switches to the memory manager key domain if the current processor state matches the expected processor state.

[0327] In Example 43, the method of claim 39 further includes: receiving the remainder of the consumer domain image from the consumer, wherein the encrypted domain launch image includes a domain manager portion of the consumer domain image; and verifying at least one page of the remainder of the consumer domain image by using a secret key included in the encrypted domain launch image.

[0328] In Example 44, a computer-readable medium for protecting consumer data in a public cloud environment includes code that, when executed, causes a machine to perform the method of any of the above examples.

[0329] In Example 45, an apparatus for protecting consumer data in a public cloud environment includes components for performing the method of any of the above examples.

[0330] In Example 46, at least one computer-readable medium includes instructions that, if executed by a processor, cause a computer to: receive a domain manager image and memory location-related address information in response to a request for a service from a cloud service provider; verify the domain manager image; identify and encrypt a key domain key, wherein the key domain key will be used to encrypt data stored in the key domain, and the key domain includes a plurality of memory locations of a memory of a server capable of having a key domain; encrypt a domain launch image by using the key domain key and the memory location-related address information such that the encrypted domain launch image is cryptographically bound to at least one memory location of the key domain; and send the encrypted domain launch image and the encrypted key domain key to the server capable of having a key domain, wherein a processor of the server capable of having a key domain executes instructions to create the key domain.

[0331] In Example 47, the memory location-related address information includes the entry point address in the memory of the server capable of having a key domain; and the encrypted domain launch image is cryptographically bound to start execution at the entry point address.

[0332] In Example 48, the encrypted launch image includes the domain manager image; and the memory location-related address information includes a page table structure that maps the virtual address of the domain manager created by executing the domain manager image to the physical memory location of the key domain.

[0333] In Example 49, the memory location-related address information includes the expected processor state that existed before the processor of the server capable of having a key domain could switch to the key domain; and if the expected processor state matches the current processor state of the processor, the processor switches to the key domain.

[0334] In Example 50, the instructions of Example 46 further cause the computer to: modify the domain manager image to include a secret key for the functionality provided by the rest of the consumer domain image, wherein the first part of the consumer domain image and the modified domain image are included in the encrypted domain launch image.

[0335] In Example 51, encrypting the key domain key includes encrypting the key domain key using the public key of the server capable of having a key domain, and the command includes the encrypted key domain key.

[0336] In Example 52, executing the instructions to create the key domain includes: decrypting the encrypted key domain key by using the private key of the server capable of having a key domain, and providing the key domain key to the memory encryption engine associated with the processor of the server capable of having a key domain, and when reading first data from a first memory location of the key domain, the memory encryption engine decrypts the first data stored in the first memory location by using the key domain key.

[0337] In Example 53, when second data is written to a second memory location of the key domain, the memory encryption engine encrypts the second data by using the key domain key.

[0338] In Example 54, the instructions of Example 46 further cause the computer to: calculate an integrity check value for at least one memory location of the key domain by using the key domain key; and send the integrity check value to the server capable of having a key domain.

[0339] In Example 55, the instructions of Example 46 further cause the computer to send a message to a domain manager executing outside the key domain via a shared area of the memory of the server capable of having a key domain.

[0340] In Example 56, at least one computer-readable medium includes instructions that, if executed by a processor, cause the computer to: provide a domain manager image to the consumer in response to a request by the consumer for a service; allocate a portion of memory for the domain manager image and provide the consumer with address information related to the location of the memory for that portion of the memory; load the encrypted domain launch image provided by the consumer into that portion of the memory; execute the encrypted domain launch image; and send a command to a processor coupled to the memory to create a key domain, where the command includes an encrypted key domain key, the key domain includes a plurality of locations of the memory, and each memory location of the key domain will be encrypted by the key domain key.

[0341] In Example 57, the instructions further cause the computer to: send a command to the processor to verify that the encrypted domain launch image includes the domain manager image and that an entry point address for the portion of the memory into which the encrypted domain launch image is loaded matches a specified entry point address specified in the address information related to the location of the memory; and execute the domain manager image if the processor verifies that the encrypted domain launch image includes the domain manager image and the entry point address matches the specified entry point address.

[0342] In Example 58, the instructions further cause the computer to: send a command to the processor to switch from the key domain to a second key domain, where the processor executes instructions to determine whether the current processor state of the processor matches an expected processor state and, if the current processor state matches the expected processor state, switch to the second key domain.

[0343] In Example 59, the instructions further cause the computer to: send a command to the processor to switch from the key domain to a memory manager key domain, where the processor executes instructions to determine whether the current processor state of the processor matches an expected processor state and, if the current processor state matches the expected processor state, switch to the memory manager key domain.

[0344] In example 60, the instructions further cause the computer to: receive the remainder of the consumer domain image from the consumer, where the encrypted domain launch image includes the domain manager portion of the consumer domain image; and verify at least one page of the remainder of the consumer domain image by using a secret key included in the encrypted domain launch image.

[0345] In example 61, a consumer of a public cloud service can control the cloud environment in which the consumer's data is processed. In another example, the consumer can provide code to execute on the cloud service provider's server for establishing a secure environment for the consumer, which is referred to herein as the key domain. In still another example, the consumer can provide all of the code to execute within the cloud service provider's server software stack, including privileged code such as hypervisor (VMM) code, operating system code, application code, and so on.

[0346] In one example, the consumer creates a password-secure code / data image that can only be decoded and executed by the cloud service provider's server hardware by using an encrypted key domain key provided by the consumer. In another example, the consumer can securely exchange password-type key domain keys that are used to exclusively utilize the cloud service provider's server hardware on which the domain image will execute to create a secure password domain image for the consumer. In still another example, cloud service provider software, administrators, and technicians outside of the consumer's password-secure environment / key domain are not provided with the consumer's encrypted key domain key and do not have the ability to view, augment, or modify the contents of the consumer domain image (whether the domain image is in transit, during storage, or during execution).

[0347] In another example, the consumer can encrypt the domain image passwordically and provide only the encrypted domain image to the cloud service provider's server. In another example, the consumer can passwordically bind the domain image to a specific memory location / address on the cloud service provider's server on which the domain image will execute. This password binding will ensure that the domain image is executed within the consumer's secure environment / key domain at the memory location specified by the consumer.

[0348] In one example, the consumer can further enable the domain image to communicate with cloud service provider software and / or other domains outside of the consumer's secure environment / key domain via a specified shared memory region. In another example, the consumer can further enable the domain image to interact with a device, perform input / output (I / O), and communicate via exchanging messages, audio, video, and so on, via a specified shared memory region.

[0349] In one example, a public cloud service provider can verify a consumer's code to provide a secure environment / key domain for the consumer on the cloud service provider's server. In particular, the cloud service provider can verify via hardware that the portion of the domain image supplied by the consumer that includes privileged code is as expected. In another example, the cloud service provider can verify that the virtual machine monitor (VMM) code supplied by the consumer, which would typically be provided by the cloud service provider, is the same VMM code expected by the cloud service provider. This verification can be performed by the cloud service provider's hardware, even though the cloud service provider's software has never been provided with an unencrypted domain image or an unencrypted key domain key.

[0350] In another example, the cloud service provider's server can correctly enter and start executing the consumer's domain image, where the hardware checks that the consumer and the cloud service provider agree on how the domain image is initially executed and the correct state of the hardware (CPU) when entering the cryptographic domain image. In another example, the cloud service provider's hardware can securely switch from one consumer's domain image to another consumer's domain image. In another example, the cloud service provider's hardware can securely switch from one consumer's domain image to the cloud service provider's software (such as a memory manager). In yet another example, the cloud service provider's hardware can cryptographically verify the integrity of the domain image to detect tampering / modification by exchanging a cryptographic integrity check value or a message authentication code. In yet another example, the cloud service provider can prevent replay of the contents of the consumer domain image by allowing writing / configuring of the integrity check value, but blocking reading of the integrity check value outside the hardware.

[0351] In another example, a method includes providing an image to execute privileged code in a memory of a remote computing device, where the image is cryptographically bound to a first physical memory location of the remote computing device such that the privileged code will not execute if loaded into a second physical memory location of the remote computing device.

[0352] It is understood that various combinations of the above examples are possible.

[0353] Embodiments can be used in many different types of systems. For example, in one embodiment, a communication device can be arranged to perform the various methods and techniques described herein. Of course, the scope of the present invention is not limited to communication devices, but instead other embodiments can be directed to other types of apparatus for processing instructions, or one or more machine-readable media including instructions that, when executed on a computing device, cause the device to implement one or more of the methods and techniques described herein.

[0354] Note that the terms "circuit" and "circuitry" are used interchangeably herein. As used herein, these terms are used to refer, individually or in any combination, to analog circuits, digital circuits, hardwired circuits, programmable circuits, processor circuits, micro-memory manager circuits, hardware logic circuits, state machine circuits, and / or any other type of physical hardware component. Embodiments may be used in many different types of systems. For example, in one embodiment, a communication device may be arranged to perform the various methods and techniques described herein. Of course, the scope of the present invention is not limited to communication devices, but instead other embodiments may be directed to other types of devices for processing instructions, or one or more machine-readable media including instructions that, when executed on a computing device, cause the device to implement one or more of the methods and techniques described herein.

[0355] Embodiments may be implemented in code and may be stored on a non-transitory storage medium that has instructions stored thereon that may be used to program a system to execute the instructions. Embodiments may also be implemented in data and may be stored on a non-transitory storage medium that, if used by at least one machine, causes the at least one machine to fabricate at least one integrated circuit to perform one or more operations. Still other embodiments may be implemented in a computer-readable storage medium that includes information that, when fabricated into a SoC or other processor, will configure the SoC or other processor to perform one or more operations. The storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, solid state drives (SSD), compact disc read only memory (CD-ROM), compact disc rewritable (CD-RW), and magneto-optical disks; semiconductor devices such as read only memory (ROM), random access memory (RAM) (such as dynamic random access memory (DRAM), static random access memory (SRAM)), erasable programmable read only memory (EPROM), flash memory, electrically erasable programmable read only memory (EEPROM), magnetic or optical cards, or any other type of medium suitable for storing electronic instructions.

[0356] While the invention has been described with respect to a limited number of embodiments, those skilled in the art will appreciate numerous modifications and variations therefrom. It is intended that the appended claims cover all such modifications and variations as fall within the true spirit and scope of this invention.

Claims

1. A computer-readable medium comprising instructions that, if executed by a processor, cause a computer to be capable of: Receiving a domain manager image and memory location-related address information in response to a request for a service from a cloud service provider; Verifying the domain manager image; Identifying a key domain key that is to be used to encrypt data stored in a key domain of a server having key domain capabilities, wherein the key domain includes a plurality of memory locations of the memory of the server having key domain capabilities; Using the key domain key and the memory location-related address information to encrypt a domain launch image such that the encrypted domain launch image is cryptographically bound to at least one memory location of the key domain; Encrypting the key domain key; And Sending the encrypted domain launch image and the encrypted key domain key to the server having key domain capabilities to cause a processor of the server having key domain capabilities to execute instructions to create the key domain.

2. The computer-readable medium according to claim 1, Wherein: The memory location-related address information includes an entry point address in the memory of the server having key domain capabilities; and the encrypted domain launch image is cryptographically bound to start execution at the entry point address.

3. The computer-readable medium according to claim 1, Wherein: The encrypted domain launch image includes the domain manager image; And the memory location-related address information includes a page table structure that maps a virtual address of a domain manager to be created by executing the domain manager image to a physical memory location of the key domain.

4. The computer-readable medium according to claim 1, wherein the instructions further cause the computer to be capable of: Calculating a hash value based on a processor state expected to exist in the server having key domain capabilities before a processor of the server having key domain capabilities can switch to the key domain; and Sending the hash value to the server having key domain capabilities.

5. The computer-readable medium according to claim 1, wherein the instructions further cause the computer to be capable of: Modifying the domain manager image to include a secret key for functions provided by the remainder of the domain image, wherein a first part of the domain image and the modified domain manager image are included in the encrypted domain launch image.

6. The computer-readable medium according to claim 1, Wherein: The operation of encrypting the key domain key includes encrypting the key domain key with a public key of the server having key domain capabilities.

7. The computer-readable medium according to claim 1, wherein the instructions further cause the computer to be capable of: Calculating an integrity check value for at least one memory location of the key domain using the key domain key; and Sending the integrity check value to the server having key domain capabilities.

8. A computer for protecting data, Comprising: A processor; At least one computer-readable medium coupled to the processor; And Instructions in the computer-readable medium, if executed by the processor, enable the computer to: In response to a request for a service from a cloud service provider, receive a domain manager image and memory location-related address information; Verify the domain manager image; Identify a key domain key to be used to encrypt data stored in a key domain of a server having key domain capabilities, where the key domain includes multiple memory locations of the memory of the server having key domain capabilities; Use the key domain key and the memory location-related address information to encrypt the domain launch image such that the encrypted domain launch image is cryptographically bound to at least one memory location of the key domain; Encrypt the key domain key; And Send the encrypted domain launch image and the encrypted key domain key to the server having key domain capabilities to cause a processor of the server having key domain capabilities to execute instructions to create the key domain.

9. The computer of claim 8, Wherein: The memory location-related address information includes an entry point address in the memory of the server having key domain capabilities; and the encrypted domain launch image is cryptographically bound to start execution at the entry point address.

10. The computer of claim 8, Wherein: The encrypted domain launch image includes the domain manager image; And the memory location-related address information includes a page table structure that maps a virtual address of a domain manager to be created by executing the domain manager image to a physical memory location of the key domain.

11. The computer of claim 8, wherein the instructions further enable the computer to: Before the processor of the server having key domain capabilities can switch to the key domain, calculate a hash value based on a processor state expected to exist in the server having key domain capabilities; and Send the hash value to the server having key domain capabilities.

12. The computer of claim 8, wherein the instructions further enable the computer to: Modify the domain manager image to include a secret key for functions provided by the remainder of the domain image, where a first part of the domain image and the modified domain manager image are included in the encrypted domain launch image.

13. The computer of claim 8, Wherein: The operation of encrypting the key domain key includes encrypting the key domain key with the public key of the server having key domain capabilities.

14. The computer of claim 8, wherein the instructions further enable the computer to: Calculate an integrity check value for at least one memory location of the key domain using the key domain key; and Send the integrity check value to the server having key domain capabilities.

15. A method for protecting data, Comprising: In response to a request for a service from a cloud service provider, receive a domain manager image and memory location-related address information; Verify the domain manager image; Identify a key domain key to be used for encrypting data stored in a key domain of a server having key domain capabilities, where the key domain includes a plurality of memory locations of the memory of the server having key domain capabilities; Use the key domain key and the memory location-related address information to encrypt a domain launch image such that the encrypted domain launch image is cryptographically bound to at least one memory location of the key domain; Encrypt the key domain key; And Send the encrypted domain launch image and the encrypted key domain key to the server having key domain capabilities to cause a processor of the server having key domain capabilities to execute instructions to create the key domain.

16. The method according to claim 15, Wherein: The memory location-related address information includes an entry point address in the memory of the server having key domain capabilities; and the encrypted domain launch image is cryptographically bound to start execution at the entry point address.

17. The method according to claim 15, Wherein: The encrypted domain launch image includes the domain manager image; And the memory location-related address information includes a page table structure that maps a virtual address of a domain manager to be created by executing the domain manager image to a physical memory location of the key domain.

18. The method according to claim 15, further Including: Modify the domain manager image to include a secret key for functions provided by the remainder of the domain image, where a first part of the domain image and the modified domain manager image are included in the encrypted domain launch image.

19. The method according to claim 15, Wherein: The operation of encrypting the key domain key includes encrypting the key domain key with the public key of the server having key domain capabilities.

20. The method according to claim 15, further Including: Calculate an integrity check value for at least one memory location of the key domain using the key domain key; And Send the integrity check value to the server having key domain capabilities.

21. An apparatus for protecting data, Including: Components for receiving a domain manager image and memory location-related address information in response to a request for a service from a cloud service provider; Components for verifying the domain manager image; Components for identifying a key domain key to be used for encrypting data stored in a key domain of a server having key domain capabilities, where the key domain includes a plurality of memory locations of the memory of the server having key domain capabilities; Components for using the key domain key and the memory location-related address information to encrypt a domain launch image such that the encrypted domain launch image is cryptographically bound to at least one memory location of the key domain; Components for encrypting the key domain key; And Components for sending the encrypted domain launch image and the encrypted key domain key to the server having key domain capabilities to cause a processor of the server having key domain capabilities to execute instructions to create the key domain.

22. The apparatus according to claim 21, wherein: the memory location-related address information includes an entry point address in the memory of the server having key domain capabilities; and the encrypted domain launch image is cryptographically bound to begin execution at the entry point address.

23. The apparatus according to claim 21, wherein: the encrypted domain launch image includes the domain manager image; and, the memory location-related address information includes a page table structure that maps the virtual address of the domain manager to be created by executing the domain manager image to the physical memory location of the key domain.

24. The apparatus according to claim 21, further comprising: means for modifying the domain manager image to include a secret key that provides functionality provided by the remainder of the domain image, wherein the first part of the domain image and the modified domain manager image are included in the encrypted domain launch image.

25. The apparatus according to claim 21, wherein: the operation of encrypting the key domain key includes encrypting the key domain key with the public key of the server having key domain capabilities.

26. The apparatus according to claim 21, further comprising: means for calculating an integrity check value for at least one memory location of the key domain using the key domain key; and means for sending the integrity check value to the server having key domain capabilities.

Images