Protection of ATS from rogue device attacks for confidential computing

Abstract

Techniques are disclosed for ensuring that address translation service (ATS) functionality is correctly and securely used for any type of device that supports ATS, even for devices that may potentially act in a rogue manner. The host performs an integrity check on the device using the ATS to prevent the device from maliciously using the locally cached HPA. A device submits a first ATS enabled request to a host. The device receives metadata including (i) a first integrity check vector (ICV) usable to authenticate the device, (ii) an HPA, and (iii) an initialization vector (IV). A device locally caches metadata in an address translation cache (ATC). The device submits a second ATS enabled request including the metadata. The host then independently authenticates the device using the metadata received in the request.

Description

Background Art

[0001] Address Translation Service (ATS) is a set of configurations and protocols that provide address translation to a device, such as a PCIe (peripheral component interconnect express) device. ATS provides the ability for a PCIe device (also known as an endpoint) to communicate with a root complex so that the PCIe device can maintain a translation cache or Address Translation Cache (ATC). By doing so, the PCIe device can assume some of the responsibilities that the root complex would typically handle. In other words, ATS offloads some of the work performed by the root complex to the PCIe device. Address translation tasks are often quite time-consuming. By enabling the PCIe device to perform address translation, the root complex can operate more efficiently.

[0002] Many computing systems now utilize virtualization technologies such as virtual machines (VMs). In fact, a VM can run on a PCIe device. A hypervisor is able to virtualize the actual physical memory so that the operating system running in the VM can use its own contiguous physical memory. To facilitate these operations, the hypervisor can be used to convert or map a so-called "Guest-Physical Address" (GPA) to a "Host-Physical Address" (HPA), where HPA refers to the native physical address space.

[0003] Because ATS enables a PCIe device running a VM to directly use the HPA, ATS eliminates the overhead that occurs when mapping the GPA to the HPA. To eliminate this overhead, ATS enables the ATC to be prepared inside the PCIe device.

[0004] The ATC is filled by requesting a physical address from a host / translation agent (TA), often the CPU (central processing unit). The PCIe device can then directly access the host's memory using this address. The host can then skip the GPA-HPA conversion when the ATS indication is set.

[0005] One problem with ATS capabilities is that rogue PCIe devices can issue requests to any memory address by enabling their ATS capabilities. The rogue PCIe device may then be able to view any memory location. A person skilled in the art can clearly realize how such a scenario may cause security problems and how it may undermine the promise of confidential computing.

[0006] Some techniques have been implemented to try to avoid such scenarios. For example, a technique called Integrity & Data Encryption (IDE) provides PCIe link protection. Another technique called the Trust Execution Environment (TEE) Device Interface Security Protocol (TDISP) provides protection for isolating data between independent virtual functions (VFs) or virtual machines (VMs). However, these security techniques do not protect against the scenario where a rogue device can access memory not allocated to it when ATS is enabled.

[0007] To address these security vulnerabilities, the host device can be configured to implement a proprietary solution. For example, an example of a proprietary solution is to actually not provide HPA; rather, the solution is to provide an intermediate address for ATS requests, and then this intermediate address is verified against the cached value. The drawback of this solution is that it does not achieve the underlying purpose of ATS (i.e., removing translation). Therefore, an improved technique is needed to provide ATS features while also ensuring that rogue PCIe devices cannot maliciously use the cached addresses.

[0008] The subject matter claimed herein is not limited to embodiments that solve any disadvantages or operate only in environments such as those described above. Rather, this background is provided to illustrate an exemplary technical field in which some embodiments described herein may be practiced. Summary of the Invention

[0009] The disclosed embodiments are directed to systems, devices, and methods for ensuring that the Address Translation Service (ATS) function is used correctly and securely for any type of device that supports ATS, even for devices that may potentially act in a rogue manner.

[0010] Some embodiments are constructed to perform integrity checks on a device configured to use ATS to prevent the device from maliciously using the HPAs cached locally. For example, the device may submit a first ATS-enabled request to the ATS host, where the first ATS-enabled request is a request for the ATS host to translate a guest physical address (GPA) into a host physical address (HPA). The device may then receive metadata from the ATS host, the metadata including (i) a first integrity check vector (ICV) that can be used to authenticate the device, (ii) the HPA, and (iii) an initialization vector (IV). The device then locally caches the metadata in an address translation cache (ATC). After the metadata has been cached, the device submits a second ATS-enabled request to the ATS host. The second ATS-enabled request includes the cached metadata. Based on the result of an inline verification performed on the metadata, where the inline verification determines that the device is authenticated, the device obtains access to the memory controller to facilitate processing of the second ATS-enabled request. The device receives new metadata from the ATS host, the new metadata including (i) a new ICV that can be used to authenticate the device subsequently, (ii) the HPA, and (iii) an incremented version of the IV. The device then locally caches the new metadata in the ATC.

[0011] In some embodiments, the ATS host receives a first ATS-enabled request from the device, where the first ATS-enabled request is a request for the ATS host to translate the GPA into an HPA. The ATS host translates the GPA into an HPA. The ATS host also accesses a key unique to the device. The ATS host uses (i) the key, (ii) the HPA, and (iii) an initialization vector (IV) to generate a first integrity check vector (ICV) that can be used to authenticate the device. The ATS host sends (i) the first ICV, (ii) the HPA, and (iii) the IV to the device. Later, the ATS host receives a second ATS-enabled request from the device, where the second ATS-enabled request includes (i) the first ICV, (ii) the HPA, and (iii) the IV. The ATS host performs an inline verification on the first ICV received in the second ATS-enabled request. The inline verification is performed by using (i) the key, (ii) the HPA received in the second ATS-enabled request, and (iii) the IV received in the second ATS-enabled request to calculate a second ICV. The inline verification is also performed by comparing the second ICV with the first ICV; and then determining whether the second ICV matches the first ICV.

[0012] In some embodiments, a system includes an ATS host and a PCIe device. The system causes the PCIe device to submit a first ATS-enabled request to the ATS host, where the first ATS-enabled request is a request for the ATS host to convert the GPA of the PCIe device into an HPA. The system causes the ATS host to convert the GPA into an HPA. The system also causes the ATS host to access a key unique to the PCIe device. Optionally, the key may be locally stored on the ATS host. The system causes the ATS host to generate a first integrity check vector (ICV) using (i) the key, (ii) the HPA, and (iii) an initialization vector. Optionally, the initialization vector is locally stored on the ATS host. The system causes the ATS host to send (i) the first ICV, (ii) the HPA, and (iii) the initialization vector to the PCIe device. The system causes the PCIe device to locally cache (i) the first ICV, (ii) the HPA, and (iii) the initialization vector in the address translation cache (ATC) of the PCIe device. After having cached (i) the first ICV, (ii) the HPA, and (iii) the initialization vector in the ATC of the PCIe device, the system causes the PCIe device to trigger a second ATS-enabled request to the ATS host, where the second ATS-enabled request includes (i) the first ICV cached in the ATC, (ii) the HPA cached in the ATC, and (iii) the initialization vector cached in the ATC. The system causes the ATS host to perform an in-line verification of the first ICV received in the second ATS-enabled request. The in-line verification is performed by: calculating a second ICV using (i) the key locally stored on the ATS host, (ii) the HPA received in the second ATS-enabled request, and (iii) the initialization vector received in the second ATS-enabled request. The verification is also performed by: comparing the second ICV with the first ICV; and then determining that the second ICV matches the first ICV. As a result, the first ICV is considered to be verified by the ATS host. The system causes the ATS host to send the second ATS-enabled request to a memory controller for processing on behalf of the PCIe device. The system also causes the ATS host to: increment the initialization vector; and locally store the incremented initialization vector. The system causes the ATS host to calculate a third ICV using (i) the key, (ii) the HPA received in the second ATS-enabled request, and (iii) the incremented initialization vector. The system causes the ATS host to send (i) the third ICV, (ii) the HPA, and (iii) the incremented initialization vector to the PCIe device. The system causes the PCIe device to locally cache (i) the third ICV, (ii) the HPA, and (iii) the incremented initialization vector in the ATC of the PCIe device.

[0013] The present invention content is provided to introduce a series of concepts that will be further described in the detailed description below in a simplified form. The present invention content is not intended to identify the key features or essential features of the claimed subject matter, nor is it intended to be used to assist in determining the scope of the claimed subject matter.

[0014] Additional features and advantages will be set forth in the following description, and in part will be obvious from the description, or may be learned by practice of the teachings herein. The features and advantages of the present invention may be realized and obtained by means of the instrumentalities and combinations particularly pointed out in the appended claims. The features of the present invention will become more apparent from the following description and the appended claims, or may be learned by practicing the present invention as described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] To describe the manner in which the above and other advantages and features can be obtained, a more specific description of the subject matter briefly described above will be presented by reference to specific embodiments illustrated in the accompanying drawings. It should be understood that these drawings depict only typical embodiments and are not to be considered limiting of the scope, and the embodiments will be described and explained with additional features and details by using the drawings, wherein:

[0016] Figure 1 An example architecture for implementing ATS is shown.

[0017] Figure 2 An example architecture for providing encrypted authentication information to an ATS-enabled device to enable the device to be authenticated later is shown.

[0018] Figure 3 An example of how encrypted authentication information (e.g., integrity check vector) can be generated is shown.

[0019] Figure 4 An example scenario of using encrypted authentication information to verify a device is shown.

[0020] Figure 5 An example processing flow outlining some activities performed by a PCIe device and an ATS host is shown.

[0021] Figure 6 A flowchart of an example method performed by a PCIe device is shown.

[0022] Figure 7 A flowchart of an example method performed by an ATS host is shown.

[0023] Figure 8 An example of a computer system that can be configured to perform any of the disclosed operations is shown. DETAILED DESCRIPTION

[0024] The disclosed embodiments are directed to systems, devices, and methods for ensuring that an address translation service (ATS) function is used correctly and securely for any type of device that supports ATS, even for devices that may potentially act in a rogue manner.

[0025] Some embodiments perform an integrity check on a device that uses ATS to prevent the device from maliciously using a host physical address (HPA) that is locally cached. For example, the device submits a first ATS-enabled request to an ATS host, where the first ATS-enabled request is a request to translate a guest physical address (GPA) into an HPA. The device receives metadata that includes (i) a first integrity check vector (ICV) that can be used to authenticate the device, (ii) the HPA, and (iii) an initialization vector (IV). The device locally caches the metadata in an address translation cache (ATC). Later, the device submits a second ATS-enabled request that includes the cached metadata. The device obtains access to a memory controller to facilitate processing of the second ATS-enabled request. The device receives new metadata from the ATS host, the new metadata including (i) a new ICV that can be used to authenticate the device subsequently, (ii) the HPA, and (iii) an incremented version of the IV. The device then locally caches the new metadata in the ATC.

[0026] In some embodiments, the ATS host receives a first ATS-enabled request to translate a GPA into an HPA. The ATS host performs the translation. The ATS host accesses a key and then uses (i) the key, (ii) the HPA, and (iii) an initialization vector (IV) to generate a first integrity check vector (ICV) that can be used to authenticate the device. The ATS host sends (i) the first ICV, (ii) the HPA, and (iii) the IV to the device. Later, the ATS host receives a second ATS-enabled request that includes (i) the first ICV, (ii) the HPA, and (iii) the IV. The ATS host performs an in-line verification of the first ICV received in the second ATS-enabled request.

[0027] In some embodiments, the system includes an ATS host and a PCIe device. The system then uses the ATS host and the PCIe device to facilitate the above operations.

[0028] Examples of Technical Advantages, Improvements, and Practical Applications

[0029] The following sections outline some example improvements and practical applications provided by the disclosed embodiments. However, it will be realized that these are merely examples and the embodiments are not limited to these improvements.

[0030] The disclosed embodiments bring many advantages, benefits, and practical applications to the field of computer security technology. Specifically, these embodiments are directed to techniques for providing an improved authentication mechanism to ensure that a device cannot access resources that it should not be able to access.

[0031] An Address Translation Service (ATS) enables a device (e.g., a PCIe device) to directly use a Host Physical Address (HPA), thereby using an Address Translation Cache (ATC) inside the device to eliminate the overhead of Guest Physical Address (GPA) to Host Physical Address (HPA) in the ATS host. The ATC is populated by requesting physical addresses from the host, and the device can use this address to directly access the host's memory, allowing the host to skip the GPA-to-HPA conversion when the ATS indication is set. A significant problem with this capability is that a rogue PCIe device can issue requests to "any" memory address by enabling the ATS capability. The rogue device can then view any memory location. This scenario creates security issues and undermines the promise of confidential computing.

[0032] The disclosed solution uses cryptographic integrity checking techniques to address these issues. An integrity check is performed on the translated address, a key that can be periodically refreshed, and an initialization vector. The resulting integrity check vector is then provided to the PCIe device. During subsequent requests, the PCIe device can submit its integrity check vector, and the host can independently verify the vector. By performing the disclosed operations, these embodiments provide a solution that significantly reduces the integrity check overhead for memory operations. Accordingly, these and many other advantages will now be discussed in more detail throughout the remaining sections of this disclosure.

[0033] PCIe Example Architecture

[0034] Attention is now turned to Figure 1 , Figure 1 FIG. 16 shows an example architecture 100 that can be used to implement the above advantages. Architecture 100 is shown as including a Central Processing Unit (CPU) 105. The CPU 105 is an electronic circuit that executes instructions to perform various actions. The CPU 105 can be implemented as a System on a Chip (SOC) or any other circuit design.

[0035] In some cases, the CPU 105 can be or can include a translation agent (TA) 110. The TA 110 is a logical or computational entity configured to translate addresses from one address space to another. The TA 110 is typically associated with a memory, and the memory often includes various different translation tables or page tables that the TA 110 will use during the address translation process. In some cases, the TA 110 can also include an address translation cache for caching different address translations.

[0036] The architecture 100 is also shown as including a root complex 115 coupled to a memory 120. The memory 120 can include the above-described translation tables. In a high-speed Peripheral Component Interconnect (PCIe) unit, the root complex 115 refers to the device that connects the CPU 105 and the memory 120 to the PCIe switch fabric, which can optionally include various different PCIe devices. For example, the architecture 100 is shown as including a first PCIe endpoint 125, and the first PCIe endpoint 125 includes an ATC 130. The architecture 100 also shows a switch 135 that connects the root complex 115 to various other PCIe endpoints, such as a PCIe endpoint 140 that includes an ATC 145 and a PCIe endpoint 150 that includes an ATC 155. The switch 135 and the PCIe endpoints 125, 140, and 150 are part of the PCIe switch fabric. The root complex 115 is configured to generate various transaction requests for the CPU 105. The root complex 115 can include multiple PCIe ports, enabling multiple PCIe devices to be connected to the root complex 115. The switch 135 enables multiple PCIe devices to be connected to a single port of the root complex 115.

[0037] Initial Caching of Addresses at the PCIe Device

[0038] Figure 2 The architecture 200 representing Figure 1 is shown. The architecture 200 is shown as including a host 205 representing the CPU 105 and a TA 210. The architecture 200 also includes a root complex 215, a memory 220, a PCIe device 225, a VM 225A operating on the PCIe device 225, and an ATC 230 in the PCIe device 225.

[0039] According to the disclosed principles, the PCIe device 225 is initially populated with various addresses (using the ATS function) that are used to access dynamic random access memory (DRAM), and may be used to access other memories or memory services of the host 205. As described above, the PCIe device 225 is provided with the ATS function. During the configuration or initial installation of the PCIe device into the architecture, the PCIe device can be initially verified and authenticated to use ATS. However, as will be discussed in more detail later, it is possible that the PCIe device later becomes a rogue device and attempts to use its ATS function in an improper manner. These embodiments are designed to prevent the PCIe device from using its ATS function improperly.

[0040] To obtain these addresses, the PCIe device 225 submits an ATS-enabled request 235 to the host 205 or possibly to the TA 210. This first ATS-enabled request 235 includes a guest physical address (GPA) 240. The GPA refers to the address of the actual physical memory resource that a virtual machine (e.g., VM 225A) uses to access memory hardware (e.g., memory 220). To enable a VM workload (e.g., an ATS-enabled read or write request) to access specific data in memory, the task of the underlying architecture will be to find the physical memory address (i.e., host physical address HPA) that corresponds to or matches the virtual address (i.e., GPA). This lookup operation is called memory translation or memory mapping. Page or translation tables are typically used to map the GPA to the HPA.

[0041] After receiving the ATS-enabled request 235, the task of the host 205 is to perform memory translation. For example, Figure 2 Shows the GPA 245 representing GPS240 and the HPA 250. The host 205 accesses various page or translation tables to map the GPA 245 to the HPA 250.

[0042] After successfully performing this memory translation operation, the host 205 then returns the metadata 255 to the PCIe device 225. According to the disclosed principles, the metadata 255 is configured to include various data units. Further details regarding this metadata 255 are shown in Figure 3 shown.

[0043] Figure 3 Shows the representative Figure 2 The host 300 in shows the host 205. The host 300 generates or perhaps accesses a key 305, which is unique to any PCIe device communicating with the host 300. For example, in this example, the key 305 is Figure 2unique to the PCIe device 225. "Unique" means that the key 305 is specifically customized and only used for operations involving the PCIe device 225; no other PCIe device can use the key 305 or perform operations using the key 305.

[0044] In some embodiments, the PCIe device is a single virtual function device. In this scenario, then a single key is generated for the PCIe device. In other embodiments, the PCIe device includes multiple virtual functions. In this scenario, different keys can be generated for each different virtual function of the PCIe device. Thus, it is possible that a single PCIe device can be associated with multiple keys, such as one key for each virtual function of the PCIe device.

[0045] Figure 3 Also shown is a transformed HPA 310 representing Figure 2 the HPA 250 in. It should be noted that the host has converted the GPA into the HPA 310.

[0046] Figure 3 Also shown is an initialization vector (IV) 315. The IV 315 refers to a variable that can be incremented and can be provided as an input, which helps to obfuscate or obscure other data so that the other data cannot be inferred or discerned by other entities. That is, the IV 315 can be used in combination with the key 305 to perform various different encryption operations. As will be discussed in more detail later, performing these encryption operations is to verify or authenticate the PCIe device to ensure that the PCIe device is not operating in a rogue manner, especially to ensure that its use of the ATS function is not operating in a rogue manner.

[0047] The host 300 uses the key 305, the HPA 310, and the IV 315 to generate metadata 320, and the metadata 320 represents Figure 2 the metadata 255 in and is a form of encrypted data. For example, the host 300 can utilize a hash based message authentication code (HMAC) to generate a so-called "integrity check vector" (ICV) that can be included in the metadata 255. More specifically, the host 300 can perform the following operation: ICV = HMAC(key, HPA, IV).

[0048] Figure 3 Shows how the metadata 320 includes the ICV 325. The metadata 320 also includes the HPA 330 representing the HPA 310 and the IV 335 representing the IV 315.

[0049] ReturnFigure 2 The host 205 transfers the metadata 255 to the PCIe device 225. Recall that the metadata 255 includes the ICV, HPA, and IV. The PCIe device 225 then locally caches the metadata 255 in the ATC 230. Thus, the PCIe device 225 locally caches the ICV, HPA, and IV.

[0050] For subsequent ATS-enabled requests, the PCIe device can submit the ICV, HPA, and IV to the host. The host can then independently calculate a new ICV using the key stored locally at the host 205 for the PCIe device 225, the received HPA, and the received IV. After calculating the new ICV, the host 205 can then compare the new ICV with the ICV received in the subsequent ATS-enabled request. If the two ICVs match, the host 205 can be confident that the PCIe device 225 is operating correctly and not in a rogue manner. The ATS-enabled request can then be processed. On the other hand, if the ICVs do not match, the host 205 can determine that the PCIe device 225 is operating in a rogue manner or using false data. Thus, the host 205 can terminate the request and potentially perform various corrective actions to attempt to correct the state of the PCIe device.

[0051] Processing Subsequent ATS-Enabled Requests

[0052] Figure 4 An example architecture 400 representing the architecture discussed so far is shown. Figure 4 The scenario shown occurs after the PCIe device has been populated with the translated address. Specifically, the architecture 400 is shown to include a host 405, a TA 410, a root complex 415, a memory controller 420 (which accesses the memory 220), a PCIe device 425, and an ATC 430 within the PCIe device 425.

[0053] After the metadata 255 in Figure 2 has been cached in the ATC 430 of the ATC 230 representing Figure 2 , the PCIe device 425 can then generate a second or subsequent ATS-enabled request 435. The ATS-enabled request 435 can be a memory read or write request that uses the ATS function. For example, the ATS-enabled request 435 can be a request to access the DRAM of the host 405.

[0054] According to the disclosed principles, the ATS-enabled request 435 is configured to include its read or write request and the ICV 440, HPA 445, and IV 450. The ICV 440, HPA 445, and IV 450 are all pre-cached in the ATC 430.

[0055] The PCIe device 425 submits the ATS-enabled request 435 to the host 405. The host 405 then performs in-line verification 455 to ensure that the PCIe device 425 is not operating in a rogue manner.

[0056] The in-line verification 455 includes multiple operations. One operation involves accessing a key unique to the PCIe device 425. The host 405 then uses the key, the HPA 445 included in the ATS-enabled request 435, and the IV 450 also included in the ATS-enabled request 435 to generate a second ICV using the HMAC process discussed earlier. The host 405 then compares the ICV 440 included in the ATS-enabled request 435 with the newly generated ICV to determine if they match. If the two ICVs match, the host 405 can be certain that the PCIe device 425 is not operating in a rogue manner. If the two ICVs do not match, the host 405 can terminate or cancel the ATS-enabled request 435.

[0057] If the in-line verification 455 finds that the PCIe device 425 is certified, the ATS-enabled request 435 can be processed by the memory controller 420. Further, the host 405 can prepare updated metadata 460 and send the updated metadata 460 to the PCIe device 425.

[0058] The updated metadata 460 includes a newly calculated ICV (a third ICV). The third ICV is calculated using the key unique to the PCIe device 425. The third ICV is further calculated using the HPA 445. Furthermore, the third ICV is calculated using an incremented version of the IV locally stored on the host 405.

[0059] The IV is provided not only to improve the cryptographic aspects of the ICV but also to prevent the PCIe device from repeatedly submitting the same request or repeatedly attempting to use the cached HPA. In other words, this prevents the PCIe device from performing a replay attack. Incrementing the IV helps frustrate the ability of a rogue actor to calculate the key used to calculate the ICV.

[0060] For each new ATS-enabled request, the host increments the IV. In some embodiments, the IV can be a 256-bit value or number. When a sufficient number of ATS-enabled requests have been transmitted, the IV can roll over due to its limited size and number of bits. When an IV rollover event occurs, the host can be triggered to generate a new key for the PCIe device (or virtual function of the PCIe device).

[0061] In some cases, the generation of the new key can be triggered earlier than the bit rollover of the IV. For example, a hysteresis point can be defined for the IV. When the hysteresis point is hit based on the increment of the IV, the host can be triggered to generate a new key. The hysteresis point can be set to any value. For example, the hysteresis point can be set to the 50% mark of the bits of the IV. In some cases, the hysteresis point can be the 75% mark. Any hysteresis point can be set. Typical points are in the range of 25% - 90% of the incremented bits. For example, if the IV is a 3-bit vector, the IV can have the following values: 0, 1, 2, 3, 4, 5, 6, 7, and 8. In one example, the hysteresis point can be set to the value 4. After the IV reaches the value 4, the IV will roll over and a new key will be generated.

[0062] When a new key is generated for the PCIe device, the host also triggers the PCIe device to flush its ATC. Since a new key is used, the ICV stored in the ATC of the PCIe device will no longer be valid.

[0063] As described above, if the inline verification 455 is successful, the metadata 460 is sent to the PCIe device 425. The metadata 460 includes the third ICV, HPA, and the incremented IV. If the inline verification 455 is successful, the ATS-enabled request 435 is also processed by the memory controller 420.

[0064] In some cases, the ATS-enabled request 435 is processed in a speculative manner. For example, assume the ATS-enabled request 435 is a memory read request. The host 405 can cause the memory controller 420 to start processing the memory read request in parallel with the inline verification 455. If the memory read request is completed before the inline verification 455, the host 405 can hold the read data read from the memory until the inline verification 455 is completed. If the inline verification 455 indicates that the PCIe device 425 is trustworthy, the read data can be transmitted back to the PCIe device 425. On the other hand, if the inline verification 455 indicates that the PCIe device 425 is not trustworthy, the host can prevent the read data from being transmitted to the PCIe device 425.

[0065] Similar to read requests, write requests can also optionally be executed speculatively, or at least partially speculatively. For example, any data preparation or calculations that need to be done before the actual write operation can be executed speculatively. If the PCIe device is determined to be trustworthy, the write operation can commence; otherwise, the write operation will not be executed.

[0066] Accordingly, in some embodiments, the ATS-enabled request 435 is a read request. In such a scenario, the host 405 can send a read request to the memory controller 420 in parallel with the execution of the inline verification 455. In response to determining that the second ICV does not match the first ICV (ICV 440) as part of the inline verification 455, the host 405 can prevent the data read as part of the read request from being transmitted to the PCIe device 425.

[0067] In some embodiments, the ATS-enabled request 435 is a read request. In such a scenario, the host 405 can issue a read request during a time period that overlaps with the execution of the inline verification 455. After determining that the inline verification 455 has failed, the host 405 can set an invalid bit into the request completion queue associated with the ATS-enabled request 435. Further, in response to obtaining the data read as part of the read request, these embodiments (e.g., perhaps the host 405 or perhaps the memory controller 420) can (i) squash (i.e., delete, scramble, or otherwise make it unavailable) the data, (ii) cause the read request to be marked as an undefined request, and (iii) generate an interrupt to notify the host 405 of an unauthorized access attempt by the PCIe device 425.

[0068] In some embodiments, the ATS-enabled request 435 is executed speculatively. As a request, the ATS-enabled request 435 can be executed at least partially before the inline verification 455 is completed.

[0069] In some embodiments, the ATS-enabled request 435 is a write request. Optionally, the write request can be executed at least partially in parallel with the inline verification 455.

[0070] Example Processing Flow

[0071] Now turn attention to Figure 5 , Figure 5 illustrates an example of PCIe devices (e.g., PCIe endpoints / devices 125, 225, and 425 of Figure 1 , Figure 2 and Figure 4 respectively) and a host (e.g., CPU 105 of Figure 1 , Figure 2host 205, Figure 3 host 300, and Figure 4 host 405) of the various operations performed by example processing flow 500. Processing flow 500 may be executed in architectures 100, 200, and 400.

[0072] Processing flow 500 generally outlines techniques for performing integrity checks on a high-speed Peripheral Component Interconnect (PCIe) device to prevent the PCIe device from maliciously using the locally cached HPA, where the PCIe device is configured to use Address Translation Service (ATS).

[0073] Processing flow 500 includes the action of causing the PCIe device to submit a first ATS-enabled request 505 to the ATS host. The first ATS-enabled request 505 includes a GPA for the PCIe device (e.g., Figure 2 GPA 240). The first ATS-enabled request 505 is a request for the ATS host to convert the GPA of the PCIe device into an HPA (e.g., Figure 2 HPA 250).

[0074] The ATS host then converts (e.g., conversion 510) the GPA into an HPA. The ATS host also accesses a key unique to the PCIe device. In some embodiments, the key is locally stored on the ATS host. In some cases, the key may be customized for the entire PCIe device. In other cases, the key may be unique to a particular virtual function of the PCIe device.

[0075] The ATS host uses (i) the key, (ii) the HPA, and (iii) an initialization vector (IV) to generate a first integrity check vector (ICV) 515. The IV is locally stored on the ATS host. In some implementations, the IV may be incremented. For example, the IV may be incremented each time a new ATS-enabled request is received. In some cases, the IV is incremented only in response to the ATS request being successfully processed. In some cases, the IV is incremented for each ATS request, regardless of whether it is successfully processed. The size of the IV may be limited. When (due to incrementing) the maximum value of the IV is reached, the IV may be flipped and restarted. In some cases, the flip may be triggered in response to the IV hitting a value other than the maximum value (such as a defined hysteresis point). The IV may be triggered to flip when it reaches the hysteresis point, rather than when it reaches the maximum value. Setting the hysteresis point may be beneficial in avoiding scenarios where a stale ICV is attempted to be used. As discussed earlier, the flip event may also trigger the generation of a new key for the PCIe device and may further trigger the PCIe device to flush its ATC.

[0076] The ATS host then sends metadata 520 to the PCIe device, where the metadata 520 includes (i) a first ICV, (ii) an HPA, and (iii) an IV. The PCIe device locally caches (e.g., cache 525) (i) the first ICV, (ii) the HPA, and (iii) the IV in the address translation cache (ATC) of the PCIe device.

[0077] After (i) the first ICV, (ii) the HPA, and (iii) the initialization vector have been cached in the ATC of the PCIe device, the PCIe device can trigger a second ATS-enabled request 530 to the ATS host. The second ATS-enabled request 530 includes (i) the first ICV cached in the ATC, (ii) the HPA cached in the ATC, and (iii) the initialization vector cached in the ATC.

[0078] The ATS host performs an in-line verification 535 on the first ICV received in the second ATS-enabled request. The in-line verification is performed by calculating a second ICV 540 using (i) a key locally stored in the ATS host, (ii) the HPA received in the second ATS-enabled request, and (iii) the initialization vector received in the second ATS-enabled request. Figure 5 Illustrated is how to calculate the second ICV 540 using the data in the ATS-enabled request 530.

[0079] As part of the in-line verification, the ATS host then compares the second ICV with the first ICV. In some cases, the ATS host can determine that the second ICV matches the first ICV. As a result, the first ICV is considered to be verified by the ATS host, and the ATS host considers the PCIe device to be trustworthy.

[0080] The ATS-enabled request 530 can be processed (sometimes in parallel with the in-line verification 535). In other words, the ATS host can send the second ATS-enabled request 530 to the memory controller for processing on behalf of the PCIe device.

[0081] In some cases, the two ICVs may not match. In such a scenario, the ATS host will terminate the ATS-enabled request 530.

[0082] In one embodiment, the ATS host increments the initialization vector (e.g., increment IV 545) and locally stores the incremented initialization vector. Optionally, the increment occurs only if the two ICVs match. In another embodiment, the increment can occur each time an ATS-enabled request is received, regardless of whether the ICVs match.

[0083] The ATS host uses (i) a key, (ii) the HPA received in a second ATS-enabled request, and (iii) an incremented initialization vector to compute a third ICV 550. The ATS host then sends metadata 555 to the PCIe device, where the metadata 555 includes (i) the third ICV, (ii) the HPA, and (iii) the incremented initialization vector. The PCIe device locally caches (e.g., cache 560) (i) the third ICV, (ii) the HPA, and (iii) the incremented initialization vector in the ATC of the PCIe device.

[0084] Example Method - Perspective of the PCIe Device

[0085] The following discussion now refers to many methods and method acts that may be performed. Although method acts may be discussed in a particular order or shown in a flowchart as occurring in a particular order, no particular order is required unless specifically stated, or required because an act depends on another act that is completed before that act is performed.

[0086] Attention is now turned to Figure 6 , Figure 6 FIG. shows a flowchart of an example method 600 for performing an integrity check on a device to prevent the device from maliciously using a locally cached HPA, where the device is configured to use address translation service (ATS). Method 600 may be performed by Figure 4 the PCIe device 425. In other words, the device may be a PCIe device. In some embodiments, the PCIe device is a single virtual function device, while in other embodiments, the PCIe device is a multi-virtual function device.

[0087] Method 600 includes an act (act 605) of submitting a first ATS-enabled request (e.g., Figure 2 the ATS-enabled request 235) to an ATS host (e.g., host 205). The first ATS-enabled request is a request for the ATS host to convert a GPA (e.g., GPA 240, 245) to an HPA (e.g., HPA 250). In some embodiments, the device runs at least one virtual machine (VM). In such a scenario, the GPA may optionally be used for the VM.

[0088] Act 610 includes receiving metadata (e.g., Figure 2 the metadata 255 in Figure 3the metadata 320). The metadata includes (i) a first integrity check vector (ICV) (e.g., ICV 325) that can be used to authenticate the device, (ii) an HPA (e.g., HPA 330), and (iii) an initialization vector (IV) (e.g., IV 335).

[0089] Action 615 includes locally caching the metadata in an address translation cache (ATC) (e.g., ATC 230). Thus, not only the HPA is locally cached at the PCIe device, but also the ICV and the IV are locally cached in the PCIe device. Further, these data units are cached in the ATC of the PCIe device.

[0090] After the metadata has been cached, action 620 includes submitting a second ATS-enabled request (e.g., Figure 4 the ATS-enabled request 435) to the ATS host. The second ATS-enabled request includes the cached metadata. In some embodiments, the second ATS-enabled request is a read request for the memory controller. Optionally, the second ATS-enabled request can be a request to access the dynamic random access memory (DRAM) of the ATS host (e.g., the CPU). In other embodiments, the second ATS-enabled request is a write request for the memory controller.

[0091] Based on the result of the in-line verification (e.g., in-line verification 455) performed on the metadata, there is an action (action 625) to obtain access to the memory controller (e.g., memory controller 420) to facilitate the processing of the second ATS-enabled request, where the in-line verification determines that the device is authenticated. Optionally, the ATS-enabled request can be executed speculatively, such as in a scenario where at least a portion of the ATS-enabled request is processed concurrently with the execution of the in-line verification.

[0092] Action 630 then includes receiving new metadata (e.g., metadata 460) from the ATS host. The new metadata includes (i) a new ICV that can be used for subsequent device authentication, (ii) an HPA, and (iii) an incremented version of the IV. Optionally, the old ICV can be marked or otherwise indicated as no longer valid. The result of executing the second ATS-enabled request can also be transmitted to the device.

[0093] Action 635 includes locally caching the new metadata in the ATC. In some embodiments, the incremented versions of the new ICV and the IV replace the previously cached ICV and IV in the ATC. Thus, in some cases, the old or invalid versions of the ICV and the IV are not maintained in the ATC.

[0094] In some cases, older versions of the ICV and IV can optionally be stored in the ATC for a limited period of time. As an example, a queue or buffer can optionally be provided or implemented in the ATC, and a running log of past ICVs and past IVs can be maintained. When the queue is full and new ICVs and new IVs are loaded into the queue, the older ICVs and older IVs can optionally be ejected from the queue.

[0095] In some cases, method 600 can also include the device flushing its ATC in response to a triggering event occurring. For example, the triggering event can be the generation of a new key for the device. The new key can be generated when the IV flips or when the IV hits a predefined hysteresis point. In some embodiments, the triggering event can be an event where the host determines that all PCIe devices to which it is connected should be re-authenticated.

[0096] Example Method – Perspective of the ATS Host

[0097] Figure 7 A flowchart of an example method 700 is shown from the perspective of the ATS host. For example, method 700 can be implemented by Figure 1 the CPU 105 of Figure 2 the host 205 of Figure 3 the host 300 and / or Figure 4 the host 405 of

[0098] Method 700 includes the action (action 705) of receiving a first ATS-enabled request (e.g., ATS-enabled request 235) from a device (e.g., PCIe device 225). The first ATS-enabled request is a request for the ATS host to convert a GPA (e.g., GPA 240, 245) into an HPA (e.g., HPA 250).

[0099] Action 710 includes converting the GPA into an HPA. The conversion can be performed using various pages or translation tables, as discussed previously.

[0100] Action 715 includes accessing a key unique to the device (e.g., key 305). Optionally, the key can be locally stored at the ATS host.

[0101] Action 720 includes generating a first integrity check vector (ICV) (ICV 325) using (i) the key (e.g., key 305), (ii) the HPA (e.g., HPA 310), and (iii) an initialization vector (IV) (IV 315). The first ICV can be used to authenticate the device. Optionally, the IV can be locally stored on the ATS host.

[0102] Action 725 includes sending to the device (i) a first ICV, (ii) an HPA, and (iii) an IV. For example, this data can be sent to the device in the form of metadata 255 of Figure 2 The metadata 255 of

[0103] Action 730 includes receiving from the device a second ATS-enabled request (e.g., ATS-enabled request 435). The second ATS-enabled request includes (i) a first ICV, (ii) an HPA, and (iii) an IV. Optionally, the second ATS-enabled request can be a request to use the memory services of a dynamic random access memory (DRAM), other memory, or an ATS host.

[0104] Action 735 includes performing an in-line verification (in-line verification 455) on the first ICV received in the second ATS-enabled request. The in-line verification is performed by: using (i) a key, (ii) the HPA received in the second ATS-enabled request, and (iii) the IV received in the second ATS-enabled request to calculate a second ICV. The in-line verification is also performed by comparing the second ICV with the first ICV. The in-line verification is also performed by determining whether the second ICV matches the first ICV.

[0105] Optionally, the second ATS-enabled request can be executed speculatively, such as by executing at least partially in parallel with the in-line verification. If the in-line verification determines that the device is authenticated, the second ATS-enabled request can be executed to completion. On the other hand, if the in-line verification determines that the device is not authenticated, the second ATS-enabled request can be terminated, rejected, or revoked. The ATS host can then maintain an indication that the device is currently untrusted.

[0106] Consider a scenario where the second ATS-enabled request is a read request. In such a scenario, method 700 can optionally also include the action of sending a read request to the memory controller in parallel with the execution of the in-line verification. In response to determining as part of the in-line verification that the second ICV matches the first ICV, method 700 can also include allowing the device to access the data read as part of the read request. Another action includes incrementing the IV. Another action includes using (i) a key, (ii) the HPA received in the second ATS-enabled request, and (iii) the incremented IV to calculate a third ICV. Yet another action includes sending to the device (i) the third ICV, (ii) the HPA, and (iii) the incremented IV.

[0107] Accordingly, by performing the disclosed operations, a significant improvement in computational security can be achieved. In fact, these embodiments help prevent rogue ATS-enabled devices from being able to access resources that they should not be able to access.

[0108] Example Computer / Computer System

[0109] Now turn attention to Figure 8 , Figure 8 FIG. shows an example computer system 800 that may include and / or be used to perform any of the operations described herein. The computer system may be implemented as any of the PCIe devices and / or hosts mentioned herein. The computer system 800 may also perform any of the disclosed methods described herein.

[0110] The computer system 800 may take on a variety of different forms. For example, the computer system 800 may be embodied as a tablet computer, a desktop computer, a laptop computer, a mobile device, or a stand-alone device, such as those described throughout this disclosure. The computer system 800 may also be a distributed system that includes one or more connected computing components / devices that communicate with the computer system 800.

[0111] In its most basic configuration, the computer system 800 includes a variety of different components. Figure 8 FIG. shows that the computer system 800 includes one or more processors 805 (also referred to as "hardware processing units") and a storage device 810.

[0112] Regarding the (multiple) processors 805, it will be appreciated that the functions described herein may be performed, at least in part, by one or more hardware logic components (e.g., the (multiple) processors 805). By way of example, and not limitation, illustrative types of hardware logic components / processors that can be used include Field-Programmable Gate Array (FPGA), Application-Specific Integrated Circuit (ASIC), Program-Specific Standard Product (ASSP), System-on-a-Chip (SOC), Complex Programmable Logic Device (CPLD), Central Processing Unit (CPU), Graphical Processing Unit (GPU), or any other type of programmable hardware.

[0113] As used herein, the terms "executable module", "executable component", "component", "module", or "engine" may refer to a hardware processing unit or a software object, routine, or method that can be executed on the computer system 800. The different components, modules, engines, and services described herein may be implemented as objects or processors (e.g., separate threads) that execute on the computer system 800.

[0114] The storage device 810 can be a physical system memory, which can be volatile, non-volatile, or some combination of both. The term "memory" can also be used herein to refer to non-volatile mass storage devices, such as physical storage media. If the computer system 800 is distributed, the processing, memory, and / or storage capabilities can also be distributed.

[0115] The storage device 810 is shown as including executable instructions 815. The executable instructions 815 represent instructions that can be executed by the processor(s) 805 of the computer system 800 to perform the disclosed operations (such as the operations described in the various methods).

[0116] The disclosed embodiments can include or utilize, for example, a special-purpose or general-purpose computer including computer hardware such as one or more processors (such as the processor(s) 805) and system memory (such as the storage device 810), as discussed in more detail below. The embodiments also include physical computer-readable media and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media can be any available media that can be accessed by a general-purpose or special-purpose computer system. A computer-readable medium that stores computer-executable instructions in data form is a "physical computer storage medium" or "hardware storage device". Further, computer-readable storage media including physical computer storage media and hardware storage devices do not include signals, carriers, and propagated signals. On the other hand, a computer-readable medium that carries computer-executable instructions is a "transmission medium" and includes signals, carriers, and propagated signals. Thus, by way of example and not limitation, the current embodiments can include at least two distinct types of computer-readable media: computer storage media and transmission media.

[0117] A computer storage medium (also known as a "hardware storage device") is a computer-readable hardware storage device such as RAM (random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable ROM), CD-ROM (compact disc ROM), a RAM-based solid state drive (SSD), flash memory, phase-change memory (PCM), or other types of memory, or other optical disc storage devices, magnetic disk storage devices, or other magnetic storage devices, or any other medium that can be used to store the required program code components in the form of computer-executable instructions, data, or data structures and that can be accessed by a general-purpose or special-purpose computer.

[0118] Computer system 800 may also be connected to external sensors (e.g., one or more remote cameras) or devices via network 820 (via a wired or wireless connection). For example, computer system 800 may communicate with any number of devices or cloud services to obtain or process data. In some cases, network 820 itself may be a cloud network. Further, computer system 800 may also be connected to remote / separate (s) computer systems configured to perform any of the processing described with respect to computer system 800 via one or more wired or wireless networks.

[0119] A "network" (such as network 820) is defined as one or more data links and / or data switches that enable the transfer of electronic data between computer systems, modules, and / or other electronic devices. When information is passed or provided to a computer via a network (hardwired, wireless, or a combination of hardwired and wireless), the computer properly treats the connection as a transmission medium. Computer system 800 will include one or more communication channels for communicating with network 820. The transmission medium includes a network that can be used to carry data or the required program code in the form of computer-executable instructions or in the form of a data structure. Further, these computer-executable instructions can be accessed by a general-purpose or special-purpose computer. Combinations of the above should also be included within the scope of computer-readable media.

[0120] After reaching various computer system components, program code means in the form of computer-executable instructions or data structures can be automatically transferred from a transmission medium to a computer storage medium (and vice versa). For example, computer-executable instructions or data structures received via a network or data link can be buffered in RAM within a network interface module (e.g., a network interface card (NIC)) and then ultimately transferred to the computer system RAM and / or non-volatile computer storage media at the computer system. Thus, it should be understood that computer storage media can be included in computer system components that also (or even primarily) utilize a transmission medium.

[0121] Computer-executable (or computer-interpretable) instructions include, for example, instructions that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a particular function or group of functions. For example, computer-executable instructions can be in binary intermediate format instructions (such as assembly language), or even source code. Although the subject matter has been described in language specific to structural features and / or method acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.

[0122] Those skilled in the art will appreciate that these embodiments can be implemented in a network computing environment having many types of computer system configurations, including personal computers, desktop computers, laptop computers, messaging processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, pagers, routers, switches, etc. These embodiments can also be practiced in a distributed system environment where local and remote computer systems linked by a network (either by a hardwired data link, a wireless data link, or a combination of a hardwired data link and a wireless data link) each perform tasks (e.g., cloud computing, cloud services, etc.). In a distributed system environment, program modules can be located in both local and remote memory storage devices.

[0123] The present invention may be embodied in other specific forms without departing from its characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. Thus, the scope of the present invention is indicated by the appended claims rather than the foregoing description. All changes that come within the meaning and range of equivalency of the claims are to be embraced within the scope of the claims.

Claims

1. A method for performing an integrity check on a device to prevent the device from maliciously using a locally cached HPA, the device being configured to use an address translation service (ATS), the method being performed by the device and comprising: Submitting a first ATS-enabled request to an ATS host, wherein the first ATS-enabled request is a request for the ATS host to translate a guest physical address (GPA) into a host physical address (HPA); Receiving metadata from the ATS host, the metadata including (i) a first integrity check vector (ICV) that can be used to authenticate the device, (ii) the HPA, and (iii) an initialization vector (IV); Locally caching the metadata in an address translation cache (ATC); After the metadata has been cached, submitting a second ATS-enabled request to the ATS host, wherein the second ATS-enabled request includes the cached metadata; Obtaining access to a memory controller based on the result of an in-line verification performed on the metadata to facilitate processing of the second ATS-enabled request, wherein the in-line verification determines that the device is authenticated; Receiving new metadata from the ATS host, the new metadata including (i) a new ICV that can be used to subsequently authenticate the device, (ii) the HPA, and (iii) an incremented version of the IV; And Locally caching the new metadata in the ATC.

2. The method according to claim 1, wherein the device is a high-speed peripheral component interconnect (PCIe) device.

3. The method according to claim 2, wherein the PCIe device is a single virtual function device.

4. The method according to claim 2, wherein the PCIe device is a multi-virtual function device.

5. The method according to claim 1, wherein the method further comprises the device flushing its ATC in response to a trigger event occurring.

6. The method according to claim 1, wherein the second ATS-enabled request is a read request for the memory controller.

7. The method according to claim 1, wherein the second ATS-enabled request is a write request for the memory controller.

8. The method according to claim 1, wherein the second ATS-enabled request is a request to access a dynamic random access memory (DRAM) of the ATS host.

9. The method according to claim 1, wherein the device runs at least one virtual machine (VM), and wherein the GPA is used for the at least one VM.

10. A method for performing an integrity check on a device to prevent the device from maliciously using a locally cached HPA, the device being configured to use an address translation service (ATS), the method being performed by an ATS host and comprising: Receive a first ATS-enabled request from the device, where the first ATS-enabled request is a request for the ATS host to convert a guest physical address (GPA) to a host physical address (HPA); Convert the GPA to the HPA; Access a key unique to the device; Use (i) the key, (ii) the HPA, and (iii) an initialization vector (IV) to generate a first integrity check vector (ICV) that can be used to authenticate the device; Send to the device (i) the first ICV, (ii) the HPA, and (iii) the IV; Receive a second ATS-enabled request from the device, where the second ATS-enabled request includes (i) the first ICV, (ii) the HPA, and (iii) the IV; Perform in-line verification on the first ICV received in the second ATS-enabled request, where the in-line verification is performed by: Calculating a second ICV using (i) the key, (ii) the HPA received in the second ATS-enabled request, and (iii) the IV received in the second ATS-enabled request; Compare the second ICV with the first ICV; and Determine whether the second ICV matches the first ICV.

11. The method according to claim 10, wherein the second ATS-enabled request is a read request, and wherein the method further comprises: Send the read request to a memory controller in parallel with the performance of the in-line verification; In response to determining that the second ICV matches the first ICV as part of the in-line verification, perform the following operations: Allow the device to access the data read as part of the read request; Increment the IV; Calculate a third ICV using (i) the key, (ii) the HPA received in the second ATS-enabled request, and (iii) the incremented IV; And Send to the device (i) the third ICV, (ii) the HPA, and (iii) the incremented IV.

12. The method according to claim 10, wherein the second ATS-enabled request is a read request, and wherein the method further comprises: Send the read request to a memory controller in parallel with the performance of the in-line verification; And In response to determining that the second ICV does not match the first ICV as part of the in-line verification, prevent the data read as part of the read request from being transmitted to the device.

13. The method according to claim 10, wherein the second ATS-enabled request is a write request, and wherein the write request is performed at least partially in parallel with the in-line verification.

14. The method according to claim 10, wherein the second ATS-enabled request is a read request, and wherein the method comprises: Issue the read request during a time period that overlaps with the performance of the in-line verification; After determining that the in-line verification has failed, set an invalid bit into a request completion queue associated with the second ATS-enabled request; and In response to obtaining data read as part of the read request, (i) squash the data, (ii) mark the read request as an undefined request, and (iii) generate an interrupt to notify the ATS host that the device has attempted an unauthorized access.

15. The method according to claim 10, wherein the second ATS-enabled request is speculatively executed such that the second ATS-enabled request is executed at least in part before the in-line verification is completed.

Images