Providing a copy of the input / output memory management unit registers to the guest operating system

Abstract

An electronic device includes a processor that executes a guest operating system, an input / output memory management unit (IOMMU), and a main memory that stores an IOMMU backup storage area. The IOMMU backup storage area includes a separate copy of an input / output (MMIO) register set of an IOMMU memory map for each guest operating system centralized for the supported guest operating systems. The IOMMU receives a communication from the guest operating system to access data in a given IOMMU MMIO register. The IOMMU then performs a corresponding access to the data in the copy of the given IOMMU MMIO register in the IOMMU backup storage area associated with the guest operating system.

Description

Background Art

[0001] Related Art

[0002] Some electronic devices (e.g., servers or desktop computers, etc.) support the “virtualization” of electronic device hardware such as input / output (I / O) devices. Virtualization involves an intermediate entity above or within the electronic device, so as to provide software instances (e.g., application programs, etc.) executing on the electronic device with the illusion that they can directly access the electronic device hardware when in fact the intermediate entity intercepts / redirects or otherwise assists the accesses made by the software instances. For example, a common intermediate entity is a “virtual machine”. A virtual machine is a software entity that abstracts the electronic device hardware and emulates or presents the known interfaces of the electronic device hardware, so that software instances can execute on various types and arrangements of underlying electronic device hardware - which may include electronic device hardware that is otherwise incompatible with the software instances. In some electronic devices, the virtual machine supports the execution of one or more operating system instances, called “guest” operating systems. The guest operating systems in turn provide an environment for executing other software instances (such as productivity applications, databases, etc.).

[0003] In some electronic devices, the virtual machine is managed and controlled by a software entity called a hypervisor. The hypervisor can start or initialize the virtual machine; control, monitor, and assist the virtual machine in accessing the electronic device hardware; terminate or shut down the virtual machine, etc. FIG. 1 presents a block diagram showing the virtual machine and the hypervisor. As can be seen in FIG. 1, there are three virtual machines (VMs) 100, under each of which a guest operating system (guest OS) 102 and one or more programs (PRGRMS) 104, such as databases, software applications, etc., are executed. The virtual machines 100 communicate with the hypervisor 106, which is coupled between the host operating system (host OS) 108 and the virtual machines 100. The host operating system 108 provides an interface between the electronic device hardware 110 and the hypervisor 106. In addition, the hypervisor 106 is coupled between the virtual machines 100 and the input / output memory management unit (IOMMU) 112, where the IOMMU 112 serves as the memory management unit and controller for the I / O device hardware 114.

[0004] Operations performed by the hypervisor include handling communication between the electronic device hardware and the guest operating system (or more generally, virtual machines). For example, the hypervisor can translate, redirect, or otherwise assist with communication between the guest operating system and the Input / Output Memory Management Unit (IOMMU). Communications handled by the hypervisor include communications such as Input / Output Memory Management Unit (IOMMU) Peripheral Page Request (PPR) logging and event logging writes, and guest operating system command buffer writes. PPR logging, event logging, and command buffer writes are described in detail in the December 2016 AMD I / O Virtualization Technology (IOMMU) Specification, Revision 3.00, the entire contents of which are incorporated herein by reference.

[0005] Figure 2 presents a block diagram showing communication between the guest operating system and the IOMMU handled by the hypervisor. In Figure 2, many elements are shown in dotted form; these elements are logs, buffers, etc. stored in memory (e.g., the main memory of the electronic device) and thus accessed through typical memory access techniques. The elements in Figure 2, together with the guest operating system 102, the hypervisor 106, and the IOMMU 112, include a guest Peripheral Page Request (PPR) log 200, a guest command buffer (CMDBUF) 202, and a guest event log 204, which are structures (e.g., lists, tables, etc.) in memory for storing communications to and from the guest operating system 102. Additionally, the elements include guest pointers (PTRS) / status registers (REGS) 206, which are a set of locations in memory for storing pointers to guest operating system structures and status information associated with the guest operating system. The elements also include an IOMMU Peripheral Page Request log 208, an IOMMU command buffer 210, and an IOMMU event log 212, which are structures (e.g., lists, tables, etc.) in memory for storing communications to and from the IOMMU 112. The elements also include IOMMU Memory-Mapped Input / Output (MMIO) pointers / status registers (REGS) 214 in the IOMMU 112, which are a set of registers in the IOMMU 112 for storing pointers to various IOMMU 112 structures and status information associated with the IOMMU 112.

[0006] In operation, and using commands as an example, the guest operating system 102 writes commands destined for the IOMMU 112 to the guest command buffer 202 (i.e., to the next available location in the buffer in memory where commands from the guest operating system 102 are stored). The hypervisor 106 (shown as a dashed line in FIG. 2) detects the write by the guest operating system to the guest command buffer 202, obtains and processes the command (e.g., replaces the guest domain ID and / or guest device ID in the command with the corresponding host domain ID and / or device ID, etc.), and stores the processed command in the IOMMU command buffer 210. The hypervisor 106 also updates the tail pointer in the IOMMU MMIO pointer / status register 214 for the IOMMU command buffer 210 to indicate the newly written command (e.g., increments the tail pointer to the next location in the IOMMU command buffer 210). Then, the IOMMU 112 retrieves the command from the IOMMU command buffer 210 using the command buffer head pointer and executes the command, which causes the IOMMU 112 to perform the corresponding action. The hypervisor 106 performs a similar operation: the IOMMU 112 writes to the IOMMU peripheral page request log 208 and the IOMMU event log 212 (e.g., replaces the host device ID with the guest device ID, etc.). Due to the relatively long latency of the memory reads and writes, pointer updates, and other operations performed by the hypervisor 106, using the hypervisor 106 to intervene between the guest operating system 102 and the IOMMU 112 causes latency in processing the communication, keeps the processor busy, and increases traffic on the memory bus of the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] FIG. 1 presents a block diagram showing a virtual machine and a hypervisor.

[0008] FIG. 2 presents a block diagram showing the communication between a guest operating system and an IOMMU processed by a hypervisor.

[0009] Figure 3 A block diagram showing a virtual machine and a hypervisor according to some embodiments is presented.

[0010] Figure 4 A block diagram showing an electronic device according to some embodiments is presented.

[0011] Figure 5 A block diagram showing a memory portion accessed by an IOMMU according to some embodiments is presented.

[0012] Figure 6 A block diagram showing values stored in a guest copy of an IOMMU MMIO register according to some embodiments is presented.

[0013] Figure 7The figure presents a block diagram showing the communication between a guest operating system processed by an IOMMU and the IOMMU according to some embodiments.

[0014] Figure 8 The figure presents a process in which the IOMMU accesses a copy of the IOMMU MMIO register in the IOMMU backing store on behalf of the guest operating system according to some embodiments.

[0015] Figure 9 The figure presents a flowchart of a process in which the guest operating system writes to a copy of the IOMMU MMIO register in the IOMMU backing store and the IOMMU reads from it according to some embodiments.

[0016] Figure 10 The figure presents a flowchart of a process in which the IOMMU writes to a copy of the IOMMU MMIO register in the IOMMU backing store and the guest operating system reads from it according to some embodiments.

[0017] Throughout the figures and the description, like reference numerals refer to like elements. Detailed Description

[0018] The following description is presented to enable any person skilled in the art to make and use the described embodiments, and the following description is provided in the context of a particular application and its requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications. Thus, the described embodiments are not limited to the embodiments shown, but are to be accorded the widest scope consistent with the principles and features disclosed herein.

[0019] Terminology

[0020] In the following description, various terms are used to describe the embodiments. The following is a simplified and general description of one of these terms. Note that the term may have important additional aspects that are not enumerated herein for clarity and brevity, and thus the description is not intended to limit the term.

[0021] Functional block: A functional block refers to a group, collection, and / or set of one or more interrelated circuit elements (such as integrated circuit elements, discrete circuit elements, etc.). Circuit elements are "interrelated" because they share at least one property. For example, interrelated circuit elements can be included in a specific integrated circuit chip or a part thereof, fabricated on a specific integrated circuit chip or a part thereof, or otherwise coupled to a specific integrated circuit chip or a part thereof, can be involved in the execution of a given function (computing or processing function, memory function, etc.), can be controlled by a common control element and / or a common block, etc. A functional block can include any number of circuit elements, from a single circuit element (e.g., a single integrated circuit logic gate) to millions or billions of circuit elements (e.g., integrated circuit memory).

[0022] Virtualization, Virtual Machines, and Hypervisors

[0023] The described embodiments support the "virtualization" of electronic device hardware such as memories, input / output (IO) devices, etc. Virtualization generally involves an intermediate entity above or within the electronic device, thus providing software instances executing on the electronic device with the illusion that they can directly access the electronic device hardware when in fact the intermediate entity intercepts / redirects, translates, or otherwise assists the access made by the software instances. For example, the intermediate entity can present a set of electronic device registers, memory locations, electronic device settings, and other functional blocks to the software instance, which appear to the software instance to be the actual device registers, memory locations, etc. of the electronic device, but are just copies presented by the intermediate entity. In this case, the intermediate entity receives, intercepts, or otherwise obtains access to the copies of the electronic device hardware and interacts with the actual electronic device hardware on behalf of the software instance. There are many benefits to the virtualization of electronic device hardware, such as enabling different electronic devices to use different arrangements of electronic device hardware, different addresses, locations, or identifiers of the electronic device hardware, etc., while the software instances have the same interface to the electronic device hardware via the intermediate entity. In addition, the intermediate entity can determine whether to allow or block access to the electronic device hardware by a given software instance, and thus the virtualization of electronic device hardware enables the protection of the electronic device hardware (or parts thereof) and / or the software instances executing on the electronic device. By controlling access as described above, the intermediate entity can share the electronic device hardware among multiple software instances and / or provide exclusive access to parts of the electronic device hardware to individual software instances.

[0024] In the described embodiments, the intermediate entity includes a "virtual machine". A virtual machine is a software entity that abstracts electronic device hardware and presents a known interface to the software instance to actual or emulated electronic device hardware. The abstracted hardware enables the software instance to execute on various types and arrangements of underlying electronic device hardware - possibly including electronic device hardware that would otherwise be incompatible with the software instance. In the described embodiments, the virtual machine supports the execution of one or more operating system instances, called "guest" operating systems. The guest operating systems, in turn, provide an environment for the execution of other software programs (such as applications, databases, etc.).

[0025] In the described embodiments, the virtual machine is managed and controlled by a software entity called a hypervisor. The hypervisor can start or initialize the virtual machine; control, monitor, and assist the virtual machine's access to the electronic device hardware; terminate or shut down the virtual machine, etc. Figure 3 The figure presents a block diagram of a virtual machine and a hypervisor according to some embodiments. As Figure 3 can be seen, there are three virtual machines (VMs) 300, under each of which a guest operating system (guest OS) 302 and one or more programs (PRGRMS) 304, such as databases, software applications, etc., are executed. The virtual machines 300 communicate with a hypervisor 306, which is coupled between a host operating system (host OS) 308 and the virtual machines 300. The host operating system 308 provides an interface between the electronic device hardware 310 and the hypervisor 306. Different from that shown in FIG. 1 for existing electronic devices, in Figure 3 this case, an IOMMU 312 is directly coupled between the guest operating system 302 and the IO device hardware 314 without the intervention of the hypervisor 306 (as indicated by the thicker line between the IOMMU 312 and the guest operating system 302). Thus, different from existing electronic devices, in the described embodiments, the hypervisor 306 is not responsible for performing at least some operations for handling the communication between the guest operating system 302 and the IOMMU 312, as described herein. However, note that some communication occurs between the IOMMU 312 and the hypervisor 306, as indicated by the line between the hypervisor 306 and the IOMMU 312. Additionally, note that in some embodiments, the host operating system 308 does not exist and the hypervisor 306 communicates more directly with the electronic device hardware 310.

[0026] Overview

[0027] In the described embodiments, an electronic device includes a processor, a memory (e.g., main memory), multiple input / output (I / O) devices (e.g., network interface devices, disk controllers, etc.), and an input / output memory management unit (IOMMU) coupled between the processor and the I / O devices. The processor executes a hypervisor, one or more virtual machines, and guest operating systems within the virtual machines. Each guest operating system in the guest operating systems is assigned a guest portion of the memory (e.g., a contiguous or non - contiguous region or block of the memory), and the guest portion is reserved for storing data and information to be accessed by that guest operating system. In the described embodiments, the IOMMU performs operations to handle communication between the guest operating systems and the IOMMU.

[0028] As part of handling communication between the guest operating systems and the IOMMU, unlike a single IOMMU copy that uses certain buffers and logs in an existing system, the IOMMU directly accesses the buffers and logs of each guest operating system in the corresponding guest portion of the memory. In other words, the IOMMU accesses not a copy of the buffers and logs maintained by the IOMMU (and the hypervisor) in the existing system, but a single copy of the buffers and logs used by each guest operating system in the guest operating systems. For example, in some embodiments, each guest operating system maintains buffers and logs in the corresponding guest portion of the memory, including command buffers, event logs, and / or peripheral page request (PPR) logs, and the IOMMU directly reads information from and / or writes information to the buffers and logs of each guest operating system from the memory portion of that guest operating system. For example, the IOMMU can read commands from the command buffer of a given guest operating system in the memory guest portion for that given guest operating system.

[0029] In existing systems where the IOMMU uses a single copy of buffers and logs, the IOMMU maintains an input / output (MMIO) register set of the IOMMU memory map for storing information about the buffers and logs. For example, the IOMMU MMIO registers include information such as a set of pointers to the buffers and logs (e.g., head and / or tail pointers for each log or buffer, etc.) and control values (e.g., size indicators and / or configuration values for each log or buffer, etc.). Since in the described embodiments, the IOMMU accesses the buffers and logs in different guest portions of memory (i.e., at different memory addresses and possibly with different control values), a single set of IOMMU MMIO registers is not sufficient. Thus, in the described embodiments, the IOMMU is provided with a separate copy of the IOMMU MMIO register set for each guest operating system, the separate copy being used to access the buffers and logs in the corresponding guest portion of memory. In other words, in some embodiments, if the system supports N guest operating systems, the IOMMU is provided with N copies of the IOMMU MMIO registers (i.e., a set of pointers and control values for the buffers and logs), one copy for each guest operating system.

[0030] Due to the number of registers (which may be very large) required to track the buffers and logs of all possible guest operating systems, the separate copies of the IOMMU MMIO registers do not actually exist in the IOMMU. Instead, the separate copies of the IOMMU MMIO register sets for the guest operating systems are stored in the IOMMU backing store, which is a portion of the main memory reserved for storing IOMMU data and information. In the described embodiments, the IOMMU "virtualizes" the IOMMU MMIO registers, or presents to the guest operating system (and possibly other entities such as a hypervisor, etc.) the impression that the guest operating system can access the IOMMU MMIO registers via a specified IOMMU MMIO address. However, there are no registers at the IOMMU MMIO address. Instead, the IOMMU intercepts the guest operating system's access to the IOMMU MMIO registers at the IOMMU MMIO address and redirects the access to the IOMMU MMIO registers to the corresponding location in the IOMMU backing store. For example, in some embodiments, the IOMMU MMIO addresses include addresses from a specified address range provided by the IOMMU in the IOMMU interface / aperture, and these addresses are used to map communications to the copies of the IOMMU registers in the IOMMU backing store.

[0031] In some embodiments, during operation, the IOMMU receives a communication from the guest operating system to access data in a given IOMMU MMIO register. For example, after the guest operating system has written a command to the command buffer of the guest portion of memory, the guest operating system may transmit a memory write to update the command buffer tail pointer received by the IOMMU. The IOMMU then performs a corresponding access to the data in the copy of the given IOMMU MMIO register in the IOMMU backing store associated with the guest operating system. Continuing with this example, the IOMMU may write the updated data (e.g., the new address of the command buffer tail pointer) to the copy of the command buffer tail pointer register in the IOMMU backing store for the guest operating system. In addition to the guest operating system accessing the copy of the IOMMU MMIO register, in some embodiments, the IOMMU itself updates the copy of the IOMMU MMIO register in the backing store. For example, in some embodiments, the IOMMU may update the command buffer head pointer (and / or other command buffer values) for the guest operating system after having processed commands from the guest command buffer in a first-in, first-out order.

[0032] In some embodiments, due to the virtualization of the addresses associated with the copies of the IOMMU MMIO registers in the IOMMU backing store, various entities perform operations to calculate, translate, or otherwise determine the physical address in the IOMMU backing store where the copy of the IOMMU MMIO register is located. In some of these embodiments, a given guest operating system accesses the IOMMU MMIO register using a corresponding guest virtual address, which is a local address generated by or provided to the given guest operating system for the IOMMU MMIO register. The guest virtual address is translated by a memory management unit (MMU) in the processor (e.g., using one or more page tables) into an IOMMU MMIO address - i.e., into a system physical address that falls within the MMIO aperture / interface of the IOMMU and within the range used to map the guest copy of the IOMMU MMIO register. Since the IOMMU MMIO address has been virtualized for use by multiple guest operating systems, the IOMMU then determines the address of the specific copy of the IOMMU MMIO register for the given guest operating system. For this process, the IOMMU first calculates an IOMMU virtual address for the copy of the IOMMU MMIO register of the given guest operating system based on the identifier of the given guest operating system and the system physical address provided by the MMU, using an algorithm, table, etc. The IOMMU then uses one or more page tables to translate the IOMMU virtual address into the system physical address in the IOMMU backing store that stores the copy of the IOMMU MMIO register for the given guest operating system.

[0033] In some embodiments, when the guest operating system detects an update to the data in the IOMMU MMIO registers, the IOMMU performs one or more corresponding processing operations. For example, in some embodiments, in response to detecting that the guest operating system has updated the tail pointer of the command buffer of the guest operating system in the corresponding IOMMU MMIO register (i.e., its copy in the backing store), the IOMMU retrieves commands from the command buffer of the guest operating system and processes the commands.

[0034] In some embodiments, the hypervisor simulates certain IOMMU MMIO registers that are not included in the separate copy of the IOMMU MMIO register set for the guest operating system stored in the IOMMU backing store. For example, in some embodiments, the hypervisor simulates the IOMMU control MMIO register, the translation table base address MMIO register, etc. of the IOMMU. In these embodiments, the hypervisor intercepts accesses by the guest operating system to the simulated IOMMU MMIO registers and performs the corresponding accesses via the IOMMU.

[0035] In some embodiments, the IOMMU stores a copy of the information of the copies of the IOMMU MMIO registers from some separate IOMMU MMIO register sets in the local cache memory in the IOMMU. For example, the IOMMU can store copies of the most recently used or frequently used values from the respective IOMMU MMIO registers for the corresponding guest operating systems.

[0036] By providing a separate copy of the IOMMU MMIO register set for accessing the buffers and logs in the guest portion of the memory, the IOMMU enables direct access to the buffers and logs in the guest portion of the memory. This means that, like existing systems, the IOMMU has no obligation to rely on the hypervisor to handle accesses to the buffers and logs. Moving these operations from the hypervisor (implemented in software) to the IOMMU (implemented in hardware) speeds up the operation, requires less memory system bandwidth, and reduces the load on the computational functional blocks in the processor, which improves the overall performance of the electronic device. The improvement in the performance of the electronic device leads to higher user satisfaction.

[0037] Electronic Device

[0038] Figure 4 The figure shows a block diagram of an electronic device 400 according to some embodiments. As Figure 4As can be seen, the electronic device 400 includes a processor 402, a memory 404, a mass storage device 406, input / output (I / O) devices 408 - 412, an input / output (I / O) hub 414, and a memory controller 416.

[0039] The processor 402 is a functional block that performs computing operations in the electronic device 400. The processor 402 includes two cores 418 - 420, and each core includes one or more computing mechanisms, such as a central processing unit (CPU) core, a graphics processing unit (GPU) core, an embedded processor, an application specific integrated circuit (ASIC), and / or other computing mechanisms. The processor 402 also includes a memory management unit (MMU) 422, which is a functional block that performs operations associated with address translation (e.g., page table traversal, translation lookaside buffer lookup, etc.), memory access protection, etc., for memory access by the cores 418 - 420.

[0040] The memory 404 is a functional block that performs the operations of the memory (e.g., “main” memory) in the electronic device 400. The memory 404 includes: a memory circuit, such as a dynamic random access memory (DRAM), a double data rate synchronous DRAM (DDR SDRAM), and / or one or more of other types of memory circuits for storing data and instructions used by other functional blocks in the electronic device 400; and a control circuit for handling access (e.g., read, write, check, delete, invalidate, etc.) to the data and instructions stored in the memory circuit.

[0041] The mass storage device 406 is a functional block and / or device that performs the operations of high - capacity non - volatile storage elements to store data and instructions for use by other functional blocks in the electronic device 400. The mass storage device 406 can be or include a high - capacity semiconductor memory (e.g., flash memory, etc.), a disk drive (hard disk drive, etc.), an optical drive, etc. A copy of the data and instructions stored in the mass storage device 406 is retrieved and stored in the memory 404 for use by other functional blocks in the electronic device 400. For example, in some embodiments, data and / or instructions are retrieved from the mass storage device 406 in blocks or “pages” of a given size (e.g., 4 kB, 2 MB, etc.) and the pages are stored in the memory 404 for access by other functional blocks. Additionally, pages can be newly created at available locations in the memory 404 (e.g., for storing computation results, etc.).

[0042] The IO devices 408 - 412 are functional blocks and / or devices that perform corresponding IO operations. The specific nature of the IO operations performed by each of the IO devices 408 - 412 depends on the nature of the IO device. For example, the IO devices 408 - 412 can include human - machine interface devices, network interface devices, audio / visual processing or providing devices, GPUs, sensor devices, disk controllers, Peripheral Component Interconnect (PCI) devices, Universal Serial Bus (USB) devices, etc., and each IO device performs associated operations such as receiving input from a person (e.g., keyboard, mouse, etc.), receiving or sending data over a network, etc. The IO devices 408 - 412 provide data and / or instructions to other functional blocks in the electronic device 400 or consume data and / or instructions from them. For example, in some embodiments, the IO devices 408 - 412 access (i.e., read, write, invalidate, etc.) data in memory pages in the guest memory 428 (i.e., the portion of the memory reserved for a given guest operating system).

[0043] The IO hub 414 is a functional block that performs the operations of an input - output hub, which is coupled between the IO devices 408 - 412 and other functional blocks in the electronic device 400 (e.g., the processor 402, the memory 404, etc.). The operations performed by the IO hub 414 include operations for ensuring that communications to the IO devices 408 - 412 reach the intended IO device, communications from the IO devices 408 - 412 reach the other functional blocks correctly, preventing unauthorized access by the IO devices 408 - 412 to the other functional blocks, and vice versa, etc. In some embodiments, the IO hub 414 is coupled between buses that use different communication standards (such as between the Peripheral Component Interconnect Express (PCIe) bus and HyperTransport and so on), and thus converts or translates the associated communications.

[0044] The I / O hub 414 includes an IOMMU 424, which is a functional block that performs operations to enable the I / O devices 408 - 412 to access data and / or instructions in the memory 404, communicate with the processor 402 (and the guest operating system executed thereby), and so on. In these embodiments, when an I / O device in the memory 404 (e.g., the I / O device 408) wants to access data and instructions, the I / O device sends a memory access request (e.g., a direct memory access request or DMA) to the IOMMU 424. Then, the IOMMU 424 sends a corresponding request to the memory 404 to satisfy the memory access request. For example, in some embodiments, if data is to be retrieved based on the memory access request, the IOMMU 424 fetches the data from the memory 404 (or the mass storage device 406 if the data does not exist in the memory 404) and forwards the data to the requesting I / O device. In some embodiments, the IOMMU 424 includes page tables, translation lookaside buffers, and / or other functional blocks for translating the "virtual" or native memory addresses used by the I / O devices 408 - 412 into the physical addresses in the memory 404 where the data actually resides.

[0045] In the described embodiments, the IOMMU 424 communicates with the guest operating system executed by the cores 418 - 420 in the virtual machine, and vice versa. For example, in some embodiments, the IOMMU 424 (or the I / O devices 408 - 412 via the IOMMU 424) conveys events and peripheral page requests (PPRs) to the guest operating system. In these embodiments, the IOMMU 424 reports events such as I / O page faults (representing page table walks for the I / O devices 408 - 412), IOMMU 424 hardware errors, etc. to the guest operating system via a shared guest event log in the memory 404. Additionally, in these embodiments, the IOMMU 424 forwards PPRs from peripheral devices (I / O devices) to the guest operating system, and the PPRs perform memory page service operations (i.e., perform operations on or associated with pages in the memory 404 accessible by the guest operating system) using well-known address translation services or ATS standards via a shared guest PPR log in the memory 404. As another example, in some embodiments, the guest operating system sends commands to the IOMMU 424. In these embodiments, the guest operating system issues commands to the IOMMU 424 via a shared guest command buffer in the memory 404 to control the IOMMU 424 and / or the I / O devices 408 - 412, such as a complete wait (which serves as a command barrier to force an earlier command to complete before the IOMMU 424 proceeds), invalidate device table entries, invalidate IOMMU 424 translation lookaside buffer entries, etc.

[0046] In some embodiments, the IOMMU 424 provides an interface to the guest operating system that includes memory-mapped locations, registers, etc. for communicating with the IOMMU 424. For example, in some embodiments, the IOMMU 424 provides a set of memory-mapped input / output (MMIO) memory locations to which the guest operating system can write values such that the IOMMU 424 will receive the values. In some embodiments, the interface is virtualized in that the memory locations, registers, etc. are not used to store values as assumed by the guest operating system but are simply presented by the IOMMU 424. In these embodiments, the IOMMU 424 can receive values from the guest operating system (e.g., addressed to the IOMMU MMIO address, etc.) via the interface but uses the IOMMU backing store 426 and / or other locations in the memory 404 to store separate copies of the values in the memory locations, registers, etc. of each guest operating system. The memory accessed by the IOMMU 424 to communicate with the guest operating system and other entities (e.g., the processor 402, etc.) is described in more detail below.

[0047] In some embodiments, although not shown in Figure 4 the IOMMU 424 includes a local cache memory for storing copies of data or information of the IOMMU 424. For example, in some embodiments, the cache memory is used to store copies of recently used or frequently used data from the IOMMU backing store 426, such as a set of values of the IOMMU MMIO registers of a given guest operating system and / or individual values therein. The cache memory is smaller than the IOMMU backing store 426 and thus may only have the capacity (i.e., memory locations) to store a portion (and potentially a small portion) of the data and information stored in the IOMMU backing store 426.

[0048] The guest memory 428 is a part of the memory 404 (e.g., one or more contiguous or non - contiguous pages or blocks of the memory), and this part is used by the corresponding guest operating system to store data and information that will be used by the guest operating system. Generally, the guest operating system and / or other entities can use the guest memory 428 to store any form of data and information used by the guest operating system and / or other entities. In some embodiments, the guest memory 428 is protected and only certain entities are allowed to access the guest memory 428. For example, the corresponding guest operating system, hypervisor, security processor, and / or operating system in the electronic device 400 can "protect" the guest memory 428 by restricting access to the guest memory 428 to the corresponding guest operating system and specified other devices and / or functional blocks. In some embodiments, the guest memory 428 is encrypted or otherwise made inaccessible to undesired entities. In some embodiments, the guest memory 428 is used to store guest event logs, guest peripheral page request (PPR) logs, and guest command buffers, which are data structures (e.g., tables, lists, etc.) for communication between the guest operating system and the IOMMU. The guest event logs, guest peripheral page request (PPR) logs, and guest command buffers are described in more detail below.

[0049] In some embodiments, communication paths are coupled between various functional blocks (processor 402, memory controller 416, memory 404, etc.) in the electronic device 400, as shown by the arrow lines between the elements. The communication paths include one or more buses, wires, conductors, and / or other connections that may be together with controllers, structural elements (switches, routers, etc.), circuit elements, etc. The communication paths are used to route commands, data, control signals, and / or other information between the functional blocks. For example, in some embodiments, a coherent bus structure or interconnect is coupled between the IO hub 414, the processor 402 (e.g., MMU 422), and the memory 404. Note that for clarity, some communication paths in the electronic device 400 are not shown in Figure 4 the figure.

[0050] In some embodiments, Figure 3 the electronic device hardware 310 in Figure 3 includes functional blocks and devices such as the processor 402 and the memory 404, and the IO device hardware 314 includes functional blocks and devices such as the IO devices 408 - 412. In these embodiments, Figure 4 the IOMMU 312 in

[0051] The electronic device 400 is shown as using a specific number and arrangement of components (e.g., functional blocks and devices such as processor 402, memory 404, etc.) and communication paths. However, for illustrative purposes, the electronic device 400 is simplified, and in some embodiments, there are different numbers or arrangements of components and / or communication paths in the electronic device 400. For example, the electronic device 400 may include a power supply subsystem, a display, etc. Generally, the electronic device 400 includes sufficient components and communication paths to perform the operations described herein.

[0052] The electronic device 400 can be or can be included in any electronic device that performs computing operations. For example, the electronic device 400 can be or can be included in an electronic device such as a desktop computer, a laptop computer, a wearable electronic device, a tablet computer, a smart phone, a server, an artificial intelligence device, virtual or augmented reality equipment, a network appliance, a toy, an audio-visual equipment, a household appliance, a controller, a vehicle, etc. and / or a combination thereof.

[0053] Memory Portion Accessed by IOMMU

[0054] In some embodiments, the IOMMU accesses data and information in different portions of a memory (e.g., memory 404) to perform the operations described herein. In some of these embodiments, the portions of the memory include an IOMMU backing store (e.g., IOMMU backing store 426), guest memory (e.g., guest memory 428), and / or hypervisor memory. Figure 5 The figure presents a block diagram showing the portions of the memory accessed by the IOMMU according to some embodiments. Although Figure 5 presented as an example, in some embodiments, the memory and / or different portions of the memory store different types and / or arrangements of information. Generally, the memory includes sufficient information to implement the operations described herein.

[0055] As Figure 5 can be seen, the IOMMU backing store 500 includes an ID translation table 502. Generally, the ID translation table 502 includes information that the IOMMU uses to translate or convert guest domain IDs and / or device IDs in communications from the guest operating system to the IOMMU (or vice versa, from the IOMMU to the guest operating system) into host domain IDs and / or device IDs. Domain IDs and device IDs are described in detail in the AMD I / O Virtualization Technology (IOMMU) Specification, Revision 3.00, December 2016, which is incorporated herein by reference as described above.

[0056] In some embodiments, the ID translation table 502 includes a table for domain IDs (shown as the domain ID mapping table 504) and a table for device IDs (shown as the device ID mapping table 506) respectively, but separate tables are not required (and thus all translations can be included in a single table). The domain ID mapping table 504 includes a set of entries, each entry for storing the identity or indication of a guest domain ID associated with or related to a specified host domain ID. The device ID mapping table 506 includes a set of entries, each entry for storing the identity or indication of a guest device ID associated with or related to a specified host device ID. In operation, when a guest or host domain ID and / or device ID is to be translated or converted in a communication, the IOMMU performs a lookup in the ID translation table 502 (i.e., the domain ID mapping table 504 and / or the device ID mapping table 506) to obtain the corresponding translation or conversion.

[0057] The IOMMU backing store 500 also includes guest control 508. Generally, the guest control 508 includes a copy of the values stored in or from the interface registers and control registers of the guest operating system in the electronic device. For each supported guest operating system, the guest control 508 includes a copy of the guest interface registers and / or guest operating system control registers (or at least the values therein) that control the interaction between the IOMMU and that guest operating system. By way of example, for each guest operating system, the guest control 508 may include mapping control registers for transmitting the domain ID and / or device ID mappings of the guest operating system to the IOMMU.

[0058] The IOMMU backing store 500 also includes guest memory mapped input / output (MMIO) 510. Generally, the guest MMIO 510 includes pointers and control information for accessing buffers and logs (such as guest command buffers, guest event logs, and guest PPR logs) of the guest operating system in the guest portion of the memory 404 (e.g., guest memory 428). More specifically, for each supported guest operating system, the guest MMIO 510 includes a copy of the values for controlling access to the buffers and logs in the guest portion of the memory 404. By way of example, in some embodiments, the IOMMU supports (can interact with, process its communications, etc.) 2 N guest operating systems, where N = 10, 16, or other values, and thus the guest MMIO 510 includes up to 2 N copies of the values, one for each supported guest operating system. In the described embodiments, the values for controlling access are similar to the values stored in the IOMMU MMIO registers in existing devices, but separate sets of values are reserved for each supported guest operating system, referring to the guest portion of the memory 404 for that guest operating system (as opposed to a single copy in the IOMMU in existing devices).

[0059] Figure 6 The figure presents a block diagram showing values stored in the guest MMIO 510 according to some embodiments. For Figure 6 the example in, the values of a single complete set of IOMMU MMIO registers for a given guest operating system and the values of portions of two adjacent sets of IOMMU MMIO registers above and below the single complete set of values are shown. Presented in this way Figure 6 to illustrate that, as described above, in some embodiments, the IOMMU backing store 426 includes multiple separate sets of values, each set associated with a given guest operating system. Although Figure 6 presented as an example, in some embodiments, the guest MMIO 510 stores different values or values in a different arrangement. Generally, the guest MMIO 510 and / or another entity stores sufficient information to access buffers and logs in the guest portion of the memory as described herein.

[0060] Each value in the guest MMIO 510 can generally be grouped into values associated with a guest command (CMD) buffer, a guest event log, and a guest PPR log of the guest operating system. For the guest command buffer, the values include a command tail pointer 600, which is a pointer or other reference indicating the end, tail, or most recently written entry of the guest command buffer in the corresponding guest portion of the memory 404. In other words, the command tail pointer 600 holds a pointer or reference, such as a memory address, indicating the location of the end or most recently written entry of the command buffer, e.g., some or all bits of a physical address in the guest portion of the memory 404. The values of the guest command buffer also include a command head pointer 602, which is a pointer or other reference indicating the head or base of the command buffer, i.e., the location where the command buffer starts and / or where the buffered commands start in the corresponding guest portion of the memory 404. The values of the guest command buffer also include a command length 604, which is a value indicating the current size or maximum size of the command buffer, such as the number of entries or the number of bytes. The values of the guest command buffer also include a command control 606, which is a bit set or sequence of bits that includes a plurality of bits (or combinations thereof) indicating the configuration of the command buffer in the guest operating system and / or the corresponding guest portion of the memory 404. By way of example, in some embodiments, the command control 606 includes bits indicating whether certain command buffer-related interrupts are enabled, whether command buffering or processing is enabled (or disabled / paused), what types of commands are present or allowed to be present in the command buffer, and the like. By way of example, in some embodiments, the command control 606 includes control values for the command buffer similar to those described in the AMD I / O Virtualization Technology (IOMMU) Specification Revision 3.00, which is incorporated herein by reference as described above - such as CmdWaitInt, CmdBufRun, CmdWaitInteEn, CmdBufEn, and / or other values.

[0061] In some embodiments, the command buffer is implemented as a circular buffer by using the command head pointer 602 to point to the next entry / location in the command buffer to be read and using the command tail pointer 600 to indicate the last / most recently added entry / location in the command buffer (when only one command is stored in the command buffer, they can be the same entry / location). In these embodiments, as commands are read from the entries and processed, the command head pointer 602 advances by one entry until the command head pointer 602 and the command tail pointer 600 indicate the same entry or adjacent entries. In some of these embodiments, the command length 604 stores the maximum length of the command buffer. Implementing a circular buffer using pointers is well known in the art and will not be described in detail herein.

[0062] For the event log, the values in the guest MMIO 510 (i.e., the event tail pointer 608, event head pointer 610, event length 612, and event control 614) are functionally similar to those described above for the command buffer, but these values are used to access the event log in the corresponding guest portion of the memory 404. The same is true for the PPR log, since the values in the guest MMIO 510 (i.e., the PPR tail pointer 616, PPR head pointer 618, PPR length 620, and PPR control 622) are functionally similar to those described above for the command buffer, but these values are used to access the PPR log in the corresponding guest portion of the memory 404.

[0063] In some embodiments, the number of guest operating systems executing in the electronic device 400 at any given time may be less than the supported number (and may be much less) - and thus, in these embodiments, the IOMMU backing store 500 is dynamically adjusted to include sufficient space for storing the guest MMIO 510, etc. In other words, when a guest operating system is initialized, the hypervisor and / or another entity may allocate, append, or activate space in the IOMMU backing store 500 (i.e., in the guest MMIO 510) for storing the IOMMU MMIO register values of the guest operating system.

[0064] In some embodiments, the hypervisor performs at least some initialization operations for the IOMMU backing store 500. For example, in some embodiments, the hypervisor allocates memory in the memory 404 for storing the IOMMU backing store, such as contiguous or discontiguous memory pages. As described above, the memory 404 is a general-purpose memory in the electronic device 400 that is used by various functional blocks (e.g., cores 418 - 420, etc.) for storing data and information - and not just the local memory in the IOMMU. Thus, this operation involves the hypervisor allocating space in the "main" memory of the electronic device 400 for storing the IOMMU backing store. As another example, in some embodiments, the hypervisor writes initial values to the copies of the IOMMU MMIO registers in the backing store of each active guest operating system via the IOMMU. Thus, the hypervisor writes the initial values of the pointers and control values of each active guest operating system to the corresponding copies of the IOMMU MMIO registers in the IOMMU backing store, such as when the guest operating system starts up.

[0065] In some embodiments, only a subset of the available IOMMU MMIO registers are included in the copy of the IOMMU MMIO registers in the IOMMU backing store. In these embodiments, the IOMMU itself may provide other registers. For example, in some embodiments, the registers in the IOMMU that store IOMMU control values (e.g., IOMMU enable / disable, etc.), page table base addresses, etc. are provided by the IOMMU. Generally, the IOMMU MMIO registers that are not used to access buffers and logs in the memory portion of the guest operating system are not presented as copies, but rather a single copy of each register is provided by the IOMMU. In some of these embodiments, the hypervisor simulates these IOMMU registers for the guest operating system, but accesses to the registers in the IOMMU are received by the IOMMU and processed within the IOMMU.

[0066] Return Figure 5 , the guest memory 512 includes a guest event log 514, a guest peripheral page request (PPR) log 516, and a guest command buffer 518 of the guest operating system. Generally, the guest event log 514, the guest PPR log 516, and the guest command buffer 518 are memory structures (e.g., lists, tables, buffers, etc.) for storing the corresponding events, PPR requests, and commands for access by the IOMMU and / or the guest operating system. In operation, the IOMMU transmits events and PPRs to the guest operating system via the corresponding logs in the guest event log 514 and the guest PPR log 516 in the guest memory 512. In addition, the guest operating system transmits commands to the IOMMU via the corresponding command buffer in the guest command buffer 518 in the guest memory 512.

[0067] In some embodiments, each guest operating system active in the electronic device 400 is associated with a corresponding separate guest portion of the memory (i.e., multiple pages in the memory 404), the separate guest portion including a guest event log, a peripheral page request log, and a guest command buffer that are used by the guest operating system and accessible by the IOMMU. This is shown in Figure 5 as additional guest memories 520 - 522 behind the guest memory 512.

[0068] The hypervisor memory 524 includes a device table 526. Generally, the device table 526 is a table that stores device-related information of devices (which can be actual / physical devices or virtual devices) associated with and / or coupled to an electronic device. The device table 526 includes a set of entries, each of which can be used to store information about a corresponding device, such as pointers to a page table and an interrupt table, control and configuration values, function indicators, mode indicators, domain IDs, security information and settings, etc. In addition, in the described embodiments—and different from existing device tables—each entry in the device table 526 includes a device ID and a guest identifier of a guest operating system that communicates with, is responsible for, or is otherwise associated with the device. In operation, in addition to using the device table to determine information about a device, the IOMMU also uses the device ID and / or the guest identifier to translate or convert a guest device ID into a host device ID.

[0069] In some embodiments, some or all and / or portions of the IOMMU backing store 500, the guest memory 512, and the hypervisor memory 524 are not contiguous, but are stored in different regions or locations of the memory. For example, the base address of the guest event log 514 (and thus the guest event log itself) can be located in the memory far from the guest PPR log 516. Therefore, the guest event log 514 may not be adjacent to the guest PPR log 516 as Figure 5 shown.

[0070] In some embodiments, the IOMMU includes a private address map that includes pointers, references, and / or other indications of the locations in the memory of various data and information accessed by the IOMMU in the memory. For example, the IOMMU private address map can include pointers to the base addresses in the memory for the guest event log 514, the guest PPR log 516, etc. In these embodiments, before accessing data and information in the memory, the IOMMU looks up the locations of the data and information in the private address map—and can also translate these addresses using a page table, etc. before use.

[0071] In some embodiments, the IOMMU backing store 500 and / or portions thereof (control bits, etc.) are accessed by other entities in the electronic device 400 via the IOMMU (e.g., by sending requests to the IOMMU) or are not accessible by other entities. For example, at least some of the data and information in the IOMMU backing store 500 can be accessed by other entities via writes to and reads from the corresponding IOMMU MIO registers.

[0072] IOMMU Communicates with Guest Operating System

[0073] In the described embodiments, an IOMMU (e.g., IOMMU 424) handles communications between the IOMMU (or an IO device served thereby) and a guest operating system. Figure 7 The figure presents a block diagram showing communications between a guest operating system 700 and an IOMMU 702 handled by the IOMMU 702 according to some embodiments. Although multiple elements are shown in a particular arrangement in Figure 7 , other embodiments use different numbers or arrangements of elements. Generally, in the described embodiments, the IOMMU 702 includes or accesses sufficient elements to implement the operations described herein. In Figure 7 , many elements are shown as dotted; these elements are logs, buffers, etc. stored in memory (e.g., in the IOMMU backing store 500, in the guest memory 512, etc.) and accessed by the IOMMU 702, the guest operating system 700, and / or other entities using typical memory access techniques. In some embodiments, the guest operating system 700, the IOMMU 702, and the hypervisor 704 are organized in a manner similar to that of the guest operating system 302, the IOMMU 312, and the hypervisor 306 in Figure 3 , but this is not required.

[0074] As in Figure 7As can be seen, and different from that shown in FIG. 2 for the existing system, in the described embodiments, the IOMMU 702 and the guest operating system 700 communicate with each other more directly. In other words, the IOMMU 702 and the guest operating system 700 communicate with each other via the guest event log 514, the guest PPR log 516, and the guest command buffer (BUFF) 518 in the memory (i.e., the guest portion of the memory of the guest operating system 700 (e.g., guest memory 428)). In addition, the guest operating system 700 and the IOMMU 702 use the guest control 508 and the guest MMIO 510 to specify how the communication is to be performed. For example, in some embodiments, the IOMMU 702 uses a pointer in the guest MMIO 510 to determine the locations in the memory of the guest event log 514, the guest command buffer 518, and the guest PPR log 516 of the guest operating system 700. The hypervisor 704 does not intervene and does not otherwise participate in some or all of the operations for completing these communications. For example, the hypervisor 704 does not perform operations such as translating domain IDs and device IDs for these communications, accessing the pointers in the guest MMIO 510, and / or accessing the buffers and logs in the guest portion of the memory. Instead, the IOMMU 702 performs these operations. Since the IOMMU 702 translates domain IDs and device IDs and obtains information from the guest MMIO 510, etc., the described embodiments avoid using the hypervisor 704 to handle at least part of the communication between the guest operating system 700 and the IOMMU 702, which may mean that the communication is completed faster, resulting in less load on the processor 402, the memory 404, etc.

[0075] In operation, taking commands as an example, the guest operating system 700 writes an invalidate_IOMMU_pages command to the guest command buffer 518, and the command causes the IOMMU 702 to invalidate a series of entries in the IOMMU translation cache, as specified by the domain ID in the command. In other words, the guest operating system performs a memory write in the corresponding guest portion of the memory to update the next open / available entry in the guest command buffer 518 to include data (i.e., bits representing the command) for the invalidate_IOMMU_pages command. Then, the guest operating system 700 sends a write command to the IOMMU to update (e.g., advance, increment, etc.) the command buffer tail pointer (e.g., command tail pointer 600) in the corresponding IOMMU MMIO register to indicate that the guest operating system 700 has written a command to the command buffer. The IOMMU 702 detects the write by the guest operating system 700 to the command buffer tail pointer, e.g., by listening for writes to the address in the corresponding guest command buffer, detecting a change in the value of the buffer tail pointer, receiving a write command from the guest operating system 700, etc. Upon detecting the write to the command buffer tail pointer, the IOMMU 702 uses the value of the command buffer head pointer (e.g., command head pointer 602) to retrieve the next command from the command buffer in the guest portion of the memory and prepares to process the command (e.g., replace the host domain ID associated with the guest domain ID in the command, etc.). Then, the IOMMU 702 processes the command, which causes the IOMMU 702 to invalidate a series of entries in the translation cache of the IOMMU 702 indicated by the host domain ID. The IOMMU 702 performs at least some similar operations: writing to the guest PPR log 516 and the guest event log 514, although in reverse, as the IOMMU 702 writes to these logs and the guest operating system 700 reads the logs.

[0076] Although the hypervisor 704 does not participate in some portions of the communication between the guest operating system 700 and the IOMMU 702, such as the translation of the guest domain ID to the host domain ID, the hypervisor 704 and the guest operating system 700 and / or the IOMMU 702 may exchange communications associated with the communication between the guest operating system 700 and the IOMMU 702 separately, or the hypervisor 704 may otherwise participate in ensuring that the guest operating system 700 and / or the IOMMU 702 correctly processes the communication. For example, the hypervisor 704 may directly (e.g., via communication) or indirectly (e.g., by listening for memory accesses) determine that the IOMMU 702 or the guest operating system 700 has performed a specified operation (e.g., written to a buffer or log) and may perform operations such as sending an interrupt signal to the guest operating system 700 and / or the IOMMU 702, updating a shared memory location used as a flag, etc. As described above, the hypervisor 704 may also initialize the IOMMU backing store, etc.

[0077] Using IOMMU Process of MMIO Register Value

[0078] In the described embodiments, the IOMMU (e.g., IOMMU 702) performs operations for providing a virtual copy of the IOMMU MMIO registers to the guest operating system (e.g., guest operating system 700). As described above, these operations include accessing, on behalf of the guest operating system, the corresponding copies of the IOMMU MMIO registers stored in the IOMMU backing store. Figure 8 The figure presents a process by which the IOMMU accesses a copy of the IOMMU MMIO registers in the IOMMU backing store on behalf of the guest operating system according to some embodiments. Note that Figure 8 The operations shown are presented as general examples of operations performed by some embodiments. Operations performed by other embodiments include different operations, operations performed in a different order, and / or operations performed by different entities or functional blocks.

[0079] Figure 8 The operations shown for accessing the copy of the IOMMU MMIO registers are general within the scope and involve various types of accesses, such as reads, writes, invalidations, etc. Figures 9 to 10 More specific examples are presented in Figure 9 including the guest operating system writing to a copy of the IOMMU MMIO registers in Figure 10 and the guest operating system reading from a copy of the IOMMU MMIO registers in

[0080] Figure 8The operation in [the context] starts with the IOMMU receiving a communication from the guest operating system to access data in a given IOMMU MMIO register (step 800). The access can be to read data in the IOMMU MMIO register and thus the communication is a read request, to write data into the IOMMU MMIO register and thus the communication is a write request, and / or another access such as an invalidate, etc. For example, the IOMMU can receive from the guest operating system a communication to write specific data into the command buffer tail pointer (e.g., command tail pointer 600) to initialize or otherwise update the command buffer tail pointer.

[0081] The IOMMU then performs the corresponding access to the data in the copy of the given IOMMU MMIO register associated with the guest operating system in the IOMMU backing store (step 802). Recall that the IOMMU virtualizes the IOMMU MMIO registers, i.e., presents to the guest operating system (and other entities) the illusion that the guest operating system is accessing the actual MMIO registers in the IOMMU, but the IOMMU actually uses separate copies of the IOMMU MMIO registers stored in the IOMMU backing store for each supported guest operating system. Thus, this operation involves the IOMMU determining which copy of the IOMMU MMIO register to access in the backing store and then performing the access.

[0082] Figure 9 The figure shows a flowchart of the process in which a guest operating system writes to a copy of an IOMMU MMIO register in the IOMMU backing store and the IOMMU reads from it according to some embodiments. Note that Figure 9 The operations shown are presented as general examples of operations performed by some embodiments. The operations performed by other embodiments include different operations, operations performed in a different order, and / or operations performed by different entities or functional blocks.

[0083] Figure 9 The operation in [the context] starts when the guest operating system (e.g., guest operating system 700) writes a command into a command buffer (e.g., guest command buffer area 518) in the guest portion of the memory (e.g., guest memory 428) (step 900). For example, in some embodiments, the guest command buffer is a circular buffer including multiple entries in the guest portion of the memory, and this operation involves the guest operating system writing a command (i.e., a sequence of bits representing the command) into the next available entry in the guest command buffer.

[0084] Because the guest operating system has added commands that need to be processed by the IOMMU (e.g., IOMMU 702) to the guest command buffer, the guest operating system notifies the IOMMU of the commands. More specifically, the guest operating system sends a write request to update the command buffer tail pointer, which points to the guest virtual address of the IOMMU MMIO register that stores the command buffer tail pointer (step 902). In the described embodiment, the guest operating system uses a local or "virtual" address to access the IOMMU MMIO register (and may not know the actual IOMMU address). The guest virtual address is translated by the memory management unit (MMU) in the processor executing the guest operating system into a system physical address in the IOMMU where the MMIO register is located (i.e., the IOMMU MMIO address) (e.g., by performing a page table walk in the page table of the guest operating system, etc.). As described above, the IOMMU virtualizes the IOMMU MMIO register, so the system physical address used by the MMU is not the address where the copy of the guest operating system's IOMMU MMIO register is located, but the address of the IOMMU in the interface / aperture that the IOMMU recognizes as an access to the MMIO register. Therefore, after translation in the MMU, the MMU forwards the write request to the IOMMU at the IOMMU MMIO address (i.e., the system physical address) associated with the guest virtual address (step 904).

[0085] Because the IOMMU MMIO address is just the address provided by the IOMMU for receiving access requests from the guest operating system, the IOMMU also performs translation to determine the physical address in memory where the copy of the guest operating system's command buffer tail pointer is stored based on the write request (step 906). During this operation, the IOMMU uses the identity of the guest operating system to determine the physical address in memory, which specifies which copy of the IOMMU MMIO register to access in the IOMMU backing store. For this process, the IOMMU first calculates an IOMMU virtual address for the copy of the guest operating system's IOMMU MMIO register using an algorithm, table, etc., based on the identifier of the guest operating system and the system physical address provided by the MMU. The IOMMU then uses one or more page tables to translate the IOMMU virtual address into the system physical address in the IOMMU backing store where the copy of the guest operating system's IOMMU MMIO register is stored.

[0086] The IOMMU then stores the data from the write request at the physical address in the memory, thereby updating the copy of the command buffer tail pointer of the guest operating system (step 908). In other words, the IOMMU stores data such as one or more bits of the updated address of the command buffer tail pointer, the entry identifier, the run count, etc. in the memory location in the IOMMU backing store that stores the copy of the command buffer tail pointer associated with the guest operating system. The update causes the command buffer tail pointer to directly or indirectly indicate that a command has been written to the guest command buffer and is thus waiting for IOMMU processing. In some embodiments, the IOMMU, the guest operating system, and / or another entity also set other pointers or indicators to indicate that a command is waiting for processing in the guest command buffer or that the guest operating system has provided a command. For example, in some embodiments, the IOMMU includes a set of command wait bits, one bit for each supported guest operating system, and sets (or clears) the command wait bit for the guest operating system to indicate that a command is waiting for processing in the guest command buffer.

[0087] Based on detecting the write request (or otherwise determining that a command is waiting for processing), the IOMMU uses the data from the copy of the command buffer head pointer (e.g., command head pointer 602) to perform one or more subsequent operations to process the command in the guest command buffer (step 910). For example, in some embodiments, the value in the copy of the command buffer head pointer stores the address of the location in the guest portion of the memory where the next entry of the guest command buffer is located, and the IOMMU directly or indirectly uses that address to obtain the command from the guest command buffer before processing the command.

[0088] Figure 10 The figure shows a flowchart of the process in which the IOMMU writes a copy of the IOMMU MIO register to the IOMMU backing store and the guest operating system reads from it according to some embodiments. Note that Figure 10 The operations shown are presented as general examples of operations performed by some embodiments. The operations performed by other embodiments include different operations, operations performed in a different order, and / or operations performed by different entities or functional blocks.

[0089] Figure 10 The operation in starts when the IOMMU (e.g., IOMMU 702) writes an event to the guest event log (e.g., guest event log 514) in the guest portion of the memory of the guest operating system (e.g., guest memory 428) (step 1000). For example, in some embodiments, the guest event log is a circular buffer in the guest portion of the memory that includes multiple entries, and the operation involves the IOMMU writing an event (i.e., a sequence of bits representing the event) to the next available entry in the guest event log.

[0090] In some embodiments, although not shown for simplicity in Figure 10 , the IOMMU first obtains information about the guest event log from a copy of the event log tail pointer associated with the guest operating system in the IOMMU backing store and / or other copies of the IOMMU MMIO pointer. The IOMMU then uses the information to perform step 1000. For example, in some embodiments, the IOMMU uses a direct or indirect indication (such as a physical address, etc.) of the location of the guest event log (or available entries therein) obtained from a copy of the event log tail pointer of the guest operating system in the IOMMU backing store to write an event to the guest event log. To obtain information about the guest event log, in some embodiments, the IOMMU determines the physical address of a copy of the event log tail pointer and / or other copies of the IOMMU MMIO pointer in a manner similar to step 1002, and then uses the physical address to obtain the information.

[0091] Since the IOMMU has added an event to the guest event log that requires processing by the guest operating system, the IOMMU notifies the guest operating system of the event. In some embodiments, notifying the guest operating system includes updating a copy of the event log tail pointer of the guest operating system in the IOMMU backing store and sending an interrupt to the guest operating system to notify the guest operating system. Accordingly, the IOMMU determines the physical address of a copy of the event log tail pointer of the guest operating system in the IOMMU backing store (step 1002). During this operation, the IOMMU uses the identity of the guest operating system to determine the physical address in memory, which identity specifies which copy of the IOMMU MMIO register and one or more IOMMU private address tables and / or page tables to access in the IOMMU backing store. The IOMMU then stores data at the physical address, thereby updating the copy of the event log tail pointer of the guest operating system (step 1004). In other words, the IOMMU stores data such as one or more bits of the updated address of the event log tail pointer, an entry identifier, a run count, etc. in the memory location in the IOMMU backing store that stores the copy of the event log tail pointer associated with the guest operating system. The update causes the event log tail pointer to directly or indirectly indicate that an event has been written to the event log and is thus awaiting processing by the guest operating system.

[0092] The IOMMU also sends an interrupt to the guest operating system that indicates that there is an event in the guest event log awaiting processing (step 1006). For this operation, the IOMMU can use an interrupt mechanism through which interrupts are passed between the IOMMU and the guest operating system, such as via a virtual advanced programmable interrupt controller (vAPIC) associated with the guest operating system.

[0093] In response to receiving an interrupt, the guest operating system sends a read request to the IOMMU to read data from the event log header pointer (e.g., event header pointer 610), and the read request points to the guest virtual address in the IOMMU MMIO register that stores the event log header pointer (step 1008). In the described embodiments, the guest operating system uses a local or "virtual" address to access the IOMMU MMIO register (and may not know the actual and / or virtual IOMMU address). The guest virtual address is translated by the memory management unit (MMU) in the processor executing the guest operating system into the system physical address in the IOMMU where the MMIO register is located (i.e., the IOMMU MMIO address) (e.g., by performing a page table walk in the page table of the guest operating system, etc.). As described above, the IOMMU virtualizes the IOMMU MMIO register, so the system physical address used by the MMU is not the address where the copy of the IOMMU MMIO register of the guest operating system is located, but the address of the IOMMU in the interface / aperture that is recognized by the IOMMU as an access to the MMIO register. Therefore, after translation in the MMU, the MMU forwards the read request to the IOMMU at the IOMMU MMIO address (i.e., the system physical address) associated with the guest virtual address (step 1010).

[0094] Since the IOMMU MMIO address is only the address provided by the IOMMU to receive access requests from the guest operating system, the IOMMU also performs translation to determine the physical address in memory where the copy of the event log header pointer of the guest operating system is stored based on the read request (step 1012). During this operation, the IOMMU uses the identity of the guest operating system to determine the physical address in memory, and the identity specifies which copy of the IOMMU MMIO register to access in the IOMMU backing store. For this process, the IOMMU first calculates the IOMMU virtual address for the copy of the IOMMU MMIO register of the guest operating system using an algorithm, table, etc. based on the identifier of the guest operating system and the system physical address provided by the MMU. The IOMMU then uses one or more page tables to translate the IOMMU virtual address into the system physical address in the IOMMU backing store where the copy of the IOMMU MMIO register of the guest operating system is stored.

[0095] The IOMMU then reads data for the read request from the memory location at the physical address in the memory (step 1014). In other words, the IOMMU reads data such as one or more bits of an address such as an event log header pointer, an entry identifier, a run count, etc. from the memory location in the IOMMU backing store that stores a copy of the event log header pointer associated with the guest operating system. The event log header pointer directly or indirectly indicates the next event that has been written to the guest event log and is waiting for the guest operating system to process. In some embodiments, the IOMMU, the guest operating system, and / or another entity also set other pointers or indicators to indicate that there is an event waiting to be processed in the guest event log or that the IOMMU has provided an event. For example, in some embodiments, the guest operating system and / or the hypervisor includes a set of event waiting bits, one bit for each supported guest operating system, and the event waiting bit is set (or cleared) for the guest operating system to indicate that there is an event waiting to be processed in the guest event log.

[0096] After reading the data from the event log header pointer, the IOMMU returns the data to the guest operating system (step 1016). The guest operating system then uses the data to perform one or more subsequent operations to process the event in the guest event log (step 1018). For example, in some embodiments, a copy of the event log header pointer of the guest operating system stores the address of the location in the guest portion of the memory where the entry of the event log is located, and the guest operating system directly or indirectly uses the address to obtain the event from the guest event log before processing the event.

[0097] In some embodiments, an electronic device (e.g., electronic device 400 and / or a portion thereof) uses code and / or data stored on a non-transitory computer-readable storage medium to perform some or all of the operations described herein. More specifically, when performing the described operations, the electronic device reads the code and / or data from the computer-readable storage medium and executes the code and / or uses the data. The computer-readable storage medium can be any device, medium, or combination thereof that stores the code and / or data for use by the electronic device. By way of example, the computer-readable storage medium can include, but is not limited to, volatile and / or non-volatile memory, including flash memory, random access memory (e.g., eDRAM, RAM, SRAM, DRAM, DDR4 SDRAM, etc.), read-only memory (ROM), and / or magnetic or optical storage media (e.g., disk drives, tapes, CDs, DVDs, etc.).

[0098] In some embodiments, one or more hardware modules perform the operations described herein. By way of example, the hardware modules can include, but are not limited to, one or more processors / cores / central processing units (CPUs), application specific integrated circuit (ASIC) chips, neural network processors or accelerators, field programmable gate arrays (FPGAs), computing units, embedded processors, graphics processing units (GPUs) / graphics cores, pipelines, accelerated processing units (APUs), caches / cache controllers, memories / memory controllers, functional blocks, and / or other programmable logic devices. When such hardware modules are activated, the hardware modules perform some or all of the operations. In some embodiments, the hardware modules include one or more general-purpose circuits that are configured by executing instructions (program code, firmware, etc.) to perform the operations.

[0099] In some embodiments, data structures (e.g., electronic device 400, IOMMU 424, and / or portions thereof) representing some or all of the structures and mechanisms described herein are stored on a non-transitory computer-readable storage medium that includes a database or other data structure that can be read by an electronic device and used, directly or indirectly, to fabricate hardware that includes the structures and mechanisms. By way of example, the data structure can be a behavioral-level description or a register transfer level (RTL) description of the hardware functionality in a high-level design language (HDL) (e.g., Verilog or VHDL). The description can be read by a synthesis tool that can synthesize the description to produce a netlist that includes a listing of gates / circuit elements from a synthesis library that represents the functionality of the hardware that includes the above-described structures and mechanisms. The netlist can then be placed and routed to produce a data set that describes the geometry to be applied to a mask. The mask can then be used in various semiconductor manufacturing steps to produce one or more semiconductor circuits (e.g., integrated circuits) corresponding to the above-described structures and mechanisms. Alternatively, the database on the computer-accessible storage medium can be a netlist (with or without a synthesis library) or a data set (as needed), or a graphics data system (GDS) II data.

[0100] In this specification, variables or unspecified values (i.e., general descriptions of values in the absence of a particular instance of a value) are represented by letters such as N. As used herein, although similar letters may be used at different locations in this specification, the variables and unspecified values are not necessarily the same in each case, and different quantities and values of variables are intended for some or all of the general variables and unspecified values. In other words, in this specification, N and any other letter used to represent variables and unspecified values are not necessarily related to each other.

[0101] As used herein, the expressions "et cetera" or "etc." are intended to present one and / or situations, namely, the equivalents of "at least one" of the elements associated with etc. in a list. For example, in the statement "The electronic device performs a first operation, a second operation, etc.", the electronic device performs at least one of the first operation, the second operation, and other operations. In addition, the elements in the list associated with etc. are merely examples in a set of examples - and at least some of the examples may not appear in some embodiments.

[0102] The previous description of the embodiments has been given for purposes of illustration and description only. The previous description is not intended to be exhaustive or to limit the embodiments to the disclosed form. Accordingly, many modifications and variations will be apparent to practitioners in the art. Additionally, the above disclosure is not intended to limit the embodiments. The scope of the embodiments is defined by the appended claims.

Claims

1. An electronic device, which comprises: a processor that executes a guest operating system; an input / output memory management unit (IOMMU); and a main memory that stores an IOMMU backing store, the IOMMU backing store including a separate copy of an input / output (MMIO) register set of an IOMMU memory map for each guest operating system among a plurality of supported guest operating systems; wherein the IOMMU is configured to: receive, from the guest operating system, a communication for accessing data in a given IOMMU MMIO register, the communication being directed by the guest operating system to an IOMMU MMIO address associated with the given IOMMU MMIO register; and perform a corresponding access to the data in the copy of the given IOMMU MMIO register in the IOMMU backing store associated with the guest operating system, the performing including redirecting, by the IOMMU, the corresponding access from the IOMMU MMIO address to an address of the copy of the given IOMMU MMIO register of the guest operating system in the IOMMU backing store.

2. The electronic device according to claim 1, wherein receiving, from the guest operating system, the communication for accessing the data in the given IOMMU MMIO register comprises: receiving, from a memory management unit (MMU) of the processor, the communication, the MMU having received the communication from the guest operating system and forwarded the communication to the IOMMU at the IOMMU MMIO address associated with the given IOMMU MMIO register.

3. The electronic device according to claim 2, wherein accessing the data involves storing data in the given IOMMU MMIO register, and wherein when redirecting the corresponding access, the IOMMU is further configured to: use a domain identifier associated with the guest operating system and one or more translation tables to determine a physical address where the copy of the given IOMMU MMIO register in the IOMMU backing store is located based on the IOMMU MMIO address; and store the data in the copy of the given IOMMU MMIO register at the physical address in the backing store associated with the guest operating system.

4. The electronic device according to claim 3, wherein the IOMMU is further configured to: detect a communication that causes the data in the copy of the given IOMMU MMIO register to be stored in the backing store; and in response to detecting the communication, perform one or more subsequent processing operations using the data from the copy of the given IOMMU MMIO register in the backing store.

5. The electronic device according to claim 1, wherein accessing the data involves reading data from the given IOMMU MMIO register, and wherein when redirecting the corresponding access, the IOMMU is further configured to: Use a domain identifier associated with the guest operating system and one or more translation tables to determine, based on the IOMMU MMIO address, the physical address in the IOMMU backing store where the copy of the given IOMMU MMIO register is located; Read the data from the copy of the given IOMMU MMIO register in the backing store associated with the guest operating system from the physical address; and Return the data to the guest operating system.

6. The electronic device according to claim 1, wherein each copy of the set of IOMMU MMIO registers in the backing store includes memory locations for storing the following for the corresponding guest operating system: Pointers to buffers and logs of the guest operating system; and Control fields associated with the buffers and logs of the guest operating system.

7. The electronic device according to claim 1, wherein the processor further executes a hypervisor, the hypervisor: During an initialization operation for the IOMMU backing store, allocate contiguous or scattered memory pages in the main memory for storing the IOMMU backing store.

8. The electronic device according to claim 7, wherein the hypervisor further simulates at least some IOMMU MMIO registers for the guest operating system, the at least some IOMMU MMIO registers not being included in the corresponding set of IOMMU MMIO registers of the guest operating system in the backing store; and Access information in the simulated IOMMU MMIO registers via the IOMMU.

9. The electronic device according to claim 8, wherein the IOMMU MMIO registers simulated by the hypervisor include a translation table base address and an IOMMU control register.

10. The electronic device according to claim 1, further comprising a cache memory in the IOMMU, wherein the IOMMU: Caches copies of information in IOMMU MMIO registers from some of the separate copies of the set of IOMMU MMIO registers.

11. A method for providing a copy of an input-output memory management unit (IOMMU) register to a guest operating system in an electronic device, the electronic device including a processor executing the guest operating system, the IOMMU, and a main memory storing an IOMMU backing store, wherein the IOMMU backing store includes a separate copy of a set of input-output (MMIO) registers of the IOMMU memory map for each guest operating system in a set of supported guest operating systems, and wherein the method comprises: Receiving, by the IOMMU, communication from the guest operating system to access data in a given IOMMU MMIO register, the communication being directed by the guest operating system to an IOMMU MMIO address associated with the given IOMMU MMIO register; and Performing, by the IOMMU, a corresponding access to the data in a copy of the given IOMMU MMIO register in the IOMMU backing store associated with the guest operating system, the performing including redirecting, by the IOMMU, the corresponding access from the IOMMU MMIO address to an address of the copy of the given IOMMU MMIO register of the guest operating system in the IOMMU backing store.

12. The method according to claim 11, wherein receiving the communication from the guest operating system to access the data in the given IOMMU MMIO register comprises: Receiving the communication from a memory management unit (MMU) of the processor, the MMU having received the communication from the guest operating system and forwarded the communication to the IOMMU at the IOMMU MMIO address associated with the given IOMMU MMIO register.

13. The method according to claim 11, wherein accessing the data involves storing data in the given IOMMU MMIO register, and the method further comprises, when the IOMMU redirects the corresponding access: Using, by the IOMMU, a domain identifier associated with the guest operating system and one or more translation tables to determine a physical address in the IOMMU backing store where the copy of the given IOMMU MMIO register is located based on the IOMMU MMIO address; and Storing, by the IOMMU, the data in the copy of the given IOMMU MMIO register at the physical address in the backing store associated with the guest operating system.

14. The method according to claim 13, further comprises: Detecting, by the IOMMU, a communication that causes the data in the copy of the given IOMMU MMIO register to be stored in the backing store; and In response to detecting the communication, performing one or more subsequent processing operations using the data from the copy of the given IOMMU MMIO register in the backing store.

15. The method according to claim 11, wherein accessing the data involves reading data from the given IOMMU MMIO register, and the method further comprises, when the IOMMU redirects the corresponding access: Using, by the IOMMU, a domain identifier associated with the guest operating system and one or more translation tables to determine a physical address in the IOMMU backing store where the copy of the given IOMMU MMIO register is located based on the IOMMU MMIO address; The IOMMU reads the data in the copy of the given IOMMU MMIO register in the backing store area associated with the guest operating system from the physical address; and the IOMMU returns the data to the guest operating system.

16. The method according to claim 11, wherein each copy of the set of IOMMU MMIO registers in the backing store area includes memory locations for storing the following for the corresponding guest operating system: pointers to buffers and logs of the guest operating system; and control fields associated with the buffers and logs of the guest operating system.

17. The method according to claim 11, wherein the processor executes a hypervisor, and the method further includes: the hypervisor allocates contiguous or scattered memory pages in the main memory for storing the IOMMU backing store area during an initialization operation for the IOMMU backing store area.

18. The method according to claim 17, which further includes: the hypervisor simulates at least some IOMMU MMIO registers for the guest operating system, the at least some IOMMU MMIO registers not being included in the corresponding set of IOMMU MMIO registers of the guest operating system in the backing store area; and the IOMMU accesses information in the simulated IOMMU MMIO registers via the IOMMU.

19. The method according to claim 18, wherein the IOMMU MMIO registers simulated by the hypervisor include a translation table base address and an IOMMU control register.

20. The method according to claim 11, which further includes a cache memory in the IOMMU, wherein the method further includes: the IOMMU caches copies of information in IOMMU MMIO registers from some of the separate copies of the set of IOMMU MMIO registers.

Images