I / O memory management unit access to guest operating system buffers and logs

Abstract

The electronic device includes a processor that executes a guest operating system, a memory having a guest portion reserved for storing data and information accessed by the guest operating system, and an input / output memory management unit (IOMMU). The IOMMU writes information in the guest portion to guest buffers and / or logs used to communicate information from the IOMMU to the guest operating system. The IOMMU also reads information from the guest portion in the guest buffers and / or logs used to communicate information from the guest operating system to the IOMMU.

Description

[Background technology]

[0001] (Related Technology) Some electronic devices (e.g., servers, desktop computers, etc.) support "virtualization" of electronic device hardware, such as input / output (I / O) devices. Virtualization involves an intermediate entity on or within the electronic device providing the illusion to an instance of software (e.g., an application program, etc.) running on the electronic device that the instance of software has direct access to the electronic device hardware, when in fact the intermediate entity intercepts / redirects or otherwise assists the instance of software in accessing it. For example, one common intermediate entity is a "virtual machine." A virtual machine is a software entity that abstracts the electronic device hardware and emulates or presents a known interface to the electronic device hardware, thereby allowing an instance of software to run on electronic devices of various types and configurations. The configuration of electronic device hardware may include electronic device hardware with which the instance of software is not compatible. In some electronic devices, a virtual machine supports the execution of one or more instances of operating systems, called "guest" operating systems, each with its own view of the (virtualized) resources of a particular machine, such as memory and central processing unit (CPU). The guest operating system in turn provides an environment for running other instances of software, such as productivity applications, databases, etc.

[0002] In some electronic devices, virtual machines are managed and controlled by a software entity known as a hypervisor. The hypervisor can launch or initialize virtual machines, control, monitor and assist virtual machine access to electronic device hardware, terminate or close virtual machines, provide specific virtualized resources in expected locations or with specific hardware programming models, arbitrate access to physical resources required by multiple virtual machines, direct input and output of external interfaces (e.g., from a user or network to the appropriate virtual machine), etc. FIG. 1 is a block diagram illustrating virtual machines and a hypervisor. As seen in FIG. 1, there are three virtual machines (VMs) 100, under each of which runs one or more programs (PRGRMS) 104, such as a guest operating system (guest OS) 102, a database, software applications, etc. The virtual machines 100 communicate with a hypervisor 106, which interfaces between a host operating system (host OS) 108 and the virtual machines 100. The host operating system 108 provides an interface between the electronic device hardware 110 and the hypervisor 106. Additionally, the hypervisor 106 interfaces between the virtual machines 100 and an input / output management unit (IOMMU) 112 which acts as a memory management unit and controller for IO device hardware 114 .

[0003] Operations performed by the hypervisor include handling communications between the electronic device hardware and guest operating systems (more broadly, virtual machines). For example, the hypervisor may translate, redirect, or otherwise assist communications between the guest operating systems and an input / output management unit (IOMMU), such as by providing memory addresses for direct memory access to memory used by the virtual machines. Communications handled by the hypervisor include communications such as peripheral page request (PPR) log and event log writing by the IOMMU, and command buffer writing by the guest operating systems. PPR log, event log, and command buffer writing are described in detail in the AMD I / O Virtualization Technology (IOMMU) Specification, rev. 3.00, December 2016, which is incorporated by reference in its entirety.

[0004] FIG. 2 is a block diagram illustrating typical communications between guest operating systems and IOMMUs handled by a hypervisor. In FIG. 2, several elements are shown dotted / dotted, which are logs, buffers, etc. stored in memory (e.g., main memory of an electronic device) and thus accessed via typical memory access techniques. The elements of FIG. 2 include guest operating system 102, hypervisor 106, and IOMMU 112, as well as guest peripheral page request (PPR) log 200, guest command buffer (CMD BUF) 202, and guest event log 204, which are structures in memory (e.g., lists, tables, etc.) used to store communications from and to guest operating system 102. Additionally, the elements include guest pointer (PTRS) / status register (REGS) 206, which is a set of pointers to guest operating system structures and locations in memory for storing status information related to the guest operating system. Additionally, elements include IOMMU peripheral page request log 208, IOMMU command buffer 210, and IOMMU event log 212, which are structures in memory (e.g., lists, tables, etc.) used to store communications from and to the IOMMU 112. Elements also include IOMMU memory mapped input / output (MMIO) pointers / status registers (REGS) 214 in the IOMMU 112, which are a set of registers in the IOMMU 112 for storing pointers to various IOMMU 112 structures and status information related to the IOMMU 112.

[0005] During operation, using commands as an example, guest operating system 102 writes commands destined for IOMMU 112 to guest command buffer 202 (i.e., to the next available location of the buffer in memory where commands from guest operating system 102 are stored). As shown by the dotted line in FIG. 2, hypervisor 106 detects the guest operating system's write to guest command buffer 202, retrieves and processes the command (e.g., replaces the guest domain ID and / or guest device ID in the command with the corresponding host domain ID and / or device ID, etc.). The processed command is stored in IOMMU command buffer 210, or hypervisor 106 updates the tail pointer in IOMMU MMIO pointer / status register 214 so that IOMMU command buffer 210 indicates the newly written command (e.g., increments the tail pointer to the next location in IOMMU command buffer 210). IOMMU 112 then retrieves the command from IOMMU command buffer 210 using the command buffer head pointer, etc., and executes the command. This causes the IOMMU 112 to perform a corresponding action. The hypervisor 106 performs a similar operation on the IOMMU 112, writing to the IOMMU peripheral page request log 208 and the IOMMU event log 212 (e.g., replacing the host device ID with the guest device ID). Memory reads and writes, pointer updates, and other operations performed by the hypervisor 106 have higher latency. Thus, using the hypervisor 106 to intervene between the guest operating system 102 and the IOMMU 112 can lead to communication delays and can cause the processor to be busy, increasing traffic on the memory bus in the electronic device. [Brief description of the drawings]

[0006] [Figure 1] FIG. 2 is a block diagram illustrating a virtual machine and a hypervisor. [Diagram 2]FIG. 2 is a block diagram illustrating communication between guest operating systems and an IOMMU that is handled by a hypervisor. [Diagram 3] FIG. 2 is a block diagram illustrating a virtual machine and a hypervisor, according to some embodiments. [Figure 4] FIG. 1 is a block diagram illustrating an electronic device according to some embodiments. [Diagram 5] 1 presents a block diagram illustrating a portion of memory accessed by an IOMMU, according to some embodiments. [Figure 6] 1 presents a block diagram illustrating values ​​stored in guest copies of IOMMU MMIO registers according to some embodiments. [Figure 7] 1 is a block diagram illustrating communication between a guest operating system and an IOMMU that is handled by the IOMMU, according to some embodiments. [Figure 8] 4 is a flowchart illustrating a process during which an IOMMU accesses memory locations in a guest portion of memory for a guest operating system, according to some embodiments. [Figure 9] 1 is a flowchart illustrating a process during which an IOMMU reads from a guest buffer or log in a guest portion of memory, according to some embodiments. [Figure 10] 1 is a flowchart illustrating a process during which an IOMMU writes to a guest buffer or log in a guest portion of memory, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Throughout the drawings and the description, like numbers refer to like drawing elements.

[0008] The following description is presented to enable any person skilled in the art to make and use the described embodiments and is provided in the context of a particular application and its requirements. Various modifications to the described embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications. Thus, the described embodiments are not limited to the embodiments shown but are to be accorded the widest scope consistent with the principles and features disclosed herein.

[0009] (term) In the following description, various terms are used to describe the embodiments. The following are simplified general descriptions of these terms. Please note that the terms may have important additional features that are not described herein for clarity and conciseness, and therefore this description is not intended to limit the terms.

[0010] Functional Block: A functional block refers to a group, collection, and / or set of one or more interrelated circuit elements, such as integrated circuit elements, discrete circuit elements, etc. The circuit elements are "interrelated" in that they share at least one characteristic. For example, circuit elements that are interrelated may be contained in, fabricated on, or otherwise coupled to a particular integrated circuit chip or portion thereof, may participate in the performance of a given function (e.g., a computational or processing function, a memory function, etc.), may be controlled by a common control element and / or a common clock, etc. A functional block may include any number of circuit elements, ranging from a single circuit element (e.g., a single integrated circuit logic gate) to millions or billions of circuit elements (e.g., integrated circuit memories).

[0011] (Virtualization, Virtual Machines and Hypervisors) The described embodiments support "virtualization" of electronic device hardware, such as memory, input / output (IO) devices, etc. Virtualization generally involves an intermediate entity on or within an electronic device providing an illusion to an instance of software running on the electronic device that the instance of software has direct access to the electronic device hardware, when in fact the intermediate entity intercepts / redirects or otherwise assists in the accesses made by the instance of software. For example, an instance of software may be presented by the intermediate entity with a set of electronic device registers, memory locations, electronic device settings, and other functional blocks that appear to the instance of software as actual device registers, memory locations, etc. of the electronic device, but are merely copies presented by the intermediate entity. In this case, the intermediate entity receives, intercepts, or otherwise obtains access of a copy of the electronic device hardware and performs corresponding interactions with the actual electronic device hardware on behalf of the instance of software. Virtualization of electronic device hardware has many advantages, such as allowing different electronic devices to use different configurations of electronic device hardware, different addresses, locations, or identifiers of electronic device hardware, etc., and the instances of software to be presented through the intermediate entity with the same interface to the electronic device hardware. Additionally, the intermediate entity can decide whether to allow or block access of the electronic device hardware by a given instance of the software, thus allowing protection of the electronic device hardware (or portions thereof) and / or instances of software running on the electronic device through virtualization of the electronic device hardware. By controlling access as described above, the intermediate entity can share the electronic device hardware among several instances of the software and / or provide individual instances of the software with exclusive access to portions of the electronic device hardware.

[0012] In the described embodiment, the intermediate entity includes a "virtual machine." A virtual machine is a software entity that abstracts the electronic device hardware and presents a known interface to actual or emulated electronic device hardware to an instance of software. The hardware abstraction allows an instance of software to run on various types and configurations of electronic device hardware, including potentially incompatible electronic device hardware. In the described embodiment, the virtual machine supports the execution of one or more instances of operating systems, referred to as "guest" operating systems. The guest operating systems provide an environment for running other software programs, such as applications, databases, and the like.

[0013] In the described embodiment, the virtual machines are managed and controlled by a software entity known as a hypervisor. The hypervisor may start or initialize the virtual machines, control, monitor and assist the virtual machines in accessing the electronic device hardware, terminate or close the virtual machines, etc. FIG. 3 is a block diagram illustrating virtual machines and a hypervisor according to some embodiments. As seen in FIG. 3, there are three virtual machines (VMs) 300, under each of which runs a guest operating system (guest OS) 302 and one or more programs (PRGRMS) 304, such as databases, software applications, etc. The virtual machines 300 communicate with a hypervisor 306, which interfaces between a host operating system (host OS) 308 and the virtual machines 300. The host operating system 308 provides an interface between the electronic device hardware 310 and the hypervisor 306. Unlike that shown in FIG. 1 for existing electronic devices, in FIG. 3 the IOMMU 312 directly interfaces between the guest operating system 302 and the IO device hardware 314 (as indicated by the thick line between the IOMMU 312 and the guest operating system 302) without the hypervisor 306 being involved. Thus, unlike existing electronic devices, in the described embodiments the hypervisor 306 is not involved in performing at least some of the operations for handling communications between the guest operating system 302 and the IOMMU 312 as described herein. However, it should be noted that certain communications do occur between the IOMMU 312 and the hypervisor 306, as indicated by the line between the hypervisor 306 and the IOMMU 312. Additionally, it should be noted that in some embodiments, the host operating system 308 is not present and the hypervisor 306 communicates more directly with the electronic device hardware 310.

[0014] (overview) In a described embodiment, an electronic device includes a processor, a memory (e.g., main memory), a number of input / output (IO) devices (e.g., network interface devices, disk controllers, etc.), and an input / output memory management unit (IOMMU) that interfaces between the processor and the IO devices. The processor executes a hypervisor, one or more virtual machines, and guest operating systems on the virtual machines. Each guest operating system is assigned a guest portion of memory (a contiguous or non-contiguous region or block of memory) reserved for storing data and information accessed by that guest operating system. In a described embodiment, the IOMMU performs operations to handle communication between the guest operating systems and the IOMMU.

[0015] As part of the communication process between the guest operating systems and the IOMMU, instead of using a single IOMMU copy of certain buffers and logs as in existing systems, the IOMMU directly accesses each guest operating system's buffers and logs in the corresponding guest portion of memory. That is, the IOMMU accesses separate copies of the buffers and logs used by each guest operating system instead of the copies of the buffers and logs maintained by the IOMMU (and hypervisor) in existing systems. For example, in some embodiments, each guest operating system maintains buffers and logs, including command buffers, event logs, and / or peripheral page request (PPR) logs, in the corresponding guest portion of memory. The IOMMU can directly read and / or write information to the memory locations (e.g., entries) of each guest operating system's buffers and logs in that guest operating system's portion of memory.

[0016] In existing systems where an IOMMU uses a single copy of buffers and logs, the IOMMU maintains a set of IOMMU memory-mapped input / output (MMIO) registers that are used to store information about the buffers and logs. For example, the IOMMU MMIO registers include information such as a set of pointers (e.g., head and / or tail pointers for each log or buffer) and control values ​​for the buffers and logs (e.g., size indicators and / or configuration values ​​for each log or buffer). In described embodiments, the IOMMU accesses buffers and logs separately in different guest portions of memory (i.e., at different memory addresses and possibly with different control values), so a single set of IOMMU MMIO registers is insufficient. Thus, in described embodiments, the IOMMU provides a separate copy of the set of IOMMU MMIO registers for each guest operating system that is used to access the buffers and logs in the corresponding guest portions of memory. In other words, in some embodiments, if N guest operating systems are supported by the system, the IOMMU provides N copies of the IOMMU MMIO registers (i.e., sets of pointers and control values ​​for the buffers and logs), one for each guest operating system.

[0017] Due to the number of registers required to keep track of all possible guest operating system buffers and logs (which can be very large), a separate copy of the IOMMU MMIO registers does not physically exist within the IOMMU (i.e., the IOMMU does not contain hardware circuitry for a copy of the IOMMU MMIO registers). Instead, a separate copy of the set of IOMMU MMIO registers for the guest operating systems is stored in an IOMMU backing store, which is a portion of main memory reserved for storing IOMMU data and information. In the described embodiment, the IOMMU "virtualizes" the IOMMU MMIO registers, or presents an appearance to the guest operating systems that the guest operating systems (and other entities, such as the hypervisor, for example) can access the IOMMU MMIO registers through a specified IOMMU MMIO address. However, there are no registers at the IOMMU MMIO address. Instead, the IOMMU intercepts accesses by the guest operating systems to the IOMMU MMIO registers at the IOMMU MMIO address and redirects the IOMMU MMIO register accesses to the corresponding locations in the IOMMU backing store. For example, in some embodiments, the IOMMU MMIO addresses include addresses from a specific range of addresses provided by the IOMMU in the IOMMU interface / aperture, and these addresses are used to map communications to copies of IOMMU registers in the IOMMU backing store.

[0018] During operation, the IOMMU performs several operations to access guest buffers or logs (or, more generally, corresponding memory locations) in the guest portion of memory. For example, the IOMMU can read information from the guest buffer or log that was written to the guest buffer or log by the guest operating system (or another entity, including the IOMMU itself). In some embodiments, an operation to read the guest buffer or log is initiated by the IOMMU when the guest operating system writes information to a guest buffer or log in the respective guest portion of memory. For example, the guest operating system can write a command to a command buffer in the guest portion of memory, which, when processed by the IOMMU, causes the IOMMU to perform a corresponding action. The guest operating system then sends a memory write to update the command buffer tail pointer in the IOMMU MMIO register (and / or other command buffer IOMMU MMIO register), which is received, intercepted, or otherwise detected by the IOMMU. The IOMMU then performs a corresponding write in the IOMMU backing store to the guest operating system's copy of the IOMMU MMIO register where the command buffer tail pointer is stored. Continuing the example, the IOMMU can write updated data, such as a new address or offset of the command buffer tail pointer, to the guest operating system's copy of the command buffer tail pointer register in the IOMMU backing store. The IOMMU further monitors such writes from the guest operating system (e.g., monitors writes to the IOMMU MMIO registers). The IOMMU then reads the command from the guest command buffer in the guest portion of memory, directly or indirectly using information from the command buffer head pointer. The IOMMU then processes the command, thereby causing the IOMMU to perform the corresponding action.After processing the command, the IOMMU may update the command buffer's head pointer in the guest operating system's copy of the IOMMU MMIO registers in the IOMMU backing store and update other command buffer information for the guest operating system in the IOMMU backing store. For example, the IOMMU may update the command buffer's head pointer in its copy of the IOMMU MMIO registers (such as by advancing the command buffer's head pointer to point to the next entry in the command buffer) to indicate that the command has been processed and is therefore no longer present in the command buffer in the guest portion of memory.

[0019] In addition to reading information from a guest buffer or log, the IOMMU may write information to the guest buffer or log to be read by the guest operating system (or another entity, including the IOMMU itself). For example, the IOMMU may write information about an event to an event log in the guest portion of the memory of the guest operating system. In some embodiments, when writing information to a guest buffer or log for a given guest operating system, the IOMMU first determines the location of the guest buffer or log (or an entry therein) in the guest portion of the memory of the given guest operating system. For this operation, the IOMMU determines the system physical address of the guest operating system's copy of the IOMMU MMIO registers, which includes the location information of the guest buffer or log in the IOMMU backing store. Continuing with the example, the IOMMU determines the system physical address of the guest operating system's copy of the IOMMU MMIO registers in the IOMMU backing store where the tail pointer of the event log is stored. The IOMMU then obtains location information for the guest operating system's guest buffer or log (e.g., system physical address of the entry, entry counter, etc.) from its copy of the IOMMU MMIO register at the system physical address. The IOMMU then uses the location information to write the information to the appropriate guest buffer or log in memory. For example, the IOMMU can use the location information to write information about the event to an event log in the guest operating system's guest portion of memory. The IOMMU then updates the copy of the relevant IOMMU MMIO register in the IOMMU backing store to indicate that a write has occurred (e.g., advances the command buffer tail pointer to the next entry in the command buffer in the guest portion of memory). The IOMMU also sends an internal interrupt to the guest operating system indicating that the IOMMU has written the information to the guest buffer or log.The guest operating system then performs operations in the guest portion of memory, based on the interrupt, to read information from the guest buffers or logs (these operations are described in more detail below).

[0020] The IOMMU allows the guest operating system to directly access buffers and logs in the guest portion of memory, i.e., to perform memory reads and writes, avoiding the dependency of guest buffer and log accesses on hypervisor intervention / processing as in existing systems. Moving these operations from the hypervisor (implemented in software) to the IOMMU (implemented in hardware) speeds up the operations, reduces memory system bandwidth requirements, and reduces the load on the processor compute functional blocks, thereby improving the overall performance of the electronic device. Better performance of electronic devices translates into higher user satisfaction.

[0021] (Electronic Devices) 4 is a block diagram illustrating an electronic device 400 according to some embodiments. As seen in FIG. 4, the electronic device 400 includes a processor 402, a memory 404, a mass storage device 406, input / output (IO) devices 408-412, an input / output (IO) hub 414, and a memory controller 416.

[0022] The processor 402 is a functional block that executes computational operations in the electronic device 400. The processor 402 includes two cores 418-420, each of which includes one or more computational mechanisms, such as a central processing unit (CPU) core, a graphics processing unit (GPU) core, an embedded processor, an application specific integrated circuit (ASIC), and / or other computational mechanisms. The processor 402 also includes a memory management unit (MMU) 422, which is a functional block that executes operations related to address translation (e.g., page table walk, translation lookaside buffer lookup, etc.), memory access protection for memory accesses by the cores 418-420, etc.

[0023] Memory 404 is a functional block that performs memory (e.g., "main" memory) operations within electronic device 400. Memory 404 includes one or more dynamic random access memory (DRAM), double data rate synchronous DRAM (DDR SDRAM), and / or other types of memory circuits that store data and instructions used by other functional blocks within electronic device 400, and control circuitry that handles accessing (e.g., reading, writing, checking, deleting, invalidating, etc.) the data and instructions stored in the memory circuits.

[0024] The mass storage device 406 is a functional block and / or device that performs the operations of a large capacity non-volatile storage element for storing data and instructions used by other functional blocks of the electronic device 400. The mass storage device 406 may be or include a large capacity semiconductor memory (e.g., flash memory, etc.), a disk drive (e.g., hard drive, etc.), an optical drive, etc. Copies of the data and instructions stored in the mass storage device 406 are retrieved and stored in the memory 404 for use by other functional blocks in the electronic device 400. For example, in some embodiments, data and / or instructions are retrieved from the mass storage device 406 in blocks or "pages" of a predetermined size (e.g., 4kB, 2MB, etc.), and the pages are stored in the memory 404 for access by the other functional blocks. Additionally, pages may be newly created in available locations in the memory 404 (e.g., to store computation results).

[0025] The IO devices 408-412 are functional blocks and / or devices that perform corresponding IO operations. The particular nature of the IO operations performed by each of the IO devices 408-412 depends on the nature of the IO device. For example, the IO devices 408-412 can include human interface devices, network interface devices, audio / visual processing or presentation devices, GPUs, sensor devices, disk controllers, peripheral component interface (PCI) devices, universal serial bus (USB) devices, etc., with each IO device performing related operations such as receiving input from a human (e.g., keyboard, mouse, etc.), receiving or transmitting data over a network, etc. The IO devices 408-412 provide data and / or instructions to or consume data and / or instructions from other functional blocks of the electronic device 400. For example, in some embodiments, the IO devices 408-412 access (i.e., read, write, invalidate, etc.) data in memory pages in the guest memory 428 (i.e., portions of memory reserved for a given guest operating system).

[0026] The IO hub 414 is a functional block that performs the operations of an input / output hub that interfaces between the IO devices 408-412 and other functional blocks (e.g., the processor 402, the memory 404, etc.) of the electronic device 400. The operations performed by the IO hub 414 include operations to ensure that communications destined for the IO devices 408-412 reach the intended IO devices, that communications from the IO devices 408-412 reach the other functional blocks properly, that the other functional blocks are secured from unauthorized access by the IO devices 408-412, and vice versa. In some embodiments, the IO hub 414 interfaces and converts or translates related communications between buses that use different communication standards (e.g., between a Peripheral Component Interface Express (PCIe) bus and a HyperTransport link (registered trademark)).

[0027] IO hub 414 includes IOMMU 424, which is a functional block that performs operations to enable IO devices 408-412 to access data and / or instructions in memory 404, communicates with processor 402 (and guest operating systems executed thereby), etc. In these embodiments, when data and instructions are accessed in memory 404 by an IO device (e.g., IO device 408), the IO device sends a memory access request (e.g., a direct memory access request or DMA) to IOMMU 424. IOMMU 424 then sends a corresponding request to memory 404 to fulfill the memory access request. For example, in some embodiments, when data is to be retrieved based on a memory access request, IOMMU 424 retrieves the data from memory 404 (or mass storage device 406 if the data is not present in memory 404) and transfers the data to the requesting IO device. In some embodiments, IOMMU 424 includes page tables, translation lookaside buffers, and / or other functional blocks used to translate the “virtual” or local memory addresses used by IO devices 408-412 into physical addresses in memory 404 where the data is actually located.

[0028] In the described embodiments, the IOMMU 424 communicates with the guest operating systems executed by the cores 418-420 in the virtual machines, and vice versa. For example, in some embodiments, the IOMMU 424 (or the IO devices 408-412 via the IOMMU 424) communicate events and peripheral page requests (PPRs) to the guest operating systems. In these embodiments, the IOMMU 424 reports events, such as IO page faults (for page table walks on behalf of the IO devices 408-412), IOMMU 424 hardware errors, etc., to the guest operating systems via a shared guest event log in memory 404. Additionally, in these embodiments, the IOMMU 424 forwards PPRs from peripherals (IO devices) using the well-known Address Translation Services or ATS standard to the guest operating systems for memory page provisioning operations (i.e., to perform operations on pages in memory 404 accessible by the guest operating systems) via a shared guest PPR log in memory 404. As another example, in some embodiments, the guest operating systems communicate commands to the IOMMU 424. In these embodiments, the guest operating system controls IOMMU 424 and / or IO devices 408-412 by issuing commands to IOMMU 424 via a shared guest command buffer in memory 404, such as wait for completion (which acts as a command barrier to allow the previous command to complete before IOMMU 424 continues), invalidate device table entries, and invalidate IOMMU 424 translation lookaside buffer entries.

[0029] In some embodiments, IOMMU 424 provides an interface to guest operating systems, which interface includes memory mapped locations, registers, etc. used to communicate with IOMMU 424. For example, in some embodiments, IOMMU 424 provides a set of memory mapped input / output (MMIO) memory locations to which guest operating systems can write values, which are received by IOMMU 424. In some embodiments, the interface is virtualized; that is, the memory locations, registers, etc. are not used to store the values ​​as expected by the guest operating systems, but instead are simply presented by IOMMU 424. In these embodiments, IOMMU 424 may receive values ​​from guest operating systems over the interface, but uses IOMMU backing store 426 and / or other locations in memory 404 to store separate copies of the values ​​for each guest operating system (e.g., addressed to an IOMMU MMIO address, etc.). Memory accessed by IOMMU 424 to communicate with guest operating systems and other entities (e.g., processor 402, etc.) is described in more detail below.

[0030] 4, IOMMU 424 includes a local cache memory that is used to store copies of data or information for IOMMU 424. For example, in some embodiments, the cache memory is used to store copies of recently or frequently used data from IOMMU backing store 426, such as the set of values ​​and / or individual values ​​therefrom of a given guest operating system's IOMMU MMIO registers. Because the cache memory is smaller than IOMMU backing store 426, it may only have the capacity (i.e., memory locations) to store a portion (and perhaps only a small portion) of the data and information stored in IOMMU backing store 426.

[0031] Guest memory 428 is a portion of memory 404 (e.g., one or more contiguous or non-contiguous pages or blocks of memory) used by a corresponding guest operating system to store data and information used by the guest operating system. In general, guest memory 428 may be used by a guest operating system and / or other entities to store any type of data and information used by the guest operating system and / or other entities. In some embodiments, guest memory 428 is protected and only certain entities are permitted to access guest memory 428. For example, the corresponding guest operating system, a hypervisor, a security processor, and / or an operating system within electronic device 400 may "protect" guest memory 428 by restricting access of guest memory 428 to the corresponding guest operating system and other designated devices and / or functional blocks. In some embodiments, guest memory 428 is encrypted or otherwise rendered inaccessible by undesirable entities. In some embodiments, guest memory 428 is used to store a guest event log, a guest peripheral page request (PPR) log, and a guest command buffer, which are data structures (e.g., tables, lists, etc.) used to communicate between the guest operating system and the IOMMU. The guest event log, guest peripheral page request (PPR) log, and guest command buffer are described in more detail below.

[0032] In some embodiments, communication paths are coupled between various functional blocks of electronic device 400 (e.g., processor 402, memory controller 416, memory 404, etc.), as indicated by arrowed lines between the elements. The communication paths include one or more buses, wires, guides, and / or other connections, possibly along with controllers, fabric elements (e.g., switches, routers, etc.), circuit elements, etc. The communication paths are used to route commands, data, control signals, and / or other information between the functional blocks. For example, in some embodiments, a coherent bus fabric or interconnect is coupled between IO hub 414, processor 402 (e.g., MMU 422), and memory 404. Note that some communication paths within electronic device 400 are not shown in FIG. 4 for clarity.

[0033] In some embodiments, electronic device hardware 310 of Figure 3 includes functional blocks and devices such as processor 402 and memory 404, and IO device hardware 314 includes functional blocks and devices such as IO devices 408-412. In these embodiments, IOMMU 312 of Figure 3 and IOMMU 424 of Figure 4 perform at least some of the same operations.

[0034] Electronic device 400 is shown with a particular number and configuration of elements (e.g., functional blocks and devices, such as processor 402, memory 404, etc.) and communication paths. However, electronic device 400 is simplified for illustrative purposes, and in some embodiments, electronic device 400 has a different number or configuration of elements and / or communication paths. For example, electronic device 400 may include a power subsystem, a display, etc. In general, electronic device 400 includes sufficient elements and communication paths to perform the operations described herein.

[0035] Electronic device 400 can be or can be included in any electronic device that performs computational operations. For example, electronic device 400 can be or can be included in electronic devices such as a desktop computer, a laptop computer, a wearable electronic device, a tablet computer, a smartphone, a server, an artificial intelligence device, a virtual or augmented reality device, a network device, a toy, an audiovisual device, a home appliance, a controller, a vehicle, and / or combinations thereof.

[0036] (Part of memory accessed by IOMMU) In some embodiments, the IOMMU accesses data and information in different portions of memory (e.g., memory 404) to perform the operations described herein. In some of these embodiments, the portions of memory include an IOMMU backing store (e.g., IOMMU backing store 426), guest memory (e.g., guest memory 428), and / or hypervisor memory. FIG. 5 is a block diagram illustrating portions of memory accessed by an IOMMU, according to some embodiments. While FIG. 5 is shown as an example, in some embodiments, the memory and / or different portions of the memory store different types and / or configurations of information. In general, the memory contains sufficient information to enable the operations described herein.

[0037] 5, the IOMMU backing store 500 includes an ID translation table 502. In general, the ID translation table 502 includes information used by the IOMMU to translate or convert guest domain IDs and / or device IDs to host domain IDs and / or device IDs in guest operating system to IOMMU communications, or vice versa. Domain IDs and device IDs are described in more detail in the AMD I / O Virtualization Technology (IOMMU) Specification, rev. 3.00, December 2016, which is incorporated herein by reference as noted above.

[0038] In some embodiments, ID translation table 502 includes separate tables for domain IDs, denoted as domain ID mapping table 504, and for device IDs, denoted as device ID mapping table 506, although separate tables are not required (so that all translations can be included in a single table). Domain ID mapping table 504 includes a set of entries, each of which is used to store an identification or index of a guest domain ID associated or related to a specified host domain ID. Device ID mapping table 506 includes a set of entries, each of which is used to store an identification or index of a guest device ID associated or related to a specified host device ID. During operation, when a guest or host domain ID and / or device ID is translated or translated in a communication, the IOMMU performs a lookup in ID translation table 502 (i.e., domain ID mapping table 504 and / or device ID mapping table 506) to obtain the corresponding translation or translation.

[0039] IOMMU backing store 500 also includes guest control 508. Generally, guest control 508 includes copies of values ​​stored in or from interface and control registers for guest operating systems in the electronic device. For each supported guest operating system, guest control 508 includes a copy of the guest interface registers and / or guest operating system control registers (or at least the values ​​therein) that control the interaction between the IOMMU and that guest operating system. For example, guest control 508 may include, for each guest operating system, a map control register that is used to communicate domain ID and / or device ID mapping for the guest operating system to the IOMMU.

[0040] IOMMU backing store 500 further includes guest memory mapped input / output (MMIO) 510. Generally, guest MMIO 510 includes pointers and control information used to access buffers and logs (e.g., guest command buffers, guest event logs, and guest PPR logs) for guest operating systems in the guest portion of memory 404 (e.g., guest memory 428). More specifically, guest MMIO 510 includes, for each supported guest operating system, a copy of values ​​used to control access of buffers and logs in the guest portion of memory 404. For example, in some embodiments, the IOMMU may include two N Supports guest operating systems (capable of interacting with, communicating with, etc.), where N=10, 16, or another value, and Guest MMIO 510 is up to 2 N, one for each supported guest operating system. In the described embodiment, the values ​​used to control access are similar to the values ​​stored in the IOMMU MMIO registers of the existing device, but a separate set of values ​​is kept for each supported guest operating system and points to that guest operating system's guest portion of memory 404 (rather than a single copy in the IOMMU on the existing device).

[0041] FIG. 6 is a block diagram illustrating values ​​stored in guest MMIO 510, according to some embodiments. In the example of FIG. 6, values ​​for a single complete set of IOMMU MMIO registers for a given guest operating system are shown along with partial values ​​for two adjacent sets of IOMMU MMIO registers above and below the single complete set of values. FIG. 6 is shown in this manner to illustrate that, as discussed above, in some embodiments, IOMMU backing store 426 includes multiple separate sets of values, each set of values ​​associated with a given guest operating system. FIG. 6 is shown by way of example, and in some embodiments, guest MMIO 510 stores different values ​​or different configurations of values. In general, guest MMIO 510 and / or another entity stores sufficient information to access buffers and logs in the guest portion of memory, as described herein.

[0042] The values ​​in each set of guest MMIO 510 values ​​can be grouped into values ​​generally associated with a guest operating system's guest command (CMD) buffer, guest event log, and guest PPR log. For a guest command buffer, the values ​​include a command tail pointer 600, which is a pointer or other reference that indicates the tail or most recently written entry of the guest command buffer in the corresponding guest portion of memory 404. In other words, the command tail pointer 600 holds some or all of the bits of a physical address in the guest portion of memory 404 that indicates a pointer or reference, such as a memory address (e.g., where the tail or most recently written entry of the command buffer is located). The guest command buffer value also includes a command head pointer 602, which is a pointer or other reference that indicates the head or base of the command buffer, i.e., where the command buffer starts in the corresponding guest portion of memory 404 and / or where the buffered commands start. The guest command buffer value further includes a command length 604, which is a value that indicates the current or maximum size of the command buffer, such as in number of entries, number of bytes, etc. The guest command buffer value further includes command controls 606, which is a set or sequence of bits that includes several bits (or combinations thereof) that indicate the configuration of the command buffer in the guest operating system and / or the corresponding guest portion of memory 404. For example, in some embodiments, command controls 606 includes bits that indicate whether a particular command buffer-related interrupt is enabled, whether command buffering or processing is enabled (or disabled / paused), what types of commands are or can be present in the command buffer, etc.For example, in some embodiments, command controls 606 include command buffer control values ​​similar to those described in the AMD I / O Virtualization Technology (IOMMU) Specification, rev. 3.00, such as CmdWaitInt, CmdBufRun, CmdWaitInteEn, CmdBufEn, and / or other values, as described above.

[0043] In some embodiments, the command buffer is implemented as a ring buffer by using a command head pointer 602 to point to the next entry / location to be read in the command buffer and a command tail pointer 600 to indicate the last / most recently added entry / location in the command buffer (which may be the same entry / location if only one command is stored in the command buffer). In these embodiments, as a command is read from an entry and processed, the command head pointer 602 is advanced by one entry until the command head pointer 602 and command tail pointer 600 point to the same entry. In some of these embodiments, the command length 604 stores the maximum length of the command buffer. Implementing ring buffers using pointers is commonly known in the art and therefore will not be described in detail herein.

[0044] For the event log, the values ​​of guest MMIO 510, namely event tail pointer 608, event head pointer 610, event length 612, and event control 614, function similarly to those described above for the command buffer, except that these values ​​are used for accessing the event log in the corresponding guest portion of memory 404. The same applies for the PPR log, where the values ​​of guest MMIO 510, PPR tail pointer 616, PPR head pointer 618, PPR length 620, and PPR control 622 function similarly to those described above for the command buffer, except that these values ​​are used for accessing the PPR log in the corresponding guest portion of memory 404.

[0045] In some embodiments, fewer guest operating systems (and possibly much fewer) than the supported number may be running on electronic device 400 at any one time. In some of these embodiments, IOMMU backing store 500 is dynamically adjusted to include sufficient space to store guest MMIO 510, etc. In other words, as guest operating systems are initialized / active, the hypervisor and / or another entity may allocate, add, or activate space in IOMMU backing store 500 (i.e., guest MMIO 510) to store the guest operating system's IOMMU MMIO register values. In some of these embodiments, IOMMU backing store 500 does not have additional free / unused space allocated for virtual machines that are not yet present (i.e., not initialized) in electronic device 400.

[0046] In some embodiments, the hypervisor performs at least some initialization operations on the IOMMU backing store 500. For example, in some embodiments, the hypervisor allocates memory, e.g., contiguous or distributed pages of memory, in memory 404 to store the IOMMU backing store. As described above, memory 404 is general-purpose memory in electronic device 400 used by various functional blocks (e.g., cores 418-420, etc.) to store data and information, and is not merely local memory in the IOMMU. Thus, this operation includes the hypervisor allocating space in the "main" memory of electronic device 400 to store the IOMMU backing store. As another example, in some embodiments, the hypervisor writes initial values, via the IOMMU, to copies of IOMMU MMIO registers in the backing store of each active guest operating system. Thus, the hypervisor writes the initial values ​​of pointers and control values ​​of each active guest operating system to their respective copies of IOMMU MMIO registers in the IOMMU backing store, such as at the start of the guest operating system.

[0047] In some embodiments, only a subset of the available IOMMU MMIO registers are included in the copy of the IOMMU MMIO registers in the IOMMU backing store. In these embodiments, other registers may be provided by the IOMMU itself. For example, in some embodiments, registers in the IOMMU that store IOMMU control values ​​(e.g., enable / disable the IOMMU, etc.), base addresses of page tables, etc. are provided by the IOMMU. In general, IOMMU MMIO registers that are not used to access some buffers and logs of memory in the guest operating system are not shown as copies, but instead a single copy of each register is provided by the IOMMU. In some of these embodiments, the hypervisor emulates these IOMMU registers for the guest operating system, but accesses are received and processed by the IOMMU for registers in the IOMMU.

[0048] Returning to FIG. 5, guest memory 512 includes a guest event log 514, a guest peripheral page request (PPR) log 516, and a guest command buffer 518 for the guest operating system. Generally, guest event log 514, guest PPR log 516, and guest command buffer 518 are memory structures (e.g., lists, tables, buffers, etc.) used to store corresponding events, PPR requests, and commands accessed by the IOMMU and / or guest operating system. For example, guest event log 514 may be a table or ring buffer that includes a number (e.g., 24, 50, etc.) of entries, each entry for holding information (e.g., a pattern of bits) representative of a particular event that occurred within the IOMMU to be processed by the corresponding guest operating system. During operation, the IOMMU communicates events and PPRs to the guest operating system via corresponding logs in guest event log 514 and guest PPR log 516 in guest memory 512. Additionally, the guest operating system communicates commands to the IOMMU via corresponding command buffers in guest command buffers 518 in guest memory 512.

[0049] In some embodiments, each guest operating system active on electronic device 400 is associated with a corresponding separate guest portion of memory (i.e., some number of pages in memory 404) that includes a guest event log, a peripheral page request log, and a guest command buffer used by that guest operating system and accessible by the IOMMU. This is illustrated in FIG. 5 as additional guest memories 520-522 behind guest memory 512.

[0050] The hypervisor memory 524 includes a device table 526. In general, the device table 526 is a table in which device-related information is stored about devices (which may be real / physical devices or virtual devices) associated with and / or coupled to an electronic device. The device table 526 includes a set of entries, each of which can be used to store information about the corresponding device, such as pointers to page tables and interrupt tables, control and configuration values, function indicators, mode indicators, domain IDs, security information and settings, etc. Additionally, in the described embodiment, unlike existing device tables, each entry in the device table 526 includes a device ID and a guest identifier for a guest operating system that communicates with, participates in, or is otherwise associated with the device. During operation, in addition to using the device table to determine information about the device, the IOMMU uses the device ID and / or the guest identifier to translate or convert the guest device ID to a host device ID.

[0051] In some embodiments, some / all of IOMMU backing store 500, guest memory 512, and hypervisor memory 524 are not contiguous, but instead are stored in different regions or locations of memory. For example, the base address of guest event log 514 (and thus the guest event log itself) may be located in memory separate from guest PPR log 516. Thus, guest event log 514 may not be adjacent to guest PPR log 516 as shown in FIG. 5.

[0052] In some embodiments, the IOMMU includes a private address map that contains pointers, references, and / or other indications to the locations in memory of various data and information in memory accessed by the IOMMU. For example, in some embodiments, the IOMMU private address map contains pointers or references to individual copies of the IOMMU MMIO registers of the guest operating systems and / or the starting / base address of a set of copies of the IOMMU MMIO registers in the IOMMU backing store 500. In these embodiments, before accessing data and information in memory, the IOMMU performs a lookup in the private address map for the location of the data and information.

[0053] In some embodiments, IOMMU backing store 500 and / or portions thereof (such as control bits) may be accessed by other entities of electronic device 400 via the IOMMU (e.g., by sending requests to the IOMMU) or may not be accessible by other entities. For example, at least some of the data and information in IOMMU backing store 500 may be accessed by other entities via writes to and reads from corresponding IOMMU MMIO registers.

[0054] (IOMMU and guest operating system communication) In the described embodiment, the IOMMU (e.g., IOMMU 424) handles communication between the IOMMU (or IO devices served by it) and the guest operating system. FIG. 7 is a block diagram illustrating communication between guest operating system 700 and IOMMU 702, as handled by IOMMU 702, according to some embodiments. While some elements are shown in a particular configuration in FIG. 7, other embodiments use a different number or configuration of elements. In general, in the described embodiments, IOMMU 702 includes or has access to sufficient elements to enable the operations described herein. In FIG. 7, some elements are shown as dotted / dotted. These elements are logs, buffers, etc. that are stored in memory (e.g., in IOMMU backing store 500, in guest memory 512, etc.) and accessed by IOMMU 702, guest operating system 700, and / or other entities using typical memory access techniques. In some embodiments, guest operating system 700, IOMMU 702, and hypervisor 704 are configured similarly to guest operating system 302, IOMMU 312, and hypervisor 306 of FIG. 3, although this is not a requirement.

[0055] As seen in FIG. 7, unlike that shown in FIG. 2 for the existing system, in the described embodiment, the IOMMU 702 and guest operating system 700 communicate with each other more directly. In other words, the IOMMU 702 and guest operating system 700 communicate with each other via the guest event log 514, guest PPR log 516, and guest command buffer (BUFF) 518 in memory (i.e., in the guest portion of the memory of the guest operating system 700 (e.g., guest memory 428)). Additionally, the guest operating system 700 and the IOMMU 702 use the guest control 508 and guest MMIO 510 to direct how the communication is to be performed. For example, in some embodiments, the IOMMU 702 uses pointers in the guest MMIO 510 to determine the location in memory of the guest event log 514, guest command buffer 518, and guest PPR log 516 for the guest operating system 700. The hypervisor 704 does not intervene or is otherwise not involved in some or all of the operations to complete these communications. For example, hypervisor 704 does not perform operations such as translating domain IDs and device IDs for these communications, accessing pointers to guest MMIO 510, and / or accessing buffers and logs in the guest portion of memory. Instead, IOMMU 702 performs these operations. Because IOMMU 702 translates domain IDs and device IDs, accesses guest buffers, logs, etc., the described embodiments avoid using hypervisor 704 to handle at least some of the communications between guest operating systems 700 and IOMMU 702. This may mean that communications are completed more quickly, reducing the load on processor 402, memory 404, etc.

[0056] In operation, using the command as an example, guest operating system 700 writes an invalidate_IOMMU_pages command to guest command buffer 518. When the command is processed by IOMMU 702, it causes IOMMU 702 to invalidate the range of entries in the IOMMU translation cache specified by the domain ID in the command. In other words, the guest operating system performs a memory write in the corresponding guest portion of memory and updates the next open / available entry in guest command buffer 518 to include the invalidate_IOMMU_pages command's information (i.e., a bit representing the command). Guest operating system 700 then sends the write command to the IOMMU and updates (e.g., advances) the command buffer tail pointer (e.g., command tail pointer 600) in the corresponding IOMMU MMIO register to indicate that guest operating system 700 has written the command to the command buffer. IOMMU 702 detects a command buffer tail pointer write by guest operating system 700, e.g., via snooping for writes to addresses in the corresponding guest command buffer, detecting a change in the value of the buffer tail pointer, receiving a write command from guest operating system 700, etc. Upon detecting a command buffer tail pointer write, IOMMU 702 uses the value of the command buffer head pointer (e.g., command head pointer 602) to retrieve the command from the command buffer in the guest portion of memory and prepares the command for processing (e.g., by substituting the host domain ID associated with the guest domain ID in the command). IOMMU 702 then processes the command, which causes IOMMU 702 to invalidate the range of entries in IOMMU 702's translation cache that are indicated by the host domain ID.IOMMU 702 performs at least some similar operations for IOMMU 702 writes to guest PPR log 516 and guest event log 514, but in the opposite direction: IOMMU 702 typically writes these logs and guest operating system 700 reads them.

[0057] Although the hypervisor 704 is not involved in certain portions of the communications between the guest operating system 700 and the IOMMU 702, e.g., translating guest domain IDs to host domain IDs, the hypervisor 704 and the guest operating system 700 and / or the IOMMU 702 may separately exchange communications associated with communications between the guest operating system 700 and the IOMMU 702. Or, the hypervisor 704 may be involved in other ways in ensuring that communications are properly handled by the guest operating system 700 and / or the IOMMU 702. For example, the hypervisor 704 may directly (e.g., via communications) or indirectly (e.g., via monitoring memory accesses) determine that the IOMMU 702 or the guest operating system 700 has performed a specified operation (e.g., writing to a buffer or log) and may perform an operation such as signaling an interrupt to the guest operating system 700 and / or the IOMMU 702, updating a shared memory location that serves as a flag, etc. As noted above, the hypervisor 704 may also initialize an IOMMU backing store.

[0058] (The process of IOMMU access to guest buffers and logs in the guest's part of memory) In the described embodiment, the IOMMU (e.g., IOMMU 702) performs operations to access (e.g., read and write) guest buffers and logs in a guest portion of memory that are used to communicate information between the IOMMU and a guest operating system (e.g., guest operating system 700). For example, in some embodiments, the IOMMU accesses entries in a guest command buffer, a guest event log, and / or a guest peripheral page request (PPR) log in the guest portion of memory for each of the guest operating systems. The IOMMU directly accesses the guest buffers and logs of each guest operating system in that the IOMMU accesses the guest buffers and logs in each guest operating system's portion of memory using normal memory access operations, without the hypervisor processing the memory access operations. FIG. 8 illustrates a flow chart illustrating a process during which the IOMMU accesses a buffer or log in a guest portion of memory for a guest operating system, according to some embodiments. It should be noted that the operations illustrated in FIG. 8 are provided as general examples of operations performed by some embodiments. Operations performed by other embodiments include different operations, operations performed in a different order, and / or operations performed by different entities or functional blocks.

[0059] The scope of operations shown in Figure 8 is general. More specific examples are shown in Figures 9-10 and include the IOMMU reading from a buffer or log in the guest portion of memory for a guest operating system (Figure 9) and the IOMMU writing to a buffer or log in the guest portion of memory for a guest operating system (Figure 10).

[0060] As shown in FIG. 8, the IOMMU performs an access to information in a guest buffer or log in a guest portion of memory for a guest operating system, and the buffer or log is used to communicate information between the IOMMU and the guest operating system (step 800). In this operation, the IOMMU first determines to perform the access. For example, the IOMMU may determine that information is to be read from a particular guest buffer or log after receiving an indication that information has been written to the guest buffer or log by the guest operating system or another entity. The IOMMU then determines the address of the guest buffer or log in the guest portion of memory (i.e., the particular memory location of the entry to be accessed). The IOMMU then performs the access to the buffer or log in the guest portion of memory. To access the buffer or log, the IOMMU performs a typical memory access operation with the address of the buffer or log. The IOMMU directly accesses the buffer or log in the guest portion of memory instead of using its own set of buffers and logs (e.g., IOMMU buffers and logs, such as IOMMU PPR log 208), thereby enabling communication with the guest operating system without hypervisor involvement.

[0061] FIG. 9 illustrates a flow chart showing the process during which a guest operating system writes to and an IOMMU reads from a guest buffer or log in a guest portion of memory, according to some embodiments. Note that the operations shown in FIG. 9 are shown as general examples of operations performed by some embodiments. Operations performed by other embodiments include different operations, operations performed in a different order, and / or operations performed by different entities or functional blocks. For example, FIG. 9 uses commands in a guest command buffer as an example, but in other embodiments, the IOMMU reads from a different buffer or log, such as a guest PPR log or a guest event log. In these embodiments, the IOMMU may perform some or all of the same operations to read a different guest buffer or log.

[0062] 9 begins when a guest operating system (e.g., guest operating system 700) writes a command to a command buffer (e.g., guest command buffer 518) in a guest portion of memory (e.g., guest memory 428) (step 900). For example, in some embodiments, the guest command buffer is a circular buffer in the guest portion of memory that contains several entries, and this operation includes the guest operating system writing the command (i.e., a sequence of bits that represent the command) to the next available entry in the guest command buffer.

[0063] As the guest operating system adds a command to the guest command buffer that requires processing by the IOMMU (e.g., IOMMU 702), the guest operating system notifies the IOMMU of the command. More specifically, the guest operating system sends a write request to update the command buffer tail pointer, the write request being directed to the guest virtual address of the IOMMU MMIO register where the command buffer tail pointer is stored (step 902). In the described embodiment, the guest operating system uses a local or "virtual" address to access the IOMMU MMIO register (without having to know the actual IOMMU address). The guest virtual address is translated by a memory management unit (MMU) in the processor running the guest operating system (e.g., by performing a page table walk in the page table of the guest operating system, etc.) to a system physical address in the IOMMU where the MMIO register is located (i.e., the IOMMU MMIO address). As described above, because the IOMMU virtualizes the IOMMU MMIO registers, the system physical address used by the MMU is not the address where the guest operating system's copy of the IOMMU MMIO register is located, but instead is the address in the IOMMU's interface / aperture that is seen by the IOMMU as an access of the MMIO register. Thus, following translation in the MMU, the MMU forwards the write request to the IOMMU at the IOMMU MMIO address (i.e., the system physical address) associated with the guest virtual address (step 904).

[0064] Because the IOMMU MMIO address is simply an address provided by the IOMMU to receive access requests from the guest operating system, the IOMMU also performs a translation to determine a physical address in memory where a copy of the command buffer tail pointer for the guest operating system is stored based on the write request (step 906). During this operation, the IOMMU determines the physical address in memory using the guest operating system's identification information, which indicates which copy of the IOMMU MMIO register is to be accessed in the IOMMU backing store. In this process, the IOMMU first calculates an IOMMU virtual address of the guest operating system's copy of the IOMMU MMIO register using an algorithm, table, or the like, based on the guest operating system's identifier and the system physical address provided by the MMU. The IOMMU then translates the IOMMU virtual address to a system physical address using one or more page tables. Here, the guest operating system's copy of the IOMMU MMIO register is stored in the IOMMU backing store.

[0065] The IOMMU then stores the data from the write request at a physical address in memory, thereby updating a copy of the command buffer tail pointer for the guest operating system (step 908). In other words, the IOMMU stores data, such as one or more bits of the updated address of the command buffer tail pointer, an entry identifier, an execution count, etc., in the memory location where the copy of the command buffer tail pointer associated with the guest operating system is saved in the IOMMU backing store. This update causes the command buffer tail pointer to directly or indirectly indicate that a command has been written to the guest command buffer and is therefore awaiting processing by the IOMMU. In some embodiments, the IOMMU, the guest operating system, and / or another entity sets other pointers or indicators to indicate that a command is awaiting processing in the guest command buffer or that the guest operating system has provided a command. For example, in some embodiments, the IOMMU includes a set of command waiting bits, one for each supported guest operating system, and the command waiting bit is set (or cleared) for the guest operating system to indicate that a command is awaiting processing in the guest command buffer.

[0066] Based on detecting a write request (or otherwise determining that a command is waiting to be processed), the IOMMU retrieves the command from the guest command buffer in the guest portion of memory using data from its copy of the command buffer head pointer (step 910). In this operation, the IOMMU directly or indirectly uses data from its copy of the command buffer head pointer, possibly along with other pointers or other information (e.g., a guest command buffer base pointer, etc.) from the copy of the IOMMU MMIO registers for the guest operating system from the backing store to determine the system physical address of the entry in the guest command buffer where the command is stored. For this operation, the IOMMU may determine the address and perform a translation, etc. (e.g., using a corresponding page table) to determine the system physical address of the entry in the guest command buffer. The IOMMU then reads the command from the system physical address. This may include allowing the operating system, hypervisor, memory management unit, etc. to allow the memory read to occur, assuming the IOMMU is allowed to access the particular address in the guest portion of memory. Upon completion of this operation, the IOMMU has the command obtained from the guest portion of memory (e.g., it has the individual bits of the command) and is therefore ready to execute the command.

[0067] The IOMMU then processes the communication (step 912). For this operation, in some embodiments, command processing logic within the IOMMU processes the command and causes the IOMMU to perform the commanded operation. Although not shown in FIG. 9, after processing the command, the IOMMU updates the system state to indicate that the command has been processed. For example, in some embodiments, the IOMMU may update the command buffer head pointer in the guest operating system's copy of the corresponding IOMMU MMIO register in the IOMMU backing store and update other command buffer information for the guest operating system in the IOMMU backing store.

[0068] FIG. 10 is a flow chart illustrating the process during which an IOMMU writes to, and a guest operating system reads from, a guest buffer or log in a guest portion of memory, according to some embodiments. Note that the operations illustrated in FIG. 10 are provided as general examples of operations performed by some embodiments. Operations performed by other embodiments include different operations, operations performed in a different order, and / or operations performed by different entities or functional blocks. For example, FIG. 10 uses events in a guest event log as an example, but in other embodiments, the IOMMU writes to a different buffer or log, such as a guest PPR log or a guest command buffer. In these embodiments, the IOMMU may perform some or all of the same operations to write to a different guest buffer or log.

[0069] The operations of FIG. 10 begin when an IOMMU (e.g., IOMMU 702) determines a system physical address of a guest event log (e.g., guest event log 514) in a guest portion of memory for a guest operating system (e.g., guest memory 428) (step 1000). In this operation, the IOMMU calculates, obtains, or otherwise determines a system physical address of a copy of the guest operating system's IOMMU MMIO registers in an IOMMU backing store that holds the guest event log's tail pointer based on an identifier for the guest operating system (e.g., a domain ID). The system then obtains the guest event log tail pointer from the system physical address and directly or indirectly uses the value of the guest event log tail pointer (and possibly values ​​from other copies of the guest operating system's IOMMU MMIO registers, such as the event log base pointer) as the system physical address of the guest event log. In some embodiments, the IOMMU performs one or more operations to calculate a system physical address based on the value of the event log tail pointer and / or to translate the value of the event log tail pointer using one or more page tables.

[0070] The IOMMU then writes the event to a guest event log in the guest portion of the guest operating system's memory using the system physical address (step 1002). For example, in some embodiments, the guest command log is a circular buffer in the guest portion of memory that contains several entries, and this operation includes the IOMMU writing the event (i.e., a sequence of bits representing the event) to the next available entry in the guest event log. The IOMMU writes the event to the guest event log using a typical memory access operation. In some embodiments, writing to a system physical address in the guest portion of memory may include the operating system, hypervisor, memory management unit, etc. allowing the memory write to occur, assuming the IOMMU is allowed to access the particular address in the guest portion of memory.

[0071] Because the IOMMU has added an event to the guest event log that requires processing by the guest operating system, the IOMMU notifies the guest operating system of the event. In some embodiments, notifying the guest operating system includes both updating the guest operating system's copy of the event log tail pointer in the IOMMU backing store and sending an interrupt to the guest operating system to notify the guest operating system. Thus, the IOMMU determines the physical address of the guest operating system's copy of the event log tail pointer in the IOMMU backing store (step 1004). During this operation, the IOMMU determines the physical address in memory using the IOMMU backing store and the guest operating system's identification information indicating which copy of the IOMMU MMIO register is accessed in one or more IOMMU private address tables and / or page tables. The IOMMU then stores the data to update the guest operating system's copy of the event log tail pointer at the physical address (step 1006). In other words, the IOMMU stores data, such as one or more bits of the updated address of the event log tail pointer, entry identifier, execution count, etc., in a memory location where a copy of the guest operating system's associated event log tail pointer is saved in the IOMMU backing store. This update causes the event log tail pointer to directly or indirectly indicate that an event has been written to the event log and is therefore awaiting processing by the guest operating system.

[0072] The IOMMU also sends an interrupt to the guest operating system, the interrupt indicating that an event is waiting in the guest event log, step 1008. In this operation, the IOMMU may use an internal interrupt mechanism, such as by using a virtual highly programmable interrupt controller (vAPIC) associated with the guest operating system, where the interrupt is passed between the IOMMU and the appropriate guest operating system.

[0073] In response to receiving the interrupt, the guest operating system sends a read request to the IOMMU to read data from the event log head pointer. The read request is directed to the guest virtual address of the IOMMU MMIO register where the event log head pointer is stored (step 1010). In the described embodiment, the guest operating system uses a local or "virtual" address (it may not be aware of the actual and / or virtual IOMMU address) to access the IOMMU MMIO register. The guest virtual address is translated by a memory management unit (MMU) in the processor executing the guest operating system (e.g., by performing a page table walk in the guest operating system's page table, etc.) to a system physical address within the IOMMU where the MMIO register is located (i.e., the IOMMU MMIO address). As described above, because the IOMMU virtualizes the IOMMU MMIO register, the system physical address used by the MMU is the address in the IOMMU's interface / aperture that is seen by the IOMMU as an access of the MMIO register, rather than the address where the guest operating system's copy of the IOMMU MMIO register is located. Thus, following the translation in the MMU, the MMU forwards the read request to the IOMMU at the IOMMU MMIO address (ie, the system physical address) associated with the guest virtual address (step 1012).

[0074] Because the IOMMU MMIO address is simply an address provided by the IOMMU to receive access requests from the guest operating system, the IOMMU also performs a translation to determine a physical address in memory where a copy of the event log head pointer for the guest operating system is stored based on the read request (step 1014). During this operation, the IOMMU determines the physical address in memory using the guest operating system's identification information indicating which copy of the IOMMU MMIO register is to be accessed in the IOMMU backing store. In this process, the IOMMU first calculates an IOMMU virtual address of the guest operating system's copy of the IOMMU MMIO register using an algorithm, table, etc. based on the guest operating system's identifier and the system physical address provided by the MMU. The IOMMU then translates the IOMMU virtual address to a system physical address using one or more page tables, where the guest operating system's copy of the IOMMU MMIO register is stored in the IOMMU backing store.

[0075] The IOMMU then reads the data of the read request from the memory location of the physical address in memory. In other words, the IOMMU reads data, such as one or more bits of the address of the event log head pointer, entry identifier, execution count, etc., at the memory location where a copy of the event log head pointer associated with the guest operating system is saved in the IOMMU backing store. The event log head pointer directly or indirectly indicates the next event to be written to the guest event log that is waiting for processing by the guest operating system. In some embodiments, the IOMMU, the guest operating system, and / or another entity sets other pointers or indicators to indicate that an event is waiting for processing in the guest event log or that the IOMMU has provided the event. For example, in some embodiments, the guest operating system and / or hypervisor includes a set of event waiting bits, one for each supported guest operating system, and the event waiting bit is set (or cleared) for the guest operating system to indicate that an event is waiting in the guest event log. The IOMMU then returns the data from the event log head pointer to the guest operating system (step 1016).

[0076] The guest operating system then uses the data to perform one or more subsequent actions to process the events in the guest event log (step 1018). For example, in some embodiments, the guest operating system's copy of the event log head pointer stores the address of the location in the guest portion of memory where the event log entries are located. The guest operating system uses the address, directly or indirectly, to retrieve the events from the guest event log before processing the events.

[0077] In some embodiments, the electronic device (e.g., electronic device 400 and / or portions thereof) uses code and / or data stored in a non-transitory computer-readable storage medium to perform some or all of the operations described herein. More specifically, the electronic device reads the code and / or data from the computer-readable storage medium and executes the code and / or uses the data in performing the described operations. The computer-readable storage medium may be any device, medium, or combination thereof that stores code and / or data used by the electronic device. For example, the computer-readable storage medium may include, but is not limited to, volatile and / or non-volatile memory, including flash memory, random access memory (e.g., eDRAM, RAM, SRAM, DRAM, DDR4 SDRAM, etc.), read-only memory (ROM), and / or magnetic or optical storage media (e.g., disk drives, magnetic tapes, CDs, DVDs, etc.).

[0078] In some embodiments, one or more hardware modules perform the operations described herein. For example, the hardware modules may include, but are not limited to, one or more processors / cores / central processing units (CPUs), application specific integrated circuits (ASICs) chips, neural network processors or accelerators, field programmable gate arrays (FPGAs), computational units, embedded processors, graphics processors (GPUs) / graphics cores, pipelines, accelerated processing units (APUs), sparsity monitors, functional blocks, and / or other programmable logic devices. When such hardware modules are activated, the hardware modules perform some or all of the operations. In some embodiments, the hardware modules include one or more general purpose circuits configured to perform the operations by executing instructions (program code, firmware, etc.).

[0079] In some embodiments, data structures representing some or all of the structures and mechanisms described herein (e.g., electronic device 400, IOMMU 424, and / or portions thereof) are stored in a non-transitory computer-readable storage medium, including a database or other data structure, that can be read by the electronic device and used directly or indirectly to manufacture hardware including the structures and mechanisms. For example, the data structure may be a behavioral or register transfer level (RTL) description of the hardware functionality in a high-level design language (HDL) such as Verilog or VHDL. The description may be read by a synthesis tool, which can synthesize the description to generate a netlist from a synthesis library that includes a list of gates / circuit elements that represent the functionality of the hardware including the structures and mechanisms described above. The netlist may then be placed and routed, and a data set may be generated that describes the geometry to be applied to a mask. The mask may then be used in various semiconductor manufacturing steps to manufacture one or more semiconductor circuits (e.g., integrated circuits) corresponding to the structures and mechanisms described above. Alternatively, the database on the computer accessible storage medium may be a netlist (with or without a synthesis library) or a data set or Graphical Data System (GDS) II data, as appropriate.

[0080] In this specification, variables or unspecified values ​​(i.e., general descriptions of values ​​without specific instances of the values) are represented by letters such as N. Similar letters may be used elsewhere in this description, but as used herein, the variables and unspecified values ​​in each instance are not necessarily the same, i.e., there may be different variable quantities and values ​​for some or all of the general variables and unspecified values. In other words, N and other letters used to represent variables and unspecified values ​​in this description are not necessarily related to each other.

[0081] As used herein, the phrase "et cetra" or "etc." is intended to present the equivalent of "and / or," i.e., "at least one" of the elements in the list to which "etc." pertains. For example, in the sentence "the electronic device performs a first operation, a second operation, etc.", the electronic device performs at least one of the first operation, the second operation, and other operations. Moreover, the elements in the list to which "etc." pertains are merely examples in a set of examples, and at least some of the examples may not appear in some embodiments.

[0082] The foregoing descriptions of the embodiments have been presented for purposes of illustration and description only. They are not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. Accordingly, many modifications and variations will be apparent to those skilled in the art. Moreover, the above disclosure is not intended to limit the embodiments. The scope of the embodiments is defined by the appended claims.

Claims

1. a processor running a guest operating system; a memory having a guest portion reserved for storing data and information accessed by the guest operating system; an input / output memory management unit (IOMMU), The IOMMU includes: writing information in the guest portion to a guest buffer and / or log used to communicate information from the IOMMU to the guest operating system; reading information from the guest portion in guest buffers and / or logs used to communicate information from the guest operating system to the IOMMU; 4. The method of claim 3, Electronic devices.

2. The IOMMU writes information to a predetermined guest buffer or log in the guest portion. the IOMMU determining a system physical address of the given guest buffer or log in the memory; the IOMMU writing information to the predetermined guest buffer or log in the memory at the system physical address; the IOMMU sending an indication to the guest operating system that the IOMMU has written the information to the designated guest buffer or log. The electronic device of claim 1 .

3. The IOMMU writes information to the specified guest buffer or log in the guest portion. the IOMMU determining, based on an identifier of the guest operating system, a system physical address of a copy of the guest operating system's IOMMU MMIO register in an IOMMU backing store; the IOMMU obtaining the system physical address from a copy of the IOMMU MMIO register of the guest operating system in the IOMMU backing store; the IOMMU, after writing the information to the guest buffer or log, updating a copy of the IOMMU MMIO register of the guest operating system in the IOMMU backing store to indicate the write to the guest buffer or log. The electronic device of claim 2.

4. Sending the indication to the guest operating system includes: the IOMMU sending an internal interrupt to the guest operating system through a virtual copy of an interrupt controller of the guest operating system; The internal interrupt notifies the guest operating system that the IOMMU has written to the given guest buffer or log. The electronic device of claim 2.

5. the guest operating system reads the information from the guest buffer or log of the guest portion at the system physical address based on the index; The electronic device of claim 2.

6. The guest operating system reading the information from the guest buffer or log of the guest portion, the guest operating system obtaining information from a copy of the guest operating system's IOMMU MMIO registers in an IOMMU backing store via the IOMMU; the guest operating system using the information from the copy of the IOMMU MMIO register to read the information from the guest buffer or log of the guest portion. The electronic device of claim 5.

7. The IOMMU reads information from a predetermined guest buffer or log of the guest portion. the IOMMU receiving a memory write request from the guest operating system to update an IOMMU MMIO register associated with the given guest buffer or log, the guest operating system independently writing to the given guest buffer or log; the IOMMU determining a system physical address of the given guest buffer or log in the guest portion based on the update; the IOMMU reading information from the given guest buffer or log in the memory at the system physical address; The electronic device of claim 1 .

8. The IOMMU includes: determining a system physical address of a copy of the IOMMU MMIO register for the guest operating system in an IOMMU backing store based on the memory write request and an identifier of the guest operating system; performing corresponding updates to the guest operating system's copies of the IOMMU MMIO registers in the IOMMU backing store; and 4. The method of claim 3, 8. The electronic device of claim 7.

9. The guest buffer and / or log may include: A command buffer; Event logs and A peripheral page request (PPR) log; Including some or all of the following: The electronic device of claim 1 .

10. the processor executing at least one other guest operating system; the memory having a separate guest portion reserved for storing data and information accessed by the at least one other guest operating system; The IOMMU includes: writing information in a guest buffer and / or log in a corresponding guest portion that is used to communicate information from the IOMMU to the at least one other guest operating system; reading information from the corresponding guest portion in a guest buffer and / or a log used to communicate information from the at least one other guest operating system to the IOMMU; 4. The method of claim 3, The electronic device of claim 1 .

11. 1. A method of accessing guest buffers and logs in an electronic device comprising: a processor executing a guest operating system; a memory having a guest portion reserved for storing data and information accessed by the guest operating system; and an input / output memory management unit (IOMMU), comprising: the IOMMU writing information in the guest portion to a guest buffer and / or log used to communicate information from the IOMMU to the guest operating system; and the IOMMU reading information from the guest portion in a guest buffer and / or a log used to communicate information from the guest operating system to the IOMMU. method.

12. The IOMMU writes information to a predetermined guest buffer or log in the guest portion. the IOMMU determining a system physical address of the given guest buffer or log in the memory; the IOMMU writing information to the predetermined guest buffer or log in the memory at the system physical address; the IOMMU sending an indication to the guest operating system that the IOMMU has written the information to the designated guest buffer or log.

12. The method of claim 11.

13. The IOMMU writes information to the specified guest buffer or log in the guest portion. the IOMMU determining, based on an identifier of the guest operating system, a system physical address of a copy of the guest operating system's IOMMU MMIO register in an IOMMU backing store; the IOMMU obtaining the system physical address from a copy of the IOMMU MMIO register of the guest operating system in the IOMMU backing store; the IOMMU, after writing the information to the guest buffer or log, updating a copy of the IOMMU MMIO register of the guest operating system in the IOMMU backing store to indicate the write to the guest buffer or log.

13. The method of claim 12.

14. Sending the indication to the guest operating system includes: the IOMMU sending an internal interrupt to the guest operating system through a virtual copy of an interrupt controller of the guest operating system; The internal interrupt notifies the guest operating system that the IOMMU has written to the given guest buffer or log.

13. The method of claim 12.

15. the guest operating system reading the information from the guest buffer or log of the guest portion at the system physical address based on the indication.

13. The method of claim 12.

16. The guest operating system reading the information from the guest buffer or log of the guest portion, the guest operating system obtaining information from a copy of the guest operating system's IOMMU MMIO registers in an IOMMU backing store via the IOMMU; the guest operating system using the information from the copy of the IOMMU MMIO register to read the information from the guest buffer or log of the guest portion.

16. The method of claim 15.

17. The IOMMU reads information from a predetermined guest buffer or log of the guest portion. the IOMMU receiving a memory write request from the guest operating system to update an IOMMU MMIO register associated with the given guest buffer or log, the guest operating system independently writing to the given guest buffer or log; the IOMMU determining a system physical address of the given guest buffer or log in the guest portion based on the update; the IOMMU reading information from the given guest buffer or log in the memory at the system physical address; 12. The method of claim 11.

18. The IOMMU includes: determining a system physical address of a copy of the IOMMU MMIO register for the guest operating system in an IOMMU backing store based on the memory write request and an identifier of the guest operating system; performing corresponding updates to the guest operating system's copies of the IOMMU MMIO registers in the IOMMU backing store; and 4. The method of claim 3, 20. The method of claim 17.

19. The guest buffer and / or log may include: A command buffer; Event logs and A peripheral page request (PPR) log; Including some or all of the following:

12. The method of claim 11.

20. the processor executing at least one other guest operating system; the memory having a separate guest portion reserved for storing data and information accessed by the at least one other guest operating system; The IOMMU includes: writing information in a guest buffer and / or log in a corresponding guest portion that is used to communicate information from the IOMMU to the at least one other guest operating system; reading information from the corresponding guest portion in a guest buffer and / or a log used to communicate information from the at least one other guest operating system to the IOMMU; 4. The method of claim 3, 12. The method of claim 11.

Images