ARCHITECTURAL INTERFACE FOR ADDRESS TRANSLATION CACHE (ATC) IN XPU FOR DIRECT COMMAND TRANSMISSION TO THE GUEST SOFTWARE

Abstract

In one embodiment, a device comprises: at least one accelerator for performing operations on data; and an address translation cache (ATC) coupled to the at least one accelerator, the ATC being configured to store address translations. The ATC is configured to: send an instruction to a pending request queue (PRQ) stored in a memory coupled to the device, the PRQ being associated with a process of guest software; and send an interrupt to inform the process about the instruction. Other embodiments are described and claimed.

Description

BACKGROUNDIncreasingly, devices are being deployed that support the address translation service (ATS) of a peripheral component interconnect express (PCIe) architecture. More specifically, the ATS provides a series of transactions for PCIe components to exchange and use translated addresses to support native input / output (I / O) virtualization in accordance with the PCIe base specification Version 5.0 (2019). Nevertheless, inefficiencies still exist.BRIEF DESCRIPTION OF THE DRAWINGSFIG. 1 is a diagram illustrating shared virtual memory (SVM) and PCIe ATS, according to various embodiments. FIG. 2 is a block diagram of a portion of a system according to an embodiment. FIG. 3 is another block diagram of a portion of a system according to an embodiment. FIG. 4 is a flow diagram of a method according to an embodiment. FIG. 5 is a flow diagram of a method according to an embodiment. FIG. 6 is an embodiment of a structure composed of point-to-point links that connect a set of components. FIG. 7 is a block diagram of a system according to another embodiment. FIG. 8 is a block diagram of a system according to another embodiment.DETAILED DESCRIPTIONFIG. 1 is a diagram illustrating shared virtual memory (SVM) and PCIe ATS, according to various embodiments. Legacy computers are heterogeneous systems that combine general purpose central processing units (CPUs) with specialized processing units (referred to herein as "XPUs"). Such systems begin to support SVM, applications running on CPUs transfer work to XPUs using virtual addresses, and expect XPUs to be able to manipulate memory using virtual addresses just like the CPU. Using the page tables established by virtual machines (VMs) and virtual machine monitors (VMMs), memory management units (MMUs) translate virtual addresses (VAs) to physical addresses (PAs) for CPUs, and IOMMUs translate virtual addresses to physical addresses for XPUs, as shown in FIG. 1. SVM allows CPUs and XPUs to manipulate complex data structures (e.g., trees) in memory without having to copy data. SVM also allows the XPUs to handle page errors, so that the XPUs need not be requested to keep their memory in, and thus have a much larger working set size, for example.In the embodiment of FIG. 1, a computing system 100 may be any type of computing platform ranging from small portable devices such as smart phones, tablet computers, etc. to larger devices such as client systems, e.g., desktop systems, server systems, etc. As shown, the system 100 includes a plurality of CPUs 110 0-110n. The CPUs 110 communicate with a memory 120, which may be implemented as an SVM, which in turn is shared in 0-1301 with a set of devices 130. Although shown generically as XPUs in FIG. 1, it should be appreciated that many different types of devices, such as different PCIe or other devices, may be present in a given system. As further illustrated, a root complex (root complex) 140 provides an interface between the memory 120 and the XPUs 130.As shown in detail in FIG. 1, CPUs 110 may include a processor complex (generally 112), which may include one or more cores or other processing engines. As can be seen, software executing on these processing units may issue VAs that may be provided to a translation lookaside buffer (TLB) 114. In general, the TLBs 114 may buffer VAto PA addresses and possibly additional information. This caching capability may be on behalf of a memory management unit (MMU) 116, which may include a larger set of address translations using a multi-level page table structure. In addition, MMU 116 may also include its page fault handling circuitry to obtain address translations, e.g., from memory 120, if not present.Similarly, root complex 140 includes another MMU, namely an IOMMU 142, which can store address translations on behalf of XPUs 130. Thus, as shown, requests for translation in root complex 140 may be received from given XPUs 130, and IOMMU 142 in turn provides a physical address. Such translations may be stored in a TLB within the XPU 130, referred to herein as a device TLB or, more particularly, an address translation cache (ATC) 132. With this physical address, an XPU may then send a memory request (e.g., read or write) to memory 120 with a particular physical address. It should be appreciated that in various implementations, the root complex 140 may be a separate component or may be present in a SoC having one or more CPUs.In various embodiments, a link 135 coupling the XPUs 130 to the root complex 140 may enable communication according to one or more communication protocols such as PCIe, Compute Express Link (CXL) (e.g., a CXL.io protocol) or an integrated scalable on-chip fabric (IOSF) (IOSF), as examples.In an arrangement as in FIG. 1, certain communications between guest software and an ATC may avoid "middle agents" such as IOMMU hardware and host software. However, without one embodiment, instructions from an ATC to guest software may still involve these middle agents. Embodiments thus improve performance and scalability by removing these middle agents from command communication between an ATC and the guest software.To this end, a translation agent (e.g., an IOMMU) may be configured to describe guest-specific command queues (the page request queue or, more generally, to be a pending request queue (both referred to as a "PRQ")) to receive commands from ATCs via a so-called PRQ descriptor. The ATC, in turn, is configured with an interface to obtain the location / property of such PRQs and then provide guest software directed commands to these PRQs and inform the guest software of its presence. In certain embodiments, an ATC is configured to allow it to send commands (e.g., page misses) to a corresponding guest queue and then inform the relevant guest software of the presence of such commands.Referring now to FIG. 2, shown is a block diagram of a portion of a system according to an embodiment. In FIG. 2, system 200 may be any type of computing platform that includes a SoC 210 or other CPU (and that may include multiple cores). As shown, guest software 212 (e.g., a VM) interacts with host software 214 (e.g., VMM). The system 200 further includes a plurality of XPUs 245 each having an ATC 240. Both SoC 210 and XPUs 245 may collectively use a memory 220 that functions as an SVM. The virtual to physical (and guest to host) translation may be performed at least in part using a translation agent, shown as IOMMU 230.As further shown in FIG. 2, an architectural interface is provided to allow an ATC to send commands directly to the guest software. To this end, embodiments provide a process address space identifier (PASID) table 222 within the memory 220. The PASID table 222 includes a guest PASID associated with a host PASID and an address in memory at which a PRQ descriptor resides. In this manner, the ATC 240 has easy access to the guest PASIDs that can be used to directly provide instructions (e.g., page requests) to the guest software 212. To this end, the ATC 240 may write such instructions to PRQ 224, e.g., using a PRQ descriptor address for a PRQ descriptor 226.In turn, the ATC 240 informs the guest software 212, e.g., via an interrupt, to indicate the presence of one or more instructions within the PRQ 224. It should be noted that this interrupt is not sent each time an instruction is written to the PRQ 224; instead, an interrupt is sent only when the guest software 212 has requested the interrupt, e.g., by setting a bit in the PRQ descriptor 226, the details of which will be described below.With this arrangement, embodiments provide an efficient interface that allows an ATC to communicate directly with the guest software without the host software involved. This improves processing efficiency because guest software can avoid leaving the virtual machine and the concomitant host processing of such requests coming from an ATC.To this end (reducing the middle agents), embodiments provide a translation agent (TA) architecture to write guest-specific command queues to directly receive commands from ATCs. Additionally, an ATC is configured, according to an embodiment, to request location / property of such guest-specific command queues from the TA. In addition, the ATC is configured to send commands (e.g., page misses) to an appropriate guest queue and then inform the relevant guest software in a manner that reduces interaction with middle agents.In FIG. 2, the host software 214 provides to the TA 230 a mapping structure that includes characteristics of a process for which the TA 230 performs the address translation. In embodiments, the host software 214 includes a PRQ descriptor address field in the mapping structure entry of each process. Guest software 212 programs the PRQ descriptor with location / head pointer / tail pointer and some other fields to describe the queue. In turn, ATC 240 may request the PRQ descriptor address from TA 230 via a ProcessInfoRe and cache it, e.g., in a ProcessInfoCache (PIC) 244. If the ATC 240 then wants to send a command to a process, it uses the PRQ descriptor address (e.g., stored in the PIC) to read the properties of the PRQ, send a write command (write) to the location pointed to by the PRQ tail, and send an interrupt to the process to inform it that there is a command that it needs to process. Since the command queues are stored in the guest memory, they scale with the number of guests and the number of processes without affecting the TA hardware.The ATS specification supports an ATC-IOMMU interface to enable an XPU to request address translations from the IOMMU and cache the results in the ATC. Thereafter, the ATC uses the results stored in the ATC to send a translated request to the random access memory. A system software ATC interface allows the system software to issue invalidates to the ATC to remove legacy translations from the ATC. In embodiments, an ATC may send commands to the guest software using simple memory write commands (writes). There is no impact on an XPU driver or XPU driver / OS interface because the command queues are in guest memory. An ATC system software interface allows an XPU ATC to report page faults to the system software and can be extended to allow the XPU ATC to send other commands to the system software.Embodiments implement a higher performance interface because the ATC can send commands directly to the guest software without TA or host software functioning as the middle agent (except initialization).As shown in FIG. 2, the host software 214 programs the address of the guest command queue 224 (referred to as a PRQ descriptor or, more generally, as a "descriptor") in an IOMMU mapping structure (e.g., PASID table 222). The IOMMU 230, in turn, provides the PRQ descriptor address to the ATC 240 to allow it to write instructions to the PRQ. The ATC 240 then informs the guest software 212 to read PRQ 224, e.g., by sending an interrupt. In one or more embodiments, the interrupt is not sent each time an instruction is written to the PRQ 224, but only when an instruction is written therein and the guest software 212 requested an interrupt using an indicator in the PRQ descriptor 226.In embodiments, each guest creates a queue (e.g., a circular queue) in memory with head / tail pointers to receive commands from the ATC. In examples described herein, this queue is a PRQ. For each PRQ, a guest creates a PRQ descriptor that includes: the base of the queue, the current value of the head pointer, the current value of the tail pointer, a pending page request (PPR) interrupt status, and a size of the queue.Since a guest may send work from a given process (identified by PASID) to more than one XPU, there may be more than one ATC that may need to send commands to the guest. Thus, a guest may create a PRQ for each XPU / process pair that may be running. In one embodiment, the XPU is identified by PCIe Bus / Device / Function (BDF) and the process is identified by PASID. In a particular embodiment, the TA maintains process specific information in a PASID table as the mapping structure. More specifically, one TA may store mapping information per BDF, such that there is one PASID table entry for the BDFx / PASIDi pair and another PASID table entry for the BDFy / PASIDi pair. If, however, without one embodiment, a guest were to create a single PRQ to receive commands from all ATCs using PASIDi, then there would be multiple ATCs that would simultaneously attempt to write to the queue, which would require them to use atomic operations to insert a command into the PRQ, which would result in unsatisfactory performance.The guest provides the host with the location of the PRQ descriptor to cause it to program the location in a corresponding PASID table entry as a PRQ descriptor address field. In addition, the host adds the PASIDj associated with the guest to the PASID table entry so that it can be sent to the XPU with the PRQ descriptor address. The XPU uses PASIDj to access the PRQ descriptor and the PRQ. To effect this guest-XPU / PASID combination, a PASID table entry may include, among other fields: PRQ descriptor address; PASIDj (host PASID associated with the guest); PASIDj_valid (to allow the host to indicate whether the XPU should access PRQ descriptor and PRQ without PASID); and PRQ interrupt index.As discussed above, the ATC requests the IOMMU to provide a location of a PRQ. In many implementations, it is impractical for an XPU to implement a large number of registers to describe any PRQ that can be used by the XPU (number of PRQ used by the XPU = number of PASIDs that run across all VMs that the XPU can use). Thus, the XPU may request PRQ information from a TA when it wants to use the PRQ. In one embodiment, the ATC may request the PRQ descriptor address and the guest-associated PASIDj (both stored in a PASID table entry) from the TA using a ProcessInfoRe interface.When an ATC receives a completion from the TA (in response to a previously sent translation request) with insufficient permissions, it regards this event as a page fault. As a result, the ATC requests the system software to provide the desired entitlement by sending a page request to the software, which is an example of a command that the ATC sends to the system software.It can be assumed that translation completion (translation completion) informs the ATC that the page fault is in the guest page tables. In one embodiment, because the software knows the BDF / PASID associated with each PRQ, the instructions themselves need not carry BDF / PASID information, thereby eliminating the complexity of mapping host BDF (used by ATC) to guest BDF (needed by guest software) and mapping host PASID (used by ATC) to guest PASID (needed by guest software).Table 1 is an implementation of information included in a page request, according to one embodiment. Note that as part of this page request, no PASID needs to be specified. TABLE 1 TABLE 1InstructionType of Command sent to the Software5Page addressAddress of the page for which ATC has determined insufficient entitlement52Page Group IndexThis field contains the identifier provided by ATC for the associated page request.9Read Access RequestedIf this field is set, it indicates that the ATC requests read access to the associated page.1Write Access RequestedIf this field is set, it indicates that the ATC requests read access to the associated page.1Last Request in PRGIf this field is set, it indicates that the associated page request is the last request of the associated PRG.1PrivilegeIf this field is set, it indicates that the ATC requests authorisation for a monitoring location. In "clear" (delete), the ATC requests authorisation for the user unit.1When the ATC has a command to send to a guest, it obtains the PRQ descriptor address from the PIC and reads the PRQ descriptor from memory using the PASIDj associated with the guest (obtained from the PIC) to obtain the latest values of all fields describing the PRQ. The ATC then generates a memory write transaction into the PRQ at the address PRQ-base (-base) + PRQ-tail (-tail). This memory write operation is performed using PASIDj, which includes the command information described above in Table 1, as a payload of the memory write operation, in one embodiment. After writing the command to the PRQ, the ATC generates a memory write to update the value of PRQ-tail at the address PRQ-descriptor-address+ tailoffset (tail offset) using PASIDj. This memory write to update the PRQ tail advances the memory write that places the command in the PRQ. In addition to updating the PRQ tail, the ATC also sets the PPR bit in the PRQ descriptor. If the ATC, upon generating the command, observes that the PRQ is full, it may stop the XPU and wait until space is available in the PRQ again.In certain embodiments, it may be assumed that the guest uses pinned memory (PRQ) for the PRQ and that attempts by the ATC to write to the PRQ do not result in page errors. However, in another embodiment, the guest may use paged memory (off-line memory) for the PRQ, and if the ATC encounters page errors in a PRQ write operation, it would report such page errors in a centralized PRQ in the host memory using page request services (PAGE REQUEST services) as described by the ATS 1.x specification. In such an implementation, the ATC may stop the XPU to prevent additional commands from having to be sent to the PRQ until the page fault is resolved and the current command is written to the associated PRQ.The ATC may request the guest software to test the PRQ by sending an interrupt. As discussed above, the interrupt is not sent each time a command is written to the PRQ, but only when the PPR bit in the PRQ descriptor is recognized as "cleared" (clear). Because the guest may have created many PRQs (one for each pair BDF / PASID), the ATC may generate a unique interrupt associated with the PRQ when writing a command to the PRQ. To this end, the ProcessInfoRequest which brought the PRQ descriptor address to the ATC may also provide a PRQ interrupt index from the TA that the uses to index into its bank of MSI-X registers the proper interrupt to send to the CPU.Upon receipt of the interrupt from the ATC, the guest software performs the steps of: (i) clearing the PPR field in the PRQ descriptor (so that the ATC can generate a new interrupt when it adds further instructions to the PRQ); (ii) reading the PRQ descriptor (to obtain the latest value of Tail); (iii) reading one or more instructions from the PRQ between Head and Tail; and (iv) updating the Head field in the PRQ descriptor.It should be noted that steps iii and iv may be iteratively performed by reading head from the PRQ after each command or group of commands; or they may be performed once with all commands between head and tail read at once and head updated once.To implement an architectural interface according to one embodiment, the guest software may perform various operations to set up a PRQ descriptor and fill it with information which may then be used to enable the ATC to use information in that PRQ descriptor to write instructions directly to a PRQ and inform the guest software of the presence of such instructions.FIG. 3 is another block diagram of a system according to an embodiment. In FIG. 3, system 300 is implemented similar to system 200 of FIG. 2 (with the same reference numerals but row "300"), and further illustrates interaction between guest and host software and an ATC, according to an embodiment.As shown, a location of a PRQ descriptor 326 (comprising information for an associated PRQ 324) is stored in an entry in a PASID table 322. The location of the PRQ descriptor 326 is stored in DevPIC 344. When the ATC 340 then attempts to write an instruction to the guest software 312, it first reads the information of the PRQ descriptor 326. Then, it writes the command to the specified location (using the tail pointer information) within the PRQ 324. Finally, the ATC 340 updates the tail pointer by a write to the PRQ descriptor 326.Referring now to FIG. 4, a flowchart of a method according to an embodiment is shown. More specifically, the method 400 of FIG. 4 is a method performed by guest software when executed on one or more cores of a processor to enable an ATC to obtain guest process information from a data structure and use the information to directly inform the guest software of the presence of instructions from the ATC. As such, method 400 may be performed by guest software executing on hardware circuitry such as one or more cores of a processor.As shown, the method 400 begins by creating a PRQ descriptor in memory (block 410). This PRQ descriptor may identify a location of a PRQ for a given XPU and process. That is, there may be multiple PRQs, each PRQ associated with a given XPU and process. Thus, there may be as many PRQs for a given XPU as there are processes executing on that XPU. For each XPU of a system, there may again be multiple such PRQs. It should be noted that in one embodiment, this PRQ descriptor may be implemented in one or more fields of a PASID table stored in memory.Still referring to FIG. 4, at block 420, the location of this PRQ descriptor is then provided to the host software. It is pointed out that this provision can be effected by means of a display of the address (namely a guest virtual address (GVA)). In response to this indication to the host software, the host may add the corresponding PASID associated with the guest to the PASID table entry so that it may be sent to an XPU along with a PRQ descriptor address. These operations at blocks 410 and 420 thus allow the PRQ descriptor to be established and made available to the ATC. The guest software may perform additional operations when a command is sent from an ATC. Specifically, as shown in FIG. 4 at block 430, the guest software may receive an interrupt indicating the presence of an instruction in the PRQ.Still referring to FIG. 4, at the next block 440, the guest software reads a command from the PRQ in response to this interrupt. For discussion, this command is assumed to be a page fault request; of course, other commands may be received in other implementations. In response to this command, the guest software may determine whether to update page permissions associated with a memory page that is the subject of the page fault to indicate whether the ATC is permitted appropriate access to the page (e.g., read or write access). Also at block 440, the guest software may update one or more fields of the PRQ descriptor. For example, in response to this interrupt, the guest software may clear the PPR bit to indicate that the ATC must send an interrupt when writing subsequent commands to the PRQ. The guest software then pulls commands from the PRQ that are between the head pointer and tail pointer and sends the commands for processing. After that, the guest software waits for a next interrupt to restart this process. Although shown in detail in this embodiment of Fig. 4, many variations and alternatives are possible.Referring now to Table 2, exemplary pseudo code for a guest software operation for handling a page request in response to receiving an interrupt as described herein is shown. Referring now to FIG. 5, a flow diagram of a method according to a further embodiment is shown. More specifically, the method 500 shown in FIG. 5 is a method that allows an ATC to communicate directly with various guest software instantiations via an architectural interface. As such, the method 500 may be performed by an ATC of an XPU, e.g., using a cache controller or other hardware circuitry associated with the ATC.As shown, the method 500 begins with a request from the ATC for a location of a PRQ (block 510). It should be appreciated that this request may be issued in response to a page fault or other indication that the ATC does not have access to a particular location, such as a page in shared memory. This request is sent to a translation agent, which in this case is an IOMMU. In response to this request, the ATC receives an address of a PRQ descriptor (block 515). It should be noted that this PRQ descriptor applies to a given XPU and process. Upon receipt of this PRQ descriptor address, the ATC stores it in a DevPIC.At block 520, the ATC then requests a translation for a given memory location from the translation agent (e.g., IOMMU). At block 525, the ATC receives a translation completion, which may include a given address translation, namely a translation to a physical address location associated with a given guest software to identify a location within a shared memory.Still referring to FIG. 5, it is then determined at diamond 530 whether the translation completion provides access authority. If so, at block 540 the address location is stored in the ATC. Otherwise, control passes to block 550. At block 550, the ATC generates a command, e.g., a page fault request. It should be noted that this page fault request may include an identification of a requested entitlement type (e.g., read, write, and / or read / write). Thereafter, at block 560, the ATC accesses the PRQ descriptor to read pointers and a PPR status. Here, the ATC reads head and tail pointers to identify a location to which the request is to be written. The PPR status, e.g., in the form of a PPR bit, indicates whether the ATC should send an interrupt to inform the guest software of the presence of this request.Still referring to FIG. 5, at block 570, the ATC writes the command to the PRQ using the pointers. Then, at block 580, the ATC performs a write to the PRQ descriptor to update the tail pointer (e.g., by incrementing the value by one). Next, at block 585, the PRQ descriptor (including the PPR bit) is read again.Based on this read of the PPR bit in the PRQ descriptor at diamond 590, it is determined whether there is a pending PRQ interrupt to the guest software, e.g., as determined based on the status of the PPR bit. More specifically, if set, this means that there is already at least one PRQ interrupt that has been sent to the software but has not yet been serviced, so that no further interrupt needs to be generated. In this case, no further operation is performed.If it is determined that no pending PRQ interrupts are pending for the software (indicated by a clear PPR bit), control passes to block 595 where the ATC writes a set value to the PPR indicator in the PRQ descriptor and sends an interrupt to inform the guest software of the presence of the page request. In one embodiment, this request may be sent to the host software, which in turn provides the interrupt to the guest software. In other cases, this interrupt may be sent directly to the guest software. It will be appreciated that although shown in detail in the embodiment of Figure 5, many variations and alternatives are possible.Referring now to Table 3, example pseudo code for an ATC operation for writing an instruction to a PRQ as described herein is shown. embodiments may be implemented in a variety of interconnect structures. Referring to FIG. 6, an embodiment of a structure composed of point-to-point links connecting a set of components is illustrated. The system 600 includes a processor 605 and flash memory 610 coupled to the controller hub 615. Processor 605 includes any processing element, e.g., a microprocessor, a host processor, an embedded processor, a co-processor, or another processor. Processor 605 is coupled to controller hub 615 via front side bus (FSB) 606. In one embodiment, the FSB 606 is a serial point-to-point connection.System memory 610 includes any memory device, such as random access memory (RAM), non-volatile (NV) memory, or other memory, that can be accessed by devices in system 600.As shown, system memory 610 is coupled to controller hub 615 via memory interface 616. Examples of a memory interface include a double data rate (DDR) memory interface, a dual channel DDR memory interface, a dynamic RAM (DRAM), and / or an SPI memory interface.In one embodiment, the control hub 615 is a root hub, root complex, or root control (root control) in a PCIe connection hierarchy. Examples of control hubs 615 are a chipset, memory controller hub (MCH), north bridge, interconnect controller hub (ICH), south bridge or peripheral control hub (PCH), and root controller / hub(s). Often, the term chipset refers to two physically separate control hubs, i.e., a memory control hub (MCH) coupled to an interconnect control hub (ICH). It should be appreciated that current systems often include the MCH integrated with processor 605, while controller 615 is configured to communicate with I / devices in a similar manner as described below. In some embodiments, peer-to-peer routing is optionally supported by root complex 615. Root complex 615 may include an IOMMU that, in an SVM model, allows graphics accelerator 630 and / or device 625 (which may include ATCs according to one embodiment) to access a shared memory area with processor 605.The control hub 615 is coupled to the switch / bridge (switch / bridge) 620 via the serial link 619. Input / output modules 617 and 621, which may also be referred to as interfaces / ports 617 and 621, include / implement a multi-layered protocol stack to enable communication between controller hub 615 and switch 620. In one embodiment, multiple devices are capable of being coupled to switch 620.Switch / bridge 620 routes packets / messages from device 625 upstream, i.e., up in a hierarchy toward a root complex, to controller hub 615 and downstream, i.e., down in a hierarchy away from a root controller, from processor 605 or system memory 610 to device 625. In one embodiment, switch 620 is referred to as a logical array of multiple virtual PCI-to-PCI bridge devices. The device 625 includes any internal / s or external / s device or component coupled to an electronic system, such as an I / O device, a network interface controller (NIC), an add-in card, an audio processor, a network processor, a hard disk, a storage device, a CD / DVD-ROM, a monitor, a printer, a mouse, a keyboard, a router, a portable storage device, a firewire device, a universal serial bus (USB) device, a scanner, and other input / output devices. In the PCIe language, such a device is often referred to as an endpoint (endpoint). Although not specifically shown, device 625 may include a PCIe-to-PCI / PCI-X bridge to support legacy or other versions of PCI devices. PCIe terminals are often classified as legacy, PCIe, or root complex integrated (PCIe) endpoints.Graphics accelerator 630 is further coupled to control hub 615 via serial link 632. In one embodiment, graphics accelerator 630 is coupled to an MCH, which in turn is coupled to an ICH. The switch 620, and thus the I / O device 625, is then coupled to the ICH. I / O modules 631 and 618 are also configured to implement a multi-layered protocol stack for communication between graphics accelerator 630 and control hub 615. A graphics controller or graphics accelerator 630 itself may be integrated with the processor 605.Referring now to FIG. 7, shown is a block diagram of a system according to another embodiment. As shown in FIG. 7, a system 700 may be any type of computing device, and in one embodiment may be a server system. In the embodiment of FIG. 7, system 700 includes a plurality of CPUs 710 a,b, which in turn are coupled to respective system memories 720 a,b, which in certain embodiments may be implemented as DIMMs, such as double data rate (DDR) memory, persistent or other types of memory. It should be appreciated that the CPUs 710 may be coupled together via an interconnect system 715 such as Intel® Ultra Path Interconnect, or other processor interconnect technology.To enable coherent accelerator devices and / or smart adapter devices to couple to the CPUs 710 via potentially multiple communication protocols, a plurality of links 730 a 1- b 2 may be present. In one embodiment, each link 730 may be a given instance of a compute express link (CXL) in which PCIe communication, including ATS communication, may occur.In the embodiment shown, the respective CPUs 710 are coupled to corresponding FPGAs / accelerators 750 a,b(which may include GPUs, or other accelerators may include ATCs, according to one embodiment). In addition, the CPUs 710 are also coupled to the smart NIC devices 760 a,b. The smart NIC devices 760 a,bin turn couple to switches 780 a,b, which in turn couple to a pooled memory 790 a,b, e.g., a persistent memory. Of course, embodiments are not limited to accelerators 750, and the techniques and structures described herein may also be implemented in other entities of a system.Referring now to FIG. 8, shown is a block diagram of a system according to another embodiment, e.g., a data center platform. As shown in FIG. 8, multiprocessor system 1000 includes a first processor 870 and a second processor 880 coupled via a point-to-point link 850. As shown in FIG. 8, each of processors 870 and 880 may be multi-core processors representatively including first and second processor cores (i.e., processor cores 874 aand 874 band processor cores 884 aand 884 b).In the embodiment of FIG. 8, processors 870 and 880 further include point-to-point connections 877, 887 coupled via connections 842 and 844 (which may be CXL buses over which PCIe communication runs) to switches 859 and 860, which may include IOMMUs, to allow devices with ATCs to access pooled memories 855 and 865.Still referring to FIG. 8, the first processor 870 further includes a memory controller hub (MCH) 872 and point-to-point (P-P) interfaces 876 and 878. Similarly, the second processor 880 includes an MCH 882 and P-P interfaces 886 and 888. As shown in FIG. 8, MCHs 872 and 882 couple the processors to respective memories, namely a memory 832 and a memory 834, which may be portions of system memory (e.g., DRAM) locally attached to the respective processors. The first processor 870 and the second processor 880 may be coupled to a chipset 890 via P-P links 876 and / or 886. As shown in FIG. 8, chipset 890 includes P-P interfaces 894 and 898.Chipset 890 further includes an interface 892 to couple chipset 890 to a high performance graphics engine 838 via a P-P link 839. As shown in FIG. 8, various input / output (I / O) devices 814 may be coupled to the first bus 1016, along with a bus bridge 818 that couples the first bus 1016 to a second bus 820. Various devices may be coupled to the second bus 820, including, for example, a keyboard / mouse 822, communication devices 826, and a data storage unit 828, such as a hard disk or other mass storage device, which may include code 830, in one embodiment. Further, an audio I / O 824 may be coupled to the second bus 820.The following examples relate to further exemplary embodiments.In one embodiment, an apparatus includes: at least one accelerator to perform operations on data; and an ATC coupled to the at least one accelerator, the ATC configured to store address translations. The ATC is configured to: send a command to a PRQ stored in a memory coupled to the device, the PRQ associated with a process of guest software; and send an interrupt to inform the process of the command.In one example, the apparatus is configured to send the command to a location in the PRQ based on information in a PRQ descriptor stored in the memory, the PRQ descriptor associated with the process.In one example, the apparatus is configured to send the command to the location in the PRQ based at least in part on a tail pointer of the PRQ descriptor.In one example, the ATC is configured to send the interrupt when a pending request interrupt indicator of the PRQ descriptor indicates that there are no untreated pending request interrupts.In one example, the ATC is configured to: send a second command to the PRQ; and not send another interrupt to inform the process regarding the second command when a pending request interrupt indicator of the PRQ descriptor indicates that there is at least one pending request interrupt pending for guest software.In one example, the device is configured to send the command to a location in the memory in response to an access entitlement violation, the memory comprising shared memory to be shared by the device and a host processor on which the guest software is to be executed.In one example, the ATC is configured to receive a translation completion that includes a virtual address-to-physical address translation for the location in the memory, the translation completion to indicate the access entitlement violation.In one example, the ATC is configured to: request an address of the PRQ from a translation agent; and receive, in response, an address of a PRQ descriptor, the PRQ descriptor associated with the PRQ.In an embodiment, the apparatus further comprises a device process information cache, wherein the apparatus is configured to store the address of the PRQ descriptor in the device process information cache.In another example, a method includes: generating, in an ATC of an accelerator, a command to be communicated to guest software when executed on a host processor coupled to the accelerator, the command being for a location in a memory; accessing a PRQ descriptor to read pointer information and an pending request status, the PRQ descriptor associated with a PRQ of the memory, the PRQ associated with the accelerator and a process of the guest software; and writing the command to the PRQ using the pointer information.In one example, the method further comprises sending an interrupt to inform the guest software of the command based at least in part on the pending request status.In one example, the method further comprises sending the interrupt to host software, wherein the host software is configured to provide the interrupt to the guest software.In one example, the method further comprises: requesting an address of the PRQ in the memory via the ATC from a translation agent; and responsive to requesting the address of the PRQ, receiving an address of the PRQ descriptor and storing the address of the PRQ descriptor in a device process information cache of the ATC, wherein the PRQ descriptor is associated with the accelerator and the process of the guest software and comprises the address of the PRQ.In an example, the method further comprises storing a plurality of PRQ descriptor addresses in the device process information cache, wherein each of the plurality of PRQ descriptor addresses is associated with a process of the guest software.In an example, the method further comprises receiving an address of the PRQ descriptor from the translation agent, wherein the PRQ descriptor comprises a head pointer, a tail pointer, and a pending request status indicator.In an example, the method further comprises generating the command in response to an access entitlement violation for the location in the memory.In another example, a computer readable medium comprising instructions is configured to perform the method of any of the above examples.In another example, a computer readable medium comprising data is to be used by at least one machine for manufacturing at least one integrated circuit to perform the method of any of the above examples.In yet another example, an apparatus comprises means for performing the method of any of the above examples.In another example, a system includes: a memory; a CPU coupled to the memory, the CPU having at least one core for executing instructions, the CPU configured to execute guest software; and processing circuitry coupled to the CPU, the processing circuitry including: an accelerator for performing operations on data; and an address translation cache (ATC) coupled to the accelerator, the ATC configured to store address translations, the ATC configured to directly send an instruction to a process of the guest software via a PRQ stored in the memory, the PRQ associated with the process of the guest software.In one example, the system further comprises a translation agent coupled to the processing circuitry, the translation agent configured to send, to the ATC, an address of a PRQ descriptor stored in the memory, the PRQ descriptor associated with the process.In an example, the processing circuit further includes a process information cache to store the address of the PRQ descriptor, the PRQ descriptor including an address of the PRQ and an indicator to indicate whether the ATC is configured to send an interrupt to inform the process of the command.In one example, the memory is configured to store a PASID table, the PASID table storing a plurality of entries, at least one of which is associated with the process and stores the address of the PRQ descriptor.It is noted that various combinations of the above examples are possible.Various embodiments may include any suitable combination of the above-described embodiments, including alternative (or) embodiments of embodiments described above in connection form (and) (e.g., the "and" may be an "and / or"). Further, some embodiments may include one or more articles of manufacture (e.g., non-transitory computer readable medium) having instructions stored thereon that, when executed, result in actions of any of the above-described embodiments. Further, some embodiments may include devices or systems that include any suitable means for performing the various operations of the embodiments described above.It is noted that the terms "circuit" and "circuitry" are used interchangeably herein. As used herein, these terms and the term "logic" refer, individually or in any combination, to analog circuitry, digital circuitry, hardwired circuitry, programmable circuitry, processor circuitry, microcontroller circuitry, hardware logic circuitry, state machine circuitry, and / or any other type of physical hardware components. Embodiments may be used in many different types of systems. For example, in one embodiment, a communication device may be arranged to perform the various methods and techniques described herein. Of course, the scope of the present invention is not limited to a communication device, and instead, other embodiments may be directed to other types of instruction processing devices or to one or more machine readable media comprising instructions that, in response to execution on a computing device, cause the device to perform one or more of the methods and techniques described herein.Embodiments may be implemented in code and may be stored on a non-transitory storage medium having instructions stored thereon that may be used to program a system for executing the instructions. Implementations may also be implemented in data and may be stored on a non-transitory storage medium that, when used by at least one machine, causes the at least one machine to fabricate at least one integrated circuit to perform one or more operations. Further embodiments may be implemented in a computer readable storage medium comprising information that, when incorporated into an SOC or other processor, is configured to configure the SOC or other processor to perform one or more operations. The storage medium may include any type of disk, including, but not limited to, floppy disks, optical disks, solid state drives (SSDs), compact disk read-only memories (CD-ROMs), rewritable compact disks (CD-RWs), and magneto-optical disks, semiconductor devices such as read-only memories (ROMs), random access memories (RAMs) such as dynamic RAMs (DRAMs), static RAMs (SRAMs), erasable programmable read-only memories (EPROMs; Electrically erasable programmable ROMs), flash memory, electrically erasable programmable ROMs (EEPROMs), magnetic or optical cards or any type of media suitable for storing electronic instructions.While the present disclosure has been described with respect to a limited number of implementations, those skilled in the art, having the benefit of the present disclosure, will recognize numerous modifications and variations thereof. It is intended that the appended claims cover any such modifications or variations thereof.

Claims

An apparatus, comprising: at least one accelerator to perform operations on data; and an address translation cache (ATC) coupled to the at least one accelerator, the ATC configured to store address translations, the ATC configured to: send a command to a pending request queue (PRQ) stored in a memory coupled to the apparatus, the PRQ associated with a process of guest software; and send an interrupt to inform the process of the command.The apparatus of claim 1, wherein the apparatus is configured to send the command to a location in the PRQ based on information in a PRQ descriptor stored in the memory, the PRQ descriptor associated with the process.The apparatus of claim 2, wherein the apparatus is configured to send the command to the location in the PRQ based at least in part on a tail pointer of the PRQ descriptor.The apparatus of claim 2 or 3, wherein the ATC is configured to send the interrupt when a pending request interrupt indicator of the PRQ descriptor indicates that there are no untreated pending request interrupts.The apparatus of claim 2, 3 or 4, wherein the ATC is configured to: send a second command to the PRQ; and not send another interrupt to inform the process of the second command when a pending request interrupt indicator of the PRQ descriptor indicates that there is at least one pending request interrupt pending for guest software.The apparatus of claim 2, 3, 4 or 5, wherein the apparatus is configured to send the command to a location in the memory in response to an access entitlement violation, the memory comprising shared memory to be shared by the apparatus and a host processor on which the guest software is to be executed.The apparatus of claim 6, wherein the ATC is configured to receive a translation completion comprising a virtual address-to-physical address translation for the location in the memory, the translation completion to indicate the access entitlement violation.The apparatus of any of claims 1-7, wherein the ATC is configured to: request an address of the PRQ from a translation agent; and receive, in response, an address of a PRQ descriptor, wherein the PRQ descriptor is associated with the PRQ.The apparatus of claim 8, wherein the apparatus further comprises a device process information cache, wherein the apparatus is configured to store the address of the PRQ descriptor in the device process information cache.A method, comprising: generating an instruction to be communicated to guest software when executed on a host processor coupled to the accelerator in an address translation cache (ATC) of an accelerator, the instruction for a location in a memory; accessing a pending request queue (PRQ) descriptor to read pointer information and a pending request status, the PRQ descriptor associated with a PRQ of the memory, the PRQ associated with the accelerator and a process of the guest software; and writing the instruction to the PRQ using the pointer information.The method of claim 10, further comprising sending an interrupt to inform the guest software of the command based at least in part on the pending request status.The method of claim 11, further comprising sending the interrupt to host software, wherein the host software is configured to provide the interrupt to the guest software.The method of claim 10, 11 or 12, further comprising: requesting an address of the PRQ in the memory via the ATC from a translation agent; and responsive to requesting the address of the PRQ, receiving an address of the PRQ descriptor and storing the address of the PRQ descriptor in a device process information cache of the ATC, wherein the PRQ descriptor is associated with the accelerator and the process of the guest software and comprises the address of the PRQ.The method of claim 13, further comprising storing a plurality of PRQ descriptor addresses in the device process information cache, wherein each of the plurality of PRQ descriptor addresses is associated with a process of the guest software.The method of any of claims 10-14, further comprising receiving an address of the PRQ descriptor from the translation agent, wherein the PRQ descriptor comprises a head pointer, a tail pointer, and a pending request status indicator.The method of any of claims 10-15, further comprising generating the command in response to an access entitlement violation for the location in the memory.A computer readable storage medium comprising computer readable instructions which, when executed, implement a method according to any one of claims 10 to 16.An apparatus comprising means for performing a method according to any of claims 10 to 16.A system comprising: a memory; a central processing unit (CPU) coupled to the memory, the CPU having at least one core for executing instructions, the CPU configured to execute guest software; and processing circuitry coupled to the CPU, the processing circuitry comprising: an accelerator for performing operations on data; and an address translation cache (ATC) coupled to the accelerator, the ATC configured to store address translations, the ATC configured to directly send a command to a process of the guest software via a pending request queue (PRQ) stored in the memory, the PRQ associated with the process of the guest software.The system of claim 19, further comprising a translation agent coupled to the processing circuitry, the translation agent configured to send to the ATC an address of a PRQ descriptor stored in the memory, the PRQ descriptor associated with the process.The system of claim 20, wherein the processing circuitry further comprises a process information cache to store the address of the PRQ descriptor, wherein the PRQ descriptor comprises an address of the PRQ and an indicator to indicate whether the ATC is configured to send an interrupt to inform the process of the command.The system of any of claims 19-21, wherein the memory is configured to store a process address space identifier (PASID) table, wherein the PASID table stores a plurality of entries, at least one of which is associated with the process, and stores the address of the PRQ descriptor.An apparatus comprising: at least one accelerator means for performing operations on data; and an address translation cache (ATC) means coupled to the at least one accelerator means, the ATC means being configured to store address translations, the ATC means being configured to: send a command to a pending request queue (PRQ) means stored in a memory coupled to the apparatus, the PRQ means associated with a process of guest software; and send an interrupt to inform the process of the command.The apparatus of claim 23, wherein the apparatus is configured to send the command to a location in the PRQ means based on information in a PRQ descriptor stored in the memory, the PRQ descriptor associated with the process.The apparatus of claim 24, wherein the apparatus is configured to send the command to the location in the PRQ means based at least in part on a tail pointer of the PRQ descriptor.

Images