Low latency accelerator

Abstract

Low latency accelerators are disclosed. Methods, apparatuses, and related techniques and mechanisms for reducing latency in accelerators. The techniques and mechanisms are implemented in a platform architecture that supports shared virtual memory (SVM) and includes an accelerator that supports SVM and use of a translation lookaside buffer (TLB). Request descriptors that define a job to be performed by the accelerator and reference a virtual address (VA) and size of one or more buffers are enqueued by executing a thread on a processor core. In one approach, the descriptor includes a hint that includes a physical address or a virtual address to physical address (VA-PA) translation that is obtained from one or more TLBs associated with the core using the buffer VA. In another approach that employs TLB snooping, the buffer VA is used as a lookup and a matching TLB entry (VA-PA translation) is used as the hint. The hint is used to speculatively prefetch the buffer data and speculatively begin processing the prefetched buffer data on the accelerator.

Description

BACKGROUND

[0001] In recent years, data analytics and "big data" processing have become increasingly important. Data analytics and big data workloads require processing large amounts of data. One approach to processing such large amounts of data is to distribute processing tasks across a large number of servers and process the workloads in parallel. For example, the Apache Hadoop software framework enables distribution of tasks across a large number of commodity servers and uses MapReduce to process the workloads. While Hadoop and MapReduce provide excellent scalability, they require a very large amount of intra-server communication (when implemented at a large scale) and do not efficiently use processor and memory resources.

[0002] Some compute and memory bandwidth intensive workloads, such as for data analytics and "big data," are difficult to achieve the required performance levels using processor cores. To address this problem, so-called "accelerators" have been developed. Accelerators were initially implemented as components coupled to CPUs (central processing units) and managed as input-output (IO) devices with their own address space, which required a significant level of IO communication to transfer data between the accelerator address space and applications running in the system memory address space. More recently, CPUs employing a system-on-a-chip (SoC) architecture with embedded accelerators have been proposed.

[0003] The capabilities of accelerators have steadily increased, and one of the most notable recent trends is accelerators that support "shared virtual memory" (SVM). Traditional accelerators are managed as input-output (IO) devices in their own private address space; this is implemented using an expensive kernel mode driver (KMD) that requires applications to cross between user and kernel space, pin pages in memory, or copy user buffers to and from dedicated buffers managed by the OS / kernel mode driver. With SVM, an accelerator or IO device can work directly on the address space of any user application thread, as it shares the same virtual -> physical address translation capabilities as the CPU threads. This is a key improvement in accelerator efficiency (from the perspective of data movement), which enables user mode to be submitted directly to the accelerator (through a "user mode driver" or UMD) and makes the programming model and adoption easier.

[0004] However, SVM also presents interesting challenges for applications that require low latency processing, especially for small buffer processing. When a job is provided to the accelerator to be processed, the job descriptor identifies some input data buffers and output data buffers in the virtual memory space that the accelerator is to access. These buffers are allocated by the user application and thus can typically include many different physical memory pages depending on the size of the buffers. The accelerator needs to be able to translate virtual addresses (VAs) to physical addresses (PAs) in order to work on the job. This address translation adds latency overhead to traditional accelerator designs. BRIEF DESCRIPTION OF DRAWINGS

[0005] The foregoing aspects and many of the attendant advantages of this application will become more readily appreciated as the same become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein:

[0006] Figure 1 is a schematic diagram of a first platform architecture including one or more on-chip accelerators;

[0007] Figure 2 is a schematic diagram of a second platform architecture including one or more off-chip accelerators;

[0008] Figure 3a is a schematic diagram illustrating further details of an on-chip accelerator according to one embodiment;

[0009] Figure 3b is a schematic diagram illustrating further details of an off-chip accelerator according to one embodiment;

[0010] Figure 4 is a schematic diagram of an address translation scheme including a translation lookaside buffer (TLB) employed by a 64-bit processor architecture;

[0011] Figure 5 is a schematic diagram illustrating further details of the TLB of Figure 4 and including a translation register;

[0012] Figure 6a is a combined schematic diagram and dataflow diagram illustrating one embodiment of an implementation of the Enq_with_translations_vl instruction using the platform architecture of Figure 3a ;

[0013] Figure 6b is a combined schematic diagram and dataflow diagram illustrating one embodiment of an implementation of the Enq_with_translations_vl instruction using the platform architecture of Figure 3b ;

[0014] Figure 7a The diagram shows the use of Figure 3a A combined schematic diagram and data flow diagram of an embodiment of a TLB snooping mechanism for a platform architecture;

[0015] Figure 7b The diagram shows the use of Figure 3b A combined schematic diagram and data flow diagram of an embodiment of a TLB snooping mechanism for a platform architecture;

[0016] Figure 8 is a flow chart illustrating operations and logic related to an enqueue request descriptor according to one embodiment;

[0017] Figure 9 is a flow chart illustrating operations and logic related to accelerator processing of request descriptors according to one embodiment;

[0018] Figure 10 is a simplified NUMA platform architecture illustrating various costs for accessing corresponding components including accelerators and memory devices;

[0019] Figure 10 is a schematic block diagram illustrating an example of an Arm-based microarchitecture suitable for implementing the instruction set architecture (ISA) instructions disclosed herein; and

[0020] Figure 11 is a schematic diagram illustrating further details of an exemplary platform architecture according to one embodiment. DETAILED DESCRIPTION

[0021] This article describes embodiments of methods and devices for reducing latency in accelerators and related technologies and mechanisms. In the following description, many specific details are set forth to provide a thorough understanding of embodiments of the present invention. However, those skilled in the relevant art will understand that the present invention can be implemented without one or more specific details, or by other methods, components, materials, etc. In other examples, well-known structures, materials, or operations are not shown or described in detail to avoid confusion about certain aspects of the present invention.

[0022] Throughout this specification, reference to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in multiple places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0023] For the sake of clarity, various components in the drawings can be labeled with labels other than particular reference numerals in this document. Moreover, components can be shown with reference numerals followed by a "(typ)" to indicate that the component is indicative of a particular type of component (as opposed to a particular component). It will be understood that the configuration of these components will be typical of the following components: similar components that can be present but are not shown in the drawings for purposes of simplicity and clarity; or otherwise similar components that are not labeled with a separate reference numeral. Conversely, "(typ)" is not to be construed to mean that the component, element, etc. is typically used for its disclosed function, implementation, purpose, etc.

[0024] As noted above, accelerators need to be able to convert virtual addresses to physical addresses, which adds a large latency overhead to traditional accelerator designs. According to aspects of embodiments disclosed herein, techniques are provided for reducing the latency associated with address translations used by an SVM-enabled accelerator. In one aspect, a conversion hint is provided from a CPU core / thread, which submits an accelerator request to the accelerator so that the accelerator can start its computation as early as possible. Various mechanisms and schemes can be used to implement this approach.

[0025] For example, in one embodiment, an instruction is implemented to submit a descriptor to the accelerator, with additional metadata for the descriptor containing valid VA-PA translations that have already been read or copied from the CPU(s) translation lookaside buffer (TLB) for the core to execute the instruction thread that includes the instruction. In general, such an instruction can be added to the processor's instruction set architecture (ISA) as a new ISA instruction, as described in further detail below.

[0026] In another approach, the IO memory management unit (IOMMU) "snoops" the CPU core TLBs and obtains the translations cached there. These can be broadcast to all CPU cores, or done more efficiently in a point-to-point fashion (which requires information of which thread / core submitted the job). Note that for maximum efficiency, this should be done as soon as the job descriptor arrives at the accelerator complex - if we wait until the descriptor is finally picked up for execution, then queue latency can cause the core thread to be swapped out during this time.

[0027] As the accelerator works on the job, it uses these received VA-PA translations as hints and starts processing the job when it can. In parallel, in one embodiment, the normal IOMMU process will be followed to obtain the requested VA-PA translations, which are validated against the hints (if already used by the engine) or replaced with the VA-PA translations from the IOMMU process if the engine has not yet used these pages.

[0028] While this is more efficient for input data, as it is read into the accelerator complex, we need to pay more attention to the output data. This can be addressed in the following ways.

[0029] 1. The above mechanism (snoop or instruction based) only provides hints for input / reads data. The engine has to wait for the translation on the output buffer from the IOMMU to enter in order to write their data.

[0030] 2. The mechanism provides hints for all translations, but the engine has enough local memory to buffer their output until the translation is validated by the IOMMU.

[0031] 3. There is a special provision in some job submission where the output data is part of the transaction. In this case, the accelerator can be allowed to write back the data in the output region in the memory based on the hints. However, if the translation misses, the accelerator signals a "transaction abort".

[0032] Note that in case of input or output translation error, the engine has to start over from the beginning or can have a mechanism to roll back to a checkpoint state.

[0033] Figure 1 and Figure 2 Exemplary platform architectures 100 and 200 are shown under which aspects of the embodiments described herein can be implemented. The platform architecture 100 includes a processor 102 having a system-on-a-chip (SoC) architecture mounted to a motherboard 103. The processor 102 includes a central processing unit (CPU) (also referred to as a core or core unit) that includes n processor cores 1061-106 n , where n is an integer. For example, n can be 2, 4, 6, 8, 10, 12, 16, or higher numbers. More generally, the processor 102 is a multi-core processor, meaning that it has at least two processor cores. Each of the processor cores 1061-106 n is coupled to a first (LI) and second level (L2) cache, depicted as LI caches 1081-108 n and L2 caches 1101-110 n . As further shown, each of the LI caches 1081-108 n includes an instruction cache (LI I) and a data cache (LID).

[0034] Each of the LI and L2 caches is associated with a respective translation lookaside buffer (TLB), such as TLBs 1121-112 n for the LI caches 1081-108 n and TLBs 1141-114 nTLBs 1141-114 n As further described and illustrated below, in some embodiments, each of the LI instruction and data caches LI I and LI D can have a respective TLB.

[0035] The processor 102 includes various interconnect circuitry for interconnecting the various components on the processor. For simplicity, the interconnect circuitry is shown as an interconnect 116, which is also referred to as a mesh. In practice, the mesh can include an interconnect fabric and one or more levels on an interconnect hierarchy, with each level can include its own separate hierarchy (e.g., a nested interconnect hierarchy). Typically, a given interconnect hierarchy can employ structures and operations defined by a standardized protocol or a proprietary protocol. Moreover, there can be bridges between layers to enable interfacing between different types of protocols.

[0036] The various components on the processor 102 are interconnected via the interconnect 116, including the L2 caches 1101-110 n , m accelerators 1181-118 m , a third level (L3) cache 122 (also referred to as a last level cache or LLC), and a pair of memory controllers 124 and 126 (also labeled MC1 and MC2). Note also that each of the components illustrated for each processor core (including the core, LI cache, and TLB) are communicably coupled to the interconnect 116 via direct or indirect connections.

[0037] Each of the memory controllers 124 and 126 also has an associated IOMMU and IO TLB, collectively depicted as IOMMU / IO TLB blocks 128 and 130. In embodiments that implement multiple accelerators, the accelerators can be interconnected in a chain, as shown by the dashed double arrow 120. As further shown in Figure 11 As further shown in the

[0038] Each of the memory controllers 124 and 126 includes one or more channels connected to one or more DRAM (dynamic random access memory) memory devices 132, such as dual in-line memory modules (DIMMs) 134. In Figure 1In this regard, the DRAM memory devices are depicted as DRAM 1-4. In the illustrated embodiment, a respective memory channel (as indicated by Ch. 1, Ch. 2, Ch. 3, etc., where "Ch." means channel) is connected to each DRAM memory device 132. However, this is merely exemplary, as a memory controller can have more than one memory channel connected to the same memory device.

[0039] Each of the DRAM memory devices 132 has a physical address space. Generally, the physical address space is divided into "page" units, which are further divided into cache line units, although other addressing schemes can be used. The physical address space of a DRAM memory device is mapped to a virtual address space, as shown by virtual address space 136. The virtual address space is typically divided into a plurality of virtual memory "pages," and is accessed at the page level, noting that individual cache lines can also be accessed with pages. The mapping between virtual and physical address spaces is generally handled by the memory controller and / or other platform components, including an IOMMU and TLBs. Depending on the particular platform, the operating system can provide further aspects of this mapping.

[0040] In one embodiment, the processor 102 is operatively coupled to a printed circuit board including the motherboard 103 via a "socket," or otherwise coupled to the motherboard via direct coupling techniques such as flip-chip bonding. In either case, it is common practice in the art to refer to the processor itself as the socket. Generally, the motherboard 103 includes electrical wiring (e.g., traces and vias) to facilitate communication with the processor 102 and other components of the platform 100. The motherboard 103 also includes a number of other components, including a memory controller 130, which is coupled to the processor 102 via a bus 134. The memory controller 130 is coupled to a number of DRAM memory devices 132, which are depicted as DRAM 1-4. In the illustrated embodiment, a respective memory channel (as indicated by Ch. 1, Ch. 2, Ch. 3, etc., where "Ch." means channel) is connected to each DRAM memory device 132. However, this is merely exemplary, as a memory controller can have more than one memory channel connected to the same memory device. Figure 1 The electrical connections corresponding to the physical structures of the various interconnects depicted in

[0041] Figure 2 The platform architecture 200 includes a processor 202, four DRAM memory devices 132, m accelerators 2041-204 m , and a network interface controller (NIC) 206 mounted or otherwise coupled to the motherboard 203. In addition to the components having the same reference numerals in Figure 1 and Figure 2 The processor 202 also includes IO components and interfaces, including a PCIe (Peripheral Component Interconnect Express) root complex 208 coupled to a plurality of PCIe interfaces, depicted as PCIe interfaces 210, 212, and 214. The PCIe interface 210 is coupled to the accelerator 2041 via a PCIe link 216, while the PCIe interface 212 is coupled to the accelerator 204 mand the PCIe interface 214 is coupled to the NIC 206 via the PCIe link 220. PCIe supports various lane widths and speeds, including lx, 2x, 4x, 8x, and 16x configurations, where the ‘x’ denotes the number of lanes. Both existing and future PCIe links and interfaces can be used Figure 2 The PCIe links and interfaces shown and discussed herein.

[0042] In general, off-chip accelerators can include chips (e.g., field programmable gate arrays (FPGAs) or chips with fixed pre-programmed logic) that are assembled on a motherboard, or can reside on accelerator boards or cards that are installed in PCIe expansion slots. There can also be multiple accelerator chips on the same board or card.

[0043] As discussed in further detail below, off-chip accelerators, such as the accelerators 2041-204 m may include a memory controller or other type of memory interface that enables the accelerator to access system memory devices over one or more memory channels. This is shown in Figure 2 where the accelerator 2041 is connected to DRAM 1 via memory channel 1, while the accelerator 204 m is connected to DRAM 4 via memory channel 4, where the dashed lines indicate that the connections are optional.

[0044] By using the direct memory access (DMA) support provided by PCIe and related components, the NIC 206 is enabled to access system memory (e.g., DRAM memory devices 1-4) without the need to use the CPU 104. In addition, DMA operations can also be used to support data transfers between the NIC 206 and one or more accelerators 2041-204 m such that packet processing operations for network traffic received from and / or sent to the network 222 can be offloaded to the one or more accelerators.

[0045] In addition to the platform architecture that uses on-chip and off-chip accelerators, a hybrid platform architecture that includes both on-chip and off-chip accelerators is also supported. The architecture generally combines applicable aspects of the on-chip and off-chip accelerators described herein.

[0046] Figure 3a and Figure 3b Further details of the processors and accelerators shown in Figure 1 and Figure 2 are shown. Figure 3a A platform architecture 100a is shown that includes a processor 300 having an on-chip accelerator 302 (also referred to as an accelerator complex when the accelerator supports multiple accelerator function instances). The processor 300 includes the CPU 104, which includes the cores 106, the memory controller 108, and the PCIe interface 110. The accelerator 302 is coupled to the CPU 104 via the PCIe interface 110.Figure 1 and Figure 2 The CPU 104 shown in FIG. 1 is representative of a CPU that includes a number of cores and TLBs, and is shown in FIG. 3 as including n cores 106 and a CPU core TLB 302; it will be recognized that the CPU 104 will further include L1 and L2 caches, and the TLB 302 is representative of Figure 1 and Figure 2 the TLBs 1121-112 n and TLBS 1141-114 n The interconnect 116 and LLC 122 of the processor 102 are collectively illustrated as a mesh structure / LLC block 304, which is coupled to a memory controller 306 with an associated IOMMU / IOTLB 308. The memory controller 306 is coupled to a memory 310, which is representative of one or more DRAM memory devices 132 of Figure 1 and Figure 2 As previously noted, the memory controller 306 will be coupled to DRAM memory devices, each of which (not shown) has one or more memory channels.

[0047] Figure 3a Also shown is an embodiment of an on-chip accelerator 312, which is representative of various types of accelerators. The on-chip accelerator 312 includes a fabric interface 314, a device TLB 316, a host interface DMA queue 318, a scheduler request / completion queue 320, and a bus 322 with a number of compression and decompression blocks (illustrated as compression blocks 324 and 326 and decompression blocks 328 and 330). The fabric interface 314 is generally illustrative of various types of IO interfaces that can connect the on-chip accelerator to the interconnect infrastructure on the processor / SoC, which is collectively illustrated and described herein as a mesh structure. The particular interconnect structure and protocol can generally include proprietary interconnects and standards-based interconnects.

[0048] For purposes of illustration, the on-chip accelerator 312 is shown as including two compression and decompression blocks. Accelerators are generally used to offload CPU intensive tasks from the cores of a processor, such as math-intensive compression and decompression functions. In addition, accelerators can include embedded circuitry and logic that is tailored to efficiently perform one or more specialized tasks, such as compression and decompression in the accelerators shown herein. The circuitry can be in the form of an ASIC (application specific integrated circuit), or can include programmable circuitry / logic, such as provided by an FPGA. Such an FPGA can include one or more FPGA blocks, such as can be available through licenses from different manufacturers. The FPGA blocks can also contain custom designs. In general, ASICs, FPGA blocks, or similar embedded circuitry and logic are referred to herein as functional units that are designed to perform respective functions.

[0049] More generally, an accelerator can also be referred to as an "engine," where the engine can be programmed to perform one or more specialized functions. In some embodiments, an engine can operate in a similar manner as an embedded processor, and be capable of executing instructions for specialized functions (e.g., accelerator application / function instructions). An engine can also combine execution of instructions in conjunction with embedded circuitry and logic.

[0050] In Figure 3b Under embodiments of platform architecture 200a, processor 202 includes a processor 202 coupled to an off-chip accelerator 332. Processor 202 generally includes components similar to processor 202 as collectively illustrated in a similar manner as Figure 2 Figure 3a In a similar manner, components and blocks of off-chip accelerator 332 and monolithic accelerator 312 having like reference numerals operate in a similar manner as described above with reference to Figure 3a

[0051] In addition to these components and blocks, off-chip accelerator 332 includes a memory controller 334 and an IO fabric block 336. In the illustrated embodiment, memory controller 334 is coupled to memory 310 as described above, which is representative of one or more DRAM memory devices. Such a memory controller 334 can generally connect to one or more DRAM memory devices via one or more memory channels.

[0052] In Figure 3b In the illustrated embodiment, off-chip accelerator 332 is connected to processor 202 via a PCIe link 338. Accordingly, each of processor 202 and off-chip accelerator 332 will also include a PCIe interface (not shown for clarity) at the opposite end of PCIe link 338. As described above, an off-chip accelerator can include an accelerator board or card that is installed in a PCIe expansion slot or the like, as well as being installed to a motherboard. Other means of communication can be employed to couple an off-chip accelerator in communication with a processor in addition to PCIe.

[0053] Host interface DMA queue 318 and scheduler request / completion queue 320 enable accelerators 312 and 332 to support DMA operations under which memory 310 and other platform components (e.g., I / O devices) are accessed. Figure 2 ​​accesses of the NIC 206) without any CPU involvement. For example, under an accelerator configured to support compression and decompression, network traffic received at a receive port on the NIC can be decompressed by the accelerator prior to forwarding the traffic for further processing by a network stack implemented via execution of software instructions on a processor core. Similarly, for outbound traffic (i.e., data to be transmitted to the network via a transmit port of the NIC), the traffic can be compressed by the accelerator.

[0054] Aspects of various embodiments described and illustrated herein, including embodiments of the Figure 6a , Figure 6b , Figure 7a and Figure 7b embodiments, read or copy TLB entries that include VA-PA translations and use (VA-PA translations or just PAs) as hints in order to facilitate speculative fetching (i.e., prefetching) and processing of data using accelerators. To better understand operations associated with reading / copying TLB entries and / or providing hints in accordance with embodiments described herein, reference is now made to Figure 4 and Figure 5 to provide a discussion of operations of an exemplary 64-bit TLB architecture.

[0055] The 64-bit architecture employs a virtual addressing model that is essentially a 64-bit flat linear address space. 64-bit registers are used as pointers to this address space. The architecture also supports 32-bit virtual linear addresses that are zero-extended into the 64-bit virtual address space.

[0056] The 64-bit virtual address space is divided into eight 2 61 byte virtual regions. A region is selected by the upper 3 bits of the virtual address. Associated with each virtual region is a region register that specifies a 24-bit region identifier for the region. There are 2 24 eight virtual address spaces can be accessed simultaneously by eight region registers. If desired, regions can be merged by assigning sequential region identifiers, for example, to produce 62, 63, or 64-bit spaces.

[0057] Figure 4The process of mapping (also referred to as "translation") virtual address 400 to physical address 402, resulting in a VA-PA translation, is shown. The bits of virtual address 400 are divided into three fields: 404, 406, and 408. Field 404 includes the three most significant bits 61-63, which are used to point to a virtual region number (VRN). Field 408, which includes the least significant bits, forms the page offset. The remaining bits, including field 406, are used to specify a virtual page number (VPN). The page offset bits (field 408) are passed through the translation process unmodified, as shown by physical address page offset 408'. The exact bit positions of the page offset and VPN bits (field 406) vary depending on the page size used in the virtual mapping. Permitted page sizes include 4K, 8K, 16K, 64K, 256K, 1M, 4M, 16M, 64M, 256M, and 4G.

[0058] At the time of a memory reference (except for inserts or clears), the VRN bits of field 404 select a region identifier (RID) 410 from one of eight region registers 412. A translation entry with matching VPN and RID values is then searched for in TLB 214. If a matching translation entry is found, the physical page number (PPN) 416 of the entry is concatenated with the page offset bits (offset 408') to form physical address 402. The eligibility of the matching translation is determined by a page-grain privilege level access permission check and an optional protection domain check by verifying the key of the translation is contained within a set of protection key registers 417, and read, write, or execute permissions are granted. The key value is defined in a key field 418 of the TLB, while the access permissions are specified in a permissions field 420.

[0059] If the translation is not resident in the TLB, the processor can optionally search a virtual hash page table (VHPT) structure in memory (not shown) for the required translation and install the entry into the TLB. If the required entry is not found in either the TLB or the VHPT, the processor issues a TLB miss fault to request the operating system to provide the translation. After the operating system installs the translation in the TLB and / or VHPT, the faulting instruction can be restarted and execution resumed.

[0060] In one embodiment, a 64-bit processor maintains two architectural TLBs, as shown in Figure 2. The first TLB 214 is a level 1 (Ll) TLB, while the second TLB 216 is a level 2 (L2) TLB. The Ll TLB 214 is a fully associative TLB, while the L2 TLB 216 is a set associative TLB. Figure 5As shown, the TLB includes instruction TLB (ITLB) 500 and data TLB (DLTB) 502. Each TLB services translation requests for instruction and data memory references, respectively. Data TLB 502 also services translation requests for RSE and VHPT walker references. Each of instruction TLB 500 and data TLB 502 is further divided into two sub-sections: translation registers (TRs) (ITR 504 and DTR 506) and translation caches (TCs) (ITC 508 and DTC 510).

[0061] The translation register portion of the TLB is a fully associative array defined to hold translations managed directly by software (e.g., OS) and / or firmware. Software / firmware can explicitly insert translations into the TR by specifying a register slot number. Translations are removed from the TR by specifying a virtual address, page size, and region identifier. The translation registers allow the operating system to "pin" key virtual memory translations in the TLB. Examples include I / O space, kernel memory regions, frame buffers, page tables, sensitive interrupt code, etc.

[0062] An entry is placed into a particular TR slot using the insert translation register (itr) instruction. Once a translation is inserted, the processor will not automatically replace the translation to make room for other translations. A local translation can only be removed by issuing a purge translation register (ptr) instruction.

[0063] Those skilled in the art will appreciate that Figure 4 and Figure 5 The foregoing description of the 64-bit TLB and related architecture illustrated in FIGS. 1-3 is merely an example of one type of TLB and is not limiting. Other types of TLB architectures can be implemented using techniques known in the art in accordance with the principles and teachings of the embodiments herein.

[0064] An SVM-capable IO device must perform address translation before submitting a read / write transaction on the IO fabric. One mechanism is to have a TLB in the device that caches some translations. If the device TLB cannot service the translation, the request is sent to the IOTLB in the IOMMU. The IOMMU performs address translation by looking up the IOTLB, and if there is no entry, it initiates a page walk. In the case of a page fault, the IOMMU must report the event to software. More details can be found in the PCISIG standard under ATS (Address Translation Services) (e.g., see Quick PCI Address Translation Services 1.1). Under PCIe, the IOTLB is also referred to as the Address Translation Cache (ATC) in order to distinguish it from the TLB used by the CPU. Optionally, the ATC can be stored separately from the IOTLB, where updates to the ATC are copied to the IOTLB.

[0065] New ISA instruction

[0066] Under aspects of some embodiments disclosed herein, a new instruction is provided that can be implemented as part of the ISA of a processor. The basic idea is that when a CPU thread needs to access an accelerator, the thread submits its request descriptor by writing it to a specific memory address that has been mapped by the memory to an input portal of the accelerator device. The thread discovers this memory address earlier through a kernel call that handles the discovery / enumeration of accelerators and their functionality.

[0067] The thread cannot read / write this memory location through normal load / store instructions, but can only write descriptors to it and receive an acknowledgement (whether accepted or not) using a special ISA from device. The CPU core's hardware also enhances the descriptor payload with additional information such as process / thread ID, and importantly, a translation list as shown in Listing 1 below, which contains the pseudo code for implementing the ISA instruction (Enq_with_translations_vl).

[0068] The core hardware only lists the translations that exist in the L1 or L2 TLB, but does not trigger fetching the translations if they do not exist. The reason is that the accelerator is likely to need the data but not the core CPU thread.

[0069] Listing 1

[0070]

[0071]

[0072] The Enq_with_translations_vl instruction includes three parameters (operands): dest, src, and len, which are stored in the first and second m64 64-bit registers and the r64 64-bit register, respectively. The m64 notation indicates that the operand is a pointer to memory, such that the contents will be used for load / store instructions. The r64 notation indicates that the operand is used as a data value. As shown in Listing 1, dest stores the address of the accelerator portal to write the descriptor to, and src stores the address of the descriptor from the user application of length len.

[0073] As further shown in lines 6-17, in the embodiment of Listing 1, the descriptor has a format that includes the number of input buffers, a pointer to each input buffer (e.g., input-buffer1-pointer, input-buffer2-pointer), and the size of each input buffer (described as input-buffer2-size, input-buffer2-size, ...), the number of output buffers, a pointer and size of each output buffer (described as output buffer1 pointer, output buffer2 size, and other payload information, such as function opcodes, completion records, etc.).

[0074] As shown in lines 19-25, the VM# (virtual machine number), process ID, thread number, and core number are written to dest. The outer for loop (lines 20-25) is used to process each input buffer in src. The inner for loop has an iterator that is the size of the page (4K in this example, note that other page sizes can also be used). Therefore, for each page in each buffer, the virtual address of the current page (the virtual address of the buffer pointer plus "i") is used to look up the physical address (PA) from the TLB, which is equal to the page size time offset from the beginning of the buffer. If the result is not NULL, the physical address of the current page is written to dest.

[0075] As further shown in lines 27-28, the user payload is written to dest by performing a memory copy (memcopy) of length len bytes from src (register) to dest (register). The thread then waits for confirmation from the accelerator (line 29) and returns a pass (processing succeeded without error) or a fail (error occurred) as shown in line 31.

[0076] Variants

[0077] The basic version of the ISA instruction with the pseudocode shown in Listing 1 can be extended to lookup transitions for input and output buffers. Note that for things like context / state buffers that are both read and written, the instruction will create transitions for them, but the accelerator will hold off on writing the state until the transition is confirmed.

[0078] In the basic form of the enqueue instruction, we report only success or failure from the accelerator, and nothing else. To enable a low-latency system, we can extend the protocol to have the accelerator return something akin to a "service time" estimate. This is similar to our experience of joining some queue where we are told our expected wait time is 'X'. Based on the expected wait time, the thread can decide to wait / poll or swap out. In addition, when the CPU thread will need to process the translations for the output buffer, it can decide to prefetch those translations. Pseudo code for one embodiment of a variant of the basic instruction of List 1 is shown in List 2.

[0079] List 2

[0080]

[0081]

[0082]

[0083] Lines 1-30 of List 1 and List 2 are the same (ignoring the different instruction names). In line 31, the time value returned from the accelerator is designated as "T". If T is less than a predetermined threshold tl, then the operations shown in lines 34-41 are performed. For each output buffer in src (line 34), and for each page of that buffer (line 35), the TLB is used to look up the physical address (PA) of the page in a similar manner as discussed above. If the PA is NULL, then page walking is initiated to find the physical address and insert it into the TLB. If T is less than a second threshold t2, as shown in line 43, the execution of the thread is stalled, otherwise it is swapped out for another thread.

[0084] As an alternative to prefetching translations through the page walker, a reverse snoop from the IOTLB for output data translations can be performed.

[0085] Accelerator processing

[0086] The operations and data flow for speculative prefetching of buffer data related to accelerator processing for an exemplary on-chip accelerator embodiment are shown in Figure 6a and Figure 7a The corresponding operations and data flow for an exemplary off-chip accelerator embodiment are shown in Figure 6b and Figure 7b In addition, the operations and logic of an exemplary embodiment corresponding to accelerator processing are shown in flowcharts 800 and 900 of Figure 8 and Figure 9

[0087] Figure 6a An illustration of using​Figure 3a implementation of the Enq_with_translations_vl instruction of the platform architecture 100a. In this example, an executing thread 600 executing on core n invokes the Enq_with_translations_vl instruction (depicted in block 602) with applicable data loaded into the operands discussed above, including a reference (e.g., a pointer) to a buffer's virtual address. The Enq_with_translations_vl instruction uses the buffer virtual address as input to the TLB(s) associated with core n. For example, in one embodiment, a data TLB will be used for the Ll cache, while in another embodiment, both a data TLB for the Ll cache and a TLB for the L2 cache will be used. Using the buffer VA as a lookup, a TLB "hit" or "miss" is checked for, where a TLB hit means that a TLB entry exists that includes a VA-PA translation for the buffer VA, where the Enq_with_translations_vl instruction returns the physical address (PA) for that VA. As described above in lines 20-25 of Listing 1, Enq_with_translations_vl uses the TLB lookup and returns the PA for each of the pages in memory where each buffer is stored.

[0088] The Enq_with_translations_vl instruction further enqueues a request descriptor 604 into the ingress of accelerator 312, depicted as a request descriptor added to the scheduler request / completion queue 320. The request descriptor contains various parameters shown in Listing 1, including the buffer VA and size 605 plus the translated PA, which are used as address hints.

[0089] In one embodiment, when the request descriptor is enqueued, a speculative data fetch operation is performed to pre-fetch (presumed) buffer data stored in the memory pages corresponding to the PA hint 606 and the buffer size using a DMA transfer, as shown in block 608. Under the platform architecture 100a, the DMA transfer is facilitated by using the host interface DMA queue 318, fabric interface 314, mesh / LLC 304, processor memory controller 306, and memory 310, as depicted by the bolded connections in Figure 6a While the speculative pre-fetch buffer data, speculative processing of the buffer data by one or more functional units on accelerator 312 can begin.

[0090] Figure 6b shows an implementation of the Enq_with_translations_vl instruction of the platform architecture 200a using Figure 3b and Figure 6a andFigure 6b The same numbered components and blocks in FIG. 6 are shown until the operations performed by the enqueue request descriptor 604 are performed for Figure 6a and Figure 6b are all the same. However, under platform architecture 200a, the DMA access mechanism is different. As depicted by block 610, DMA operations are facilitated by using the memory controller (334) of the onboard accelerator 332. As shown by the bold lines, the DMA access path includes the host interface DMA queue 318, the IO structure 336, the memory controller 334, and the memory 310. As described above, when speculative prefetching buffer data, speculative processing of the buffer data by one or more functional units on the accelerator 312 can begin.

[0091] Figure 7a and Figure 7b An embodiment of a scheme for supporting speculative prefetching of buffer data using IOTLB snooping is illustrated. Figure 6a and Figure 6b , an execution thread 700 executes on core n and uses an accelerator entry to enqueue a request descriptor 702, depicted as the scheduler request / completion queue 320. However, instead of including the physical address of the buffer, the request descriptor 702 includes the virtual address of the buffer.

[0092] As described above, once the request descriptor 702 is enqueued, speculative prefetching and processing of the buffer data is performed using DMA transfers. Figure 7a In the embodiment, the path for DMA transfer is Figure 6a is the same as in . However, in an embodiment of IOTLB snooping, DMA prefetch accesses to memory 310 reference the virtual address of the buffer rather than the physical address. Since the memory controller uses physical addresses to access memory 310, a VA-PA translation needs to be performed before accessing the memory. This is facilitated by using the IOMMU / IOTLB 308. First, the IOTLB is checked to see if the existing TLB entry for the VA results in a hit. If so, the returned PA is used to access the memory page for the buffer. If the TLB lookup by the IOMMU / IOTLB 308 results in a miss, the IOMMU / IOTLB performs a snoop of the TLB associated with core n using the VA as the TLB lookup, as shown in block 706. For a TLB hit, the corresponding TLB entry (VA-PA translation) is copied from the (multiple) application core TLBs to the IOMMU / IOTLB 308, as shown in block 708. The applicable memory page for the buffer in memory 310 may then be accessed using the applicable TLB entry in the IOMMU / IOTLB 308 , as shown in block 710 .

[0093] likeFigure 7a and Figure 7b As further depicted in FIG. 7, the TLB entries in the IOMMU / IOTLB can be copied into the device TLB 316. Under this approach, the device TLB can be checked for a TLB hit prior to submitting the DMA operation to the memory controller 306. If the VA lookup to the device TLB results in a hit, the PA from the VA-PA translation for the memory page can be used to submit the DMA access request, thereby eliminating the need to use the IOMMU / IOTLB 308.

[0094] In one embodiment, the IOTLB snooping of the CPU core TLB 302 is performed in conjunction with the associated DMA transfer. In another embodiment, speculative IOTLB snooping can be performed prior to the DMA transfer. Under this approach, the IOTLB performs speculative pre-caching of the current CPU core TLB entries. Depending on the size of the IOTLB and other considerations, such as interconnect bandwidth and latency considerations consumed by using IOTLB snooping, various pre-caching approaches can be employed.

[0095] Figure 7b An embodiment of a TLB snooping approach for the platform architecture 200a using Figure 3b is depicted. The request descriptor and its enqueue operation are the same as shown in Figure 7a and described above. However, under the platform architecture 200a, the DMA access path is via the memory controller (334) on the accelerator 332 instead of the memory controller 306. Under the platform architecture 200a, the memory controller 334 utilizes the device TLB 316 to perform the VA-PA translation. If the TLB lookup of the device TLB 316 results in a miss, the IOMMU / IOTLB snooping is performed in a manner similar to that described for Figure 7a Once the physical address of the buffer memory page is available (either from the device TLB or read or snooped from the IOMMU / IOTLB), the accelerator memory controller uses the PA to access the buffer memory page in the memory 310, as depicted in block 712.

[0096] In one embodiment, the IOTLB TLB entries resulting from the IOMMU / IOTLB snooping performed on behalf of an accelerator are copied to the device TLB for that accelerator. In general, the copying of the entries from the IOTLB to the device TLB can be performed in conjunction with the current DMA operation, or it can be asynchronous to any DMA operation under which multiple TLB entries can be copied.

[0097] Depending on the particular platform configuration, a platform can include more than one off-chip accelerator. Embodiments including a combination of off-chip and on-chip accelerators can also be supported. In addition, as Figure 1 andFigure 2 As shown, the platform architecture can include multiple memory controllers, each with its own IOMMU / IOTLB. It should also be noted that IOTLB entries can result from memory accesses by other IO devices (other than accelerators) in the platform, such as DMA operations to transfer data between system memory and a NIC.

[0098] In addition to speculative prefetching and processing of buffer data, mechanisms are provided for verifying that the accessed buffer data is in fact correct data. In one embodiment, buffer data is verified by verifying that the physical address used to prefetch the data is a correct physical address. The reason for verifying the physical address (and thus the buffer data) is that TLB entries can become stale or not reflect the actual physical location of a given memory page at a particular point in time. For example, a memory buffer can be swapped in and out of physical memory by an operating system (e.g., to a disk to effectively increase the size of the physical memory of the platform). The given VA-PA mapping can also change, resulting in accessing the wrong memory page.

[0099] As described above, Figure 8 and Figure 9 Flowcharts 800 and 900 illustrate operations and logic corresponding to exemplary embodiments of accelerator processing. Reference is made to Figure 8 Block 802 of flowchart 800, an enqueue ISA instruction is used to write a descriptor to a special address that maps to an entry in the accelerator. This causes the accelerator to receive a "doorbell" in block 804. The doorbell or doorbell ring is in effect a signal or similar signal received by the accelerator indicating that it has new work to perform. In response to receiving the doorbell, the accelerator checks its hardware queue to make sure that an empty slot exists (for adding a descriptor to the hardware queue).

[0100] As depicted by decision block 806, if an empty slot exists, the logic flow goes to block 808 where the descriptor is added to the request queue of the accelerator and a success is returned to the thread on the CPU that submitted the descriptor. For example, in the context of the accelerator of Figure 3a and Figure 3b The descriptor is added to the request descriptor queue in the scheduler request / completion queue 320 (enqueued). If no empty slot exists, the logic flow goes to block 810 where the accelerator can optionally return a value representing the current queue depth, which serves as a proxy for the expected wait time until a slot becomes available.

[0101] At this point, the logic proceeds to flowchart 900. As shown in block 902, upon reading the descriptor, the ATS request is started for all buffers identified by the descriptor. In parallel, in decision block 904, it is determined whether the descriptor is ready to be processed by the functional unit on the accelerator. If the answer is no, one or more subsequent checks of decision block 904 are made after a predetermined delay and / or using a polling scheme. Once the answer is yes, the logic proceeds to block 906, where the ATS status of the input and status buffers is checked.

[0102] In decision block 908, it is determined whether the status indicates that the ATS for the input and status buffers is available. If the answer is yes, the logic proceeds to block 910, where the DMA operation and execution of the functional unit is started. If the answer to decision block 908 is no, the logic proceeds to decision block 912, where it is determined whether a hint was provided for the descriptor for the first input and status page. If the answer is yes, the DMA operation is started speculatively, while the speculative execution of the functional unit is started with the read data. If the answer to decision block 912 is no, the logic loops back to block 906.

[0103] Meanwhile, the operations and logic shown in dashed block 916 are performed asynchronously in a continuous fashion. In block 918, a check is made to detect any new ATS mapping for any speculative uses of the translation. As noted above, the details of the operation of the ATS are defined by the PCISIG standard (e.g., Fast PCI Address Translation Services 1.1).

[0104] In decision block 920, it is determined whether there is a match. If the answer is yes, the logic proceeds to block 922, where processing continues. However, if the answer to decision block 920 is no, the process is restarted from the beginning after waiting for all ATS operations to complete, or otherwise the processing is resumed from a previous checkpoint known to be good.

[0105] As further illustrated, each of blocks 922, 910, and 914 feed asynchronously to block 926. In this block, all output buffers are buffered into temporary local memory until the ATS provides a valid translation for the first page(s) of the output buffer(s). The process is then completed in block 928, where the output stream is written out (non-speculatively).

[0106] In some embodiments, PCIe ATS is accessed through a simulated PCIe device. For example, in embodiments of the on-chip accelerator described herein, as Figure 3aAs illustrated in the middle, the accelerator is coupled to the mesh fabric through a fabric interface that does not employ PCIe. To access PCIe ATS, the accelerator emulates a PCIe device by using PCIe protocol messages encapsulated within messages sent via the fabric interface. The mesh fabric interconnect fabric includes a PCIe-to-fabric bridge configured to interface a PCIe interconnect with a fabric interconnect. At the bridge, the encapsulated PCIe messages are decapsulated and forwarded to the appropriate PCIe components on the processor, such as a PCIe root complex.

[0107] In one embodiment, each of the foregoing Enq_with_translations_vl and Enq_with_translations_v2 instructions can be implemented in a processor employing the x86 ISA. However, this is merely exemplary and not limiting as variants of the foregoing instructions can be implemented on various processor architectures. For example, consider a RISC type Arm processor. ARM instructions can generally allow for 3 operands. They have integer scalar instructions that work on general purpose registers (GPRs) (e.g., 16 or 32 registers), as well as vector / float instructions that work on 128-bit SIMD (called Neon) registers.

[0108] Figure 10 An example of one embodiment of an Arm processor microarchitecture 1000 is shown. The microarchitecture 1000 includes a branch prediction unit (BPU) 1002, a fetch unit 1004, an instruction translation lookaside buffer (ITLB) 1006, a 64 KB (kilobyte) instruction store 1008, a fetch queue 1010, a plurality of decoders (DECs) 1012, a register renaming block 1014, a reorder buffer (ROB) 1016, a reservation station unit (RSU) 1018, 1020, and 1022, a branch arithmetic logic unit (BR / ALU) 1024, an ALU / MUL (multiplier) / BR 1026, a shift / ALU 1028 and 1030, and a load / store block 1032 and 1034. The microarchitecture 1000 further includes vector / float (VFP) Neon blocks 1036 and 1038, and VFP Neon crypto block 1040, an L2 control block 1042, integer registers 1044, 128-bit VFP and Neon registers 1046, an ITLB 1048, and a 64 KB instruction store 1050.

[0109] In general, each of the Enq_with_translations_v1 and Enq_with_translations_v2 instructions described herein may be implemented using embedded logic (eg, via circuitry), microcode, or a combination of both. In the Arm microarchitecture, general purpose registers may be used for 64-bit m64 and r64 operands.

[0110] Due to space limitations, the Figure 1 and 2 Processors 102 and 202 in. Figure 11 A platform architecture 1100 is shown, which illustrates what may generally be included in Figure 1 and 2 1 and 2. Further details of the processor 1102 and platform components in the platform architectures 100 and 200 are provided. The platform architecture 1100 includes a motherboard 1104 to which various components are mounted and / or otherwise communicatively coupled, including the processor 1102. As described above, the processor 1102 includes a CPU 104 that includes L1 caches 108 1-108 coupled to the L1 caches 108 1-108 n and L2 cache 1101-110 n n processor cores 1061-106 n As further shown, each L1 cache 1081-108 n Includes an instruction cache (L1I) and a data cache (L1D), and each of the L1 and L2 caches is associated with a corresponding translation lookaside buffer (TLB), as shown by the L1 caches 1081-108 n TLB 1121-112 n and for L2 cache 1101-110 n TLB 1141-114 n As shown. L2 cache 1101-110 n Connected to interconnect 116 .

[0111] Also connected to the interconnect 116 is a PCIe root complex 1106, and L3 cache 122, accelerators 1181-118 m , IO interface 1108, and memory controllers 124 and 126 (also labeled MC1 and MC2).

[0112] The PCIe root complex 1106 will generally be coupled to one or more PCIe interfaces, depicted as PCIe interfaces 1114, 1116, and 1118. In general, all or a portion of the PCIe interfaces and PCIe links can be connected to a PCIe expansion slot (not shown) that is populated on a motherboard 1104. The PCIe interface 1114 is depicted as being connected to an off-chip accelerator 1120 via a PCIe link 1122. As discussed above, an off-chip accelerator can include an accelerator chip or the like that is populated on a motherboard of a platform, or that is mounted on an accelerator board or card that is populated in a PCIe expansion slot.

[0113] The PCIe interface 1116 is connected to a NIC 1126 that provides access to a network 1128 via a PCIe link 1124. In general, the NIC 1126 is representative of various types of network interface adapters and fabric adapters, including but not limited to Ethernet adapters, InfiniBand Host Controller Adapters (HCAs), and OmniPath Host Fabric Interface (HFI).

[0114] The PCIe interface 1118 is connected to a solid state drive (SSD) 1130 via a PCIe link 1132. Optionally, other types of IO interfaces can be used to enable the processor to communicate with the SSD. As shown, system software 1134 is stored in the SSD 1130. The system software can generally include an operating system and one or more applications that run on the operating system. The system software can also support various types of virtualization embodiments, including virtualization platforms that implement Type 1 and Type 2 hypervisors, as well as container-based virtualization environments. As further depicted by the software 1136, all or a portion of the system software can be loaded during platform boot on the network 1128.

[0115] The IO interface 1108 is connected to a firmware storage device, such as a flash memory device 1138, via an IO link 1140. The flash memory device 1138 stores system firmware 1142 that is loaded as part of platform initialization. In general, depending on the platform, various types of firmware can be used, including firmware that employs the Unified Extensible Firmware Interface (UEFI) architecture. For historical reasons, all or a portion of the firmware can also be referred to as BIOS (Basic Input Output System).

[0116] In general, the processor can include one or more memory controllers, each including one or more memory channels that are connected to one or more memory devices, as discussed above. Figure 11Embodiments include two memory controllers 124 and 126. Memory controller 124 is coupled to memory devices DRAM 1 and DRAM 2 via respective memory channels 1 and 2. Memory controller 126 is coupled to memory devices DRAM 3 and DRAM 4 via respective memory channels 3 and 4.

[0117] In addition to off-chip accelerators that have a memory controller and are configured to directly access system memory via the memory controller, off-chip accelerators can not include a memory controller and access system memory through DMA operations that are forwarded through a processor via a memory controller on the processor. For example, one or more accelerators can be installed in an expansion card or board that is installed in a PCIe expansion slot. From an architectural perspective, each of the accelerators on the card or board operates as a PCIe device. However, since the PCIe link that connects the processor to the PCIe expansion slot is not coupled to system memory, the PCIe devices on the card or board installed in the PCIe expansion slot cannot directly access system memory and thus use the DMA operations described above.

[0118] Further aspects of the subject matter described herein are set forth in the following numerous clauses:

[0119] A method performed on a computing platform having a multi-core processor coupled to one or more memory devices, the memory devices comprising system memory implemented as shared virtual memory (SVM), the multi-core processor having a plurality of cores each associated with at least one translation lookaside buffer (TLB), the multi-core processor further comprising at least one memory controller operably coupled to the one or more memory devices through at least one memory channel, the computing platform further comprising one or more SVM-enabled accelerators having one or more functional units, the method comprising:

[0120] enqueuing, by execution of a thread on one of the plurality of cores, a request descriptor for an accelerator, the request descriptor defining a job to be performed by the accelerator and including information associated with one or more buffers in the system memory to be accessed by the accelerator during processing of the request descriptor;

[0121] speculatively prefetching buffer data from the system memory using the information in the request descriptor associated with the one or more buffers;

[0122] speculatively starting processing of the job by processing at least a portion of the buffer data that was prefetched using one or more functional units of the accelerator; and

[0123] verify whether the buffer data of the speculative prefetch is the correct buffer data to be processed by the job.

[0124] 2. The method of clause 1, wherein the information associated with the one or more buffers in the system memory comprises a virtual address, a size, and an address hint for each of the one or more buffers, the method further comprising:

[0125] for each of the one or more buffers,

[0126] speculatively prefetching buffer data for the buffer using the address hint for the buffer.

[0127] 3. The method of clause 2, wherein the address hint for each of the one or more buffers comprises:

[0128] a physical address for at least a first page of one or more memory pages containing data corresponding to the buffer; or

[0129] a virtual address to physical address (VA-PA) translation for at least the first page of one or more memory pages containing data corresponding to the buffer.

[0130] 4. The method of clause 2, wherein the address hint for each of the one or more buffers is obtained by using an instruction set architecture (ISA) instruction in the execution thread executed by the processor core.

[0131] 5. The method of clause 4, wherein execution of the ISA instruction by the processor core causes the processor to:

[0132] for each of the one or more buffers,

[0133] use the virtual address of the buffer as a TLB lookup in one or more TLBs associated with the processor core, the TLB lookup returning a physical memory address comprising a hint.

[0134] 6. The method of clause 1, wherein the information associated with the one or more buffers in the system memory comprises a virtual address and a size for each of the one or more buffers, the method further comprising:

[0135] for each of the one or more buffers,

[0136] using the virtual address for the buffer as a TLB lookup in one or more TLBs associated with the core executing the thread, the TLB lookup identifying at least one of a physical memory address or a virtual address to physical address (VA-PA) translation hint; and

[0137] speculatively prefetching buffer data for the buffer using the hint.

[0138] 7. The method of any of clauses 2-6, further comprising:

[0139] for each of the one or more buffers, converting the virtual address for the buffer to a physical address to obtain a current valid physical address for the buffer; and

[0140] for each of the one or more buffers, verifying the address hint for the buffer with the current valid physical address for the buffer.

[0141] 8. The method of clause 7, wherein converting a virtual address for the buffer to a physical address for the buffer is facilitated by using a peripheral component interconnect express (PCIe) address translation service.

[0142] 9. The method of clause 8, further comprising:

[0143] evaluating the PCIe address translation service by emulating a PCIe device.

[0144] 10. The method of any of the preceding clauses, wherein verifying that the buffer data speculatively prefetched is the correct buffer data to be processed by the job, the method further comprising:

[0145] continuing processing of the job by processing all of the buffer data prefetched using one or more functional units of the accelerator.

[0146] 11. A computer system comprising:

[0147] a multi-core processor, the multi-core processor comprising:

[0148] a plurality of processor cores, each processor core having at least one associated translation lookaside buffer (TLB), each TLB to store a plurality of TLB entries comprising virtual address to physical address (VA-PA) translations;

[0149] an interconnect fabric communicably coupled to each processor core and each TLB associated with each processor core;

[0150] a memory controller communicably coupled to the interconnect fabric and having one or more memory channels;

[0151] an input-output (IO) interface communicably coupled to the interconnect fabric; and

[0152] an IOTLB to store a plurality of TLB entries including VA-PA translations for one or more IO devices;

[0153] a system memory including one or more memory devices each communicably coupled to at least one memory channel of the memory controller;

[0154] an accelerator supporting shared virtual memory (SVM) having one or more functional units and embedded on the processor or communicably coupled to the processor through the IO interface;

[0155] wherein the system is configured to,

[0156] enqueuing, through execution of an execution thread on one of the plurality of cores, a request descriptor for the accelerator, the request descriptor defining a job to be executed by the accelerator and including information associated with one or more buffers in system memory to be accessed by the accelerator during processing of the request descriptor;

[0157] speculatively prefetching buffer data from the system memory through use of the information in the request descriptor associated with the one or more buffers;

[0158] speculatively starting processing of the job through processing at least a portion of the buffer data prefetched using one or more functional units of the accelerator; and

[0159] verifying whether the speculatively prefetched buffer data is the correct buffer data to be processed by the job.

[0160] 12. The computer system of clause 11, wherein the information associated with the one or more buffers in system memory includes a virtual address, a size, and an address hint for each of the one or more buffers, wherein the system is further configured to:

[0161] for each of the one or more buffers,

[0162] The address hint for the buffer is used to speculatively prefetch buffer data for the buffer.

[0163] 13. The computer system of clause 12, wherein the address hint for each of the one or more buffers comprises:

[0164] a physical address for at least a first page of one or more memory pages containing data corresponding to the buffer; or

[0165] a virtual address to physical address (VA-PA) translation for at least the first page of one or more memory pages containing data corresponding to the buffer.

[0166] 14. The computer system of clause 12 or 13, wherein each of the processor cores has an instruction set architecture (ISA), and the address hint for each of the one or more buffers is obtained by using an ISA instruction in the execution thread executed by the processor core.

[0167] 15. The computer system of clause 14, wherein execution of the ISA instruction by the processor core causes the processor to:

[0168] for each of the one or more buffers,

[0169] use the virtual address of the buffer as a TLB lookup in one or more TLBs associated with the processor core, the TLB lookup returning a physical memory address comprising the hint.

[0170] 16. The computer system of any of clauses 12-15, wherein the system is further configured to:

[0171] for each of the one or more buffers, convert the virtual address for the buffer to a physical address to obtain a currently valid physical address for the buffer; and

[0172] for each of the one or more buffers, verify the address hint for the buffer with the currently valid physical address for the buffer.

[0173] 17. The computer system of clause 16, wherein the IO interface comprises a Peripheral Component Interconnect Express (PCIe) interface, and the processor comprises a PCIe root complex coupled to the interconnect fabric and to the PCIe interface, and the processor is configured to support PCIe address translation services (ATS), and wherein translating a virtual address for a buffer to a physical address for the buffer is facilitated using the PCIe ATS.

[0174] 18. The computer system of clause 11, wherein the information associated with the one or more buffers in system memory comprises a virtual address and a size of each of the one or more buffers, and wherein the system is further configured to:

[0175] for each of the one or more buffers,

[0176] use the virtual address of the buffer as a TLB lookup in one or more TLBs associated with the core executing the thread, the TLB lookup identifying at least one of a physical memory address comprising a hint or a virtual address to physical address (VA-PA) translation comprising a hint; and

[0177] speculatively prefetch buffer data for the buffer using the hint.

[0178] 19. The computer system of clause 18, wherein the system is further configured to:

[0179] translate the virtual address for each of the one or more buffers to a physical address to obtain a currently valid physical address; and

[0180] for each of the one or more buffers, verify the hint for that buffer with the currently valid physical address for that buffer.

[0181] 20. The computer system of any of clauses 11-19, wherein responsive to verifying that the speculatively prefetched buffer data is the correct buffer data to be processed by the job, the system is further configured to continue processing of the job by processing all of the buffer data prefetched using one or more functional units of the accelerator.

[0182] 21. A multi-core processor, the multi-core processor comprising:

[0183] a plurality of processor cores, each processor core having at least one associated translation lookaside buffer (TLB), each TLB for storing a plurality of TLB entries comprising virtual address to physical address (VA-PA) translations;

[0184] an interconnect structure communicably coupled to each processor core and each TLB associated with each processor core;

[0185] a memory controller communicably coupled to the interconnect structure and having one or more memory channels;

[0186] an accelerator communicably coupled to the interconnect structure, having one or more functional units and at least one of a request queue and an entry,

[0187] wherein the multi-core processor is configured to be installed in a computer system that includes system memory, the system memory including one or more memory devices coupled to the one or more memory channels when the processor is installed in the computer system, the computer system employing a shared virtual memory (SVM) architecture, and wherein each processor core has an instruction set architecture (ISA) that includes instructions that, when executed, cause the processor to:

[0188] write a request descriptor into the accelerator entry or enqueue a request descriptor into the accelerator request queue, the request descriptor defining a job to be performed by the accelerator and including an address hint for each of one or more buffers in system memory, the address hint to be accessed by the accelerator during processing of the request descriptor,

[0189] wherein the processor speculatively prefetches buffer data corresponding to the buffer from the system memory using the address hint for the buffer.

[0190] 22. The multi-core processor of clause 21, wherein the plurality of processors are further configured to:

[0191] in response to enqueuing the request descriptor into the accelerator request queue,

[0192] speculatively prefetch buffer data for each of the one or more buffers from the system memory using the address hint for that buffer;

[0193] speculatively begin processing the job defined by the request descriptor by processing at least a portion of the buffer data that was prefetched using one or more functional units of the accelerator; and

[0194] verify whether the speculatively prefetched buffer data is the correct buffer data to be processed by the job.

[0195] 23. The multicore processor of clause 21 or 22, wherein the ISA instruction comprises: a first operand in a first register to store an address of the accelerator entry for a write request descriptor; a second operand in a second register to store an address of the request descriptor from a user application running in the system memory; and a third operand in a third register to store a length of the request descriptor.

[0196] 24. The multicore processor of clause 23, wherein execution of the ISA instruction by a processor core causes the processor to:

[0197] extract, from the request descriptor, data identifying a virtual address of each of one or more buffers;

[0198] for each of the one or more buffers,

[0199] use the virtual address of the buffer as a TLB lookup in one or more TLBs associated with the processor core, the TLB lookup returning a physical memory address comprising a hint; and

[0200] update the request descriptor to include the hint.

[0201] 25. The multicore processor of clause 24, wherein at least one of the buffers comprises data stored in multiple pages in the system memory, and wherein execution of the ISA instruction by a processor core causes the processor to:

[0202] update the request descriptor for each buffer comprising data stored in multiple pages in the system memory with a physical memory address for each of the multiple pages.

[0203] 26. The multicore processor of clause 24 or 25, wherein the request descriptor identifies virtual addresses for one or more input buffers and one or more output buffers, and execution of the instruction causes the processor to generate one or more hints for each of the one or more input buffers but not for any of the one or more output buffers.

[0204] 27. The multicore processor of clause 24 or 25, wherein the request descriptor identifies virtual addresses for one or more input buffers and one or more output buffers, and execution of the instruction causes the processor to generate one or more hints for each of the one or more input buffers and for each of the one or more output buffers.

[0205] 28. The multi-core processor of clause 24 or 25, wherein the request descriptor identifies virtual addresses for one or more input buffers and one or more output buffers, and execution of the instruction causes the processor to:

[0206] generate one or more hints for each of the one or more input buffers;

[0207] send information to the accelerator;

[0208] receive a time value from the accelerator;

[0209] if the time value is less than a threshold,

[0210] generate one or more hints for each of the one or more output buffers.

[0211] 29. The multi-core processor of any of clauses 22-28, wherein the accelerator further comprises a device TLB, and the processor is further configured to add a new VA-PA translation to the device TLB in conjunction with at least one of the following actions: accessing a physical address through a hint and verifying a VA-PA translation associated with the hint.

[0212] 30. The multi-core processor of any of clauses 22-29, wherein the accelerator is further configured to access the system memory using a direct memory access (DMA) operation.

[0213] 31. The multi-core processor of clause 30, wherein the accelerator further comprises a plurality of DMA queues.

[0214] 32. The multi-core processor of any of clauses 21-31, further comprising an input- output memory management unit (IOMMU) coupled to the memory controller.

[0215] 33. The multi-core processor of any of clauses 21-32, wherein the processor is further configured to:

[0216] for each of the one or more buffers, convert a virtual address for the buffer to a physical address to obtain a currently valid physical address for the buffer; and

[0217] for each of the one or more buffers, verify the address hint for the buffer with the currently valid physical address for the buffer.

[0218] 34. The multicore processor of clause 33, wherein translating a virtual address for the buffer to a physical address for the buffer is facilitated by using a Peripheral Component Interconnect Express (PCIe) address translation service.

[0219] 35. The multicore processor of clause 34, wherein the accelerator is enabled to access the PCIe address translation service by emulating a PCIe device.

[0220] 36. The multicore processor of any one of clauses 22-35, further comprising an input-output translation lookaside buffer (IOTLB), wherein the multicore processor is configured to add a VA-PA translation to the IOTLB in conjunction with at least one of the following actions: generating a VA-PA translation associated with a hint and validating a VA-PA translation associated with a hint.

[0221] 37. A multicore processor, the multicore processor comprising:

[0222] a plurality of processor cores, each processor core having at least one associated translation lookaside buffer (TLB), each TLB to store a plurality of TLB entries comprising virtual address to physical address (VA-PA) translations;

[0223] an interconnect structure communicably coupled to each processor core and each TLB associated with each processor core;

[0224] a memory controller communicably coupled to the interconnect structure and having one or more memory channels;

[0225] an input-output (IO) interface;

[0226] an accelerator communicably coupled to the interconnect structure, having one or more functional units and a request queue,

[0227] wherein the multicore processor is configured to be installed in a computer system comprising a system memory, the system memory comprising one or more memory devices coupled to the one or more memory channels and a shared virtual memory (SVM)-capable accelerator coupled to the IO interface when the processor is installed in the computer system, the computer system employing a shared virtual memory architecture, and wherein each processor core has an instruction set architecture (ISA) comprising instructions that, when executed, cause the processor to perform the following operations:

[0228] sending, through the IO interface, a request descriptor to the accelerator, the request descriptor defining a job to be performed by the accelerator and including an address hint for each of one or more buffers in system memory, the buffers being accessed by the accelerator during processing of the request descriptor,

[0229] wherein the accelerator employs the address hint for a buffer to speculatively prefetch buffer data corresponding to the buffer from the system memory.

[0230] 38. The multi-core processor of clause 37, wherein the ISA instruction comprises: a first operand in a first register to store an address of an entry on the accelerator for writing a request descriptor; a second operand in a second register to store an address of the request descriptor from a user application running in the system memory; and a third operand in a third register to store a length of the request descriptor.

[0231] 39. The multi-core processor of clause 38, wherein execution of the ISA instruction by a processor core causes the processor to:

[0232] extract, from the request descriptor, data identifying a virtual address of each of one or more buffers;

[0233] for each of the one or more buffers,

[0234] use the virtual address of the buffer as a TLB lookup in one or more TLBs associated with the processor core, the TLB lookup returning a physical memory address including a hint; and

[0235] update the request descriptor to include the hint.

[0236] 40. The multi-core processor of clause 39, wherein at least one of the buffers comprises data stored in multiple pages in the system memory, and wherein execution of the ISA instruction by a processor core causes the processor to:

[0237] update the request descriptor for each buffer comprising data stored in multiple pages in the system memory with a physical memory address for each of the multiple pages.

[0238] 41. The multicore processor of any of clauses 39 or 40, wherein the request descriptor identifies virtual addresses for one or more input buffers and one or more output buffers, and execution of the instruction causes the processor to generate one or more hints for each of the one or more input buffers, but not for any of the one or more output buffers.

[0239] 42. The multicore processor of any of clauses 39 or 40, wherein the request descriptor identifies virtual addresses for one or more input buffers and one or more output buffers, and execution of the instruction causes the processor to generate one or more hints for each of the one or more input buffers and each of the one or more output buffers.

[0240] 43. The multicore processor of any of clauses 39 or 40, wherein the request descriptor identifies virtual addresses for one or more input buffers and one or more output buffers, and execution of the instruction causes the processor to:

[0241] generate one or more hints for each of the one or more input buffers;

[0242] send information to the accelerator;

[0243] receive a time value from the accelerator;

[0244] if the time value is less than a threshold,

[0245] generate one or more hints for each of the one or more output buffers.

[0246] 44. The multicore processor of any of clauses 34-37, wherein the accelerator is further configured to:

[0247] speculatively prefetch buffer data for each of the one or more buffers from the system memory using the address hints for that buffer;

[0248] speculatively begin processing the job defined by the request descriptor by processing at least a portion of the buffer data prefetched using one or more functional units of the accelerator,

[0249] and wherein the multicore processor is further configured to,

[0250] receive an address translation request from the accelerator including a virtual address for each of the one or more buffers for which data is to be speculatively prefetched using hints;

[0251] converting the virtual address to a physical address to produce a VA-PA translation; and

[0252] for each address translation request, returning the VA-PA translation to the accelerator.

[0253] 45. The multi-core processor of clause 44, wherein the IO interface comprises a Peripheral Component Interconnect Express (PCIe) interface, and wherein converting the virtual address to the physical address is facilitated using a PCIe address translation service.

[0254] 46. The multi-core processor of any of clauses 44 or 45, further comprising an input-output translation lookaside buffer (IOTLB) to store a plurality of TLB entries including VA-PA translations for one or more IO devices, including the accelerator, wherein the multi-core processor is configured to add a VA-PA translation to the IOTLB in conjunction with producing a VA-PA translation.

[0255] 47. An accelerator supporting shared virtual memory (SVM), comprising:

[0256] an input-output (IO) interface;

[0257] a first memory controller;

[0258] an ingress; and

[0259] one or more functional units;

[0260] wherein the accelerator is configured to be installed in a computer system comprising a multi-core processor having a plurality of processor cores and a second memory controller coupled to a system memory comprising one or more memory devices, wherein each processor core is associated with one or more translation lookaside buffers (TLBs), the computer system employing a shared virtual memory architecture, wherein when installed in the computer system, the accelerator is coupled to the multi-core processor through the IO interface, the first memory controller of the accelerator is coupled to at least one of the one or more memory devices, and wherein the accelerator is configured, when installed in the computer system, to:

[0261] receive a request descriptor from an execution thread executing on one of the processor cores of the multi-core processor, the request descriptor defining a job to be performed by the accelerator and including information associated with one or more buffers in the system memory to be accessed by the accelerator during processing of the request descriptor;

[0262] speculatively prefetch buffer data from the system memory by using the information in the request descriptor associated with the one or more buffers; and

[0263] speculatively begin processing the job by processing at least a portion of the buffer data that was prefetched using one or more functional units of the accelerator.

[0264] 48. The SVM-enabled accelerator of clause 47, further configured to verify whether the speculatively prefetched buffer data is the correct buffer data to be processed by the job.

[0265] 49. The SVM-enabled accelerator of clause 47 or 48, wherein the information associated with the one or more buffers in system memory comprises a virtual address, a size, and an address hint for each of the one or more buffers, further configured to:

[0266] for each of the one or more buffers,

[0267] speculatively prefetch buffer data for the buffer using the address hint for the buffer.

[0268] 50. The SVM-enabled accelerator of clause 49, wherein the address hint for each of the one or more buffers comprises:

[0269] a physical address for at least a first page of one or more memory pages containing data corresponding to the buffer; or

[0270] a virtual address to physical address (VA-PA) translation for at least the first page of one or more memory pages containing data corresponding to the buffer.

[0271] 51. The SVM-enabled accelerator of clause 47, wherein the information associated with the one or more buffers in system memory comprises a virtual address and a size for each of the one or more buffers, the request descriptor comprising information identifying a processor core that executed a thread that generated the request descriptor, wherein the SVM-enabled accelerator is further configured to:

[0272] for each of the one or more buffers,

[0273] use the virtual address for the buffer as a TLB lookup in one or more TLBs associated with the core that executed the thread, the TLB lookup identifying at least one of a hinting physical memory address or a hinting virtual address to physical address (VA-PA) translation; and

[0274] speculatively prefetch buffer data for the buffer using the hint.

[0275] 52. The SVM-enabled accelerator of any of clauses 47-51, further comprising a device TLB in which a plurality of VA-PA translations are to be stored, wherein the accelerator is configured to employ the device TLB in order to access the system memory.

[0276] 53. The SVM-enabled accelerator of clause 52, wherein the accelerator is configured to receive a VA-PA translation from the multi-core processor and use the received VA-PA translation to update one of existing TLB entries in the device TLB or add a new TLB entry in the device TLB using the received VA-PA translation.

[0277] 54. The SVM-enabled accelerator of any of clauses 49-53, further configured to:

[0278] send address translation requests to the multi-core processor including a virtual address for each of the one or more buffers, wherein a hint is employed to speculatively prefetch data for the buffer;

[0279] receive a VA-PA translation from the multi-core processor for each address translation request; and

[0280] verify the hint for each hint by comparing the received VA-PA translation from the multi-core processor.

[0281] 55. The SVM-enabled accelerator of clause 54, wherein the IO interface comprises a Peripheral Component Interconnect Express (PCIe) interface and the multi-core processor is configured to implement a PCIe address translation service, and wherein the address translation service request comprises a PCIe address translation service request.

[0282] 56. The SVM-enabled accelerator of any of clauses 47-55, further comprising a request queue, wherein in response to receiving the request descriptor, the accelerator enqueues the request descriptor into the request queue.

[0283] 57. The SVM-enabled accelerator of clause 56, further configured to:

[0284] receive a doorbell, the doorbell indicating that a request descriptor has been received;

[0285] check the request queue in order to determine whether an empty slot exists; and

[0286] add the request descriptor if an empty slot exists.

[0287] 58. The SVM-enabled accelerator of clause 57, wherein the accelerator is further configured to return a current queue depth to the execution thread if it is determined that there is no empty slot.

[0288] 59. A method performed on a computing platform, the computing platform having a multi-core processor coupled to one or more memory devices, the memory devices comprising a system memory implemented as shared virtual memory (SVM), the multi-core processor having a plurality of cores, each core having a first level cache and a second level cache associated with at least one translation lookaside buffer (TLB), the multi-core processor further comprising at least one memory controller operably coupled to the one or more memory devices by at least one memory channel, the computing platform further comprising one or more SVM-enabled accelerators, each SVM-enabled accelerator having a device TLB and comprising one or more functional units configured to perform at least one function, the method comprising:

[0289] enqueuing a request descriptor for an accelerator by executing a thread on one of the plurality of cores, the request descriptor defining a job to be performed by the accelerator and identifying virtual addresses stored in the system memory for a plurality of buffers to be accessed by the accelerator during processing of the request descriptor;

[0290] speculatively starting processing of the job by,

[0291] for each of one or more of the plurality of buffers,

[0292] identifying a speculative VA-PA translation having a virtual address corresponding to the buffer and performing a direct memory access (DMA) operation using a physical address from the speculative VA-PA translation to read data from the system memory that speculatively corresponds to the buffer; and

[0293] starting processing of the data read from the one or more buffers using one or more functional units on the accelerator.

[0294] 60. The method of clause 59, further comprising:

[0295] in parallel with speculatively starting processing of the job,

[0296] converting the virtual addresses for each of the one or more buffers to physical addresses to create current valid VA-PA translations; and

[0297] The speculative VA-PA translation is verified by comparing the speculative VA-PA translation for each of the one or more buffers with the current active VA-PA translation for that buffer.

[0298] 61. The method of clause 59 or 60, further comprising:

[0299] copying the speculative (VA-PA) translation from a TLB associated with the core; and

[0300] providing a copy of the speculative VA-PA translation using the request descriptor.

[0301] 62. The method of clause 59, further comprising:

[0302] identifying a respective virtual address in the request descriptor for the one or more buffer request descriptors; and

[0303] snooping the TLB associated with the core using the respective virtual address to obtain the copy of the speculative VA-PA translation from the TLB.

[0304] 63. The method of any of clauses 1-10 and 59-62, further comprising:

[0305] in response to enqueuing the request descriptor at the accelerator,

[0306] reading the request descriptor; and

[0307] starting address translation service (ATS) requests for all of the plurality of buffers.

[0308] 64. The method of clause 63, further comprising:

[0309] determining whether the descriptor is ready for processing by a functional unit on the accelerator.

[0310] 65. The method of clause 64, wherein the descriptor is determined to be ready for processing by a functional unit on the accelerator, and wherein the plurality of buffers comprises an input and a state buffer, further comprising:

[0311] checking the ATS status of the input or state buffer;

[0312] if the ATS status indicates that the address translation for the input or state buffer is available,

[0313] starting a direct memory access (DMA) operation to access the input or state buffer and start execution of the functional unit;

[0314] Otherwise, if the ATS state indicates that the address translation for the input or state buffer is not available,

[0315] speculatively initiating a DMA operation using one of the virtual-to-physical address translations provided in the request descriptor under which the accelerator reads data in the system memory speculatively corresponding to the input or state buffer; and

[0316] speculatively initiating execution of the function unit to perform an accelerator function that operates on the read data.

[0317] 66. A computer system comprising:

[0318] a multi-core processor comprising:

[0319] a plurality of processor cores each having at least one associated translation lookaside buffer (TLB) each for storing a plurality of TLB entries comprising virtual address to physical address (VA-PA) translations;

[0320] an interconnect structure communicably coupled to each processor core and each TLB associated with each processor core;

[0321] a memory controller communicably coupled to the interconnect structure and having one or more memory channels;

[0322] an input-output (IO) interface communicably coupled to the interconnect structure; and

[0323] an IOTLB for storing a plurality of TLB entries comprising VA-PA translations for one or more IO devices;

[0324] a system memory comprising one or more memory devices each communicably coupled to at least one memory channel of the memory controller;

[0325] an accelerator supporting shared virtual memory (SVM) having one or more function units and embedded on the processor or communicably coupled to the processor through the IO interface;

[0326] wherein the system further comprises means for:

[0327] enqueues, through execution of an execution thread on one of the plurality of cores, a request descriptor for the accelerator, the request descriptor defining a job to be performed by the accelerator and including information associated with one or more buffers in system memory, the buffers to be accessed by the accelerator during processing of the request descriptor;

[0328] speculatively prefetches buffer data from the system memory using the information in the request descriptor associated with the one or more buffers;

[0329] speculatively begins processing the job by processing at least a portion of the buffer data that was speculatively prefetched using one or more functional units of the accelerator; and

[0330] verifies whether the speculatively prefetched buffer data is the correct buffer data to be processed by the job.

[0331] 67. The computer system of clause 66, wherein the information associated with the one or more buffers in system memory includes, for each of the one or more buffers, a virtual address, a size, and an address hint, wherein the system further comprises means for:

[0332] for each of the one or more buffers,

[0333] speculatively prefetches buffer data for the buffer using the address hint for the buffer.

[0334] 68. The computer system of clause 67, wherein the address hint for each of the one or more buffers includes:

[0335] a physical address for at least a first page of one or more memory pages containing data corresponding to the buffer; or

[0336] a virtual address to physical address (VA-PA) translation for at least the first page of one or more memory pages containing data corresponding to the buffer.

[0337] 69. The computer system of clause 67 or 68, wherein each of the processor cores has an instruction set architecture (ISA), and the address hint for each of the one or more buffers is obtained using an ISA instruction in the execution thread executed by the processor core.

[0338] 70. The computer system of clause 69, wherein execution of the ISA instruction by the processor core causes the processor to:

[0339] for each of the one or more buffers,

[0340] using the virtual address for the buffer as a TLB lookup in one or more TLBs associated with the processor core, the TLB lookup returning a physical memory address including a hint.

[0341] 71. The computer system of any of clauses 67-70, wherein the system further comprises means for:

[0342] for each of the one or more buffers, converting the virtual address for the buffer to a physical address to obtain a currently valid physical address for the buffer; and

[0343] for each of the one or more buffers, verifying the address hint for the buffer with the currently valid physical address for the buffer.

[0344] 72. The computer system of clause 71, wherein the IO interface comprises a Peripheral Component Interconnect Express (PCIe) interface, and the processor comprises a PCIe root complex coupled to the interconnect fabric and to the PCIe interface, and the processor is configured to support PCIe address translation services (ATS), and wherein converting a virtual address for a buffer to a physical address for the buffer is facilitated using the PCIe ATS.

[0345] 73. The computer system of clause 66, wherein the information associated with the one or more buffers in system memory comprises a virtual address and a size for each of the one or more buffers, and wherein the system further comprises means for:

[0346] for each of the one or more buffers,

[0347] using the virtual address for the buffer as a TLB lookup in one or more TLBs associated with the core executing the thread, the TLB lookup identifying at least one of a physical memory address including a hint or a virtual address to physical address (VA-PA) translation including a hint; and

[0348] using the hint to speculatively prefetch buffer data for the buffer.

[0349] 74. The computer system of clause 73, wherein the system further comprises means for:

[0350] converting the virtual address for each of the one or more buffers to a physical address to obtain a current valid physical address; and

[0351] For each of the one or more buffers, verifying the hint for that buffer with the current valid physical address for that buffer.

[0352] 75. The computer system of any of clauses 66-74, wherein the system further comprises means for:

[0353] In response to verifying that the buffer data for the speculative prefetch is the correct buffer data to be processed by the job,

[0354] continuing processing of the job by processing all of the buffer data that was prefetched using one or more functional units of the accelerator.

[0355] While some embodiments have been described with reference to particular implementations, other implementations are possible according to some embodiments. Additionally, the arrangement and / or order of elements or other features illustrated in the figures and / or described herein need not be arranged and / or ordered as illustrated and / or described. Many other arrangements are possible according to some embodiments.

[0356] In each of the apparatuses or systems illustrated in the figures, in some cases, elements can each have the same or different reference labels to indicate that the represented elements can be different and / or similar. However, the elements can be flexible enough to have different implementations and operate with some or all of the systems illustrated or described herein. The various elements illustrated in the figures can be the same or different. It is arbitrary which is called a first element and which is called a second element.

[0357] In the specification and claims, the terms "coupled" and "connected," along with their derivatives, can be used. It should be understood that these terms are not intended as synonyms for each other. Rather, in particular embodiments, "connected" can be used to indicate that two or more elements are in direct physical and / or electrical contact with each other. "Coupled" can mean that two or more elements are in direct physical or electrical contact with each other. However, "coupled" can also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other. Further, "communicatively coupled" means that two or more elements can or can not be in direct contact with each other, but are enabled to communicate with each other. For example, if component A is connected to component B, which is in turn connected to component C, then component A can be communicatively coupled to component C using component B as an intermediary component.

[0358] Embodiments are implementations or examples of the application. References in the specification to "an embodiment", "one embodiment", "some embodiments", or "other embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least some embodiments of the application, but not necessarily all embodiments of the application. The various appearances of "embodiment", "one embodiment", or "some embodiments" do not necessarily all refer to the same embodiments.

[0359] Not all components, features, structures, characteristics, etc. described and shown in the descriptions herein need to be included in a particular embodiment or embodiments of the application. Furthermore, unless otherwise stated, the descriptions and examples are not intended to be exhaustive or to be otherwise limit the application. Indeed, descriptions and examples of the application are intended to be illustrative and not exclusionary. Furthermore, although embodiments of the application have been described and shown, the application is not limited to the described or illustrated embodiments. Numerous changes and modifications can be made herein without departing from the scope of the application. Additionally, where the descriptions and examples concern articles and methods, it is contemplated that the articles and methods include the steps of the application, unless otherwise stated.

[0360] As discussed above, aspects of the embodiments herein can be facilitated by corresponding software and / or firmware components and applications, such as software and / or firmware executed by embedded processors, etc. Accordingly, embodiments of the application can be used as or to support software programs, software modules, firmware, and / or virtual machines that execute on some form of processor, processing core, or embedded logic, a virtual machine running on or otherwise realized or achieved within a non-transitory computer- or machine-readable storage medium. Non-transitory computer- or machine-readable storage media include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, non-transitory computer- or machine-readable storage media include read-only memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc. Content can be directly executable ("object" or "executable" form), source code, or difference code ("delta" or "patch" code). Non-transitory computer- or machine-readable storage media can also include memories stored on or databases accessed via a communication network. Non-transitory computer- or machine-readable storage media can also include devices or products that store content as it is being manufactured and / or shipped to customers, such that the content is included with the device or product when it is purchased and / or used for the first time. Accordingly, transporting a device or product with content stored thereon, or providing content for download over a communication medium, can be understood to provide an article of manufacture including a non-transitory computer- or machine-readable storage medium having such content as described herein.

[0361] The various components described above as being called processes, servers, or tools can be means for performing the described functions. The operations and functions performed by various components described herein can be implemented by software running on a processing element, via embedded hardware, or any combination of hardware and software. Such components can be implemented as software modules, hardware modules, special-purpose hardware (e.g., application specific hardware, ASICs, DSPs, etc.), embedded controllers, hardwired circuitry, hardware logic, etc. Software content (e.g., data, instructions, configuration information, etc.) can be provided via an article of manufacture including a non-transitory computer or machine readable storage medium that provides content that represents instructions that are executable by a machine. The content can cause a machine to perform various functions / operations described herein.

[0362] As used herein, a list of items connected by a term such as “at least one of’ can mean any combination of the listed terms. For example, the phrases “at least one of A, B, or C” can mean A; B; C; A and B; A and C; B and C; or A, B, and C.

[0363] Italicized letters such as ‘n’ and ‘m’ in the above DETAILED DESCRIPTION are used to depict integers, and the use of a particular letter is not limited to a particular embodiment. Further, the same letter can be used in different claims to represent different integers or different letters can be used. Further, the use of a particular letter in the DETAILED DESCRIPTION can or can not match the letter used in the claims referring to the same subject matter in the DETAILED DESCRIPTION.

[0364] The above description of the illustrated embodiments of the application (including what is in the Abstract) is not intended to be exhaustive or to be assumed to represent the full scope of the application. Although specific embodiments of the application have been described above, various modifications are possible that fall within the scope of the application. The scope of the application should be determined by the appended claims and their equivalents.

[0365] These modifications can be made in light of the above Specific Embodiments. The terms used in the following claims should not be construed to limit the application to the specific embodiments disclosed in the specification and drawings. Rather, the scope of the application is to be determined entirely by the following claims, which are to be construed in accordance with the principles of claim drafting, regardless of how those claims later might be interpreted during enforcement of the claims by the patent office.

Claims

1. A method executed on a computing platform, the computing platform having a multi-core processor coupled to one or more memory devices, the memory devices including system memory implemented as a shared virtual memory (SVM), the multi-core processor having multiple cores, each core associated with at least one translation lookaside buffer (TLB), the multi-core processor further comprising at least one memory controller operatively coupled to the one or more memory devices via at least one memory channel, the computing platform further comprising one or more SVM-capable accelerators, the SVM-capable accelerators having one or more functional units, the method comprising: enqueueing, by execution of an execution thread on one of the plurality of cores, a request descriptor for an accelerator on the accelerator, the request descriptor defining a job to be performed by the accelerator and including information associated with one or more buffers in the system memory that are accessed by the accelerator during processing of the request descriptor; speculatively prefetching buffer data from the system memory via the accelerator using the information in the request descriptors associated with the one or more buffers; speculatively starting processing of the job by processing at least a portion of the buffer data prefetched using one or more functional units of the accelerator; as well as Verifying whether the speculatively prefetched buffer data is correct buffer data to be processed by the job.

2. The method according to claim 1, wherein The information associated with the one or more buffers in the system memory includes a virtual address, a size, and an address hint for each of the one or more buffers, the method further comprising: For each of the one or more buffers, Buffer data for the buffer is speculatively prefetched using the address hint for the buffer.

3. The method according to claim 2, wherein The address hint for each of the one or more buffers comprises: a physical address of at least a first page of one or more memory pages containing data corresponding to the buffer; or A virtual address to physical address (VA-PA) translation is provided for at least the first of one or more memory pages containing data corresponding to the buffer.

4. The method according to claim 2, wherein The address hint for each of the one or more buffers is obtained by using an instruction set architecture (ISA) instruction in the execution thread executed by the processor core.

5. The method according to claim 4, wherein Execution of the ISA instructions by the processor core causes the processor to: For each of the one or more buffers, The virtual address of the buffer is used as a TLB lookup in one or more TLBs associated with the processor core, the TLB lookup returning a physical memory address including a hint.

6. The method according to claim 1, wherein The information associated with the one or more buffers in system memory includes a virtual address and a size of each of the one or more buffers, the method further comprising: For each of the one or more buffers, using the virtual address of the buffer as a TLB lookup in one or more TLBs associated with the core executing the thread, the TLB lookup identifying at least one of a physical memory address including a hint or a virtual address to physical address (VA-PA) translation including a hint; and Buffer data for the buffer is speculatively prefetched using the hint.

7. The method according to any one of claims 2 to 6, wherein: Also includes: for each of the one or more buffers, translating the virtual address for the buffer into a physical address to obtain a current effective physical address for the buffer; as well as For each of the one or more buffers, the address hint for the buffer is validated using the current valid physical address for the buffer.

8. The method according to claim 7, wherein Converting a virtual address for the buffer to a physical address for the buffer is facilitated by utilizing a Peripheral Component Interconnect Express (PCIe) address translation service.

9. The method according to claim 8, wherein Also includes: The PCIe address translation service is evaluated by simulating a PCIe device.

10. The method according to claim 1, wherein Verifying that the speculatively prefetched buffer data is the correct buffer data to be processed by the job, the method further comprising: Processing of the job continues by processing all of the buffer data prefetched using one or more functional units of the accelerator.

11. A computer system comprising: A multi-core processor, comprising: a plurality of processor cores, each processor core having at least one associated translation lookaside buffer (TLB), each TLB for storing a plurality of TLB entries including virtual address to physical address (VA-PA) translations; an interconnect structure communicatively coupled to each processor core and each TLB associated with each processor core; a memory controller communicatively coupled to the interconnect structure and having one or more memory channels; an input-output (IO) interface communicatively coupled to the interconnect structure; and An IOTLB, the IOTLB being used to store a plurality of TLB entries including VA-PA translations for one or more IO devices; a system memory comprising one or more memory devices, each memory device communicatively coupled to at least one memory channel of the memory controller; an accelerator supporting a shared virtual memory (SVM), the accelerator supporting the shared virtual memory (SVM) having one or more functional units and being embedded on the processor or communicatively coupled to the processor via the IO interface; Wherein, the system is configured as follows: enqueueing, by execution of an execution thread on one of the plurality of cores, a request descriptor for the accelerator on the accelerator, the request descriptor defining a job to be performed by the accelerator and including information associated with one or more buffers in system memory that are accessed by the accelerator during processing of the request descriptor; speculatively prefetching buffer data from the system memory via the accelerator using the information in the request descriptors associated with the one or more buffers; speculatively starting processing of the job by processing at least a portion of the buffer data prefetched using one or more functional units of the accelerator; and Verifying whether the speculatively prefetched buffer data is correct buffer data to be processed by the job.

12. The computer system according to claim 11, wherein: The information associated with the one or more buffers in the system memory includes a virtual address, a size, and an address hint for each of the one or more buffers, wherein the system is further configured to: For each of the one or more buffers, The address hint for the buffer is employed to speculatively prefetch buffer data for the buffer.

13. The computer system of claim 12, wherein: The address hint for each of the one or more buffers comprises: a physical address of at least a first page of one or more memory pages containing data corresponding to the buffer; or A virtual address to physical address (VA-PA) translation is provided for at least the first of one or more memory pages containing data corresponding to the buffer.

14. The computer system according to claim 12 or 13, wherein: Each of the processor cores has an instruction set architecture (ISA), and the address hint for each of the one or more buffers is obtained by using ISA instructions in the execution thread executed by the processor core.

15. The computer system of claim 14, wherein: Execution of the ISA instructions by the processor core causes the processor to: For each of the one or more buffers, The virtual address of the buffer is used as a TLB lookup in one or more TLBs associated with the processor core, the TLB lookup returning a physical memory address including a hint.

16. The computer system of claim 12, wherein: The system is further configured to: For each of the one or more buffers, converting the virtual address for the buffer to a physical address to obtain a current effective physical address for the buffer; and For each of the one or more buffers, the address hint for the buffer is validated using the current valid physical address for the buffer.

17. The computer system of claim 16, wherein: The IO interface includes a Peripheral Component Interconnect Express (PCIe) interface, and the processor includes a PCIe root complex coupled to the interconnect structure and to the PCIe interface, and the processor is configured to support a PCIe Address Translation Service (ATS), and wherein translating a virtual address for a buffer into a physical address for the buffer is facilitated by using the PCIe ATS.

18. The computer system of claim 11, wherein: The information associated with the one or more buffers in system memory includes a virtual address and a size of each of the one or more buffers, and wherein the system is further configured to: For each of the one or more buffers, using the virtual address of the buffer as a TLB lookup in one or more TLBs associated with the core executing the thread, the TLB lookup identifying at least one of a physical memory address including a hint or a virtual address to physical address (VA-PA) translation including a hint; and Buffer data for the buffer is speculatively prefetched using the hint.

19. The computer system of claim 18, wherein: The system is further configured to: converting the virtual address for each of the one or more buffers into a physical address to obtain a current effective physical address; and For each of the one or more buffers, the hint for the buffer is validated using the current valid physical address for the buffer.

20. The computer system of claim 11, wherein: In response to verifying that the speculatively prefetched buffer data is the correct buffer data to be processed by the job, the system is further configured to continue processing of the job by processing all of the buffer data prefetched using one or more functional units of the accelerator.

21. A multi-core processor, comprising: a plurality of processor cores, each processor core having at least one associated translation lookaside buffer (TLB), each TLB for storing a plurality of TLB entries including virtual address to physical address (VA-PA) translations; an interconnect structure communicatively coupled to each processor core and each TLB associated with each processor core; a memory controller communicatively coupled to the interconnect structure and having one or more memory channels; an accelerator communicatively coupled to the interconnect structure, having one or more functional units and at least one of a request queue and a portal, wherein the multi-core processor is configured to be installed in a computer system including a system memory, the system memory including one or more memory devices coupled to the one or more memory channels when the processor is installed in the computer system, the computer system employing a shared virtual memory (SVM) architecture, and wherein each processor core has an instruction set architecture (ISA) including instructions that, when executed, cause the processor to perform the following operations: writing a request descriptor to the accelerator entry or enqueuing a request descriptor into the accelerator request queue, the request descriptor defining a job to be performed by the accelerator and including an address hint for each of one or more buffers in system memory that are accessed by the accelerator during processing of the request descriptor, Wherein the processor employs the address hint for a buffer to speculatively prefetch buffer data corresponding to the buffer from the system memory.

22. The multi-core processor according to claim 21, wherein: The plurality of processors are further configured to: In response to enqueuing the request descriptor into the accelerator request queue, employing the address hint for each of the one or more buffers to speculatively prefetch buffer data for that buffer from the system memory; speculatively starting processing of the job defined by the request descriptor by processing at least a portion of the buffer data prefetched using one or more functional units of the accelerator; and Verifying whether the speculatively prefetched buffer data is correct buffer data to be processed by the job.

23. The multi-core processor according to claim 21 or 22, wherein: The ISA instruction includes: a first operand in a first register for storing the address of the accelerator entry for writing a request descriptor; a second operand in a second register for storing the address of the request descriptor from a user application running in the system memory; and a third operand in a third register for storing the length of the request descriptor.

24. The multi-core processor according to claim 23, wherein: Execution of the ISA instructions by the processor core causes the processor to: extracting data identifying a virtual address of each of the one or more buffers from the request descriptor; For each of the one or more buffers, using the virtual address of the buffer as a TLB lookup in one or more TLBs associated with the processor core, the TLB lookup returning a physical memory address including a hint; and The request descriptor is updated to include the hint.

25. The multi-core processor according to claim 24, wherein: At least one of the buffers includes data stored in a plurality of pages in the system memory, and wherein execution of the ISA instructions by the processor core causes the processor to: The request descriptor is updated for each buffer including data in a plurality of pages stored in the system memory using a physical memory address for each of the plurality of pages.

Images