Apparatus and method for remote display and content protection in a virtualized graphics processing environment

Abstract

Apparatus and methods relating to remote display and content protection in a virtualized graphics processing environment. The apparatus includes a display engine that renders a sequence of video images; an encoder that compresses the sequence to produce a compressed sequence; a network interface controller that transmits the compressed sequence over a network link to a remote display; a buffer pointer register that stores read and write pointers in a frame buffer and a compressed stream buffer; a central processing unit that initializes the read and write pointers and processes the images; the display engine accesses a first write pointer to write a designated location in the frame buffer, the encoder reads from the frame buffer based on a first read pointer value, the encoder writes to the compressed stream buffer based on a second write pointer value, the network interface controller reads from the compressed stream buffer based on a second read pointer value, the first and second write and read pointer values are updated without intervention from the CPU as the display engine writes to the frame buffer, the encoder reads from the frame buffer and writes to the compressed stream buffer, and the network interface controller reads therefrom.

Description

TECHNICAL FIELD

[0001] The present invention relates generally to the field of computer processors. More particularly, the present invention relates to an apparatus and method for remote display and content protection in a virtualized graphics processing environment.

[0002] Description of the Related Art

[0003] Recent advances have been made in the area of graphics processor unit (GPU) virtualization. Virtualized graphics processing environments are used, for example, in media clouds, remote workstations / desktops, interchangeable virtual appliances (IVI), rich client virtualization, and the like. Certain architectures perform full GPU virtualization by trapping and emulation to emulate a full-featured virtual GPU (vGPU) while providing near-native performance by passing graphics memory resources that are critical to performance.

[0004] As the importance of GPUs in servers to support 3D, media, and GPGPU workloads increases, GPU virtualization is becoming more and more prevalent. How to virtualize GPU memory accesses from a virtual machine (VM) is one of the key design factors. GPUs have their own graphics memory: either dedicated video memory or shared system memory. When system memory is used for graphics, guest physical addresses (GPAs) need to be translated to host physical addresses (HPAs) before being accessed by hardware.

[0005] There are multiple ways to perform the translation for a GPU. Some implementations perform the translation by hardware support, but can only pass the GPU to one VM. Another solution is a software approach to build shadow structures for the translation. For example, shadow page tables are employed with certain architectures to implement, such as in the full GPU virtualization solution mentioned above, which can support multiple VMs sharing a physical GPU.

[0006] In some implementations, guest / VM memory pages are backed by host memory pages. A virtual machine monitor (VMM) (sometimes referred to as a "hypervisor") maps from guest physical addresses (PAs) to host PAs using, for example, an extended page table (EPT). A variety of memory sharing techniques can be used, such as kernel same-page merging (KSM) techniques.

[0007] KSM merges pages from multiple VMs that have the same content into a single page with write protection. That is, if a memory page in VM1 (mapped from guest PA1 to host PA1) has the same content as another memory page in VM2 (mapped from guest PA2 to host PA2), then only one host page (e.g., HPA_SH) can be used to back the guest memory. That is, both guest PA1 of VM1 and PA2 of VM2 are mapped to HPA_SH with write protection. This saves memory for the system, and is particularly useful for read-only memory pages of the guests, such as code pages and zero pages. With KSM, once a VM modifies the page content, a copy-on-write (COW) technique can be used to remove the sharing.

[0008] Mediation passes for device performance and sharing in virtualized systems, where a single physical GPU is presented as multiple virtual GPUs to multiple guests with direct DMA, while privileged resource access from the guests is still trapped and emulated. In certain embodiments, each guest can run a native GPU driver, and device DMA goes directly into memory without the need for hypervisor intervention. BRIEF DESCRIPTION OF DRAWINGS

[0009] A better understanding of the present application can be obtained from the following detailed description in conjunction with the following drawings, in which:

[0010] FIG. 1 is a block diagram of an embodiment of a computer system with a processor having one or more processor cores and a graphics processor;

[0011] FIG. 2 is a block diagram of an embodiment of a processor having one or more processor cores, an integrated memory controller, and an integrated graphics processor;

[0012] FIG. 3 is a block diagram of an embodiment of a graphics processor that can be a discrete graphics processing unit, or can be a graphics processor integrated with a number of processing cores;

[0013] FIG. 4 is a block diagram of an embodiment of a graphics processing engine for a graphics processor;

[0014] FIG. 5 is a block diagram of another embodiment of a graphics processor;

[0015] FIG. 6 is a block diagram of thread execution logic including an array of process elements;

[0016] FIG. 7FIG. illustrates a graphics processor execution unit instruction format according to an embodiment;

[0017] FIG. 8 is a block diagram of another embodiment of a graphics processor that includes a graphics pipeline, a media pipeline, a display engine, thread execution logic, and a render output pipeline.

[0018] FIG. 9A is a block diagram illustrating a graphics processor command format according to an embodiment;

[0019] FIG. 9B is a block diagram illustrating a graphics processor command sequence according to an embodiment;

[0020] FIG. 10 illustrates an exemplary graphics software architecture for a data processing system according to an embodiment;

[0021] FIG. 11 illustrates an exemplary IP core development system that can be used to fabricate an integrated circuit to perform operations according to embodiments;

[0022] FIG. 12 illustrates an exemplary system on a chip integrated circuit that can be fabricated using one or more IP cores according to an embodiment;

[0023] FIG. 13 illustrates an exemplary graphics processor of a system on a chip integrated circuit that can be fabricated using one or more IP cores;

[0024] FIG. 14 illustrates an additional exemplary graphics processor of a system on a chip integrated circuit that can be fabricated using one or more IP cores;

[0025] FIG. 15 illustrates an exemplary graphics processing system;

[0026] FIG. 16 illustrates an exemplary architecture for full graphics virtualization;

[0027] FIG. 17 illustrates an exemplary virtualized graphics processing architecture including a virtual graphics processing unit (vGPU);

[0028] FIG. 18 illustrates one embodiment of a virtualization architecture with an IOMMU;

[0029] FIG. 19 illustrates one embodiment in which graphics processing is performed on a server;

[0030] FIG. 20 illustrates one embodiment of a virtualized graphics architecture for automotive implementation;

[0031] FIG. 21 An exemplary allocation of graphical physical addresses and openings for each virtual machine is illustrated.

[0032] FIG. 22 A remote display system is illustrated in accordance with one embodiment of the present application.

[0033] FIG. 23 A method is illustrated in accordance with one embodiment of the present application.

[0034] FIG. 24 is a block diagram that illustrates a computer system configured to implement one or more aspects of the embodiments described herein.

[0035] FIGS. 25A-25D A parallel processor component is illustrated in accordance with an embodiment;

[0036] FIGS. 26A-26B is a block diagram of a graphics multiprocessor in accordance with an embodiment;

[0037] FIGS. 27A-27F An exemplary architecture in which multiple GPUs are communicatively coupled to multiple multi-core processors is illustrated; and

[0038] FIG. 28 A graphics processing pipeline is illustrated in accordance with an embodiment. DETAILED DESCRIPTION

[0039] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the application described below. However, it will be apparent to one skilled in the art that the embodiments of the present application can be practiced without some or all of these specific details. In other instances, well known structures and devices are shown in block diagram form to avoid obscuring the underlying principles of the embodiments of the present application.

[0040] Exemplary graphics processor architecture and data types

[0041] System Overview

[0042] FIG. 1 is a block diagram of a processing system 100 in accordance with an embodiment. In various embodiments, system 100 includes one or more processors 102 and one or more graphics processors 108, and can be a single processor desktop system, a multiprocessor workstation system, or a server system

[0043] Embodiments of system 100 can include or incorporate a server-based gaming platform, a gaming console, including a game and media console, a mobile gaming console, a handheld gaming console, or an online game console. In some embodiments, system 100 is a mobile telephone, a smartphone, a tablet computing device, or a mobile Internet device. Data processing system 100 can also include, be coupled to, or integrated in a wearable device, such as a smartwatch wearable device, a smartglasses device, an augmented reality device, or a virtual reality device. In some embodiments, data processing system 100 is a television or set-top box device having one or more processors 102 and a graphical interface generated by one or more graphics processors 108.

[0044] In some embodiments, the one or more processors 102 each include one or more processor cores 107 for processing instructions that, when executed, perform operations for system and user software. In some embodiments, each of the one or more processor cores 107 is configured to process a specific instruction set 109. In some embodiments, instruction set 109 can facilitate complex instruction set computing (CISC), reduced instruction set computing (RISC), or computing via a very long instruction word (VLIW). Multiple processor cores 107 can each process a different instruction set 109, which can include instructions to facilitate the emulation of other instruction sets. Processor core(s) 107 can also include other processing devices, such as digital signal processors (DSPs).

[0045] In some embodiments, processor 102 includes cache memory 104. Depending upon the architecture, the processor 102 can have a single internal cache or multiple levels of internal cache. In some embodiments, the cache memory is shared among the various components of the processor 102. In some embodiments, the processor 102 also uses an external cache (e.g., a level 3 (L3) cache or last level cache (LLC)) (not shown), which can be shared among the processor cores 107 using known cache coherency techniques. Additionally, a register file 106 is included in the processor 102, which can include different types of registers for storing different types of data (e.g., integer registers, floating point registers, status registers, and instruction pointer registers). Some of the registers can be general registers, while others can be specific to the design of the processor 102.

[0046] In some embodiments, processor 102 is coupled to processor bus 110 for communicating information between processor 102 and other components in system 100, such as address, data, or control signals. In one embodiment, system 100 uses an exemplary 'hub' system architecture including a memory controller hub 116 and an input / output (I / O) controller hub 130. The memory controller hub 116 facilitates communication between memory device and other components in system 100, while the I / O controller hub 130 provides connections between the I / O devices and the other components in the system. In one embodiment, the logic of memory controller hub 116 is integrated within the processor.

[0047] Memory device 120 can be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase- change memory device, or some other memory device having suitable performance to serve as processing memory. In one embodiment, memory device 120 can operate as system memory for system 100, storing data 122 and instructions 121 for use when one or more processors 102 executes an application or process. Memory controller hub 116 is also coupled to an optional external graphics processor 112, which can communicate with one or more graphics processors 108 within the processor(s) 102 through the use of a graphics bus (not illustrated). The external graphics processor 112 can communicate with the memory controller hub 116 through the use of a point-to-point interface 113.

[0048] In some embodiments, ICH 130 allows peripheral components to connect to memory device 120 and processor 102 via a high-speed I / O bus. I / O peripherals include, but are not limited to, an audio controller 146, a firmware interface 128, a wireless transceiver 126 (e.g., Wi-Fi, Bluetooth), a data storage device 124 (e.g., hard disk drive, flash memory, etc.), and a

[0049] FIG. 2is a block diagram of an embodiment of a processor 200 having one or more processor cores 202A-202N, an integrated memory controller 214, and an integrated graphics processor 208. FIG. 2 Those elements of with the same reference numbers (or names) as elements in any other figure herein can operate or function in any manner similar to the manner described elsewhere herein, but are not limited to such. The processor 200 can include additional cores up to and including the additional core 202N represented by the dashed line. Each of the processor cores 202A-202N includes one or more internal cache units 204A-204N. In some embodiments, each processor core can also access one or more shared cache units 206.

[0050] The internal cache units 204A-204N and shared cache units 206 represent a cache memory hierarchy within the processor 200. The cache memory hierarchy can include at least one level of instruction and data caches within each processor core and one or more levels of shared mid-level caches, such as level 2 (L2), level 3 (L3), level 4 (L4), or other levels of caches, with the highest level cache being classified as an LLC before external memory. In some embodiments, cache coherency logic maintains coherency among the cache units 206 and 204A-204N.

[0051] In some embodiments, the processor 200 can also include a set of one or more bus controller units 216 and a system agent core 210. The one or more bus controller units 216 manage a set of peripheral buses, such as one or more Peripheral Component Interconnect buses (e.g., PCI, PCI Express). The system agent core 210 provides management functionality for the various processor components. In some embodiments, the system agent core 210 includes one or more integrated memory controllers 214 to manage access to various external memory devices (not shown).

[0052] In some embodiments, one or more of the processor cores 202A-202N include support for simultaneous multi-threading. In such embodiments, the system agent core 210 includes components to coordinate and operate the cores 202A-202N during multi-threaded processing. Additionally, the system agent core 210 can also include a power control unit (PCU) including logic and components to adjust the power state of the processor cores 202A-202N and the graphics processor 208.

[0053] In some embodiments, the processor 200 additionally includes a graphics processor 208 for performing graphics processing operations. In some embodiments, the graphics processor 208 is coupled to the shared cache unit 206 set and the system agent core 210, which includes one or more integrated memory controllers 214. In some embodiments, a display controller 211 is coupled with the graphics processor 208 for driving graphics processor output to one or more coupled displays. In some embodiments, the display controller 211 can be separate from the graphics processor, or can be integrated within the graphics processor 208 or system agent core 210.

[0054] In some embodiments, a ring-based interconnect unit 212 is used to couple the internal components of the processor 200. However, alternative interconnect units, such as point-to-point interconnects, switch fabrics, or other technologies can be used, as are known in the art. In some embodiments, the graphics processor 208 is coupled with the ring interconnect 212 via an I / O link 213.

[0055] The exemplary I / O link 213 represents at least one of a plurality of categories of multiple I / O interconnects, including a package I / O interconnect that facilitates communication between various processor components and a high performance embedded memory module 218, such as an eDRAM module. In some embodiments, each of the processor cores 202A-202N and the graphics processor 208 use the embedded memory module 218 as a shared last level cache.

[0056] In some embodiments, the processor cores 202A-202N are homogeneous cores executing the same instruction set architecture. In another embodiment, the processor cores 202A-202N are heterogeneous in terms of instruction set architecture (ISA), where one or more of the processor cores 202A-202N execute a first instruction set while at least one of the other cores executes a subset of the first instruction set or a different instruction set. In one embodiment, the processor cores 202A-202N are homogeneous in terms of microarchitecture where one or more of the cores have a relatively high power consumption and one or more power cores have a lower power consumption. Additionally, the processor 200 can be implemented on one or more chips or as a SoC integrated circuit with the illustrated components, among other components.

[0057] FIG. 3is a block diagram of a graphics processor 300 that can be a discrete graphics processing unit, or can be graphics processor integrated with a multiple of processor cores. In some embodiments, the graphics processor communicates via a memory mapped I / O interface to registers on the graphics processor, and utilizes commands placed into the processor memory to execute graphics processing operations. In some embodiments, the graphics processor 300 includes a memory interface 314 to access a memory. The memory interface 314 can be an interface to local memory, one or more internal caches, one or more shared external caches, and / or to system memory.

[0058] In some embodiments, the graphics processor 300 also includes a display controller 302 to drive display output data to a display device 320. The display controller 302 includes hardware for one or more overlay planes for the display and compositing of multiple layers of video or user interface elements. In some embodiments, the graphics processor 300 includes a video codec engine 306 to

[0059] In some embodiments, the graphics processor 300 includes a block image transfer (BLIT) engine 304 to perform two-dimensional (2D) rasterizer operations including, for example, bit-boundary block transfers. However, in one embodiment, 2D graphics operations are performed using one or more components of the graphics-processing engine (GPE) 310. In some embodiments, GPE 310 is a compute engine for performing graphics operations, including three-dimensional (3D) graphics operations and media operations.

[0060] In some embodiments, GPE 310 includes a 3D pipeline 312 for performing 3D operations, such as rendering three-dimensional graphics shapes representing 3D objects and scenes along with the associated texture and lighting effects. The 3D pipeline 312 stages programmable and fixed function elements that perform various tasks within execution threads to execute the 3D graphics operations. While the 3D pipeline 312 can be used to perform media operations, an embodiment of GPE 310 also includes a media pipeline 316 that is specifically used to execute media operations, such as video post-processing and image enhancements.

[0061] In some embodiments, media pipeline 316 includes fixed function or programmable logic for accelerating media operations including video encode, video decode, and / or photo

[0062] In some embodiments, 3D / media subsystem 315 includes logic to execute threads generated by 3D pipeline 312 and media pipeline 316. In one embodiment, the pipelines send thread execution requests to 3D / media subsystem 315, which includes thread dispatch logic to arbitrate and dispatch the requests to available thread execution resources. Execution resources include an array of graphics execution units to process 3D and media threads. In some embodiments, 3D / media subsystem 315 includes one or more internal caches to cache it thread instructions and data. In some embodiments, the subsystem also includes shared memory (including registers and / or addressable memory) to share data between threads and to store output data.

[0063] Graphics Processing Engines

[0064] FIG. 4 Figure 4 is a block diagram of a graphics processing engine 410 of a graphics processor of some embodiments in accordance with some embodiments. In one embodiment, the graphics processing engine (GPE) 410 is a version of the GPE 310 shown and described herein. FIG. 3 FIG. 4 Those elements of Figure 4 having the same reference numbers (or names) as the elements shown in any other figure herein can operate or function in any manner similar to that described elsewhere herein, but are not limited to such. For example, the 3D pipeline 312 and media pipeline 316 of Figure 4 are illustrated as separate entities, but are not required to be implemented as such. For example, and in at least one embodiment, a single media and / or image processor is coupled to the GPE 410. FIG. 3

[0065] ​​In some embodiments, GPE 410 is coupled with or includes a command streamer 403 that provides a command stream to 3D pipeline 312 and / or media pipeline 316. In some embodiments, command streamer 403 is coupled with memory, which can be system memory, or one or more of internal cache memory and shared cache memory. In some embodiments, command streamer 403 receives commands to perform from memory and sends these commands to 3D pipeline 312 and / or media pipeline 316. The commands are instructions to fetch, decode and execute graphics processing operations, such as those defined by a graphics processing language, such as, but not limited to, the OpenGL® or Direct3D® graphics processing languages. In one embodiment, further to fetching, decoding and executing instructions, the ring buffer also includes a batch command buffer that stores batches of multiple commands. The commands for 3D pipeline 312 can further include references to data stored in memory, such as, but not limited to, vertex and geometry data for 3D pipeline 312 and / or image data and memory objects for media pipeline 316. 3D pipeline 312, media pipeline 316, and / or general-purpose processing pipeline 304 process the commands and data to produce results, including, but not limited to, pixel data. These results are output to memory.

[0066] In various embodiments, 3D pipeline 312 can execute one or more shader programs, such as a vertex shader, a geometry shader, a pixel shader, a fragment shader, a compute shader, or other shader programs, by processing instructions and dispatching execution threads to graphics core array 414. Graphics core array 414 provides unified execution resources for all shader programs, shader program types, and / or

[0067] In some embodiments, graphics core array 414 also includes execution logic to perform media functions, such as video and / or image processing. In one embodiment, execution units additionally include fixed function or programmable execution logic to perform FIG. 1 processing operations in parallel with or in conjunction with the general-purpose logic within FIG. 2 processor core(s) 107 of FIG. 1 or cores 202A-202N in FIG. 2.

[0068] Output data generated by threads executing on graphics core array 414 can be output to memory in a unified return buffer (URB) 418. URB 418 can store data for multiple threads. In some embodiments, URB 418 can be used to transmit data between different threads executing on graphics core array 414. In some embodiments, URB 418 can additionally be used for synchronization between threads on graphics core array and fixed function logic within shared function logic 420.

[0069] In some embodiments, graphics core array 414 is scalable such that the array includes a variable number of graphics cores each having a variable number of execution units based on target performance and power levels of the GPE 410. In one embodiment, the execution resources are dynamically scalable such that execution resources can be enabled or disabled as needed.

[0070] Graphics core array 414 is coupled with shared function logic 420, which includes a number of resources shared between graphics cores in the graphics core array. Shared functions within shared function logic 420 are hardware logic units that provide specialized supplemental functionality to graphics core array 414. In various embodiments, shared function logic 420 includes, without limitation, sampler 421, math 422, and inter-thread communication (ITC) 423 logic. Additionally, some embodiments implement one or more caches 425 within shared function logic 420. Shared functions are implemented in cases where demand for a given specialized function is insufficient to include within graphics core array 414. Instead, a single instance of the specialized function is implemented as a standalone entity within shared function logic 420 and shared between execution resources within graphics core array 414. The exact set of functions shared between graphics core array 414 and included within graphics core array 414 varies between embodiments.

[0071] FIG. 5 is a block diagram of another embodiment of a graphics processor 500. FIG. 5 Those elements of having the same reference number (or name) in the figures herein as elements in any other figure herein can operate or function in any manner similar to that described elsewhere herein, but are not limited to such.

[0072] In some embodiments, graphics processor 500 includes a ring interconnect 502, a pipeline front-end 504, a media engine 537, and graphics cores 580A-580N. In some embodiments, ring interconnect 502 couples the graphics processor to other processing units including other graphics processors or one or more general-purpose processor cores. In some embodiments, the graphics processor is one of a plurality of processors integrated within a multi-core processing system.

[0073] In some embodiments, graphics processor 500 receives batches of commands via ring interconnect 502. The incoming commands are interpreted by a command streamer 503 in pipeline front-end 504. In some embodiments, graphics processor 500 includes scalable execution logic to perform 3D geometry processing and media processing via graphics core(s) 580A to 580N. For 3D geometry processing commands, command streamer 503 supplies commands to geometry pipeline 536. For at least some media processing commands, command streamer 503 supplies commands to video front end 534, which couples with media engine 537. In some embodiments, media engine 537 includes a video quality engine (VQE) 530 for video and image post-processing, and a multi-format encode / decode (MFX) 533 engine to

[0074] In some embodiments, graphics processor 500 includes a number of graphics core representations module cores 580A to 580N (sometimes referred to as core tiles), each having a number of sub-cores 550A to 550N, 560A to 560N (sometimes referred to as core sub-tiles). In some embodiments, graphics processor 500 can have any number of graphics cores 580A to 580N. In some embodiments, graphics processor 500 includes a graphics core 580A having at least a first sub-core 550A and a second sub-core 560A. In other embodiments, the graphics processor is a low power processor with a single sub-core (e.g., 550A). In some embodiments, graphics processor 500 includes a number of graphics cores 580A to 580N each including a set of first sub-cores 550A to 550N and a set of second sub-cores 560A to 560N. Each sub-core in the set of first sub-cores 550A to 550N includes at least a first set of execution units 552A to 552N and a media / texture sampler 554A to 554N. Each sub-core in the set of second sub-cores 560A to 560N includes at least a second set of execution units 562A to 562N and a sampler 564A to 564N. In some embodiments, each sub-core 550A to 550N, 560A to 560N shares a set of shared resources 570A to 570N. In some embodiments, the shared resources include shared cache memory and pixel operation logic. Other shared resources can also be included in embodiments of graphics processors.

[0075] Execution Units

[0076] FIG. 6Figure illustrates thread execution logic 600, including an array of processing elements employed in some embodiments of GPEs. FIG. 6 Those elements of 536 having the same reference number (or name) as elements in any other figure herein can operate or function in any manner similar to that described elsewhere herein, but are not limited to such.

[0077] In some embodiments, thread execution logic 600 includes a shader processor 602, a thread dispatcher 604, an instruction cache 606, a scalable execution unit array including a number of execution units 608A-608N, a sampler 610, a data cache 612, and a data port 614. In one embodiment, the scalable execution unit array can dynamically scale by enabling or disabling one or more execution units (e.g., any of execution units 608A, 608B, 608C, 608D, up to 608N-1 and 608N) based on computed workload demands of the application. In one embodiment, the included components are interconnected via an interconnection fabric that links the components. In some embodiments, thread execution logic 600 includes one or more connections to memory (such as system memory or cache memory) via instruction cache 606, data port 614, sampler 610, and one or more of execution units 608A-608N. In some embodiments, each execution unit (e.g., 608A) is a separate programmable general-purpose computing unit capable of executing multiple simultaneous hardware threads with each thread handling multiple data elements in parallel.

[0078] In some embodiments, execution units 608A-608N are primarily for executing shader programs. Shader processor 602 can process various shader programs and dispatch execution threads associated with the shader programs via thread dispatcher 604. In one embodiment, thread dispatcher includes logic to arbitrate thread initiation requests and instance the requested threads on one or more execution units 608A-608N. For example, a geometry pipeline (e.g., of 536) can dispatch a vertex processing, tessellation or geometry shader to thread execution logic 600 (of 536) for processing. FIG. 5 FIG. 6 In some embodiments, thread dispatcher 604 can also process runtime thread generation requests from execution of shader programs.

[0079] ​In some embodiments, execution units 608A-608N support single program multiple instruction (SPIM) with each execution unit 608A-608N supporting a large number of instructions and a deep instruction pipeline. Most instructions are single cycle except for a few more complex instructions. An instruction is typically entered into the execution unit 608A-608N on each cycle of the graphics processor 600. In some embodiments, SPIM is used often in the graphics pipeline during the following stages: vertex setup / classification, vertex shading, fragment shading, and results read back out. In some embodiments, the graphics processing unit 600 supports vertex programs having 2-8 vertex attribute inputs plus vertex constant input, programmable vertex fixed function, programmable vertex shader, programable fragment shader, and programmable fragment fixed function. In some embodiments, the graphics processing unit 600 supports fragment programs having 2-8 fragment attribute inputs plus fragment constant input, programmable fragment fixed function, programmable fragment shader, and programmable fragment shader.

[0080] Each of the execution units 608A-608N operate on arrays of data elements. The number of data elements is the "execution size," or the number of channels that the instruction operates on. The execution channels are the logical units of execution for the data element. The number of channels may be independent of the number of physical Arithmetic Logic Units (ALUs) or Floating Point Units (FPUs) for a particular graphics processor. In some embodiments, execution units 608A-608N support integer and floating-point data types.

[0081] The instruction set of the execution units includes SIMD instructions. Various data elements can be stored in registers and the execution units will process data in single or double precision, depending on the mode setting. The execution units support a wide spectrum of programming models including, but not limited to, multiple thread programming models, data parallel programming models, and image processing for example. All graphics processor 600 resources are available to processors running programs under a programming model. In some embodiments, the memory semantics and access control between graphics processor 600 address spaces are controlled via a coherent virtual address (VA) space, providing a coherent abstraction over individual address spaces.

[0082] One or more internal instruction caches (e.g., 606) are included in the thread execution logic 600 to cache thread instructions for the execution units. In some embodiments, one or more data caches (e.g., 612) are included to cache thread data during execution. In some embodiments, a sampler 610 is included to provide texture sampling for 3D operations and media sampling for media operations. In some embodiments, the sampler 610 includes specialized texture or media sampling functionality to process texture or media data during sampling before the sampled data is provided to the execution units.

[0083] During execution, the graphics and media pipeline sends thread initiation requests to the thread execution logic 600 via thread generation and dispatch logic. Once a set of geometric objects has been processed and rasterized into pixel data, pixel processor logic (e.g., pixel shader logic, fragment shader logic, etc.) within the shader processor 602 is invoked to further calculate output values and cause resulting command information to be written to an output surface (e.g., color buffer, depth buffer, stencil buffer, etc.). In some embodiments, pixel or fragment shaders calculate values of per-vertex attributes that are interpolated across rasterized objects. In some embodiments, the pixel processor logic within the shader processor 602 then executes an application programming interface (API)-supplied pixel or fragment shader program. To execute the shader program, the shader processor 602 dispatches threads to the execution units (e.g., 608A) via the thread dispatcher 604. In some embodiments, the pixel shader 602 uses texture sampling logic in the sampler 610 to access texture data stored in a texture map stored in memory. Arithmetic operations on the texture data and input geometry data calculate pixel color data for each geometric fragment, or discard one or more pixels without further processing.

[0084] In some embodiments, a data port 614 provides a memory access mechanism for the thread execution logic 600 to output processed data to memory for processing on the graphics processor output pipeline. In some embodiments, the data port 614 includes or couples with one or more cache memories (e.g., data cache 612) to cache data for memory access via a data port cache.

[0085] FIG. 7is a block diagram illustrating a graphics processor instruction format 700 according to some embodiments. In one or more embodiments, a graphics processor execution unit supports an instruction set that includes a number of instructions in multiple formats. The solid lined boxes illustrate the format of a typical instruction used by an execution unit in one embodiment and provided for illustrative purposes. The dashed line illustrates an alternate instruction format that can be used by the graphics processor execution unit in some embodiments. In some embodiments, the described and illustrated instruction formats 700 are macroinstructions, with each macroinstruction comprising one or more sub operations for the graphics processor execution unit to perform. In some embodiments, instructions are split into sub operations to enable greater flexibility in programming and / or parallel processing.

[0086] In some embodiments, a graphics processor execution unit natively supports the 128-bit instruction format 710. Some instructions are supported in the 64-bit compressed instruction format 730, which includes one or more 128-bit instructions packed into a 64-bit instruction slot. The 64-bit instruction format 730 is used for some graphics processor instructions, but not all.

[0087] For each format, the instruction opcode 712 defines the operation that the execution unit is to perform. The execution units execute each instruction in parallel across the multiple data elements in each operand. For example, in response to an add instruction, the execution units perform a simultaneous add operation across each color channel for the elements in the two source registers / memory addresses 720 and 722 and stores the result in a destination register / memory address 718. In some embodiments, the execution units execute each instruction in parallel across the elements of two source registers / memory addresses 720 and 722 and store the result in a destination register / memory address 718. In one or more embodiments, the graphics processor is a Mali® graphics processor as offered by ARM Holdings of San Jose, CA when used in conjunction with the ARM® core processing units also offered by ARM Holdings.

[0088] Some execution units, in some embodiments, have multiple data elements named for the data element positions 0, 1, and 2, which are used in some graphics processing operations. In some embodiments, the number of data elements is adjustable based on the number of channels in an operation. For example, one or more channels can be used for multiple vertex attributes, physics connected attributes, or other data. The number of channels used can be 1, 2, 3, 4, 8, 16, 32, and other amounts. The number of channels used can be fixed or programmable for a channel.

[0089] In some embodiments, the 128-bit instruction format 710 includes an access / address mode field 726 that, for example, defines whether a direct register addressing mode or an indirect register addressing mode is used. When the direct register addressing mode is used, the register address(es) for the operand(s) are provided directly by bits in the instruction.

[0090] In some embodiments, the 128-bit instruction format 710 includes an access / address mode field 726 that specifies the address mode and / or access mode for the instruction. In one embodiment, the access mode is used to define the data access alignment for the instruction. Some embodiments support access modes including a 16-byte aligned access mode and a 1-byte aligned access mode, where the byte alignment of the access mode determines the access alignment of the instruction operands. For example, when in a first mode, the instruction can use byte-aligned addressing for source and destination operands, and when in a second mode, the instruction can use 16-byte aligned addressing for all source and destination operands.

[0091] In one embodiment, the address mode portion of the access / address mode field 726 determines whether the instruction uses direct addressing or indirect addressing. When the direct register addressing mode is used, the register address(es) for the operand(s) are provided directly by bits in the instruction. When the indirect register addressing mode is used, the register address(es) for the operand(s) can be calculated based on an address register value and an address immediate field in the instruction.

[0092] In some embodiments, instructions are grouped based on the opcode 712-bit field to simplify opcode decoding 740. For 8-bit opcodes, bits 4, 5, and 6 allow the execution unit to determine the opcode type. The exact opcode grouping shown is exemplary only. In some embodiments, the move and logic opcode group 742 includes data move and logic instructions (e.g., move (mov), compare (cmp)). In some embodiments, the move and logic group 742 shares the five most significant bits (MSBs), where move (mov) instructions use the form 0000xxxxb, while logic instructions use the form 0001xxxxb. The flow control instruction group 744 (e.g., call, jump (jmp)) includes instructions using the form 0010xxxxb (e.g., 0x20). The miscellaneous instruction group 746 includes a mix of instructions, including synchronization instructions (e.g., wait, send) using the form 0011xxxxb (e.g., 0x30). The parallel math instruction group 748 includes component-wise arithmetic instructions (e.g., add, mul) in the form of 0100xxxxb (e.g., 0x40). The parallel math group 748 performs arithmetic operations in parallel across data lanes. The vector math group 750 includes arithmetic instructions (e.g., dp4) in the form of 0101xxxxb (e.g., 0x50). The vector math group performs arithmetic operations on vector operands, such as dot products.

[0093] Graphics Pipelines

[0094] FIG. 8 is a block diagram of another embodiment of a graphics processor 800 . FIG. 8 Those elements having the same reference numbers (or names) as elements in any other figures herein may operate or function in any manner similar to, but not limited to, those described elsewhere herein.

[0095] In some embodiments, graphics processor 800 includes a graphics pipeline 820, a media pipeline 830, a display engine 840, thread execution logic 850, and a render output pipeline 870. In some embodiments, graphics processor 800 is a graphics processor within a multi-core processing system that includes one or more general-purpose processing cores. The graphics processor is controlled by register writes to one or more control registers (not shown) or by commands issued to graphics processor 800 via ring interconnect 802. In some embodiments, ring interconnect 802 couples graphics processor 800 to other processing components, such as other graphics processors or general-purpose processors. Commands from ring interconnect 802 are interpreted by command streamer 803, which supplies instructions to individual components of graphics pipeline 820 or media pipeline 830.

[0096] In some embodiments, the command streamer 803 directs operations of a vertex fetcher 805, which reads vertex data from memory and executes vertex processing commands provided by the command streamer 803. In some embodiments, the vertex fetcher 805 provides vertex data to a vertex shader 807, which performs coordinate space transformation and lighting operations on each vertex. In some embodiments, the vertex fetcher 805 and vertex shader 807 execute vertex processing instructions by dispatching execution threads to execution units 852A-B via thread dispatcher 831.

[0097] In some embodiments, the execution units 852A-B are vector processors having instruction sets specifically configured to perform graphics and media operations. In some embodiments, the execution units 852A-B have attached Ll caches 851, which are dedicated to each array or shared between arrays. The caches can be configured as data caches, instruction caches, or single caches that are partitioned into different regions for data and instructions.

[0098] In some embodiments, the graphics pipeline 820 includes tessellation components for performing hardware-accelerated tessellation of 3D objects. In some embodiments, a programmable hull shader 811 configures tessellation operations. A programmable domain shader 817 provides post-processing of tessellation output. A tessellator 813 operates in the direction of the hull shader 811 and includes specialized logic for generating a detailed set of geometric objects based on a coarse geometric model provided as input to the graphics pipeline 820. In some embodiments, the tessellation components (e.g., hull shader 811, tessellator 813, and domain shader 817) can be bypassed if tessellation is not used.

[0099] In some embodiments, full geometric objects can be processed by a geometry shader 819 via one or more threads dispatched to the execution units 852A-B, or can proceed directly to a clipper 829. In some embodiments, the geometry shader operates on entire geometric objects (rather than vertices or vertex patches as in previous stages of the graphics pipeline). If tessellation is disabled, the geometry shader 819 receives input from the vertex shader 807. In some embodiments, the geometry shader 819 can be programmed by a geometry shader program to perform geometric tessellation when the tessellation unit is disabled.

[0100] A clipper 829 processes vertex data prior to rasterization. The clipper 829 can be a fixed function clipper or a programmable clipper with clip and geometry shader functionality. In some embodiments, the rasterizer and depth test components 873 in the rendering output pipeline 870 dispatch pixel shaders to convert geometry objects into their per-pixel representation. In some embodiments, the pixel shader logic is included in the thread execution logic 850. In some embodiments, an application can bypass the rasterizer and depth test components 873 and access un-rasterized vertex data via the outgoing unit 823.

[0101] The graphics processor 800 has an interconnect bus, interconnect fabric, or some other interconnect mechanism to allow data and messages to be passed between components of the graphics processor 800. In some embodiments, the execution units 852A-852B and associated sub-cores 851, the texture and media samplers 854, and the texture / sampler caches 858 are interconnected via an internal data bus. In some embodiments, the samplers 854, caches 851, 858, and execution units 852A-852B each have separate memory access ports onto the internal data bus to allow fetches and stores from / to system memory (e.g., main memory) without disturbing (e.g., pipeline stalling due to) memory operations by the other components.

[0102] In some embodiments, the rendering output pipeline 870 includes a rasterizer and depth test component 873 that converts based on vertex data into associated pixel-based representations. In some embodiments, the rasterizer logic includes a windower / masker unit for fixed function triangle and line rasterization. An associated render cache 878 and depth cache 879 are also available in some embodiments. Pixel operation components 877 perform pixel-based operations, although in some instances pixel operations associated with 2D operations (e.g., bit block image transfers with blitting) are performed by 2D engine 841, or replaced at display time by display controller 843 using overlapping display planes. In some embodiments shared L3 cache 875 is used for all graphics components to allow for shared data use without needing to go to main system memory.

[0103] In some embodiments, the graphics processor media pipeline 830 includes a media engine 837 and a video front-end 834. In some embodiments, the video front-end 834 receives pipeline commands from the command streamer 803. In some embodiments, the media pipeline 830 includes a separate command streamer. In some embodiments, the video front-end 834 processes media commands before sending the media commands to the media engine 837. In some embodiments, the media engine 837 includes thread spawning functionality to process media instructions in parallel via the thread execution logic 850.

[0104] In some embodiments, graphics processor 800 includes a display engine 840. In some embodiments, display engine 840 is external to processor 800 and coupled to the graphics processor via the ring interconnect 802, or some other interconnect bus or fabric. In some embodiments, display engine 840 includes a 2D engine 841 and a display controller 843. In some embodiments, display engine 840 contains special purpose logic that is configured to operate independently of the 3D pipeline. In some embodiments, display controller 843 is coupled with a display device (not illustrated) that can be a system integrated display device, as in a laptop computer, or an external display device attached via an display device connector.

[0105] In some embodiments, graphics pipeline 820 and media pipeline 830 can be configured to perform operations based on a number of graphics and media programming interfaces and are not specific to any one application programming interface (API). In some embodiments, driver software for a graphics processor will translate API calls that are specific to particular graphics or media libraries into commands that can be processed by the graphics processor. In some embodiments, support is provided for the Open Graphics Library (OpenGL) from the Khronos Group, the Open Computing Language (OpenCL), and / or the Vulkan graphics and compute API. In some embodiments, support can also be provided for the Direct3D library from the Microsoft Corporation. In some embodiments, a combination of these libraries can be supported. Support can also be provided for the Open Source Computer Vision Library (OpenCV). Future APIs that are compatible with the 3D pipeline of the graphics processor will also be supported, if a mapping of the pipeline of the future API to the pipeline of the graphics processor can be made.

[0106] Graphics Pipeline Programming

[0107] FIG. 9A FIG. 9 is a block diagram illustrating a graphics processor command format 900 according to some embodiments. FIG. 9B FIG. 10 is a block diagram illustrating a graphics processor command sequence 910 according to an embodiment. FIG. 9A The solid lined boxes in FIG. 10 illustrate a general graphics processor command sequence that is typically performed by a graphics processor, in concert with the graphics processor command format of FIG. 9. The command sequence includes specific sub-operations FIG. 9A The exemplary graphics processor command format 900 includes fields for identifying a target for the command 902, a command operation code (opcode) 904, and data field(s) for the command 906. Sub-opcodes 905 and a command size 908 are also included in some commands.

[0108] In some embodiments, the client 902 defines a client unit of the graphics device that processes the command data. In some embodiments, the graphics processor command parser examines a client field of each command to determine the further processing to be performed on the command and routes the command data to the appropriate client unit. In some embodiments, a graphics processor client unit includes a memory interface unit, render units, a 2D unit, a 3D unit, and a media unit. Each client unit has a respective processing pipeline to process the commands. Once a command is received by a client unit, the client unit reads the operation code 904 and sub-opcode 905 (if present) to determine the operation to be performed. The client unit uses information in the data field 906 to perform the command. For some commands, an explicit command size 908 is desired to define the size of the command. In some embodiments, the command parser automatically determines the size of at least some of the commands based on the command opcode. In some embodiments, the commands are aligned via multiples of double-words.

[0109] FIG. 9B The flow diagram in FIG. 9 illustrates an exemplary graphics processor command sequence 910. In some embodiments, a software or firmware of a data processing system featuring embodiments of a graphics processor uses a version of the command sequence shown to set up, execute, and terminate a set of graphics operations. A sample command sequence is shown and described for illustrative purposes only. The embodiments are not limited to these particular commands or to this command sequence. Moreover, the described commands can be issued as batch of commands in a command sequence, such that the graphics processor will process the sequence of commands in at least partially simultaneous fashion.

[0110] In some embodiments, the graphics processor command sequence 910 can begin with a pipeline flush command 912 to cause any active graphics pipelines to complete any current pending commands in the pipeline. In some embodiments, the 3D pipeline 922 and media pipeline 924 are not operating simultaneously. The pipeline flush is performed to ensure that active graphics pipelines complete any pending commands. In response to the pipeline flush, the command parser for the graphics processor will stop processing commands until the active draw engine completes the pending operations and the relevant read caches are invalidated. Optionally, any data in the render caches that is marked 'dirty' can be flushed to memory. In some embodiments, the pipeline flush command 912 can be used in conjunction with a pipeline synchronization or before bringing the graphics processor to a low power state.

[0111] In some embodiments, a pipeline selection command 913 is used when the command sequence requires the graphics processor to explicitly switch between pipelines. In some embodiments, only one pipeline selection command 913 is needed in the execution context before issuing the pipeline commands, unless the context is issuing commands to two pipelines. In some embodiments, a pipeline flush clear command 912 is needed immediately before the pipeline switch via the pipeline selection command 913.

[0112] In some embodiments, pipeline control commands 914 configure the graphics pipeline for operation and for programming the 3D pipeline 922 and the media pipeline 924. In some embodiments, the pipeline control commands 914 configure the pipeline state for the active pipeline. In one embodiment, the pipeline control commands 914 are used for pipeline synchronization and for clearing data from one or more cache memories within the active pipeline before processing a batch of commands.

[0113] In some embodiments, commands 916 for return buffer state are used to configure a set of return buffers for a respective pipeline to write data. Some pipeline operations require allocation, selection, or configuration of one or more return buffers in which to write intermediate data during processing. In some embodiments, the graphics processor also uses one or more return buffers to store output data and to perform cross-thread communication. In some embodiments, configuring return buffer state 916 includes selecting the size and number of return buffers for a set of pipeline operations.

[0114] The remaining commands in the command sequence differ based on the active pipeline for operation. Based on the pipeline determination 920, the command sequence is tailored for either the 3D pipeline 922 starting with 3D pipeline state 930, or the media pipeline 924 starting at media pipeline state 940.

[0115] The commands for configuring the 3D pipeline state 930 include 3D state setting commands for vertex buffer state, vertex element state, constant color state, depth buffer state, and other state variables to be configured before processing 3D primitive commands. The values for these commands are determined at least in part based on the particular 3D API in use. In some embodiments, the 3D pipeline state 930 commands can also selectively disable or bypass certain pipeline elements if those elements will not be used.

[0116] In some embodiments, 3D primitive 932 commands are used to submit 3D primitives to be processed by the 3D pipeline. Commands and associated parameters passed to the graphics processor via a 3D primitive 932 command are forwarded to a vertex fetch function in the graphics pipeline. The vertex fetch function uses the 3D primitive 932 command data to generate a number of vertex data structures. The vertex data structures are stored in one or more vertex buffers. In some embodiments, 3D primitive 932 commands are used to perform vertex operations on 3D primitives via a vertex shader. To process the vertex shader, the 3D pipeline 922 dispatches shader execution threads to the graphics processor execution units.

[0117] In some embodiments, 3D pipeline 922 is triggered via an execute 934 command or an event. In some embodiments, a register write triggers command execution. In some embodiments, execution is triggered via a 'go' or 'kick' command in the command sequence. In one embodiment, a pipeline synchronization command is used to trigger command execution in order to flush the command sequence with no-ops through the graphics pipeline. The 3D pipeline will perform geometry processing for 3D primitives. Once the processing is completed, the resulting geometry is rasterized and the pixel engine shades the resulting pixels. For these operations, additional commands can also be included to control the pixel shading and pixel back end operations.

[0118] In some embodiments, when performing media operations, graphics processor command sequence 910 follows media pipeline 924 path. In general, the specific use and manner of programming for media pipeline 924 depends on the media or compute operations to be performed. In media decode processes, specific media decode operations can be offloaded to the media pipeline. In some embodiments, the media pipeline can also be bypassed and media decode can be performed in whole or in part using resources provided by one or more general purpose processing cores. In one embodiment, the media pipeline also includes elements for general purpose graphics processor unit (GPGPU) operations, where the graphics processor is used to perform SIMD vector operations using a compute shader program that is not explicitly related to the rendering of a graphics primitive.

[0119] In some embodiments, the media pipeline 924 is configured in a similar manner as the 3D pipeline 922. A set of commands to configure the media pipeline state 940 is dispatched or placed into the command queue, prior to the media object command 942. In some embodiments, the commands 940 for the media pipeline state include data to configure the media pipeline elements that will be used to process the media object. This includes data to configure the video decode and video encode logic within the media pipeline, such as the encoding or decoding format. In some embodiments, the commands 940 for the media pipeline state also support the use of one or more pointers to "indirect" state elements that contain a batch of state settings.

[0120] In some embodiments, the media object command 942 supplies a pointer to a media object for processing by the media pipeline. The media object includes a memory buffer that contains video data to be processed. In some embodiments, all of the media pipeline state must be valid prior to the media object command 942 being issued. Once the pipeline state is configured and the media object command 942 is queued, the media pipeline 924 is triggered via an execute 944 command or equivalent execution event (e.g., register write). The output from the media pipeline 924 can then be post-processed by operations provided by the 3D pipeline 922 or the media pipeline 924. In some embodiments, GPGPU operations are configured and executed in a similar manner as media operations.

[0121] Graphics Software Architecture

[0122] FIG. 10 FIGURE 1 illustrates an exemplary data processing system 1000, according to some embodiments. In some embodiments, the data processing system 1000 includes a 3D graphics application 1010, an operating system 1020, and at least one processor 1030. In some embodiments, the processor 1030 includes a graphics processor 1032 and one or more general-purpose processor cores 1034. The graphics application 1010 and the operating system 1020 each execute in the system memory 1050 of the data processing system.

[0123] In some embodiments, the 3D graphics application 1010 contains one or more shader programs including shader instructions 1012. The shader language instructions can be in a high-level shader language, such as the High-Level Shader Language (HLSL) or the OpenGL Shader Language (GLSL). The application also includes executable instructions 1014 in a machine language suitable for execution by the general- purpose processor cores 1034. The application also includes graphics objects defined by vertex data 1016.

[0124] In some embodiments, operating system 1020 is a Microsoft® Windows® operating system from Microsoft Corporation, a proprietary UNIX-like operating system, or an open source UNIX-like operating system using a Linux kernel variant. Operating system 1020 can support a graphics API 1022, such as a Direct3D API, an OpenGL API, or a Vulkan API. When the Direct3D API is in use, operating system 1020 uses a front-end shader compiler 1024 to compile any shader instructions 1012 written in HLSL into a lower-level shader language. The compilation can be a just-in-time (JIT) compilation or the application can perform the shader pre- compilation. In some embodiments, high-level shaders are compiled into low-level shaders in the process of compiling a 3D graphics application 1010. In some embodiments, shader instructions 1012 are provided in an intermediate form, such as a version of the Standard Portable Intermediate Representation (SPIR) used by the Vulkan API.

[0125] In some embodiments, user mode graphics driver 1026 includes a back-end shader compiler 1027 to translate shader instructions 1012 into representations specific to hardware. When using an OpenGL API, shader instructions 1012 in GLSL high-level language are passed to user mode graphics driver 1026 for compilation. In some embodiments, user mode graphics driver 1026 uses operating system kernel mode functions 1028 to communicate with kernel mode graphics driver 1029. In some embodiments, kernel mode graphics driver 1029 communicates with graphics processor 1032 to dispatch

[0126] IP Core Implementation

[0127] One or more aspects of at least one embodiment can be implemented by representative code stored on a machine-readable medium which represents and / or defines logic within an integrated circuit such as a processor. For example, a machine-readable medium can include instructions that represent various logic within the processor. When read by a machine, the instructions can cause the machine to fabricate the logic to perform the techniques described herein. Such representations, known as "IP cores," are reusable units of logic for an integrated circuit that can be stored on a tangible, machine-readable medium as hardware models, simulated models, or other implementations. The computer readable medium can be a machine readable storage on which is stored software / simulations / or executable instructions that cause machine to fabricate logic to perform techniques described herein. The software / software simulations that cause machines to fabricate logic to perform techniques described herein can be stored as hardware descriptions in a hardware description language on a tangible, machine-readable medium (e.g., a computer readable storage media). Specifically, the computer readable medium does not include propagated signals. Accordingly, the computer readable medium can be a machine readable storage on which is stored software, simulations, or executable instructions that cause a machine to fabricate logic to perform techniques described herein.

[0128] FIG. 11 is a block diagram illustrating an IP core development system 1100 that can be used to fabricate integrated circuits employed in implementations described herein. The IP core development system 1100 can be used to generate modular, re-usable designs that can be incorporated into a larger design or used to construct an entire integrated circuit (e.g., an SOC integrated circuit). A design facility 1130 can employ a high-level programming language (e.g., C / C++) to generate a software simulation 1110 of the IP core design. The software simulation 1110 can be used to design, test, and verify the

[0129] The RTL design 1115 or equivalent can be further synthesized, using the design facility, into a hardware model 1120, which can be in a hardware description language (HDL) or some other representation of physical design data. The HDL can be further simulated or tested to verify the IP core design. The IP core design can be stored in non-volatile memory 1140 (e.g., hard disk, flash memory, or any non-volatile storage medium) for delivery to a third party fabrication facility 1165. Alternatively, the IP core design can be transmitted (e.g., via the Internet) over a wired 1150 or wireless 1160 connection. The fabrication facility 1165 can then fabricate an integrated circuit based at least in part on the IP core design. The fabricated integrated circuit can be configured to perform operations in accordance with at least one embodiment described herein.

[0130] Exemplary System on a Chip Integrated Circuit

[0131] FIGS. 12-14 FIG. 1 illustrates an exemplary integrated circuit and associated graphics processor that can be fabricated using one or more IP cores in accordance with various embodiments described herein. Other logic and circuitry can also be included, including additional graphics processors / cores, peripheral interface controllers, or general purpose processor cores.

[0132] FIG. 12is a block diagram illustrating an exemplary system on a chip integrated circuit 1200 that can be fabricated using one or more IP cores, according to an embodiment. The exemplary integrated circuit 1200 includes one or more application processors 1205 (e.g., CPUs), at least one graphics processor 1210, and can additionally include an image processor 1215 and / or a video processor 1220, any of which can be a modular IP core from the same or multiple different design facilities. The integrated circuit 1200 includes peripheral or bus logic including USB controllers 1225, UART controllers 1230, SPI / SDIO controllers 1235, and I2S / I2C controllers 1240. The integrated circuit can additionally include a display controller 1245 that is coupled with one or more of a high-definition multimedia interface (HDMI) controller 1250 and a mobile industry processor interface (MIPI) display interface 1255. Storage can be provided by a flash memory subsystem 1260 including flash memory and a flash memory controller. Memory interfaces can be provided via a memory controller 1265 to access SDRAM or SRAM memory devices. The integrated circuit additionally includes an embedded security engine 1270 in some embodiments.

[0133] FIG. 13 is a block diagram illustrating an exemplary graphics processor 1310 of a system on a chip integrated circuit that can be fabricated using one or more IP cores, according to an embodiment. The graphics processor 1310 can be a variant of the graphics processor 1210 of FIG. 12 . The graphics processor 1310 includes a vertex processor 1305 and one or more fragment processor(s) 1315A-1315N (e.g., 1315A, 1315B, 1315C, 1315D, up to 1315N-1, and 1315N). The graphics processor 1310 can execute different shader programs via separate logic for vertex shader programs, which are executed by the vertex processor 1305, and / or geometry and / or fragment shader programs, which are executed by the one or more fragment processor(s) 1315A-1315N. The vertex processor 1305 is optimized to execute operations for vertex shader programs, which can generate vertices and connectivity, per-fraction, and / or vertex attributes. The one or more fragment processor(s) 1315A-1315N generate a finished image from vertex data, per-fragment, and / or the like. The graphics processor 1310 can further include a rasterizer and / or tile unit 1320 to perform rasterization operations and / or tile-based processing functions. The graphics processor 1310 can further include one or more memory download units 1325 to perform memory read and write operations over a bus. The memory download unit 1325 can perform reads from and writes to memory storage devices and inputs data received from the bus.

[0134] In addition, the graphics processor 1310 also includes one or more memory management units (MMU) 1320A-1320B, cache memory 1325A-1325B, and circuit interconnects 1330A-1330B. One or more MMU(s) 1320A-1320B provide for virtual to physical address mapping, for read and write translation lookaside buffer (TLB) operations, and for cache control operations. One or more MMU(s) 1320A-1320B can be synchronized with other MMUs within the system, including one or more MMUs associated with application processor(s) 1205, image processor 1215, and / or video processor 1220, such that each processor 1205-1220 can participate in a shared or unified virtual memory system. FIG. 12 The one or more circuit interconnects 1330A-1330B enable the graphics processor 1310 to interface with other IP cores within the SoC, either via an internal bus, as shown, or via a direct connection. In either case, the circuit interconnects 1330A-1330B include serial or other types of connection that can be faster than traditional bus- based connections.

[0135] FIG. 14 is a block diagram illustrating an additional exemplary graphics processor 1410 that can be fabricated using one or more IP cores, according to embodiments. The graphics processor 1410 can be a variant of the graphics processor 1210 of FIG. 12 The graphics processor 1410 includes one or more MMU(s) 1320A-1320B, cache memory 1325A-1325B, and circuit interconnects 1330A-1330B, which can be shared between FIG. 13 the integrated circuit 1300 of

[0136] Graphics processor 1410 includes one or more shader cores 1415A-1415N (e.g., 1415A, 1415B, 1415C, 1415D, 1415E, 1415F, up to 1315N-1 and 1315N) that provide a unified shader core architecture, in which a single core or type or core can execute all types of programmable shader code, including shader program code to implement vertex shaders, fragment shaders, and / or compute shaders. The exact number of shader cores present can vary in different embodiments and implementations. Additionally, graphics processor 1410 also includes an inter-core task manager 1405 that acts as a thread dispatcher and task manager for dispatching execution threads to one or more of shader cores 1415A-1415N and to speed up graphics processing by pre-processing tasks for a graphics pipeline, such as the vertex shading stage, and to pre-fetch tasks based on one or more tasks in the graphics pipeline.

[0137] Example graphics virtualization architecture

[0138] Some embodiments of the present invention are implemented on a platform that utilizes full graphics processor unit (GPU) virtualization. As such, an overview of the GPU virtualization technology employed in one embodiment of the present invention is provided below, followed by a detailed description of the apparatus and method for mode-driven page table shadowing.

[0139] One embodiment of the present invention employs a full GPU virtualization environment that runs native graphics drivers in the guest and implements mediated passes for good performance, scalability, and security isolation between guests. This embodiment provides each virtual machine (VM) with a virtual full-featured GPU that can directly access performance-critical resources in most cases without intervention from the hypervisor, while trapping and emulating privileged operations from the guest at minimal cost. In one embodiment, a virtual GPU (vGPU) with full GPU features is presented to each VM. In most cases, the VM can directly access performance-critical resources without intervention from the hypervisor, while trapping and emulating privileged operations from the guest to provide security isolation between VMs. Each quantum switches the vGPU context to share the physical GPU among multiple VMs.

[0140] FIG. 15A high-level system architecture is illustrated in which embodiments of the present application can be implemented, including a graphics processing unit (GPU) 1500, a central processing unit (CPU) 1520, and a system memory 1510 shared between the GPU 1500 and the CPU 1520. A render engine 1502 fetches GPU commands from a command buffer 1512 in the system memory 1510 to accelerate graphics rendering using a variety of different features. A display engine 1504 fetches pixel data from a frame buffer 1514 and then sends the pixel data to an external monitor for display.

[0141] Certain architectures use the system memory 1510 as graphics memory, while other GPUs can use on-die memory. The system memory 1510 can be mapped into multiple virtual address spaces by a GPU page table 1506. A 2 GB global virtual address space, referred to as global graphics memory, is accessible from both the GPU 1500 and the CPU 1520, mapped by a global page table. A local graphics memory space is supported in the form of multiple 2 GB local virtual address spaces, but is limited to access from the render engine 1502 by a local page table. The global graphics memory is mostly the frame buffer 1514, but is also used as the command buffer 1512. A large amount of data access to the local graphics memory occurs while hardware acceleration is in progress. GPUs with on-die memory employ a similar page table mechanism.

[0142] In one embodiment, the CPU 1520 programs the GPU 1500 through GPU-specific commands in a producer-consumer model, as shown in FIG. 15 According to high-level programming APIs such as OpenGL and DirectX, a graphics driver programs GPU commands into the command buffer 1512, including a primary buffer and batch buffers. The GPU 1500 then fetches and executes the commands. The primary buffer (ring buffer) can link together other batch buffers. The terms "primary buffer" and "ring buffer" are used interchangeably hereinafter. Batch buffers are used to deliver the majority of commands (up to ~98%) for each programming model. Register tuples (head, tail) are used to control the ring buffer. In one embodiment, the CPU 1520 submits commands to the GPU 1500 by updating the tail, while the GPU 1500 fetches commands from the head and then notifies the CPU 1520 after the commands have completed execution by updating the head.

[0143] As described above, one embodiment of the present application is implemented in a full GPU virtualization platform with mediated pass-through. Thus, each VM is presented with a full-featured GPU to run native graphics drivers inside the VM. However, there are significant challenges in the following three areas: (1) the complexity of virtualizing the entire complex modern GPU, (2) performance due to multiple VMs sharing the GPU, and (3) complete security isolation between VMs without any compromise.

[0144] FIG. 16 A GPU virtualization architecture according to one embodiment of the present application is illustrated, including a hypervisor 1610 running on a GPU 1600, a privileged virtual machine (VM) 1620, and one or more user VMs 1631-1632. A virtualization stub module 1611 running in the hypervisor 1610 extends memory management to include an extended page table (EPT) 1614 for the user VMs 1631-1632, and a privileged virtual memory management unit (PVMMU) 1612 for the privileged VM 1620 to implement a trap-and-pass-through policy. In one embodiment, each VM 1620, 1631-1632 runs a native graphics driver 1628 that can directly access performance critical resources of the frame buffer and command buffer using resource partitioning as described below. To protect privileged resources, i.e., I / O registers and PTEs, respective accesses from the graphics drivers 1628 in the user VMs 1631-1632 and the privileged VM 1620 are trapped and forwarded to a virtualization mediator 1622 in the privileged VM 1620 for emulation. In one embodiment, as illustrated, the virtualization mediator 1622 uses hypercalls to access the physical GPU 1600.

[0145] In addition, in one embodiment, the virtualization mediator 1622 implements a GPU scheduler 1626 running concurrently with a CPU scheduler 1616 in the hypervisor 1610 to share the physical GPU 1600 among the VMs 1631-1632. One embodiment uses the physical GPU 1600 to directly execute all commands submitted from the VMs, thus it avoids the complexity of emulating the rendering engine, which is the most complex part inside the GPU. At the same time, resource pass-through of the frame buffer and command buffer minimizes the intervention of the hypervisor 1610 on CPU accesses, while the GPU scheduler 1626 guarantees that each VM quantum is used for direct GPU execution. Thus, the illustrated embodiment achieves good performance when sharing the GPU among multiple VMs.

[0146] In one embodiment, the virtualization stub 1611 selectively traps or passes through guest access of certain GPU resources. The virtualization stub 1611 manipulates EPT 1614 entries to selectively present or hide certain address ranges to user VMs 1631-1632, while using reserved bits PTEs in the PV MMU 1612 for privileged VM 1620 for selectively trapping or passing through guest access to certain address ranges. In both cases, peripheral input / output (PIO) accesses are trapped. All trapped accesses are forwarded to the virtualization mediator 1622 for emulation, while the virtualization mediator 1611 uses hypercalls to access the physical GPU 1600.

[0147] As mentioned above, in one embodiment, the virtualization mediator 1622 emulates a virtual GPU (vGPU) 1624 for privileged resource access, and performs context switching between vGPUs 1624. Meanwhile, the privileged VM 1620 graphics driver 1628 is used to initialize the physical device and manage power. One embodiment employs a flexible publish model by implementing the virtualization mediator 1622 as a kernel module in the privileged VM 1620 for mitigating the binding between the virtualization mediator 1622 and the hypervisor 1610.

[0148] The separate CPU / GPU scheduling mechanism is implemented via the CPU scheduler 1616 and the GPU scheduler 1626. This is done because the cost of a GPU context switch can be more than 1000 times the cost of a CPU context switch (e.g., ~700 us vs. ~300 ns). Additionally, the number of CPU cores can differ from the number of GPU cores in a computer system. Therefore, in one embodiment, the GPU scheduler 1626 is implemented separately from the existing CPU scheduler 1616. The separate scheduling mechanism results in the need for concurrent access to resources from both the CPU and the GPU. For example, when the CPU is accessing the graphics memory of VMl 1631, the GPU can be concurrently accessing the graphics memory of VM2 1632.

[0149] As discussed above, in one embodiment, a native graphics driver 1628 is executed inside each VM 1620, 1631-1632, which directly accesses a portion of performance critical resources through privileged operations emulated by the virtualization mediator 1622. The separate scheduling mechanism results in the resource partitioning design described below. To better support resource partitioning, one embodiment reserves memory mapped I / O (MMIO) register windows to convey resource partitioning information to the VMs.

[0150] In one embodiment, the location and definition of virt_info has been incorporated into the hardware specification as a virtualization extension, so the graphics driver 1628 natively handles the extension, and future GPU generations adhere to the specification for backward compatibility.

[0151] While illustrated in FIG. 16 as separate components, in one embodiment, the privileged VM 1620 including the virtualization mediator 1622 (and its vGPU instances 1624 and GPU scheduler 1626) is implemented as a module within the hypervisor 1610.

[0152] In one embodiment, the virtualization mediator 1622 manages the vGPU 1624 for all VMs by trapping and emulating privileged operations. The virtualization mediator 1622 handles physical GPU interrupts, and can generate virtual interrupts to the designated VMs 1631-1632. For example, a physical completion interrupt for a command execution can trigger a virtual completion interrupt, and is delivered to the rendering owner. The idea of emulating a vGPU instance for each semantic is simple; however, the implementation involves a great deal of engineering effort and deep understanding of the GPU 1600. For example, some graphics drivers can access approximately 700 I / O registers.

[0153] In one embodiment, the GPU scheduler 1626 implements a coarse-grained quality of service (QoS) policy. A particular time quantum is chosen as the time slice for each VM 1631-1632 to share the GPU 1600 resources. For example, in one embodiment, a time quantum of 16 ms is chosen as the scheduling time slice because this value results in a low perception of image changes by humans. This relatively large quantum is also chosen because the cost of a GPU context switch is more than 1000 times the cost of a CPU context switch, so the quantum cannot be as small as the time slice in the CPU scheduler 1616. Commands from the VMs 1631-1632 are continuously submitted to the GPU 1600 until the guest / VM exhausts its time slice. In one embodiment, the GPU scheduler 1626 waits for the guest ring buffer to become idle before switching, because most GPUs today are non-preemptible, which can impact fairness. To minimize the waiting overhead, a coarse-grained flow control mechanism can be implemented by tracking command submission to guarantee that the accumulated commands are within a certain limit at any time. Thus, the time drift between the allocated time slice and the execution time is relatively small compared to a large quantum, so a coarse-grained QoS policy is implemented.

[0154] In one embodiment, at render context switch, when switching between rendering engines of vGPUs 1624, internal pipeline state and I / O register state are saved and restored, and cache / TLB flushes are performed. Internal pipeline state is not visible to the CPU, but can be saved and restored by GPU commands. Saving / restoring I / O register state can be achieved by reading / writing a list of registers in the render context. Internal caches and translation lookaside buffers (TLBs) included in modern GPUs for accelerating data access and address translation must be flushed using commands at the render context switch to guarantee isolation and correctness. Steps for switching contexts in one embodiment are: 1) save current I / O state, 2) flush current context, 3) use additional commands to save current context, 4) use additional commands to restore new context, and 5) restore I / O state of new context.

[0155] As mentioned, one embodiment uses a dedicated ring buffer to carry additional GPU commands. The (audited) guest ring buffer can be reused for execution, but it is not safe to insert commands directly into the guest ring buffer because the CPU can continue to queue more commands, causing the contents to be overwritten. To avoid contention, one embodiment switches from the guest ring buffer to its own dedicated ring buffer. At the end of the context switch, the embodiment switches from the dedicated ring buffer to the guest ring buffer of the new VM.

[0156] One embodiment reuses the privileged VM 1620 graphics driver to initialize the display engine, and then manages the display engine to display different VM frame buffers.

[0157] When two vGPUs 1624 have the same resolution, only the frame buffer location is switched. For different resolutions, the privileged VM can use hardware scalers, a common feature in modern GPUs, to automatically scale the resolution. Both techniques take only a few milliseconds. In many cases, display management can not be needed, for example when a VM is not displayed on a physical display (e.g., when the VM is hosted on a remote server).

[0158] As FIG. 16 One embodiment passes access to frame buffers and command buffers to accelerate performance critical operations from VMs 1631-1632, as illustrated in FIG. 17. For a global graphics memory space of 2 GB size, graphics memory resource partitioning and address space augmentation techniques can be employed. For local graphics memory spaces, each also has a size of 2 GB, since local graphics memory is only accessible by GPU 1600, each VM local graphics memory can be implemented by a render context switch.

[0159] As mentioned, one embodiment partitions the global graphics memory between VMs 1631-1632. As explained above, the separate CPU / GPU scheduling mechanisms require that the CPU and GPU can access the global graphics memory of different VMs simultaneously, so each VM must be ready to present using its own resources at all times, resulting in a resource partitioning approach to the global graphics memory.

[0160] FIG. 17 FIG. illustrates additional details for one embodiment of a graphics virtualization architecture 1700, including multiple VMs (e.g., VM 1730 and VM 1740) managed by a hypervisor 1710, including access to the full array of GPU features in GPU 1720. In embodiments, hypervisor 1710 can enable VM 1730 or VM 1740 to use graphics memory and other GPU resources for GPU virtualization. Based on the GPU virtualization techniques, one or more virtual GPUs (vGPUs) (e.g., vGPUs 1760A and 1760B) can access the full functionality provided by GPU 1720 hardware. In various embodiments, hypervisor 1710 can track, manage the resources and lifecycle of vGPUs 1760A and 1760B as described herein.

[0161] In some embodiments, vGPUs 1760A-B can include virtual GPU devices presented to VMs 1730, 1740 and can be used to interact with native GPU drivers (e.g., as described above with respect to FIG. 16 VM 1730 or VM 1740 can then access the full array of GPU features and use the virtual GPU devices in vGPUs 1760A-B to access virtual graphics processors. For example, once VM 1730 is trapped in hypervisor 1710, hypervisor 1710 can manipulate a vGPU instance (e.g., vGPU 1760A) and determine whether VM 1730 can access the virtual GPU devices in vGPU 1760A. vGPU contexts can be switched per quantum or event. In some embodiments, context switching can occur per GPU rendering engine (such as 3D rendering engine 1722 or bitblitter rendering engine 1724). Regularly switching allows multiple VMs to share a physical GPU in a manner that is transparent to the workloads of the VMs.

[0162] GPU virtualization can take various forms. In some embodiments, device pass-through can be utilized to enable a VM 1730, in which the entire GPU 1720 is presented to the VM 1730 as if they are directly connected. Much like a single central processing unit (CPU) core can be assigned for the exclusive use of the VM 1730, a GPU 1720 can be assigned for the exclusive use of the VM 1730 (e.g., even for a limited time). Another virtualization model is time-sharing, in which a GPU 1720 or a portion thereof can be shared in a multiplexed fashion by multiple VMs (e.g., VM 1730 and VM 1740). In other embodiments, device 1700 can also use other GPU virtualization models. In various embodiments, graphics memory associated with GPU 1720 can be partitioned and allocated to vGPUs 1760A-B in hypervisor 1710.

[0163] In various embodiments, a graphics translation table (GTT) can be used by a VM or GPU 1720 to map graphics processor memory to system memory or to convert GPU virtual addresses to physical addresses. In some embodiments, hypervisor 1710 can manage graphics memory mappings via a shadow GTT, which can be maintained in a vGPU instance (e.g., vGPU 1760A). In various embodiments, each VM can have a corresponding shadow GTT for maintaining mappings between graphics memory addresses and physical memory addresses (e.g., machine memory addresses in a virtualized environment). In some embodiments, a shadow GTT can be shared and maintain mappings for multiple VMs. In some embodiments, each VM 1730 or VM 1740 can include both a per-process GTT and a global GTT.

[0164] In some embodiments, device 1700 can use system memory as graphics memory. System memory can be mapped into multiple virtual address spaces by a GPU page table. Device 1700 can support a global graphics memory space and a per-process graphics memory address space. The global graphics memory space can be a virtual address space (e.g., 2 GB) mapped by a global graphics translation table (GGTT). The lower portion of this address space is sometimes referred to as the aperture, which is accessible from both GPU 1720 and CPUs (not shown). The upper portion of this address space is referred to as the high graphics memory space or hidden graphics memory space, which can only be used by GPU 1720. In various embodiments, a shadow global graphics translation table (SGGTT) can be used by VM 1730, VM 1740, hypervisor 1710, or GPU 1720 to convert graphics memory addresses to corresponding system memory addresses based on the global memory address space.

[0165] With full GPU virtualization, a static global graphics memory space partitioning scheme can face scalability issues. For example, for a 2 GB global graphics memory space, a first 512 megabyte (MB) virtual address space can be reserved for an aperture, and the rest (1536 MB) can become a high graphics memory space. With the static global graphics memory space partitioning scheme, each VM enabled for full GPU virtualization can be allocated with a 128 MB aperture and a 384 MB high graphics memory space. Thus, the 2 GB global graphics memory space can only accommodate up to four VMs.

[0166] In addition to the scalability issue, a VM with limited graphics memory space can also suffer from performance degradation. Sometimes, when a media application extensively uses GPU media hardware acceleration, severe performance degradation can be observed in some media intensive workloads of the media application. As an example, at least 40 MB of graphics memory can be needed to decode a 1080p H.264 / Advanced Video Coding (AVC) bitstream for one channel. Thus, decoding 1080p H264 / AVC bitstreams for 10 channels can require at least 400 MB of graphics memory space. Meanwhile, some graphics memory space can need to be reserved for surface composition / color conversion, switching display frame buffers during decoding, etc. In this case, 512 MB of graphics memory space per VM can not be enough for the VM to run multi-video encoding or decoding.

[0167] In embodiments, device 100 can implement GPU graphics memory overcommitment with an on-demand SGGTT. In some embodiments, hypervisor 1710 can build an SGGTT on demand, which can include all pending translations for graphics memory virtual addresses of owner VMs from different GPU components.

[0168] In embodiments, at least one VM managed by hypervisor 1710 can be allocated with more than a statically partitioned global graphics memory address space and memory. In some embodiments, at least one VM managed by hypervisor 1710 can be allocated with or have access to the entire high graphics memory address space. In some embodiments, at least one VM managed by hypervisor 1710 can be allocated with or have access to the entire graphics memory address space.

[0169] The hypervisor / VMM 1710 can use the command parser 1718 to detect the potential memory working set of the GPU rendering engine for commands submitted by the VM 1730 or the VM 1740. In embodiments, the VM 1730 can have a corresponding command buffer (not shown) for holding commands from the 3D workload 1732 or the media workload 1734. Similarly, the VM 1740 can have a corresponding command buffer (not shown) for holding commands from the 3D workload 1742 or the media workload 1744. In other embodiments, the VM 1730 or the VM 1740 can have other types of graphics workloads.

[0170] In embodiments, the command parser 1718 can scan the commands from the VM and determine whether the commands contain memory operands. If so, the command parser can read the relevant graphics memory space mapping from the VM's GTT, for example, and then write it into the workload-specific portion of the SGGTT. After scanning the entire command buffer of a workload, the SGGTT holding the memory address space mapping associated with this workload can be generated or updated. In addition, by scanning the pending commands from the VM 1730 or the VM 1740, the command parser 1718 can also improve the security of GPU operations, such as by mitigating malicious operations.

[0171] In some embodiments, one SGGTT can be generated to hold translations for all workloads from all VMs. In some embodiments, one SGGTT can be generated to hold translations for all workloads from, for example, only one VM. The workload-specific SGGTT portion can be built on-demand by the command parser 1718 for holding translations for a particular workload, e.g., the 3D workload 1732 from the VM 1730 or the media workload 1744 from the VM 1740. In some embodiments, the command parser 1718 can insert the SGGTT into the SGGTT queue 1714 and the corresponding workload into the workload queue 1716.

[0172] In some embodiments, the GPU scheduler 1712 can build such on-demand SGGTT at execution time. A particular hardware engine can use only a small portion of the graphics memory address space allocated to the VM 1730 at execution time, and GPU context switches rarely occur. To take advantage of such GPU features, the hypervisor 1710 can use the SGGTT of the VM 1730 to hold only the executed and pending translations for the individual GPU components, rather than the entire portion of the global graphics memory address space allocated to the VM 1730.

[0173] The GPU scheduler 1712 for the GPU 1720 can be separate from the scheduler of the CPU in the device 1700. In some embodiments, to exploit hardware parallelism, the GPU scheduler 1712 can schedule workloads for different GPU engines (e.g., 3D rendering engine 1722, bitblitter rendering engine 1724, video command streamer (VCS) rendering engine 1726, and video enhanced command streamer (VECS) rendering engine 1728) separately. For example, a VM 1730 can be 3D intensive, and a 3D workload 1732 can need to be scheduled to the 3D rendering engine 1722 at one time. Meanwhile, a VM 1740 can be media intensive, and a media workload 1744 can need to be scheduled to the VCS rendering engine 1726 and / or the VECS rendering engine 1728. In this case, the GPU scheduler 1712 can schedule the 3D workload 1732 from the VM 1730 and the media workload 1744 from the VM 1740 separately.

[0174] In embodiments, the GPU scheduler 1712 can track SGGTTs in execution used by the corresponding rendering engines in the GPU 1720. In this case, the hypervisor 1710 can maintain a per-rendering engine SGGTT for tracking graphics memory working sets in all executions in the corresponding rendering engine. In some embodiments, the hypervisor 1710 can maintain a single SGGTT for tracking graphics memory working sets in all executions for all rendering engines. In some embodiments, such tracking can be based on separate execution SGGTT queues (not shown). In some embodiments, such tracking can be based on tags (e.g., using a registry) on the SGGTT queue 1714. In some embodiments, such tracking can be based on tags (e.g., using a registry) on the workload queue 1716.

[0175] In the scheduling process, the GPU scheduler 1712 can check SGGTTs from the SGGTT queue 1714 for a pending workload from the workload queue 1716 to schedule. In some embodiments, to schedule the next VM for a particular rendering engine, the GPU scheduler 1712 can check whether the graphics memory working set for the particular workload used by the VM of that rendering engine conflicts with the graphics memory working set executed or to be executed by this rendering engine. In other embodiments, such a conflict check can extend to checking by all other rendering engines with executed or to be executed graphics memory working sets. In embodiments, such a conflict check can be based on the respective SGGTTs in the SGGTT queue 1714 or based on the SGGTT maintained by the hypervisor 1710 for tracking graphics memory working sets in all executions in the corresponding rendering engine as discussed above.

[0176] If there is no conflict, GPU scheduler 1712 can integrate the in-flight and pending graphics memory worksets together. In some embodiments, the resulting SGGTTs for the in-flight and pending graphics memory worksets for a particular rendering engine can also be generated and stored, for example, in SGGTT queue 1714 or other data store. In some embodiments, the resulting SGGTTs for the in-flight and pending graphics memory worksets for all rendering engines associated with one VM can also be generated and stored if the graphics memory addresses for all of these workloads do not conflict with each other.

[0177] Before submitting the selected VM workload to GPU 1720, hypervisor 1710 can write the corresponding SGGTT pages into GPU 1720 (e.g., to graphics translation table 1750). Thus, hypervisor 1710 can enable this workload to execute with the correct mappings in the global graphics memory space. In embodiments, all of these translation entries can be written into graphics translation table 1750, to lower memory space 1754 or upper memory space 1752. In some embodiments, graphics translation table 1750 can contain a separate table per VM to hold these translation entries. In other embodiments, graphics translation table 1750 can also contain a separate table per rendering engine to accommodate these translation entries. In embodiments, graphics translation table 1750 can contain at least the pending graphics memory addresses.

[0178] However, if there is a conflict determined by GPU scheduler 1712, then GPU scheduler 1712 can delay the scheduling of this VM and instead attempt to schedule another workload for the same or a different VM. In some embodiments, this conflict can be detected if two or more VMs can attempt to use the same graphics memory address (e.g., for the same rendering engine or two different rendering engines). In some embodiments, GPU scheduler 1712 can change the scheduler policy to avoid selecting one or more of the rendering engines that can conflict with each other. In some embodiments, GPU scheduler 1712 can stall the execution of a hardware engine to mitigate the conflict.

[0179] In some embodiments, memory overcommit as discussed herein can coexist with a static global graphics memory space partitioning scheme. As an example, the hole of lower memory space 1754 can still be used for static partitioning of all VMs. The high bits graphics memory space in upper memory space 1752 can be used for the memory overcommit scheme. Compared to the static global graphics memory space partitioning scheme, the memory overcommit scheme for GPU virtualization can enable each VM to use the entire high bits graphics memory space in upper memory space 1752, which can allow some applications within each VM to use larger graphics memory space for improved performance.

[0180] With the static global graphics memory space partitioning scheme, a VM that initially requires a large portion of the memory can only use a small portion at runtime, while other VMs can be in a memory-starved state. With memory overcommit, the hypervisor can allocate memory to VMs on demand, and the saved memory can be used to support more VMs. With SGGTT-based memory overcommit, only the graphics memory space used by the workloads to be executed can be allocated at runtime, which saves graphics memory space and supports more VMs to access GPU 1720.

[0181] Current architectures enable hosting of GPU workloads in cloud and datacenter environments. Full GPU virtualization is one of the fundamental enabling technologies used in GPU clouds. In full GPU virtualization, a virtual machine monitor (VMM), in particular a virtual GPU (vGPU) driver, captures and emulates guest access to privileged GPU resources for security and multiplexing, while CPU accesses performance critical resources such as graphics memory. GPU commands, once submitted, are executed directly by the GPU without VMM intervention. As a result, near-native performance is achieved.

[0182] Current systems use system memory for GPU engines to access a global graphics translation table (GGTT) and / or a per-process graphics translation table (PPGTT) to translate from GPU graphics memory addresses to system memory addresses. A shadowing mechanism can be used for the GGTT / PPGTT of a guest GPU page table.

[0183] The VMM can use a shadow PPGTT that is synchronized with the guest PPGTT. The guest PPGTT is write-protected so that the shadow PPGTT can be continuously synchronized with the guest PPGTT by capturing and emulating the guest's PPGTT modifications. Currently, the GGTT for each vGPU is shadowed and partitioned between each VM, and the PPGTT is shadowed and each VM is shadowed (e.g., on a per-process basis). Since the GGTT PDE table resides in the PCI barO MMIO range, the shadowing for the GGTT page tables is simple. However, the shadowing for the PPGTT relies on write protection to the guest PPGTT page tables, and traditional shadow page tables are very complex (and thus also vulnerable) and inefficient. For example, the performance overhead of the CPU shadow page tables in current architectures is ~30%. Thus, in some of these systems, an enlightened shadow page table is used that modifies the guest graphics driver to cooperatively identify pages for the page tables and / or modifies the guest graphics driver when they are freed.

[0184] Embodiments of the present invention include a memory management unit (MMU), such as an I / O memory management unit (IOMMU), to remap from a GPN (guest page number) mapped by the guest PPGTT to a HPN (host page number) without relying on inefficient / complex shadow PPGTTs. At the same time, one embodiment preserves a global shadow GGTT page table for address expansion. These techniques are generally referred to as hybrid layer address mapping (HLAM).

[0185] By default, IOMMU cannot be used for some mediation pass-through architectures because multiple VMs can use just a single second level translation. One embodiment of the present invention solves this problem using the following techniques:

[0186] 1. Use the IOMMU to do two levels of translation without a shadow PPGTT. Specifically, in one embodiment, the GPU translates from a graphics memory address (GM_ADDR) to a GPN, and the IOMMU translates from the GPN to a HPN, instead of a shadow PPGTT that translates from the GM_ADDR to the HPN, where write protection is applied to the guest PPGTT.

[0187] 2. In one embodiment, the IOMMU page tables are managed per VM, and are switched when the vGPU is switched (or can be partially switched). That is, when a VM / vGPU is scheduled, the IOMMU page tables for the corresponding VM are loaded.

[0188] 3. However, in one embodiment the addresses of the GGTT mapping are shared, and since the vCPUs can access the addresses of the GGTT mapping (such as, for example, the openings), this global shadow GGTT must remain valid even when the vGPU of this VM is not scheduled. As such, one embodiment of the invention uses a hybrid layer address translation that preserves the global shadow GGTT but uses the guest PPGTT directly.

[0189] 4. In one embodiment, the GPN address space is partitioned to move the GPN addresses of the GGTT mapping (which become inputs to the IOMMU like GPNs) to a dedicated address range. This can be achieved by trapping and emulating the GGTT page tables. In one embodiment, the GPNs are modified from the GGTT with large offsets to avoid overlapping with the PPGTT in the IOMMU mapping.

[0190] FIG. 18 An architecture employed in one embodiment is illustrated, in which an IOMMU 1830 is enabled for device virtualization. The illustrated architecture includes two VMs 1801, 1811 executing on a hypervisor / VMM 1820 (although the underlying principles of the invention can be implemented with any number of VMs). Each VM 1801, 1811 includes a driver 1802, 1812 (e.g., native graphics driver) that manages a guest PPGTT and GGTT 1803, 1813, respectively. The illustrated IOMMU 1830 includes a HLAM module 1831 for implementing the hybrid layer address mapping techniques described herein. Note that in this embodiment, there is no shadow PPGTT.

[0191] In one embodiment, the GPN-to-HPN translation tables 1833 for the entire guest VM (guest VM 1811 in the example) are prepared in the IOMMU mapping, and each vGPU switch triggers an IOMMU page table swap. That is, when each VM 1801, 1811 is scheduled, its corresponding GPN-to-HPN translation table 1833 is swapped. In one embodiment, the HLAM 1831 distinguishes between GGTT GPNs and PPGTT GPNs and modifies the GGTT GPNs so that they do not overlap with the PPGTT GPNs when performing lookups in the translation table 1833. Specifically, in one embodiment, virtual GPN generation logic 1832 converts the GGTT GPNs to virtual GPNs, which are then used to perform lookups in the translation table 1833 to identify the corresponding HPN.

[0192] In one embodiment, virtual GPNs are generated by moving the GGTT by a designated (possibly large) offset to ensure that the mapped addresses do not overlap / conflict with the PPGTT GPNs. Additionally, in one embodiment, since the CPU can access GGTT mapped addresses at any time (e.g., open), the global shadow GGTT will always be valid and maintained in the IOMMU mapping 1833 for each VM.

[0193] In one embodiment, the hybrid layer address mapping 1831 solution partitions the IOMMU address range into two parts: a lower part reserved for PPGTT GPN to HPN translation, and an upper part reserved for GGTT virtual GPN to HPN translation. Since the GPNs are provided by the VM / guest 1811, the GPNs should be within the range of guest memory size. In one embodiment, the guest PPGTT page tables remain unchanged, and all GPNs from the PPGTT are sent directly to the graphics translation hardware / IOMMU by the workload execution. However, in one embodiment, MMIO reads / writes from the guest VM are trapped, and GGTT page table changes are captured and altered as described herein (e.g., a large offset is added to the GPNs so as to ensure no overlap with the PPGTT mapping in the IOMMU).

[0194] Remote virtualized graphics processing

[0195] In some embodiments of the invention, a server performs graphics virtualization, virtualizing physical GPUs on behalf of clients and running graphics applications. FIG. 19 One such embodiment is illustrated, in which two clients 1901-1902 are connected to a server 1930 over a network 1910, such as the Internet and / or a private network. The server 1930 implements a virtualized graphics environment in which a hypervisor 1960 allocates resources from one or more physical GPUs 1938, presenting the resources as virtual GPUs 1934-1935 to VMs / applications 1932-1933. The graphics processing resources can be allocated according to a resource allocation policy 1961, which can cause the hypervisor 1960 to allocate resources based on the requirements of the applications 1932-1933 (e.g., higher performance graphics applications require more resources), user accounts associated with the applications 1932-1933 (e.g., certain users pay extra for higher performance), and / or current load on the system. The allocated GPU resources can include, for example, groups of graphics processing engines such as 3D engines, bitblit engines, execution units, and media engines, to name a few.

[0196] In one embodiment, the user of each client 1901-1902 has an account on a service hosted by server(s) 1930. For example, the service can provide a subscription service to provide users with remote access to online applications 1932-1933, such as video games, productivity applications, and multi-player virtual reality applications. In one embodiment, the applications are remotely executed on a virtual machine in response to user input 1907-1908 from clients 1901-1902. Although not illustrated in FIG. 19, one or more CPUs can also be virtualized and used to execute applications 1932-1933, with graphics processing operations offloaded to vGPUs 1934-1935. FIG. 19

[0197] In one embodiment, in response to execution of graphics operations, vGPUs 1934-1935 generate a sequence of image frames. For example, in a first-person shooter game, a user can specify input 1907 to move a character around a fantasy world. In one embodiment, the resulting images are compressed (e.g., by compression circuitry / logic, not shown) and streamed over network 1910 to clients 1901-1902. In one implementation, a video compression algorithm such as H.261 can be used; however, a variety of different compression techniques can be used. Decoders 1905-1906 decode the incoming video stream and then render it on respective displays 1903-1904 of clients 1901-1902.

[0198] Using the system illustrated in FIG. 19, high-performance graphics processing resources such as GPU 1938 can be allocated to different clients of a subscription service. In an online gaming implementation, for example, server 1930 can host a new video game when the new video game is released. The video game program code is then executed in a virtualized environment, and the resulting video frames are compressed and streamed to each client 1901-1902. Clients 1901-1902 in this architecture do not need to have substantial graphics processing resources. For example, even a relatively low-power smartphone or tablet with decoders 1905-1906 will be able to decompress the video stream. Thus, the latest graphics-intensive video games can be played on any type of client that is able to compress video. While video games are described as one possible implementation, the underlying principles of the application can be used for any form of application that requires graphics processing resources (e.g., graphics design applications, interactive and non-interactive ray-tracing applications, productivity software, video editing software, etc.). FIG. 19 Virtualized display support

[0199]

[0200] ​​In one embodiment, a display engine has various pipelines that can be combined to render each image frame. In a virtualized environment, for example, there can be multiple VMs, such as VM0 and VM1. The operating systems (OS) in VM 0 and VM 1 can each create multiple windows in memory. For example, two windows on each of VM 0-1 can be designated as VM 0 - W 1 (window 1 on virtual machine 0), VM 0 - W 2 (window 2 on virtual machine 0), VM 1 - W 1 (window 1 on virtual machine 1), and VM 1 - W 2. Once generated, these windows can be composited together in one image frame. Normally, VMs are isolated, so it is difficult to get them to combine data to form an integrated image frame. The normal operation is to have a hypervisor VM, VM 0, that has access to all of the windows of VM 1. In order to display anything, VM 1 then needs to make a request to VM 0. The renderer will then run the workload as if it were running the workload of VM 0, including all of the necessary buffers containing all of the windows of VM 1.

[0201] This same environment is being converted into a car usage model, where there is one piece of graphics hardware that runs both a cluster of gauges (e.g., speedometer) and an in-vehicle infotainment (IVI) display (e.g., navigation, radio, etc.). It is generally undesirable to have the same operating system implement both the cluster of gauges and the IVI display, so current implementations use one OS to perform the main cluster of gauges and then perform a separate VM for the IVI display. The reason for the isolation is because (1) the cluster of gauges needs to have high reliability and real-time operation and (2) IVI applications will update more frequently (e.g., install new maps for navigation).

[0202] In one embodiment of the invention, the display processing is split so that each pipeline and the assets associated with it are partitioned between VMs using 4 kB page boundaries. Then one part of the display surface is passed to VM 0 (e.g., cluster of gauges) and another part of the display surface is passed to VM 1. In one implementation, CPU page tables are used to perform the split to form two different pipelines: one part of the page tables is assigned to IVI / VM 1 and another part is assigned to the cluster of gauges / VM 0. The end result is that both VM 0 and VM 1 see their own assigned portion of system memory. Configuring the system in this way allows VM 0 to operate completely unimpeded by VM 1, which has no control over its own display buffers. Nonetheless, there is a set of designated assets that are utilized by the pipelines that are still controlled by VM 0, such as urgency and / or priority associated with memory requests. These are set globally by VM 0, which can be designated as a trusted VM.

[0203] FIG. 20An example mapping is illustrated that includes a trusted VM0 2011 for instrument cluster display pipe 2031 and VM1 2012 for IVI display pipe 2032. In the illustrated example, a first operating system 2021 is executed in VM0 2011 with two windows W1 and W2. Similarly, a second operating system 2022 is executed on VM1 with two windows W1 and W2. In addition to CPU page tables, as illustrated in FIG. 21 In a virtualized graphics environment, a global guest translation table (GTT) 2100 can be partitioned between VM0 and VM1, as illustrated in

[0204] Low latency composition and transmission

[0205] Current flows for remote / wireless display are driven by software executing on the CPU. For example, the CPU must individually (1) instruct the rendering pipeline to render the next frame, (2) instruct the display hardware to display the results in the frame buffer (potentially compositing multiple windows), (3) instruct the encoder to encode / compress the frame buffer; and (4) instruct the network interface to stream the frame buffer to the destination (e.g., remote display).

[0206] Embodiments of the present invention include additional circuitry and logic to reduce the need for continuous CPU intervention during video frame composition and transmission. In particular, the CPU can set up each functional block from one end to the other and then disengage for a period of time. Each stage in the graphics processing pipeline then communicates with the previous stage and the next stage to perform remote display processing without CPU intervention.

[0207] To reduce the CPU interactions needed to render frames for remote / wireless display, one embodiment of the present invention provides a mechanism for coordination between the various remote / wireless display processing stages. In particular, inter-stage messaging is performed and registers are updated at each stage to indicate the current head (H) and / or tail (T) buffer pointers for the shared buffer (e.g., indicating the buffer location that each stage is currently processing).

[0208] As FIG. 22As illustrated in FIG. 22, one embodiment of a remote display system includes a display engine 2211 that includes 2D / 3D image rendering circuitry for rendering 2D or 3D images for one or more windows (e.g., within an operating system executing on a VM). The display engine 2211 can also include multi-plane compositing circuitry for combining multiple rendered windows in a final image frame. The final image frame is stored within one or more frame buffers 2221 that are input to an encoder 2212. In one embodiment, the encoder 2212 implements an image / video compression algorithm (e.g., H.264 or similar) to compress a sequence of frames and generate a compressed video stream that it stores in a compressed stream buffer 2221. Finally, a network interface controller (NIC) 2213 streams each frame from the compressed stream buffer 2232 to a remote display 2214. In one embodiment, the NIC 2213 streams each frame over a wireless link to the remote display 2214, implementing a wireless display technology such as WiDi.

[0209] In one embodiment, a set of registers 2221-2223 is initially updated by the CPU 2204 to store pointers to each buffer 2231-2232 and any other information needed for each stage 2211-2213 to communicate with each other, such as by identifying memory regions through which messages are exchanged. Although the registers 2221-2223 are illustrated within the particular processing stages in FIG. 22, a general or specialized register file accessible by all processing stages 2211-2213 can be used. FIG. 22

[0210] In one embodiment, the display engine registers 2221 are updated to identify a point in the frame buffer 2231 where the display engine 2211 will begin writing the first frame, and the encoder registers 2222 are updated to include a first pointer to a location in the frame buffer 2231 from which the encoder 2212 will read and a second pointer to a location in the compressed stream buffer 2232 to which the encoder 2212 will write. Similarly, the NIC registers 2223 are updated with a pointer to a location in the compressed stream buffer 2232 where the NIC 2213 will begin reading the next compressed frame.

[0211] ​In operation, once the CPU writes the initial values ​​to registers 2221-2223, it no longer needs to interact with the various image processing stages 2211-2213 (as in current CPU-managed systems). Specifically, when display engine 2211 writes to frame buffer 2231, it updates the current write position in register 2221, sometimes referred to as the head pointer (H). Once it has reached a specified point in writing to frame buffer 2231 (e.g., 16 pixels or a macroblock wide), it notifies encoder 2212, which begins reading from the position stored in its register 2222, continuously updating the most recently read position. It may also send notifications to display engine 2211 to indicate its status (i.e., the current position it is reading from).

[0212] Similarly, as the encoder 2212 writes compressed frames to the compressed stream buffer 2232, it may update its write pointer (or "head" pointer) within its register 2222 and periodically notify the NIC 2213 of its progress (e.g., when it has reached a certain threshold and periodically thereafter). The NIC 2213, in turn, updates its register 2223 to include the current read pointer value as it reads from the compressed stream buffer 2232 and streams the data over the network to the remote display 2214 (which decompresses and displays the image data).

[0213] In one embodiment, using FIG. 22 The architecture shown in implements a virtualized execution environment with a VMM / hypervisor and one or more virtual machines. In this embodiment, threads from each virtual machine can utilize the display engine 2211, encoder 2212 and NIC 2213 and the buffer management technology described herein.

[0214] In this manner, through intercommunication between the various processing stages 2211 - 2213 , the CPU is relieved of the responsibility of coordinating between the various stages 2211 - 2213 as they perform their respective operations.

[0215] FIG. 23 The middle figure shows a method according to one embodiment. The method can be implemented on the above-mentioned GPU and processor architectures, but is not limited to any particular architecture.

[0216] At 2301, to initiate a remote display sequence, the CPU writes buffer pointers for the display engine, encoder, and network interface controller (NIC), and then disengages at 2302 once the buffer pointers are written. As used herein, disengagement means that the CPU does not need to participate in image frame rendering, encoding, and transmitting image frames. Depending on the implementation, disengagement can occur for a single image frame or multiple image frames.

[0217] At 2303, the display engine begins rendering an image frame and writing the results to the first buffer, updating the display engine write pointer as it writes. In one embodiment, the display engine write pointer is stored in a register accessible by both the display engine and the encoder. At 2304, when the display engine write pointer has reached a certain threshold (e.g., when the width of an image frame capable of being compressed, such as a macroblock, has been stored), the display engine can notify the encoder. Alternatively, the encoder can periodically check the write pointer to determine the current buffer position being written. At 2305, the encoder begins reading from the first buffer as indicated by the encoder read pointer, encoding the image data and updating the read pointer as it continues. At 2306, the encoder writes the encoded portion of the image frame to the second buffer as indicated by the encoder write pointer, continuing to update the encoder write pointer as data is written.

[0218] At 2307, when the encoder write pointer reaches a threshold, such as a block of data that can be packaged and transmitted by the NIC, the encoder notifies the NIC. Alternatively, the NIC can periodically check the encoder's write pointer to determine when to begin reading. At 2308, the NIC begins reading the encoded image data from the second buffer as indicated by the NIC read pointer, updating the read pointer as the encoded data is transmitted over the network connection to the remote display, and updating the read pointer as data is transmitted.

[0219] At 2309, the display engine, encoder, and NIC continue to process image frames, updating the read / write pointers and providing notifications to each other until the image frame has been rendered, encoded, and transmitted. The process moves to the next image frame at 2310. Depending on the implementation, the process can then return to 2301, where the CPU initiates the process by writing the buffer pointer for the next frame. Alternatively, the process can return to 2303 and continue to the next image frame without CPU intervention.

[0220] In some embodiments, a graphics processing unit (GPU) is communicatively coupled to the host / processor cores to accelerate graphics operations, machine learning operations, pattern analysis operations, and various general purpose GPU (GPGPU) functions. The GPU can be communicatively coupled to the host processor / cores over a bus or another interconnect (e.g., a high-speed interconnect such as PCIe or NVLink). In other embodiments, the GPU can be integrated on the same package or chip as the cores and communicatively coupled to the cores through an internal processor bus / interconnect (i.e., internal to the package or chip). Regardless of the manner in which the GPU is connected to the processor cores, the processor cores can allocate work to the GPU in the form of sequences of commands / instructions contained in a work descriptor. The GPU then uses dedicated circuitry for processing these commands / instructions in accordance with the GPU's native instruction set.

[0221] In the following description, numerous specific details are set forth to provide a more thorough explanation. However, it will be apparent to one skilled in the art that the embodiments described herein can be practiced without one or more of these specific details. In other instances, well-known features have not been described in order to avoid obscuring the details of the present embodiments.

[0222] System Overview

[0223] FIG. 24 FIG. 24 is a block diagram illustrating a computing system 2400 configured to implement one or more aspects of the embodiments described herein. The computing system 2400 includes a processing subsystem 2401 having one or more processor(s) 2402 and a system memory 2404, which are communicatively coupled via an interconnection path 2405, which can include a memory bus, a memory controller, a peripheral bus, a peripheral controller, a local bus, a point-to-point connection, or another suitable connection. The interconnection path 2405 can implement multiple unidirectional or bidirectional data paths between the processing subsystem 2401 and the system memory 2404. The interconnection path 2405 can also include one or more bridges, controllers, or other components that are used to

[0224] In one embodiment, processing subsystem 2401 includes one or more parallel processor(s) 2412 coupled to memory hub 2405 via a bus or other communication link 2413. Communication link 2413 can be one of any number of standards-based communication links, or proprietary communication links such as a high-speed implementation of the PCI Express interface. In one embodiment, one or more parallel processor(s) 2412 form a computationally-intensive, parallel, or vector processing system that can include a number of processor cores, each with its own memory, cache, bus controller, or processor core la nding. In one embodiment, one or more parallel processor(s) 2412 form a graphics processing subsystem that can output pixels to one or more display device(s) 241 OA coupled via I / O hub 2407. One or more parallel processor(s) 2412 can also include a display controller and display interface (not shown) to enable a direct connection to one or more display device(s) 2410B.

[0225] Within I / O subsystem 2411, system storage unit 2414 can connect to I / O hub 2407 to provide storage

[0226] Computing system 2400 can include other components not explicitly shown, including USB or other port connections, optical storage drives, video capture devices, etc., which can also be connected to I / O hub 2407. FIG. 24 Communication paths interconnecting various components can use any suitable protocols, such as those discussed above, and / or any other suitable communication interface and / or protocol.

[0227] In one embodiment, one or more parallel processors 2412 incorporate circuitry optimized for graphics and video processing, including, for example, video output circuitry, and the circuitry constitutes a graphics processing unit (GPU). In another embodiment, one or more parallel processors 2412 incorporate circuitry optimized for general use applications

[0228] It should be appreciated that the computing system 2400 illustrated herein is illustrative and not limiting. Variations and modifications are possible. The connection topology, including the number and arrangement of bridges, the number and

[0229] Some of the particular components shown herein are optional and can not be included in all implementations of the computing system 2400. For example, any number of add-in cards or peripheral devices can be supported, or some components might be omitted. Additionally, some architectures can use different terminology for components similar to those described herein. For example, in some architectures, the memory hub 2405 might be called a north bridge, and the I / O hub 2407 might be called a south bridge. FIG. 24

[0230] ​FIG. 25A A parallel processor 2500 is shown in accordance with an embodiment. Various components of the parallel processor 2500 can be implemented using one or more integrated circuits, such as programmable processor(s), application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The illustrated parallel processor 2500 is one variant of one or more parallel processors 2412 shown in FIG. 24. FIG. 24 The parallel processor 2500 includes a parallel processing unit 2502. The parallel processing unit includes an I / O unit 2504 that enables communication with other devices, including other instances of the parallel processing unit 2502. The I / O unit 2504 can be directly connected to the other devices. In one embodiment, the I / O unit 2504 is connected to the other devices via use of a hub or switch interface, such as the memory hub 2405. Connections between the memory hub 2405 and the I / O unit 2504 form a communication link 2513. Within the parallel processing unit 2502, the I / O unit 2504 is connected to a host interface 2506, which communicates with the main memory 2404 and which mirrors the functionality of the hub 2405, and with a crossbar switch 2516, which allows the parallel processing unit 2502 to communicate with the

[0231] When the host interface 2506 receives a command buffer from the host via the I / O unit 2504, the host interface 2506 can direct the work operations for executing the commands to a front end 2508. In one embodiment, the front end 2508 is coupled with a scheduler 2510 that is configured to distribute the commands or other work items to a processing cluster array 2512. In one embodiment, the scheduler 2510 ensures that the processing cluster array 2512 is correctly configured and that tasks are distributed in a manner that efficiently uses the processing cluster array 2512. In one embodiment, the scheduler 2510 is implemented via firmware logic executing on a microcontroller. The microcontroller implemented scheduler 2510 can be configured to perform complex scheduling and work distribution operations, such as prioritizing tasks based on the state of the processing cluster array 2512, designating one or more processing clusters 2514 to complete tasks, and so on. In one embodiment, the host software can prove a workload for scheduling on the processing cluster array 2512 via one of a number of graphics processing doorbells. The workload can then be automatically distributed by the scheduler 2510 logic within the scheduler microcontroller in the processing cluster array 2512.

[0232] When the host interface 2506 receives a command buffer from the host via the I / O unit 2504, the host interface 2506 can direct the work operations for executing the commands to a front end 2508. In one embodiment, the front end 2508 is coupled with a scheduler 2510 that is configured to distribute the commands or other work items to a processing cluster array 2512. In one embodiment, the scheduler 2510 ensures that the processing cluster array 2512 is correctly configured and that tasks are distributed in a manner that efficiently uses the processing cluster array 2512. In one embodiment, the scheduler 2510 is implemented via firmware logic executing on a microcontroller. The microcontroller implemented scheduler 2510 can be configured to perform complex scheduling and work distribution operations, such as prioritizing tasks based on the state of the processing cluster array 2512, designating one or more processing clusters 2514 to complete tasks, and so on. In one embodiment, the host software can prove a workload for scheduling on the processing cluster array 2512 via one of a number of graphics processing doorbells. The workload can then be automatically distributed by the scheduler 2510 logic within the scheduler microcontroller in the processing cluster array 2512.

[0233] The processing cluster array 2512 can include up to "N" processing clusters (e.g., cluster 2514A, 2514B, through 2514N). Each cluster 2514A-2514N of the processing cluster array 2512 can execute a plurality of concurrent threads. The scheduler 2510 can allocate work to the clusters 2514A-2514N of the processing cluster array 2512 using various scheduling and / or work distribution algorithms. The scheduling can be dynamic and can be performed by the scheduler 2510 or can be assisted by compiler logic during compilation of program logic configured for execution by the processing cluster array 2512. In one embodiment, different clusters 2514A-2514N of the processing cluster array 2512 can be allocated for processing different types of programs or for performing different types of computations.

[0234] The processing cluster array 2512 can be configured to perform a wide variety of parallel processing operations. In one embodiment, the processing cluster array 2512 is configured to perform general-purpose parallel compute operations. For example, the processing cluster array 2512 can include logic to perform processing tasks including filtering of video and / or audio data, performance of modeling operations including physics operations, and performance of data transformations.

[0235] In one embodiment, the processing cluster array 2512 is configured to perform parallel graphics processing operations. In embodiments in which the parallel processor 2500 is configured to perform graphics processing operations, the processing cluster array 2512 can include additional logic to support the execution of such graphics processing operations, including without limitation texture sampling logic to perform texture parameterization operations; and mosaicking logic and other vertex processing logic. Further, the processing cluster array 2512 can be configured to execute shader programs, including, but not limited to, vertex shaders, mosaicking shaders, geometry shaders, and pixel shaders. The parallel processing unit 2502 can transfer data to be processed from the system memory over the I / O unit 2504. During processing, the transferred data can be stored to on-chip memory (e.g., parallel processor memory 2522) that can include, without limitation, cache memory. Once processing is complete, results can be written to system memory over the I / O unit 2504.

[0236] In one embodiment, when parallel processing unit 2502 is used to perform graphics processing, scheduler 2510 can be configured to divide the processing workload into substantially equal sized tasks to better enable distribution of the graphics processing operations to the multiple clusters 2514A-2514N of processing cluster array 2512. In some embodiments, portions of processing cluster array 2512 can be configured to perform different types of processing. For example, a first portion can be configured to perform vertex shading and topology generation, a second portion can be configured to perform tessellation and geometry shading, and a third portion can be configured to perform pixel shading or other screen space operations to produce a rendered image for display. Intermediate data produced by one or more of clusters 2514A-2514N can be stored in buffers to allow the intermediate data to be transmitted between clusters 2514A-2514N for further processing.

[0237] During operation, processing cluster array 2512 can receive processing tasks to be executed via scheduler 2510, which receives commands defining the processing tasks from front end 2508. For graphics processing operations, the processing tasks can include data to be processed, such as surface (patch) data, geometric data, vertex data, and / or pixel data, and state parameters and commands (e.g., indices into a jump table of a particular procedure) that define how the data is to be processed. Scheduler 2510 can be configured to fetch the indices corresponding to the tasks or can receive the indices from front end 2508. Front end 2508 can be configured to ensure processing cluster array 2512 is configured in an effective state prior to a workload specified by an incoming command buffer (e.g., a batch buffer, an in-stall buffer, etc.) being launched.

[0238] Each of the one or more instances of parallel processing unit 2502 can be coupled to parallel processor memory 2522. Parallel processor memory 2522 can be accessed by the processing clusters 2512 and the I / O unit 2504 via memory crossbar 2516. The memory crossbar 2516 can be accessed via a memory interface 2518 from the processing clusters 2512 and the I / O unit 2504. The memory interface 2518 can include a number of memory partitions, such as memory partition 2520A, memory partition 2520B, up to memory partition 2520N. In one embodiment, the number of memory partitions 2520A-2520N can be configured to be equal to the number of memory units, such that a first memory partition 2520A has a corresponding first memory unit 2524A, a second memory partition 2520B has a corresponding second memory unit 2524B, and an Nth memory partition 2520N has a corresponding Nth memory unit 2524N. In other embodiments, the number of memory partitions 2520A-2520N can not be equal to the number of memory units.

[0239] In various embodiments, the memory units 2524A-2524N can include various types of memory devices including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. In one embodiment, the memory units 2524A-2524N can also include 3D stacked memory, including but not limited to high bandwidth memory (HBM). One of skill in the art will understand that the specific implementation of the memory units 2524A-2524N can vary and can be selected from one of various conventional designs. Render targets, such as frame buffers or texture maps, can be stored on the memory units 2524A-2524N, allowing the memory partitions 2520A-2520N to write portions of each render target in parallel, to effectively use the available bandwidth of the parallel processor memory 2522. In some embodiments, to support a unified memory design that utilizes system memory in conjunction with local cache memory, local instances of the parallel processor memory 2522 can be excluded.

[0240] In one embodiment, any of the clusters 2514A-2514N of the processing cluster array 2512 can process data to be written into any of the memory units 2524A-2524N within the parallel processor memory 2522. The memory crossbar 2516 can be configured to communicate the output of each cluster 2514A-2514N to any partition unit 2520A-2520N or another cluster 2514A-2514N, which can perform additional processing operations on the output. Each cluster 2514A-2514N can communicate through the memory crossbar 2516 with the memory interface 2518 to perform loads and stores containing data values between various external memory devices and the parallel processor memory 2522. In one embodiment, memory crossbar 2516 has a connection to the memory interface 2518 to communicate with the I / O unit 2504, and a connection to a local instance of the parallel processor memory 2522, enabling processing elements within the different processing clusters 2514A-2514N to communicate with system memory or other memories not located locally to the parallel processor units 2502. In one embodiment, memory crossbar 2516 can use virtual channels to separate traffic streams between the clusters 2514A-2514N and the partition units 2520A-2520N.

[0241] While a single instance of the parallel processor unit 2502 is shown in the parallel processor 2500, any number of instances of the parallel processor unit 2502 can be included. For example, a multi-socket platform can have multiple instances of the parallel processor unit 2502 provided in different sockets. In one embodiment, different instances of the parallel processor unit 2502 can be configured to interoperate to enable parallel processing of data. For example, and in one embodiment, some instances of the parallel processor unit 2502 can be configured to process

[0242] FIG. 25B is a block diagram of a partition unit 2520 in accordance with an embodiment. In one embodiment, the partition unit 2520 is a FIG. 25AThe partition unit 2520 is an example of one of the partition units 2520A-2520N. As illustrated, the partition unit 2520 includes an L2 cache 2521, a frame buffer interface 2525, and an ROP 2526 (raster operations unit). The L2 cache 2521 is a read / write cache that is configured to perform load and store operations received from the memory crossbar 2516 and the ROP 2526. Read misses and urgent write-back requests are output by the L2 cache 2521 to the frame buffer interface 2525 for processing. Updates can also be sent to the frame buffer via the frame buffer interface 2525 for processing. In one embodiment, the frame buffer interface 2525 interfaces with one of the memory units in the parallel processor's memory, such as the memory units 2524A-2524N in FIG. 25 (e.g., within the parallel processor's memory 2522).

[0243] In graphics applications, the ROP 2526 is a processing unit that performs raster operations such as stencil, z-test, blending, and so forth. The ROP 2526 then outputs primed graphics data that is stored in graphics memory. In some embodiments, the ROP 2526 includes compression logic to compress depth or color data that is written to memory and to decompress depth or color data that is read from memory. The compression logic can be lossless compression logic that uses one or more of a variety of compression algorithms. The type of compression that is performed by the ROP 2526 can vary according to statistical properties of the data to be compressed. For example, in one embodiment, delta color compression is performed on depth and color data on a per-tile basis.

[0244] In some embodiments, the ROP 2526 is included within each processing cluster (e.g., clusters 2514A-2514N of FIG. 25) instead of in the partition unit 2520. In such embodiments, read and write requests for pixel data are transmitted through the memory crossbar 2516 instead of pixel fragment data. Processed graphics data can be displayed on one or more display devices 2410, routed to the processor 2402 for further processing, or routed to another processing entity for further processing. FIG. 24 FIG. 25A

[0245] FIG. 25C ​​is a block diagram of a processing cluster 2514 within a parallel processing unit according to an embodiment. In one embodiment, the processing cluster is an instance of one of the processing clusters 2514A-2514N of Figure 25. A processing cluster 2514 can be configured to execute a plurality of threads in parallel, where the term "thread" refers to an instance of a particular program executing on a particular set of input data. In some embodiments, single-instruction, multiple-data (SIMD) instruction issue techniques are used to enable parallel execution of a large number of threads on a single processor. In other embodiments, single-instruction, multiple-thread (SIMT) techniques are used to enable parallel execution of a large number of generally synchronous threads on each of multiple processors.

[0246] Operation of a processing cluster 2514 can be controlled via a pipeline manager 2532 that distributes processing tasks to SIMT parallel processors. The pipeline manager 2532 receives instructions from the scheduler 2510 of Figure 25 and manages execution of those instructions via a graphics multiprocessor 2534 and / or a texture unit 2536. The illustrated graphics multiprocessor 2534 is an exemplary instance of a SIMT parallel processor. However, various types of SIMT parallel processors of differing architectures can be included within a processing cluster 2514. One or more instances of the graphics multiprocessor 2534 can be included within a processing cluster 2514. Graphics multiprocessor 2534 can process data and a data crossbar 2540 can be used to distribute the processed data to one of multiple possible destinations. The pipeline manager 2532 can facilitate distribution by specifying destinations for processed data as a function of that processed data.

[0247] Each graphics multiprocessor 2534 within a processing cluster 2514 can include an identical set of functional execution logic (e.g., arithmetic logic units, load store units, etc.). The functional execution logic can be configured to implement a wide variety of operations and / or data types on registers and / or memory using both single-threaded and multi-threaded execution techniques. In one embodiment, graphics multiprocessor 2534 can be configured to execute shader programs that define various custom processing operations, such as blending, smoothing, or other various graphics processing operations.

[0248] The instructions transmitted to the processing cluster 2514 form a thread. A set of threads executed on the set of parallel processing engines is a threadblock. A threadblock executes the same program on different input data. Each thread in the threadblock can be assigned to a different processing engine within the graphics multiprocessor 2534. A threadblock can include fewer threads than are available processing engines within the graphics multiprocessor 2534. When a threadblock includes fewer threads than are available processing engines, one or more of the processing engines can be idle during the cycle that the threadblock is being processed. A threadblock can also include more threads than are available processing engines within the graphics multiprocessor 2534. When a threadblock includes more threads than are available processing engines, multiple threadblocks can be executed simultaneously by the processing engines available within the graphics multiprocessor 2534. In one embodiment, a plurality of threadblocks can be executed concurrently on the graphics multiprocessor 2534.

[0249] In one embodiment, the graphics multiprocessor 2534 includes an internal cache memory for storing commands and data. In one embodiment, the graphics multiprocessor 2534 can discard the internal cache memory and use the cache memory within the processing cluster 2514 (e.g., the LI cache 308). Each graphics multiprocessor 2534 can also have access to L2 Cache within the partition unit (e.g., L2 caches 2520A-2520N of FIG. 25), which can be shared among all of the processing clusters 2514 in the processing unit 2502, and can be used to transfer data between threads. The graphics multiprocessor 2534 can also have access to off-chip global memory, which can include one or more of a local parallel processor memory and a system memory. Any memory external to the parallel processor unit 2502 can be accessed by the processing clusters 2514 via the memory interface 2518. Embodiments in which the processing cluster 2514 includes multiple instances of graphics multiprocessor 2534 can share the common instructions and data stored in the LI cache 308.

[0250] Each processing cluster 2514 can include an MMU 2545 (memory management unit) configured to translate virtual addresses into physical addresses, in other embodiments, one or more instances of the MMU 2545 can reside in the memory interface 2518 of FIG. 25. The MMU 2545 includes a set of page table entries (PTEs) used to translate virtual addresses into physical addresses by mapping the virtual addresses to physical addresses at which tiles (more on tiles) and optionally cache lines are stored. The MMU 2545 can include address translation lookaside buffers (TLBs) or caches which can reside within the graphics multiprocessor 2534 or the LI cache or the processing cluster 2514. The physical addresses are processed to distribute surface data access locality to allow efficient request interweaving between processing clusters. The cache line index can be used to determine a request for a cache line is a hit or miss.

[0251] In graphics and compute applications, processing cluster 2514 can be configured such that each graphics multiprocessor 2534 is coupled to a texture unit 2536 for performing texture mapping operations, e.g., determining texture sample positions, reading texture data, filtering texture data. Texture data can be read from an internal texture Ll cache (not shown) or in some embodiments from the Ll cache within graphics multiprocessor 2534, and is fetched from an L2 cache, local parallel processor memory, or system memory, as needed. Each graphics multiprocessor 2534 outputs processed tasks to data crossbar 2540 to provide processed task data to another processing cluster 2514 for further processing or to store processed task data in an L2 cache, local parallel processor memory, or system memory via memory crossbar 2516. PreROP 2542 (pre-raster operations unit) is configured to receive data from graphics multiprocessor 2534, direct data to ROP units, which can be located in the same or different processing cluster 2514 as graphics multiprocessor 2534 as set forth herein. PreROP 2542 unit can perform optimizations and reorganizations on the pixel

[0252] It should be understood that the core architecture described herein is illustrative and that variations and modifications are possible. Any number of processing units, e.g., graphics multiprocessor 2534, texture unit 2536, preROP 2542, etc., can be included in processing cluster 2514. In addition, while only one processing cluster 2514 is shown, any number of parallel processing units can be included in processing cluster 2514, as described herein. In one embodiment, each processing cluster 2514 can be configured to operate independently of the other processing clusters 2514 using separate and different processing units, Ll caches, etc.

[0253] FIG. 25D A graphics multiprocessor 2534 according to one embodiment is shown. In such embodiments, graphics multiprocessor 2534 is coupled with a pipeline manager 2532 of a processing cluster 2514. Graphics multiprocessor 2534 has an execution pipeline that includes, without limitation, an instruction cache 2552, an instruction unit 2554, an address mapping unit 2556, a register file 2558, one or more general-purpose graphics processing unit (GPGPU) cores 2562, and one or more load / store units 2566. GPGPU cores 2562 and load / store units 2566 are coupled with a memory and cache interconnect 2568 to cache memory 2572 and shared memory 2570.

[0254] In one embodiment, instruction cache 2552 receives a stream of instructions to be executed by the pipeline manager 2532. These instructions are cached in the instruction cache 2552 and dispatched for execution to the instruction unit 2554. The instruction unit 2554 can dispatch instructions as threads to different execution units of the GPGPU cores 2562. Each thread can be allocated resources on a graphics processing cluster 2590 (GPC 2590) for execution. The resources can include register files, memory space, and memory bandwidth. The resources can be allocated by the scheduler 2592. The scheduler 2592 can also monitor the resources allocated to each thread and, in one embodiment, can deallocate resources when the thread is stalled.

[0255] Register file 2558 provides temporary storage for operands of the data

[0256] The GPGPU cores 2562 can each include floating point, integer, and / or single instruction, multiple data (SIMD) execution units supporting multiple warps. In one embodiment, the execution units can implement IEEE 754-2008 floating point arithmetic. The GPGPU cores 2562 can also include single precision (32-bit) floating point, double precision (64-bit) floating point, and / or integer (32-bit) execution units. In one embodiment, the GPGPU cores 2562 can include a shared instruction cache for the GPGPU cores 2562. In one embodiment, the GPGPU cores 2562 can also include a shared memory space for the GPGPU cores 2562.

[0257] In one embodiment, the GPGPU cores 2562 include SIMD logic capable of performing a single instruction on multiple sets of data. In one embodiment GPGPU cores 2562 can physically execute SIMD8, SIMD4 instructions and logically execute SIMDl, SIMD2, and SIMD32 instructions. The SIMD instructions for GPGPU cores can be generated at compile time by a shader compiler or dynamically generated by a just-in-time compiler. A plurality of threads of a program configured for the SIMT execution model can be executed via a single SIMD instruction where the threads are singly invoked or dynamically generated. For example and in one embodiment, eight SIMT threads can be executed concurrently via a single SIMD8 logic unit.

[0258] The memory and cache interconnect 2568 is an interconnect network that connects each of the functional units of graphics multiprocessor 324 to the register file 2558 and shared memory 2570. In one embodiment the memory and cache interconnect 2568 is a crossbar interconnect that allows load / store units 2566 to initiate loads and stores between shared memory 2570 and register file 2558. The register file 2558 can operate at similar frequencies as GPGPU cores 2562, thus allowing for very short latency accesses to register file 2558 by GPGPU cores 2562. The shared memory 2570 can be used to communicate between threads executing on functional units within graphics multiprocessor 2534. For example, cache memory 2572 can be used by a texture unit 2536 to cache texture data to speed up texture accesses during graphics processing operations. The shared memory 2570 can also be used to store cached managed program. In addition to the automatically cached data stored in cache memory 2572, a thread executing on a GPGPU core 2562 can store program instructions loaded from the shared memory 2570.

[0259] FIGS. 26A-26B An additional graphics multiprocessor is illustrated in accordance with an embodiment. The illustrated graphics multiprocessor 2625, 2650 is a variation of graphics multiprocessor 2534 of FIG. 25C FIG. 5. The illustrated graphics multiprocessor 2625, 2650 can be configured to execute a large number of execution threads in a streaming multi-processor (SM).

[0260] FIG. 26A A graphics multiprocessor 2625 in accordance with an additional embodiment is shown. The graphics multiprocessor 2625 includes a variation of the graphics multiprocessor 2534 of FIG. 25Dadditional instances of execution resources of the graphics multiprocessor 234. For example, graphics multiprocessor 525 can include multiple instances of the instruction units 2632A-B, register file 2634A-B, and texture unit(s) 2644A-B. Graphics multiprocessor 2625 also includes multiple groups of graphics or compute execution units (e.g., GPGPU cores 2636A-B, GPGPU cores 2637A-B, GPGPU cores 2638A-B) and groups of load / store units 2640A-B. In one embodiment, the execution resources units have a common instruction cache 2630, texture and / or data cache memory 2642, and shared memory 2646.

[0261] The various components can communicate via an interconnect 2627. In one embodiment, the interconnect 2627 includes one or more crossbars that facilitate communication between the various components of the graphics multiprocessor 2625. In one embodiment, the interconnect 2627 is a separate high-speed network structure layer on which each component of the graphics multiprocessor 2625 is stacked. The components of the graphics multiprocessor 2625 communicate with remote components via the interconnect 2627. For example, the GPGPU cores 2636A-B, 2637A-B, and 2678A-B can each communicate with the shared memory 2646 via the interconnect 2627. The interconnect 2627 can arbitrate communications between the components within the graphics multiprocessor 2625 to ensure fair bandwidth allocation.

[0262] FIG. 26B A graphics multiprocessor 2650 according to an additional embodiment is shown. As FIG. 25D and FIG. 26A illustrated, the graphics processor includes multiple groups of execution resources 2656A-D, where each group of execution resources includes multiple instruction units, register file, GPGPU cores, and load store units. The execution resources 2656A-D can work with the texture unit(s) 2660A-D to perform texture operations, while sharing the instruction cache 2654 and shared memory 2662. In one embodiment, the execution resources 2656A-D can share the instruction cache 2654 and shared memory 2662, as well as multiple instances of the texture and / or data cache memory 2658A-B. The various components can communicate via an interconnect 2652 that is similar to the interconnect 2627 of FIG. 26A

[0263] Those of skill in the art will appreciate that FIG. 24 ,​FIGS. 25A-25D and FIGS. 26A-26B The architecture described in FIGS. 25 and 26 is descriptive, and does not limit the scope of embodiments of the application. Thus, the techniques described herein can be implemented on any suitably configured processing unit, including but not limited to: one or more mobile application processors; one or more desktop computer or server central processing units (CPUs), including multi-core CPUs; one or more parallel processing units such as parallel processing unit 2502 of FIG. 25; and one or more graphics processors or specialized processing units, without departing from the scope of embodiments described herein.

[0264] In some embodiments, parallel processors or GPGPUs as described herein are communicatively coupled to host / processor cores to accelerate graphics operations, machine learning operations, pattern analysis operations, and various general purpose GPU (GPGPU) functions. The GPU can be communicatively coupled to the host processor / cores over a bus or other interconnect (e.g., a high-speed

[0265] Techniques for GPU to Host Processor Interconnect

[0266] FIG. 27A FIG. 26 illustrates an exemplary architecture in which multiple GPUs 2610-2612 are communicatively coupled to multiple multi-core processors 2605-2606 over high-speed links 2640-2643 (e.g., buses, point-to-point interconnects, etc.). In one embodiment, the high-speed links 2640-2643 support communication at a throughput level of 4 GB / s, 30 GB / s, 80 GB / s or higher, depending on implementation. Various interconnect protocols can be used including, but not limited to, PCIe 4.0 or 5.0 and NVLink 2.0. However, the underlying principles of this application are not limited to any particular communication protocol or throughput level.

[0267] Further, in one embodiment, two or more of GPUs 2710-2713 are interconnected over high-speed links 2744-2745, which can be implemented using the same or different protocol / links as used for high-speed links 2740-2743. Similarly, two or more of multicore processors 2705-2706 can be connected over a high-speed link 2733, which can be a symmetric multi-processor (SMP) bus that runs at 20 GB / s, 30 GB / s, 120 GB / s, or higher. Alternatively, FIG. 27A All communication between the various system components shown in FIG. 27 can be accomplished using the same protocol / link (e.g., over a common interconnect fabric). However, as mentioned, the underlying principles of the application are not limited to any particular type of interconnect technology.

[0268] In one embodiment, each of multicore processors 2705-2706 is communicatively coupled to processor memories 2701-2702 via memory interconnects 2730-2731, respectively, and each of GPUs 2710-2713 is communicatively coupled to GPU memories 2720-2723 over GPU memory interconnects 2750-2753, respectively. Memory interconnects 2730-2731 and 2750-2753 can utilize the same or different memory access technologies. By way of example and not limitation, processor memories 2701-2702 and GPU memories 2720-2723 can be volatile memory such as dynamic random access memory (DRAM) (including stacked DRAM), graphics DDR SDRAM (GDDR) (e.g., GDDR5, GDDR6), or high-bandwidth memory (HBM), and / or can be non-volatile memory such as 3D XPoint or Nano-Ram. In one embodiment, some portion of the memory can be volatile memory while another portion can be non-volatile memory (e.g., using a two-level memory (2LM) hierarchy).

[0269] As described below, although each of various processors 2705-2706 and GPUs 2710-2713 can be physically coupled to a particular memory 2701-2702, 2720-2723, respectively, a unified memory architecture can be implemented in which the same virtual system address space (also referred to as an “effective address” space) is distributed across all of the various physical memories. For example, processor memories 2701-2702 can each include 64 GB of system memory address space, and GPU memories 2720-2723 can each include 32 GB of system memory address space (resulting in a total of 256 GB of addressable memory in the example).

[0270] FIG. 27BAdditional details of the interconnection between multicore processor 2707 and graphics acceleration module 2746 are illustrated in accordance with one embodiment. Graphics acceleration module 2746 can include one or more GPU chips integrated on a line card that is coupled to processor 2707 via high-speed link 2740. Alternatively, graphics acceleration module 2746 can be integrated on the same package or chip as processor 2707.

[0271] The illustrated processor 2707 includes a number of cores 2760A-2760D, each of which has a translation lookaside buffer 2761A-2761D and one or more caches 2762A-2762D. The cores can include various other components (e.g., instruction fetch unit, branch prediction unit, decoder, execution unit, reorder buffer, etc.) to execute instructions and process data that would obfuscate the principles of the application. The caches 2762A-2762D can include level 1 (Ll) and level 2 (L2) caches. In addition, one or more shared caches 2726 can be included in the cache hierarchy and shared by the groups of cores 2760A-2760D. For example, one embodiment of processor 2707 includes 24 cores, each of which has its own Ll cache, 12 shared L2 caches, and 12 shared L3 caches. In this embodiment, one of the L2 caches and L3 caches is shared by two adjacent cores. Processor 2707 and graphics accelerator integration module 2746 are connected with system memory 2741, which can include processor memory 2701-2702.

[0272] Data and instructions stored in various caches 2762A-2762D, 2756, and system memory 2741 are kept coherent across the cores via inter-core communications over coherency bus 2764. For example, each cache can have cache coherency logic / circuitry associated therewith to communicate over coherency bus 2764 in response to a detected read or write to a particular cache line. In one embodiment, a snoop protocol is implemented over coherency bus 2764 to snoop cache accesses. Cache snoop / coherency techniques are well understood by one skilled in the art, and are not described in detail here in order to avoid obscuring the principles of the application.

[0273] In one embodiment, proxy circuit 2725 communicatively couples graphics acceleration module 2746 to coherency bus 2764, allowing graphics acceleration module 2746 to participate in the cache coherency protocol as a peer to the cores. Specifically, interface 2735 provides connectivity from proxy circuit 2725 to high-speed link 2740 (e.g., PCIe bus, NVLink, etc.) and interface 2737 connects graphics acceleration module 2746 to link 2740.

[0274] In one embodiment, accelerator integrated circuit 2736 provides cache management, memory access, context management, and interrupt management services on behalf of multiple graphics processing engines 2731, 2732, and N of graphics acceleration module 2746. Graphics processing engines 2731, 2732, and N may each comprise a separate graphics processing unit (GPU). Alternatively, graphics processing engines 2731, 2732, and N may include different types of graphics processing engines within the GPU, such as a graphics execution unit, a media processing engine (e.g., a video encoder / decoder), a sampler, and a block image transfer engine. In other words, the graphics acceleration module may be a GPU comprising multiple graphics processing engines 2731 to 2732, and N, or graphics processing engines 2731 to 2732, and N may be separate GPUs integrated into a common package, line card, or chip.

[0275] In one embodiment, the accelerator integrated circuit 2736 includes a memory management unit (MMU) 2739 for performing various memory management functions, such as virtual-to-physical memory translation (also known as effective-to-real memory translation), and memory access protocols for accessing system memory 2741. The MMU 2739 may also include a translation lookaside buffer (TLB) (not shown) for caching virtual / effective-to-physical / real address translations. In one embodiment, cache 2738 stores commands and data for efficient access by graphics processing engines 2731-2732, N. In one embodiment, data stored in cache 2738 and graphics memories 2733-2734, N is maintained coherently with core caches 2762A-2762D, 2756, and system memory 2711. As mentioned, this can be accomplished via proxy circuitry 2725, which participates in cache coherence mechanisms on behalf of cache 2738 and memories 2733 to 2734, N (e.g., sending updates to cache 2738 related to modifications / accesses of cache lines on processor caches 2762A to 2762D, 2756 and receiving updates from cache 2738).

[0276] A set of registers 2745 store context data for threads executed by the graphics processing engines 2731-2732, N, and context management circuit 2748 manages thread contexts. For example, the context management circuit 2748 can perform save and restore operations to save and restore the context of various threads during context switches (e.g., where a first thread is saved and a second thread is stored such that the second thread can be executed by the graphics processing engines). For example, upon context switch, the context management circuit 2748 can store the current register values into a designated area in memory (e.g., identified by a context pointer). The context management circuit can then restore the register values upon return from context. In one embodiment, the interrupt management circuit 2747 receives and processes interrupts received from the system.

[0277] In one implementation, virtual / effective addresses from the graphics processing engines 2731 are translated to real / physical addresses in system memory 2711 by MMU 2739. One embodiment of accelerator integration circuit 2736 supports multiple (e.g., 4, 8, 16) graphics processor modules 2746 and / or other accelerator devices. The graphics processor modules 2746 can be dedicated to a single application executing on processor 2707 or shared between multiple applications. In one embodiment, a virtualized graphics execution environment is presented in which resources of the graphics processing engines 2731-2732, N are shared between multiple applications or virtual machines (VMs). The resources can be subdivided into "tiles" that are allocated to different VMs and / or applications based on processing requirements and priorities associated with the VMs and / or applications.

[0278] Thus, the accelerator integration circuit functions as a bridge to the system for the graphics acceleration module 2746 and provides address translation and system memory cache services. In addition, the accelerator integration circuit 2736 can provide virtualization facilities for a host processor to manage virtualization of graphics processing engines, interrupts, and memory management.

[0279] Since the hardware resources of the graphics processing engines 2731-2732, N are explicitly mapped to the real address space seen by the host processor 2707, any host processor can use effective address values to directly address these resources. In one embodiment, a function of the accelerator integration circuit 2736 is to physically separate the graphics processing engines 2731-2732, N so that they appear as independent units on the system.

[0280] As mentioned, in the illustrated embodiment, one or more graphics memory 2733-2734, M is coupled to each of the graphics processing engines 2731-2732, N, respectively. The graphics memory 2733-2734, M stores instructions and data for processing by each of the graphics processing engines 2731-2732, N. The graphics memory 2733-2734, M can be volatile memory, such as DRAM (including stacked DRAM), GDDR memory (e.g., GDDR5, GDDR6), or HBM, and / or can be non-volatile memory such as 3D XPoint or Nano-Ram.

[0281] In one embodiment, to reduce the data traffic on the link 2740, a biasing technique is used to ensure that data stored in the graphics memory 2733-2734, M is that which is most frequently used by the graphics processing engines 2731-2732, N and that data which is preferably not used by the cores 2760A-2760D (at least not frequently). Similarly, the biasing mechanism attempts to keep data required by the cores (and preferably not the graphics processing engines 2731-2732, N) within the caches 2762A-2762D, 2756 of the cores and system memory 2711.

[0282] FIG. 27C Another embodiment is illustrated in which the accelerator integration circuit 2736 is integrated within the processor 2707. In this embodiment, the graphics processing engines 2731-2732, N communicate directly over high-speed link 2740 to the accelerator integration circuit 2736 via the interface 2737 and interface 2735 (which can be utility work distribution, or other) in addition to the high-speed link 2740 (which can utilize any form of bus or interface protocol). The accelerator integration circuit 2736 can execute same operations as those described with regard to FIG. 27B the accelerator integration circuit 2736, but can operate at higher throughput given its close proximity to the coherence bus 2762 and caches 2762A-2762D, 2726.

[0283] One embodiment supports different programming models, including a dedicated process programming model (without graphics acceleration module virtualization) and a shared programming model (with virtualization). The shared programming model can include a programming model controlled by the accelerator integration circuit 2736 and a programming model controlled by the graphics acceleration module 2746.

[0284] In one embodiment of the dedicated process model, the graphics processing engines 2731-2732, N are dedicated to a single application or process under a single operating system. The single application can centralize other application requests to the graphics engines 2731-2732, N, providing virtualization within the VM / partition.

[0285] In a dedicated process programming model, graphics processing engines 2731-2732, N can be shared by multiple VM / application partitions. The shared model requires a system hypervisor that is used to virtualize the graphics processing engines 2731-2732, N to allow access by each operating system. For single-partition systems without a hypervisor, the graphics processing engines 2731-2732, N are owned by the operating system. In both cases, the operating system can virtualize the graphics processing engines 2731-2732, N to provide access to each process or application.

[0286] For the shared programming model, graphics acceleration module 2746 or individual graphics processing engines 2731-2732, N use a process handle to select a process element. In one embodiment, the process element is stored in system memory 2711 and is addressable using the effective to real address translation techniques described herein. The process handle can be an implementation-specific value provided to the host process when it registers its context with the graphics processing engines 2731-2732, N (i.e., calls system software to add the process element to a process element link table). The lower 16 bits of the process handle can be an offset into the process element link table of the process element.

[0287] FIG. 27D An exemplary accelerator integration tile 2790 is illustrated. As used herein, a "tile" includes a specified portion of the processing resources of accelerator integration circuit 2736. Application effective address space 2782 within system memory 2711 stores process elements 2783. In one embodiment, process elements 2783 are stored in response to GPU calls 2781 from applications 2780 executing on processor 2707. Process elements 2783 contain process state for respective applications 2780. Work descriptors (WDs) 2784 contained in process elements 2783 can be a single job requested by an application, or can contain a pointer to a queue of jobs. In the latter case, WD 2784 is a pointer to a job request queue in application address space 2782.

[0288] Graphics acceleration module 2746 and / or individual graphics processing engines 2731-2732, N can be shared by all or a subset of processes in a system. Embodiments of the present invention include infrastructure for setting up process state and sending WDs 2784 to graphics acceleration module 2746 to start jobs in a virtual environment.

[0289] In one embodiment, the dedicated process programming model is implementation specific. In this model, a single process owns a graphics acceleration module 2746 or individual graphics processing engines 2731. Since the graphics acceleration module 2746 is owned by a single process, the hypervisor initializes the accelerator integration circuit 2736 for the owning partition, and the operating system initializes the accelerator integration circuit 2736 for the owning process when the graphics acceleration module 2746 is allocated.

[0290] In operation, a WD fetch unit 2791 in the accelerator integration slice 2790 fetches the next WD 2784, which includes an indication of work to be done by one of the graphics processing engines of the graphics acceleration module 2746. As illustrated, data from the WD 2784 can be stored in registers 2745 and used by the MMU 2739, interrupt management circuit 2747, and / or context management circuit 2746. For example, one embodiment of the MMU 2739 includes a segment / page walk circuit to access a segment / page table 2786 within the OS virtual address space 2785. The interrupt management circuit 2747 can handle interrupt events 2792 received from the graphics acceleration module 2746. When performing graphics operations, effective addresses 2793 generated by the graphics processing engines 2731-2732, N are translated to real addresses by the MMU 2739.

[0291] In one embodiment, a same set of registers 2745 is replicated for each graphics processing engine 2731-2732, N and / or graphics acceleration module 2746, and can be initialized by the hypervisor or operating system. Each of these replicated registers can be included in an accelerator integration slice 2790. Exemplary registers that can be initialized by the hypervisor are shown in Table 1.

[0292] Table 1 - Hypervisor Initialized Registers

[0293] 1 Slice Control Register 2 Real Address (RA) Dispatch Process Region Pointer 3 Authorization Mask Override Register 4 Interrupt Vector Table Entry Offset 5 Interrupt Vector Table Entry Limit 6 Status Register 7 Logical Partition ID 8 Real Address (RA) Hypervisor Accelerator Utilization Record Pointer 9 Storage Descriptor Register

[0294] Exemplary registers that can be initialized by the operating system are shown in Table 2.

[0295] Table 2 - Operating System Initialized Registers

[0296] 1 Process and Thread Identification 2 Effective Address (EA) Context Save / Restore Pointer 3 Virtual Address (VA) Accelerator Utilization Record Pointer 4 Virtual Address (VA) Storage Segment Table Pointer 5 Authorization Mask 6 Work Descriptor

[0297] In one embodiment, each WD 2784 is specific to a particular graphics acceleration module 2746 and / or graphics processing engines 2731-2732, N. The WD includes all information that the graphics processing engines 2731-2732, N need to do their work, or the WD can be a pointer to a memory location where the application has set up a command queue of work to be completed.

[0298] FIG. 27E Additional details of one embodiment of the shared model are illustrated. The embodiment includes a hypervisor real address space 2798 in which a list of process elements 2799 is stored. The hypervisor real address space 2798 is accessible via the hypervisor 2796, which virtualizes the graphics acceleration module engines for the operating system 2795.

[0299] The shared programming model allows all processes or a subset of processes from all partitions or a subset of partitions in the system to use the graphics acceleration module 2746. There are two programming models in which the graphics acceleration module 2746 is shared by multiple processes and partitions: time-sliced sharing and graphics direct sharing.

[0300] In this model, the system hypervisor 2796 owns the graphics acceleration module 2746 and makes its functionality available to all operating systems 2795. In order for the graphics acceleration module 2746 to support the hypervisor 2796 virtualization, the graphics acceleration module 2746 can follow the following requirements:

[0301] 1) Application job requests must be autonomous (i.e., no need to maintain state between jobs), or the graphics acceleration module 2746 must provide a context save and restore mechanism. 2) The graphics acceleration module 2746 guarantees that an application job request will complete in a specified amount of time, including any translation faults, or the graphics acceleration module 2746 provides the ability to preempt job processing. 3) When operating in the direct shared programming model, the graphics acceleration module 2746 must guarantee fairness between processes.

[0302] In one embodiment, for a shared model, an application 2780 is required to utilize a graphics acceleration module 2746 type, a work descriptor (WD), an authorization mask register (AMR) value, and a context save / restore area pointer (CSRP) for an operating system 2795 system call. The graphics acceleration module 2746 type describes the target acceleration function for the system call. The graphics acceleration module 2746 type can be a system specific value. The WD is formatted specifically for the graphics acceleration module 2746 and can be in the form of a graphics acceleration module 2746 command; a pointer to a user defined structure; a pointer to a command queue; or any other data structure used to describe work to be performed by the graphics acceleration module 2746. In one embodiment, the AMR value is the AMR state for the current process. The value passed to the operating system is similar to how an application sets the AMR. If the implementation of the accelerator integration circuit 2736 and graphics acceleration module 2746 does not support a user authorization mask override register (UAMOR), the operating system can apply the current UAMOR value to the AMR value before passing the AMR in a hypervisor call. The hypervisor 2796 can optionally apply the current authorization mask override register (AMOR) value before placing the AMR in the process element 2783. In one embodiment, the CSRP is one of the registers 2745 containing the effective address of an area in the application address space 2782 for the graphics acceleration module 2746 to save and restore context state. This pointer is optional if there is no need to save state between jobs or when a job is preempted. The context save / restore area can be paged system memory.

[0303] Upon receiving the system call, the operating system 2795 can verify that the application 2780 is registered and authorized to use the graphics acceleration module 2746. The operating system 2795 then calls the hypervisor 2796 with the information shown in Table 3.

[0304] Table 3 - Operating System Call to Hypervisor Parameters

[0305] 1 Work Descriptor (WD) 2 Authorization Mask Register (AMR) Value (possibly masked) 3 Effective Address (EA) Context Save / Restore Region Pointer (CSRP) 4 Process ID (PID) and optional Thread ID (TID) 5 Virtual Address (VA) Accelerator Utilization Record Pointer (AURP) 6 Virtual Address of Storage Segment Table Pointer (SSTP) 7 Logical Interrupt Service Number (LISN)

[0306] Upon receiving the hypervisor call, the hypervisor 2796 verifies that the operating system 2795 is registered and authorized to use the graphics acceleration module 2746. The hypervisor 2796 then places the process element 2783 in a process element link table for the corresponding graphics acceleration module 2746 type. The process element can contain the information shown in Table 4.

[0307] Table 4 - Process Element Information

[0308] 1 Work Descriptor (WD) 2 Authorization Mask Register (AMR) Value (possibly masked) 3 Effective Address (EA) Context Save / Restore Region Pointer (CSRP) 4 Process ID (PID) and optional Thread ID (TID) 5 Virtual Address (VA) Accelerator Utilization Record Pointer (AURP) 6 Virtual Address of Storage Segment Table Pointer (SSTP) 7 Logical Interrupt Service Number (LISN) 8 Interrupt Vector Table, derived from hypervisor call parameters 9 Status Register (SR) Value 10 Logical Partition ID (LPID) 11 Real Address (RA) Hypervisor Accelerator Utilization Record Pointer 12 Storage Descriptor Register (SDR)

[0309] In one embodiment, the hypervisor initializes the multiple accelerator integration slices 2790 of the register 2745.

[0310] As FIG. 27F illustrated, one embodiment of the present application employs a unified memory that can be addressed via a common virtual memory address space for accessing physical processor memory 2701-2702 and GPU memory 2720-2723. In this implementation, operations performed on the GPUs 2710-2713 utilize the same virtual / effective memory address space to access processor memory 2701-2702 and vice versa, thereby simplifying programmability. In one embodiment, a first portion of the virtual / effective address space is allocated to processor memory 2701, a second portion is allocated to second processor memory 2702, a third portion is allocated to GPU memory 2720, and so on. The entire virtual / effective memory space (sometimes referred to as the effective address space) is thereby distributed across each of the processor memory 2701-2702 and GPU memory 2720-2723, allowing any processor or GPU to access any physical memory with a virtual address mapped to said memory.

[0311] In one embodiment, bias / coherence management circuitry 2794A-2794E within one or more of the MMUs 2739A-2739E ensures cache coherency between the host processor (e.g., 2705) and the caches of the GPUs 2710-2713, as well as implements bias techniques that dictate the physical memory in which certain types of data should be stored. Although multiple instances of bias / coherence management circuitry 2794A-2794E are illustrated in FIG. 27F , bias / coherence circuitry can also be implemented within the MMU(s) of the host processor(s) 2705 and / or within the accelerator integration circuit 2736.

[0312] One embodiment allows GPU-attached memory 2720-2723 to be mapped as part of system memory and accessed using shared virtual memory (SVM) techniques, but without suffering the typical performance penalties associated with full system cache coherency. The ability for GPU-attached memory 2720-2723 to be accessed as system memory does not incur the heavy cache coherency overhead, which provides a favorable operating environment for GPU offload. This arrangement allows the host processor 2705 software to set operands and access computation results without the overhead of traditional I / O DMA data copies. These traditional copies involve driver calls, interrupts, and memory-mapped I / O (MMIO) accesses, which are inefficient relative to simple memory accesses. At the same time, the ability to access GPU-attached memory 2720-2723 without cache coherency overhead can be critical to the execution time of offloaded computations. For example, in the case of a large amount of streaming write memory traffic, cache coherency overhead can significantly reduce the effective write bandwidth seen by the GPU 2710-2713. The efficiency of operand setup, the efficiency of result access, and the efficiency of GPU computation all play a significant role in determining the effectiveness of GPU offload.

[0313] In one implementation, the selection between GPU bias and host processor bias is driven by a bias tracker data structure. For example, a bias table can be used, which can be a page-granularity structure (i.e., controlled at the granularity of a memory page) that includes 1 or 2 bits per GPU-attached memory page. The bias table can be implemented within the stolen memory range of one or more GPU-attached memories 2720-2723, with or without a bias cache in the GPU 2710-2713 (e.g., to cache frequently / recently used entries of the bias table). Alternatively, the entire bias table can be kept within the GPU.

[0314] In one embodiment, the bias table entry associated with each access to GPU-attached memory 2720-2723 is accessed prior to actually accessing the GPU memory, resulting in the following operations. First, local requests from GPUs 2710-2713 that find their pages in the GPU bias are forwarded directly to the corresponding GPU memory 2720-2723. Local requests from the GPUs that find their pages in the host bias are forwarded to the processor 2705 (e.g., over the high-speed link as discussed above). In one embodiment, requests from the processor 2705 that find the requested pages in the host processor bias complete the request as a normal memory read. Alternatively, requests for GPU-biased pages can be forwarded to the GPUs 2710-2713. If the GPU is not currently using the page, the GPU can convert the page to the host processor bias.

[0315] The bias state of a page can be changed by a software-based mechanism, a hardware- assisted software-based mechanism, or for a limited set of cases, a hardware-only mechanism.

[0316] One mechanism for changing the bias state employs an API call (e.g., OpenCL) that in turn invokes a device driver of the GPU, which in turn sends a message (or enqueues a command descriptor) to the GPU, directing the GPU to change the bias state and, for certain conversions, to perform a cache flush operation in the host. The cache flush operation is necessary for a conversion from the host processor 2705 bias to the GPU bias, but not for the reverse conversion.

[0317] In one embodiment, cache coherency is maintained by temporarily presenting GPU-biased pages that are not cacheable by the host processor 2705. To access these pages, the processor 2705 can request access from the GPU 2710, which can or can not grant access immediately, depending on the embodiment. Thus, to reduce the communication between the processor 2705 and the GPU 2710, it is advantageous to ensure that the GPU-biased pages are those that are needed by the GPU but not by the host processor 2705, and vice versa.

[0318] Graphics Processing Pipeline

[0319] FIG. 28 A graphics processing pipeline 2800, in accordance with an embodiment, is illustrated. In one embodiment, a graphics processor can implement the graphics processing pipeline 2800 illustrated. The graphics processor can be included within a parallel processing subsystem such as the parallel processor 2500 of Figure 25 as described herein, in one embodiment, the parallel processor is a GPU FIG. 24of the parallel processor(s) 2412. As described herein, various parallel processor systems can implement the graphics processing pipeline 2800 via one or more instances of the parallel processing units (e.g., parallel processing unit 2502 of FIG. 25). For example, a shader unit (e.g., graphics multiprocessor 2634 of FIG. 26) can be configured to perform the functions of one or more of the vertex processing unit 2804, the tessellation control processing unit 2808, the tessellation evaluation processing unit 2812, the geometry processing unit 2816, and the fragment / pixel processing unit 2824. The functions of the data assembler 2802, the primitive assembler 2806, 2814, 2818, the tessellation unit 2810, the rasterizer 2822, and the raster operations unit 2826 can also be performed by other processing engines within the processing clusters (e.g., processing clusters 214 of FIG. 21) and the corresponding partition units (e.g., partition units 220A to 220N of FIG. 22). The graphics processing pipeline 2800 can also be implemented using dedicated processing units for one or more of the functions. In one embodiment, one or more parts of the graphics processing pipeline 2800 can be executed by parallel processing logic within a general purpose processor (e.g., a CPU). In one embodiment, one or more parts of the graphics processing pipeline 2800 can be accessed via a memory interface 2828 by on-chip memory (e.g., parallel processor memory 2522 of FIG. 25), which can be an instance of the memory interface 2518 of FIG. 25. FIG. 3 FIG. 2 The graphics processing pipeline 2800 can also be implemented using dedicated processing units for one or more of the functions. In one embodiment, one or more parts of the graphics processing pipeline 2800 can be executed by parallel processing logic within a general purpose processor (e.g., a CPU). In one embodiment, one or more parts of the graphics processing pipeline 2800 can be accessed via a memory interface 2828 by on-chip memory (e.g., parallel processor memory 2522 of FIG. 25), which can be an instance of the memory interface 2518 of FIG. 25.

[0320] In one embodiment, the data assembler 2802 is a processing unit that collects vertex data for surfaces and primitives. The data assembler 2802 then outputs the vertex data, including vertex attributes, to the vertex processing unit 2804. The vertex processing unit 2804 is a programmable execution unit that is configured to execute vertex shader programs to transform and illuminate vertices of a vertex array. The vertex processing unit 2804 reads data stored in cache, local or system memory that contains vertex data for processing and can be programmed to transform the vertex data from an object-based coordinate representation to the world space coordinate space or the normalized device coordinate space.

[0321] A first instance of the primitive assembler 2806 receives vertex attributes from the vertex processing unit 2804. The primitive assembler 2806 reads stored vertex attributes as needed and constructs graphics primitives for processing by the tessellation control processing unit 2808. The graphics primitives include triangles, lines, points, patches, and so on, as supported by various graphics processing application programming interfaces (APIs).

[0322] ​The tessellation control processing unit 2808 treats the input vertices as control points for the geometry patch. These control points are transformed from the input representation from the patch (e.g., the basis for the patch) to a representation suitable for surface evaluation by the tessellation evaluation processing unit 2812. The tessellation control processing unit 2808 can also compute tessellation factors for the edges of the geometry patch. The tessellation factors are applicable to individual edges and quantify a level of view-dependent detail associated with the edge. The tessellation unit 2810 is configured to receive the tessellation factors for the edges of the patch and tessellate the patch into a plurality of geometric primitives such as line, triangle, or quadrilateral primitives that are transmitted to the tessellation evaluation processing unit 2812. The tessellation evaluation processing unit 2812 operates on the parametric coordinates of the subdivided patch to generate surface representations and vertex attributes for each vertex associated with the geometric primitives.

[0323] A second instance of the primitive assembler 2814 receives the vertex attributes from the tessellation evaluation processing unit 2812, reads the stored vertex attributes as needed, and constructs graphics primitives for processing by the geometry processing unit 2816. The geometry processing unit 2816 is a programmable execution unit that executes a geometry shader program to transform the graphics primitives received from the primitive assembler 2814 as specified by the geometry shader program. In one embodiment, the geometry processing unit 2816 is programmed to subdivide the graphics primitives into one or more new graphics primitives and compute parameters for rasterizing the new graphics primitives.

[0324] In some embodiments, the geometry processing unit 2816 can add or delete elements in the geometry stream. The geometry processing unit 2816 outputs parameters and vertices that specify new graphics primitives to the primitive assembler 2818. The primitive assembler 2818 receives the parameters and vertices from the geometry processing unit 2816 and constructs graphics primitives for processing by the viewport scale, cull, and clip unit 2820. The geometry processing unit 2816 reads data stored in the parallel processor memory or system memory for processing the geometry data. The viewport scale, cull, and clip unit 2820 performs clipping, culling, and viewport scaling and outputs the processed graphics primitives to the rasterizer 2822.

[0325] The rasterizer 2822 can perform depth culling and other depth-based optimizations. The rasterizer 2822 also performs scan conversion of new graphics primitives to generate fragments and outputs these fragments and associated coverage data to the fragment / pixel processing units 2824. The fragment / pixel processing units 2824 are programmable execution units configured to perform fragment shader programs or pixel shader programs. The fragment / pixel processing units 2824 transform the fragments or pixels received from the rasterizer 2822 as specified by the fragment or pixel shader programs. For example, the fragment / pixel processing units 2824 can be programmed to perform operations including, but not limited to, texture mapping, shading, blending, texture correction, and perspective correction to produce shaded fragments or pixels that are output to the raster operations unit 2826. The fragment / pixel processing units 2824 can read data stored in the parallel processor memory or system memory to use when processing the fragment data. The fragment or pixel shader programs can be configured to color at a sample, pixel, tile, or other granularity depending on how the processing units are configured.

[0326] The raster operations unit 2826 is a processing unit that performs raster operations including, but not limited to, stencil operations, z-test, blending, and the like and outputs pixel data as processed graphics data to be stored in graphics memory (e.g., the parallel processor memory 2522 in FIG. 25 and / or the system memory 2404 in FIG. 24 as illustrated in FIG. 25) for display on the one or more display devices 2410 or for further processing by one of the one or more processors 2402 or the parallel processor(s) 2412. In some embodiments, the raster operations unit 2826 is configured to compress z or color data that is written to memory and decompress z or color data that is read from memory. FIG. 24

[0327] Embodiments of the present invention provide significant benefits over current procedures for software-driven remote / wireless display performed on a CPU. The techniques described herein significantly reduce the involvement of the CPU during frame rendering, encoding, and transmission, thereby improving performance.

[0328] In embodiments, the term "engine" or "module" or "logic" can refer to, be part of, or include an application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group), and / or memory (shared, dedicated, or group) that execute one or more software or firmware programs, a combinational logic circuit, and / or other suitable components that provide the described functionality. In embodiments, an engine or module can be implemented in firmware, hardware, software, or any combination of firmware, hardware, and software.

[0329] ​Embodiments of the application can include each of the steps described above. These steps can be embodied in machine-executable instructions which can be used to cause a general-purpose or special-purpose processor to perform the steps. Alternatively, these steps can be performed by specific hardware components that contain hardwired logic for performing the steps, or by any combination of programmed computer components and custom hardware components.

[0330] As described herein, instructions can refer to specific configurations of hardware such as configuration as an Application Specific Integrated Circuit (ASIC) configured to perform certain operations or having certain pre-determined functionality or software instructions stored in memory that implement the techniques described herein. Accordingly, the techniques shown in the figures can be implemented using code and data stored and executed on one or more electronic devices (e.g., an end station, a network element, etc.). Such electronic devices store and communicate (internally and / or with other electronic devices over a network) code and data using computer machine- readable media such as non-transitory computer machine-readable storage media (e.g., magnetic disks; optical disks; random access memory; read only memory; flash memory devices; phase-change memory) and transitory computer machine-readable communication media (e.g., electrical, optical, acoustical or other form of propagated signals - such as carrier waves, infrared signals, digital signals, etc.).

[0331] In addition, such electronic devices typically include a set of one or more processors coupled to one or more other components, such as one or more storage devices (non-transitory machine-readable storage media), user input / output devices (e.g., a keyboard, a touchscreen, and / or a display), and network connections. The coupling of the set of processors and other components is typically through one or more buses and bridges (also termed as bus controllers). Storage devices and signals carrying the network traffic respectively represent one or more machine-readable storage media and machine-readable communication media. Thus, a storage device of a given electronic device typically stores code and / or data for execution on the set of one or more processors of that electronic device. Of course, one or more parts of an embodiment of the application can be implemented using different combinations of software, firmware, and / or hardware. Throughout this detailed description, for the purposes of explanation, numerous specific details were set forth in order to provide a thorough understanding of the present application. It will be apparent, however, to one skilled in the art that the present application can be practiced without some of these specific details. In certain instances, well-known structures and functions were not described in elaborate detail in order to avoid obscuring the subject matter of the present application. Accordingly, the scope and spirit of the application should be judged in terms of the claims following, rather than the above description.

Claims

1. A remote display device, comprising: a display engine to render a sequence of video images; an encoder to compress the sequence of video images to produce a sequence of compressed video images; a network interface controller to transmit the compressed video images over a network link to a remote display; a plurality of buffer pointer registers to store read and write pointers that respectively identify read and write locations in a frame buffer and a compressed stream buffer; a central processing unit (CPU) to initialize the read and write pointers to process one or more video images; and the display engine to access a first write pointer to write to a specified location in the frame buffer, the encoder to read from the frame buffer based on a first read pointer value, the encoder to write to the compressed stream buffer based on a second write pointer value, and the network interface controller to read from the compressed stream buffer based on a second read pointer value, the first and second write and read pointer values being updated without intervention from the CPU as the display engine writes to the frame buffer, the encoder reads from the frame buffer and writes to the compressed stream buffer, and the network interface controller reads from the compressed stream buffer.

2. The remote display device of claim 1, wherein the display engine is to transmit a notification signal to the encoder when the first write pointer reaches a specified threshold, the encoder responsively to begin reading from the frame buffer at the first read pointer value.

3. The remote display device of claim 1 or 2, wherein the encoder is to transmit a notification signal to the network interface controller when the second write pointer reaches a specified threshold, the network interface controller responsively to begin reading from the frame buffer at the second read pointer value.

4. The remote display device of claim 1 or 2, wherein the network interface controller is to transmit the compressed video images over a wireless link to the remote display.

5. The remote display device of claim 4, wherein the wireless link comprises a WiDi link.

6. The remote display device of claim 1 or 2, further comprising: circuitry to execute a thread of one or more virtual machines, the thread to cause execution of graphics commands, thereby causing the display engine to render the sequence of video images.

7. The remote display device of claim 1 or 2, wherein the buffer pointer registers comprise general purpose registers.

8. A method for remote display and content protection in a virtualized graphics processing environment, comprising: rendering a sequence of video images; compressing the sequence of video images to produce a sequence of compressed video images; transmitting the compressed video images over a network link to a remote display; storing read and write pointers in a plurality of buffer pointer registers, the read and write pointers respectively identifying read and write locations in a frame buffer and a compressed stream buffer; initializing the read and write pointers to process one or more video images by a CPU; accessing, by a display engine, a first write pointer to write to a specified location in the frame buffer, reading, by an encoder, from the frame buffer based on a first read pointer value, writing, by the encoder, to the compressed stream buffer based on a second write pointer value, and reading, by a network interface controller, from the compressed stream buffer based on a second read pointer value, the first and second write and read pointer values being updated without intervention from the CPU as the display engine writes to the frame buffer, the encoder reads from the frame buffer and writes to the compressed stream buffer, and the network interface controller reads from the compressed stream buffer. writing, by the encoder, to the compressed stream buffer based on a second write pointer value, reading, by the network interface controller, from the compressed stream buffer based on a second read pointer value, the first and second write and read pointer values being updated without interference from the CPU as the display engine writes to the frame buffer, the encoder reads from the frame buffer and writes to the compressed stream buffer, and the network interface controller reads from the compressed stream buffer.

9. The method of claim 8, further comprising: communicating a notification signal from the display engine to the encoder when the first write pointer reaches a specified threshold, the encoder responsively starting to read from the frame buffer at a first read pointer value.

10. The method of claim 8 or 9, further comprising: communicating a notification signal from the encoder to the network interface controller when the second write pointer reaches a specified threshold, the network interface controller responsively starting to read from the frame buffer at a second read pointer value.

11. The method of claim 8 or 9, wherein the network interface controller is configured to communicate the compressed video images to a remote display over a wireless link.

12. The method of claim 11, wherein the wireless link comprises a WiDi link.

13. The method of claim 8, further comprising: executing a thread of one or more virtual machines, the thread causing execution of graphics commands, thereby causing the display engine to render a sequence of video images.

14. The method of claim 8 or 9, wherein the buffer pointer registers comprise general purpose registers.

15. A machine-readable medium having stored thereon program code, which when executed by a machine, causes the machine to: render a sequence of video images; compress the sequence of video images to produce a sequence of compressed video images; communicate the compressed video images to a remote display over a network link; store read and write pointers in a plurality of buffer pointer registers, the read and write pointers respectively identifying read and write locations in a frame buffer and a compressed stream buffer; initialize the read and write pointers to process one or more video images by a CPU; access, by the display engine, a first write pointer to write to a specified location in the frame buffer, read, by the encoder, from the frame buffer based on a first read pointer value, write, by the encoder, to the compressed stream buffer based on a second write pointer value, read, by the network interface controller, from the compressed stream buffer based on a second read pointer value, the first and second write and read pointer values being updated without interference from the CPU as the display engine writes to the frame buffer, the encoder reads from the frame buffer and writes to the compressed stream buffer, and the network interface controller reads from the compressed stream buffer.

16. The machine-readable medium of claim 15, further comprising: communicate a notification signal from the display engine to the encoder when the first write pointer reaches a specified threshold, the encoder responsively starting to read from the frame buffer at a first read pointer value.

17. The machine-readable medium of claim 15 or 16, further comprising: a notification signal is communicated from the encoder to the network interface controller when the second write pointer reaches a specified threshold, the network interface controller responsively begins reading from the frame buffer at a second read pointer value.

18. The machine-readable medium of claim 15 or 16, wherein the network interface controller is to communicate the compressed video images to a remote display over a wireless link.

19. The machine-readable medium of claim 18, wherein the wireless link comprises a WiDi link.

20. The machine-readable medium of claim 15 or 16, further comprising: a thread to execute one or more virtual machines, the thread to cause execution of graphics commands, thereby causing a display engine to render a sequence of video images.

21. The machine-readable medium of claim 15 or 16, wherein the buffer pointer registers comprise general purpose registers.

22. An apparatus for remote display and content protection in a virtualized graphics processing environment, comprising: means for rendering a sequence of video images; means for compressing the sequence of video images to produce a sequence of compressed video images; means for communicating the compressed video images to a remote display over a network link; means for storing read and write pointers in a plurality of buffer pointer registers, the read and write pointers to identify read and write locations in a frame buffer and a compressed stream buffer, respectively; means for initializing the read and write pointers to process one or more video images by a CPU; means for accessing a first write pointer by a display engine to write to a specified location in the frame buffer, means for reading from the frame buffer by an encoder based on a first read pointer value, means for writing to a compressed stream buffer by the encoder based on a second write pointer value, means for reading from the compressed stream buffer by a network interface controller based on a second read pointer value, the first and second write and read pointer values being updated without intervention from the CPU as the display engine writes to the frame buffer, the encoder reads from the frame buffer and writes to the compressed stream buffer, and the network interface controller reads from the compressed stream buffer.

23. The apparatus of claim 22, further comprising: means for communicating a notification signal from the display engine to the encoder when the first write pointer reaches a specified threshold, the encoder responsively beginning to read from the frame buffer at a first read pointer value.

24. The apparatus of claim 22 or 23, further comprising: means for communicating a notification signal from the encoder to the network interface controller when the second write pointer reaches a specified threshold, the network interface controller responsively beginning to read from the frame buffer at a second read pointer value.

25. The apparatus of claim 22 or 23, wherein the network interface controller is to communicate the compressed video images to a remote display over a wireless link.

Images