Efficient memory layout for enabling intelligent data compression under machine learning environment

Abstract

A mechanism for facilitating efficient memory layout for enabling intelligent data compression in a machine learning environment is described. As described herein, one method of an embodiment includes facilitating dividing an initial tile representing an image into a primary plurality of tiles such that each tile of the primary plurality of tiles is treated as an independent image for processing by one or more processors of a computing device. The method can further include computing the primary plurality of tiles into a secondary plurality of tiles compatible with a size of a local buffer. The method can further include merging the secondary plurality of tiles into a final tile representing the image, and compressing the final tile.

Description

TECHNICAL FIELD

[0001] Embodiments described herein relate generally to computers. More particularly, embodiments are described for facilitating efficient memory layout for enabling intelligent data compression in a machine learning environment. BACKGROUND

[0002] To better enable efficiency in data processing, compression schemes and memory layouts have been proposed over time. However, conventional techniques are severely limited due to resource constraints, which is especially true in a machine learning environment. For example, conventional schemes are known to require multiple convolution operations to be performed on layers in a neural network, which often results in over-fetching of data around boundaries. BRIEF DESCRIPTION OF DRAWINGS

[0003] Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference numerals refer to similar elements.

[0004] FIG. 1 is a block diagram of a processing system in accordance with embodiments.

[0005] FIG. 2 is a block diagram of an embodiment of a processor having one or more processor cores, an integrated memory controller, and an integrated graphics processor.

[0006] FIG. 3 is a block diagram of a graphics processor that can be a discrete graphics processing unit, or can be a graphics processor integrated with a plurality of processing cores.

[0007] FIG. 4 is a block diagram of a graphics processing engine of a graphics processor in accordance with some embodiments.

[0008] FIG. 5 is a block diagram of hardware logic of a graphics processor core in accordance with some embodiments.

[0009] FIGS. 6A-6B Thread execution logic in accordance with some embodiments is shown, including an array of processing elements employed in a graphics processor.

[0010] FIG. 7 is a block diagram showing a graphics processor instruction format in accordance with some embodiments.

[0011] FIG. 8 is a block diagram of another embodiment of a graphics processor.

[0012] FIG. 9A is a block diagram showing a graphics processor command format in accordance with embodiments.

[0013] FIG. 9Bis a block diagram illustrating a graphics processor command sequence according to an embodiment.

[0014] FIG. 10 An exemplary graphics software architecture of a data processing system is illustrated in accordance with some embodiments.

[0015] FIG. 11A is a block diagram illustrating an IP core development system which can be used to manufacture an integrated circuit for performing operations according to embodiments.

[0016] FIG. 11B A cross-sectional side view of an integrated circuit package assembly is illustrated in accordance with some embodiments.

[0017] FIG. 12 is a block diagram illustrating an exemplary system on a chip integrated circuit which can be manufactured using one or more IP cores according to an embodiment.

[0018] FIGS. 13A-13B is a block diagram illustrating an exemplary graphics processor for use within a system on a chip (SoC) in accordance with the embodiments described herein.

[0019] FIGS. 14A-14B Additional exemplary graphics processor logic is illustrated in accordance with the embodiments described herein.

[0020] FIG. 15 is a generalized graph of a machine learning software stack.

[0021] FIG. 16A Various layers of a convolutional neural network (CNN) are illustrated.

[0022] FIG. 16B A computation stage within a convolution layer of a CNN is illustrated.

[0023] FIG. 17 An exemplary recurrent neural network is illustrated.

[0024] FIG. 18 Training and deployment of a deep neural network is illustrated.

[0025] FIG. 19 is a block diagram illustrating distributed learning.

[0026] FIG. 20 A computing device hosting an efficient memory layout mechanism is illustrated in accordance with one embodiment.

[0027] FIG. 21 An efficient memory layout mechanism is illustrated in accordance with one embodiment.

[0028] FIG. 22A An input feature map tile of an image having an overlap region is illustrated in accordance with one embodiment.

[0029] FIG. 22B A tile is shown as an independent picture according to one embodiment.

[0030] FIG. 22C A tile is shown that is split into multiple tiles representing independent pictures or images according to one embodiment.

[0031] FIG. 23 A method for facilitating efficient memory layout for smart data compression according to one embodiment is shown. DETAILED DESCRIPTION

[0032] In the following description, numerous specific details are set forth. However, embodiments, as described herein, can be practiced without the specific details. In other instances, well-known circuits, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0033] Embodiments provide for optimizing memory layout in order to work with compressed data when images are tiled. Typically, due to resource constraints, input feature maps (“IFMs” or simply “FMs”) are divided into various three-dimensional (3D) tiles, and since regular operations are performed on layers, data around the borders needs to be over-fetch, which is one of the problems with compressing and decompressing using regular techniques, since overlapping portions of data need to be obtained from multiple compressed tiles. This can require partial decompression, which cannot be performed more easily due to the variable nature of compression; alternatively, the entire tile needs to be simply decompressed in order to fetch the overlapping portion.

[0034] Embodiments provide a novel technique for providing a superior solution for partial decompression by computing tile sizes backward from the layer where multiple tiles need to be merged. In one embodiment, tiles are treated as independent images, such that these independent images can be decompressed in raster scan order without requiring random access decompression for overlapping portions. This enables a parallel implementation of a hardware accelerator (e.g., a machine learning hardware accelerator) that works with compressed 3D surfaces.

[0035] Embodiments provide a novel technique for performing a backward computation from a particular layer in order to find tile sizes that need to be fetched to treat FM tiles as independent units. This, for example, avoids any need for partial reads and thus partial decompression. In other words, this novel technique allows tiles to be treated as independent images throughout the process and enables a fairly simple parallel implementation of a hardware accelerator such as a machine or deep learning hardware accelerator.

[0036] It is contemplated that throughout this document, terms such as "request," "query," "job," "work," "work item," and "workload" can be used interchangeably. Similarly, "application" and "agent" can refer to or include computer programs, software applications, games, workstation applications, and the like provided through an application programming interface (API) such as a free-rendered API (e.g., Open Graphics Library 12and the like), where "dispatch" can be referred to interchangeably as a "work unit" or "draw," and similarly, "application" can be referred to interchangeably as a "workflow" or simply "agent." For example, a workload such as a three-dimensional (3D) game can include and issue any number and type of "frames," where each frame can represent an image (e.g., a sailboat, a human face). Further, each frame can include and provide any number and type of work units, where each work unit can represent a portion of the image (e.g., a mast of a sailboat, a forehead of a human face) represented by its respective frame. However, for consistency, throughout this document, each can be referred to by a single term (e.g., "dispatch," "agent," and the like).

[0037] In some embodiments, terms such as "display screen" and "display surface" can be used interchangeably to refer to the viewable portion of a display device, while the remaining portion of the display device can be embedded into a computing device such as a smartphone, a wearable device, and the like. It is contemplated and noted that embodiments are not limited to any particular computing device, software application, hardware component, display device, display screen or surface, protocol, standard, and the like. For example, embodiments can be applied to and used with any number and type of real-time applications on any number and type of computers such as desktops, laptops, tablets, smartphones, head-mounted displays, and other wearable devices, and the like. Further, for example, the range of scenarios that can be rendered efficiently using this novel technology can range from simple scenarios such as desktop compositing to complex scenarios such as 3D games, augmented reality applications, and the like.

[0038] It is noted that throughout this document, terms such as convolutional neural network (CNN), CNN, neural network (NN), NN, deep neural network (DNN), DNN, recurrent neural network (RNN), RNN, and the like can be used interchangeably. Additionally, throughout this document, terms such as "autonomous machine" or simply "machine," "autonomous vehicle" or simply "vehicle," "autonomous agent" or simply "agent," "autonomous device," or "computing device," "robot," and the like can be used interchangeably.

[0039] System Overview

[0040] FIG. 1is a block diagram of a processing system 100 according to embodiments. In embodiments, the system 100 includes one or more processors 102 and one or more graphics processors 108, and can be a single processor desktop system, a multiprocessor workstation system, or a server system that includes a large number of processors 102 or processor cores 107. In one embodiment, the system 100 is a processing platform incorporated within a chip that is used in a mobile device, hand-held device, or embedded device.

[0041] In one embodiment, the system 100 can include or incorporate a server-based gaming platform, a game console, including a game and media console, a mobile gaming console, a handheld game console, or an online game console. In some embodiments, the system 100 is a mobile phone, a smart phone, a tablet, or a mobile internet device. The processing system 100 can also include, couple with, or be integrated within a wearable device, such as a smart watch wearable device, a smart eyewear device, an augmented reality device, or a virtual reality device. In some embodiments, the processing system 100 is a television or set-top box device having one or more processors 102 and a graphical interface generated by one or more graphics processors 108.

[0042] In some embodiments, the one or more processors 102 each include one or more processor cores 107 to process instructions which, when executed, perform operations for system and user software. In some embodiments, each of the one or more processor cores 107 is configured to process a specific instruction set 109. In some embodiments, instruction set 109 can facilitate Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or computing via a Very Long Instruction Word (VLIW). Multiple processor cores 107 can each process a different instruction set 109, which can include instructions to facilitate the emulation of other instruction sets. A processor core 107 can also include other processing devices, such as a digital signal processor (DSP).

[0043] In some embodiments, the processor 102 includes cache memory 104. Depending upon the architecture, the processor 102 can have a single internal cache or multiple levels of internal caches. In some embodiments, the cache memory is shared among the various components of the processor 102. In some embodiments, the processor 102 also uses an external cache (e.g., a level 3 (L3) cache or last level cache (LLC)) (not shown), which can be shared among the processor cores 107 using known cache coherency techniques. Additionally, a register file 106 is included in the processor 102, which can include different types of registers to store different types of data (e.g., integer registers, floating point registers, status registers, and instruction pointer registers). Some registers can be general-purpose registers, while other registers can be specific to the design of the processor 102.

[0044] In some embodiments, the one or more processors 102 are coupled with one or more interface buses 110 for communicating information between the processor 102 and other components of the system 100, such as address, data, or control signals. In one embodiment, the interface bus 110 can be a processor bus, such as a version of the direct media interface (DMI) bus. However, the processor bus is not limited to the DMI bus and can include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In one embodiment, the processor(s) 102 include an integrated memory controller 116 and platform controller hub 130. The memory controller 116 facilitates communication between the memory devices and other components of the system 100, while the platform controller hub (PCH) 130 provides connections to I / O devices via a local I / O bus.

[0045] The memory device 120 can be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase- change memory device, or some other memory device having suitable performance as a processing memory. In one embodiment, the memory device 120 can operate as system memory for the system 100, for storing data 122 and instructions 121 for use when the one or more processors 102 execute an application or process. The memory controller 116 is also coupled to an optional external graphics processor 112, which can communicate with the one or more graphics processors 108 in the processors 102 to perform graphics and media operations. In some embodiments, a display device 111 can be coupled to the processor(s) 102. The display device 111 might be an internal device, such as in a mobile electronic device or a laptop device, or an external device, such as a monitor or a television. In one embodiment, the display device 111 can be a head-mounted display (HMD) for a virtual reality (VR) application or an augmented reality (AR) application.

[0046] In some embodiments, the platform controller hub 130 enables peripherals to connect to the memory device 120 and the processor 102 via a high-speed I / O bus. The I / O peripherals include, but are not limited to, an audio controller 146, a network controller 134, a firmware interface 128, a wireless transceiver 126, a touch sensor 125, a data storage device 124 (e.g., hard disk drive, flash memory, etc.). The data storage device 124 can connect via a storage interface (e.g., SATA) or via a peripheral bus, such as a Peripheral Component Interconnect bus (e.g., PCI, PCI Express). The touch sensor 125 can include a touch screen sensor, a pressure sensor, or a fingerprint sensor. The wireless transceiver 126 can be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver, such as a 3G, 4G, or Long Term Evolution (LTE) transceiver. The firmware interface 128 enables communication with the system firmware, and can be, for example, a unified extensible firmware interface (UEFI). The network controller 134 can enable network connectivity to a wired network. In some embodiments, a high-performance network controller (not shown) couples with the interface bus 110. The audio controller 146 is, in one embodiment, a multi-channel high definition audio controller. In one embodiment, the system 100 includes an optional legacy I / O controller 140 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to the system. The platform controller hub 130 can also connect to one or more Universal Serial Bus (USB) controllers 142 connected input devices, such as keyboard and mouse 143 combinations, a camera 144, or other USB input devices.

[0047] It will be recognized that the illustrated system 100 is exemplary and not limiting as other types of data processing systems configured in different ways can also be used. For example, instances of the memory controller 116 and platform controller hub 130 can be integrated into a discrete external graphics processor, such as external graphics processor 112. In one embodiment, the platform controller hub 130 and / or memory controller 160 can be external to the one or more processor(s) 102. For example, the system 100 can include an external memory controller 116 and platform controller hub 130 that can be configured as a memory controller hub and peripheral controller hub within a system chipset that is in communication with the processor(s) 102.

[0048] FIG. 2 is a block diagram of an embodiment of a processor 200 having one or more processor cores 202A to 202N, an integrated memory controller 214, and an integrated graphics processor 208. FIG. 2 Those elements of having the same reference number (or name) in the foregoing figure(s) as a corresponding element in any other figure(s) herein can operate or function in any manner similar to that described herein, but are not limited to such. The processor 200 can include additional cores beyond those illustrated in the dashed lined boxes. The processor cores 202A to 202N can each include one or more internal cache units 204A to 204N. In some embodiments, each processor core can also have access to one or more shared cache units 206.

[0049] The internal cache units 204A to 204N and the shared cache unit 206 represent a cache memory hierarchy within the processor 200. The cache memory hierarchy can include at least one level of instruction and data caches within each processor core and one or more levels of shared mid-level caches, such as level 2 (L2), level 3 (L3), level 4 (L4), or other levels of caches, with the highest level cache being classified as an LLC, external to the memory. In some embodiments, cache coherency logic maintains coherency among the cache units 206 and 204A to 204N.

[0050] In some embodiments, the processor 200 also includes a set of one or more bus controller units 216 and a system agent unit 210. The one or more bus controller units 216 manage a set of peripheral buses, such as one or more PCI or PCI express busses. The system agent unit 210 provides management functionality for the various processor components. In some embodiments, the system agent unit 210 includes one or more integrated memory controllers 214 to manage access to various external memory devices (not shown).

[0051] In some embodiments, one or more of the processor cores 202A-202N include support for simultaneous multi-threading. In such embodiments, the system agent unit 210 includes components for coordinating and operating the cores 202A-202N during multi-threaded processing. Additionally, the system agent unit 210 can also include a power control unit (PCU), including logic and components to regulate the power state of the processor cores 202A-202N, as well as the graphics processor 208.

[0052] In some embodiments, the processor 200 additionally includes a graphics processor 208 to perform graphics processing operations. In some embodiments, the graphics processor 208 is coupled to the shared cache unit 206 set and the system agent unit 210 including the one or more integrated memory controllers 214. In some embodiments, the system agent unit 210 also includes a display controller 211 to drive graphics processor output to one or more coupled displays. In some embodiments, the display controller 211 can also be a separate module coupled with the graphics processor via at least one interconnect, or can be integrated within the graphics processor 208.

[0053] In some embodiments, a ring-based interconnect unit 212 is used to couple the internal components of the processor 200. However, an alternative interconnect unit can be used, such as a point-to-point interconnect, a switched interconnect, or other technology, including technologies well known in the art. In some embodiments, the graphics processor 208 is coupled with the ring interconnect 212 via an I / O link 213.

[0054] The exemplary I / O link 213 represents at least one of a variety of I / O interconnects, including a package I / O interconnect to facilitate communication between the various processor components and a high performance embedded memory module 218, such as an eDRAM module. In some embodiments, each of the processor cores 202A-202N and the graphics processor 208 use the embedded memory module 218 as a shared last level cache.

[0055] In some embodiments, the processor cores 202A-202N are homogeneous cores executing the same instruction set architecture. In another embodiment, the processor cores 202A-202N are heterogeneous in terms of instruction set architecture (ISA), where one or more of the processor cores 202A-202N execute a first instruction set and at least one of the other cores executes a subset or a different instruction set. In one embodiment, the processor cores 202A-202N are homogeneous in terms of microarchitecture, where one or more of the cores have a relatively high power consumption and one or more power cores have a lower power consumption. Additionally, the processor 200 can be implemented on one or more chips or as a SoC integrated circuit having, among other components, the illustrated components.

[0056] FIG. 3 is a block diagram of a graphics processor 300, which can be a discrete graphics processing unit, or can be a graphics processor integrated with a number of processing cores. In some embodiments, the graphics processor communicates with memory via a mapped I / O interface to registers on the graphics processor and utilizes commands placed into processor memory. In some embodiments, the graphics processor 300 includes a memory interface 314 for accessing memory. The memory interface 314 can be an interface to local memory, one or more internal caches, one or more shared external caches, and / or to system memory.

[0057] In some embodiments, the graphics processor 300 also includes a display controller 302 for driving display output data to a display device 320. The display controller 302 includes hardware for one or more overlay planes for compositing and for compositing layers of video or user interface elements. The display device 320 can be an internal or external display device. In one embodiment, the display device 320 is a head mounted display device, such as a virtual reality (VR) display device or an augmented reality (AR) display device. In some embodiments, the graphics processor 300 includes a video codec engine 306 for encoding, decoding, or trans-coding media into media data in one or more media data formats, including, but not limited to, Moving Picture Experts Group (MPEG) formats such as MPEG-2, Advanced Video Coding (AVC) formats such as H.264 / MPEG-4 AVC, and Society of Motion Picture & Television Engineers (SMPTE) 421M / VC-1, and Joint Photographic Experts Group (JPEG) formats such as JPEG, and Motion JPEG (MJPEG) formats.

[0058] In some embodiments, graphics processor 300 includes a block image transfer (BLIT) engine 304 to perform two-dimensional (2D) rasterizer operations including, for example, bit- block transfer. However, in one embodiment, 2D graphics operations are performed using one or more components of graphics processing engine (GPE) 310. In some embodiments, GPE 310 is a compute engine for performing graphics operations, including three-dimensional (3D) graphics operations and media operations.

[0059] In some embodiments, GPE 310 includes a 3D pipeline 312 for performing 3D operations, such as rendering three-dimensional graphics shapes representing 3D objects and scenes along with the associated data. 3D pipeline 312 includes programmable and fixed function elements that perform various tasks to generate output to threadsawing elements and / or to a display. While 3D pipeline 312 can be used to perform media operations, an embodiment of GPE 310 also includes a media pipeline 316 to specifically carry out media operations such as video post-processing and image enhancements.

[0060] In some embodiments, media pipeline 316 includes fixed function or programmable logic unit to accelerate one or more specialized media operations, such as video

[0061] In some embodiments, 3D / media subsystem 315 includes logic to execute threads generated by 3D pipeline 312 and media pipeline 316. In one embodiment, the pipelines send thread execution requests to 3D / media subsystem 315, which includes thread dispatch logic to arbitrate and dispatch the requests to available thread execution resources. The execution resources include an array of graphics execution units to be used by 3D and media threads. In some embodiments, 3D / media subsystem 315 includes one or more internal caches to cache itinerant and data. In some embodiments, the subsystems also include shared memory (including registers and addressable memory) to share data between threads and to store output data.

[0062] Graphics Processing Engine

[0063] FIG. 4is a block diagram of a graphics processing engine 410 of a graphics processor according to some embodiments. In one embodiment, the graphics processing engine (GPE) 410 is a version of the GPE 310 shown and described previously. FIG. 3 FIG. 4 Those elements of the GPE 310 having the same reference number (or name) as that of any other figure herein can operate or function in any manner similar to that described elsewhere herein, but are not limited to such. For example, the 3D pipeline 312 and media pipeline 316 are shown. FIG. 3

[0064] In some embodiments, the GPE 410 is coupled with or includes a command stream translator 403 that provides a command stream to the 3D pipeline 312 and / or media pipeline 316. In some embodiments, the command stream translator 403 is a separate component from the GPE 410, while in other embodiments, the command stream translator 403 is internal to the GPE 410. In some embodiments, the command stream translator 403 is a software component that is converted to one or more hardware components such as, for example, a graphics

[0065] ​​In various embodiments, the 3D pipeline 312 includes fixed function logic and programmable logic to process one or more shader programs, such as vertex shaders, geometry shaders, pixel shaders, fragment shaders, compute shaders, or other shader programs. The graphics processor array 414 provides unified execution resources to

[0066] In some embodiments, the graphics processor array 414 also includes execution logic to perform media functions, such as video and / or image processing. In one embodiment, the execution units include programmable tissue to perform parallel general-purpose computing operations, in addition to graphics processing operations. FIG. 1 The (multiple) processor cores 107 or FIG. 2 The general-purpose logic within the cores 202A-202N in the

[0067] Output data generated by threads executing on graphics processor array 414 can be stored in a unified return buffer (URB) 418 in memory. The URB 418 can store data for multiple threads executing on graphics processor array 414. In some embodiments, the URB 418 can be used to transmit data between different threads executing on the graphics processor array 414. In some embodiments, the URB 418 can additionally be used for synchronization purposes between threads on the graphics processor array and fixed function logic within shared function logic 420.

[0068] In some embodiments, the graphics processor array 414 is scalable, such that the array includes varying numbers of graphics cores each having varying numbers of execution units based on the target power and performance level of the GPE 410. In one embodiment, the execution resources are dynamically scalable such that execution resources can be enabled or disabled as needed.

[0069] The graphics processor array 414 is coupled with shared function logic 420 that includes resources shared among the graphics cores within the graphics processor array. Shared functions within the shared function logic 420 are hardware logic units that provide specialized supplemental functionality to the graphics processor array 414. In various embodiments, the shared function logic 420 includes, but is not limited to, samplers 421, math 422, and inter-thread communication (ITC) 423 logic. Further, some embodiments implement one or more caches 425 within the shared function logic 420.

[0070] Shared functions are implemented in cases where the demand for a given special-purpose function is insufficient to be included in the graphics core array 414. Instead, a single instance of the special-purpose function is implemented as a standalone entity in shared function logic 420 and shared among the execution resources within the graphics core array 414. The exact set of functions that are shared among the graphics core arrays 414 and included within the graphics core array 414 varies between embodiments. In some embodiments, particular shared functions that are heavily used by the graphics core array 414 within the shared function logic 420 can be included within the shared function logic 416 within the graphics core array 414. In various embodiments, the shared function logic 416 within the graphics core array 414 can include some or all of the logic within the shared function logic 420. In one embodiment, all of the logic elements within the shared function logic 420 can be duplicated within the shared function logic 416 of the graphics core array 414. In one embodiment, the shared function logic 420 is executed in order to support the shared function logic 416 within the graphics core array 414.

[0071] FIG. 5 is a block diagram of hardware logic of a graphics processor core 500 in accordance with some embodiments described herein. FIG. 5 Those elements of having the same reference number (or name) in any other figure(s) herein as elements in the figure(s) of can operate or function in any manner similar to that described elsewhere herein, but are not limited to such. In some embodiments, the illustrated graphics processor core 500 includes a number of sub-cores FIG. 4 within the graphics core array 414 of In some embodiments, the graphics processor core 500 is sometimes referred to as a core slice. The graphics processor core 500 can be one or more graphics cores within a modular graphics processor. An example of the graphics processor core 500 is one graphics core slice, and a graphics processor as described herein can include multiple graphics core slices based on target power and performance envelopes. Each graphics core 500 can include fixed function blocks 530 coupled with a number of sub-cores 501A-501F (also referred to as sub-slices) that include modular general purpose logic blocks and fixed function logic blocks.

[0072] In some embodiments, the fixed function blocks 530 include a geometry / fixed function pipeline 536 that can be shared by all of the sub-cores in the graphics processor 500, for example, in low performance and / or low power graphics processor implementations. In various embodiments, the geometry / fixed function pipeline 536 includes a 3D fixed function pipeline (e.g., like 3D pipeline 312 in FIG. 3 and FIG. 4 a thread dispatcher, and a unified return buffer manager that manages a unified return buffer like unified return buffer 418 of FIG. 4 ​

[0073] In one embodiment, fixed function block 530 also includes a graphics SoC interface 537, a graphics microcontroller 538, and a media pipeline 539. Graphics SoC interface 537 provides an interface between graphics processing core 500 and other processor cores within a system on a chip integrated circuit. Graphics microcontroller 538 is a programmable sub-processor that is configurable to manage various functions of graphics processor 500 including thread dispatch, scheduling, and pre-emption. Media pipeline 539 (e.g., similar to media pipeline 316 of FIG. 3) includes logic to accelerate decoding, encoding, pre-processing, and / or post-processing of multimedia data, including image and video data. Media pipeline 539 implements media operations via requests to compute or sample logic within sub-cores 501-501F. FIG. 3 and FIG. 4 Media pipeline 539 (e.g., similar to media pipeline 316 of FIG. 3) includes logic to accelerate decoding, encoding, pre-processing, and / or post-processing of multimedia data, including image and video data. Media pipeline 539 implements media operations via requests to compute or sample logic within sub-cores 501-501F.

[0074] In one embodiment, SoC interface 537 enables graphics processing core 500 to communicate with general-purpose application processor cores (e.g., CPUs) and / or other components within the SoC including memory hierarchy elements such as shared L2 cache, system RAM, and / or embedded on-chip or package on-lens DRAM. SoC interface 537 can also enable communication with fixed function devices within the SoC such as camera imaging pipeline and to use and / or implement global memory atoms that can be shared between graphics processing core 500 and a CPU within the SoC. SoC interface 537 can also implement power management controls for graphics processing core 500 and enable an interface between a clock domain of graphics processing core 500 with other clock domains within the SoC. In one embodiment, SoC interface 537 enables receiving command buffers from a command streamer and global thread dispatcher that is configured to supply commands and instructions to each of one or more graphics processing cores within a graphics processor. These commands and instructions can be dispatched to media pipeline 539 when media operations are to be performed, or to a geometry and fixed function pipeline (e.g., geometry and fixed function pipeline 536, geometry and fixed function pipeline 514) when graphics processing operations are to be performed.

[0075] The graphics microcontroller 538 can be configured to perform various scheduling and management tasks for the graphics core 500. In one embodiment, the graphics microcontroller 538 can perform graphics and / or compute workload scheduling on individual graphics processing engines within execution unit (EU) arrays 502A-502F, 504A-504F within the corelets 501A-501F. In this scheduling model, host software executing on a CPU core of a SoC including the graphics core 500 can submit a workload via one of a number of graphics processor doorbells, which invokes a scheduling operation on the appropriate graphics engine. The scheduling operation includes determining which workload to run next, submitting the workload to a command streamer, pre-empting existing workloads running on the engine, monitoring progress of the workload, and notifying host software when the workload completes. In one embodiment, the graphics microcontroller 538 can also facilitate low power or idle states for the graphics core 500, providing the graphics core 500 with the ability to save and restore registers across low power state transitions independently of operating systems and / or graphics driver software on the system.

[0076] The graphics core 500 can have more or fewer than the illustrated number of sub-cores 501A-501F, up to N modular sub-cores. For each set of N sub-cores, the graphics core 500 can also include shared function logic 510, shared memory and / or cache memory 512, geometry / fixed function pipeline 514, and additional fixed function logic 516 for performing various graphics and compute operations. The shared function logic 510 can include logic units (e.g., sampler logic, math logic, and / or inter-thread communication logic) that are shared among each N sub-core within the graphics core 500. The shared memory and / or cache memory 512 can be a last level cache for the set of N sub-cores 501A-501F within the graphics core 500, and can also act as shared memory accessible by a number of the sub-cores. The geometry / fixed function pipeline 514 can be included within the fixed function block 530 instead of the geometry / fixed function pipeline 536, and can include the same or similar logic units. FIG. 4 The shared function logic 420 is associated with logic units (e.g., sampler logic, math logic, and / or inter-thread communication logic). The shared memory and / or cache memory 512 can be a last level cache for the set of N sub-cores 501A-501F within the graphics core 500, and can also act as shared memory accessible by a number of the sub-cores. The geometry / fixed function pipeline 514 can be included within the fixed function block 530 instead of the geometry / fixed function pipeline 536, and can include the same or similar logic units.

[0077] In one embodiment, graphics core 500 includes additional fixed function logic 516 which can include various fixed function acceleration logic to be utilized by graphics core 500. In one embodiment, additional fixed function logic 516 includes an additional geometry pipeline for use in position only shading. In position only shading, there are two geometry pipelines: a full geometry pipeline within geometry / fixed function pipeline 516, 536; and a cull pipeline, which is an additional geometry pipeline that can be included within additional fixed function logic 516. In one embodiment, the cull pipeline is a slimmed down version of the full geometry pipeline. The full pipeline and the cull pipeline can execute different instances of the same application, each with a separate context. Position only shading can hide the long cull run of discarded triangles, enabling completion of shading earlier in some instances. For example, and in one embodiment, cull pipeline logic within additional fixed function logic 516 can execute position shaders in parallel with the main application, and often generate critical results faster than the full pipeline because the full pipeline only fetches and shades position attributes of vertices, without performing rasterization and rendering of pixels to a frame buffer. The cull pipeline can use the generated critical results to compute visibility information for all triangles, without having to consider whether those triangles are culled. The full pipeline, which in this instance can be referred to as a replay pipeline, can consume the visibility information in order to skip culled triangles to only shade visible triangles that are ultimately passed to a rasterization stage.

[0078] In one embodiment, additional fixed function logic 516 can also include machine learning acceleration logic, such as fixed function matrix multiplication logic, for implementations including machine learning training or inferencing.

[0079] Within each graphics sub-core 501A-501F includes a set of execution resources that can be leveraged to perform graphics operations, media operations, and compute operations in response to requests by graphics pipeline, media pipeline, or shader programs. The graphics sub-cores 501A-501F include: multiple EU arrays 502A-502F, 504A-504F; thread dispatch and inter-thread communication (TD / IC) logic 503A-503F; 3D (e.g., texture) samplers 505A-505F; media samplers 506A-506F; shader processors 507A-507F; and shared local memory (SLM) 508A-508F. The EU arrays 502A-502F, 504A-504F each include multiple execution units, which are general-purpose graphics processing units capable of performing floating-point and integer / fixed-point logic operations to service graphics, media, or compute shader programs. The TD / IC logic 503A-503F performs local thread dispatch and thread control operations for the execution units within a sub-core and facilitate communication between threads executing on the execution units of the sub-core. The 3D samplers 505A-505F can read texture or other 3D graphics-related data into memory. The 3D samplers can read texture data differently based on a configured sample state and a texture format associated with a given texture. The media samplers 506A-506F can perform similar read operations based on a type and format associated with media data. In one embodiment, each graphics sub-core 501A-501F can alternately include unified 3D and media samplers. Threads executing on the execution units within each of the sub-cores 501A-501F can make use of shared local memory 508A-508F within each sub-core, to enable threads executing within a thread group to execute using a common pool of on-chip memory.

[0080] Execution Units

[0081] FIGS. 6A-6B Thread execution logic 600 including a processing element array employed in a graphics processor core is shown in accordance with an embodiment described herein. FIGS. 6A-6B Those elements of Figure 1 having the same reference number (or name) as elements in any other figure herein can operate or function in any manner similar to that described elsewhere herein, but are not limited to such. FIG. 6A An overview of thread execution logic 600 is shown, which can include a set of shared function units (SFUs) 610A-610B and shared function control logic 620A-620B, which can be shared under the control of a shared function unit controller 630A-630B. The SFUs 610A-610B can be shared by a set of execution units in the associated processing element. The SFUs 610A-610B can be configured and operated in any manner similar to that described elsewhere herein, but are not limited to such. FIG. 5 Variations of the hardware logic of each sub-core 501A-501F. FIG. 6B Exemplary internal details of an execution unit are shown.

[0082] like FIG. 6A As shown in , in some embodiments, thread execution logic 600 includes a shader processor 602, a thread dispatcher 604, an instruction cache 606, a scalable execution unit array including a plurality of execution units 608A to 608N, a sampler 610, a data cache 612, and a data port 614. In one embodiment, the scalable execution unit array can be dynamically scaled by enabling or disabling one or more execution units (e.g., execution units 608A, 608B, 608C, 608D, up to any one of 608N-1 and 608N) based on the computational requirements of the workload. In one embodiment, the included components are interconnected via an interconnect structure that links to each of the components. In some embodiments, thread execution logic 600 includes one or more connections to a memory (such as system memory or cache memory) through the instruction cache 606, the data port 614, the sampler 610, and one or more of the execution unit arrays 608A to 608N. In some embodiments, each execution unit (e.g., 608A) is an independently programmable general-purpose computing unit capable of executing multiple simultaneous hardware threads while processing multiple data elements in parallel for each thread. In various embodiments, the array of execution units 608A to 608N is scalable to include any number of individual execution units.

[0083] In some embodiments, execution units 608A to 608N are primarily used to execute shader programs. Shader processor 602 can process various shader programs and dispatch execution threads associated with the shader programs via thread dispatcher 604. In one embodiment, the thread dispatcher includes logic for arbitrating thread initiation requests from graphics and media pipelines and instantiating the requested threads on one or more execution units 608A to 608N. For example, the geometry pipeline can dispatch vertex processing, tessellation, or geometry processing threads to thread execution logic for processing. In some embodiments, thread dispatcher 604 can also handle runtime thread generation requests from executing shader programs.

[0084] In some embodiments, execution units 608A-608N support single program multiple instruction instructions 608A-608N (SPMD) with each execution unit 608A-608N being configured to provide one or more SMIDs for a thread. SPMD is a technique by which a single instruction is modified to include an operand that identifies data to be operated on by that instruction. The data is divided into a plurality of subsets, with each subset corresponding to a different thread. The execution units 608A-608N operate on the different subsets of data in parallel. The execution units 608A-608N can be configured for use in 32 single instruction multiple data (SIMD) or vector processing operations, 64 SIMIDs, or other targeted SIMD or vector length.

[0085] Each of the execution units 608A-608N operates on arrays of data elements. The number of data elements is the "execution size," or the number of channels that the instruction operates on. An execution channel is a logical unit of execution for data element access, masking, and flow control. The number of channels may

[0086] The execution unit instruction set includes SIMD instructions. Various data elements can be stored in registers and the execution units will process data in 64-bit, 128-bit, 256-bit, and 512-bit wide data registers, which can contain 1x64 bit, 2x32 bit, 4x16 bit, or 8x8 bit data elements. Each execution lane can follow its own control and data flow, allowing independent processing operations.

[0087] In one embodiment, one or more execution units can be combined into a fused execution unit 609A-609N that has thread control logic (607A-607N) common to the fused EU. Multiple EU's can be fused into an EU group. Each EU in the fused EU group can be configured to execute a separate SIMD hardware thread. The number of EU's in a fused EU group can vary from one embodiment to another. In addition, different SIMD widths can be executed per EU including, but not limited to, SIMD8, SIMD16, and SIMD32. Each fused graphics execution unit 609A-609N comprises at least two execution units. For example, fused execution unit 609A comprises a first EU 608A, a second EU 608B, and thread control logic 607A common to first EU 608A and second EU 608B. Thread control logic 607A controls threads executing on fused graphics execution unit 609A, allowing each EU within fused execution units 609A-609N to use a common instruction pointer register to be executed.

[0088] One or more internal instruction caches (e.g., 606) are included in the thread execution logic 600 to cache thread instructions for the execution units. In some embodiments, one or more data caches (e.g., 612) are included to cache thread data during thread execution. In some embodiments, a sampler 610 is included to provide texture sampling for 3D operations and to provide media sampling for media operations. In some embodiments, the sampler 610 includes specialized texture or media sampling functionality to handle texture or media data before it is provided to the execution units during a sampling process.

[0089] During execution, the graphics and media pipeline sends thread initiation requests to the thread execution logic 600 via thread generation and dispatch logic. Once a set of geometry objects has been processed and rasterized into pixel data, pixel processor logic (e.g., pixel shader logic, fragment shader logic, etc.) within the shader processor 602 is invoked to further calculate output information and cause results to be written to an output surface (e.g., color buffer, depth buffer, stencil buffer, etc.). In some embodiments, the pixel or fragment shader calculates values for various vertex attributes that are interpolated across the rasterized object. In some embodiments, the pixel processor logic within the shader processor 602 then executes an application programming interface (API) supplied pixel or fragment shader program. To execute the shader program, the shader processor 602 dispatches threads to execution units (e.g., 608A) via the thread dispatcher 604. In some embodiments, the shader processor 602 uses texture sampling logic in the sampler 610 to access texture data stored in a texture map stored in memory. Arithmetic operations on the texture data and input geometry data calculate pixel color data for each geometric fragment, or discard one or more pixels without further processing.

[0090] In some embodiments, the data port 614 provides a memory access mechanism for the thread execution logic 600 to output processed data to memory for further processing on a graphics processor output pipeline. In some embodiments, the data port 614 includes or is coupled to one or more cache memories (e.g., data cache 612) to cache data for memory access via the data port.

[0091] As shown in FIG. 6B The graphics execution unit 608 can include, in one embodiment, an instruction fetch unit 637, a general register file array (GRF) 624, an architecture register file array (ARF) 626, a thread arbiter 622, an issue unit 630, a branch unit 632, a set of SIMD floating point units (FPUs) 634, and in one embodiment, a set of dedicated integer SIMD ALUs 635. The GRF 624 and ARF 626 include the set of general and architecture register files associated with each synchronized hardware thread that can be active in the graphics execution unit 608. In one embodiment, per-thread architecture state is maintained in the ARF 626, while data used during thread execution is stored in the GRF 624. The execution state of each thread, including the instruction pointer of each thread, can be held in thread-specific registers in the ARF 626.

[0092] In one embodiment, graphics processing unit 608 has an architecture that is a combination of simultaneous multi-threading (SMT) and fine-grained interleaved multi-threading (IMT). The architecture has a modular configuration that can be tuned at design time based on a target number of simultaneous threads and a target number of registers per execution unit, in which execution unit resources are partitioned across logic used to execute multiple simultaneous threads.

[0093] In one embodiment, graphics processing unit 608 can issue multiple instructions collectively that can each be different instructions. Thread arbiter 622 of graphics processing unit thread 608 can dispatch instructions to one of send unit 630, branch unit 642, or SIMD FPU(s) 634 for execution. Each execution thread has access to 128 general purpose registers within GRF 624, where each register can store 32 bytes that can be accessed as a SIMD 8-element vector with 32-bit data elements. In one embodiment, each execution unit thread has access to 4 kilobytes within GRF 624, although embodiments are not so limited, and more or less register resources can be provided in other embodiments. In one embodiment, up to seven threads can execute synchronously, although the number of threads per execution unit can vary according to embodiments. In an embodiment in which seven threads have access to 4 kilobytes, GRF 624 can store a total of 28 kilobytes. Flexible addressing modes can permit multiple registers to be addressed together, efficiently establishing wider registers or representing stride rectangular block data structures.

[0094] In one embodiment, memory operations, sampler operations, and other longer latency system communications are dispatched via “send” instructions executed by send unit 630 over a message passing fabric. In one embodiment, branch instructions are dispatched to a dedicated branch unit 632 to facilitate SIMD divergence and eventual convergence.

[0095] In one embodiment, graphics processing unit 608 includes one or more SIMD floating point units (FPUs) 634 to perform floating point operations. In one embodiment, FPU(s) 634 also support integer computation. In one embodiment, FPU(s) 634 can SIMD execute up to a number M of 32-bit floating point (or integer) operations, or up to 2M of 16-bit integer or 16-bit floating point operations. In one embodiment, at least one of FPU(s) 634 provides an extended math capability that supports high throughput transcendental math functions and double precision 64-bit floating point. In some embodiments, a set of 8-bit integer SIMD ALUs 635 also represent and can be specifically optimized to perform operations associated with machine learning computations.

[0096] In one embodiment, an array of instances of the graphics execution unit 608 can be instantiated in a graphics sub-core (e.g., a sub-slice) for graphics processing. In one embodiment, each graphics sub-core shares one instance of the graphics execution unit 608. In another embodiment, multiple instances of the graphics execution unit 608 are provided for graphics processing. For scalable performance, product architectures can include different amounts of execution units 608 for each sub-core. In one embodiment, the product architecture can include a number of execution units 608 for each sub-core, where the number of execution units 608 is based on the performance and power considerations.

[0097] FIG. 7 FIG. 7 is a block diagram illustrating a graphics processor instruction format 700 according to some embodiments. In one or more embodiments, a graphics processor execution unit supports an instruction set that includes a number of different types of instructions. The solid lined boxes illustrate the components that are generally included in an execution unit or instruction. The graphics processor is a pipelined architecture that processes instruction packets in a graphics processing pipeline. A front-end retrieves the packets from a queue and decodes them. Packets flow through the graphics processing pipeline with different instructions being processed at different stages. The graphics processor executes the decoded instructions by operating on the data elements in the graphic processing pipeline. The solid lined boxes represent the components that are generally included in an execution unit or instruction packet. The graphics processor is a pipelined architecture that processes instruction packets in a graphics processing pipeline. A front-end retrieves the packets from a queue and decodes them. Packets flow through the graphics processing pipeline with different instructions being processed at different stages. The graphics processor executes the decoded instructions by operating on the data elements in the graphic processing pipeline. The solid lined boxes represent the components that are generally included in an execution unit or instruction packet.

[0098] In some embodiments, a graphics processor execution unit natively supports instructions in a 128-bit instruction format 710. Some instructions are compacted from the 128-bit instruction format 710 into a 64-bit compact instruction format 730 based on a set of instruction options and number of operands. The native 128-bit instruction format 710 provides access to all instruction options for the graphics processor execution units. Some instruction options and operations are restricted to the 64-bit compact instruction format 730. Not all instruction options for an execution unit are supported in the 64-bit instruction format 730. In some embodiments, a set of index values in the instruction's index field 713 are used to partially compress the instruction. The graphics processor execution unit hardware references a set of compression tables based on the index values in the instruction. Values in the compression tables are used to refiector the native 128-bit instruction format 710 from the compacted 64-bit instruction format 730. The reference to the compression table is determined during the decoding of the instruction packet. Whether to use the compression table is a programmatic choice the programmer makes as part of the development of the graphics software.

[0099] For each format, the instruction opcode 712 defines the operation that the execution unit is to perform. The execution units execute each instruction in parallel across the data elements of each operand. For example, in response to an add instruction, the execution units perform a simultaneous add operation across each color channel for the operands. Many instructions are performed on all data channels of each operand, but in some embodiments, there is support for integer and memory data types that can be operated on in parallel across the data elements of a group of operands. The instruction control field 714 for the graphics processor execution logic 700 enables control over a number of execution options, including control over data channels with bit 0 enabling a first data channel and bit 1 enabling a second data channel (not all bits are used in all implementations). The instruction control field 714 also includes a number of other bit fields that control various other features of the graphics processor execution units, including bit 2 to control the number of threads executed per clock, bit 3 to control data address masking, bit 4 to indicate integer or floating point data, bit 5 to indicate a move instruction, bit 6 to indicate a cache flush, bit 7 to indicate a cache invalidate, bit 8 to indicate a bind into a graphics processor thread, bit 9 to indicate a thread spawning, bit 10 to indicate a thread halt, bit 11 to indicate a thread kill, and bit 12 to indicate a method call. In some embodiments, the graphics processor execution units support a like call for the thread operation control field 714 bit 11.

[0100] Some execution unit instructions have up to three operands, including two source operands (src0 720, src1 722) and one destination 718. In some embodiments, the execution unit supports dual destination instructions in which one of the destinations is implicit. Data operation instructions can have a third source operand (e.g., SRC2 724), where the instruction opcode 712 determines the number of source operands. The last source operand of an instruction can be an immediate (e.g., hard coded) value passed with the instruction.

[0101] In some embodiments, the 128-bit instruction format 710 includes an access / address mode field 726 that specifies, for example, whether to use a direct register addressing mode or an indirect register addressing mode. When using the direct register addressing mode, the register address for one or more operands is provided directly by bits in the instruction.

[0102] In some embodiments, the 128-bit instruction format 710 includes an access / address mode field 726 that specifies the address mode and / or access mode of the instruction. In one embodiment, the access mode is used to qualify data access alignment for the instruction. Some embodiments support access modes including a 16-byte aligned access mode and a 1-byte aligned access mode, where the byte alignment of the access mode determines the access alignment of the instruction operands. For example, when in a first mode, the instruction can use byte-aligned addressing for source and destination operands, and when in a second mode, the instruction can use 16-byte aligned addressing for all source and destination operands.

[0103] In one embodiment, the address mode portion of the access / address mode field 726 determines whether the instruction uses direct addressing or indirect addressing. When using the direct register addressing mode, the register address for one or more operands is provided directly by bits in the instruction. When using the indirect register addressing mode, the register address for one or more operands can be calculated based on an address register value and an address immediate field in the instruction.

[0104] In some embodiments, instructions are grouped based on the opcode 712 bit field to simplify opcode decoding 740. For 8-bit opcodes, bits 4, 5, and 6 allow the execution units to determine the type of opcode. The exact opcode grouping shown is exemplary only. In some embodiments, the move and logic opcode group 742 includes data move and logic instructions (e.g., move (mov), compare (cmp)). In some embodiments, the move and logic group 742 shares five most significant bits (MSBs), with move (mov) instructions taking the form 0000xxxxb and logic instructions taking the form 0001xxxxb. The flow control instruction group 744 (e.g., call, jmp) includes instructions that take the form 0010xxxxb (e.g., Ox20). The miscellaneous instruction group 746 includes a mix of instructions, including synchronization instructions (e.g., wait, send) that take the form 0011xxxxb (e.g., Ox30). The parallel math instruction group 748 includes per-component arithmetic instructions (e.g., add, mul) that take the form 0100xxxxb (e.g., Ox40). The parallel math group 748 performs arithmetic operations in parallel across data lanes. The vector math group 750 includes arithmetic instructions (e.g., dp4) that take the form 0101xxxxb (e.g., Ox50). The vector math group performs arithmetic operations on vector operands, such as a dot product operation.

[0105] Graphics Pipeline

[0106] FIG. 8 is a block diagram of another embodiment of a graphics processor 800. FIG. 8 Those elements of having the same reference number (or name) in the figures herein as elements in any other figure herein can operate or function in any manner similar to that described elsewhere herein, but are not limited to that. Those elements of having the same reference number (or name) in the figures herein as elements in any other figure herein can operate or function in any manner similar to that described elsewhere herein, but are not limited to that.

[0107] In some embodiments, the graphics processor 800 includes a geometry pipeline 820, a media pipeline 830, a display engine 840, thread execution logic 850, and a render output pipeline 870. In some embodiments, graphics processor 800 is a graphics processor included in a multi-core processor that includes one or more general-purpose processing cores. The graphics processor is controlled by register writes to one or more control registers (not shown) or via commands issued to the graphics processor 800 via the ring interconnect 802. In some embodiments, ring interconnect 802 couples graphics processor 800 to other processing components such as other graphics processors or general-purpose processors. Commands from the ring interconnect 802 are interpreted by the command streamer 803, which supplies instructions to the individual components of graphics processor 800.

[0108] In some embodiments, the command streamer 803 directs the operation of the vertex fetcher 805, which reads vertex data from memory and executes vertex processing commands provided by the command streamer 803. In some embodiments, the vertex fetcher 805 provides vertex data to a vertex shader 807, which performs coordinate space transformation and lighting operations on each vertex. In some embodiments, the vertex fetcher 805 and vertex shader 807 execute vertex processing instructions by dispatching execution threads to execution units 852A-B via thread dispatcher 831.

[0109] In some embodiments, the execution units 852A-B are vector processors having instruction sets specifically configured to perform graphics and media operations. In some embodiments, the execution units 852A-B have attached Ll caches 851, which are dedicated to each array or shared between arrays. The caches can be configured as data caches, instruction caches, or single caches that are partitioned into different regions for data and instructions.

[0110] In some embodiments, the geometry pipeline 820 includes a tessellation component for performing hardware-accelerated tessellation of 3D objects. In some embodiments, a programmable hull shader 811 configures tessellation operations. A programmable domain shader 817 provides post-processing of tessellation output. A tessellator 813 operates in the direction of the hull shader 811 and contains specialized logic for generating a detailed set of geometric objects based on a coarse geometric model provided as input to the geometry pipeline 820. In some embodiments, the tessellation component (e.g., hull shader 811, tessellator 813, domain shader 817) can be bypassed if tessellation is not used.

[0111] In some embodiments, complete geometric objects can be processed by a geometry shader 819 via one or more threads dispatched to the execution units 852A-B, or can pass directly to the clipper 829. In some embodiments, the geometry shader operates on entire geometric objects (rather than vertices or vertex patches as in previous stages of the graphics pipeline). If tessellation is disabled, the geometry shader 819 receives input from the vertex shader 807. In some embodiments, the geometry shader 819 can be programmed by a geometry shader program to perform geometric tessellation when the tessellation unit is disabled.

[0112] The clipper 829 processes vertex data prior to rasterization. The clipper 829 can be a fixed function clipper or a programmable clipper with clip and geometry shader functionality. In some embodiments, the rasterizer and depth test components 873 in the render output pipeline 870 dispatch pixel shaders to convert a geometric object into a per-pixel representation. In some embodiments, the pixel shader logic is included in the thread execution logic 850. In some embodiments, the application can bypass the rasterizer and depth test components 873 and access un-rasterized vertex data via the outflow unit 823.

[0113] The graphics processor 800 has an interconnect bus, interconnect fabric or some other interconnect mechanism to allow data and messages to be passed between components of the graphics processor, including the main unit, memory unit, and the graphics processing array. In some embodiments, the execution units 852A-852B and the associated logic units (e.g., LI cache 851, sampler 854, texture cache 858, etc.) are interconnected via a data port 856 to perform memory accesses and communicate with other processor rendering output pipeline components. In some embodiments, the sampler 854, caches 851, 858, and execution units 852A-852B each have separate memory access ports to the data port 856.

[0114] In some embodiments, the render output pipeline 870 includes a rasterizer and depth test component 873 that converts based on vertex data into associated pixel-based representations. In some embodiments, the rasterizer logic includes a windower / masker unit for performing fixed function triangle and line rasterization. An associated render cache 878 and depth cache 879 are also available in some embodiments. Pixel operation components 877 perform pixel-based operations on the data, although in some instances pixel operations associated with 2D operations (e.g., bit block image transfers with blending) are performed by the 2D engine 841 or replaced at display time by the display controller 843 using an overlay display plane. In some embodiments, a shared L3 cache 875 is available for all graphics components, allowing sharing of data without use of main system memory.

[0115] In some embodiments, graphics processor media pipeline 830 includes a media engine 837 and a video front-end 834. In some embodiments, video front-end 834 receives pipeline commands from the command streamer 803. In some embodiments, media pipeline 830 includes a separate command streamer. In some embodiments, video front-end 834 processes media instructions before sending the media instructions to media engine 837. In some embodiments, media engine 837 includes thread spawning functionality to process media instructions.

[0116] In some embodiments, graphics processor 800 includes a display engine 840. In some embodiments, display engine 840 is external to processor 800 and couples with the graphics processor via the ring interconnect 802, or some other interconnect bus or fabric. In some embodiments, display engine 840 includes a 2D engine 841 and a display controller 843. In some embodiments, display engine 840 contains special purpose logic that is configured to operate independently of the 3D pipeline. In some embodiments, display controller 843 couples with a display device (not illustrated) that can be a system integrated display device, as in a laptop computer, or an external display device attached via an display device connector.

[0117] In some embodiments, geometry pipeline 820 and media pipeline 830 can be configured to perform operations based on a number of graphics and media programming interfaces. In some embodiments, driver software for the graphics processor translates API calls received by the graphics processor into commands

[0118] Graphics Pipeline Programming

[0119] FIG. 9A is a block diagram illustrating a graphics processor command format 900 according to some embodiments. FIG. 9B is a block diagram illustrating a graphics processor command sequence 910 according to an embodiment. FIG. 9AThe solid lined boxes illustrate the components typically that are included in a graphics command while the dashed lined boxes illustrate optional components that can or can not be included in the graphics command. FIG. 9A The exemplary graphics processor command format 900 includes a data field for identifying a client 902, a command operation code (opcode) 904, and data for the command 906. Some commands also include a sub-opcode 905 and a command size 908.

[0120] In some embodiments, the client 902 specifies a client unit of the graphics device that will process the command data. In some embodiments, a graphics processor command parser examines the client field of each command to direct further processing of the command and to route the command data to the appropriate client unit. In some embodiments, a graphics processor client unit includes a memory interface unit, a rendering unit, a 2D unit, a 3D unit, and a media unit. Each client unit has a respective processing pipeline that processes the commands. Once a command is received by a client unit, the client unit reads the opcode 904 and the sub-opcode 905 (if present) to determine the operation to perform. The client unit uses information in the data field 906 to perform the command. For some commands, an explicit command size 908 is expected to define the size of the command. In some embodiments, the command parser automatically determines the size of at least some of the commands based on the command opcode. In some embodiments, commands are aligned via multiples of a doubleword.

[0121] FIG. 9B The flowchart in FIG. 10 illustrates an exemplary graphics processor command sequence 910. In some embodiments, a software or firmware of a data processing system featuring an embodiment of a graphics processor uses a version of the command sequence shown to initiate, perform, and terminate a set of graphics operations. A sample command sequence is shown and described for illustrative purposes only, as embodiments are not limited to these specific commands or to this command sequence. Moreover, the commands can be issued as a batch of commands in a command sequence, such that a graphics processor will process the command sequence in at least partially simultaneous fashion.

[0122] In some embodiments, the graphics processor command sequence 910 can begin with a pipeline flush command 912 to ensure that any active pipeline does not complete pending commands. In some embodiments, the 3D pipeline 922 and media pipeline 924 are not operating at the same time. A pipeline flush can be performed to ensure that the active pipeline completes any uncompleted commands before the other pipeline is allowed to operate. The pipeline flush can be followed by a pipeline context save command 913 to save the pipeline application context so that these contexts can be restored for the next pipeline operation. In some embodiments, the graphics processor core 101 is a dual pipeline core, and the pipeline context save and restore commands are used to context switch between the two pipelines. In some embodiments, following the pipeline flush, the graphics processor core 101 is placed in an inactive state.

[0123] In some embodiments, when the command sequence requires the graphics processor to explicitly switch between processing tasks on the pipeline, a pipeline select command 913 is used. In some embodiments, only one pipeline selection command 913 is needed in the sequence for each pipeline switch. In some embodiments, the pipeline flush command 912 is only required at the beginning of the command sequence when the application pipeline context save command is to be issued.

[0124] In some embodiments, pipeline control commands 914 configure the graphics pipeline for operation and set pipeline parameters. In some embodiments, pipeline control commands 914 configure the pipeline state for the active pipeline. In one embodiment, pipeline control commands 914 are used to set pipeline state for the pipeline being operated on. In some embodiments, pipeline control commands 914 are used to set the pipeline state for one or more pipes within the active pipeline. In some embodiments, pipeline control commands 914 are used to set the pipeline state for the active pipeline and one or more inactive pipelines. In some embodiments, the pipeline control commands 914 are used to set the pipeline state for all pipelines.

[0125] In some embodiments, return buffer state commands 916 are used to configure a set of return buffers for a corresponding pipeline to write data to. Some pipeline operations require allocation, selection, or configuration of one or more return buffers that the operation will write intermediate data into during processing. In some embodiments, the graphics processor also uses one or more return buffers to store output data and to perform cross-thread communication. In some embodiments, the return buffer state 916 includes selection of a size and number of return buffers for a set of pipeline operations.

[0126] The remaining commands in the command sequence vary depending on the active pipeline for operation. Based on the pipeline determination 920, the command sequence is customized for the 3D pipeline 922 beginning at 3D pipeline state 930, or for the media pipeline 924 beginning at media pipeline state 940.

[0127] Commands used to configure 3D pipeline state 930 include 3D state setting commands for vertex buffer state, vertex element state, constant color state, depth buffer state, and other state variables to be configured prior to processing 3D primitive commands. The values for these commands are determined based at least in part on the particular 3D API in use. In some embodiments, 3D pipeline state 930 commands can also selectively disable or bypass certain pipeline elements if those elements will not be used.

[0128] In some embodiments, 3D primitive 932 commands are used to submit 3D primitives to be processed by the 3D pipeline. The commands and associated parameters passed to the graphics processor via 3D primitive 932 commands are forwarded to a vertex fetch function in the graphics pipeline. The vertex fetch function uses the 3D primitive 932 command data to generate a plurality of vertex data structures. The vertex data structures are stored in one or more return buffers. In some embodiments, 3D primitive 932 commands are used to perform vertex operations on 3D primitives via a vertex shader. To process the vertex shader, 3D pipeline 922 dispatches shader execution threads to graphics processor execution units.

[0129] In some embodiments, 3D pipeline 922 is triggered via an execute 934 command or event. In some embodiments, a register write triggers command execution. In some embodiments, execution is triggered via a 'go' or 'kick' command in a command sequence. In one embodiment, a pipeline synchronization command is used to trigger command execution in order to flush a command sequence through the graphics pipeline. The 3D pipeline will perform geometry processing for 3D primitives. Once the operations are complete, the resulting geometry is rasterized, and a pixel engine shades the resulting pixels. Additional commands to control pixel shading and pixel back-end operations can also be included for these operations.

[0130] In some embodiments, graphics processor command sequence 910 follows the media pipeline 924 path when performing media operations. In general, the specific use and manner of programming for media pipeline 924 depends on the media or compute operations to be performed. In media decode processes, specific media decode operations can be offloaded to the media pipeline. In some embodiments, the media pipeline can also be bypassed and media decode can be performed in whole or in part using resources provided by one or more general purpose processing cores. In one embodiment, the media pipeline also includes elements for general purpose graphics processor unit (GPGPU) operations, where the graphics processor is used to perform SIMD vector operations using a compute shader program that is not explicitly related to the rendering of a graphics primitive.

[0131] In some embodiments, the media pipeline 924 is configured in a similar manner as the 3D pipeline 922. A set of commands to configure the media pipeline state 940 is dispatched or placed into the command queue, prior to the media object command 942. In some embodiments, the commands 940 for the media pipeline state include data to configure the media pipeline elements that will be used to process the media object. This includes data to configure the video decode and video encode logic within the media pipeline, such as the encoding or decoding format. In some embodiments, the commands 940 for the media pipeline state also support the use of one or more pointers to "indirect" state elements that contain a batch of state settings.

[0132] In some embodiments, the media object command 942 supplies a pointer to a media object for processing by the media pipeline. The media object includes a memory buffer that contains video data to be processed. In some embodiments, all of the media pipeline state must be valid prior to issuing the media object command 942. Once the pipeline state is configured and the media object command 942 is queued, the media pipeline 924 is triggered via an execute 944 command or equivalent execution event (e.g., register write). The output from the media pipeline 924 can then be post-processed by operations provided by the 3D pipeline 922 or the media pipeline 924. In some embodiments, GPGPU operations are configured and executed in a similar manner as media operations.

[0133] Graphics Software Architecture

[0134] FIG. 10 An exemplary graphics software architecture of a data processing system 1000 is shown in accordance with some embodiments. In some embodiments, the software architecture includes a 3D graphics application 1010, an operating system 1020, and at least one processor 1030. In some embodiments, the processor 1030 includes a graphics processor 1032 and one or more general-purpose processor cores 1034. The graphics application 1010 and the operating system 1020 each execute in a system memory 1050 of the data processing system.

[0135] In some embodiments, the 3D graphics application 1010 contains one or more shader programs including shader instructions 1012. The shader language instructions can be in a high-level shader language, such as the High-Level Shader Language (HLSL) or the OpenGL Shader Language (GLSL). The application also includes executable instructions 1014 in a machine language suitable for execution by the general-purpose processor cores 1034. The application also includes graphics objects 1016 defined by vertex data.

[0136] In some embodiments, operating system 1020 is a version of Windows® from Microsoft Corporation Operating system, a special UNIX®-based operating system, or an open source UNIX®-based operating system using a Linux kernel variant. Operating system 1020 can support a graphics API 1022, such as a Direct3D API, an OpenGL API, or a Vulkan API. When the Direct3D API is in use, operating system 1020 uses a front-end shader compiler 1024 to compile any shader instructions 1012 in HLSL into a lower-level shader language. The compilation can be a just-in-time (JIT) compilation or the application can perform shader pre- compilation. In some embodiments, high-level shaders are compiled into low-level shaders in the process of compiling 3D graphics application 1010. In some embodiments, shader instructions 1012 are provided in an intermediate form, such as versions of the Standard Portable Intermediate Representation (SPIR) used by the Vulkan API.

[0137] In some embodiments, user mode graphics driver 1026 contains a back-end shader compiler 1027 to translate shader instructions 1012 into representations for a hardware- specific use. When the OpenGL API is in use, shader instructions 1012 in GLSL high-level language are passed to user mode graphics driver 1026 for compilation. In some embodiments, user mode graphics driver 1026 uses operating system kernel mode functions 1028 to communicate with kernel mode graphics driver 1029. In some embodiments, kernel mode graphics driver 1029 communicates with graphics processor 1032 in order to dispatch

[0138] IP Core Implementation

[0139] One or more aspects of at least one embodiment can be implemented by representative code stored on a machine-readable medium which represents and / or defines logic utilizing a plurality of lines of code. For example, the machine- readable medium can include a plurality of lines of code that represent various logical operations employed by a processor to implement techniques described herein. Such representations, known as "IP cores" can be stored on a tangible, machine-readable medium and supplied to various customers or manufacturing facilities to load into the fabrication machines that make the integrated circuit. A machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, a machine-readable medium includes magnetic storage media (e.g., magnetic disks); optical storage media (e.g., optical disks); electrical storage media (e.g., solid-state

[0140] FIG. 11A is a block diagram illustrating an IP core development system 1100 that can be used to fabricate integrated circuits to perform operations according to embodiments. The IP core development system 1100 can be used to generate modular, reusable designs that can be incorporated into larger designs or used to construct an entire integrated circuit (e.g., an SOC integrated circuit). A design facility 1130 can employ a high-level programming language (e.g., C / C++) to generate a software simulation 1110 of the IP core design. The software simulation 1110 can be used to design, test, and verify the behavior of the IP core using a simulation model 1112. The simulation model 1112 can include functional, behavioral, and / or timing simulations. The register transfer level (RTL) design 1115 can then be created or synthesized from the simulation model 1112. The RTL design 1115 is an abstraction of the behavior of the integrated circuit (including associated logic executed by the modeled digital signals) that models the flow of digital signals between hardware registers, including the associated logic performed thereon. In addition to an RTL design 1115, a lower-level design, such as a logic level or transistor level design, can also be created, designed, or synthesized.

[0141] The RTL design 1115, or equivalent, can be further synthesized, created, or fabricated into a hardware model 1120 by the design facility, which can employ a hardware description language (HDL) or some other representation of the design. The HDL can be further simulated or tested to verify the IP core design. The IP core design can be stored using non-volatile memory 1140 (e.g., hard disk, flash, or any non-volatile storage medium) for delivery to a third party fabrication facility 1165. Alternatively, the IP core design can be transmitted (e.g., via the Internet) over a wired 1150 or wireless 1160 connection. The fabrication facility 1165 can then fabricate an integrated circuit based at least in part on the IP core design. The fabricated integrated circuit can be configured to perform operations in accordance with at least one embodiment described herein.

[0142] FIG. 11BA cross-sectional side view of an integrated circuit package assembly 1170 is shown in accordance with some embodiments described herein. The integrated circuit package assembly 1170 shows an implementation of one or more processor or accelerator devices as described herein. The package assembly 1170 includes multiple hardware logic units 1172, 1174 connected to a substrate 1180. The logic 1172, 1174 can be implemented at least partially in configurable logic or fixed function logic hardware and can include one or more portions of processor core(s), graphics processor(s), or any of the other accelerator devices described herein. Each logic unit 1172, 1174 can be implemented within a semiconductor die and coupled with the substrate 1180 via an interconnect structure 1173. The interconnect structure 1173 can be configured to route electrical signals between the logic 1172, 1174 and the substrate 1180 and can include interconnects such as, but not limited to, bumps or pillars. In some embodiments, the interconnect structure 1173 can be configured to route electrical signals such as, for example, input / output (I / O) signals associated with the operation of the logic 1172, 1174 and / or power or ground signals. In some embodiments, the substrate 1180 is an epoxy-based laminate substrate. In other embodiments, the package substrate 1180 can include other suitable types of substrates. The package assembly 1170 can be connected to other electrical devices via a package interconnect 1183. The package interconnect 1183 can be coupled to a surface of the substrate 1180 in order to route electrical signals to other electrical devices such as a motherboard, other chipsets, or multi-chip modules.

[0143] In some embodiments, the logic units 1172, 1174 are electrically coupled with a bridge 1182 that is configured to route electrical signals between the logic 1172, 1174. The bridge 1182 can be a dense interconnect structure that provides routing for electrical signals. The bridge 1182 can include a bridge substrate composed of glass or a suitable semiconductor material. Circuitry features can be formed on the bridge substrate in order to provide chip-to-chip connections between the logic 1172, 1174.

[0144] Although two logic units 1172, 1174 and a bridge 1182 are shown, embodiments described herein can include more or fewer logic units on one or more dies. The one or more dies can be connected by zero or more bridges, as the bridge 1182 can be excluded when the logic is included on a single die. Alternatively, multiple dies or logic units can be connected by one or more bridges. Additionally, multiple logic units, dies, and bridges can be connected together in other possible configurations including three-dimensional configurations.

[0145] Exemplary System on a Chip Integrated Circuit

[0146] FIGS. 12-1 4 illustrates exemplary integrated circuits and related graphics processors that can be fabricated using one or more IP cores in accordance with various embodiments described herein. In addition to the graphics processors illustrated, other logic and circuits can also be included, including additional graphics processors / cores, peripheral interface controllers or general purpose processor cores.

[0147] FIG. 12 is a block diagram illustrating an exemplary system on a chip integrated circuit 1200 that can be fabricated using one or more IP cores in accordance with embodiments. The exemplary integrated circuit 1200 includes one or more application processors 1205 (e.g., CPUs), at least one graphics processor 1210, and can additionally include an image processor 1215 and / or a video processor 1220, any of which can be a modular IP core from the same or multiple different design facilities. The integrated circuit 1200 includes peripheral or bus logic including USB controllers 1225, UART controllers 1230, SPI / SDIO controllers 1235, and I2S / I2C controllers 1240. In addition, the integrated circuit can also include a display device 1245 coupled to one or more of a high-definition multimedia interface (HDMI) controller 1250 and a mobile industry processor interface (MIPI) display interface 1255. Storage can be provided by a flash memory subsystem 1260, including flash memory and a flash memory controller. Memory interfaces can be provided via a memory controller 1265 to access SDRAM or SRAM memory devices. In addition, some integrated circuits include an embedded security engine 1270. 2 S / I 2 C controller 1240. In addition, the integrated circuit can also include a display device 1245 coupled to one or more of a high-definition multimedia interface (HDMI) controller 1250 and a mobile industry processor interface (MIPI) display interface 1255. Storage can be provided by a flash memory subsystem 1260, including flash memory and a flash memory controller. Memory interfaces can be provided via a memory controller 1265 to access SDRAM or SRAM memory devices. In addition, some integrated circuits include an embedded security engine 1270.

[0148] FIGS. 13A-13B is a block diagram illustrating an exemplary graphics processor for use within a SoC in accordance with embodiments described herein. FIG. 13A illustrates an exemplary graphics processor 1310 of a system on a chip integrated circuit that can be fabricated using one or more IP cores in accordance with embodiments. FIG. 13B illustrates an additional exemplary graphics processor 1340 of a system on a chip integrated circuit that can be fabricated using one or more IP cores in accordance with embodiments. FIG. 13A The graphics processor 1310 of is an example of a low power graphics processor core. FIG. 13B The graphics processor 1340 of is an example of a higher performance graphics processor core. Each of the graphics processors 1310, 1340 can be a modular IP core that can be FIG. 12 a variant of the graphics processor 1210.

[0149] As FIG. 13AAs shown in FIG, the graphics processor 1310 includes a vertex processor 1305 and one or more fragment processors 1315A to 1315N (e.g., 1315A, 1315B, 1315C, 1315D, all the way to 1315N-1 and 1315N). The graphics processor 1310 can execute different shader programs via separate logic, such that the vertex processor 1305 is optimized to perform the operations of the vertex shader program, while the one or more fragment processors 1315A to 1315N perform fragment (e.g., pixel) shading operations for the fragment or pixel shader program. The vertex processor 1305 executes the vertex processing stage of the 3D graphics pipeline and generates primitives and vertex data. The fragment processor(s) 1315A to 1315N use the primitives and vertex data generated by the vertex processor 1305 to generate a frame buffer for display on a display device. In one embodiment, the fragment processor(s) 1315A through 1315N are optimized to execute fragment shader programs provided in the OpenGL API, which can be used to perform similar operations as pixel shader programs provided in the Direct 3D API.

[0150] In addition, the graphics processor 1310 also includes one or more memory management units (MMUs) 1320A-1320B, one or more caches 1325A-1325B, and one or more circuit interconnects 1330A-1330B. The one or more MMUs 1320A-1320B provide virtual-to-physical address mappings for the graphics processor 1310, including for the vertex processor 1305 and / or (multiple) fragment processors 1315A-1315N, and the virtual-to-physical address mappings can also reference vertex or image / texture data stored in memory in addition to the vertex or image / texture data stored in the one or more caches 1325A-1325B. In one embodiment, the one or more MMUs 1320A-1320B can communicate with the system, including those within the system. FIG. 12 The graphics processor 1310 may be synchronized with other MMUs, including one or more MMUs associated with the one or more application processors 1205, the image processor 1215, and / or the video processor 1220, so that each processor 1205 to 1220 may participate in a shared or unified virtual memory system. In accordance with an embodiment, the one or more circuit interconnects 1330A to 1330B enable the graphics processor 1310 to interact with other IP cores within the SoC via an internal bus of the SoC or via a direct connection.

[0151] like FIG. 13B As shown in FIG, the graphics processor 1340 includes FIG. 13Athe one or more MMUs 1320A-B, caches 1325A-B, and circuit interconnect 1330A-B of the graphics processor 1310. The graphics processor 1340 includes one or more shader cores 1355A-1355N (e.g., 1455A, 1355B, 1355C, 1355D, 1355E, 1355F, up to 1355N-1 and 1355N) that provide a unified shader core architecture in which a single core or type or core can execute all types of programmable shader code, including shader program code to implement vertex shaders, fragment shaders, and / or compute shaders. The exact number of shader cores present can vary in an embodiment and implementation. Additionally, the graphics processor 1340 includes an inter-core task manager 1345 that acts as a thread dispatcher and task manager to accelerate tasks with thread dispatch granularity that is coarser than an individual thread, and to manage a number of on-chip or on-processor shared resources. In cases where tasks are threaded at a level finer than an individual thread, the inter-core task manager 1345 can dispatch multiple threads to the graphics processor 1340 per graphics processor clock cycle. The inter-core task manager 1345 can also act as a resource manager to manage shared resources such as caches, arbitration for the memory bus, thread dispatch, and submission front-ends.

[0152] FIGS. 14A-14B Additional exemplary graphics processor logic in accordance with embodiments described herein is shown. FIG. 14A A graphics core 1400 is shown that can be included within a graphics processor 1210 as in FIG. 1 and can be as in FIG. 2. FIG. 12 The graphics processor 1210 of FIG. 1. FIG. 13B The unified shader cores 1355A-1355N in FIG. 2. FIG. 14B A highly parallel general purpose graphics processing unit 1430 suitable for deployment on a multi-chip module is shown.

[0153] As in FIG. 2 FIG. 14AAs shown, the graphics core 1400 includes a shared instruction cache 1402, a texture unit 1418, and a cache memory / shared memory 1420 common to the execution resources within the graphics core 1400. The graphics core 1400 can include multiple slices 1401A-1401N or partitions for each core, and the graphics processor can include multiple instances of the graphics core 1400. The slices 1401A-1401N can include support logic including a local instruction cache 1404A-1404N, a thread scheduler 1406A-1406N, a thread dispatcher 1408A-1408N, and a set of registers 1410A. To perform

[0154] Some of these compute units operate at a specific precision. For example, the FPUs 1414A-1414N can perform single precision (32-bit) and half precision (16-bit) floating point operations, while the DPFPUs 1415A-1415N perform double precision (64-bit) floating point operations. The ALUs 1416A-1416N can perform variable precision integer operations at 8-bit precision, 16-bit precision, and 32-bit precision, and can be configured for mixed precision operations. The MPUs 1417A-1417N can also be configured for mixed precision matrix operations, including half precision floating point operations and 8-bit integer operations. The MPUs 1417A-1417N can perform a wide variety of matrix operations to accelerate machine learning application frameworks, including enabling support for accelerated general matrix to matrix multiplication (GEMM). The AFUs 1412A-1412N can perform additional logical operations not supported by the floating point units or integer units, including trigonometric operations (e.g., sine, cosine, etc.).

[0155] As FIG. 14BAs shown in , a general purpose processing unit (GPGPU) 1430 can be configured to enable highly parallel computing operations to be performed by an array of graphics processing units. In addition, GPGPU 1430 can be directly linked to other instances of GPGPU to create a multi-GPU cluster, thereby increasing the training speed of deep neural networks in particular. GPGPU 1430 includes a host interface 1432 for implementing a connection with a host processor. In one embodiment, host interface 1432 is a PCI Express interface. However, the host interface can also be a supplier-specific communication interface or communication structure. GPGPU 1430 receives commands from the host processor and uses a global scheduler 1434 to distribute the execution threads associated with those commands to a group of computing clusters 1436A to 1436H. Computing clusters 1436A to 1436H share cache memory 1438. Cache memory 1438 can act as a higher-level cache of the cache memory within computing clusters 1436A to 1436H.

[0156] GPGPU 1430 includes memory 1434A-1434B coupled to compute clusters 1436A-1436H via a set of memory controllers 1442A-1442B. In various embodiments, memory 1434A-1434B may include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory.

[0157] In one embodiment, computing clusters 1436A through 1436H each include a set of graphics cores, such as FIG. 14A The graphics core 1400 may include multiple types of integer logic units and floating-point logic units that can perform computational operations within a range of precision, including those suitable for machine learning computations. For example, and in one embodiment, at least a subset of the floating-point units in each of the compute clusters 1436A through 1436H may be configured to perform 16-bit or 32-bit floating-point operations, while a different subset of the floating-point units may be configured to perform 64-bit floating-point operations.

[0158] Multiple instances of GPGPU 1430 can be configured to operate as a compute cluster. The compute mechanism used by compute clusters to synchronize and exchange data varies across embodiments. In one embodiment, multiple instances of GPGPU 1430 communicate over host interface 1432. In one embodiment, GPGPU 1430 includes an I / O hub 1439 that couples the GPGPU 1430 with GPU links 1440 that enable direct connections to other instances of the GPGPU. In one embodiment, GPU links 1440 couple to specialized GPU-to-GPU bridges that enable communication and synchronization between multiple instances of GPGPU 1430. In one embodiment, GPU links 1440 couple with a high-speed interconnect to transmit and receive data to other GPGPUs or parallel processors. In one embodiment, multiple instances of GPGPU 1430 are located in separate data processing systems and communicate over a network device that is accessible via host interface 1432. In one embodiment, GPU links 1440 can be configured to enable connections to host processors in addition to, or as an alternative to, host interface 1432.

[0159] While the illustrated configuration of GPGPU 1430 can be configured to train neural networks, one embodiment provides an alternative configuration of GPGPU 1430 that can be configured to be deployed within a high-performance or low-power inference platform. In an inference configuration, GPGPU 1430 includes fewer compute clusters of compute clusters 1436A-1436H than in the training configuration. Additionally, the memory technology associated with memory 1434A-1434B can differ between the inference configuration and the training configuration, with higher bandwidth memory technology being dedicated to the training configuration. In one embodiment, the inference configuration of GPGPU 1430 can support inference-specific instructions. For example, the inference configuration can provide support for one or more 8-bit integer dot product instructions that are commonly used during inference operations of a deployed neural network.

[0160] Machine Learning Overview

[0161] Machine learning algorithms are algorithms that can learn based on a set of data. Embodiments of machine learning algorithms can be designed to model high-order abstractions within a data set. For example, image recognition algorithms can be used to determine which of several categories a given input belongs to; regression algorithms can output a numerical value given an input; and pattern recognition algorithms can be used to generate translated text or perform text-to-speech and / or speech recognition.

[0162] One example type of machine learning algorithm is a neural network. There are many types of neural networks; one simple type of neural network is a feedforward network. A feedforward network can be implemented as a directed acyclic graph, with nodes arranged in layers. Typically, a feedforward network topology includes an input layer and an output layer, separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation useful for generating output in the output layer. Network nodes are fully connected to nodes in adjacent layers via edges, but there are no edges between nodes within each layer. Data received at the nodes of the input layer of a feedforward network is propagated (i.e., "fed forward") to the nodes of the output layer via an activation function that computes the state of the nodes of each successive layer in the network based on coefficients ("weights") respectively associated with each of the edges connecting these layers. The output from a neural network algorithm can take various forms, depending on the particular model represented by the algorithm being executed.

[0163] Before a machine learning algorithm can be used to model a particular problem, the algorithm is trained using a training data set. Training a neural network involves selecting a network topology, using a set of training data representing the problem being modeled by the network, and adjusting the weights until the network model performs with minimal error for all instances of the training data set. For example, during a supervised learning training process for a neural network, the output produced by the network in response to input representing an instance in the training data set is compared to the "correct" labeled output for the instance; an error signal representing the difference between the output and the labeled output is computed; and as the error signal is propagated backwards through the layers of the network, the weights associated with the connections are adjusted to minimize the error. When the error for each output generated from an instance of the training data set is minimized, the network is considered to have been "trained."

[0164] The accuracy of a machine learning algorithm can be greatly affected by the quality of the data set used to train the algorithm. The training process can be computationally intensive and can require a large amount of time on a conventional general-purpose processor. Therefore, parallel processing hardware is used to train many types of machine learning algorithms. This is particularly useful for optimizing the training of neural networks, as the computations performed in adjusting the coefficients in a neural network lend themselves naturally to a parallel implementation. In particular, many machine learning algorithms and software applications have been adapted to use parallel processing hardware within a general-purpose graphics processing device.

[0165] FIG. 15is a generalized diagram of a machine learning software stack 1500. Machine learning applications 1502 can be configured to train neural networks using training data sets or to implement machine intelligence using trained deep neural networks. Machine learning applications 1502 can include training and inference functionality of neural networks and / or specialized software that can be used to train neural networks prior to deployment. Machine learning applications 1502 can implement any type of machine intelligence, including but not limited to: image recognition, mapping and localization, autonomous navigation, speech synthesis, medical imaging, or language translation.

[0166] Hardware acceleration for machine learning applications 1502 can be implemented via a machine learning framework 1504. The machine learning framework 1504 can provide a library of machine learning primitives. A machine learning primitive is a basic operation that is commonly performed by machine learning algorithms. Without the machine learning framework 1504, developers of machine learning algorithms would be required to create and optimize the primary computational logic associated with the machine learning algorithms, and then re-optimize the computational logic when new parallel processors are developed. Instead, machine learning applications can be configured to perform the necessary computations using primitives provided by the machine learning framework 1504. Exemplary primitives include tensor convolutions, activation functions, and pooling, which are computational operations performed when training a convolutional neural network (CNN). The machine learning framework 1504 can also provide primitives for implementing basic linear algebra subprograms, such as matrix and vector operations, that are performed by many machine learning algorithms.

[0167] The machine learning framework 1504 can process input data received from the machine learning applications 1502 and generate appropriate inputs to a compute framework 1506. The compute framework 1506 can abstract the underlying instructions provided to a GPGPU driver 1508 to enable the machine learning framework 1504 to leverage hardware acceleration via GPGPU hardware 1510 without the machine learning framework 1504 needing to be very familiar with the architecture of the GPGPU hardware 1510. Additionally, the compute framework 1506 can enable hardware acceleration for the machine learning framework 1504 across multiple types and generations of GPGPU hardware 1510.

[0168] Machine Learning Neural Network Implementation

[0169] The computing architecture provided by the embodiments described herein can be configured to perform these types of parallel processing that are particularly well suited for training and deploying neural networks for machine learning. Neural networks can be generalized as networks of functions having graph relationships. As is known in the art, there are multiple types of neural network implementations used in machine learning. One exemplary type of neural network is a feedforward network as previously described.

[0170] A second exemplary type of neural network is a convolutional neural network (CNN). CNNs are specialized feed-forward neural networks used to process data having a known, grid-like topology, such as image data. Thus, CNNs are commonly used in computer vision and image recognition applications, but they can also be used for other types of pattern recognition, such as speech and language processing. Nodes in the input layer of a CNN are organized into groups of "filters" (feature detectors inspired by the receptive fields found in the retina), and the output of each group of filters is propagated to nodes in successive layers of the network. The computations used for a CNN include applying a convolution mathematical operation to each filter to produce the output of the filter. Convolution is a specialized mathematical operation performed by two functions to produce a third function that is a modified version of one of the original functions. In convolution network terminology, the first function with respect to the convolution can be referred to as the input, and the second function can be referred to as the convolution kernel. The output can be referred to as a feature map. For example, the input to a convolution layer can be a multidimensional data array that defines various color components of an input image. The convolution kernel can be a multidimensional array of parameters that are adapted through a training process for the neural network.

[0171] A recurrent neural network (RNN) is a type of feed-forward neural network that includes feedback connections between layers. RNNs enable modeling of sequential data by sharing parameter data across different parts of the neural network. The architecture of an RNN includes loops. These loops represent the influence of a current value of a variable on its own value at a future time, as at least a portion of the output data from the RNN is used as feedback for processing subsequent input in the sequence. This feature makes RNNs particularly useful for language processing due to the variable nature in which language data can be composed.

[0172] The diagrams described below present exemplary feed-forward, CNN, and RNN networks, and describe general processes for training and deploying each of those types of networks, respectively. It will be understood that these descriptions are exemplary and non-limiting with respect to any particular embodiment described herein, and that the concepts illustrated can generally be applied to deep neural networks and machine learning techniques in general.

[0173] The exemplary neural networks described above can be used to perform deep learning. Deep learning is machine learning using deep neural networks. In contrast to shallow neural networks that include only a single hidden layer, deep neural networks used in deep learning are artificial neural networks composed of multiple hidden layers. More deeply neural networks are generally more computationally intensive to train. However, the additional hidden layers of the network enable multi-step pattern recognition that results in reduced output error relative to shallow machine learning techniques.

[0174] The deep neural networks used in deep learning typically include a front-end network for performing feature recognition coupled to a back-end network representing a mathematical model that can perform operations (e.g., object classification, speech recognition, etc.) based on the feature representations provided to the model. Deep learning enables machine learning to be performed without the need to perform manual feature engineering for the model. Instead, deep neural networks can learn features based on statistical structures or correlations within the input data. The learned features can be provided to a mathematical model that can map the detected features to an output. The mathematical model used by the network is typically dedicated to a specific task to be performed, and different models will be used to perform different tasks.

[0175] Once a neural network is structured, a learning model can be applied to the network to train it to perform a specific task. The learning model describes how to adjust the weights within the model to reduce the network's output error. Backpropagation of error is a common method for training neural networks. An input vector is presented to the network for processing. The network's output is compared to the desired output using a loss function, and an error value is calculated for each neuron in the output layer. These error values ​​are then propagated backward until each neuron has an associated error value that roughly represents its contribution to the original output. The network can then learn from those errors using an algorithm (such as stochastic gradient descent) to update the weights of the neural network.

[0176] FIGS. 16A-16B Shows an example convolutional neural network. FIG. 16A Show the various layers in CNN. FIG. 16A As shown in , an exemplary CNN for modeling image processing can receive an input 1602 that describes the red, green, and blue (RGB) components of an input image. The input 1602 can be processed by multiple convolutional layers (e.g., a first convolutional layer 1604, a second convolutional layer 1606). Optionally, the outputs from the multiple convolutional layers can be processed by a set of fully connected layers 1608. The neurons in the fully connected layer have full connections to all activation functions in the previous layer, as previously described for feedforward networks. The outputs from the fully connected layer 1608 can be used to generate output results from the network. Matrix multiplication can be used instead of convolution to calculate the activation function within the fully connected layer 1608. Not all CNN implementations use the fully connected layer 1608. For example, in some implementations, the second convolutional layer 1606 can generate the output of the CNN.

[0177] Convolutional layers are sparsely connected, unlike the traditional neural network configuration found in fully connected layers 1608. Traditional neural network layers are fully connected, such that each output unit interacts with every input unit. However, convolutional layers are sparsely connected, in that the output of the convolution of the receptive field (rather than the respective state value of each node in the receptive field) is input to the nodes of the subsequent layer, as illustrated. The kernel associated with the convolutional layer performs a convolution operation, the output of which is sent to the next layer. The dimensionality reduction performed within the convolutional layer is one aspect that enables the CNN to scale to handle large images.

[0178] FIG. 16B An exemplary computation stage within a convolutional layer of a CNN is illustrated. The input 1612 to the convolutional layer of the CNN can be processed in three stages of the convolutional layer 1614. The three stages can include a convolution stage 1616, a detector stage 1618, and a pooling stage 1620. The convolutional layer 1614 can then output data to a successive convolutional layer. The last convolutional layer of the network can generate output feature map data or provide input to a fully connected layer, for example, to generate a classification value to the input to the CNN.

[0179] Several convolutions are performed in parallel in the convolution stage 1616 to produce a set of linear activation functions. The convolution stage 1616 can include an affine transformation, which is any transformation that can be specified as a linear transformation plus a translation. Affine transformations include rotation, translation, scaling, and combinations of these transformations. The convolution stage computes the output (e.g., a neuron) of a function connected to a particular region in the input, which can be determined as a local region associated with the neuron. The neuron computes a dot product between the weights of the neuron and the region in the local input to which the neuron is connected. The output from the convolution stage 1616 defines a set of linear activation functions that are processed by successive stages of the convolutional layer 1614.

[0180] The linear activation functions can be processed by the detector stage 1618. In the detector stage 1618, each linear activation function is processed by a non-linear activation function. The non-linear activation function increases the non-linear properties of the overall network without affecting the receptive field of the convolutional layer. Several types of non-linear activation functions can be used. One particular type is a rectified linear unit (ReLU), which uses an activation function defined as f(x) = max(0, x) such that the activation function is thresholded to zero.

[0181] The pooling stage 1620 uses a pooling function that replaces the output of the second convolutional layer 1606 with summary statistics of nearby outputs. Pooling functions can be used to introduce translational invariance into a neural network, such that slight translations to the input do not change the pooling output. Local translational invariance can be useful in cases where the presence of a feature in the input data is more important than the precise location of the feature. Various types of pooling functions can be used during the pooling stage 1620, including max pooling, average pooling, and L2 norm pooling. Additionally, some CNN implementations do not include a pooling stage. Instead, such implementations substitute an additional convolutional stage with an increased stride relative to the previous convolutional stage.

[0182] The output from the convolutional layer 1614 can then be processed by a next layer 1622. The next layer 1622 can be an additional convolutional layer or one of the fully connected layers 1608. For example, FIG. 16A The first convolutional layer 1604 of the CNN 1600 can output to the second convolutional layer 1606, which can output to the first of the fully connected layers 1608.

[0183] FIG. 17 An example recurrent neural network is shown. In a recurrent neural network (RNN), the previous state of the network influences the output of the current state of the network. A wide variety of functions can be used in a wide variety of ways to establish an RNN. The use of RNNs often revolves around using a mathematical model to make predictions of the future based on a sequence of previous inputs. For example, an RNN can be used to perform statistical language modeling to predict an upcoming word given a sequence of previous words. The RNN 1700 shown can be described as having an input layer 1702 that receives an input vector, a hidden layer 1704 that implements a recurrent function, a feedback mechanism 1705 that implements a'memory' of previous states, and an output layer 1706 that outputs a result. The RNN 1700 operates based on time steps. The state of the RNN at a given time step is influenced by previous time steps via the feedback mechanism 1705. The state of the hidden layer 1704 is defined for a given time step by the previous state and the input at the current time step. An initial input (x1) at a first time step can be processed by the hidden layer 1704. A second input (x2) can be processed by the hidden layer 1704 using state information determined during processing of the initial input (x1). The given state can be computed as s t = f(Ux t + Ws t-1), where U and W are parameter matrices. The function f is typically non-linear, such as the hyperbolic tangent function (Tanh) or a variant of the rectified function f(x) = max(0, x). However, the particular mathematical function used in the hidden layer 1704 can vary depending on the particular implementation details of the RNN 1700.

[0184] In addition to the basic CNN and RNN networks described, variations of those networks can be implemented. One example RNN variant is a long short-term memory (LSTM) RNN. LSTM RNNs are capable of learning long-term dependencies that can be necessary for processing longer language sequences. A variant of a CNN is a convolutional deep belief network, which has a structure similar to a CNN and is trained in a manner similar to a deep belief network. A deep belief network (DBN) is a generative neural network composed of multiple layers of stochastic (random) variables. A DBN can be trained layer by layer using greedy unsupervised learning. The learned weights of a DBN can then be used to provide a pre-trained neural network by determining a set of optimal initial weights for a neural network.

[0185] FIG. 18 Training and deployment of deep neural networks is shown. Once a given network has been structured for a task, the neural network is trained using a training dataset 1802. Various training frameworks have been developed for implementing hardware acceleration of the training process. For example, FIG. 15 The machine learning framework 1504 can be configured as a training framework 1804. The training framework 1804 can hook into an untrained neural network 1806 and enable the untrained neural network to be trained using the parallel processing resources described herein to generate a trained neural network 1808. To begin the training process, initial weights can be selected randomly or by pre-training using a deep belief network. The training loop is then performed in a supervised or unsupervised manner.

[0186] Supervised learning is a method of learning in which training is performed as an arbitration operation, such as when the training dataset 1802 includes inputs paired with expected outputs for the inputs, or in cases where the training dataset includes inputs with known outputs and the output of the neural network is manually graded. The network processes the inputs, and the resulting outputs are compared to a set of expected or desired outputs. Errors are then backpropagated through the system. The training framework 1804 can make adjustments to adjust the weights that control the untrained neural network 1806. The training framework 1804 can provide tools for monitoring how well the untrained neural network 1806 is converging to a model that is suitable for generating correct answers based on known input data. The training process occurs repeatedly as the weights of the network are adjusted to improve the outputs generated by the neural network. The training process can continue until the neural network reaches a statistically expected level of accuracy associated with a trained neural network 1808. The trained neural network 1808 can then be deployed to implement any number of machine learning operations.

[0187] Unsupervised learning is a method of learning in which the network attempts to train itself using unlabeled data. Thus, for unsupervised learning, the training dataset 1802 will include input data without any associated output data. The untrained neural network 1806 can learn groupings within the unlabeled inputs, and can determine how individual inputs relate to the overall dataset. Unsupervised training can be used to generate self-organizing maps, which are a type of trained neural network 1807 that can perform operations useful in data reduction. Unsupervised training can also be used to perform anomaly detection, which allows for the identification of data points in an input dataset that deviate from the normal patterns of the data.

[0188] Variations of supervised and unsupervised training can also be employed. Semi-supervised learning is a technique in which the training dataset 1802 includes a mix of labeled and unlabeled data of the same distribution. Incremental learning is a variation of supervised learning in which input data is used continuously for further training of the model. Incremental learning enables a trained neural network 1808 to adapt to new data 1812 without forgetting the knowledge rooted within the network during initial training.

[0189] Regardless of whether supervised or unsupervised, the training process for particularly deep neural networks can be too computationally intensive for a single computing node. Rather than using a single computing node, a distributed network of computing nodes can be used to speed up the training process.

[0190] FIG. 19is a block diagram illustrating distributed learning. Distributed learning is training a model that uses multiple distributed computing nodes to perform supervised or unsupervised training of a neural network. The distributed computing nodes can each include one or more host processors and one or more general purpose processing nodes. As illustrated, distributed learning can perform model parallelism 1902, data parallelism 1904, or a combination of model and data parallelism 1904.

[0191] In model parallelism 1902, different computing nodes in a distributed system can perform training computations for different parts of a single network. For example, each layer of a neural network can be trained by different processing nodes of a distributed system. Benefits of model parallelism include the ability to scale to particularly large models. Splitting computations associated with different layers of a neural network enables training of super large neural networks, where the weights of all layers would not fit into the memory of a single computing node. In some instances, model parallelism can be particularly useful in performing unsupervised training of large neural networks.

[0192] In data parallelism 1904, different nodes of a distributed network have a complete instance of a model, and each node receives a different portion of data. Results from different nodes are then combined. While different approaches for data parallelism are possible, data parallel training approaches all require a technique to combine results and synchronize model parameters between each node. Exemplary approaches for combining data include parameter averaging and update-based data parallelism. Parameter averaging trains each node on a subset of training data and sets global parameters (e.g., weights, biases) to the average of the parameters from each node. Parameter averaging uses a central parameter server that holds parameter data. Update-based data parallelism is similar to parameter averaging, except that updates to the model are passed instead of passing parameters from nodes to a parameter server. Additionally, update-based data parallelism can be performed in a decentralized manner, where updates are compressed and passed between nodes.

[0193] For example, combined model and data parallelism 1906 can be implemented in a distributed system where each computing node includes multiple GPUs. Each node can have a complete instance of a model, with individual GPUs within each node used to train different parts of the model.

[0194] Distributed training has increased overhead relative to training on a single machine. However, parallel processors and GPGPUs described herein can each implement techniques for reducing the overhead of distributed training, including techniques for implementing high-bandwidth GPU-GPU data transfer and accelerated remote data synchronization.

[0195] Exemplary Machine Learning Applications

[0196] Machine learning can be applied to solve a number of technical problems, including but not limited to computer vision, autonomous driving and navigation, speech recognition, and language processing. Computer vision has traditionally been one of the most active areas of research for machine learning applications. Applications of computer vision range from replicating human vision capabilities (e.g., recognizing human faces) to creating new classes of vision capabilities. For example, a computer vision application can be configured to recognize sound waves from vibrations induced in objects visible in a video. Parallel processor-accelerated machine learning enables training of computer vision applications using training data sets significantly larger than previously feasible, and enables deployment of inference- use systems using low-power parallel processors.

[0197] Parallel processor-accelerated machine learning has applications in autonomous driving, including lane and road sign recognition, obstacle avoidance, navigation, and driving control. Accelerated machine learning techniques can be used to train driving models based on data sets that define appropriate responses to particular training inputs. Parallel processors described herein can enable fast training of increasingly complex neural networks for autonomous driving solutions, and enable deployment of low-power inference-use processors in mobile platforms suitable for integration into autonomous vehicles.

[0198] Parallel processor-accelerated deep neural networks have enabled machine learning approaches for automatic speech recognition (ASR). ASR includes creating a function that computes the most likely sequence of words given an input acoustic sequence. Accelerated machine learning using deep neural networks has enabled replacement of Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs) previously used for ASR.

[0199] Parallel processor-accelerated machine learning can also be used to accelerate natural language processing. Automated learning programs can use statistical inference algorithms to produce models that are robust to erroneous or unfamiliar input. An exemplary natural language processor application includes automatic machine translation between human languages.

[0200] Parallel processing platforms for machine learning can be divided into training platforms and deployment platforms. Training platforms are typically highly parallel and include optimizations to accelerate multi-GPU single-node training and multi-node multi-GPU training, while deployed machine learning (e.g., inference) platforms typically include lower-power parallel processors suitable for use in products such as cameras, autonomous robots, and autonomous vehicles.

[0201] FIG. 20A computing device 2000 that hosts a master efficient memory layout mechanism ("memory layout mechanism") 2010 is shown in accordance with one embodiment. The computing device 2000 represents a communication and data processing device, including (but not limited to) a smart wearable device, a smart phone, a virtual reality (VR) device, a head-mounted display (HMD), a mobile computer, an Internet of Things (IoT) device, a laptop computer, a desktop computer, a server computer, and the like, and is associated with FIG. 1 the computing device 100; thus, for the sake of brevity, clarity, and understanding, numerous details that are FIGS. 1-1 described above in reference to the many details of the computing device 100 are not further discussed or repeated herein.

[0202] The computing device 2000 can further include (but is not limited to) an autonomous machine or artificial intelligence agent, such as a mechanical agent or machine, an electronic agent or machine, a virtual agent or machine, an electromechanical agent or machine, and the like. Examples of autonomous machines or artificial intelligence agents can include (but are not limited to) a robot, an autonomous vehicle (e.g., an autonomous car, an autonomous flying aircraft, an autonomous sailing ship, and the like), an autonomous equipment (e.g., an autonomously operated construction vehicle, an autonomously operated medical equipment, and the like), and the like. Throughout this document, a "computing device" can be referred to interchangeably as an "autonomous machine" or an "artificial intelligence agent" or simply as a "robot."

[0203] It is contemplated that, although reference is made throughout this document to "autonomous vehicles" and "autonomous driving," embodiments are not so limited. For example, an "autonomous vehicle" is not limited to a car, but rather it can include any number and type of autonomous machines, such as robots, autonomous equipment, home autonomous devices, and the like, and any one or more tasks or operations related to such autonomous machines can be referred to interchangeably with autonomous driving.

[0204] The computing device 2000 can further include (but is not limited to) a large computing system, such as a server computer, a desktop computer, and the like, and can further include a set-top box (e.g., an Internet-based cable television set-top box, and the like), a Global Positioning System (GPS)-based device, and the like. The computing device 2000 can include a mobile computing device that functions as a communication device, such as a cellular phone that includes a smart phone, a personal digital assistant (PDA), a tablet computer, a laptop computer, an e-reader, a smart television, a television platform, a wearable device (e.g., glasses, a watch, a bracelet, a smart card, jewelry, clothing, and the like), a media player, and the like. For example, in one embodiment, the computing device 600 can include a mobile computing device that employs a computer platform of an integrated circuit ("IC") that hosts a master, such as a system on a chip ("SoC" or "SOC"), which integrates various hardware and / or software components of the computing device 2000 on a single chip.

[0205] As shown, in one embodiment, computing device 2000 can include any number and type of hardware and / or software components such as, but not limited to, a graphics processing unit ("GPU" or simply "graphics processor") 2014 hosting one or more hardware accelerators 2011, a graphics driver (also referred to as a "GPU driver," "graphics driver logic," "driver logic," a user mode driver (UMD), UMD, user mode driver framework (UMDF), UMDF, or simply "driver") 2016, a central processing unit ("CPU" or simply "application processor") 2012, a memory 2008, a network device, a driver, and the like, as well as an input / output (I / O) source 2004 such as a touchscreen, touchpad, touchpanel, virtual or regular keyboard, virtual or regular mouse, port, connector, and the like. Computing device 2000 can include an operating system (OS) 2006 that serves as an interface between the hardware and / or physical resources of computer device 2000 and the user. It is contemplated that graphics processor 614 and application processor 2012 can be one or more of the processor(s) 102. FIG. 1

[0206] It should be appreciated that for some embodiments, systems less or more equipped than the examples described above can be preferred. Thus, depending on numerous factors, such as price constraints, performance requirements, technological improvements, or other considerations, the configuration of computing device 2000 can be varied from embodiment to embodiment.

[0207] Embodiments can be implemented as any or a combination of: one or more microchips or integrated circuits interconnecting hardware and / or software, hardwired logic, software stored by a memory device and executed by a microprocessor, firmware, an application specific integrated circuit (ASIC), and / or a field programmable gate array (FPGA). By way of example, the terms "logic," "module," "component," "engine," and "mechanism" can include, but are not limited to, software or hardware and / or combinations of software and hardware.

[0208] ​In one embodiment, as illustrated, the memory layout mechanism 2010 can be hosted by the graphics driver 2016. In another embodiment, the memory layout mechanism 2010 can be hosted or facilitated by the operating system 2006 of the computing device 2000. In yet another embodiment, the memory layout mechanism 2010 can be hosted by or can be part of the graphics processing unit ("GPU" or simply "graphics processor") 2014 or the firmware of the graphics processor 2014. For example, the memory layout mechanism 2010 can be embedded in or can be implemented as part of the processing hardware of the graphics processor 2014. Similarly, in yet another embodiment, the memory layout mechanism 2010 can be hosted by or can be part of the central processing unit ("CPU" or simply "application processor") 2012. For example, the memory layout mechanism 2010 can be embedded in or can be implemented as part of the processing hardware of the application processor 2012. In yet another embodiment, the memory layout mechanism 2010 can be hosted by or can be part of any number and type of components of the computing device 2000, as a portion of the memory layout mechanism 2010 can be hosted by or can be part of the operating system 2006, another portion can be hosted by or can be part of the graphics processor 2014, another portion can be hosted by or can be part of the application processor 2012, and one or more portions of the memory layout mechanism 2010 can be hosted by or can be part of the operating system 2006 and / or any number and type of devices of the computing device 2000. It is contemplated that embodiments are not limited to any particular implementation or hosting of the memory layout mechanism 2010, and one or more portions or components of the memory layout mechanism 2010 can be used as or implemented as hardware, software, or any combination thereof, such as firmware.

[0209] The computing device 2000 can host network interface(s) to provide access to a network, such as a LAN, a wide area network (WAN), a metropolitan area network (MAN), a personal area network (PAN), Bluetooth, a cloud network, a mobile network (e.g., 3rd Generation (3G), 4th Generation (4G), etc.), an intranet, the Internet, etc. The network interface(s) can include, for example, a wireless network interface with an antenna, which can represent one or more antennas. The network interface(s) can also include, for example, a wired network interface to communicate with a remote device via a network cable, which can be, for example, an Ethernet cable, a coaxial cable, an optical cable, a serial cable, or a parallel cable.

[0210] The embodiments may be provided, for example, as a computer program product that may include one or more machine-readable media having machine-executable instructions stored thereon, which, when executed by one or more machines (e.g., computers, computer networks, or other electronic devices), may cause the one or more machines to perform operations according to the embodiments described herein. The machine-readable media may include, but are not limited to, floppy disks, optical disks, CD-ROMs (compact disk read-only memories), and magneto-optical disks, ROMs, RAMs, EPROMs (erasable programmable read-only memories), EEPROMs (electrically erasable programmable read-only memories), magnetic or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions.

[0211] Furthermore, embodiments may be downloaded as a computer program product, wherein the program may be transmitted from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a communication link (e.g., a modem and / or a network connection) by means of one or more data signals embodied in and / or modulated by a carrier wave or other propagation medium.

[0212] Throughout this document, the term "user" may be referred to interchangeably as a "viewer," "observer," "person," "individual," "end-user," etc. It should be noted that throughout this document, terms such as "graphics domain" may be referred to interchangeably with "graphics processing unit," "graphics processor," or simply "GPU," and similarly, "CPU domain" or "host domain" may be referred to interchangeably with "computer processing unit," "application processor," or simply "CPU."

[0213] It should be noted that throughout this document, terms such as "node," "computing node," "server," "server device," "cloud computer," "cloud server," "cloud server computer," "machine," "host," "device," "computing device," "computer," "computing system," etc., may be used interchangeably. It should be further noted that throughout this document, terms such as "application," "software application," "program," "software program," "package," "software software package," etc., may be used interchangeably. Furthermore, throughout this document, terms such as "job," "input," "request," "message," etc., may be used interchangeably.

[0214] FIG. 21 Shown according to one embodiment FIG. 20 For the sake of brevity, the memory layout mechanism of 2010 will not be repeated or discussed hereafter. FIGS. 1-20Many of the details discussed. In one embodiment, the 9 memory layout mechanism 2010 can include any number and type of components, such as (but not limited to): detection / observation logic 2101; division / merge logic 2103; computation / prediction 2105; communication / compatibility logic 2107; and compression / decompression logic 2109.

[0215] As previously described, embodiments provide novel techniques for treating tiles as independent images throughout the process, which then enables a simple parallel implementation of one or more hardware accelerators 2011 (e.g., machine or deep learning hardware accelerators) at the graphics processor 2014.

[0216] In conventional techniques, a deep learning topology can include several layers, such as: layer 1 (e.g., kernel size 11, stride 4, IFM size 227, and output feature map (OFM) size 55); layer 2 (e.g., kernel size 3, stride 2, IFM size 55, and OFM size 27); layer 3 (kernel size 3, stride 2, IFM size 27, and OFM size 13); layer 4 (kernel size 3, stride 2, IFM size 13, and OFM size 6); fully connected layer 1; fully connected layer 2; and fully connected layer 3, etc. Now, assuming K represents kernel size, S represents stride, P represents pooling, W represents width, and H represents height, the OFM can be as follows: OFM_size = (W - K + 2P) / S + 1, where W is replaced by H in the case of H and W being different.

[0217] As discussed above, in addition to stride and padding, the OFM size can be considered a function of the input picture width and kernel size. Now, for example, if an input image is divided into two horizontal tiles and these tiles are to be combined at layer 13, which can be 6 and 7 in size, respectively, such tiles can be combined because they are small and can be easily handled in internal memory.

[0218] Now, for example, in the instance of an image being sliced in this manner with the fourth layer being 16x16 (assuming the input image size is somewhat large and there can be at least two tiles), the IFM size of the OFM size can be calculated as follows: layer 1 (kernel size 5, stride 4, IFM size 547, and OFM size 135); layer 2 (kernel size 3, stride 2, IFM size 135, and OFM size 67); layer 3 (kernel size 3, stride 2, IFM size 67, and OFM size 33); and layer 4 (kernel size 3, stride 2, IFM size 33, and OFM size 16). In this case, the OFM can be as follows: ((OFM_size - 1) * S) + K - 2P = IFM_size.

[0219] For example, as withFIG. 22A As shown, the read tiles along with the over-fetch data are filtered by a filter size of 1, where the filter size is equal to 11 and 5 adjacent pixels are over-fetched. As one of the drawbacks, this conventional technique makes compression difficult because when the overlapping portion is fetched, the local portion of the image must be decompressed, which can be somewhat difficult because the data can be decompressed in very large chunks. This results in the need to decompress intermittently across several sub-chunks.

[0220] Embodiments provide novel techniques for saving tiles as independent pictures along with any over-fetched data in memory, as facilitated by the memory layout mechanism 2010. Now, assuming that the layers at the third level are to be merged, the various sizes of the individual layers can be calculated backwards until the beginning, as facilitated by the compute / predict logic 2105, as described with respect to FIG. 21

[0221] In one embodiment, the detect / observe logic 2101 can be used to detect and / or observe one or more tiles having overlapping regions that can be divided into multiple chunks. For example, as shown with respect to FIG. 21 As shown, the tiles having overlapping regions can be detected and / or observed by the detect / observe logic 2101 and considered as input images. Similarly, for example, the detect / observe logic 2101 can be used to detect or determine the size or availability of the input local buffer. This information can then be used by the divide / merge logic 2103 to determine and decide how small or large the final chunks or images need to be to fit adequately in the available input local buffer. As facilitated by the divide / merge logic 2103, this input image can then be divided into multiple portions, such as two portions, and having a small amount of over-fetch, the two tiles can represent or then be considered as two independent images.

[0222] In one embodiment, when the two tiles are considered and treated as two independent images, no overlapping portion read can be needed throughout the process, as facilitated by the compute / predict logic 2105. As described above, the size of the input local buffer can be used to determine how large the independent chunks need to be, as determined by the divide / merge logic 2103, and similarly, how far or up to which level the process needs to continue to make the final merged chunk or image fit the available size of the input local buffer, as facilitated by the compute / predict logic 2105. For example, after a certain level, such as the third level, the two images or tiles can then be combined into a combined tile, but the combined tile can still be considered as an independent tile without any overlapping read. For example, using this novel technique, a real image size of 1080p can have an overhead of 5% to 6% in the over-fetch region.

[0223] ​This final block can then be used for compression, as facilitated by compression / decompression logic 2109, which can be much smaller than the block corresponding to the original image, which in turn saves processing resources. For example, a small block of 13x13 takes up much less processing resources than a block of 227x227; however, this compression is lossless, as the blocks can be put back together at any time without losing any data at all, as facilitated by division / merging logic 2103. Further, knowing which layer to combine also prevents data loss, where the layer is determined or predicted by compute / predict logic 2105 based on the available size of the input local buffer.

[0224] In one embodiment, this compressed version of the merged tile or block can then be obtained and used by one or more hardware accelerators 2011 at graphics processor 2014 for various processing tasks, such as performing machine or deep learning processing on an image corresponding to the compressed tile. In one embodiment, this novel technique for memory layout and compression makes the processing by one or more hardware accelerators 2011 more efficient and streamlined than having to process the compressed version of the original much larger tile or block.

[0225] Computing device 2000 is further shown in communication with one or more repositories, datasets, and / or databases, such as database(s) 2130 (e.g., cloud storage devices, non-cloud storage devices, etc.), where database(s) 2130 can reside at local storage or remote storage over a communication medium(s) 2125, such as one or more networks (e.g., cloud networks, proximity networks, mobile networks, intranets, the Internet, etc.).

[0226] It is contemplated that a software application running at computing device 2000 can be responsible for using one or more components of computing device 2000 (e.g., GPU 2014, graphics driver 2016, CPU 2012, etc.) to perform or facilitate the performance of any number and type of tasks. When performing such tasks, one or more components, such as GPU 2014, graphics driver 2016, CPU 2012, etc., can communicate with each other in order to ensure accurate and timely processing and completion of those tasks, as defined by the software application.

[0227] Communication / compatibility logic 2107 can be used to facilitate the required communication and compatibility between any number of devices of computing device 600 and various components of memory layout mechanism 2010.

[0228] The communication / compatibility logic 2107 can be used to facilitate dynamic communication and compatibility between the computing device 2000 and any number and type of other computing devices (such as mobile computing devices, desktop computers, server computing devices, etc.), processing devices or components (such as CPUs, GPUs, etc.), acquisition / sensing / detection devices (such as acquisition / sensing components including cameras, depth sensing cameras, camera sensors, red-green-blue (RGB) sensors, microphones, etc.), display devices (such as output components including display screens, display areas, display projectors, etc.), user / context awareness components and / or identification / authentication sensors / devices (such as biometric sensors / detectors, scanners, etc.), database(s) 2130 such as memory or storage devices, databases, and / or data sources (such as data storage devices, hard drives, solid state drives, hard disks, memory cards or devices, memory circuitry, etc.), communication medium(s) 2125 such as one or more communication channels or networks (e.g., cloud networks, the Internet, intranets, cellular networks, proximity networks such as Bluetooth, Bluetooth Low Energy (BLE), Smart Bluetooth, Wi-Fi proximity, radio frequency identification (RFID), near field communication (NFC), body area networks (BAN), etc.), wireless or wired communication and related protocols (e.g., Wi-Fi, WiMAX, Ethernet, etc.), connectivity and location management technologies, software applications / websites (e.g., social and / or business networking websites, business applications, games and other entertainment applications, etc.), and programming languages, etc., while ensuring compatibility with ever-changing technologies, parameters, protocols, standards, etc.

[0229] Throughout this document, terms like “logic,” “component,” “module,” “framework,” “engine,” “mechanism,” and the like can be used interchangeably and include, for example, software, hardware, and / or any combination of software and hardware, such as firmware. In one example, “logic” can refer to or include a software component capable of working with one or more of an operating system (e.g., operating system 2006) of a computing device, such as computing device 2000, a graphics driver (e.g., graphics driver 2016), and the like. In another example, “logic” can refer to or include a hardware component capable of being physically installed with or as part of one or more system hardware elements of a computing device, such as computing device 2000, such as an application processor (e.g., CPU 2012), a graphics processor (e.g., GPU 2014), and the like. In yet another example, “logic” can refer to or include a firmware component capable of being part of system firmware of a computing device, such as computing device 2000, such as firmware of an application processor (e.g., CPU 2012) or a graphics processor (e.g., GPU 2014), and the like.

[0230] ​Furthermore, no use is made of certain trademarks, words, terms, phrases, names and / or acronyms - such as "memory layout", "feature map", "IFM", "OFM", "over-fetch", "backward computation", "independent graphics", "hardware accelerator", "GPU", "GPU domain", "GPGPU", "CPU", "CPU domain", "graphics driver", "workload", "application", "graphics pipeline", "pipeline process", "register", "register file", "RF", "extended register file", "ERF", "execution unit", "EU", "instruction", "API", "3D API", Any use of "fragment shader", "YUV texture", "shader execution", "existing UAV capabilities", "existing backend", "hardware", "software", "agent", "graphics driver", "kernel-mode graphics driver", "user-mode driver", "user-mode driver framework", "buffer", "graphics buffer", "task", "process", "operation", "software application", "game", etc. should not be construed as limiting the embodiments to software or devices carrying such labels in a product or in literature outside of this document.

[0231] It is contemplated that any number and type of components may be added to and / or removed from the memory layout mechanism 2010 to facilitate various embodiments, including adding, removing, and / or enhancing certain features. For the sake of brevity, clarity, and ease of understanding of the memory layout mechanism 2010, many standard and / or well-known components, such as components of computing devices, are not shown or discussed herein. It is contemplated that the embodiments described herein are not limited to any particular technology, topology, system, architecture, and / or standard, and are dynamic enough to adopt and adapt to any future changes.

[0232] FIG. 22A An input feature map 2201A of an image with an overlapping region 2207 according to one embodiment is shown. For the sake of brevity, the previously referenced FIGS. 1-21 Many details are discussed in detail. It is contemplated that the embodiments are not limited to this description. As shown, the 22×227 IFM block 2201A includes a left tile 2203A and a right tile 2205A, wherein the two tiles 2203A, 2205A overlap with the 5×5 area 2207. For example, reading the tile together with the over-removed data may be worth a filter size of -1, while the filter size is equal to 11, over-removing 5 adjacent pixels. For example, this can make compression difficult because when the overlapping portion is removed, part of the image may be decompressed. Further, this may result in data being compressed at very large blocks, which may further result in the need to intermittently decompress several sub-blocks.

[0233] As demonstrated with reference to FIG. 22B Embodiments provide novel techniques for providing superior solutions for partial decompression by calculating the block size of a block 2201A backward from the layers where the blocks need to be merged. In one embodiment, individual tiles are treated as independent images, enabling these independent images to be decompressed in raster scan order without requiring random access decompression for overlapping portions. This enables parallel implementations of hardware accelerators (e.g., machine learning hardware accelerators) that work with compressed 3D surfaces.

[0234] FIG. 22B Tiles as independent pictures are demonstrated in accordance with one embodiment. For the sake of brevity, many of the details previously discussed with reference to FIGS. 1-22A may not be discussed or repeated hereafter. It is contemplated that embodiments are not limited to this illustration.

[0235] As described with reference to FIG. 21 For compression purposes, a block or tile 2201A representing an image can be detected for efficient processing. This 227x227 block 2201 can be split into smaller blocks 2203A and 2205A with overlapping regions, such as 59x59 overlapping regions 2207. In one embodiment, for efficient lossless compression of the block 2201A, the smaller blocks are then divided into two smaller blocks 169x169 2203B and 119x119 2205B for convolution and pooling at layer 1 2211. After this initial process of convolution and pooling is successfully completed, the blocks 169x169 2203B and 119x119 2205B are further reduced to blocks 40x40 2203C and 28x28 2205C, respectively, and further reduced to blocks 20x20 2203D and 18x18 2205D, respectively, for further pooling and convolution at layer 2 2213.

[0236] In one embodiment, as described with reference to FIG. 21 How far or up to which layer this process can proceed without losing any data at compression or when the merging of tiles can be efficiently and sufficiently performed can be based on the available size of the input local buffer. For example, in the demonstrated embodiment, the process continues by further reducing the blocks 20x20 2203D and 18x18 2205D to blocks 16x16 2203E and 14x14 2206E, respectively, and further to blocks 8x8 2203F and 9x9 2205F for further pooling and convolution processing at layer 3 2205.

[0237] In one embodiment, based on the available size of the input local buffer, the blocks 8x8 2203F and 9x9 2205F can be reduced again to blocks 6x13 2203G and 7x13 2205G, at which they are then merged back together to form a block 13x13 2201B. This final block 2201B can be responsive to the image and compressed to fit into the input local buffer without losing any data, as resulting in lossless compression.

[0238] FIG. 22C Tiling 2250 that is split into multiple tiles 2251, 2253, 2255, 2257 representing independent pictures or images is shown in accordance with one embodiment. For brevity, many of the details previously discussed with reference to FIGS. 1-22B the details discussed can not be discussed or repeated hereafter. It is contemplated that embodiments are not limited to this illustration.

[0239] As previously discussed throughout this document and shown here, a tile such as 1024x1024 tile 2250 can be divided or split into multiple smaller tiles such as tiles 2251, 2253, 2255, 2257, where each of tiles 2251, 2253, 2255, 2257 is treated as an independent image or picture.

[0240] FIG. 23 A method 2300 for facilitating efficient memory layout for intelligent data compression is shown in accordance with one embodiment. For brevity, many of the details previously discussed with reference to FIGS. 1-22C the details discussed can not be discussed or repeated hereafter. Any of the processes associated with method 2300 can be performed by processing logic that can comprise hardware (e.g., circuitry, dedicated logic, programmable logic, etc.), software (such as instructions run on a processing device), or a combination thereof, as facilitated by memory layout mechanism 2010 of FIG. 20 For brevity and clarity of presentation, the processes associated with method 2300 can be shown or described in a linear sequence; however, it is contemplated that any number of the processes can be performed in parallel, asynchronously, or in a different order.

[0241] The method 2300 begins at block 2301 where a block or tile representing an image in a machine or deep learning process is detected, the machine or deep learning process being performed by one or more hardware accelerators hosted by a graphics processor at an autonomous machine, where the block is shown to have an overlap region when cut into multiple slices. At block 2303, the available size of an input local buffer is determined. At block 2305, the block is divided or cut into multiple smaller blocks or tiles that are treated as independent images. At block 2306, in one embodiment, the size of these independent images and / or sub-images can be pre-computed based on reaching a future layer that is predicted or pre-computed or pre-determined, such that the independent smaller tiles are merged at the reaching of this future layer (similar to the merging of block 2309), where each sub-image is made or generated using the over-fetch data.

[0242] At block 2307, the multiple blocks are processed by the convolution and pooling at each layer of the neural network until the processing reaches a predicted or pre-determined layer at which the size of the multiple blocks is just enough to fit into the input local buffer and enable lossless compression. In one embodiment, any intermediate results obtained by this point are compressed (similar to the compression of block 2311). At block 2309, the multiple blocks are merged into a single block that is much smaller than the original block without any overlap region or data. At block 2311, this smaller block is compressed by lossless compression and used by the one or more hardware accelerators of the graphics processor for the machine or deep learning process.

[0243] Reference to “one embodiment,” “an embodiment,” “example embodiment,” “various embodiments,” etc., indicate that the embodiment(s) so described can include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Further, some embodiments can have a subset of the features of other embodiments, all of the features of other embodiments, or some combination thereof.

[0244] In the above description, embodiments have been described with reference to particular exemplary embodiments. However, as is readily apparent: various modifications and changes can be made to these embodiments by those skilled in the art without departing from the spirit and scope of the embodiments as set forth in the following claims. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

[0245] In the following description and claims, the term “coupled” and its derivatives can be used. “Coupled” is used to indicate that two or more elements co-operate or interact with each other, but they are not necessarily in physical or electrical contact.

[0246] As used in the claims, unless otherwise specified, the ordinal adjectives “first,” “second,” “third,” etc. merely denote different instances of a similar element. The use of the ordinal adjectives is not to be construed as indicating that the instances so denoted must occur in that given order.

[0247] The following clauses and / or examples relate to further embodiments or examples. Details in the examples can be used anywhere in one or more embodiments. Various features of the different embodiments or examples can be combined to produce various adaptations of the described implementations in various applications. Examples can include subject matter such as a method, means for performing acts of the method, at least one machine-readable medium including instructions that, when performed by a machine cause the machine to perform acts of the method, or of an apparatus or system for facilitating hybrid communications according to embodiments and examples described herein.

[0248] Some embodiments relate to example 1, which includes an apparatus for facilitating efficient memory layout for enabling intelligent data compression of hardware accelerators under a computing environment, the apparatus comprising: partitioning / merging logic to partition an initial tile representing an image into a primary plurality of tiles such that each of the primary plurality of tiles is treated as an independent image processed by one or more processors of the apparatus; and computing / predicting logic to compute the primary plurality of tiles into a secondary plurality of tiles compatible with a size of a local buffer, wherein the partitioning / merging logic is further to merge the secondary plurality of tiles into a final tile representing the image, wherein the computing / predicting logic is to compress the final tile.

[0249] Example 2 includes the subject matter of example 1, further comprising: detecting / observing logic to detect one or more of the initial tile and the size of the local buffer.

[0250] Example 3 includes the subject matter of examples 1-2, wherein the computing / predicting logic is further to predict at least one of the one or more layers necessary to compute or process the primary plurality of tiles into the secondary tiles based on the size of the local buffer.

[0251] Example 4 includes the subject matter of examples 1-3, wherein computing the primary plurality of tiles into the secondary plurality of tiles comprises one or more of a convolution and a pooling performed on the primary plurality of tiles at each of the one or more layers.

[0252] Example 5 includes the subject matter of Examples 1-4, wherein the compression of the final tile is lossless, wherein the compressed final tile is used by one or more hardware accelerators to perform machine or deep learning processing of the image associated with the compressed final tile.

[0253] Example 6 includes the subject matter of Examples 1-5, wherein the one or more processors include a graphics processor that hosts the one or more hardware accelerators, the one or more hardware accelerators including one or more machine learning hardware accelerators.

[0254] Example 7 includes the subject matter of Examples 1-6, wherein the graphics processor is co-located with an application processor on a common semiconductor package.

[0255] Some embodiments relate to Example 8, which includes a method for facilitating efficient memory layout for enabling intelligent data compression of hardware accelerators in a computing environment, the method comprising: dividing an initial tile representing an image into a primary plurality of tiles such that each tile of the primary plurality of tiles is treated as an independent image processed by one or more processors of a computing device; computing the primary plurality of tiles into a secondary plurality of tiles compatible with a size of a local buffer; merging the secondary plurality of tiles into a final tile representing the image; and compressing the final tile.

[0256] Example 9 includes the subject matter of Example 8, further comprising: detecting one or more of the initial tile and the size of the local buffer.

[0257] Example 10 includes the subject matter of Examples 8-9, further comprising: predicting, based on the size of the local buffer, at least one of one or more layers necessary to compute or process the primary plurality of tiles into the secondary tiles.

[0258] Example 11 includes the subject matter of Examples 8-10, wherein computing the primary plurality of tiles into the secondary plurality of tiles includes one or more of a convolution and a pooling performed on each of the one or more layers of the primary plurality of tiles.

[0259] Example 12 includes the subject matter of Examples 8-11, wherein the compression of the final tile is lossless, wherein the compressed final tile is used by one or more hardware accelerators to perform machine or deep learning processing of the image associated with the compressed final tile.

[0260] Example 13 includes the subject matter of Examples 8-12, wherein the one or more processors include a graphics processor that hosts the one or more hardware accelerators, the one or more hardware accelerators including one or more machine learning hardware accelerators.

[0261] Example 14 includes the subject matter of Examples 8-13, wherein the graphics processor is co-located with an application processor on a common semiconductor package.

[0262] Some embodiments relate to example 15, which includes a data processing system comprising a storage device having instructions and a processing device for executing the instructions to divide an initial tile representing an image into a primary plurality of tiles such that each tile of the primary plurality of tiles is treated as a separate image processed by one or more processors of the data processing system, compute the primary plurality of tiles into a secondary plurality of tiles compatible with a size of a local buffer, merge the secondary plurality of tiles into a final tile representing the image, and compress the final tile.

[0263] Example 16 includes the subject matter of Example 15, wherein the processing device is further for detecting one or more of the initial tile and the size of the local buffer.

[0264] Example 17 includes the subject matter of Examples 15-16, wherein the processing device is further for predicting, based on the size of the local buffer, at least one of one or more layers necessary to compute or process the primary plurality of tiles into the secondary tiles.

[0265] Example 18 includes the subject matter of Examples 15-17, wherein computing the primary plurality of tiles into the secondary plurality of tiles includes one or more of a convolution and a pooling of the primary plurality of tiles on each of the one or more layers.

[0266] Example 19 includes the subject matter of Examples 15-18, wherein the compression of the final tile is lossless, wherein the compressed final tile is used by one or more hardware accelerators to perform machine or deep learning processing on the image associated with the compressed final tile.

[0267] Example 20 includes the subject matter of Examples 15-19, wherein the one or more processors include a graphics processor that hosts the one or more hardware accelerators, the one or more hardware accelerators including one or more machine learning hardware accelerators.

[0268] Example 21 includes the subject matter of Examples 15-20, wherein the graphics processor is co-located with an application processor on a common semiconductor package.

[0269] Some embodiments relate to example 22, which includes a device comprising: means for dividing an initial tile representing an image into a primary plurality of tiles such that each of the primary plurality of tiles is treated as an independent image processed by one or more processors of the data processing system; means for computing the primary plurality of tiles into a secondary plurality of tiles compatible with a size of a local buffer; means for merging the secondary plurality of tiles into a final tile representing the image; and means for compressing the final tile.

[0270] Example 23 includes the subject matter of Example 22, further comprising means for detecting one or more of the initial tile and the size of the local buffer.

[0271] Example 24 includes the subject matter of Examples 22-23, further comprising: means for predicting, based on the size of the local buffer, at least one of one or more layers necessary to compute or process the primary plurality of tiles into the secondary tiles.

[0272] Example 25 includes the subject matter of Examples 22-24, wherein computing the primary plurality of tiles into the secondary plurality of tiles comprises one or more of a convolution and a pooling performed on the primary plurality of tiles at each of the one or more layers.

[0273] Example 26 includes the subject matter of Examples 22-25, wherein the compression of the final tile is lossless, wherein the compressed final tile is used by one or more hardware accelerators to perform machine or deep learning processing on the image associated with the compressed final tile.

[0274] Example 27 includes the subject matter of Examples 22-26, wherein the one or more processors comprise a graphics processor hosting the one or more hardware accelerators, the one or more hardware accelerators comprising one or more machine learning hardware accelerators.

[0275] Example 28 includes the subject matter of Examples 22-27, wherein the graphics processor is co-located with an application processor on a common semiconductor package.

[0276] Example 29 includes at least one non-transitory or tangible machine- readable medium including a plurality of instructions, the plurality of instructions when executed on a computing device are used to implement or perform the method of any of claims or examples 8-14.

[0277] Example 30 includes at least one machine readable medium comprising a plurality of instructions stored therein, which when executed by a computing device, implement or perform the method of any of claims or examples 8-14.

[0278] Example 31 includes a system comprising mechanisms for implementing or performing the method of any of claims or examples 8-14.

[0279] Example 32 includes an apparatus comprising means for performing the method of any of claims or examples 8-14.

[0280] Example 33 includes a computing device arranged to implement or perform the method of any of claims or examples 8-14.

[0281] Example 34 includes a communications device arranged to implement or perform the method of any of claims or examples 8-14.

[0282] Example 35 includes at least one machine readable medium comprising a plurality of instructions stored therein, which when executed by a computing device, implement or perform the method of any preceding claim or realize the apparatus of any preceding claim.

[0283] Example 36 includes at least one non-transitory or tangible machine readable medium comprising a plurality of instructions stored therein, which when executed by a computing device, implement or perform the method of any preceding claim or realize the apparatus of any preceding claim.

[0284] Example 37 includes a system comprising mechanisms arranged to implement or perform the method of any preceding claim or realize the apparatus of any preceding claim.

[0285] Example 38 includes an apparatus comprising means for performing the method of any preceding claim.

[0286] Example 39 includes a computing device arranged to implement or perform the method of any preceding claim or realize the apparatus of any preceding claim.

[0287] Example 40 includes a communications device arranged to implement or perform the method of any preceding claim or realize the apparatus of any preceding claim.

[0288] The drawings and the foregoing description give examples of embodiments. Those skilled in the art will appreciate that one or more of the described elements can well be combined into a single function. Alternatively, certain elements can be split into multiple functions. Elements from one embodiment can be added to another embodiment. For example, the order of processes described herein can be changed and other processes be added. Moreover, any action described as being taken by the processor can be taken by other functionality of the system, e.g., a display or alert. Furthermore, not all of the actions described need be taken. Those skilled in the art will understand that the embodiments can be practiced with less than all of the actions explicitly mentioned herein. The scope of the embodiments is, therefore, indicated by the following claims, rather than by the specific disclosure of certain embodiments.

Claims

1. A device for facilitating efficient memory layout for enabling intelligent data compression, the device comprising: partition / merge logic to partition an initial tile representing an image into a primary plurality of tiles such that each of the primary plurality of tiles is treated as an independent image for processing by one or more processors of the device; and compute / predict logic to compute the primary plurality of tiles into a secondary plurality of tiles compatible with a size of a local buffer, wherein the partition / merge logic is further to merge the secondary plurality of tiles into a final tile representing the image, wherein the compute / predict logic is to compress the final tile. detect / observe logic to detect one or more of the initial tile and the size of the local buffer.

2. The apparatus of claim 1, further comprising: The compute / predict logic is further to predict, based on the size of the local buffer, at least one of one or more layers necessary to compute or process the primary plurality of tiles into the secondary plurality of tiles.

3. The apparatus of claim 1, wherein, Computing the primary plurality of tiles into the secondary plurality of tiles includes one or more of convolution and pooling of the primary plurality of tiles on each of the one or more layers.

4. The apparatus of claim 3, wherein, The compression of the final tile is lossless, wherein the compressed final tile is used by one or more hardware accelerators to perform machine or deep learning processing on the image associated with the compressed final tile.

5. The apparatus of claim 1, wherein, The one or more processors include a graphics processor that hosts one or more hardware accelerators including one or more machine learning hardware accelerators.

6. The apparatus of claim 1, wherein, The graphics processor is co-located with an application processor on a common semiconductor package.

7. The apparatus of claim 6, wherein, 8. A method for facilitating efficient memory layout for enabling intelligent data compression, the method comprising: partitioning an initial tile representing an image into a primary plurality of tiles such that each of the primary plurality of tiles is treated as an independent image for processing by one or more processors of a computing device; computing the primary plurality of tiles into a secondary plurality of tiles compatible with a size of a local buffer; merging the secondary plurality of tiles into a final tile representing the image; and compressing the final tile.

9. The method of claim 8, further comprising detecting one or more of the initial tile and the size of the local buffer. predicting, based on the size of the local buffer, at least one of one or more layers necessary to compute or process the primary plurality of tiles into the secondary plurality of tiles. Computing the primary plurality of tiles into the secondary plurality of tiles includes one or more of convolution and pooling of the primary plurality of tiles on each of the one or more layers.

10. The method of claim 8, further comprising: The compression of the final tile is lossless, wherein the compressed final tile is used by one or more hardware accelerators to perform machine or deep learning processing on the image associated with the compressed final tile.

11. The method of claim 10, wherein, ​ 12. The method of claim 8, wherein, ​ 13. The method of claim 8, wherein, The one or more processors include a graphics processor that hosts one or more hardware accelerators, including one or more machine learning hardware accelerators.

14. The method of claim 13, wherein, The graphics processor is co-located with an application processor on a common semiconductor package.

15. At least one machine readable medium comprising a plurality of instructions to, when executed on a computing device, perform the method of any one of claims 8 to 14.

16. A system for data processing comprising a storage device having instructions and a processing device for executing the instructions to perform the operations of the method of any one of claims 8 to 14.

17. A processing device for executing instructions to perform the operations of the method of any one of claims 8 to 14.

18. A computing device arranged to perform the method of any one of claims 8 to 14.

19. A communications device arranged to perform the method of any one of claims 8 to 14.

Images