Fine-grained compute communication execution for deep learning frameworks

Abstract

One embodiment provides a system for configuring distributed training of a neural network. The system includes a memory to store a library to facilitate data transfer during distributed training of the neural network, a network interface to transmit and receive gradient data associated with the trainable parameters, a general purpose processor to execute instructions provided by the library that cause the general purpose processor to configure the network interface to transmit and receive the gradient data associated with the trainable parameters during a workflow of a machine learning framework, and a graphics processor to perform a compute operation associated with a machine learning framework workflow to generate the gradient data associated with the trainable parameters, wherein the library interleaves the compute operation on the graphics processor with transmission and reception of gradient data via the network interface based on the machine learning framework workflow.

Description

[0001] Cross-references

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 502,453, filed May 5, 2017, which is hereby incorporated by reference herein. TECHNICAL FIELD

[0003] Embodiments relate generally to data processing, and more particularly to data processing via general purpose graphics processing units. BACKGROUND

[0004] Current parallel graphics data processing includes systems and methods developed to perform specific operations on graphics data, such as, for example, linear interpolation, tessellation, rasterization, texture mapping, depth testing, etc. Traditionally, graphics processors used fixed-function compute units to process graphics data; however, recently, portions of graphics processors have become programmable, enabling such processors to support a wider variety of operations for processing vertex and fragment data.

[0005] To further improve performance, graphics processors typically implement processing techniques, such as pipelining operations, that attempt to process as much graphics data in parallel as possible throughout different portions of the graphics pipeline. Parallel graphics processors with single instruction multiple thread (SIMT) architectures are designed to maximize the amount of parallel processing in the graphics pipeline. In SIMT architectures, groups of parallel threads attempt to execute program instructions together as often as possible to improve processing efficiency. A general overview of software and hardware for SIMT architectures can be found in Shane Cook, CUDA Programming, Chapter 3, pages 37-51 (2013); and / or Nicholas Wilt, CUDA Handbook (A Comprehensive Guide to GPU Programming), Chapters 2.6.2 to 3.1.2 (June 2013). BRIEF DESCRIPTION OF DRAWINGS

[0006] For a more detailed description of the features of the present application, reference will now be made to the embodiments illustrated in the accompanying drawings in which:

[0007] Figure 1 is a block diagram that illustrates a computer system configured to implement one or more aspects of the embodiments described herein;

[0008] Figures 2A-2D A parallel processor component according to an embodiment is shown;

[0009] Figures 3A-3B is a block diagram of a graphics multiprocessor according to an embodiment;

[0010] Figures 4A-4F An exemplary architecture in which multiple GPUs are communicatively coupled to multiple multi-core processors is shown;

[0011] Figure 5 A graphics processing pipeline according to an embodiment is shown;

[0012] Figure 6 A machine learning software stack according to an embodiment is shown;

[0013] Figure 7 A highly parallel general purpose graphics processing unit according to an embodiment is shown;

[0014] Figure 8 A multi-GPU computing system according to an embodiment is shown;

[0015] Figures 9A-9B Layers of an exemplary deep neural network are shown;

[0016] Figure 10 An exemplary recurrent neural network is shown;

[0017] Figure 11 Training and deployment of a deep neural network is shown;

[0018] Figure 12 is a block diagram showing distributed learning;

[0019] Figure 13 An exemplary inference chip on system (SOC) suitable for performing inference using a trained model is shown;

[0020] Figures 14A-14E Communication patterns used during distributed machine learning computation operations performed across multiple compute nodes according to embodiments described herein are shown;

[0021] Figures 15A-15C Architecture details of a machine learning extension library provided by embodiments described herein are shown;

[0022] Figures 16A-16B Distributed machine learning training enabled by embodiments described herein is shown;

[0023] Figure 16C Inter-node communication using point-to-point primitives according to an embodiment is shown;

[0024] Figure 17AA multi-node computing system is shown in accordance with an embodiment;

[0025] Figure 17B A point-to-point network with distributed virtual addresses is shown in accordance with an embodiment;

[0026] Figure 18 An alternative MLSL architecture is shown in accordance with an embodiment;

[0027] Figure 19A Tensor computation operations suitable for fine-grained computation and communication overlap are shown;

[0028] Figure 19B Synchronized memory access between multi-node systems is shown in accordance with an embodiment;

[0029] Figure 19C Memory communication semantics extended to implement coarse-grained cache coherency for cache memory data are shown;

[0030] Figures 20A-20B A flow diagram describing operations for implementing distributed machine learning via an MLSL API is shown;

[0031] Figures 21A-21B A method of performing distributed training of a neural network in accordance with embodiments described herein is shown;

[0032] Figure 22 is a block diagram of a data processing system in accordance with embodiments described herein;

[0033] Figure 23 is a block diagram of a processing system in accordance with an embodiment.

[0034] Figure 24 is a block diagram of a processor in accordance with an embodiment;

[0035] Figure 25 is a block diagram of a graphics processor in accordance with an embodiment;

[0036] Figure 26 is a block diagram of a graphics processing engine of a graphics processor in accordance with some embodiments;

[0037] Figure 27 is a block diagram of a graphics processor provided by additional embodiments;

[0038] Figure 28 Thread execution logic is shown that includes an array of processing elements employed in some embodiments;

[0039] Figure 29 is a block diagram showing a graphics processor instruction format in accordance with some embodiments;

[0040] Figure 30 is a block diagram of a graphics processor according to another embodiment;

[0041] Figures 31A-31B graphics processor command formats and command sequences according to some embodiments are shown;

[0042] Figure 32 exemplary graphics software architecture of a data processing system according to some embodiments is shown;

[0043] Figure 33 is a block diagram showing an IP core development system according to an embodiment;

[0044] Figure 34 is a block diagram showing an exemplary system on a chip integrated circuit according to an embodiment;

[0045] Figure 35 is a block diagram showing an additional graphics processor according to an embodiment; and

[0046] Figure 36 is a block diagram showing an additional exemplary graphics processor of a system on a chip integrated circuit according to an embodiment. DETAILED DESCRIPTION

[0047] In some embodiments, a graphics processing unit (GPU) is communicatively coupled to a host / processor core to accelerate processing for graphics, machine learning, pattern analysis, and various general purpose GPU (GPGPU) functions. The GPU can be communicatively coupled to the host processor / cores over a bus or another interconnect (e.g., a high-speed

[0048] In the following description, numerous specific details are set forth to provide a more thorough understanding. However, it will be apparent to one of skill in the art that one or more of the embodiments described herein can be practiced without one or more of the specific details. In other instances, well-known features have not been described in order to avoid obscuring the details of the present embodiments.

[0049] System Overview

[0050] Figure 1is a block diagram illustrating a computer system 100 configured to implement one or more aspects of the embodiments described herein. The computing system 100 includes a processing subsystem 101 having one or more processors 102 and a system memory 104, which are in communication via an interconnection path 105. The interconnection path 105 can include one or more buses (as is true with the interconnection paths 106 and 108 described below) and / or a network. The interconnection path 105 can be configured to perform data communications using one or more protocols, including an I2C protocol, a PCI Express protocol, and / or a proprietary protocol.

[0051] In one embodiment, the processing subsystem 101 includes one or more parallel processor(s) 112, which can communicate with the memory subsystem 105 via a bus or other communication link 113. The communication link 113 can be one of a number of different communication links, such as a standard bus, a point-to-point connection, a message passing interface, a pipeline, a video bus, an I2C bus, and / or a proprietary high-speed link, depending upon the particular embodiment of the computer system 100. In one embodiment, the one or more parallel processor(s) 112 form a computationally-intensive, parallel or vector processing system that can include a number of processor cores, each of which can be configured similarly to the one or more processors 102. In one embodiment, the one or more parallel processor(s) 112 form a graphics processing subsystem that can output pixels to one or more display device(s) 110A coupled via the I / O subsystem 107. The one or more parallel processor(s) 112 can also include a display controller and display interface (not shown) to enable a direct connection to one or more display device(s) 110B.

[0052] Within I / O subsystem 111, system storage 114 can be connected to I / O hub 107 to provide storage mechanisms for computing system 100. I / O switch 116 can be used to provide an interface mechanism to enable connections between I / O hub 107 and other components that can be integrated into a platform such as network adapter 118 and / or wireless network adapter 119, as well as various other devices that can be added via one or more add-in devices 120. Network adapter 118 can be an Ethernet adapter or another wired network adapter. Wireless network adapter 119 can include one or more of Wi-Fi, Bluetooth, Near Field Communication (NFC), or other network equipment including one or more radios.

[0053] Computing system 100 can include other components not explicitly shown, including USB or other port connections, optical storage drives, video capture devices, and so on, that can also be connected to I / O hub 107. Figure 1 Communication paths interconnecting various components of computing system 100 can use any suitable protocol, such as a PCI (Peripheral Component Interconnect) - based protocol (e.g., PCI-Express), or any other bus or point-to-point communication interface and / or protocol, such as NV-Link high-speed interconnect, or interconnect protocols known in the art.

[0054] In one embodiment, one or more parallel processors 112 incorporate circuitry optimized for graphics and video processing, including, for example, video output circuitry, and the circuitry constitutes a graphics processing unit (GPU). In another embodiment, one or more parallel processors 112 incorporate circuitry optimized for general purpose processing, along with circuitry that constitutes a GPU configured to perform a graphics process. In yet another embodiment, components of computing system 100 can be integrated with one or more other system elements on a single integrated circuit. For example, one or more parallel processors 112, memory hub 105, processor(s) 102, and I / O hub 107 can be integrated into a

[0055] It should be appreciated that the computing system 100 illustrated herein is exemplary and variations and modifications are possible. The connection topology can be modified according to needs, including the number and arrangement of bridges, the number of processor(s) 102, and the number of parallel processor(s) 112. For example, in some embodiments, system memory 104 is connected to the processor(s) 102 directly rather than through a bridge, while other devices communicate with system memory 104 via the memory hub 105 and the processor(s) 102. In other alternative topologies, the parallel processor(s) 112 are connected to the I / O hub 107 or directly to one of the processor(s) 102, rather than to the memory hub 105. In still other embodiments, the I / O hub 107 and the memory hub 105 are integrated into a single chip. Some embodiments can include two or more groups of the processor(s) 102 attached via multiple sockets, which can couple with two or more instances of the parallel processor(s) 112.

[0056] Some of the particular components shown herein are optional and can not be included in all implementations of the computing system 100. For example, any number of add-in cards or peripheral devices can be supported, or some components can be omitted. Additionally, some architectures can use different terminology to describe what is Figure 1 similar to the components shown herein. For example, in some architectures, the memory hub 105 can be called a north bridge, while the I / O hub 107 can be called a south bridge.

[0057] Figure 2A A parallel processor 200 according to an embodiment is shown. The various components of the parallel processor 200 can be implemented using one or more integrated circuits, such as programmable processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). According to an embodiment, the illustrated parallel processor 200 is a variant of one or more parallel processors 112 shown and described herein. Figure 1

[0058] ​In one embodiment, parallel processor 200 includes a parallel processing unit 202. The parallel processing unit includes an I / O unit 204 that enables communication with other devices including other instances of parallel processing unit 202. The I / O unit 204 can be directly connected to the other devices. In one embodiment, the I / O unit 204 communicates with other devices via the use of an interconnect fabric, such as a memory hub 105. The connection to the memory hub 105 forms a communication link 113. Within the parallel processing unit 202, the I / O unit 204 is connected to a host interface 206 and a memory crossbar switch 216, where the host interface 206 receives commands required to perform processing tasks and the memory crossbar switch 216 facilitates communication with memory elements.

[0059] When the host interface 206 receives a command buffer via the I / O unit 204, the host interface 206 can direct the work operations for executing those commands to a front end 208. In one embodiment, the front end 208 is coupled with a scheduler 210 that is configured to distribute the commands or other work items to a processing cluster array 212. In one embodiment, the scheduler 210 ensures that the processing cluster array 212 is correctly configured and is in an enabled state before tasks are distributed to the processing clusters within the processing cluster array 212. In one embodiment, the scheduler 210 is implemented via firmware logic executing on a microcontroller. The scheduler 210 implemented via the microcontroller can be configured to perform complex scheduling and work distribution operations at a coarse and fine granularity, enabling pre-emption and context switching of threads executing on the processing array 212 at a fine granularity. In one embodiment, host software can evidence workloads for scheduling on the processing array 212 via one of multiple graphics processing doorbell mechanisms. These workloads can then be automatically distributed by the scheduler 210 logic within the scheduler microcontroller across the processing array 212.

[0060] The processing cluster array 212 can include up to "N" processing clusters (e.g., cluster 214A, 214B, through 214N). Each cluster 214A-214N of the processing cluster array 212 can execute a number of concurrent threads. The scheduler 210 can allocate work to the clusters 214A-214N of the processing cluster array 212 using a variety of scheduling and / or work distribution algorithms. The scheduling can be dynamic, with scheduling and / or work distribution algorithms adapted or changed from one type of program or computation to another, depending on the requirements of each type of program or computation. In one embodiment, different clusters 214A-214N of the processing cluster array 212 can be allocated for use in processing different types of programs or for performing different types of computations.

[0061] The processing cluster array 212 can be configured to perform various types of parallel processing operations. In one embodiment, the processing cluster array 212 is configured to perform general-purpose parallel compute operations. For example, the processing cluster array 212 can include logic to perform processing tasks including filtering of video and / or audio data, performance of modeling operations including physics operations, and performance of data transformations.

[0062] In one embodiment, the processing cluster array 212 is configured to perform parallel graphics processing operations. In an embodiment in which the parallel processor 200 is configured to perform graphics processing operations, the processing cluster array 212 can include additional logic to support the execution of such graphics processing operations including without limitation texture mapping logic used to perform texture operations, and surface rendering logic used to perform surface rendering operations. In addition, the processing cluster array 212 can be configured to execute shader programs, including, but not limited to, vertex shaders, tessellation shaders, geometry shaders, and pixel

[0063] In one embodiment, when the parallel processing unit 202 is used to perform graphics processing, the scheduler 210 can be configured to divide the processing workload into approximately equal sized tasks, in order to better enable distribution of the graphics processing operations to multiple clusters 214A-214N of the processing cluster array 212. In some embodiments, portions of the processing cluster array 212 can be configured to perform different types of processing. For example, a first portion can be configured to perform vertex shading and topology generation, a second portion can be configured to perform surface shading and geometry

[0064] During operation, the processing cluster array 212 can receive processing tasks to be performed via the scheduler 210, which receives commands defining the processing tasks from the front end 208. For graphics processing operations, the processing task can include data to be processed, such as surface (patch) data, geometric data, vertex data, and / or pixel data, and state parameters and commands defining how the data is to be processed (e.g., which program is to be executed). The scheduler 210 can be configured to fetch the indices of the tasks corresponding to the indices or can receive the indices from the front end 208. The front end 208 can be configured to ensure the processing cluster array 212 is configured in an effective state prior to a workload specified by an incoming command buffer (e.g., a batch buffer, an in-stall buffer, etc.) being launched.

[0065] Each of the one or more instances of the parallel processing unit 202 can be coupled with a parallel processor memory 222. The parallel processor memory 222 can be accessed via a memory crossbar 216, which can receive memory requests from the processing cluster array 212 as well as the I / O unit 204. The memory crossbar 216 can access the parallel processor memory 222 via a memory interface 218. The memory interface 218 can include a number of partition units (e.g., partition unit 220A, partition unit 220B, up to partition unit 220N), which can each be coupled to a portion (e.g., memory unit) of the parallel processor memory 222. In one implementation, the number of partition units 220A-220N is configured to be equal to the number of memory units, such that the first partition unit 220A has a corresponding first memory unit 224A, the second partition unit 220B has a corresponding memory unit 224B, and the Nth partition unit 220N has a corresponding Nth memory unit 224N. In other embodiments, the number of partition units 220A-220N can not be equal to the number of memory devices.

[0066] In various embodiments, memory units 224A-224N can include various types of memory devices including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. In one embodiment, memory units 224A-224N can also include 3D stacked memory, including but not limited to high bandwidth memory (HBM). As will be appreciated by one skilled in the art, the specific implementation of memory units 224A-224N can vary from one embodiment to the next and can be selected from one of various conventional designs. Render targets such as frame buffers or texture maps can be stored on memory units 224A-224N, allowing partition units 220A-220N to write portions of each render target in parallel for efficient use of the available bandwidth of parallel processor memory 222. In some embodiments, to support a unified memory design that utilizes system memory in conjunction with local cache memory, a local instance of parallel processor memory 222 can be excluded.

[0067] In one embodiment, any of clusters 214A-214N of processing cluster array 212 can process data to be written to any of memory units 224A-224N within parallel processor memory 222. Memory crossbar 216 can be configured to pass the output of each cluster 214A-214N to any partition unit 220A-220N or another cluster 214A-214N, which can perform additional processing operations on the output. Each cluster 214A-214N can communicate with memory interface 218 through memory crossbar 216 to perform read or write operations for various external memory devices. In one embodiment, memory crossbar 216 can be connected to memory interface 218 to communicate with I / O unit 204 and to a local instance of parallel processor memory 222, enabling processing units within the different processing clusters 214A-214N to communicate with system memory or other memory not local to parallel processor 202. In one embodiment, memory crossbar 216 can use virtual channels to separate traffic streams between clusters 214A-214N and partition units 220A-220N.

[0068] While a single instance of parallel processing unit 202 is shown within parallel processor 200, any number of instances of parallel processing unit 202 can be included. For example, a multiprocessor system can include multiple instances of parallel processing unit 202, or parallel processing unit 202 can be inter-connected to other parallel processing units. Even though different instances of parallel processing unit 202 can have different processing capabilities, different amounts of local parallel processor memory, and / or different configurations, they can be configured to operate as a single parallel processor. For example, and in one embodiment, some instances of parallel processing unit 202 can include a higher precision floating point unit relative to other instances. A system incorporating one or more instances of parallel processing unit 202 or parallel processor 200 can be implemented in a variety of configurations and form factors, including but not limited to desktop, laptop, or hand-held personal computers, servers, workstations, game consoles, and / or embedded systems.

[0069] Figure 2B is a block diagram of a partition unit 220 according to an embodiment. In one embodiment, partition unit 220 is an instance of one of partition units 220A-220N of FIG. 2. As shown, partition unit 220 includes an L2 cache 221, a frame buffer interface 225, and an ROP 226 (raster operations unit). L2 cache 221 is a read / write cache that is configured to perform loads and stores received from memory crossbar 216 and ROP 226. Read misses and urgent write-back requests are output to frame buffer interface 225 for processing by L2 cache 221. Updates can also be sent to a frame buffer via frame buffer interface 225 for processing. In one embodiment, frame buffer interface 225 interfaces with one of the memory units in the parallel processor memory, such as memory units 224A-224N of FIG. 2 (e.g., within parallel processor memory 222). Figure 2A

[0070] In graphics applications, ROP 226 is a processing unit that performs raster operations, such as stencil, z-test, blend, and so forth. ROP 226 then outputs processed graphics data that is stored in graphics memory. In some embodiments, ROP 226 includes compression logic to compress depth or color data that is written to memory and decompress depth or color data that is read from memory. The compression logic can be lossless compression logic that uses one or more of a variety of compression algorithms. The type of compression that is performed by ROP 226 can vary based on the statistical characteristics of the data to be compressed. For example, in one embodiment, delta color compression is performed on depth and color data on a tile-by-tile basis.

[0071] ​In some embodiments, ROP 226 is included within each processing cluster (e.g., clusters 214A-214N of FIG. 2) instead of being included within partition unit 220. In this embodiment, read and write requests to pixel data are transmitted through memory crossbar 216 instead of pixel fragment data. Processed graphics data can be displayed on one of one or more display devices 110, routed by processor(s) 102 for further processing, or routed by one of the processing entities within parallel processor 200 for further processing. Figure 1 Figure 2A

[0072] Figure 2C is a block diagram of a processing cluster 214 within a parallel processing unit according to an embodiment. In one embodiment, the processing cluster is an instance of one of the processing clusters 214A-214N of FIG. 2. A processing cluster 214 can be configured to execute a plurality of threads in parallel, where the term "thread" refers to a particular instance of a particular program executed by the processing cluster. In some embodiments, single instruction, multiple data (SIMD) instruction issue techniques are used to support parallel execution of a large number of threads on the processing cluster. In other embodiments, single instruction, multiple thread (SIMT) techniques are used to support parallel execution of a large number of generally synchronous threads on the processing cluster. Unlike SIMD execution mechanisms, where all processing engines typically execute identical instructions, SIMT execution allows for divergent execution paths from thread to thread.

[0073] Operation of a processing cluster 214 can be controlled via a pipeline manager 232 that distributes processing tasks to SIMT parallel processors. Pipeline manager 232 receives instructions from the scheduler 210 of FIG. 2 and manages execution of those instructions via a graphics multiprocessor 234 and / or a texture unit 236. The illustrated graphics multiprocessor 234 is an exemplary instance of a SIMT parallel processor. However, various types of SIMT parallel processors of differing architectures can be included within processing cluster 214. One or more instances of the graphics multiprocessor 234 can be included within a processing cluster 214. Graphics multiprocessor 234 can process data, and a data crossbar 240 can be used to distribute the processed data to one of a number of possible destinations. The pipeline manager 232 can facilitate distribution by specifying destinations in data crossbar 240 for data to be distributed via data crossbar 240.

[0074] ​​Each graphics multiprocessor 234 within processing cluster 214 can include the same functional execution logic, which can be configured in a pipelined manner to process audio and video instructions. The functional execution logic can support a variety of operations including integer and floating-point arithmetic, comparison operations, Boolean operations, bit-shifting operations, and computation of various algebraic functions. In one embodiment, same functional -unit hardware can be leveraged to perform different operations using different software instructions or microcodes and / or any combination of hardware state and a software instruction.

[0075] Instructions transmitted to the processing cluster 214 form a thread. A set of threads executed on a group of parallel processing engines forms a warp. Threads in a warp are executed in parallel in a multiprocessor. Each thread can be assigned a unique thread identifier, thread ID, which can be used to determine the unique address of each thread's local data store. Each thread can also be assigned a unique lane ID. The lane ID can be used in situations where a thread handling an instruction branch can fork a thread to be executed, and each forked thread can be assigned a unique lane ID.

[0076] In one embodiment, the graphics multiprocessor 234 includes an internal cache memory to perform load and store operations. In one embodiment, the graphics multiprocessor 234 can discard the internal cache memory and use the cache memory within the processing cluster 214 (e.g., the LI cache 308). Each graphics multiprocessor 234 can also have access to the L2 cache within the partition unit (e.g., partition units 220A-220N of FIG. 2) shared between all processing clusters 214, and can be used to transfer data between threads. The graphics multiprocessor 234 can also have access to off-chip global memory, which can include one or more of a local parallel processor memory and a system memory. Any memory external to the parallel processor unit 202 can be accessed by the graphics multiprocessors 234 through the off-chip memory interface. Embodiments in which the processing cluster 214 includes multiple instances of graphics multiprocessor 234 can share common instructions and data stored within the LI cache 308.

[0077] Each processing cluster 214 can include an MMU 245 (memory management unit) configured to translate virtual addresses into physical addresses, in other embodiments, one or more instances of the MMU 245 can reside in the memory interface 218 of FIG. 2. The MMU 245 includes a set of page table entries (PTEs) used to translate virtual addresses into physical addresses and optionally cache line indices. The MMU 245 can include an address translation lookaside buffer (TLB) or cache to reduce the latency of address translations. The physical addresses are processed to distribute surface data access locality to achieve efficient request interleaving among partition units. The cache line indices can be used to determine whether requests for cache lines are a hit or miss.

[0078] In graphics and compute applications, the processing cluster 214 can be configured such that each graphics multiprocessor 234 is coupled to a texture unit 236 for performing texture mapping operations, e.g., determining texture sample positions, reading texture data, and filtering texture data. Texture data can be read from an internal texture LI cache (not shown) or, in some embodiments, from the LI cache within the graphics multiprocessor 234, and is fetched from the L2 cache, local parallel processor memory, or system memory, as needed. Each graphics multiprocessor 234 outputs processed tasks to a data crossbar 240 that couples to another processing cluster 214 for further processing or to a memory crossbar 216 for processing different tasks, or to store tasks in system memory. A preROP 242 (pre-raster operations unit) is configured to receive data from the graphics multiprocessor 234, direct data to a ROP unit which can be located within the partition unit (e.g., partition units 220A-220N of FIG. 2) as described herein. The preROP 242 unit can optimize color mixing, organize pixel color data, and perform address translations.

[0079] It should be understood that the core architecture described herein is illustrative and that variations and modifications are possible. For example, any number of processing units, such as the graphics multiprocessor 234, texture unit 236, preROP 242, etc., can be included within processing cluster 214. Further, while only one processing cluster 214 is shown, parallel processing units can include any number of instances of processing cluster 214 as described herein. In one embodiment, each processing cluster 214 can be configured to operate independently of the other processing clusters 214 using separate and different processing units, LI caches, etc.

[0080] Figure 2DA graphics multiprocessor 234 according to one embodiment is shown. In such an embodiment, the graphics multiprocessor 234 is coupled with the pipeline manager 232 of the processing cluster 214. The graphics multiprocessor 234 has an execution pipeline that includes, without limitation, an instruction cache 252, an instruction unit 254, an address mapping unit 256, a register file 258, one or more general-purpose graphics processing unit (GPGPU) cores 262, and one or more load / store units 266. The GPGPU cores 262 and the load / store units 266 are coupled with a memory and cache interconnect 268 to cache memory 272 and shared memory 270.

[0081] In one embodiment, the instruction cache 252 receives a stream of instructions to execute from the pipeline manager 232. These instructions are cached in the instruction cache 252 and dispatched for execution by the instruction unit 254. The instruction unit 254 can dispatch the instructions as threads to different ones of the GPGPU cores 262, each thread allocated to a different one of the execution units within the GPGPU cores 262. The instructions can access the local, shared, or global address spaces by specifying addresses within a unified address space. The address mapping unit 256 can be used to convert addresses in the unified address space to different memory addresses available to the load / store units 266.

[0082] The register file 258 provides a set of registers to the functional units of the graphics multiprocessor 234. The register file 258 provides temporary storage for operands to the data paths connected to the functional units (e.g., GPGPU cores 262, load / store units 266) of the graphics multiprocessor 234. In one embodiment, the register file 258 is divided into a number of separate pools each of which is assigned to be used by particular functional units. One embodiment can use this partitioning to more effectively train and issue threads for execution.

[0083] The GPGPU cores 262 can each include floating point, integer, and / or single-instruction multiple-data (SIMD) execution units supporting multiple programming models. In one embodiment, the GPGPU cores 262 are similar in architecture and functionality to the GPGPU cores 262. The GPGPU cores 262 can be similar to the core of Intel Corporation's® Core™ i7 processor in architecture and functionality.

[0084] In one embodiment, the GPGPU cores 262 include SIMD execution units to perform a single -instruction on multiple-data (SIMD) operations (e.g., common data operations). In another embodiment, the GPGPU cores 262 also include single-instruction multiple-thread (SIMT) execution units to perform SIMT operations. In one embodiment, the GPGPU cores 262 can each execute both a common thread instruction and a thread instruction.

[0085] The memory and cache interconnect 268 is an interconnection network that connects each of the functional units of the graphics multiprocessor 234 to the register file 258 and shared memory 270. In one embodiment, the memory and cache interconnect 268 is a crossbar switch interconnect that allows the load / store units 266 to implement load and store operations between the shared memory 270 and the register file 258. The register file 258 can operate at the same frequency as the GPGPU cores 262, so that the data transfer between the GPGPU cores 262 and the register file 258 has very low latency. The shared memory 270 can be used to enable communication between threads executing on the functional units within the graphics multiprocessor 234. For example, the cache memory 272 can be used as a data cache to cache texture data communicated between the functional units and the texture unit 236. The shared memory 270 can also be used as a cached managed program. In addition to the auto-cached data stored within the cache memory 272, threads executing on the GPGPU cores 262 can also programmatically store data within the shared memory.

[0086] Figures 3A-3B Additional graphics multiprocessors according to embodiments are shown. The illustrated graphics multiprocessors 325, 350 are variants of the graphics multiprocessor 234 of Figure 2C FIG. 2. The illustrated graphics multiprocessors 325, 350 can be configured to concurrently execute a large number of execution threads as a streaming multiprocessor (SM).

[0087] Figure 3A A graphics multiprocessor 325 according to additional embodiments is shown. The graphics multiprocessor 325 includes a number of additional instances of execution resource units relative to the graphics multiprocessor 234 of Figure 2D FIG. 2. For example, the graphics multiprocessor 325 can include multiple instances of instruction units 332A-B, register files 334A-B, and texture unit(s) 344A-B. The graphics multiprocessor 325 also includes multiple sets of graphics or compute execution units (e.g., GPGPU cores 336A-B, GPGPU cores 337A-B, GPGPU cores 338A-B) and multiple sets of load / store units 340A-B. In one embodiment, the execution resource units have a common instruction cache 330, texture and / or data cache memory 342, and shared memory 346.

[0088] The various components can communicate via an interconnect structure 327. In one embodiment, the interconnect structure 327 includes one or more crossbars to enable communication between components of the graphics processing unit 325. In one embodiment, the interconnect structure 327 is a separate, high-speed network structure layer on which each component of the graphics processing unit 325 is stacked. The components of the graphics processing unit 325 communicate with remote components via the interconnect structure 327. For example, the GPGPU cores 336A-336B, 337A-337B, and 3378A-338B can each communicate with the shared memory 346 via the interconnect structure 327. The interconnect structure 327 can arbitrate communications within the graphics processing unit 325 to ensure fair bandwidth allocation between components.

[0089] Figure 3B A graphics processing unit 350 is shown in accordance with additional embodiments. As shown in Figure 2D and Figure 3A the graphics processor includes multiple sets of execution resources 356A-356D, where each set of execution resources includes multiple instruction units, register file, GPGPU cores, and load store units. The execution resources 356A-356D can work with the texture unit(s) 360A-360D to perform texture operations, while sharing the instruction cache 354 and shared memory 362. In one embodiment, the execution resources 356A-356D can share multiple instances of the instruction cache 354 and shared memory 362, as well as the texture and / or data cache memory 358A-358B. The various components can communicate via an interconnect structure 352 similar to the interconnect structure 327 of Figure 3A .

[0090] Those skilled in the art will appreciate that the architecture described in Figure 1 , Figures 2A-2D and Figures 3A-3B is descriptive, and does not limit the scope of embodiments of the present application. Thus, the techniques described herein can be implemented on any suitably configured processing unit, including but not limited to: one or more mobile application processors; one or more desktop computer or server central processing units (CPUs), including multi-core CPUs; one or more parallel processing units such as the parallel processing unit 202 of FIG. 2; and one or more graphics processors or specialized processing units, without departing from the scope of the embodiments described herein.

[0091] In some embodiments, a parallel processor or GPGPU as described herein is communicatively coupled to a host / processor core to accelerate graphics operations, machine learning operations, pattern analysis operations, and various general purpose GPU (GPGPU) functions. The GPU can be communicatively coupled to the host processor / cores over a bus or other interconnect (e.g., a high-speed

[0092] Techniques for GPU-to-host processor interconnect

[0093] Figure 4A An exemplary architecture is shown in which a plurality of GPUs 410-413 are communicatively coupled to a plurality of multi-core processors 405-406 over high-speed links 440-443 (e.g., buses, point-to-point interconnects, etc.). In one embodiment, the high-speed links 440-443 support communications at a throughput level of 4GB / s, 30GB / s, 80GB / s or higher, depending on the implementation. Various interconnect protocols can be used including, but not limited to, PCIe 4.0 or 5.0 and NVLink 2.0. However, the underlying principles of the application are not limited to any particular communication protocol or throughput level.

[0094] In addition, in one embodiment, two or more of the GPUs 410-413 are interconnected over high-speed links 444-445, which can be implemented using the same or different protocol / links than used for high-speed links 440-443. Similarly, two or more of the multi-core processors 405-406 can be connected over high-speed link 433, which can be an SMP bus that runs at 20GB / s, 30GB / s, 120GB / s or higher. Alternatively, Figure 4A All communication between the various system components shown in FIG. 4 can be accomplished using the same protocol / s (e.g., over a common interconnect architecture). However, as mentioned, the underlying principles of the application are not limited to any particular type of interconnect technology.

[0095] In one embodiment, each of the many-core processors 405-406 is communicatively coupled to processor memories 401-402 via memory interconnects 430-431, respectively, and each of the GPUs 410-413 is communicatively coupled to GPU memories 420-423 through GPU memory interconnects 450-453, respectively. The memory interconnects 430-431 and 450-453 can utilize the same or different memory access technologies. By way of example and not limitation, the processor memories 401-402 and the GPU memories 420-423 can be volatile memory such as dynamic random access memory (DRAM) including stacked DRAM, graphics DDR SDRAM (GDDR) such as GDDR5, GDDR6, or high bandwidth memory (HBM), and / or can be non-volatile memory such as 3D XPoint or Nano-Ram. In one embodiment, some portion of the memory can be volatile memory while another portion can be non-volatile memory (e.g., using a two-level memory (2LM) hierarchy).

[0096] As described below, although each of the various processors 405-406 and GPUs 410-413 can be physically coupled to a particular memory 401-402, 420-423, respectively, a unified memory architecture can be implemented in which the same virtual system address space (also referred to as "effective address" space) is distributed across all of the various physical memories. For example, the processor memories 401-402 can each include 64 GB of system memory address space, and the GPU memories 420-423 can each include 32 GB of system memory address space (resulting in a total of 256 GB of addressable memory space in the example described).

[0097] Figure 4B Additional details of the interconnection between the many-core processor 407 and the graphics acceleration module 446 are shown according to one embodiment. The graphics acceleration module 446 can include one or more GPU chips integrated on a line card coupled to the processor 407 via a high-speed link 440. Alternatively, the graphics acceleration module 446 can be integrated on the same package or chip as the processor 407.

[0098] The illustrated processor 407 includes multiple cores 460A-D, each with a translation lookaside buffer 461A-D and one or more caches 462A-D. The cores can include various other components (e.g., instruction fetch unit, branch prediction unit, decoders, execution units, reorder buffer, etc.) to execute instructions and process data not shown to avoid obscuring the inventive subject matter. The caches 462A-D can include level one (LI) and level two (L2) caches. In addition, one or more shared caches 426 can be included in the cache hierarchy and shared by the various groups of cores 460A-D. For example, one embodiment of the processor 407 includes 24 cores each with its own LI cache, 12 shared L2 caches, and 12 shared L3 caches. In this embodiment, one of the L2 and L3 caches is shared by two adjacent cores. The processor 407 and graphics accelerator integrated module 446 are connected with system memory 441, which can include processor memories 401-402.

[0099] Data and instructions stored in various caches 462A-D, 456 and system memory 441 are kept coherent across the various caches and system memory by inter-core communications over coherency bus 464. For example, each cache can have cache coherency logic / circuitry associated therewith to communicate over coherency bus 464 in response to a detected read or write to a particular cache line. In one implementation, a cache snoop protocol is implemented over coherency bus 464 to snoop cache accesses. Cache snoop / coherency techniques are well understood by those skilled in the art to avoid obscuring the inventive subject matter and are not described in detail here.

[0100] In one embodiment, the proxy circuit 425 communicatively couples the graphics acceleration module 446 to the coherency bus 464, allowing the graphics acceleration module 446 to participate in the cache coherency protocol as a peer to the cores. Specifically, the interface 435 provides connectivity to the proxy circuit 425 over a high-speed link 440 (e.g., a PCIe bus, NVLink, etc.), and the interface 437 connects the graphics acceleration module 446 to the link 440.

[0101] In one implementation, the accelerator integration circuit 436 provides cache management, memory access, context management, and interrupt management services on behalf of multiple graphics processing engines 431, 432, 43N of the graphics acceleration module 446. The graphics processing engines 431, 432, 43N can each comprise a separate graphics processing unit (GPU). Alternatively, the graphics processing engines 431, 432, 43N can comprise different types of graphics processing engines such as graphics execution units, media processing engines (e.g., video encoders / decoders), samplers, and block image transfer engines within a GPU. In other words, the graphics acceleration module can be a GPU with a plurality of graphics processing engines 431, 432, 43N, or the graphics processing engines 431, 432, 43N can be separate GPUs integrated on a common package, line card, or chip.

[0102] In one embodiment, the accelerator integration circuit 436 includes a memory management unit (MMU) 439 to translate virtual addresses into physical addresses, perform memory protection, and provide memory management operations. In one embodiment, the MMU 439 can include memory hardware to support a hierarchical page table

[0103] A set of registers 449 stores context data for threads executed by the graphics processing engines 431-432, 43N, and the context management circuit 448 manages thread contexts. For example, the context management circuit 448 can perform save and restore operations to save and restore the context of various threads during context switches (e.g., where a first thread is saved and a second thread is stored so that the second thread can be executed by the graphics processing engines). For example, upon context switch, the context management circuit 448 can store the current register values to a designated area in memory (e.g., identified by a context pointer). The context management circuit can restore the register values upon return from context. In one embodiment, the interrupt management circuit 447 receives and processes interrupts received from system devices.

[0104] In one implementation, virtual / effective addresses from the graphics processing engines 431 are translated to real / physical addresses in system memory 411 by the MMU 439. One embodiment of the accelerator integration circuit 436 supports multiple (e.g., 4, 8, 16) graphics processor modules 446 and / or other accelerator devices. The graphics processor modules 446 can be dedicated to a single application executing on the processor 407 or shared between multiple applications. In one embodiment, a virtualized graphics execution environment is presented in which resources of the graphics processing engines 431-432, 43N are shared between multiple applications or virtual machines (VMs). The resources can be subdivided into "tiles" that are allocated to different VMs and / or applications based on their processing requirements and priorities.

[0105] Thus, the accelerator integration circuit functions as a bridge of the system for the graphics acceleration module 446 and provides address translation and system memory cache services. In addition, the accelerator integration circuit 436 can provide virtualization facilities for the host processor to manage virtualization of graphics processing engines, interrupts, and memory management.

[0106] Since the hardware resources of the graphics processing engines 431-432, 43N are explicitly mapped to the real address space seen by the host processor 407, any host processor can use effective address values to directly address these resources. In one embodiment, a function of the accelerator integration circuit 436 is to physically separate the graphics processing engines 431-432, 43N so that they appear as independent units on the system.

[0107] As mentioned, in the illustrated embodiment, one or more graphics memory 433-434, 43M is coupled to each of the graphics processing engines 431-432, 43N. The graphics memory 433-434, 43M stores instructions and data for processing by each of the graphics processing engines 431-432, 43N. The graphics memory 433-434, 43M can be volatile memory such as DRAM (including stacked DRAM), GDDR memory (e.g., GDDR5, GDDR6), or HBM, and / or can be non-volatile memory such as 3D XPoint or Nano-Ram.

[0108] In one embodiment, to reduce the data traffic on the link 440, a biasing technique is used to ensure that the data stored in the graphics memory 433-434, 43M is that which is most frequently used by the graphics processing engines 431-432, 43N and that which is preferably not used (at least not frequently) by the cores 460A-D. Similarly, the biasing mechanism attempts to keep data required by the cores (and preferably not the graphics processing engines 431-432, 43N) within the caches 462A-D, 456 of the cores and system memory 411.

[0109] Figure 4C Another embodiment is illustrated in which the accelerator integration circuit 436 is integrated within the processor 407. In this embodiment, the graphics processing engines 431-432, 43N communicate directly with the accelerator integration circuit 436 via the interface 437 and the interface 435 (which can also utilize any form of bus or interface protocol) over the high-speed link 440. The accelerator integration circuit 436 can execute same operations as those described with respect to the accelerator 436 but can operate at higher throughput given its close proximity to the coherence bus 462 and caches 462A-D, 426. Figure 4B

[0110] One embodiment supports different programming models, including a dedicated process programming model (without graphics acceleration module virtualization) and a shared programming model (with virtualization). The shared programming model can include a programming model controlled by the accelerator integration circuit 436 and a programming model controlled by the graphics acceleration module 446.

[0111] In one embodiment of the dedicated process model, the graphics processing engines 431-432, 43N are dedicated to a single application or process under a single operating system. The single application can centralize other application requests to the graphics engines 431-432, 43N, providing virtualization within the VM / partition.

[0112] ​In a dedicated process programming model, graphics processing engines 431-432, 43N can be shared by multiple VM / application partitions. The shared model requires a system hypervisor that is used to virtualize the graphics processing engines 431-432, 43N to allow access by each operating system. For single-partition systems without a hypervisor, the graphics processing engines 431-432, 43N are owned by the operating system. In both cases, the operating system can virtualize the graphics processing engines 431-432, 43N to provide access to each process or application.

[0113] For the shared programming model, graphics acceleration module 446 or individual graphics processing engines 431-432, 43N use a process handle to select a process element. In one embodiment, the process element is stored in system memory 411 and is addressable using the effective-to-real address translation techniques described herein. The process handle can be an implementation-specific value provided to the host process when it registers its context with the graphics processing engines 431-432, 43N (i.e., calls system software to add the process element to a process element linked list). The lower 16 bits of the process handle can be an offset into the process element linked list of the process element.

[0114] Figure 4D An exemplary accelerator integration tile 490 is shown. As used herein, a "tile" includes a specified portion of the processing resources of accelerator integration circuit 436. Application effective address space 482 within system memory 411 stores process elements 483. In one embodiment, process elements 483 are stored in response to GPU invocations 481 from an application 480 executing on processor 407. Process elements 483 contain processing state for the respective application 480. A work descriptor (WD) 484 contained in process element 483 can be a single job requested by an application or can contain a pointer to a job queue. In the latter case, WD 484 is a pointer to a job request queue in application address space 482.

[0115] Graphics acceleration module 446 and / or individual graphics processing engines 431-432, 43N can be shared by all or a subset of processes in a system. Embodiments of the present invention include infrastructure for setting up processing state and sending WDs 484 to graphics acceleration module 446 to start jobs in a virtual environment.

[0116] In one implementation, the dedicated process programming model is specific to the implementation. In this model, a single process owns a graphics acceleration module 446 or individual graphics processing engines 431. As the graphics acceleration module 446 is owned by a single process, the hypervisor initializes the accelerator integration circuit 436 for the owning partition and the operating system initializes the accelerator integration circuit 436 for the owning process when the graphics acceleration module 446 is allocated.

[0117] In operation, the WD fetch unit 491 in the accelerator integration slice 490 fetches the next WD 484, which includes an indication of work to be done by one of the graphics processing engines of the graphics acceleration module 446. As shown, data from the WD 484 can be stored in registers 449 and used by the MMU 439, interrupt management circuit 447, and / or context management circuit 446. For example, one embodiment of the MMU 439 includes segment / page walk circuitry to access segment / page tables 486 within the OS virtual address space 485. The interrupt management circuit 447 can handle interrupt events 492 received from the graphics acceleration module 446. As graphics operations are performed, effective addresses 493 generated by the graphics processing engines 431-432, 43N are translated to real addresses by the MMU 439.

[0118] In one embodiment, a same set of registers 449 is replicated for each graphics processing engine 431-432, 43N and / or graphics acceleration module 446 and can be initialized by the hypervisor or operating system. Each of these replicated registers can be included in an accelerator integration slice 490. Exemplary registers that can be initialized by the hypervisor are shown in Table 1.

[0119] Table 1 - Hypervisor Initialized Registers

[0120] 1 Slice Control Register 2 Real Address (RA) Dispatch Process Region Pointer 3 Authorization Mask Override Register 4 Interrupt Vector Table Entry Offset 5 Interrupt Vector Table Entry Limit 6 Status Register 7 Logical Partition ID 8 Real Address (RA) Hypervisor Accelerator Utilization Record Pointer 9 Storage Description Register

[0121] Exemplary registers that can be initialized by the operating system are shown in Table 2.

[0122] Table 2 - Operating System Initialized Registers

[0123] 1 Process and Thread Identification 2 Effective Address (EA) Context Save / Restore Pointer 3 Virtual Address (RA) Accelerator Utilization Record Pointer 4 Virtual Address (RA) Storage Segment Table Pointer 5 Authorization Mask 6 Work Descriptor

[0124] In one embodiment, each WD 484 is specific to a particular graphics acceleration module 446 and / or graphics processing engine 431-432, 43N. The WD contains all information needed by the graphics processing engine 431-432, 43N to finish its work or the WD can be a pointer to a memory location where an application has set up a command queue of work to be done.

[0125] Figure 4E Additional details are presented that illustrate one embodiment of a shared model. The embodiment includes a hypervisor real address space 498 in which a list of process elements 499 is stored. The hypervisor real address space 498 is accessible via a hypervisor 496 that virtualizes the graphics acceleration module engine of the operating system 495.

[0126] The shared programming model allows all or some of the processes from all or some of the partitions in the system to use the graphics acceleration module 446. There are two programming models in which the graphics acceleration module 446 is shared by multiple processes and partitions: time-sliced sharing and graphics direct sharing.

[0127] In this model, the system hypervisor 496 owns the graphics acceleration module 446 and makes its functionality available to all operating systems 495. To enable the graphics acceleration module 446 to support the virtualization of the system hypervisor 496, the graphics acceleration module 446 can adhere to the following requirements: 1) Application job requests must be autonomous (i.e., no need to maintain state between jobs), or the graphics acceleration module 446 must provide a context save and restore mechanism. 2) The graphics acceleration module 446 guarantees that an application job request will complete within a specified amount of time, including any translation faults, or the graphics acceleration module 446 provides the ability to preempt job processing. 3) When operating in the direct shared programming model, fairness of the graphics acceleration module 446 among processes must be guaranteed.

[0128] In one embodiment, for a shared model, an application 480 is required to utilize a graphics acceleration module 446 type, a work descriptor (WD), an authorization mask register (AMR) value, and a context save / restore area pointer (CSRP) for an operating system 495 system call. The graphics acceleration module 446 type describes the target acceleration function for the system call. The graphics acceleration module 446 type can be a system specific value. The WD is formatted specifically for the graphics acceleration module 446 and can be in the form of a graphics acceleration module 446 command; a pointer to a user defined structure; a pointer to a command queue; or any other data structure used to describe work to be performed by the graphics acceleration module 446. In one embodiment, the AMR value is the AMR state for the current process. The value passed to the operating system is similar to how an application sets the AMR. If the accelerator integration circuit 436 and graphics acceleration module 446 implementation does not support a user authorization mask override register (UAMOR), the operating system can apply the current UAMOR value to the AMR value before passing the AMR in the hypervisor call. The hypervisor 496 can optionally apply the current authorization mask override register (AMOR) value before placing the AMR in the process element 483. In one embodiment, the CSRP is one of the registers 449 containing the effective address of an area in the application address space 482 for the graphics acceleration module 446 to save and restore context state. This pointer is optional if there is no need to save state between jobs or when a job is preempted. The context save / restore area can be paged system memory.

[0129] Upon receiving the system call, the operating system 495 can verify that the application 480 is registered and authorized to use the graphics acceleration module 446. The operating system 495 then invokes the hypervisor 496 with the information shown in Table 3.

[0130] Table 3 - Operating System to Hypervisor Invocation Parameters

[0131] 1 Work Descriptor (WD) 2 Authorization Mask Register (AMR) Value (possibly masked) 3 Effective Address (EA) Context Save / Restore Region Pointer (CSRP) 4 Process ID (PID) and optional Thread ID (TID) 5 Virtual Address (VA) Accelerator Utilization Record Pointer (AURP) 6 Virtual Address of Storage Segment Table Pointer (SSTP) 7 Logical Interrupt Service Number (LISN)

[0132] Upon receiving the hypervisor call, the hypervisor 496 can verify that the operating system 495 is registered and authorized to use the graphics acceleration module 446. The hypervisor 496 then places the process element 483 in a process element linked list for the corresponding graphics acceleration module 446 type. The process element can contain the information shown in Table 4.

[0133] Table 4 - Process Element Information

[0134]

[0135]

[0136] In one embodiment, the hypervisor initializes the plurality of registers 449 of the accelerator integration slice 490.

[0137] As Figure 4F embodiment of the application employs a unified memory that can be addressed via a common virtual memory address space for accessing physical processor memory 401-402 and GPU memory 420-423. In this implementation, operations performed on the GPUs 410-413 utilize the same virtual / effective memory address space to access processor memory 401-402 and vice versa, thereby simplifying programmability. In one embodiment, a first portion of the virtual / effective address space is allocated to processor memory 401, a second portion is allocated to second processor memory 402, a third portion is allocated to GPU memory 420, and so on. The entire virtual / effective memory space (sometimes referred to as the effective address space) is thereby distributed across each of the processor memory 401-402 and GPU memory 420-423, allowing any processor or GPU to access any physical memory with a virtual address mapped to said memory.

[0138] In one embodiment, bias / coherence management circuitry 494A-494E within one or more of the MMUs 439A-439E ensures cache coherence between the host processor (e.g., 405) and the caches of the GPUs 410-413, and implements a bias technique that indicates the physical memory in which certain types of data should be stored. Although multiple instances of bias / coherence management circuitry 494A-494E are shown in Figure 4F

[0139] ​One embodiment allows GPU-attached memory 420-423 to be mapped as part of system memory and accessed using shared virtual memory (SVM) technology, but without suffering the typical performance penalties associated with full system cache coherency. The ability for GPU-attached memory 420-423 to be accessed as system memory does not incur the heavy cache coherency overhead, which provides a favorable operating environment for GPU offload. This arrangement allows host processor 405 software to set operands and access computation results without the overhead of traditional I / O DMA data copies. These traditional copies involve driver calls, interrupts, and memory-mapped I / O (MMIO) accesses, which are inefficient relative to simple memory accesses. At the same time, the ability to access GPU-attached memory 420-423 without cache coherency overhead can be critical to the execution time of offloaded computations. For example, in the case of a large amount of streaming write memory traffic, cache coherency overhead can significantly reduce the effective write bandwidth seen by GPUs 410-413. The efficiency of operand setup, the efficiency of result access, and the efficiency of GPU computation all play a significant role in determining the effectiveness of GPU offload.

[0140] In one implementation, the selection between GPU bias and host processor bias is driven by a bias tracker data structure. For example, a bias table can be used, which can be a page-granularity structure (i.e., controlled at the granularity of a memory page) that includes 1 or 2 bits per GPU-attached memory page. The bias table can be implemented within the stolen memory range of one or more GPU-attached memories 420-423, with or without a bias cache in GPUs 410-413 (e.g., to cache frequently / recently used entries of the bias table). Alternatively, the entire bias table can be kept within the GPU.

[0141] In one implementation, the bias table entry associated with each access to GPU-attached memory 420-423 is accessed prior to the actual access to GPU memory, such that the following operations. First, local requests from GPUs 410-413 that find their pages in the GPU bias are forwarded directly to the corresponding GPU memory 420-423. Local requests from the GPUs that find their pages in the host bias are forwarded to processor 405 (e.g., over the high-speed link as described above). In one embodiment, requests from processor 405 that find the requested page in the host processor bias complete the request as a normal memory read. Alternatively, requests for GPU-biased pages can be forwarded to GPUs 410-413. If the page is not currently in use by the GPU, the GPU can convert the page to the host processor bias.

[0142] The bias state of a page can be changed by a software-based mechanism, a hardware-assisted software-based mechanism, or for a limited set of cases, a hardware-only mechanism.

[0143] One mechanism for changing bias states employs an API call (e.g., OpenCL) that in turn invokes a GPU device driver that in turn sends a message to the GPU (or enqueues a command descriptor) that directs the GPU to change the bias state and, for certain transitions, to perform a cache flush clear operation in the host. The cache flush clear operation is necessary for a transition from the host processor 405 bias to the GPU bias, but not for the reverse transition.

[0144] In one embodiment, cache coherency is maintained by temporarily presenting GPU-biased pages that are not cacheable by the host processor 405. To access these pages, the processor 405 can request access from the GPU 410, which can or can not grant access depending on the implementation. Thus, to reduce communication between the processor 405 and the GPU 410, it is advantageous to ensure that the GPU-biased pages are pages that are needed by the GPU but not by the host processor 405, and vice versa.

[0145] Graphics Processing Pipeline

[0146] Figure 5 A graphics processing pipeline 500 according to an embodiment is shown. In one embodiment, a graphics processor can implement the graphics processing pipeline 500 shown. The graphics processor can be included within a parallel processing subsystem such as the parallel processor 200 of Figure 2 as described herein, in one embodiment the parallel processor is a GPU Figure 1embodiments, the graphics processing pipeline 500 can be implemented using a variant of the parallel processor 112. As described herein, various parallel processing systems can implement the graphics processing pipeline 500 via one or more instances of a parallel processing unit (e.g., the parallel processing unit 202 of FIG. 2). For example, a shader unit (e.g., the graphics multiprocessor 234 of FIG. 3) can be configured to perform the functions of one or more of the vertex processing unit 504, the tessellation control processing unit 508, the tessellation evaluation processing unit 512, the geometry processing unit 516, and the fragment / pixel processing unit 524. The functionality of the data assembler 502, the primitive assemblers 506, 514, 518, the tessellation unit 510, the rasterizer 522, and the raster operations unit 526 can also be performed by other processing engines and corresponding partition units within a processing cluster (e.g., processing cluster 214 of FIG. 3). The graphics processing pipeline 500 can be implemented using a dedicated processing unit for one or more of the functions. In one embodiment, one or more portions of the graphics processing pipeline 500 can be performed by parallel processing logic within a general purpose processor (e.g., a CPU). In one embodiment, one or more portions of the graphics processing pipeline 500 can access on-chip memory (e.g., parallel processor memory 222 as shown in FIG. 2) via a memory interface 528, which can be an instance of the memory interface 218 of FIG. 2.

[0147] In one embodiment, the data assembler 502 is a processing unit that collects vertex data for surfaces and primitives. The data assembler 502 then outputs the vertex data, including vertex attributes, to the vertex processing unit 504. The vertex processing unit 504 is a programmable execution unit that executes a vertex shader program to transform and illuminate vertices of a vertex shader program as specified. The vertex processing unit 504 reads data stored in cache, local or system memory that contains vertex data for processing and can be programmed to transform the vertex data from an object-based coordinate representation to a world space coordinate space or a normalized device coordinate space.

[0148] A first instance of the primitive assembler 506 receives vertex attributes from the vertex processing unit 504. The primitive assembler 506 reads stored vertex attributes as needed, and constructs graphics primitives for processing by the tessellation control processing unit 508. The graphics primitives include triangles, lines, points, patches, and the like as supported by various graphics processing application programming interfaces (APIs).

[0149] The tessellation control processing unit 508 treats the input vertices as control points for the geometry patch. These control points are transformed from an input representation from the patch, such as a basis for the patch, to a representation suitable for surface evaluation by the tessellation evaluation processing unit 512. The tessellation control processing unit 508 can also compute tessellation factors for the edges of the geometry patch. The tessellation factors are applicable to individual edges and quantify a view-dependent level of detail associated with the edge. The tessellation unit 510 is configured to receive the tessellation factors for the edges of the patch and subdivide the patch into a plurality of geometric primitives, such as line, triangle, or quadrilateral primitives, which are transmitted to the tessellation evaluation processing unit 512. The tessellation evaluation processing unit 512 operates on the parametric coordinates of the subdivided patch to generate surface representations and vertex attributes for each vertex associated with the geometric primitives.

[0150] A second instance of the primitive assembler 514 receives the vertex attributes from the tessellation evaluation processing unit 512, reads the stored vertex attributes as needed, and constructs graphics primitives for processing by the geometry processing unit 516. The geometry processing unit 516 is a programmable execution unit that executes a geometry shader program to transform the graphics primitives received from the primitive assembler 514 as specified by the geometry shader program. In one embodiment, the geometry processing unit 516 is programmed to subdivide the graphics primitives into one or more new graphics primitives and compute parameters for rasterizing the new graphics primitives.

[0151] In some embodiments, the geometry processing unit 516 can add or delete elements in the geometry stream. The geometry processing unit 516 outputs parameters and vertices that specify new graphics primitives to the primitive assembler 518. The primitive assembler 518 receives the parameters and vertices from the geometry processing unit 516 and constructs graphics primitives for processing by the viewport scale, cull, and clip unit 520. The geometry processing unit 516 reads data stored in the parallel processor memory or system memory for processing the geometry data. The viewport scale, cull, and clip unit 520 performs clipping, culling, and viewport scale and outputs the processed graphics primitives to the rasterizer 522.

[0152] The rasterizer 522 can perform depth culling and other depth-based optimizations. The rasterizer 522 also performs scan conversion on new graphics primitives to generate fragments and outputs these fragments and associated coverage data to the fragment / pixel processing units 524. The fragment / pixel processing units 524 are programmable execution units that are configured to perform fragment shader programs or pixel shader programs. The fragment / pixel processing units 524 transform the fragments or pixels received from the rasterizer 522 as specified by the fragment or pixel shader programs. For example, the fragment / pixel processing units 524 can be programmed to perform operations including, but not limited to, texture mapping, shading, blending, texture correction, and perspective correction to produce shaded fragments or pixels that are output to the raster operations unit 526. The fragment / pixel processing units 524 can read data stored in the parallel processor memory or system memory to use when processing the fragment data. The fragment or pixel shader programs can be configured to shade with samples, pixels, tiles, or other granularity, depending on how the processing units are configured.

[0153] The raster operations unit 526 is a processing unit that performs raster operations including, but not limited to, stencil operations, z-test, blending, and the like and outputs pixel data as processed graphics data to be stored in graphics memory (e.g., the parallel processor memory 222 in FIG. 2, and / or system memory 104 in FIG. 1 to be displayed on one or more display devices 110 or further processed by one of the one or more processors 102 or parallel processor(s) 112, as further processing is required. In some embodiments, the raster operations unit 526 is configured to compress z or color data that is written to memory and decompress z or color data that is read from memory. Figure 1

[0154] Machine Learning Overview

[0155] Machine learning algorithms are algorithms that can learn based on a set of data. Embodiments of machine learning algorithms can be designed to model high-order abstractions within a data set. For example, image recognition algorithms can be used to determine which of several categories a given input belongs to; regression algorithms can output a numerical value given an input; and pattern recognition algorithms can be used to generate translated text or perform text-to-speech and / or speech recognition.

[0156] ​One example type of machine learning algorithm is a neural network. There are many types of neural networks; one simple type of neural network is a feedforward network. A feedforward network can be implemented as a directed acyclic graph, with nodes arranged in layers. Typically, a feedforward network topology includes an input layer and an output layer, separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation useful for generating output in the output layer. Network nodes are fully connected to nodes in adjacent layers via edges, but there are no edges between nodes within each layer. Data received at the nodes of the input layer of a feedforward network is propagated (i.e., "fed forward") to the nodes of the output layer via an activation function that computes the state of the nodes of each successive layer in the network based on coefficients ("weights") respectively associated with each of the edges connecting these layers. The output from a neural network algorithm can take various forms, depending on the particular model represented by the algorithm being executed.

[0157] Before a machine learning algorithm can be used to model a particular problem, the algorithm is trained using a training data set. Training a neural network involves selecting a network topology, using a set of training data representing the problem being modeled by the network, and adjusting the weights until the network model performs with minimal error for all instances of the training data set. For example, during a supervised learning training process for a neural network, the output produced by the network in response to input representing an instance in the training data set is compared to the "correct" labeled output for the instance; an error signal representing the difference between the output and the labeled output is computed; and as the error signal is propagated backwards through the layers of the network, the weights associated with the connections are adjusted to minimize the error. When the error for each output generated from an instance of the training data set is minimized, the network is considered to have been "trained."

[0158] The accuracy of a machine learning algorithm can be greatly affected by the quality of the data set used to train the algorithm. The training process can be computationally intensive and can require a large amount of time on a conventional general-purpose processor. Therefore, parallel processing hardware is used to train many types of machine learning algorithms. This is particularly useful for optimizing the training of neural networks, as the computations performed in adjusting the coefficients in a neural network lend themselves naturally to a parallel implementation. In particular, many machine learning algorithms and software applications have been adapted to use parallel processing hardware within a general-purpose graphics processing device.

[0159] Figure 6is a generalized diagram of a machine learning software stack 600. Machine learning applications 602 can be configured to train neural networks using training data sets or implement machine intelligence using trained deep neural networks. Machine learning applications 602 can include training and inference functionality of neural networks and / or specialized software that can be used to train neural networks prior to deployment. Machine learning applications 602 can implement any type of machine intelligence, including but not limited to: image recognition, mapping and localization, autonomous navigation, speech synthesis, medical imaging, or language translation.

[0160] Hardware acceleration for machine learning applications 602 can be implemented via a machine learning framework 604. Machine learning framework 604 can provide a library of machine learning primitives. Machine learning primitives are basic operations that are commonly performed by machine learning algorithms. Without machine learning framework 604, developers of machine learning algorithms would be required to create and optimize the primary computational logic associated with machine learning algorithms, then re-optimize the computational logic when new parallel processors are developed. Instead, machine learning applications can be configured to perform necessary computations using primitives provided by machine learning framework 604. Exemplary primitives include tensor convolutions, activation functions, and pooling, which are computational operations performed when training a convolutional neural network (CNN). Machine learning framework 604 can also provide primitives for implementing basic linear algebra subroutines, such as matrix and vector operations, that are performed by many machine learning algorithms.

[0161] Machine learning framework 604 can process input data received from machine learning applications 602 and generate appropriate inputs to a compute framework 606. Compute framework 606 can abstract underlying instructions provided to GPGPU drivers 608 to enable machine learning framework 604 to leverage hardware acceleration via GPGPU hardware 610 without machine learning framework 604 being very familiar with the architecture of GPGPU hardware 610. Additionally, compute framework 606 can enable hardware acceleration for machine learning framework 604 across multiple types and generations of GPGPU hardware 610.

[0162] GPGPU Machine Learning Acceleration

[0163] Figure 7 A highly parallel general purpose graphics processing unit 700 according to an embodiment is shown. In one embodiment, general purpose processing unit (GPGPU) 700 can be configured to be particularly efficient in processing this type of computational workload associated with training deep neural networks. Additionally, GPGPU 700 can be directly linked to other instances of GPGPUs for creating multi-GPU clusters to improve the speed of training particularly deep neural networks.

[0164] GPGPU 700 includes a host interface 702 for enabling connectivity to a host processor. In one embodiment, the host interface 702 is a PCI Express interface. However, the host interface can be a proprietary

[0165] GPGPU 700 includes memory 714A-B coupled with the compute clusters 706A-H via a set of memory controllers 712A-B. In various embodiments, the memory 714A-B can include various types of memory devices including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. In one embodiment, the memory units 224A-N can also include 3D stacked memory, including but not limited to high bandwidth memory (HBM).

[0166] In one embodiment, each compute cluster 706A-H includes a set of graphics processing units, such as graphics processing unit 400 of FIG. 4. Figure 4A The graphics processing units of a compute cluster include multiple types of integer and floating point logic units that can perform compute operations at a range of precisions, including precisions suitable for machine learning computations. For example, and in one embodiment, at least a subset of the floating point units of each of the compute clusters 706A-H can be configured to perform 16-bit or 32-bit floating point operations, while a different subset of the floating point units can be configured to perform 64-bit floating point operations.

[0167] Multiple instances of GPGPU 700 can be configured to operate as a compute cluster. The communication mechanism used by the compute cluster for synchronization and data exchange varies across embodiments. In one embodiment, multiple instances of GPGPU 700 communicate over host interface 702. In one embodiment, GPGPU 700 includes I / O hub 708 that couples GPGPU 700 to GPU links 710, which enable direct connections to other instances of GPGPU. In one embodiment, GPU links 710 are coupled to a specialized GPU-GPU bridge that enables communication and synchronization between multiple instances of GPGPU 700. In one embodiment, GPU links 710 are coupled with a high-speed interconnect for transmitting and receiving data to other GPGPUs or parallel processors. In one embodiment, multiple instances of GPGPU 700 are located in separate data processing systems and communicate via a network appliance that is accessible via host interface 702. In one embodiment, GPU links 710 can be configured to enable connection to a host processor in addition to or as an alternative to host interface 702.

[0168] While the illustrated configuration of GPGPU 700 can be configured to train a neural network, one embodiment provides an alternative configuration of GPGPU 700 that can be configured for deployment in a high-performance or low-power inferencing use platform. In an inferencing configuration, GPGPU 700 includes fewer compute clusters 706A-H relative to the training configuration. Additionally, the memory technology associated with memory 714A-B can vary between the inferencing and training configurations. In one embodiment, the inferencing configuration of GPGPU 700 can support inferencing-specific instructions. For example, the inferencing configuration can provide support for one or more 8-bit integer dot product instructions, which are commonly used during inferencing operations for a deployed neural network.

[0169] Figure 8 A multi-GPU computing system 800 according to an embodiment is shown. Multi-GPU computing system 800 can include a processor 802 coupled to a plurality of GPGPUs 806A-D via a host interface switch 804. In one embodiment, host interface switch 804 is a PCI Express switch device that couples processor 802 to a PCI Express bus over which processor 802 can communicate with the set of GPGPUs 806A-D. Each of the plurality of GPGPUs 806A-D can be Figure 7instances of GPGPUs 700. GPGPUs 806A-D can be interconnected via a set of high-speed point-to-point GPU-GPU links 816. The high-speed GPU-GPU links can be connected to each of GPGPUs 806A-D via a dedicated GPU link, such as GPU link 710 as described in Figure 7 P2P GPU links 816 enable direct communication between each of GPGPUs 806A-D without having to communicate through the host interface bus to which processor 802 is connected. In cases where GPU-GPU traffic is for P2P GPU links, the host interface bus can still be used for system memory access or to communicate with other instances of multi-GPU computing system 800 (e.g., via one or more network devices). While in the illustrated embodiment GPGPUs 806A-D are connected to processor 802 via host interface switch 804, in one embodiment processor 802 includes direct support for P2P GPU links 816 and can be directly connected to GPGPUs 806A-D.

[0170] Machine Learning Neural Network Implementation

[0171] The computing architecture provided by the embodiments described herein can be configured to perform these types of parallel processing that are particularly well suited for training and deploying neural networks for machine learning. Neural networks can be generalized as networks of functions having a graph relationship. As is well known in the art, there are multiple types of neural network implementations used in machine learning. One exemplary type of neural network is a feedforward network as previously described.

[0172] A second exemplary type of neural network is a convolutional neural network (CNN). CNNs are specialized feed-forward neural networks used to process data having a known, grid-like topology, such as image data. Thus, CNNs are commonly used in computer vision and image recognition applications, but they can also be used for other types of pattern recognition, such as speech and language processing. Nodes in the input layer of a CNN are organized into groups of "filters" (feature detectors inspired by the receptive fields found in the retina), and the output of each group of filters is propagated to nodes in successive layers of the network. The computations used for a CNN include applying a convolution mathematical operation to each filter to produce the output of the filter. Convolution is a specialized mathematical operation performed by two functions to produce a third function that is a modified version of one of the original functions. In convolution network terminology, the first function with respect to the convolution can be referred to as the input, and the second function can be referred to as the convolution kernel. The output can be referred to as a feature map. For example, the input to a convolutional layer can be a multidimensional data array that defines various color components of an input image. The convolution kernel can be a multidimensional array of parameters, where the parameters are adapted through a training process for the neural network.

[0173] A recurrent neural network (RNN) is a type of feed-forward neural network that includes feedback connections between layers. RNNs enable modeling of sequential data by sharing parameter data across different parts of the neural network. The architecture of an RNN includes loops. These loops represent the influence of a current value of a variable on its own value at a future time, as at least a portion of the output data from the RNN is used as feedback for processing subsequent input in the sequence. This feature makes RNNs particularly useful for language processing due to the variable nature in which language data can be composed.

[0174] The diagrams described below present exemplary feed-forward, CNN, and RNN networks, and describe general processes for training and deploying each of those types of networks, respectively. It will be understood that these descriptions are exemplary and non-limiting with respect to any particular embodiment described herein, and that the concepts illustrated can generally be applied to deep neural networks and machine learning techniques in general.

[0175] The exemplary neural networks described above can be used to perform deep learning. Deep learning is machine learning using deep neural networks. In contrast to shallow neural networks that include only a single hidden layer, deep neural networks used in deep learning are artificial neural networks composed of multiple hidden layers. More deeply neural networks are generally more computationally intensive to train. However, the additional hidden layers of the network enable multi-step pattern recognition that results in reduced output error relative to shallow machine learning techniques.

[0176] Deep neural networks used in deep learning typically include a front-end network for performing feature recognition coupled to a back-end network representing a mathematical model, which can then perform operations (e.g., object classification, speech recognition, etc.) based on the feature representations provided to the model. Deep learning enables machine learning to be performed without requiring manual feature engineering on the model. Instead, deep neural networks can learn features based on statistical structure or correlations within the input data. The learned features can be provided to a mathematical model, which can then map the detected features to the output. The mathematical model used by the network is typically specialized for a specific task to be performed, and different models will be used to perform different tasks.

[0177] Once a neural network is structured, a learning model can be applied to it to train it to perform a specific task. The learning model describes how weights are adjusted within the model to reduce the network's output error. Backpropagation of error is a common method used to train neural networks. An input vector is presented to the network for processing. The network's output is compared to the expected output using a loss function, and an error value is calculated for each neuron in the output layer. These error values ​​are then backpropagated until each neuron has an associated error value that roughly represents its contribution to the original output. The network can then learn from those errors using an algorithm (e.g., stochastic gradient descent) to update the neural network's weights.

[0178] Figure 9A To B, we will demonstrate an exemplary convolutional neural network. Figure 9A Show the individual layers within a CNN. For example... Figure 9A As shown, an exemplary CNN for modeling image processing can receive input 902, which describes the red, green, and blue (RGB) components of an input image. Input 902 can be processed by multiple convolutional layers (e.g., convolutional layer 904, convolutional layer 906). Optionally, the output from the multiple convolutional layers can be processed by a set of fully connected layers 908. Neurons in a fully connected layer have full connections to all activation functions in the previous layer, as previously described for feedforward networks. The output from the fully connected layer 908 can be used to generate an output from the network. Matrix multiplication can be used instead of convolution to compute the activation functions within the fully connected layer 908. Not all CNN implementations use fully connected layers 908. For example, in some implementations, convolutional layers 906 can generate the CNN output.

[0179] Convolutional layers are sparsely connected, unlike the traditional neural network configuration found in fully connected layers 908. Traditional neural network layers are fully connected, such that each output unit interacts with every input unit. However, convolutional layers are sparsely connected, in that the output of the convolution of the receptive field (rather than the respective state value of each node in the receptive field) is input to the nodes of the subsequent layer, as illustrated. The kernel associated with the convolutional layer performs a convolution operation, the output of which is sent to the next layer. The dimensionality reduction performed within the convolutional layer is one aspect that enables the CNN to scale to handle large images.

[0180] Figure 9B An exemplary computation stage within a convolutional layer of a CNN is illustrated. The input 912 to the convolutional layer of a CNN can be processed in three stages of the convolutional layer 914. The three stages can include a convolution stage 916, a detector stage 918, and a pooling stage 920. The convolutional layer 914 can then output data to a successive convolutional layer. The last convolutional layer of the network can generate output feature map data or provide input to a fully connected layer, for example, to generate a classification value to the input to the CNN.

[0181] Several convolutions are performed in parallel in the convolution stage 916 to produce a set of linear activation functions. The convolution stage 916 can include an affine transformation, which is any transformation that can be specified as a linear transformation plus a translation. Affine transformations include rotation, translation, scaling, and combinations of these transformations. The convolution stage computes the output (e.g., neuron) of a function connected to a particular region in the input, which can be determined as a local region associated with the neuron. The neuron computes a dot product between the neuron's weights and the region in the local input to which the neuron is connected. The output from the convolution stage 916 defines a set of linear activation functions that are processed by successive stages of the convolutional layer 914.

[0182] The linear activation functions can be processed by the detector stage 918. In the detector stage 918, each linear activation function is processed by a non-linear activation function. The non-linear activation function increases the non-linear properties of the overall network without affecting the receptive field of the convolutional layer. Several types of non-linear activation functions can be used. One particular type is a rectified linear unit (ReLU), which uses an activation function defined as f(x) = max(0, x) such that the activation function is thresholded to zero.

[0183] The pooling stage 920 uses a pooling function that replaces the output of the convolutional layer 906 with summary statistics of nearby outputs. Pooling functions can be used to introduce translational invariance into a neural network, such that slight translations to the input do not change the pooling output. Local translational invariance can be useful in cases where the presence of a feature in the input data is more important than the exact location of the feature. Various types of pooling functions can be used during the pooling stage 920, including max pooling, average pooling, and L2 norm pooling. Additionally, some CNN implementations do not include a pooling stage. Instead, such implementations substitute an additional convolutional stage with an increased stride relative to the previous convolutional stage.

[0184] The output from the convolutional layer 914 can then be processed by a next layer 922. The next layer 922 can be an additional convolutional layer or one of the fully connected layers 908. For example, Figure 9A The first convolutional layer 904 of the convolutional neural network 900 can output to a second convolutional layer 906, which can output to a first layer of the fully connected layers 908.

[0185] Figure 10 An exemplary recurrent neural network 1000 is shown. In a recurrent neural network (RNN), the previous state of the network influences the output of the current state of the network. A wide variety of functions can be used in a wide variety of ways to establish an RNN. The use of RNNs often revolves around using a mathematical model to make predictions of the future based on a sequence of previous inputs. For example, an RNN can be used to perform statistical language modeling to predict an upcoming word given a sequence of previous words. The RNN 1000 shown can be described as having an input layer 1002 that receives an input vector, a hidden layer 1004 that implements a recurrent function, a feedback mechanism 1005 that implements a'memory' of previous states, and an output layer 1006 that outputs a result. The RNN 1000 operates based on time steps. The state of the RNN at a given time step is influenced by previous time steps via the feedback mechanism 1005. The state of the hidden layer 1004 is defined for a given time step by the previous state and the input at the current time step. An initial input (xi) at a first time step can be processed by the hidden layer 1004. A second input (x2) can be processed by the hidden layer 1004 using state information determined during processing of the initial input (xi). The given state can be computed as s t = f(Ux t + Ws t-1), where U and W are parameter matrices. The function f is typically non-linear, such as the hyperbolic tangent function (Tanh) or a variant of the rectified function f(x) = max(0, x). However, the particular mathematical function used in the hidden layer 1004 can vary depending on the particular implementation details of the RNN 1000.

[0186] In addition to the basic CNN and RNN networks described, variations of those networks can be implemented. One example RNN variant is a long short-term memory (LSTM) RNN. LSTM RNNs are capable of learning long-term dependencies that can be necessary for processing longer language sequences. A variant of a CNN is a convolutional deep belief network, which has a structure similar to a CNN and is trained in a manner similar to a deep belief network. A deep belief network (DBN) is a generative neural network composed of multiple layers of stochastic (random) variables. A DBN can be trained layer by layer using greedy unsupervised learning. The learned weights of a DBN can then be used to provide a pre-trained neural network by determining a set of optimal initial weights for a neural network.

[0187] Figure 11 Training and deployment of deep neural networks is shown. Once a given network has been structured for a task, the neural network is trained using a training data set 1102. Various training frameworks 1104 have been developed for implementing hardware acceleration of the training process. For example, Figure 6 The machine learning framework 604 of FIG. 6 can be configured as a training framework 604. The training framework 604 can hook into an untrained neural network 1106 and enable the untrained neural network to be trained using the parallel processing resources described herein to generate a trained neural network 1108.

[0188] To begin the training process, the initial weights can be selected randomly or by pre-training using a deep belief network. The training loop is then performed in a supervised or unsupervised manner.

[0189] Supervised learning is a method of learning in which training is performed as an arbitration operation, such as when the training dataset 1102 includes inputs paired with expected outputs for the inputs, or in cases where the training dataset includes inputs with known outputs and the output of the neural network is manually graded. The network processes the inputs, and the resulting outputs are compared to a set of expected or desired outputs. Errors are then backpropagated through the system. The training framework 1104 can make adjustments to adjust the weights that control the untrained neural network 1106. The training framework 1104 can provide tools for monitoring how well the untrained neural network 1106 is converging to a model that is suitable for generating correct answers based on known input data. The training process occurs repeatedly as the weights of the network are adjusted to improve the output generated by the neural network. The training process can continue until the neural network reaches a statistically expected level of accuracy associated with a trained neural network 1108. The trained neural network 1108 can then be deployed to implement any number of machine learning operations.

[0190] Unsupervised learning is a method of learning in which the network attempts to train itself using unlabeled data. Thus, for unsupervised learning, the training dataset 1102 will include input data without any associated output data. The untrained neural network 1106 can learn groupings within the unlabeled inputs, and can determine how individual inputs relate to the overall dataset. Unsupervised training can be used to generate self-organizing maps, which are a type of trained neural network 1107 that can perform operations useful in data reduction. Unsupervised training can also be used to perform anomaly detection, which allows for the identification of data points in an input dataset that deviate from the normal patterns of the data.

[0191] Variations of supervised and unsupervised training can also be employed. Semi-supervised learning is a technique in which the training dataset 1102 includes a mix of labeled and unlabeled data of the same distribution. Incremental learning is a variation of supervised learning in which input data is used continuously for further training of the model. Incremental learning enables the trained neural network 1108 to adapt to new data 1112 without forgetting the knowledge rooted within the network during initial training.

[0192] Regardless of whether supervised or unsupervised, the training process for particularly deep neural networks can be too computationally intensive for a single computing node. Rather than using a single computing node, a distributed network of computing nodes can be used to speed up the training process.

[0193] Figure 12is a block diagram illustrating distributed learning. Distributed learning is training a model that uses multiple distributed computing nodes to perform supervised or unsupervised training of a neural network. The distributed computing nodes can each include one or more host processors and one or more of general purpose processing nodes, such as highly parallel general purpose graphics processing units 700 as in FIG. 700. As illustrated, distributed learning can perform model parallelization 1202, data parallelization 1204, or a combination of model and data parallelization 1204.

[0194] In model parallelization 1202, different computing nodes in a distributed system can perform training computations on different parts of a single network. For example, each layer of a neural network can be trained by different processing nodes of a distributed system. Benefits of model parallelization include the ability to scale to particularly large models. Splitting computations associated with different layers of a neural network enables training of very large neural networks where the weights of all layers would not fit in the memory of a single computing node. In certain instances, model parallelization is particularly useful for performing unsupervised training of large neural networks. In another example of model parallelization, computations in one or more layers of a neural network model can be split across multiple computing nodes across the feature map dimension to reduce the size of each node model parameter.

[0195] In data parallelization 1204, different nodes of a distributed network have a complete instance of a model, and each node receives a different portion of data. Results from different nodes are then combined. While different approaches for data parallelization are possible, data parallel training approaches all require a technique to combine results and synchronize model parameters between each node. Exemplary methods for combining data include parameter averaging and update-based data parallelization. Parameter averaging trains on a subset of training data on each node and sets global parameters (e.g., weights, biases) to the average of the parameters from each node. Parameter averaging uses a central parameter server that holds parameter data. Update-based data parallelization is similar to parameter averaging except that updates to the model are passed instead of passing parameters from nodes to a parameter server. Additionally, update-based data parallelization can be performed in a decentralized manner where updates are compressed and passed between nodes.

[0196] For example, combined model and data parallelization 1206 can be implemented in a distributed system where each computing node includes multiple GPUs. Each node can have a complete instance of a model, with individual GPUs within each node used to train different portions of the model.

[0197] Distributed training has increased overhead relative to training on a single machine. However, the parallel processors and GPGPUs described herein can each implement techniques for reducing the overhead of distributed training, including techniques for implementing high-bandwidth GPU-GPU data transfers and accelerated remote data synchronization.

[0198] Exemplary Machine Learning Applications

[0199] Machine learning can be applied to solve a number of technical problems, including but not limited to computer vision, autonomous driving and navigation, speech recognition, and language processing. Computer vision has traditionally been one of the most active areas of research for machine learning applications. Applications of computer vision range from replicating human visual capabilities (e.g., recognizing human faces) to creating new classes of visual capabilities. For example, a computer vision application can be configured to recognize sound waves from vibrations induced in objects visible in a video. Parallel processor-accelerated machine learning enables training of computer vision applications using training data sets significantly larger than previously feasible, and enables deployment of inference- use systems using low-power parallel processors.

[0200] Parallel processor-accelerated machine learning has applications in autonomous driving, including lane and road sign recognition, obstacle avoidance, navigation, and driving control. Accelerated machine learning techniques can be used to train driving models based on data sets that define appropriate responses to particular training inputs. The parallel processors described herein can enable fast training of increasingly complex neural networks for autonomous driving solutions, and enable deployment of low-power inference-use processors in mobile platforms suitable for integration into autonomous vehicles.

[0201] Parallel processor-accelerated deep neural networks have implemented machine learning approaches for automatic speech recognition (ASR). ASR includes creating a function that computes the most likely sequence of language given an input acoustic sequence. Accelerated machine learning using deep neural networks has implemented in place of hidden Markov models (HMMs) and Gaussian mixture models (GMMs) previously used for ASR.

[0202] Parallel processor-accelerated machine learning can also be used to accelerate natural language processing. Automated learning programs can use statistical inference algorithms to produce models that are robust to erroneous or unfamiliar input. An exemplary natural language processor application includes automatic machine translation between human languages.

[0203] Parallel processing platforms for machine learning can be divided into training platforms and deployment platforms. Training platforms are typically highly parallel and include optimizations for accelerating multi-GPU single-node training and multi-node multi-GPU training. Example parallel processors suitable for training include the highly parallel general-purpose graphics processing unit 700 of FIG. 700 and the multi-GPU computing system 800 of FIG. 800. In contrast, deployment machine learning platforms typically include low-power parallel processors suitable for use in products such as cameras, autonomous robots, and autonomous vehicles.

[0204] Figure 13 An example inference system-on-a-chip (SOC) 1300 suitable for performing inference using trained models is shown. The SOC 1300 can integrate multiple processing components, including a media processor 1302, a vision processor 1304, a GPGPU 1306, and a multi-core processor 1308. The SOC 1300 can additionally include on-chip memory 1305, which can implement a shared on-chip data pool accessible by each of the processing components. The processing components can be optimized for low-power operation for enabling deployment to a wide variety of machine learning platforms, including autonomous vehicles and autonomous robots. For example, one implementation of the SOC 1300 can be used as part of a main control system for an autonomous vehicle. In cases where the SOC 1300 is configured for use in an autonomous vehicle, the SOC is designed and configured to meet relevant functional safety standards for the deployment jurisdiction.

[0205] During operation, the media processor 1302 and the vision processor 1304 can work in concert to accelerate computer vision operations. The media processor 1302 can enable low-latency decoding of multiple high-resolution (e.g., 4K, 8K) video streams. The decoded video streams can be written to a buffer in the on-chip memory 1305. The vision processor 1304 can then parse the decoded video and perform preliminary processing operations on frames of the decoded video in preparation for processing the frames using trained image recognition models. For example, the vision processor 1304 can accelerate convolution operations for CNNs used to perform image recognition on high-resolution video data, while back-end model computations are performed by the GPGPU 1306.

[0206] The multi-core processor 1308 can include control logic to facilitate ordering and synchronization of data transfers and shared memory operations performed by the media processor 1302 and the vision processor 1304. The multi-core processor 1308 can also act as an application processor to execute software applications that can use the inferencing computing capabilities of the GPGPU 1306. For example, at least a portion of the navigation and driving logic can be implemented in software that executes on the multi-core processor 1308. Such software can issue computing workloads directly to the GPGPU 1306, or can issue computing workloads to the multi-core processor 1308, which can offload at least a portion of those operations to the GPGPU 1306.

[0207] The GPGPU 1306 can include compute clusters, such as the low-power configuration of the compute clusters 706A-706H within the highly parallel general-purpose graphics processing unit 700. The compute clusters within the GPGPU 1306 can support instructions that are explicitly optimized for performing inferencing computations on trained neural networks. For example, the GPGPU 1306 can support instructions for performing low-precision computations, such as 8-bit and 4-bit integer vector operations.

[0208] Abstraction Layer for Scalable Distributed Machine Learning

[0209] Currently, data scientists developing applications using distributed deep learning explicitly implement the communication system between computing nodes. Implementing the underlying communication system for distributed deep learning requires some knowledge of distributed or networking computing node communication techniques, including the libraries required to implement such techniques. For example, to implement a distributed deep learning model, such as data parallelism, model parallelism, or hybrid parallelism (a mix of data and model parallelism), an application developer can be required to explicitly construct the communication infrastructure using a low-level communication library, such as a Message Passing Interface (MPI) library. The application developer would then need to determine the specific data units to be communicated and the specific nodes that will transmit and receive such information. Since a deep learning application developer can not be an expert in the field of constructing distributed computing infrastructures, many best practices and optimizations can not be included in the communication implementation developed for a given deep learning application. Distributed processing for deep learning is often performed to accelerate the training process for neural networks. Various forms of distributed deep learning, such as model parallelism 1202, data parallelism 1204, and model and data parallelism 1206, can be performed, as shown. Model and data parallelism can also be referred to as hybrid parallelism, and generally describes any combination of model and data parallelism. Figure 12

[0210] Distributed machine learning can be implemented using various parallel modes, such as data parallelism, model parallelism, or a hybrid of data and model parallelism, as shown in​Figure 12 As described with respect to Figure 12 Data parallelism uses the same model for each compute node, each node processing a different portion of the data. Model parallelism uses the same data for each compute node, the model split across the compute nodes.

[0211] To enable communication, multiple types of low-level communication patterns are used to pass data between nodes. The low-level communication patterns used are shown in Table 5 below.

[0212] Table 5 - Low-level communication operations

[0213]

[0214] Figures 14A-14E Communication patterns used during distributed machine learning computation operations performed across multiple compute nodes according to embodiments described herein are shown. Figure 14A Data passing for machine learning computation using data parallelism is shown. Figure 14B Data passing for distributed machine learning computation using model parallelism is shown. Figure 14C Splitting a machine learning computation across multiple nodes using hybrid parallelism is shown. Figure 14D Distributed machine learning computation using hybrid parallelism across multiple nodes and across multiple layers is shown. Figure 14E A set of example messaging pattern operations that can be used for distributed machine learning is shown. In Figures 14A-14E In each of the figures in

[0215] As shown in Figure 14A Data parallelism can be implemented where input data 1402 is split along the mini-batch dimension and the same model is replicated across nodes. The mini-batch is split across several compute nodes, each node responsible for computing gradients for all model parameters using a subset of samples in the mini-batch. Forward propagation is done independently on each node. In one embodiment, only one communication is performed during backward pass to compute the average of the gradients for the learnable parameters. A global reduce operation 1405 is used to update the weights for each layer for the next forward pass. In one embodiment, distributed weight updates can be enabled where a reduce_scatter is used to compute the average of the gradients before performing stochastic gradient descent and a global gather operation is used to synchronize the weights across nodes after stochastic gradient descent.

[0216] As shown in Figure 14BAs shown, model parallelism can be achieved by splitting the model or a set of weights across multiple nodes. Typically, model parallelism executes different parts of the model's computation simultaneously on different nodes within the same batch of examples. For model parallelism, the input data is also split (e.g., along the channel dimension), such as... Figure 14B As shown in the diagram. Using the method shown, a reduction operation is performed to calculate the total number of activations to obtain the actual output, and then the activations are spread to be used to calculate the activations for the next layer. A reduction_spread1407 operation can be performed to pass data in a single communication operation. In the backpass, a full aggregation operation is performed to combine the gradient bars calculated at each node.

[0217] like Figure 14C As shown, hybrid parallelism can be performed, where partitioning occurs on activations and weights to minimize the skew matrix. For a layer of the neural network, input data 1402, weight data 1404, and / or activation data 1406 are partitioned and distributed across multiple computation nodes (e.g., nodes 0-3). Node 0 receives a first input data block 1402A and weight data 1404A. Computation is performed at node 0 to generate a first partial activation 1406A. Similarly, node 1 receives a second input data block 1402B and weight data 1404B. Computation is performed at node 1 to generate a second partial activation 1406B. Node 2 can perform computation on a third input data 1402C and weight data 1404C to generate a third partial activation 1406C. Node 3 can perform computation on a fourth input data 1402D and weight data 1404D to generate a fourth partial activation 1406D.

[0218] Figure 14D The transfer of partial activation data 1406A-1406B from a given layer (layer N-1) of a neural network to subsequent layers (layer N) is illustrated. A set of partial activations 1406A-1406B is generated via multiple nodes (node ​​0, node 1) based on the application of mathematical operations (e.g., convolution) to the input data 1402A-1402B and the weight data 1404A-1404B. For example, in one embodiment, a reduction-spreading operation 1410 is used, which performs a reduction operation on the partial activations 1406A-1406B from the multiple nodes of layer N-1 and spreads the result to multiple nodes as activations for use in layer N of the neural network.

[0219] Figure 14EExemplary communication operations for transferring data for distributed training of a neural network for machine learning operations are shown. A low-level messaging library is used to enable data transfer for weight and activation data during distributed training of a neural network. An exemplary neural network having N layers 1421A, 1421B, 1421N (e.g., layer 1, layer 2 through layer N) can be trained distributively by performing successive forward computation operations on successive layers to enable forward propagation 1426 of activation data through the neural network. During forward propagation 1426, activation data is transferred from a first layer 1421A to a successive layer 1421B using an all-to-all communication operation 1409, for example, where the first and successive layers are hidden or non-output layers. The all-to-all 1409 operation transfers unique data from the computing node that generated the activation or partial activation data to all available recipients that use the activation data as input data for operations on the successive layer. When transferring data to the final layer (e.g., layer N), a reduce scatter operation 1410 is performed, which is described with respect to Figure 14B forward propagation 1426. During backward propagation 1428, distributed stochastic gradient descent is performed to generate updated weight data. An initial all-reduce operation 1412 is performed on layer N, and a set of all-reduce operations 1411A, 1411B, 1411N are performed to update the weights for each layer for the next forward pass. An all-reduce operation is a reduce operation whose result is broadcast or transferred to the receive buffers of all processes in the communication group. Backward propagation 1428 can also include all-gather 1413 and all-to-all 1414 communication operations. For the all-gather operation 1413, data is gathered from all tasks and the combined data is distributed to all tasks. For the all-to-all operation 1414, data from all processes is transferred to all processes.

[0220] Any low-level messaging library (such as MPI, gRPC, or zeroMQ) can be used to implement the data transfer required to perform distributed computation operations for machine learning. However, without domain-level expertise in multi-processor communication libraries, it can be difficult to implement exemplary communication operations. Furthermore, it can be difficult to scale these operations to very large numbers of nodes. Without specific domain knowledge about distributed computing techniques, an implementation can handle scalable communication systems for machine learning that communicate between hundreds or thousands of nodes can significantly lengthen development time for machine learning applications.

[0221] The embodiments described herein provide various techniques to abstract distributed communication system details for deep learning applications. In one embodiment, a machine learning extension library (MLSL) is provided that enables deep learning application developers to develop distributed deep learning applications without the need for knowledge of specific communication details required to enable multi-node deep learning. Application developers of deep learning applications can specify the type of distributed computing system used by the application and library techniques provided by the embodiments described herein using deep learning domain specific terminology, which can enable the specific underlying communication methods required to enable the requested distributed computing system.

[0222] Figures 15A-15C Architecture details of the machine learning extension library provided by the embodiments described herein are shown. Figure 15A An exemplary machine learning architecture stack 1500 is shown. Figure 15B Details of the MLSL architecture 1511 are shown. Figure 15C An exemplary communication endpoint enabled by the embodiments is shown.

[0223] Figure 15A An exemplary machine learning architecture stack 1500 is shown, which can be a variant of the machine learning software stack 600 of Figure 6 . The machine learning architecture stack 1500 includes multiple software and hardware layers from input data 1502 provided from a sensor array to hardware 1514 elements that perform various computing, storage, or communication operations. Each layer of the exemplary machine learning architecture stack 1500 can be an opaque abstraction layer that hides implementation details of higher layers while using functions provided by lower layers to implement functions required by higher layers.

[0224] Input data 1502 is provided to an application layer 1504. In one embodiment, input data 1502 is multi-modal input, including but not limited to video and / or image data, data from multiple sensors, and external signal data. Application 1504 includes a multi-modal fusion and decision application that can process the input to enable machine learning tasks such as image understanding, video summarization, speech and natural language processing, path planning, navigation, or any other machine learning implementation described herein. Application 1504 communicates with one or more machine learning frameworks 1506, such as but not limited to Caffe, Theano, Torch, TensorFlow, or any other script-based machine learning framework, to implement machine learning specific operations. Machine learning framework 1506 can enable machine learning operations to be performed using any number of neural network topologies 1508, including but not limited to CNNs, RNNs, LSTMs, general deep neural networks, and reinforcement learning networks. Machine learning framework 1506 implements neural network topologies 1508 via one or more building blocks 1510. Exemplary building blocks 1510 include single-precision floating point general matrix multiplication (SGEMM) blocks, convolution building blocks, fast Fourier transform / Winograd blocks, single-source shortest path (SSSP) computation blocks, sparse matrix-matrix multiplication (SpGEMM) blocks, and a machine learning extension library (MLSL) 1511 provided by embodiments described herein. Building blocks 1510 can each implement a variety of algorithms 1512 to enable the computational operations requested by framework 1506 to implement neural network topologies 1508. Algorithms 1512 include optimizations to enhance statistical and architectural efficiency, enable cloud deployment, and enable scaling to a large number of nodes. In one embodiment, MLSL 1511 includes algorithms 1512 to enable scaling of machine learning operations to a large number of nodes. In one embodiment, building blocks 1510 can be implemented via software libraries that can be accelerated by one or more elements of hardware 1514. In one embodiment, at least a portion of building blocks 1510 can be implemented within hardware 1514. For example, FPGA or ASIC based accelerators can include custom logic to enable a portion of MLSL 1511 or one or more GEMM libraries.

[0225] The functions of the higher layers of the machine learning architecture stack 1500 can be implemented using various components of the hardware 1514. The components of the hardware 1514 include, but are not limited to, a CPU or another general purpose processor tasked with performing computations related to a compute and / or operating system. The hardware 1514 also includes parallel processing systems based on integrated many-core (MIC) or general purpose GPUs. In some embodiments, the hardware 1514 includes deep learning accelerators based on FPGAs or ASICs. The structural interconnect components of the hardware 1514 are used to enable high speed communication between the various components and high bandwidth volatile or non-volatile memory. Volatile memory technologies can include any of the graphics memory technologies described herein, including HBM and GDDR memory. Non-volatile memory technologies can include flash memory (including 3D NAND flash memory), or other memory technologies such as 3D Xpoint memory.

[0226] Figure 15B Details of the MLSL architecture 1511 are illustrated in accordance with embodiments. The MLSL architecture 1511 includes abstraction layers with machine learning specific abstractions 1513 as well as non-machine learning specific abstractions 1515. The abstractions interface with a communication module 1517 that drives a lower level messaging library 1519. The messaging library 1519 uses optimized low level communication routines to transfer data over a high performance communication fabric 1521.

[0227] The MLSL architecture 1511 enables developers of machine learning software to develop scalable machine learning applications using machine learning specific abstractions 1513. In one embodiment, the machine learning specific abstractions 1513 enable application developers to use machine learning domain specific knowledge to drive scalable performance of compute operations for neural network layers. The machine learning abstractions 1513 enable applications to be developed in a manner that is transparent to the underlying architecture, enabling machine learning applications to automatically adapt to any number of hardware 1514 elements, including multiple types of compute and fabric elements. In addition to the machine learning specific abstractions 1513, a set of non-machine learning specific abstractions 1515 can also be provided by the MLSL architecture 1511. The non-machine learning specific abstractions 1515 enable developers of machine learning applications to define one or more non-machine learning details of the application at a higher level of abstraction, such as one or more implementation specific details or operating system details that are not related to machine learning.

[0228] In one embodiment, the machine learning specific abstraction 1513 enables neural network layers to be adapted to support multiple types of parallelism (e.g., data, machine, hybrid). The machine learning specific abstraction 1513 also supports layer-to-layer communication abstraction in order to allow developers to easily implement communication patterns for different layer types and parallelism. Different layer types and parallelism are defined using machine learning specific terminology that uses the machine learning specific abstraction 1513 and communication for those layer types is enabled through the communication module 1517, messaging library 1519, and high performance communication fabric 1521. The machine learning specific abstraction 1513 also enables intelligent message scheduling across defined neural network layers while abstracting data layout and transformations needed to implement machine learning techniques at the application level.

[0229] In one embodiment, the communication module 1517 includes logic that drives the underlying messaging library 1519. The communication module 1517 includes various optimizations to enable efficient driving of the network while transferring machine learning data between computing nodes for performing distributed machine learning. The communication module 1517 includes logic for optimizing network bandwidth and enabling low latency communication. The machine learning specific abstraction 1513 and / or the non-machine learning specific abstraction 1515 can specify or prove interfaces to enable application developers to specify processor resources that manage distributed communication. In one embodiment, a specific processor can be specified. In one embodiment, a number of processors associated with communication is specified. In one embodiment, a mix between compute and communication resources can be specified. In one embodiment, the communication module 1517 includes logic for adaptively allocating processor cores for driving and performing operations for the communication module 1517 and / or messaging library 1519. In one embodiment, the communication module 1517 can adaptively allocate processing resources for communication without explicit direction from the machine learning specific abstraction 1513 or the non-machine learning specific abstraction 1515. In one embodiment, the communication module 1517 can adaptively adjust or allocate processing resources in an attempt to fully saturate available network resources, in an attempt to minimize latency effects of communication within a distributed system. For example, if the communication module 1517 determines that a high performance communication fabric 1521 is not fully saturated with data, additional processors or processor cores can be allocated to perform network tasks if the overall throughput of the distributed computing system would be increased. In one embodiment, the amount of compute resources allocated to drive the messaging library 1519 can vary based on the bandwidth of the high performance communication fabric. For higher bandwidth fabrics, greater compute resources can be needed to saturate the network. The high performance communication fabric 1521 can be implemented through any number of high speed network connection technologies, including but not limited to Ethernet, InfiniBand technology, Omni-Path interconnect, or a mesh via point-to-point interconnects such as NvLink.

[0230] In one embodiment, the communication module 1517 includes logic that enables asynchronous communication between processing nodes to ensure forward progress of distributed computing operations. The asynchronous communication enabled by the communication module 1517 allows for overlapping computation and communication operations that are efficiently interleaved to optimize computation and communication efficiency and throughput. In one embodiment, the communication module 1517 also supports prioritized communication channels to enable prioritized resolution of competing communication requests.

[0231] The messaging library 1519 uses optimized low-level communication routines to transfer data over the high-performance communication fabric 1521. The MLSL architecture 1511 is agnostic to the underlying messaging library 1519 and high-performance communication fabric 1521. In one embodiment, the messaging library is an MPI-based library. In such an embodiment, MPI functions (e.g., MPI_Alltoall, MPI_Allreduce, MPI_Allgather, etc.) are used to implement the communication patterns used by the machine learning application. In some embodiments, gRPC or zeroMQ libraries and associated functions are used for messaging. In one embodiment, NCCL collective communication routines can also be used. NCCL provides communication routines such as allgather, reduce, and broadcast to accelerate multi-GPU machine learning training across multiple GPGPUs.

[0232] Figure 15C Exemplary communication endpoints enabled by the embodiments described herein are illustrated. While the described techniques are not limited to MPI implementations, the concepts provided by the embodiments are illustrated with respect to MPI libraries. In a conventional communicator 1525, a process is associated with a rank or another communication ID. The process can support communication for multiple threads, each thread being associated with the rank or identifier of the process. The embodiments described herein utilize network endpoints to enable communication between various computing nodes of a distributed computing system. Each endpoint communicator 1530 allows for flexible arrangements between processes, communication ranks or IDs, and various threads that use the endpoints for communication. The endpoint communicator 1530 can be dynamically configured so that a process can be associated with multiple ranks and each rank can be associated with a separate process. In such a configuration, each thread can send data through the multi-processor messaging system without regard to thread contention between ranks. Alternatively, one thread can be associated with multiple ranks, enabling a single thread to have multiple communication channels.

[0233] In one embodiment, one or more instances of endpoint communicator 1530 are explicitly specified via machine learning specific abstraction 1513. In one embodiment, the number of instances of endpoint communicator 1530 is directly related to the number of cores that are allocated for performing network communications. In one embodiment, machine learning specific abstraction 1513 enables programmers to specify the type of network and the required degree of parallelism using machine learning specific terminology, and the MLSL architecture 1511 can dynamically build the communication infrastructure, including the number of cores allocated to networking operations and the associated number of communication endpoints 1530.

[0234] In various embodiments, Figure 15C The communication system of FIG. 16 can be built using explicit developer-defined variables or dynamically built based on a machine learning infrastructure defined by an application developer. In one embodiment, a machine learning application can define a plurality of application processes 1536 that perform computational operations for the machine learning application. The MLSL 1534 can expose an interface to the application processes 1536 enabling a communication system that is scalable to a large number of computational nodes. In such a configuration, multiple communication ranks or identifiers are supported for each application process 1536 (e.g., process 0, process 1, process 2), which in one embodiment can be MPI processes. A set of endpoint processes 1532 can be initiated by the MLSL 1534, with a separate endpoint process defined to support each rank or identifier of the processes within the application processes 1536. In one embodiment, machine learning specific domain awareness can be combined with a global view of the communication operations to determine how many endpoints to use. Based on the communication needs, the MLSL 1534 can dynamically scale the number of endpoint processes 1532.

[0235] Figures 16A-16B A distributed machine learning training enabled by embodiments described herein is shown. Figure 16A A training process for a neural network performed using multiple nodes is shown. The MLSL API can be used to define a distributed training system that includes multiple nodes. In one embodiment, the multiple nodes can include a first node 1610 (node 0) and a second node 1620 (node 1). Each node 1610, 1620 is configured to perform forward computation operations 1612, 1622 and backward computation operations 1614, 1624. For the backward computation 1614, 1624, the weights Δ 1616, 1626 are computed and a stochastic gradient descent 1618, 1628 is performed to generate weight value updates. The communication operations enabled by the MLSL API are represented in letter / number blocks, which perform the operations shown in Table 6.

[0236] Table 6 - MLSL communication operations

[0237] Communication Actions Communication Phases 1. Activation a. Start Communication 2. Activation Gradient b. Wait for End Communication 3. Weight Gradient 4. Updated Weights

[0238] like Figure 16A As shown, the MLSL API uses distributed forward computation 1612, 1622 for forward propagation, which is within the scope of a first communication block that waits for the end of communication of incoming data and a second communication block that begins communication for the data to be computed before starting the forward computation operation. For example, a developer can use a first MLSL API command (node ​​0 [1b]) to configure the first layer forward computation operation 1612 at the first node 1610 to wait for the end of communication of activation data that will be used as input data for the forward computation 1612. The forward computation 1612 starts automatically after the communication of activation data is completed. After the forward computation 1612 is completed, the second MLSL API command (node ​​0 [1a]) can be used to start the communication of activation data. The activation data output from the first node 1610 for communication is the activation data generated by the first layer and is used as input data for the second layer with forward computation 1622 performed at the second node 1620. The forward computation 1622 at the second node 1620 waits for the communication of the activation data to end before starting the computation operation (node ​​1 [1b]), and starts the communication of the activation data (node ​​1 [1b]) generated by the forward computation 1622 when it is completed.

[0239] In one embodiment, the MLSL API uses distributed backward computation 1624, 1612 for backpropagation, which waits for the third MLSL API enabled communication block (node ​​1 [2b]) to finish communication of incoming activation gradients and the fourth MLSL API enabled communication block (node ​​1 [2a]) to begin communication of the computed activation gradients before starting backward computation operation 1624. Similarly, the MLSL API enables the transmission and reception of weight gradients for weight Δ computation 1626 from 1616 and updated weights determined via distributed stochastic gradient updates 1628, 1618.

[0240] like Figure 16B As shown, each node 1610, 1620 can also be used to perform computational operations for multiple layers of the neural network. In one embodiment, Figure 16A The forward computation operations 1612 and 1622 shown are executed as multiple computation operations 1612A-1612B and 1622A-1622B on multiple layers (layer N, layer N+1). Similarly, Figure 16AThe backward pass operations 1624, 1614 shown in the middle can also be performed as multiple compute operations 1624A-1624B, 1614A-1614B. For each node 1610, 1620, the MLSL API can enable the activations 1602, 1604 to pass between multiple neural network layers on each node, while distributing the updated weights 1606, 1608 after the backward pass operations 1624A-1624B, 1614A-1614B.

[0241] In one embodiment, the MLSL API enables the use of different types of parallelization for different layers of the same neural network. The choice of parallelism can be made automatically by the MLSL based on the layer properties, such as the number of learnable parameters and the number of activations. Based on the determined parallelism for a layer, the type of communication required can also be determined. For example, when the current layer uses data parallelism and the next layer uses model parallelism, an all-to-all communication pattern is invoked to redistribute the data. In general, the communication pattern and the scenario can change significantly. By abstracting the communication details, the MLSL API can significantly simplify the life of the machine learning framework developer. Various machine learning structures can be implemented through the MLSL API.

[0242] Hardware-implemented point-to-point primitives for machine learning

[0243] Load and store operations are sufficient to implement higher-order distributed shared memory operations in software. However, load and store primitives alone can not be sufficient to implement an appropriate hardware solution for point-to-point communication. Embodiments described herein can extend simple point-to-point primitives known in the art to implement more rich semantics, such as but not limited to, support for remote atomics, remote procedure calls (RPCs), load with gather list, store with scatter list, and store with notify, including a notification synchronized with the communication to allow more efficient communication and synchronization between nodes and between GPGPUs, MICs, or parallel processor cards within a node, particularly when a point-to-point communication protocol such as a version of the NVLink interconnect / protocol is in use.

[0244] Figure 16CInter-node communication using point-to-point primitives is shown in accordance with an embodiment. In one embodiment, a computing cluster 1630 can perform a distributed neural network training process across multiple nodes. For P number of nodes, each node 1631A-1631P (e.g., node 1 through node P) can perform forward propagation and back propagation as described herein. In one embodiment, each node processes a different set of layers 1632A-1632P, one set per node, where each set has layers L0-Ln. During forward propagation (FP) and back propagation (BP) operations of the neural network, data associated with the layers 1632A-1632P can be exchanged between the nodes. The precise model of data exchange can vary based on the parallel model in use (e.g., data, model, hybrid, etc.). For example and in one embodiment, each node 1631A-1631P can have a copy of the neural network model, and different input data can be provided to each node. A number of weights Δ 1634 can be generated by each node 1631A-1631P. In one implementation, an all-reduce operation can be performed where a summation unit 1636 receives the set of weights Δ in a receive buffer and generates a sum 1637 of the weight Δ data. The summation unit 1636 can be any of the nodes 1631A-1631P or a separate control node. A stochastic gradient descent logic unit (SGD 1638) can generate a new set of weights 1640 which is then sent to each node.

[0245] In one embodiment, to facilitate the exchange of information between the nodes 1631A-1631P, each node can include a hardware-implemented point-to-point primitives unit (PPP 1634A-1634P) to enable the nodes to exchange data in a synchronized manner. The point-to-point techniques described herein can be used in a variety of parallel computing environments such as Figure 15Bimplemented within the communication module 1517 in each node, which can exist within each node. The point-to-point primitive logic within each node can be associated with a set of transmit buffers (TX 1641A-1641P) as well as receive buffers (RX 1642A-1642P). To exchange data between node 1 1631A and node P 1631P, the point-to-point primitive unit 1643P of node P can issue a request to node 1 to send a particular data block to node P. Once the dependencies for that data block are satisfied (such as, for example, when the computation operations associated with that data are complete), node 1 1631A sends the data block to node P 1631P (e.g., via transmit buffer 1641A). Node P 1631P can continue to perform other operations until it is notified by node 1 1631A. Once the computation operations for the requested data are complete, node 1 1631A can perform a remote write to the receive buffer 1642P on node P to write the requested data. In one embodiment, node 1 1631A can notify node P 1631P that the write is complete. In one embodiment, node P 1631P can monitor the address provided to receive buffer 1642P of node 1 1631PA for memory updates, and once node P 1631P notes that a memory write to receive buffer 1642P has occurred, copies out the written memory.

[0246] Various underlying methods can be used to implement point-to-point communication. In one embodiment, remote procedure calls can be used by nodes to remotely execute logical operations on different nodes. Remote logic can monitor the address space associated with the data required by the receiving node, and once the computation operations are complete, initiate a transfer from the sending node. In addition to having "store-with-notification" and remote procedure call support, various additional primitives can be implemented, as described above, where such primitives include, but are not limited to, remote atomics, load with gather list, store with scatter list, and other types of synchronous communication.

[0247] For example, in one embodiment, a remote atomic store can be performed from node 1 1631A to node P 1631P such that the data in node 1's memory is stored to the memory in node P as a single indivisible operation. The atomic store can be performed using hardware primitives supported by the interface logic of node 1 and node P. Node 1 can send a message to node P to implement the atomic store, which can be received by the interface logic of node P. The interface logic in node P can then lock a memory address within node P's memory, perform the store operation, and release the lock on the memory address such that node P's memory can be atomically updated with the data provided from node 1. In one implementation, such an operation can be performed entirely within the fabric interface logic of the nodes. In various embodiments, primitive operations are used to perform load with gather list, store with gather list, or other types of synchronous communication operations within the interface logic of the nodes.

[0248] In computing nodes arranged in a mesh, torus, or hypercube, the routers at each node allow messages to be directed along the most efficient path to a destination for messages between non-directly connected nodes. Embodiments described herein provide fabric interconnect logic that can be configured to route memory writes or messages between indirectly connected nodes. Instead of using node identifiers, in one embodiment, the fabric interconnect logic can route data based on a target memory address associated with a message, write, or packet to be relayed. Such techniques can enable an efficient distributed memory system in hardware, particularly for point-to-point fabric interconnects.

[0249] Figure 17A A multi-node computing system 1700 according to an embodiment is shown. Each node of the multi-node computing system 1700 is coupled to the other nodes via a fabric interconnect 1702. The fabric interconnect 1702 can be a point-to-point interconnect, a mesh, a torus, a hypercube, or other type of interconnect. In one embodiment, the fabric interconnect 1702 is a point-to-point interconnect. Figure 8The multi-GPU computing system 800 is similar. The GPGPUs 806A-806D are each interconnected via a set of P2P GPU links 816. The GPGPUs 1706A-1706D are each interconnected via P2P GPU links 1716. The GPGPUs 1706A-1706D can each communicate with the processor 1702 via the host interface switch 1704. The GPGPUs 806A-806D and the GPGPUs 1706A-1706D are not directly interconnected. However, the GPGPU 806B can communicate with the GPGPU 1706A via the P2P GPU bridge link 1708. In some prior implementations, to transfer data between indirectly connected GPGPUs, such as between GPGPU 806A to GPGPU 1706D, multiple manual transfers can be required. Embodiments described herein enable automatic routing of communications between GPGPUs connected via point-to-point links. In one embodiment, the automatic routing is performed by mapping nodes to a distributed virtual memory system.

[0250] Figure 17BA point-to-point network with distributed virtual addresses is shown in accordance with an embodiment. In one embodiment, a set of nodes 1721 (nodes 1-3) can participate in distributed training of a multi-layer neural network 1722 (layers 0-3). A negotiation can be performed such that memory addresses within each node are associated with virtual addresses within a distributed virtual address space 1730. In one embodiment, a particular physical address range in each node is mapped to a virtual address associated with that node such that the same physical address in each node is mapped to the distributed virtual address space 1730. The distributed virtual address mapping is exchanged between nodes such that each node is aware of the address range of each other node. For example, node 1 can be assigned a memory address within a first memory address range 1723 ([0x1000 to 0x1500]). Node 2 can be assigned a memory address within a second memory address range 1725 ([0x2000 to 0x2500]). Node 3 can be assigned a memory address within a third memory address range 1727 ([0x3000 to 0x3500]). Node 1 can request data from node 3 by issuing a request for data to node 3 and providing node 3 with an address within the first address range 1723. Node 1 knows that node 3 has an address range of [0x3000 to 0x3500] and has a receive buffer in memory at an address defined by the following equation: node-based address + receive buffer offset. Node 1 can request a synchronous write to the receive buffer at node 3, where the write is a message requesting that data be read at a given read address within the address space of node 3 (e.g., the third address range 1727). A fabric interface on node 1 can determine that the message is destined for node 3 based on the write address. The fabric interface can then determine that, in order to communicate with node 3, the message can be routed via node 2. Node 1 can then request a relay via node 2. A fabric interface on node 2 can determine that the message is intended for node 3 based on the destination address and relay the message to the receive buffer in node 3. Such techniques can significantly improve the efficiency of communication operations, such as all-reduce or all-gather operations involving a communication exchange between all nodes.

[0251] Fine-grained compute communication

[0252] Embodiments described herein provide a unified API for expressing compute and communication operations, and corresponding implementations that enable compute and communication operations to be overlapped in a fine-grained manner. Fine-grained overlapping compute provides benefits for machine learning compute frameworks and facilitates higher compute and communication efficiency in multi-card and / or multi-node implementations. For example and in one embodiment, data parallelism is used in weight update operations during back propagation in deep learning training. Back propagation, as above (e.g., inFigure 16A As described in the background section, including weight delta computation, inter-node / inter-card weight delta communication, and updated weight computation using stochastic gradient descent. Several implementations using different computation and communication APIs and implementations can not be able to achieve maximum parallelization, as these operations are sequentially executed in a fixed order and at a coarse-grained level, while such parallelization can be exploited within a compute node or parallel processor card, or across all compute nodes or parallel processors. Embodiments described herein achieve improved computation and communication parallelization by implementing scheduling and execution of computation operations and communication operations at a fine-grained level. By implementing fine-grained computation and communication execution, more efficient computation and communication overlap can be achieved, resulting in improved parallelization and utilization of computation and communication resources.

[0253] Figure 18 An alternative MLSL architecture 1811 according to an embodiment is shown. The alternative MLSL architecture 1811 is similar to the MLSL architecture 1511 of Figure 15B the background section, with the exception that the machine learning specific abstraction and communication module 1816 is merged into a single module. The merging of the abstraction of machine learning computation functions and communication functions enables finer-grained computation and communication overlap. Consumer threads and / or cores that are blocked on pending operations can be put to sleep and woken up once the blocked operations are completed. Computation operations can be performed at the granularity of individual data dependencies, such that as soon as a dependency to a computation unit is satisfied, the computation unit can be executed and immediately transferred to other nodes. A merged API that combines computation and communication operations can be provided, such that computation operations can be associated with communication or synchronization operations or primitives, and as soon as a computation unit is completed, the data of the computation can be transferred.

[0254] Figure 19A Tensor computation operations suitable for fine-grained computation and communication overlap are shown. The merged computation and communication API described herein can enable larger computation operations to be subdivided into smaller blocks, rather than waiting for the entire communication operation to complete before the results are communicated to other nodes, the results can be communicated at a finer granularity as soon as individual portions of the computation operation are completed. For example, Figure 19AFine-grained blocked xGEMM operations are shown that are formulated as multiple small block GEMM operations with parallelism in the M, N, and K dimensions. Intra-mode GEMM kernels optimized for peak architecture performance can be applied in parallel to several sub-units of the computation operation. For example and in one embodiment, a portion of the computation operation can be performed using a first data block 1902 and a second data block 1904 to generate a portion of an N x M matrix 1906. The operation can be performed on the first data block 1902 and the second data block 1904 and the results can be distributed before the entire matrix operation is complete. In such an embodiment, the output [m,n] 1916 can be generated based on a set of multiple fine-grained computation operations (e.g., summations) between weights 1912 and inputs 1914. This technique allows computation and communication operations to overlap not only between layers but also within layers during neural network distributed training. When a sub-computation operation is complete, the operation can be transmitted to the relevant node, e.g., using the point-to-point primitives described herein, enabling greater overlap between computation and communication operations and improving overall system efficiency. This technique is different from existing computation and communication solutions that perform a full set of computation operations before transmitting the results to other nodes.

[0255] Fine-grained execution of computation and communication requires a flexible synchronization mechanism to enable efficient use of orchestration of computation and communication resources. In one embodiment, fine-grained execution and communication is achieved via extended memory semantics at the granularity of arbitrary tensors. Consumer threads and / or cores that are blocked on pending operations to be processed can be put to sleep and upon completion of the blocked operation, they are woken up without relying on polling. Traditional full / empty bit defined at memory / cache line granularity is too fine-grained to be optimal for this purpose. Other solutions such as monitor / mwait can only track contiguous addresses and any memory write within the address range wakes up the consumer. As an improvement over existing solutions, one embodiment provides a method and apparatus for associating tensor operations with synchronization semantics that allows consumer threads to be woken up only when all operations defined on a tensor are complete.

[0256] Figure 19BSynchronized memory access within a multi-node system 1920 is shown in accordance with an embodiment. As shown, a producer node 1930 and a consumer node 1940 each have access to a shared memory 1950, which can be a distributed and shared virtual address space mapped across multiple nodes. The producer node 1930 produces data to be consumed by the consumer node 1940. When the consumer node 1940 has a dependency on tensor data 1955 to be computed by the producer node 1930, the consumer node 1940 can block and sleep until all of the required tensor data 1955 is written to memory 1950. With existing semantics, the consumer node 1940 can be interrupted when any write is performed to the address. However, the use of such semantics can be inappropriate when the consumer node 1940 has a dependency on all of the data to be written. If the consumer node 1940 depends on all of the tensor data 1955, rather than any given unit of tensor data, then waking up any memory access within the tensor data 1955 would result in a large number of false wakeups. Furthermore, in one embodiment, tensors are stored as multi-dimensional arrays in memory that do not have contiguous addresses. Existing semantics can not be able to monitor data that does not reside within a set of contiguous memory addresses. Thus, existing semantics such as monitor and wait semantics cannot be easily used to monitor tensors.

[0257] Embodiments described herein demonstrate a technique that enables a consumer node 1940 to monitor a set of non-contiguous addresses associated with a tensor stored as a multi-dimensional array in memory. In such embodiments, the consumer node 1940 can indicate that tensor data 1955 is to be monitored, and all memory addresses associated with the tensor data 1955 can be monitored for updates. In one embodiment, the consumer node 1940 can request that the monitoring be applied to a particular type of update, such as a remote write to the tensor data 1955 memory addresses from the producer node 1930. Furthermore, a semantic is provided that allows the consumer node 1940 can request to be notified when all of the monitored addresses have been written to. Further, the mechanism can be used to adjust the granularity and maintenance of hardware cache coherency at the synchronization boundary, as shown in Figure 19C

[0258] Figure 19C Figure 19B ​​Memory communication semantics can be extended to implement coarse-grained cache coherency for cache memory data. Because tensor data spans multiple cache lines on both producer nodes and consumer nodes, the techniques described herein can implement cache coherency at a coarser granularity that is more suitable for tensor data than cache line coherency used in conventional systems. For example, in one embodiment, producer node 1930 includes cache memory 1935 that can cache tensor data 1955 generated by compute clusters 1932A-1932B within producer node 1930 prior to being stored to shared memory 1950. Moreover, low-latency access to tensor data 1955 for compute clusters 1942A-1942B of consumer node 1940 can be implemented by cache memory 1945 in consumer node 1940. Instead of maintaining cache coherency at cache line granularity, consistency updates between cache memory 1935 of producer node 1930 and cache memory 1945 of consumer node 1940 can be performed at the monitored granularity of tensor data 1955. These batched consistency updates can result in more efficient use of memory bus bandwidth in the context of coarse memory updates within a multi-node, cache-coherent system such as a multi-GPU system coupled via a cache-coherent interconnect.

[0259] Figures 20A-20B A flow diagram depicting operations for enabling distributed machine learning through an MLSL API is shown. Figure 20A General operations are shown in Figure 20B MLSL settings are shown in

[0260] As shown in Figure 20A As shown in block 2002, general MLSL operations include operations to create a global view of communication operations to be performed among multiple compute nodes of a distributed computing system. The global view is structured using machine learning specific languages that are within the domain knowledge of application developers of machine learning applications. In one embodiment, the global view is internal to the MLSL machine learning abstraction layer and is specified using API commands provided by the MLSL machine learning abstraction layer. The internal global view enables the MLSL to perform an operation called introspection. Introspection enables a deep understanding of the communication performed for machine learning implementations using the global view.

[0261] In one embodiment, the self-test is used to determine the communication cost of communication operations in the graph and to track the overlap of computation and communication operations, as shown in block 2004. In one embodiment, the communication loss is quantified in terms of the computation latency introduced by the data communication between the computation nodes. In addition, the self-test continuously tracks the computation cost of two consecutive communication operations. This information can then be used to efficiently schedule the communication operations. For example, if the computation cost is high, multiple communication operations can be scheduled during this time. For smaller computation cost, the scheduling can choose to send shorter communication operations or split a larger communication operation into smaller parts that can be hidden behind useful computation operations.

[0262] The tracking of the overlap can be used to determine the degree of overlap of the communication time with the useful computation. In one embodiment, the computation and communication times can be obtained by running the actual machine learning topology for a small number of epochs or using a pre-populated database with the latency for different message sizes and layer types for the platform. In one embodiment, the overlap is quantified in terms of the number of cycles of overlapping computation and communication times or the amount of time that the overlapping computation and communication operations are performed. In some embodiments, a combination of such operations can be performed. In one embodiment, the self-test capability can be exposed to the application developer to enable the identification of performance bottlenecks and / or for performing debugging operations.

[0263] In one embodiment, the application developer can explicitly specify the type of parallelism to be used for each layer of the neural network. In one embodiment, the global view and the self-test can be used to automatically determine the type of parallelism to be used for a given layer and a given platform, as shown in block 2006. The optimal type of parallelism to be used can be determined by trying different combinations at runtime or using heuristics. For example, one heuristic can indicate the use of data parallelism when the size of the weights is greater than the size of the activations or model parallelism. The size of the weights or activations is determined based in part on a comparison of the x and y dimensions of the activation and weight data. In one embodiment, during distributed training, multiple types of parallelism can be evaluated at runtime for each layer of the neural network. The parallelism to be used for a layer of the neural network can then be determined based on the indicated communication efficiency of the runtime evaluation.

[0264] In one embodiment, as represented by block 2008, the general MLSL logic additionally performs operations to automatically determine scheduling and prioritization of messages using a global view. For example, a self-test capability enables the MLSL logic to perform operations that attempt various combinations and select an optimal combination of scheduling and prioritization. Scheduling mechanisms include using prioritized channels, delaying low-priority messages, and / or splitting large messages. Such techniques can be implemented at least in part by software logic included within the MLSL library object. The MLSL logic can be performed at least in part in hardware, for example, within a network interface controller or host fabric interface. While some embodiments are capable of automatically determining scheduling and priority of communications, in one embodiment, an application developer can explicitly schedule and prioritize message transmission by scheduling communication start and wait operations for optimal performance, as optimal scheduling and prioritization can vary by platform and neural network topology.

[0265] In one embodiment, as represented by block 2010, based on machine learning specific information specified using the MLSL API, the general MLSL logic additionally performs operations to automatically determine an optimal number of network endpoints and number of cores. In one embodiment, the MLSL API can use machine learning topology to implement specification of a distributed machine learning system, including a neural network topology and number of layers of an associated machine learning network. In one embodiment, the number of endpoints and network communication cores can be inferred using the MLSL global view and platform data. For example, when more cores are used to drive a network, communication performance can be improved. However, allocation of network cores is performed considering a tradeoff between the number of cores used for computation and the number of cores used for communication. Thus, in one embodiment, MLSL self-test allows balancing of computation and communication time to optimize overall system performance. For example, the number of cores used for communication can be dynamically varied and an optimal number of cores can be selected based on an impact on total computation time. Alternatively, automatic determination can be bypassed and an application developer can explicitly specify a number of cores to use for communication and an associated number of network endpoints.

[0266] As Figure 20B As represented by block 2012, operations for MLSL setup include a first operation to initialize the MLSL library to enable use of the MLSL API. A session object can then be created and a global batch size can be set using the MLSL logic, as represented by block 2014. The global batch size can be determined based on a sum of local batch sizes. A distribution object can then be created using the MLSL logic, the distribution object indicating a number of partitions for data parallelism and a number of partitions for model parallelism, as represented by block 2016.

[0267] As represented by block 2018, the MLSL logic can then be used to create operation objects for each layer of the neural network. In one embodiment, creating operation objects for each layer includes creating auxiliary OperationRegInfo objects that hold information about learnable parameters and activation shapes. These parameters define the specific relationship between input and output activations and the parameters of the operation. The MLSL API enables developers to add input / output activation shapes and parameter shapes to the OperationRegInfo objects. Using the MLSL API, developers can create operation objects, delete OperationRegInfo objects, and set dependencies between operations. Using the MLSL API, developers can then allocate buffers for parameters, input activations, and gradients with respect to input activations using information about batch size and shape. Since linked operations share a common activation, linked operations can be allocated on one side of a transaction and reused on other sized transactions. (They should only be allocated on one side and reused on the other). In one embodiment, the MLSL library provides specialized allocators that implement specialized optimizations. In one embodiment, the session object created at block 2014 includes a commit method that can be used to complete the creation of the operation graph.

[0268] The MLSL logic can then be used to execute the machine learning framework workflow, with MLSL API calls wrapping the computational portions of the workflow, as represented by block 2020. In one embodiment, the MLSL API calls implement automatic swapping of activations, gradients with respect to activations, and gradients with respect to parameters. The MLSL logic can then be used to update the parameters based on the machine learning framework workflow executed at block 2022.

[0269] Figures 21A-21B A method of performing distributed training of a neural network according to the present embodiment is shown. Distributed training of a neural network can be performed using graphics processing logic as described herein, with the graphics processing logic being included within a plurality of interconnected worker nodes. In various embodiments, the plurality of interconnected worker nodes can be within a single computing device or chassis, or can be distributed across multiple computing devices or chassis. For example, the plurality of interconnected worker nodes can be configured in a similar manner to the multi-node computing system 1700 of FIG. 17.

[0270] In one embodiment, as Figure 21AAs shown, the method includes storing a library in a memory, where the library is configured to facilitate transmission of data during distributed training of a neural network, as shown at block 2102. The data can be associated with trainable parameters of the neural network. During training, a plurality of nodes can send and receive gradient data associated with the trainable parameters via a network interface, as shown at block 2104. The nodes can execute instructions provided by the library via a general purpose processor, as shown at block 2106. The instructions can cause the general purpose processor to configure the network interface to send and receive gradient data associated with the trainable parameters during a workflow of a machine learning framework. Additionally, the worker nodes can execute computational operations associated with the machine learning framework workflow via a graphics processor to generate gradient data associated with the trainable parameters, as shown at block 2108. The computational operations can be interleaved with the sending and receiving of gradient data via the network interface. The interleaved computational operations can overlap, such that a first portion of data can be sent while a second portion of data is being executed on a computational operation.

[0271] Figure 21B A method of distributed training of a neural network that can be performed using the graphics processing logic described herein is shown. In embodiments, a plurality of interconnected worker nodes can be within a single computing device or chassis, or can be distributed across a plurality of computing devices or chassis. The method can be implemented by one or more processing devices within a general purpose processor, a graphics processor, or a processor within a fabric interface. The first or second instructions described below can be instructions provided by a library within a machine learning extension architecture 1511, 1811 in Figure 15B 18 such as the NCCL library, described herein.

[0272] In one embodiment, as shown at block 2112, the method includes storing a first set of instructions in a first memory, the first set of instructions including a machine learning framework. The first memory can be a system memory that is accessible by all or most components within the data processing system. For example, the first memory can be a system memory 104 as in Figure 1 As shown at block 2114, the method additionally includes implementing sending and receiving of data associated with the set of trainable machine learning parameters via a fabric interface. The sending and receiving of data can be performed or facilitated by a message passing library that is in communication with a high performance communication fabric, such as a messaging library 1519 and a high performance communication fabric 1521 of Figure 15B

[0273] ​​In one embodiment, as shown at block 2116, the method additionally includes executing a first set of instructions via a set of general purpose processor cores. Executing the first set of instructions includes providing a training workflow for computing gradients of trainable machine learning parameters and communicating with a second set of instructions. The second set of instructions can facilitate sending and receiving gradients via a fabric interface. As shown at block 2118, the method additionally includes executing a compute operation associated with the training workflow via a graphics processor. Executing the compute operation can include generating a gradient of a trainable machine learning parameter. In one embodiment, the gradient can be stored in a memory shared between the graphics processor and the fabric interface, allowing the gradient data to be sent without needing to store the data in system memory prior to sending.

[0274] Figure 22 is a block diagram of a data processing system 2200 according to embodiments described herein. The data processing system 2200 is a heterogeneous processing system having a processor 2202, a unified memory 2210, and a GPGPU 2220 including machine learning acceleration logic. The processor 2202 and the GPGPU 2220 can be any processor and GPGPU / parallel processor as described herein. The processor 2202 can execute instructions for a compiler 2215 stored in system memory 2212. The compiler 2215 executes on the processor 2202 to compile source code 2214A into compiled code 2214B. The compiled code 2214B can include code that can be executed by the processor 2202 and / or code that can be executed by the GPGPU 2220. During compilation, the compiler 2215 can perform operations to insert metadata including hints about the level of data parallelism present in the compiled code 2214B and / or hints about the data locality associated with threads to be dispatched based on the compiled code 2214B. The compiler 2215 can include information necessary to perform such operations or can perform the operations with the aid of a runtime library 2216, such as a machine learning extension library (MLSL) as described herein. The runtime library 2216 can also facilitate the compiler 2215 compiling the source code 2214A and include instructions to link with the compiled code 2214B at runtime to facilitate executing the compiled instructions on the GPGPU 2220.

[0275] Unified memory 2210 represents a unified address space that can be accessed by processor 2202 and GPGPU 2220. Unified memory includes system memory 2212 as well as GPGPU memory 2218. GPGPU memory 2218 includes GPGPU local memory 2228 within GPGPU 2220 and can also include some or all of system memory 2212. For example, compiled code 2214B stored in system memory 2212 can also be mapped into GPGPU memory 2218 for access by GPGPU 2220.

[0276] GPGPU 2220 includes a number of compute blocks 2224A-2224N, each including one or more instances of one or more processing clusters 214A-214N or processing arrays 212 as in FIG. 2. GPGPU 2220 also includes a set of registers 2224, cache memory 2226, and power and performance module 2225 that can be used as shared resources for compute blocks 2224A-2224N. Power and performance module 2225 can be configured to adjust power delivery and clock frequency of compute blocks 2224A-2224N to power idle components within compute blocks 2224A-2224N under heavy workloads. GPGPU 2220 includes GPGPU local memory 2228, which is physical memory shared with a graphics card or multi-chip module of GPGPU 2220.

[0277] In one embodiment, GPGPU 2220 includes graphics and compute acceleration logic including instruction fetch and decode unit 2221, scheduler unit 2222, and machine learning fixed function unit 2223. Fetch and decode unit 2221 is a fetch and decode unit including logic to fetch and decode instructions to be computed by GPGPU 220. In one embodiment, the instructions executed can order and / or sequence a set of operations and / or micro-operations performed via compute blocks 2224A-2224N and / or machine learning fixed function unit 2223 via scheduler unit 2222.

[0278] In one embodiment, machine learning fixed function unit 2223 is an application specific integrated circuit explicitly and specially configured to perform a large number of parallel matrix multiplication operations. In one embodiment, machine learning fixed function unit 2223 is configured to perform matrix multiplication on convolution filters having non-power-of-two filter sizes. In one embodiment, machine learning fixed function unit 2223 is a field programmable gate array (FPGA) that provides fixed function logic that can be updated between workloads.

[0279] In some embodiments, GPGPU 2220 includes an integrated fabric interface 2230 and a fabric interface cache 2232. In one embodiment, integrated fabric interface 2230 additionally includes an MLSL fabric module 2231 that enables the fabric interface to provide hardware acceleration for certain MLSL operations. Fabric interface 2230 can enable communication with a high performance communication fabric, such as high performance communication fabric 1521 of FIG. 15. Figure 15B Fabric interface 2230 has an address space that is mapped to at least a portion of GPGPU local memory 2228, and in one embodiment can participate in unified memory 2210 that is shared by processor 2202 and GPGPU 2220. Fabric interface cache 2232 is used to cache data received from or to be sent to a communication fabric that enables data communication between computing nodes. In one embodiment, when a computation result is computed by GPGPU 2220 and stored within GPGPU local memory 2228, fabric interface 2230 can send the data from GPGPU local memory 2228 to other computing nodes. In such an embodiment, there is no need to send the data to system memory 2212 unless the data is needed for use by an application executing on processor 2202.

[0280] MLSL fabric module 2231 is configured to facilitate low latency transfer of data between nodes. In embodiments, MLSL fabric module 2231 can be a firmware module configured to execute firmware provided by an MLSL or equivalent machine learning extension library. In one embodiment, MLSL fabric module 2231 is a processor or microcontroller configured to execute instructions to implement point-to-point communication primitives as described herein. In one embodiment, MLSL fabric module 2231 can receive a set of addresses within GPGPU local memory 2228 associated with a data object managed by an MLSL runtime (e.g., runtime library 2216). For example, an address range for an output buffer storing activation data to be generated by GPGPU 2220 can be provided to MLSL fabric module 2231. MLSL fabric module 2231 can then be configured to monitor updates in the address range. When the address range receives a write of activation data output by GPGPU 2220, MLSL fabric module 2231 can directly dispatch a transfer to fabric interface 2230 to transfer the output activation data. MLSL fabric module 2231 can also be used to implement the fine-grained communication and point-to-point hardware communication techniques described herein.

[0281] The protocol supported by fabric interface 2230 can vary. In one embodiment, fabric interface 2230 is a high-speed Ethernet interface. In one embodiment, fabric interface 2230 is an Omni-Path interconnect interface. In one embodiment, fabric interface 2230 is an InfiniBand interface. In one embodiment, fabric interface 2230 is some version of an NVLink interface, or another NVHS-based interconnect. Other fabric interface technologies can also be supported.

[0282] It should be appreciated that for some implementations, systems less or more equipped than the examples described above can be preferred. Thus, depending on numerous factors, such as price constraints, performance requirements, technological improvements, or other circumstances, the configuration of data processing system 2200 can be varied. Embodiments described herein can find wide applicability in high-performance computing and machine learning training environments. Accordingly, the present description contemplates that data processing system 2200, as well as other data processing and computing systems described herein, are implemented as high-performance servers or server arrays within distributed computing systems. Such distributed computing systems can be implemented within data centers or server farms. However, embodiments are not limited to such implementations, and the techniques described herein can also be used in large-scale distributed computing systems of lower-performance devices, such as but not limited to mobile or handheld devices, tablet computing devices, or connected consumer electronics devices.

[0283] Additional example graphics processing systems

[0284] Details of the embodiments described above can be included within the graphics processing systems and devices described below. FIGS. 23-36 The graphics processing systems and devices of FIGS. 1-3 illustrate alternative systems and graphics processing hardware that can implement any and all of the techniques described above.

[0285] Additional example graphics processing system overview

[0286] FIG. 23 is a block diagram of a processing system 2300 according to an embodiment. In embodiments, system 2300 includes one or more processors 2302 and one or more graphics processors 2308, and can be a single processor desktop system, a multiprocessor workstation system, or a server system having many processors 2302 or processor cores 2307. In one embodiment, system 2300 is a processing platform incorporated within a chip package

[0287] Embodiments of system 2300 can include or incorporate a server-based gaming platform, a gaming console, including a game and media console, a mobile gaming console, a handheld gaming console, or an online game console. In some embodiments, system 2300 is a mobile telephone, a smartphone, a tablet computing device, or a mobile Internet device. Data processing system 2300 can also include, be coupled to, or integrated within a wearable device such as a smart watch wearable device, a smart glasses device, an augmented reality device, or a virtual reality device. In some embodiments, data processing system 2300 is a television or set-top box device having one or more processors 2302 and a graphical interface generated by one or more graphics processors 2308.

[0288] In some embodiments, one or more processors 2302 each include one or more processor cores 2307 for processing instructions that, when executed, carry out operations for system and user software. In some embodiments, each of the one or more processor cores 2307 is configured to process a specific instruction set 2309. In some embodiments, instruction set 2309 can facilitate complex instruction set computing (CISC), reduced instruction set computing (RISC), or computing via a very long instruction word (VLIW). Multiple processor cores 2307 can each process a different instruction set 2309, which can include instructions to facilitate the emulation of other instruction sets. Processor core 2307 can also include other processing devices, such as a digital signal processor (DSP).

[0289] In some embodiments, processor 2302 includes cache memory 2304. Depending upon the architecture, the processor 2302 can have a single internal cache or multiple levels of internal cache. In some embodiments, cache memory is shared among multiple components of the processor 2302. In some embodiments, processor 2302 also uses an external cache (e.g., a 3- level (L3) cache, or last level cache (LLC)) (not shown), which can be shared among the processor cores 2307 using known cache coherence techniques. Additionally, a register file 2306 is included in the processor 2302, which can include different types of registers for storing different kinds of data (e.g., integer registers, floating point registers, status registers, and instruction pointer registers). Some of the registers can be general registers, while others can be specific to the design of the processor 2302.

[0290] In some embodiments, processor 2302 is coupled to a processor bus 2310 that transmits communication signals between processor 2302 and other components in system 2300, such as storage memory 2320, input / output (I / O) devices 2332, and a display controller 2314 coupled to a display device 2318. In some embodiments, system 2300 uses an exemplary 'hub' system architecture, including a memory controller hub 2316 and an I / O controller hub (ICH) 2330. The memory controller hub 2316 facilitates communication between the memory 2320 and other components in system 2300, while the ICH 2330 provides connections between the ICH 2330 and other components in system 2300. In one embodiment, the memory controller hub 2316 is integrated within the processor 2302.

[0291] The memory 2320 can be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase- change memory device, or some other memory device having suitable performance to serve as process memory. In one embodiment, memory 2320 can operate as system memory for system 2300, to store data 2322 and instructions 2321 for use when executing application programs or processes by one or more processors 2302. The memory controller hub 2316 also couples with an optional external graphics processor 2312, which can communicate with the one or more graphics processors 2308 within the processor 2302 to perform graphics and media operations.

[0292] In some embodiments, the ICH 2330 enables peripherals to connect to the memory 2320 and processor 2302 via a high-speed I / O bus. The I / O peripherals include, but are not limited to, an audio controller 2346, a firmware interface 2328, a wireless transceiver 2326 (e.g., Wi-Fi, Bluetooth), a data storage device 2324 (e.g., hard disk drive, flash memory, etc.), and a

[0293] FIG. 24 is a block diagram of an embodiment of a processor 2400 having one or more processor cores 2402A-2402N, an integrated memory controller 2414, and an integrated graphics processor 2408. FIG. 24 Those elements of having the same reference number (or name) in the foregoing figure as a corresponding element in any other figure herein can operate or function in any manner similar to that described herein in other instances, but are not limited to such. The processor 2400 can include additional cores up to and including the additional core 2402N denoted by the dashed line. Each processor core 2402A-2402N includes one or more internal cache units 2404A-2404N. In some embodiments, each processor core can also access one or more shared cache units 2406.

[0294] The internal cache units 2404A-2404N and shared cache units 2406 represent a cache memory hierarchy within the processor 2400. The cache memory hierarchy can include at least one level of instruction and data caches within each processor core and one or more levels of shared mid-level caches, such as level two (L2), level three (L3), level four (L4), or other levels of caches, with the highest level cache being classified as an LLC before external memory. In some embodiments, cache coherency logic maintains coherency among the cache units 2406 and 2404A-2404N.

[0295] In some embodiments, the processor 2400 can also include a set of one or more bus controller units 2416 and a system agent core 2410. The one or more bus controller units 2416 manage a set of peripheral buses, such as one or more

[0296] In some embodiments, one or more of the processor cores 2402A-2402N include support for simultaneous multi-threading. In such embodiments, the system agent core 2410 includes components used to coordinate and operate the cores 2402A-2402N during multi-threaded processing. Additionally, the system agent core 2410 can also include a power control unit (PCU), including logic and components to regulate the power state of the processor cores 2402A-2402N and the graphics processor 2408.

[0297] In some embodiments, the processor 2400 also includes a graphics processor 2408 for performing graphics processing operations. In some embodiments, the graphics processor 2408 is coupled to the shared cache unit 2406 set and the system agent 2410, which includes one or more integrated memory controllers 2414. In some embodiments, a display controller 2411 is coupled with the graphics processor 2408 to drive graphics processor output to one or more coupled displays. In some embodiments, the display controller 2411 can be separate from the graphics processor, or can be integrated within the graphics processor 2408 or system agent 2410.

[0298] In some embodiments, a ring-based interconnect unit 2412 is used to couple the internal components of the processor 2400. However, alternative interconnects such as point-to-point interconnects, switched interconnects, or other technologies can be used, including technologies other than those well known in the art. In some embodiments, the graphics processor 2408 is coupled with the ring interconnect 2412 via an I / O link 2413.

[0299] The exemplary I / O link 2413 represents at least one of a variety of different I / O interconnects, including a package I / O interconnect that facilitates communication between the various processor components and a high performance embedded memory module 2418, such as an eDRAM module. In some embodiments, each of the processor cores 2402A-2402N and the graphics processor 2408 use the embedded memory module 2418 as a shared last level cache.

[0300] In some embodiments, the processor cores 2402A-2402N are homogeneous cores executing the same instruction set architecture. In another embodiment, the processor cores 2402A-2402N are heterogeneous in terms of instruction set architecture (ISA), where one or more of the processor cores 2402A-2402N execute a first instruction set and at least one of the other cores executes a subset of the first instruction set or a different instruction set. In one embodiment, the processor cores 2402A-2402N are homogeneous in terms of microarchitecture, where one or more of the cores have a relatively high power consumption and one or more power cores have a lower power consumption. Additionally, the processor 2400 can be implemented on one or more chips or as a SoC integrated circuit having the illustrated components, among other components.

[0301] FIG. 25is a block diagram of a graphics processor 2500 that can be a discrete graphics processing unit, or can be graphics processor integrated with a multiple core processor. In some embodiments, the graphics processor communicates with the processor memory via a hub interface to map I / O to appropriate registers in the graphics processor. In some embodiments, the graphics processor 2500 includes a memory interface 2514 to access a memory. The memory interface 2514 can be an interface to local memory, one or more internal caches, one or more shared external caches, and / or to system memory.

[0302] In some embodiments, the graphics processor 2500 also includes a display controller 2502 to drive display output data to a display device 2520. The display controller 2502 includes hardware for one or more overlay planes for compositing and for planar reflections and multiple layers of video or user interface elements. In some embodiments, the graphics processor 2500 includes a video codec engine 2506 to encode, decode, or transcode media

[0303] In some embodiments, the graphics processor 2500 includes a block image transfer (BLIT) engine 2504 to perform two-dimensional (2D) rasterizer operations including, for example, bit-boundary block transfers. However, in one embodiment, 2D graphics operations are performed using one or more components of the graphics-processing engine (GPE) 2510. In some embodiments, GPE 2510 is a compute engine for performing graphics operations, including three-dimensional (3D) graphics operations and media operations.

[0304] In some embodiments, GPE 2510 includes a 3D pipeline 2512 for executing 3D graphics operations, such as rendering three-dimensional graphics shapes (e.g., rectangles, triangles, etc.) using processing functions that act upon vertices of the shapes. The 3D pipeline 2512 includes programmable and fixed function elements that perform various tasks for executing the 3D graphics operations and / or generate execution threads to be dispatched for execution on the processing array. While the 3D pipeline 2512 can be used to perform media operations, embodiments of GPE 2510 also include a media pipeline 2516 that is specifically used to execute media operations, such as video post-processing and image enhancements.

[0305] In some embodiments, the media pipeline 2516 includes fixed-function or programmable logic units to perform one or more specialized media operations, such as video decoding acceleration, video deinterleaving, and video encoding acceleration, in place of or on behalf of the video codec engine 2506. In some embodiments, the media pipeline 2516 further includes a thread generation unit to generate threads for execution on the 3D / media subsystem 2515. The generated threads perform calculations on media operations on one or more graphics execution units included in the 3D / media subsystem 2515.

[0306] In some embodiments, the 3D / media subsystem 2515 includes logic for executing threads generated by the 3D pipeline 2512 and the media pipeline 2516. In one embodiment, the pipelines send thread execution requests to the 3D / media subsystem 2515, the 3D / media subsystem including thread dispatch logic for arbitrating and dispatching requests to available thread execution resources. Execution resources include an array of graphics execution units for processing 3D and media threads. In some embodiments, the 3D / media subsystem 2515 includes one or more internal caches for thread instructions and data. In some embodiments, the subsystem also includes shared memory (including registers and addressable memory) for sharing data between threads and for storing output data.

[0307] Exemplary additional graphics processing engines

[0308] FIG. 26 This is a block diagram of a graphics processing engine 2610 of a graphics processor according to some embodiments. In one embodiment, the graphics processing engine (GPE) 2610 is... FIG. 25 The image shows a version of GPE 2510. FIG. 26 Those elements having the same reference numerals (or names) as elements in any other figure herein may operate or function in any manner similar to, but not limited to, those described elsewhere herein. For example, shown FIG. 25 The 3D pipeline 2512 and media pipeline 2516. The media pipeline 2516 is optional in some embodiments of the GPE 2610 and may not be explicitly included within the GPE 2610. For example, and in at least one embodiment, a separate media and / or image processor is coupled to the GPE 2610.

[0309] In some embodiments, GPE 2610 is coupled with or includes a command streamer 2603 that provides a command stream to 3D pipeline 2512 and / or media pipeline 2516. In some embodiments, command streamer 2603 is coupled with memory, which can be system memory, or one or more of internal cache memory and shared cache memory. In some embodiments, command streamer 2603 receives commands to perform from memory and sends these commands to 3D pipeline 2512 and / or media pipeline 2516. The commands are instructions for the 3D pipeline 2512 and media pipeline 2516 to process objects (e.g., vertices and / or fragments). In one embodiment, additionally, the ring buffer also includes a batch command buffer that stores batches of multiple commands. The commands for the 3D pipeline 2512 can also include references to data stored in memory, such as, but not limited to, vertex and geometry data for the 3D pipeline 2512 and / or image data and memory objects for the media pipeline 2516. 3D pipeline 2512 and media pipeline 2516 process the commands by performing operations via logic within the respective pipelines or by dispatching one or more threads to execution units array 2614.

[0310] In various embodiments, 3D pipeline 2512 can execute one or more shader programs, such as a vertex shader, a geometry shader, a pixel shader, a fragment shader, a compute shader, or other shader programs, by processing instructions and dispatching execution threads to the graphics core array 2614. The graphics core array 2614 provides unified execution resources for all shader programs, shader program types, and shader program

[0311] In some embodiments, graphics core array 2614 also includes execution logic to perform media functions, such as video and / or image processing. In one embodiment, the execution units are FIG. 1 parallel or in conjunction with the general-purpose logic within the FIG. 24 processor core(s) 107 of FIG. 1 or

[0312] Output data generated by threads executing on graphics core array 2614 can be stored in memory in a unified return buffer (URB) 2618. URB 2618 can store data from multiple threads. In some embodiments, URB 2618 can be used for transmitting data between different threads executing on graphics core array 2614. In some embodiments, URB 2618 can additionally be used for synchronization between threads on graphics core array and fixed function logic within shared function logic 2620.

[0313] In some embodiments, graphics core array 2614 is scalable, such that the array includes varying numbers of graphics cores each having varying numbers of execution units based on target performance and power profiles of the GPE 2610. In one embodiment, the execution resources are dynamically scalable such that execution resources can be enabled or disabled as needed.

[0314] Graphics core array 2614 is coupled with shared function logic 2620, which includes resources shared between the graphics cores in the graphics core array. Shared functions within shared function logic 2620 are hardware logic units that provide specialized supplemental functionality to graphics core array 2614. In various embodiments, shared function logic 2620 includes, without limitation, sampler 2621, math 2622, and inter-thread communication (ITC) 2623 logic. Additionally, some embodiments implement one or more caches 2625 within shared function logic 2620. Shared functions are implemented in cases where demand for a given specialized function is insufficient to include multiple instances of the specialized function within graphics core array 2614. Instead, a single instance of the specialized function is implemented as a standalone entity within shared function logic 2620 and shared between execution resources within graphics core array 2614. The precise set of functions shared between graphics core array 2614 and included within graphics core array 2614 varies between embodiments.

[0315] FIG. 27 is a block diagram of another embodiment of a graphics processor 2700. FIG. 27 Those elements of having the same reference number (or name) in the figures herein as elements in any other figure herein can operate or function in any manner similar to that described elsewhere herein, but are not limited to such.

[0316] In some embodiments, graphics processor 2700 includes a ring interconnect 2702, a front-end 2704, a media engine 2737, and graphics cores 2780A-2780N. In some embodiments, ring interconnect 2702 couples the graphics processor to other processing units including other graphics processors or one or more general-purpose processor cores. In some embodiments, the graphics processor is one of a plurality of processors integrated within a multi-core processing system.

[0317] In some embodiments, graphics processor 2700 receives batches of commands via ring interconnect 2702. The incoming commands are interpreted by a command streamer 2703 in pipeline front-end 2704. In some embodiments, graphics processor 2700 includes scalable execution logic to perform 3D geometry processing and media processing via the graphics core(s) 2780A-2780N. For 3D geometry processing commands, command streamer 2703 supplies commands to geometry pipeline 2736. For at least some media processing commands, command streamer 2703 supplies commands to video front end 2734, which couples with a media engine 2737. In some embodiments, media engine 2737 includes a video quality engine (VQE) 2730 for video and image post-processing, and a multi-format encode / decode (MFX) 2733 engine to

[0318] In some embodiments, graphics processor 2700 includes a number of graphics core(s) 2780A-2780N (sometimes referred to as core slices) each having a number of sub-cores 2750A-2750N, 2760A-2760N (sometimes referred to as core sub-slices). In some embodiments, graphics processor 2700 can have any number of graphics cores 2780A-2780N. In some embodiments, graphics processor 2700 includes a graphics core 2780A having at least a first sub-core 2750A and a second sub-core 2760A. In other embodiments, graphics processor is a low power processor with a single sub-core (e.g., 2750A). In some embodiments, graphics processor 2700 includes a number of graphics cores 2780A-2780N each including a set of first sub-cores 2750A-2750N and a set of second sub-cores 2760A-2760N. Each sub-core in the set of first sub-cores 2750A-2750N includes at least a first set of execution units 2752A-2752N and a media / texture sampler 2754A-2754N. Each sub-core in the set of second sub-cores 2760A-2760N includes at least a second set of execution units 2762A-2762N and a sampler 2764A-2764N. In some embodiments, each sub-core 2750A-2750N, 2760A-2760N shares a set of shared resources 2770A-2770N. In some embodiments, the shared resources include shared cache memory and pixel operation logic. Other shared resources can also be included in graphics processor embodiments.

[0319] Exemplary additional execution units

[0320] FIG. 28 Thread execution logic 2800 is shown including an array of processing elements employed in some embodiments of GPEs. FIG. 28 Those elements of 2736 having the same reference number (or name) as elements herein in any other figure can operate or function in any manner similar to that described elsewhere herein, but are not limited to such.

[0321] In some embodiments, thread execution logic 2800 includes a shader processor 2802, a thread dispatcher 2804, an instruction cache 2806, an array of scalable execution units 2808A-2808N including one or more execution units 2808A, 2808B, 2808C, 2808D, up to 2808N-1, and 2808N, a sampler 2810, a data cache 2812, and a thread port 2814. In one embodiment, the array of scalable execution units 2808A-2808N is dynamically scalable to include any number of execution units 2808. In one embodiment, the array of execution units 2808A-2808N includes a variable number of execution units based on the number of execution units needed to perform the software pipelined loop on the central processing unit 2700. Like the execution unit array 2708A-2708N of FIG. 27, in embodiments, the array of execution units 2808A-2808N can be scalar execution unit cores, SIMD scalar execution units or SIMD double precision execution units as described herein.

[0322] In some embodiments, the execution units 2808A-2808N include dedicated logic to perform one or more graphics processing functions such as, for example, a graphics processing cluster including one or more graphics processing cores to perform graphics processing tasks such as, for example, three dimensional (3D) graphics processing tasks or two dimensional (2D) graphics processing tasks. In one embodiment, the array of execution units 2808A-2808N include single precision data FIG. 27 FIG. 28 ​) for processing. In some embodiments, thread dispatcher 2804 can also handle run-time thread generation requests from executing shader programs.

[0323] In some embodiments, execution units 2808A-2808N support single program multiple

[0324] Each of execution units 2808A-2808N operate on arrays of data elements. The number of data elements is the "execution size," or the number of channels that the instruction operates on. An execution channel is a logical unit of execution for a data element. The number of channels can be independent of the number of physical ALUs or FPUs for a particular graphics processor. In some embodiments, execution units 2808A-2808N also support integer and floating-point data types.

[0325] The instruction set of the execution unit includes SIMD instructions. Various data elements can be stored as a packed data type within the registers and the execution unit will process the data elements according to their data type. For example, when processing SIMD single-precision floating-point instructions, the execution unit processes a 32-bit floating-point data element in each column of the register on a column-by-column basis. In another example, when processing SIMD single-precision floating-point instructions, the execution unit processes a 64-bit double-precision floating-point data element in each column of the register on a column-by-column basis. In yet another example, when processing SIMD single-precision floating-point instructions, the execution unit processes a quad-word of four 128-bit quad-precision floating-point data elements in each column of the register on a column-by-column basis. While the 128-bit data elements are processed by the execution unit in each column of the register, only one 128-bit data element is processed at a time. Thus, the execution unit processes four 128-bit data elements for each instruction execution. One or more integer instructions can also be processed using the execution unit via a second integer ALU. In one embodiment, the execution unit can also include logic to handle SIMD and PMU instructions in a single instruction cycle.

[0326] One or more internal instruction caches (e.g., 2806) are included in the thread execution logic 2800 to cache thread instructions for the execution units. In some embodiments, one or more data caches (e.g., 2812) are included to cache thread data during thread execution. In some embodiments, a sampler 2810 is included to provide texture sampling for 3D operations and media sampling for media operations. In some embodiments, the sampler 2810 includes specialized texture or media sampling functionality to handle texture or media data during sampling before the data is provided to the execution units.

[0327] During execution, the graphics and media pipeline sends thread initiation requests to the thread execution logic 2800 via the thread generation and dispatch logic. Once a set of geometric objects has been processed and rasterized into pixel data, pixel processor logic (e.g., pixel shader logic, fragment shader logic, etc.) within the shader processor 2802 is invoked to further calculate output values and cause the results to be written to an output surface (e.g., color buffer, depth buffer, stencil buffer, etc.). In some embodiments, the pixel shader or fragment shader calculates values of per-vertex attributes that are interpolated across the rasterized object. In some embodiments, the pixel processor logic within the shader processor 2802 then executes an application programming interface (API)-supplied pixel or fragment shader program. To execute the shader program, the shader processor 2802 dispatches threads to the execution units (e.g., 2808A) via the thread dispatcher 2804. In some embodiments, the pixel shader 2802 uses texture sampling logic within the sampler 2810 to access texture data stored in a texture map stored in memory. Arithmetic operations on the texture data and the input geometry data calculate pixel color values of each geometric fragment, or discard one or more pixels without further processing.

[0328] In some embodiments, data ports 2814 provide a memory access mechanism for thread execution logic 2800 to output processed data to memory for processing on a graphics processor output pipeline. In some embodiments, data ports 2814 include or are coupled to one or more cache memories (e.g., data cache 2812) to cache data for memory access via data ports.

[0329] FIG. 29 FIG. 29 is a block diagram illustrating a graphics processor instruction format 2900 according to some embodiments. In one or more embodiments, a graphics processor execution unit supports an instruction set that includes a variety of different instructions for performing various graphics processing operations. The solid lined boxes illustrate the components that are generally common to all instructions, while the optional components in the dashed lined boxes are only available on some

[0330] In some embodiments, a graphics processor execution unit natively supports the 128-bit instruction format 2910. An executable 64-bit compact instruction format 2930 can be used for some instructions based on a number of instruction options and operands used by the instruction. A native 128-bit instruction format 710 provides access to all instruction options available while some options and operations are restricted in the 64-bit format 2930. Whether a 64-bit or 128-bit instruction format is used by an execution unit is based on the graphics processor's design and in some embodiments can vary for different graphics processors. In some embodiments, a set of index values in the instruction header portion of the instruction template 2912 is used to select between compact and full instruction formats. The instruction can also include an operation code to indicate one of a variety of different graphics processing operations. The graphics processor execution unit includes multiple data operation units for performing the graphics operations identified by the operation codes. The execution unit's operation code field is used to determine which of the graphics processor's operation units are used on the instruction. One or more graphic processor parameter registers 2918 are belted to the graphics processor's data bus. In some embodiments, an address generator 2920 is also belted to the data bus to provide register addresses generated by the instruction decoder to the register file unit. In some embodiments, the graphics processor's registers also store context specific state and parameter information. Some graphics processors use graphics processor parameter registers that are used to provide caching of parameters that are closely related to specific and applications in order to provide for efficient processing. In some embodiments, X bit compact instructions and 128-bit instructions share the graphics processor parameter registers.

[0331] For each format, the instruction operation code 2912 defines the operation that will be performed by the graphics processor execution units. The graphics processor execution units execute each instruction in parallel across the multiple data elements in each of its operands. The results of the graphics processor execution units are undefined at the end of each clock cycle unless a specific instruction defines the operation in which the results should be forwarded. In some embodiments, the graphics processor includes a broadcast stage (not pictured) after each ALU stage to move the

[0332] Some execution unit instructions have up to three operands, including two source operands (src0 2920, src1 2922) and one destination 2918. In some embodiments, an execution unit supports dual destination instructions, where one of the destinations is implied. Data operation instructions can have a third source operand (e.g., SRC2 2924), where the instruction opcode 2912 determines the number of source operands. The last source operand can be an immediate (e.g., hard coded) value passed with the instruction.

[0333] In some embodiments, the 128-bit instruction format 2910 includes a access / address mode field 2926 that, for example, defines whether a direct register addressing mode or an indirect register addressing mode is used. When using direct register addressing mode, the register address for one or more operands is provided directly by bits in the instruction.

[0334] In some embodiments, the 128-bit instruction format 2910 includes an access / address mode field 2926 that specifies the address mode and / or access mode of the instruction. In one embodiment, the access mode is used to define the data access alignment for the instruction. Some embodiments support access modes including a 16-byte aligned access mode and a 1-byte aligned access mode, where the byte alignment of the access mode determines the access alignment of the instruction operands. For example, when in a first mode, the instruction can use byte-aligned addressing for source and destination operands, and when in a second mode, the instruction can use 16-byte aligned addressing for all source and destination operands.

[0335] In one embodiment, the address mode portion of the access / address mode field 2926 determines whether the instruction uses direct addressing or indirect addressing. When using direct register addressing mode, the register address for one or more operands is provided directly by bits in the instruction. When using indirect register addressing mode, the register address for one or more operands can be calculated based on an address register value and an address immediate field in the instruction.

[0336] In some embodiments, instructions are grouped based on the opcode 2912 bit field to simplify opcode decoding 2940. For 8-bit opcodes, bits 4, 5, and 6 allow the execution unit to determine the type of opcode. The precise opcode grouping shown is exemplary only. In some embodiments, move and logic opcode group 2942 includes data movement and logic instructions (e.g., move (mov), compare (cmp)). In some embodiments, the move and logic group 2942 shares five most-significant bits (MSB), with move (mov) instructions taking the form 0000xxxxb, and logic instructions taking the form 0001xxxxb. Flow control instruction group 2944 (e.g., call, jmp) includes instructions that take the form 0010xxxxb (e.g., Ox20). Hybrid instruction group 2946 includes a mix of instructions, including synchronization instructions (e.g., wait, send) that take the form 0011xxxxb (e.g., Ox30). Parallel mathematical instruction group 2948 includes per-component arithmetic instructions (e.g., add, mul) that take the form 0100xxxxb (e.g., Ox40). The parallel mathematical group 2948 performs arithmetic operations across data lanes in parallel. Vector mathematical group 2950 includes arithmetic instructions (e.g., dp4) that take the form 0101xxxxb (e.g., Ox50). The vector mathematical group performs arithmetic operations on vector operands, such as a dot product operation.

[0337] Exemplary additional graphics pipeline

[0338] FIG. 30 is a block diagram of another embodiment of a graphics processor 3000. FIG. 30 Those elements of having the same reference number (or name) in the figures herein as elements in any other figure herein can operate or function in any manner similar to that described elsewhere herein, but are not limited to that.

[0339] In some embodiments, graphics processor 3000 includes a graphics processing pipeline 3020, a media pipeline 3030, a display engine 3040, thread execution logic 3050, and a render output pipeline 3070. In some embodiments, graphics processor 3000 is a graphics processor included in a multi-core processing system that includes one or more general purpose processing cores. The graphics processor is controlled by register writes to one or more control registers (not shown) or via commands issued to the graphics processor 3000 via the ring interconnect 3002. In some embodiments, ring interconnect 3002 couples graphics processor 3000 to other processing components such as other graphics processors or general purpose processors. Commands from the ring interconnect 3002 are interpreted by a command streamer 3003 which supplies instructions to individual components of the graphics pipeline 3020 or media pipeline 3030.

[0340] In some embodiments, command streamer 3003 directs the operation of a vertex fetcher 3005 that reads vertex data from memory and executes vertex processing commands provided by command streamer 3003. In some embodiments, vertex fetcher 3005 provides the vertex data to a vertex shader 3007 that performs coordinate space transformation and lighting operations on each vertex. In some embodiments, vertex fetcher 3005 and vertex shader 3007 execute vertex processing instructions via dispatch of execution threads to execution units 3052A-3052B by thread dispatcher 3031.

[0341] In some embodiments, execution units 3052A-3052B are vector processors that are configured to execute a vector processing instruction set. In some embodiments, execution units 3052A-3052B include fixed function and programmable integer and floating point logic to perform graphics and media operations such as

[0342] In some embodiments, graphics pipeline 3020 includes a tessellation component for hardware-accelerated tessellation of 3D objects. In some embodiments, programmable hull shader 811 configures the tessellation operation. Programmable domain shader 817 provides postback evaluation of the tessellation output. Tessellator 3013 operates in the direction of hull shader 3011 and includes specialized logic for generating a detailed set of geometric objects based on a coarse geometric model provided as input to graphics pipeline 3020. In some embodiments, the tessellation components (e.g., hull shader 3011, tessellator 3013, domain shader 3017) can be bypassed if tessellation is not used.

[0343] In some embodiments, complete geometric objects can be processed by geometry shader 3019 via one or more threads dispatched to execution units 3052A-B, or can proceed directly to clipper 3029. In some embodiments, the geometry shader operates on entire geometric objects (rather than vertices or vertex patches as in previous stages of the graphics pipeline). If tessellation is disabled, geometry shader 3019 receives input from vertex shader 3007. In some embodiments, geometry shader 3019 can be programmed by a geometry shader program to perform geometric tessellation when the tessellation unit is disabled.

[0344] Clipper 3029 processes vertex data prior to rasterization. Clipper 3029 can be a fixed function clipper or a programmable clipper with clip and geometry shader functionality. In some embodiments, rasterizer and depth test component 3073 in render output pipeline 3070 dispatches pixel shaders to convert geometric objects into their per-pixel representation. In some embodiments, pixel shader logic is included in thread execution logic 3050. In some embodiments, an application can bypass rasterizer and depth test component 3073 and access un-rasterized vertex data via egress unit 3023.

[0345] Graphics processor 3000 has an interconnect bus, interconnect fabric, or some other interconnect mechanism to allow data and messages to be passed between components of the graphics processor. In some embodiments, execution units 3052A-B and associated cache(s) 3051, texture and media sampler 3054, and texture / sampler caches 3058 are interconnected via a data port 3056 to perform memory accesses and to communicate with each other and with other components of the processor. In some embodiments, sampler 3054, caches 3051, 3058, and execution units 3052A-B each have separate memory access ports to the data port 3056.

[0346] In some embodiments, the render output pipeline 3070 includes a rasterizer and depth test component 3073 that converts vertex-based objects into associated pixel-based representations. In some embodiments, the rasterizer logic includes a windower / masker unit for performing fixed function triangle and line rasterization. An associated render cache 3078 and depth cache 3079 are also available in some embodiments. Pixel operation component 3077 performs pixel-based operations on the data, although in some instances pixel operations associated with 2D operations (e.g., bit block image transfers with blending) are performed by 2D engine 3041 or replaced at display time by display controller 3043 using overlay display planes. In some embodiments, a shared L3 cache 3075 is available for all graphics components, allowing sharing of data without use of main system memory.

[0347] In some embodiments, graphics processor media pipeline 3030 includes a media engine 3037 and a video front-end 3034. In some embodiments, video front-end 3034 receives pipeline commands from the command streamer 3003. In some embodiments, media pipeline 3030 includes a separate command streamer. In some embodiments, video front-end 3034 processes media commands before sending the commands to the media engine 3037. In some embodiments, media engine 3037 includes thread spawning functionality to process media

[0348] In some embodiments, graphics processor 3000 includes a display engine 3040. In some embodiments, display engine 3040 is external to graphics processor 3000 and couples with the graphics processor via the ring interconnect 3002, or some other interconnect bus or fabric. In some embodiments, display engine 3040 includes a 2D engine 3041 and a display controller 3043. In some embodiments, display engine 3040 contains special purpose logic that is, optionally, used to accelerate only certain high-priority tasks or requests for use with a operating system or high

[0349] In some embodiments, graphics pipeline 3020 and media pipeline 3030 can be configured to perform operations based on a number of graphics and media programming interfaces and not be specific to any one application programming interface (API). In some embodiments, driver software for a graphics processor will translate API calls or other calls made by an application into commands that can be processed by the graphics processor. In some embodiments, support is provided for APIs including, but not limited to, those provided by the Khronos Group such as the Open Graphics Library (OpenGL), the Open Computing Language (OpenCL), and / or the Vulkan graphics and compute API. In some embodiments, support can also be provided for the Direct3D library from the Microsoft Corporation. In some embodiments, a combination of these APIs can be supported. Support can also be provided for the Open Source Computer Vision Library (OpenCV). Future APIs with compatible 3D pipelines will also be supported if a mapping of the pipeline of the future API to the pipeline of the graphics processor can be made.

[0350] Graphics pipeline programming

[0351] FIG. 31A FIG. 3 is a block diagram illustrating a graphics processor command format 3100 according to some embodiments. FIG. 31B FIG. 4 is a block diagram illustrating a graphics processor command sequence 3110 according to an embodiment. FIG. 31A The solid lined boxes in FIG. 4 illustrate the components of a graphics processor that are generally included in a graphics processor, while the dashed lined boxes represent optional components that are not included in every graphics processor. The ultimate configuration of graphics processor will depend on the graphics processor design and the application. FIG. 31A The exemplary graphics processor command format 3100 includes data fields to identify a target client 3102 for the command, a command operation code (opcode) 3104, and data 3106 for the command. Some commands will include sub-opcodes 3105 and a command size 3108.

[0352] In some embodiments, the client 3102 defines a client unit of the graphics device that processes the command data. In some embodiments, the graphics processor command parser examines a client field of each command to determine the further processing to be performed on the command and routes the command data to the appropriate client unit. In some embodiments, a graphics processor client unit includes a memory interface unit, render units, a 2D unit, a 3D unit, and a media unit. Each client unit has a respective processing pipeline to process the commands. Once a command is received by a client unit, the client unit reads the operation code 3104 and sub-op code 3105 (if present) to determine the operation to be performed on the command. The client unit uses information in the data field 3106 to perform the command. For some commands, an explicit command size 3108 is expected to define the size of the command. In some embodiments, the command parser automatically determines the size of at least some of the commands based on the command operation code. In some embodiments, commands are aligned via multiples of double-words.

[0353] FIG. 31B The flow diagram in FIG. 31 illustrates an exemplary graphics processor command sequence 3110. In some embodiments, a software or firmware of a data processing system featuring embodiments of a graphics processor uses a version of the command sequence shown to set up, execute, and terminate a set of graphics operations. A sample command sequence is shown and described for illustrative purposes only. The embodiments are not limited to these particular commands or to this command sequence. Moreover, the described commands can be issued as batch of commands in a command sequence, such that the graphics processor will process the sequence of commands in at least partially concurrence.

[0354] In some embodiments, the graphics processor command sequence 3110 can begin with a pipeline flush clear command 3112 to cause any active graphics pipelines to finish any currently pending commands for the pipeline. In some embodiments, the 3D pipeline 3122 and media pipeline 3124 are not operating at the same time. The pipeline flush clear is performed to cause the active graphics pipeline to finish any pending commands. In response to the pipeline flush clear, the command parser for the graphics processor will stop command processing until the active draw engine finishes the pending operations and invalidates the relevant read caches. Optionally, any data in the render caches that is marked as 'dirty' can be flushed to memory. In some embodiments, the pipeline flush clear command 3112 can be used for pipeline synchronization or before placing the graphics processor in a low power state.

[0355] In some embodiments, a pipeline selection command 3113 is used when the command sequence requires the graphics processor to explicitly switch between pipelines. In some embodiments, only one pipeline selection command 3113 is needed in the execution context before issuing the pipeline commands, unless the context is issuing commands to two pipelines. In some embodiments, a pipeline flush clear command 3112 is needed just before the pipeline switch via the pipeline selection command 3113.

[0356] In some embodiments, pipeline control commands 3114 configure the graphics pipeline for operation and for programming the 3D pipeline 3122 and media pipeline 3124. In some embodiments, the pipeline control commands 3114 configure the pipeline state of the active pipeline. In one embodiment, the pipeline control commands 3114 are used for pipeline synchronization and to clear data from one or more cache memories within the active pipeline before processing a batch of commands.

[0357] In some embodiments, commands 3116 for return buffer state are used to configure a set of return buffers for a respective pipeline to write data. Some pipeline operations require allocation, selection, or configuration of one or more return buffers in which intermediate data is written during processing. In some embodiments, the graphics processor also uses one or more return buffers to store output data and to perform cross-thread communication. In some embodiments, the return buffer state 3116 includes selecting the size and number of return buffers for a set of pipeline operations.

[0358] The remaining commands in the command sequence differ based on the active pipeline for operation. Based on the pipeline determination 3120, the command sequence is tailored for either the 3D pipeline 3122 starting with 3D pipeline state 3130, or the media pipeline 3124 starting at media pipeline state 3140.

[0359] The commands for 3D pipeline state 3130 include 3D state setting commands for vertex buffer state, vertex element state, constant color state, depth buffer state, and other state variables to be configured prior to processing 3D primitive commands. The values for these commands are determined at least in part on the particular 3D API in use. In some embodiments, the 3D pipeline state 3130 commands can also selectively disable or bypass certain pipeline elements if those elements will not be used.

[0360] In some embodiments, 3D primitive 3132 commands are used to submit 3D primitives to be processed by the 3D pipeline. Commands and associated parameters passed to the graphics processor via a 3D primitive 3132 command are forwarded to a vertex fetch function in the graphics pipeline. The vertex fetch function uses the 3D primitive 3132 command data to generate a number of vertex data structures. The vertex data structures are stored in one or more vertex buffers. In some embodiments, 3D primitive 3132 commands are used to perform vertex operations on 3D primitives via a vertex shader. To process the vertex shader, the 3D pipeline 3122 dispatches shader execution threads to the graphics processor execution units.

[0361] In some embodiments, the 3D pipeline 3122 is triggered via an execute 3134 command or an event. In some embodiments, a register write triggers command execution. In some embodiments, execution is triggered via a 'go' or 'kick' command in the command sequence. In one embodiment, a pipeline synchronization command is used to trigger command execution in order to flush the command sequence with no-ops through the graphics pipeline. The 3D pipeline will perform geometry processing for 3D primitives. Once the processing is completed, the resulting geometry is rasterized and the pixel engine is used to shade the resulting pixels. For these operations, additional commands can also be included to control the pixel shading and pixel back end operations.

[0362] In some embodiments, when performing media operations, the graphics processor command sequence 3110 follows the media pipeline 3124 path. In general, the specific use and manner of programming for the media pipeline 3124 depends on the media or compute operations to be performed. In media decode processes, specific media decode operations can be offloaded to the media pipeline. In some embodiments, the media pipeline can also be bypassed and media decode can be performed entirely or partially using resources provided by one or more general purpose processing cores. In one embodiment, the media pipeline also includes elements for general purpose graphics processor unit (GPGPU) operations, where the graphics processor is used to execute SIMD vector operations using computational shader programs that are not explicitly related to the rendering of graphics primitives.

[0363] In some embodiments, media pipeline 3124 is configured in a similar manner as 3D pipeline 3122. A set of commands to configure media pipeline state 3140 is dispatched or placed into a command queue, prior to media object command 3142. In some embodiments, media pipeline state commands 3140 include data to configure media pipeline elements that will be used to process the media object. This includes data to configure video decode and video encode logic within the media pipeline, such as encoding or decoding formats. In some embodiments, commands 3140 to configure media pipeline state also support the use of one or more pointers to "indirect" state elements that contain a batch of state settings.

[0364] In some embodiments, media object command 3142 supplies a pointer to a media object for processing by the media pipeline. The media object includes a memory buffer containing video data to be processed. In some embodiments, all media pipeline state must be valid prior to issuing media object command 3142. Once the pipeline state is configured and media object command 3142 is queued, the media pipeline 3124 is triggered via an execute 3144 command or equivalent execution event (e.g., register write). Output from the media pipeline 3124 can then be post-processed by operations provided by 3D pipeline 3122 or media pipeline 3124. In some embodiments, GPGPU operations are configured and executed in a similar manner as media operations.

[0365] Graphics software architecture

[0366] FIG. 32 An exemplary graphics software architecture of a data processing system 3200 is shown in accordance with some embodiments. In some embodiments, the software architecture includes a 3D graphics application 3210, an operating system 3220, and at least one processor 3230. In some embodiments, processor 3230 includes a graphics processor 3232 and one or more general-purpose processor cores 3234. Graphics application 3210 and operating system 3220 each execute in system memory 3250 of the data processing system.

[0367] In some embodiments, 3D graphics application 3210 contains one or more shader programs including shader instructions 3212. The shader language instructions can be in a high-level shader language, such as High-Level Shader Language (HLSL) or OpenGL Shader Language (GLSL). The application also includes executable instructions 3214 in a machine language suitable for execution by the general- purpose processor cores 3234. The application also includes graphics objects 3216 defined by vertex data.

[0368] In some embodiments, the operating system 3220 is from Microsoft Corporation. The operating system 3220 may be a dedicated UNIX-like operating system or an open-source UNIX-like operating system using a variant of the Linux kernel. The operating system 3220 may support graphics APIs 3222, such as the Direct3D API, OpenGL API, or Vulkan API. When the Direct3D API is in use, the operating system 3220 uses a front-end shader compiler 3224 to compile any shader instructions 3212 in HLSL into a lower-level shader language. This compilation may be just-in-time (JIT) compilation or pre-compilation of the application-executable shaders. In some embodiments, high-level shaders are compiled into low-level shaders during the compilation of the 3D graphics application 3210. In some embodiments, the shader instructions 3212 are provided in an intermediate form, such as a version of the standard Portable Intermediate Representation (SPIR) used by the Vulkan API.

[0369] In some embodiments, the user-mode graphics driver 3226 includes a back-end shader compiler 3227 for translating shader instructions 3212 into a hardware-specific representation. When using the OpenGL API, shader instructions 3212 in the GLSL high-level language are passed to the user-mode graphics driver 3226 for compilation. In some embodiments, the user-mode graphics driver 3226 uses operating system kernel-mode functionality 3228 to communicate with the kernel-mode graphics driver 3229. In some embodiments, the kernel-mode graphics driver 3229 communicates with the graphics processor 3232 to dispatch commands and instructions.

[0370] IP core implementation

[0371] One or more aspects of at least one embodiment can be implemented by representative code stored on a machine-readable medium that represents and / or defines logic within an integrated circuit, such as a processor. For example, the machine-readable medium may include instructions representing various logic within a processor. When read by a machine, these instructions can cause the machine to manufacture logic for performing the techniques described herein. Such representations (referred to as “IP cores”) are reusable units of logic for an integrated circuit, which can be stored on a tangible, machine-readable medium as a hardware model describing the structure of the integrated circuit. The hardware model can be supplied to various consumers or manufacturing facilities that load the hardware model onto manufacturing machines that manufacture integrated circuits. Integrated circuits can be manufactured such that the circuits perform the operations described in association with any of the embodiments described herein.

[0372] FIG. 33 is a block diagram showing an IP core development system 3300 that can be used to manufacture integrated circuits to perform operations in accordance with embodiments. The IP core development system 3300 can be used to generate modular, re-usable designs that can be incorporated into larger designs or used to construct an entire integrated circuit (e.g., an SOC integrated circuit). A design facility 3330 can employ a high-level programming language (e.g., C / C++) to generate a software simulation 3310 of the IP core design. The software simulation 3310 can be used to design, test, and verify the behavior of the IP core using a simulation model 3312. The simulation model 3312 can include functional, behavioral, and / or timing simulations. The simulation model 3312 can then be used to create or synthesize a register transfer level (RTL) design 3315. The RTL design 3315 is an abstraction of the behavior of the integrated circuit (including associated logic that executes using modeled digital signals) that models the flow of digital signals between hardware registers, including the associated logic performed by the hardware registers. In addition to an RTL design 3315, a lower- level design, such as a logic level or transistor level design can also be created, designed, or synthesized.

[0373] The RTL design 3315, or equivalent, can be further synthesized, created, or fabricated into a hardware model 3320 by the design facility, which can employ a hardware description language (HDL) or another representation of the design. The HDL can be further simulated or tested to verify the IP core design. The IP core design can be stored using non-volatile memory 3340 (e.g., hard disk, flash memory, or any non-volatile storage medium) for delivery to a third party fabrication facility 3365. Alternatively, the IP core design can be transmitted (e.g., via the Internet) over a wired connection 3350 or wireless connection 3360. The fabrication facility 3365 can then fabricate an integrated circuit that is based at least in part on the IP core design. The fabricated integrated circuit can be configured to perform operations in accordance with at least one embodiment described herein.

[0374] Exemplary system on a chip integrated circuit

[0375] FIGS. 34-36 Exemplary integrated circuits and related graphics processors that can be manufactured using one or more IP cores in accordance with various embodiments described herein are shown. Other logic and circuitry can also be included, including additional graphics processors / cores, peripheral interface controllers, or general purpose processor cores.

[0376] FIG. 34is a block diagram that shows an exemplary system on a chip integrated circuit 3400 that can be fabricated using one or more IP cores according to an embodiment. The exemplary integrated circuit 3400 includes one or more application processors 3405 (e.g., CPUs), at least one graphics processor 3410, and can additionally include an image processor 3415 and / or a video processor 3420, any of which can be a modular IP core from the same or multiple different design facilities. The integrated circuit 3400 includes peripheral or bus logic including USB controllers 3425, UART controllers 3430, SPI / SDIO controllers 3435, and I2C controllers 3440. In addition, the integrated circuit can also include a display device 3445 coupled to one or more of a high-definition multimedia interface (HDMI) controller 3450 and a mobile industry processor interface (MIPI) display interface 3455. Storage can be provided by a flash memory subsystem 3460 including flash memory and a flash memory controller. Memory interfaces can be provided via a memory controller 3465 to access SDRAM or SRAM memory devices. In addition, some integrated circuits include an embedded security engine 3470. 2 S / I 2 C controllers 3440. In addition, the integrated circuit can also include a display device 3445 coupled to one or more of a high-definition multimedia interface (HDMI) controller 3450 and a mobile industry processor interface (MIPI) display interface 3455. Storage can be provided by a flash memory subsystem 3460 including flash memory and a flash memory controller. Memory interfaces can be provided via a memory controller 3465 to access SDRAM or SRAM memory devices. In addition, some integrated circuits include an embedded security engine 3470.

[0377] FIG. 35 is a block diagram that shows an exemplary graphics processor 3510 that can be fabricated using one or more IP cores according to an embodiment. The graphics processor 3510 can be a variant of the graphics processor 3410 of FIG. 34 . The graphics processor 3510 includes a vertex processor 3505 and one or more fragment processor(s) 3515A-3515N (e.g., 3515A, 3515B, 3515C, 3515D, up to 3515N-1, and 3515N). The graphics processor 3510 can execute different shader programs via separate logic for vertex processing, such as vertex

[0378] In addition, the graphics processor 3510 also includes one or more memory management units (MMUs) 3520A-3520B, cache memory 3525A-3525B, and circuit interconnects 3530A-3530B. One or more MMU(s) 3520A-3520B provide for virtual to physical address mapping for vertex processor 3505, and / or the one or more fragment processor(s) 3515A-3515N, in addition to other graphics processor 3510 components. The one or more MMU(s) 3520A-3520B can be synchronized with other MMUs within the system, for example, one or more MMUs associated with one or more application processor(s) 3405, a GPU(s) 3415, and / or video processor(s) 3420, such that each processor 3405-3420 can participate in a shared or unified virtual memory system. The one or more circuit interconnects 3530A-3530B enable graphics processor 3510 to interface with other IP cores within the SoC, either via an internal bus, as shown, via a direct connection, or via some other interconnect mechanism. FIG. 34

[0379] FIG. 36 is a block diagram that illustrates an additional exemplary graphics processor 3610 that can be fabricated using one or more IP cores, according to embodiments. The graphics processor 3610 can be a variant of the graphics processor 3410 of FIG. 34 . The graphics processor 3610 includes FIG. 35 one or more MMUs 3520A-3520B, caches 3525A-3525B, and circuit interconnects 3530A-3530B of the integrated circuit 3500.

[0380] ​The graphics processor 3610 includes one or more shader cores 3615A-3615N (e.g., 3615A, 3615B, 3615C, 3615D, 3615E, 3615F, up to 3615N-1, and 3615N) that provide a unified shader core architecture, in which single instruction multiple thread (SIMT) instruction processing can be performed on multiple threads by one or more of the cores. The exact number of shader cores to be active and / or included can vary in different embodiments and implementations. In addition, the graphics processor 3610 includes an inter-core task manager 3605 that serves as a thread dispatcher and scheduler to facilitate parallel processing between and among the shader cores 3615A-3615N and / or task manager 3605. A task manager 3605 can also manage communication and data sharing between functions and / or threads on the graphics processor 3610. In some embodiments, the task manager 3605 can also control the low power mode for the graphics processor 3610.

[0381] Embodiments described herein can be implemented in any combination of the following: one or more microchips or integrated circuits using motherboard interconnects, hardwired logic, software stored by a memory device and executed by a microprocessor, firmware, application specific integrated circuit (ASIC), and / or field programmable gate array (FPGA). By way of example, the term "logic" can include software or hardware, and / or combinations of software and hardware.

[0382] Embodiments can be provided, for example, as a computer program product which can include one or more machine-readable media having stored thereon machine-executable instructions that, when executed by one or more machines such as a computer, network of computers, or other electronic devices, can cause the one or more machines to carry out operations according to embodiments described herein. Machine-readable media can include, but is not limited to, floppy diskettes, optical disks, CD-ROMs (Compact Disc-Read Only Memories), and magneto-optical disks, ROMs, RAMs, EPROMs (Erasable Programmable Read Only Memories), EEPROMs (Electrically Erasable Programmable Read Only Memories), magnetic or optical cards, flash memory, or other type of media suitable for storing machine-executable instructions. Embodiments can also be provided as a web service.

[0383] The following clauses and / or examples relate to particular embodiments or examples thereof. Details in the examples can be used anywhere in one or more embodiments. Various features of the different embodiments or examples can be combined to produce various modifications of the included examples that rely on the principles of the described embodiments. Examples can include subject matter such as a method, means for performing acts of the method, at least one machine-readable medium including instructions that, when performed by a machine cause the machine to carry out acts of the method, or of an apparatus or system according to embodiments and examples described herein. Various components can be means for performing the operations or functions described.

[0384] One embodiment provides a method of transmitting data between a plurality of compute nodes of a distributed computing system, the method comprising: creating a global view of a communication operation to be performed between the plurality of compute nodes of the distributed computing system, the global view being created using information specific to a machine learning model associated with the distributed computing system; determining a communication cost of the communication operation using the global view; and automatically determining a number of network endpoints for transmitting the data between the plurality of compute nodes of the distributed computing system.

[0385] One embodiment provides a system for computing and distributing data for distributed training of a neural network, the system comprising a system memory to store a set of trainable machine learning parameters and a library to facilitate data transfer during distributed training of the neural network; a fabric interface to enable transmission and reception of data associated with the set of trainable machine learning parameters; a first set of general purpose processor cores to execute instructions provided by the library to control a data transfer library; and a general purpose graphics processor to perform computational operations associated with a machine learning framework workflow to generate gradient data for the trainable machine learning parameters, wherein the first set of general purpose processor cores are to control the data transfer library to send and receive training data via the fabric interface during the machine learning framework workflow.

[0386] One embodiment provides a system to configure distributed training of a neural network, the system comprising: a memory to store a library to facilitate data transfer during distributed training of a neural network, the data associated with trainable parameters of the neural network; a network interface to transmit and receive gradient data associated with the trainable parameters; a general purpose processor to execute instructions provided by the library, the instructions causing the general purpose processor to configure the network interface to transmit and receive the gradient data associated with the trainable parameters during a workflow of a machine learning framework; and a graphics processor to perform a compute operation associated with the machine learning framework workflow to generate the gradient data associated with the trainable parameters, wherein based on the machine learning framework workflow, the library interleaves the compute operation on the graphics processor with the transmission and reception of the gradient data via the network interface.

[0387] One embodiment provides a graphics processing unit, the graphics processing unit comprising: a compute block comprising one or more processing clusters to perform a compute operation associated with a machine learning framework workflow, the compute operation to generate gradient data associated with trainable parameters of a neural network; a memory device to store the gradient data during distributed training of the neural network; and a fabric interface coupled with the memory device, the fabric interface to transmit and receive the gradient data associated with the trainable parameters.

[0388] One embodiment provides a method to perform distributed training of a neural network, the method comprising: storing a library in a memory, the library to facilitate data transfer during distributed training of a neural network, the data associated with trainable parameters of the neural network; transmitting and receiving gradient data associated with the trainable parameters via a network interface; executing instructions provided by the library via a general purpose processor, the instructions to cause the general purpose processor to configure the network interface to transmit and receive the gradient data associated with the trainable parameters during a workflow of a machine learning framework; and performing a compute operation associated with the machine learning framework workflow via a graphics processor to generate the gradient data associated with the trainable parameters, the compute operation interleaved with the transmission and reception of the gradient data via the network interface.

[0389] Those skilled in the art can appreciate from the foregoing description that the broad techniques of the disclosed embodiments can be implemented in a variety of forms. Therefore, although the embodiments of the present disclosure have been described in connection with particular examples thereof, the true scope of the embodiments should not be so limited since other modifications will become apparent to the skilled practitioner upon a reading of the foregoing description.

Claims

1. A system for configuring distributed training of a neural network using a plurality of interconnected worker nodes of a distributed training network, the plurality of interconnected worker nodes of the distributed training network being interconnected via a communication structure, the system comprising: A memory, which serves as a repository for facilitating data transfer during distributed training of the neural network, the data being associated with trainable parameters of the neural network; Multiple worker nodes, each worker node comprising: A structural interface configured to connect to the communication structure, the structural interface being used to send and receive gradient data associated with the trainable parameters, wherein during training, the worker node sends and receives the gradient data associated with the trainable parameters via the structural interface; A general-purpose processor, configured to execute instructions provided by the library, which cause the general-purpose processor to configure the architecture interface to send and receive the gradient data associated with the trainable parameters during the workflow of the machine learning framework; and A graphics processing unit (GPU) is configured to perform computational operations associated with a machine learning framework workflow to generate gradient data associated with the trainable parameters, wherein, based on the machine learning framework workflow, one or more instructions provided by the library interleave the computational operations on the GPU with the transmission and reception of gradient data via the architecture interface.

2. The system as described in claim 1, characterized in that, The computational operations are configured to overlap with communication operations that send or receive gradient data via the structure interface.

3. The system as described in claim 2, characterized in that, The graphics processor is used to perform computational operations associated with the workflow of the machine learning framework, the computational operations being associated with a first portion of a first layer of the neural network.

4. The system as described in claim 3, characterized in that, In response to a notification that the computational operation associated with the first part of the first layer of the neural network has been completed, one or more of the instructions provided by the library are used to cause the structure interface to send the result of the computational operation.

5. The system of claim 4, wherein the structural interface is configured to send the result according to a communication mode for messages to be transmitted between the plurality of worker nodes during the distributed training of the neural network.

6. The system as described in claim 5, characterized in that, The communication modes are aggregation, dissemination, full aggregation, full interchange, reduction, reduction-dissemination, or full reduction.

7. The system as described in claim 1, characterized in that, The structure interface is a fast interface for interconnecting peripheral components.

8. The system as described in claim 1, characterized in that, The interface of the structure is the NVLink interface.

9. The system as described in any one of claims 1-8, characterized in that, The graphics processor includes at least a portion of the structure interface.

10. A method for performing distributed training of a neural network using a plurality of interconnected worker nodes of a distributed training network, the plurality of interconnected worker nodes of the distributed training network being interconnected via a communication structure and each including a structural interface configured for connection to the communication structure, the method comprising: The library is stored in memory to facilitate data transfer during distributed training of the neural network, the data being associated with trainable parameters of the neural network; In each worker node: The worker node sends and receives gradient data associated with the trainable parameters via the structural interface, wherein during training, the worker node sends and receives the gradient data associated with the trainable parameters via the structural interface. The general-purpose processor of the worker node executes instructions provided by the library, which cause the general-purpose processor to configure the architecture interface to send and receive the gradient data associated with the trainable parameters during the workflow of the machine learning framework; as well as The worker node's graphics processor performs computational operations associated with the machine learning framework workflow to generate the gradient data associated with the trainable parameters, the computational operations being interleaved with the sending and receiving of gradient data via the structural interface.

11. The method of claim 10, further comprising configuring computational operations to overlap with communication operations that send or receive gradient data via the structure interface.

12. The method of claim 11, further comprising performing computational operations associated with the machine learning framework workflow via the graphics processor, the computational operations being associated with a first portion of a first layer of the neural network.

13. The method as described in claim 12, characterized in that, In response to a notification that the computational operation associated with the first part of the first layer of the neural network has been completed, the structure interface sends the result of the computational operation via one or more of the instructions provided by the library.

14. The method of claim 13, further comprising sending the result according to a communication mode for transmitting messages between the plurality of worker nodes during distributed training of the neural network.

15. A non-transient machine-readable medium storing instructions, said instructions, when executed by one or more processors, causing said one or more processors to perform operations including: A library is stored in memory to facilitate data transfer during distributed training of a neural network, the data being associated with trainable parameters of the neural network, wherein the distributed training of the neural network is performed using a plurality of interconnected worker nodes of a distributed training network, the plurality of interconnected worker nodes of the distributed training network being interconnected via a communication structure and each including a structural interface configured for connection to the communication structure. In each worker node: The worker node sends and receives gradient data associated with the trainable parameters via the structural interface, wherein during training, the worker node sends and receives the gradient data associated with the trainable parameters via the structural interface. The general-purpose processor of the worker node executes instructions provided by the library, which cause the general-purpose processor to configure the architecture interface to send and receive the gradient data associated with the trainable parameters during the workflow of the machine learning framework; as well as The worker node's graphics processor performs computational operations associated with the machine learning framework workflow to generate the gradient data associated with the trainable parameters, the computational operations being interleaved with the sending and receiving of gradient data via the structural interface.

16. The non-transient machine-readable medium of claim 15, wherein the operation additionally includes configuring computational operations to overlap with communication operations for sending or receiving gradient data via the structural interface.

17. The non-transient machine-readable medium of claim 16, wherein the operation additionally includes performing computational operations associated with the workflow of the machine learning framework via the graphics processor, the computational operations being associated with a first portion of a first layer of the neural network.

18. The non-transient machine-readable medium as claimed in claim 17, characterized in that, In response to a notification that the computational operation associated with the first part of the first layer of the neural network has been completed, the structure interface sends the result of the computational operation via one or more of the instructions provided by the library.

19. The non-transient machine-readable medium of claim 18, additionally comprising transmitting the result according to a communication mode for transmitting messages between the plurality of worker nodes during distributed training of the neural network.

20. A graphics processing unit, comprising: A memory device for storing gradient data during distributed training of a neural network; A structural interface coupled to the memory device is configured to connect to a communication structure via which multiple worker nodes of a distributed training network are interconnected. The structural interface is used to send and receive gradient data associated with trainable parameters of the neural network, wherein during training, the worker nodes send and receive the gradient data associated with the trainable parameters via the structural interface. as well as The computation block includes one or more processing clusters for performing computational operations associated with a machine learning framework workflow, the computational operations for generating gradient data associated with the trainable parameters.

21. The graphics processing unit of claim 20, further comprising a structure interface cache for storing the gradient data before the gradient data is sent by the structure interface or after the gradient data is received.

22. The graphics processing unit as claimed in claim 21, characterized in that, The structure interface cache is associated with the memory device.

23. The graphics processing unit as claimed in claim 20, characterized in that, The structure interface has an address space that is mapped to at least a portion of the memory device.

24. The graphics processing unit as claimed in claim 23, characterized in that, The computation block and the interface structure share a unified address space.

Images