Systems and methods for improving cache efficiency and utilization

Abstract

Systems and methods for improving cache efficiency and utilization are disclosed. In one embodiment, a graphics processor includes: a processing resource for performing graphics operations; and a cache controller for a cache coupled to the processing resource. The cache controller is configured to control cache priority by determining whether default settings or instructions control cache operations that control the cache.

Description

[0001] This invention patent application is a divisional application of the invention patent application with international application number PCT / US2020 / 022837, international application date of March 14, 2020, and Chinese national phase application number 202080011587.X, entitled "System and method for improving cache efficiency and utilization".

[0002] Cross-reference of related applications

[0003] This application relates to U.S. Provisional Application 62 / 819,337, entitled "GRAPHICS PROCESSING," filed March 15, 2019, by Abhishek Appu et al. (Agent's Case No. AC0271-Z); U.S. Provisional Application 62 / 819,435, entitled "GRAPHICS DATA PROCESSING," filed March 15, 2019, by Lakshminarayanan Striramassarma et al. (Agent's Case No. AC0285-Z); and U.S. Provisional Application 62 / 819,435, entitled "GRAPHICS DATA PROCESSING," filed March 15, 2019, by Subramaniam Maiyuran et al. (Agent's Case No. AC0285-Z); and U.S. Provisional Application 62 / 819,435, entitled "SYSTEMS AND METHODS FOR PARTITIONING CACHE TO REDUCE CACHE ACCESS," filed March 15, 2019, by Subramaniam Maiyuran et al. U.S. Provisional Application 62 / 819,361 (Agent's Case No. AC0286-Z) entitled "LATENCY (System and method for partitioning a cache to reduce cache access latency)," and claims the benefit and priority of these applications pursuant to 35 U.S. SC119(e), the contents of which are incorporated herein by reference. Technical Field

[0004] This disclosure relates generally to data processing, and more specifically to data processing via a general-purpose graphics processing unit. Background Technology

[0005] Current parallel graphics data processing includes systems and methods developed to perform specific operations on graphics data, such as, for example, linear interpolation, tessellation, rasterization, texture mapping, depth testing, etc. Traditionally, graphics processing units (GPUs) use fixed-function computing units to process graphics data; however, recent advancements have made multiple parts of GPUs programmable, enabling such processors to support a wider range of operations for processing vertex and fragment data.

[0006] To further enhance performance, graphics processing units (GPUs) typically implement processing techniques such as pipelined processing, which attempt to process as much graphics data as possible in parallel across different parts of the graphics pipeline. Parallel GPUs with a Single Instruction Multiple Thread (SIMT) architecture are designed to maximize the amount of parallel processing in the graphics pipeline. In a SIMT architecture, groups of parallel threads attempt to execute program instructions together synchronously as frequently as possible to increase processing efficiency. A general overview of the software and hardware of the SIMT architecture can be found in Chapter 3, pp. 37-51 (2013), of Shane Cook's CUDA Programming. Attached Figure Description

[0007] Therefore, in order to understand in detail the features of the present embodiments described above, a more specific description of the embodiments briefly summarized above can be made with reference to the embodiments, some of which are illustrated in the accompanying drawings. However, it should be noted that the accompanying drawings illustrate only typical embodiments and should therefore not be considered as limiting the scope of the embodiments.

[0008] Figure 1 This is a block diagram illustrating a computer system configured to implement one or more aspects of the embodiments described herein;

[0009] Figures 2A-2D The diagram illustrates the parallel processor components;

[0010] Figures 3A-3C This is a block diagram of graphics multiprocessors and multiprocessor-based GPUs;

[0011] Figures 4A-4F The illustration shows an exemplary architecture in which multiple GPUs are communicatively coupled to multiple multi-core processors;

[0012] Figure 5 The diagram illustrates a graphics processing pipeline;

[0013] Figure 6 Illustrated machine learning software stack;

[0014] Figure 7 The diagram illustrates a general-purpose graphics processing unit;

[0015] Figure 8 The diagram illustrates a multi-GPU computing system.

[0016] Figures 9A-9B The diagram illustrates a layer of an exemplary deep neural network;

[0017] Figure 10 The diagram illustrates an exemplary recurrent neural network;

[0018] Figure 11 The diagram illustrates the training and deployment of a deep neural network;

[0019] Figure 12 This is a block diagram illustrating distributed learning;

[0020] Figure 13 The illustration shows an exemplary inference system-on-a-chip (SOC) suitable for performing inference using a trained model;

[0021] Figure 14 This is a block diagram of the processing system;

[0022] Figures 15A-15C Graphical computing systems and graphics processors;

[0023] Figures 16A-16C The diagram shows an additional block diagram of the graphics processor and computing accelerator architecture;

[0024] Figure 17 It is a block diagram of the graphics processing engine of a graphics processor;

[0025] Figures 18A-18B The diagram illustrates the thread execution logic of the array of processing elements used in the graphics processor core;

[0026] Figure 19 The diagram shows an additional execution unit;

[0027] Figure 20 It is a block diagram illustrating the instruction format of a graphics processor;

[0028] Figure 21 This is a block diagram of the additional graphics processor architecture;

[0029] Figures 22A-22B The diagram illustrates the command format and command sequence of a graphics processor.

[0030] Figure 23 The diagram illustrates an exemplary graphical software architecture for a data processing system.

[0031] Figure 24A This is a block diagram illustrating the IP core development system;

[0032] Figure 24B The illustration shows a cross-sectional side view of an integrated circuit packaging assembly.

[0033] Figure 24C The diagram illustrates a package assembly that includes hardware logic chips (e.g., base dies) connected to multiple units of a substrate.

[0034] Figure 24D The diagram shows the packaging components for interchangeable chiplets;

[0035] Figure 25 This is a block diagram illustrating an exemplary system-on-a-chip integrated circuit; and

[0036] Figures 26A-26B This is a block diagram illustrating an exemplary graphics processor used within a SoC.

[0037] Figure 27 The illustration, according to one embodiment, shows a table of supported first-level cache attributes for each message type.

[0038] Figure 28 The illustration, according to one embodiment, shows a table of supported multi-level caching attributes for each message type.

[0039] Figure 29 The illustration shows a method 2900 according to an embodiment having an exemplary sequence of operations for controlling the priority of a first-level cache to improve cache efficiency.

[0040] Figure 30 The illustration, according to one embodiment, shows a table of L1 cache priorities for instructions relative to default cache attributes.

[0041] Figure 31 The illustration, according to another embodiment, shows a table of L1 cache priorities for instructions relative to default cache attributes.

[0042] Figure 32 The illustration, according to one embodiment, shows a table of L2 / L3 cache priorities for instructions relative to the default L2 / L3 cache control.

[0043] Figure 33 The illustration, according to another embodiment, shows a table of L2 / L3 cache priorities relative to PTE settings for instructions and default settings.

[0044] Figure 34 The figure shows a flowchart 3400 of a partitioning scheme for use in a shared cache, according to one embodiment.

[0045] Figure 35 The illustration shows how changing the number of workgroups (or warp groups) in one embodiment alters the availability of a traditional cache.

[0046] Figure 36A The diagram illustrates a flowchart for selecting hardware cache coherency according to one embodiment.

[0047] Figure 36B The diagram illustrates a memory hierarchy structure according to one embodiment.

[0048] Figure 37A , Figure 37B and Figure 37CThe illustration shows a method according to one embodiment having an exemplary sequence of operations for controlling cache operations to improve cache efficiency.

[0049] Figure 38 The diagram illustrates a memory architecture 3800 with decoupled control data and data arrays.

[0050] Figure 39 The illustration shows a compression operation based on additional compression for a GPU according to one embodiment.

[0051] Figure 40 The illustration shows a method 4000 according to one embodiment having an exemplary sequence of operations for controlling cache operations to improve cache efficiency. Detailed Implementation

[0052] The Graphics Processing Unit (GPU) is communicatively coupled to the host / processor core to accelerate tasks such as graphics operations, machine learning operations, pattern analysis operations, and / or various general-purpose GPU (GPGPU) functions. The GPU can be communicatively coupled to the host processor / core via a bus or another interconnect (e.g., a high-speed interconnect such as PCIe or NVLink). Alternatively, the GPU may be integrated with the core on the same package or chip and communicatively coupled to the core via an internal processor bus / interconnect (i.e., within the package or chip). Regardless of how the GPU is connected, the processor core can assign work to the GPU in the form of a sequence of commands / instructions contained in a job descriptor. The GPU then uses dedicated circuitry / logic to efficiently process these commands / instructions.

[0053] In the following description, numerous specific details are set forth to provide a more thorough understanding. However, it will be apparent to those skilled in the art that the embodiments described herein can be practiced without one or more of these specific details. In other instances, well-known features have not been described so as not to obscure the details of the present embodiments.

[0054] System Overview

[0055] Figure 1This is a block diagram illustrating a computing system 100 configured to implement one or more aspects of the exemplary embodiments described herein. The computing system 100 includes a processing subsystem 101 having one or more processors 102 and system memory 104, the processors 102 communicating with the system memory 104 via an interconnect path that may include a memory hub 105. The memory hub 105 may be a separate component within a chipset assembly or may be integrated within one or more processors 102. The memory hub 105 is coupled to an I / O subsystem 111 via a communication link 106. The I / O subsystem 111 includes an I / O hub 107 that enables the computing system 100 to receive input from one or more input devices 108. Additionally, the I / O hub 107 enables a display controller to provide output to one or more display devices 110A, which may be included in one or more processors 102. In one embodiment, the one or more display devices 110A coupled to the I / O hub 107 may include local, internal, or embedded display devices.

[0056] Processing subsystem 101 may include, for example, one or more parallel processors 112 coupled to memory hub 105 via a bus or other communication link 113. Communication link 113 may be any number of standards-based communication link technologies or protocols, such as, but not limited to, PCI Express, or may be vendor-specific communication interfaces or communication structures. The one or more parallel processors 112 may form a computationally focused parallel or vector processing system that may include a large number of processing cores and / or processing clusters, such as an integrated many-core (MIC) processor. For example, the one or more parallel processors 112 may form a graphics processing subsystem capable of outputting pixels to one of one or more display devices 110A coupled via I / O hub 107. The one or more parallel processors 112 may also include a display controller and a display interface (not shown) for implementing direct connections to one or more display devices 110B.

[0057] Within the I / O subsystem 111, system storage unit 114 can be connected to I / O hub 107, thereby providing a storage mechanism for computing system 100. I / O switch 116 can be used to provide an interface mechanism for connecting I / O hub 107 to other components, such as network adapter 118 and / or wireless network adapter 119 which can be integrated into the platform, and various other devices that can be added via one or more plug-in devices 120. The plug-in devices 120 may also include, for example, one or more external graphics processing units and / or computing accelerators. Network adapter 118 can be an Ethernet adapter or another wired network adapter. Wireless network adapter 119 can include one or more of Wi-Fi, Bluetooth, Near Field Communication (NFC), or other network devices including one or more wireless devices.

[0058] The computing system 100 may include other components not explicitly shown, including USB or other port connections, optical storage drives, video capture devices, etc., which may also be connected to the I / O hub 107. Figure 1 The communication paths for interconnecting the components can be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect) based protocols (e.g., PCI-Express) or any other bus or point-to-point communication interface and / or (multiple) protocols, such as NVLink high-speed interconnect, or interconnect protocols known in the art.

[0059] One or more parallel processors 112 may include circuitry optimized for graphics and video processing (including, for example, video output circuitry) and constitute a graphics processing unit (GPU). Alternatively or additionally, as described in more detail herein, one or more parallel processors 112 may include circuitry optimized for general-purpose processing while preserving the underlying computing architecture. Components of the computing system 100 may be integrated with one or more other system elements on a single integrated circuit. For example, one or more parallel processors 112, a memory hub 105, a processor 102, and an I / O hub 107 may be integrated into a system-on-a-chip (SoC) integrated circuit. Alternatively, components of the computing system 100 may be integrated into a single package to form a system-in-package (SIP) configuration. In one embodiment, at least a portion of the components of the computing system 100 may be integrated into a multi-chip module (MCM) that can interconnect with other MCMs to form a modular computing system.

[0060] It will be understood that the computing system 100 shown herein is illustrative, and variations and modifications are possible. The connectivity topology can be modified as needed, including the number and arrangement of bridges, the number of processors(102), and the number of parallel processors(112). For example, system memory 104 may be connected directly to processors(102) instead of via bridges, while other devices communicate with system memory 104 through memory hub 105 and processors(102). In other alternative topologies, parallel processors(112) are connected to I / O hub 107 or directly to one or more processors(102) instead of memory hub 105. In other embodiments, I / O hub 107 and memory hub 105 may be integrated into a single chip. It is also possible for two or more sets of processors(102) to be attached via multiple slots, and these processors(102) may be coupled to two or more instances of parallel processors(112).

[0061] Some of the specific components shown in this document are optional and not included in all implementations of the computing system 100. For example, any number of plug-in cards or peripherals may be supported, or some components may be eliminated. Furthermore, some architectures may be designed for similar... Figure 1 The components illustrated in the diagram use different terminology. For example, in some architectures, the memory hub 105 may be referred to as the Northbridge, while the I / O hub 107 may be referred to as the Southbridge.

[0062] Figure 2A The figure illustrates a parallel processor 200. The parallel processor 200 can be a GPU, GPGPU, etc., as described herein. The components of the parallel processor 200 can be implemented using one or more integrated circuit devices such as a programmable processor, an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). The illustrated parallel processor 200 can be... Figure 1 The parallel processors 112 shown, or Figure 1 One of the (multiple) parallel processors 112 shown.

[0063] Parallel processor 200 includes parallel processing unit 202. The parallel processing unit includes I / O unit 204 for communication with other devices, including other instances of parallel processing unit 202. I / O unit 204 can be directly connected to other devices. For example, I / O unit 204 is connected to other devices via a hub or switch interface (such as memory hub 105). The connection between memory hub 105 and I / O unit 204 forms a communication link 113. Within parallel processing unit 202, I / O unit 204 is connected to host interface 206 and memory crossbar switch 216, wherein host interface 206 receives commands relating to performing processing operations, and memory crossbar switch 216 receives commands relating to performing memory operations.

[0064] When host interface 206 receives a command buffer via I / O unit 204, host interface 206 can route work operations for executing those commands to front end 208. In one embodiment, front end 208 is coupled to scheduler 210, which is configured to distribute commands or other work items to processing cluster array 212. Scheduler 210 ensures that processing cluster array 212 is properly configured and active before tasks are distributed to processing clusters within processing cluster array 212. Scheduler 210 can be implemented via firmware logic executed on a microcontroller. The microcontroller-implemented scheduler 210 can be configured to perform complex scheduling and work distribution operations at both coarse and fine granular levels, enabling fast preemption and context switching of threads executing on processing array 212. Preferably, host software can validate workloads for scheduling on processing array 212 via one of a plurality of graphics processing doorbell mechanisms. The workloads can then be automatically distributed across processing array 212 by scheduler 210 logic within the scheduler microcontroller.

[0065] Processing cluster array 212 may include up to "N" processing clusters (e.g., clusters 214A, 214B to 214N). Each cluster 214A-214N in processing cluster array 212 can execute a large number of concurrent threads. Scheduler 210 may use various scheduling and / or work distribution algorithms to distribute work to clusters 214A-214N in processing cluster array 212, which may vary depending on the workload for each type of program or computation. Scheduling may be handled dynamically by scheduler 210 or may be partially assisted by compiler logic during the compilation of program logic configured for execution by processing cluster array 212. Optionally, different clusters 214A-214N in processing cluster array 212 may be assigned to process different types of programs or to perform different types of computations.

[0066] The processing cluster array 212 can be configured to perform various types of parallel processing operations. For example, the cluster array 212 can be configured to perform general-purpose parallel computing operations. For example, the processing cluster array 212 may include logic for performing processing tasks, such as filtering video and / or audio data, performing modeling operations (including physical operations), and performing data transformations.

[0067] Processing cluster array 212 is configured to perform parallel graphics processing operations. In such embodiments where the parallel processor 200 is configured to perform graphics processing operations, processing cluster array 212 may include additional logic for supporting the execution of such graphics processing operations, including but not limited to texture sampling logic for performing texture operations, as well as tessellation logic and other vertex processing logic. Additionally, processing cluster array 212 may be configured to execute shader programs related to graphics processing, such as, but not limited to, vertex shaders, tessellation shaders, geometry shaders, and pixel shaders. Parallel processing unit 202 may transfer data from system memory for processing via I / O unit 204. During processing, the transferred data may be stored in on-chip memory (e.g., parallel processor memory 222) and subsequently written back to system memory.

[0068] In embodiments where the parallel processing unit 202 is used to perform graphics processing, the scheduler 210 can be configured to divide the processing workload into tasks of approximately equal size to better distribute graphics processing operations to multiple clusters 214A-214N in the processing cluster array 212. In some of these embodiments, portions of the processing cluster array 212 can be configured to perform different types of processing. For example, a first portion can be configured to perform vertex shading and topology generation, a second portion can be configured to perform tessellation and geometry shading, and a third portion can be configured to perform pixel shading or other screen-space operations to produce a rendered image for display. Intermediate data generated by one or more clusters in clusters 214A-214N can be stored in a buffer to allow the intermediate data to be transferred between clusters 214A-214N for further processing.

[0069] During operation, the processing cluster array 212 may receive processing tasks to be executed via scheduler 210, which receives commands defining the processing tasks from front-end 208. For graphics processing operations, processing tasks may include data to be processed, along with state parameters and indexes of commands defining how that data will be processed (e.g., what program will be executed). This data may include, for example, surface (patch) data, primitive data, vertex data, and / or pixel data. Scheduler 210 may be configured to retrieve the index corresponding to the task, or may receive the index from front-end 208. Front-end 208 may be configured to ensure that the processing cluster array 212 is configured to be active before a workload specified by an incoming command buffer (e.g., a batch buffer, a push buffer, etc.) is initiated.

[0070] Each instance of one or more instances of parallel processing unit 202 may be coupled to parallel processor memory 222. Parallel processor memory 222 may be accessed via memory crossbar switch 216, which may receive memory requests from processing cluster array 212 and I / O unit 204. Memory crossbar switch 216 may access parallel processor memory 222 via memory interface 218. Memory interface 218 may include multiple partition units (e.g., partition units 220A, 220B through 220N), each of which may be coupled to a portion (e.g., a memory cell) of parallel processor memory 222. The number of partition units 220A-220N may be configured equal to the number of memory cells, such that a first partition unit 220A has a corresponding first memory cell 224A, a second partition unit 220B has a corresponding memory cell 224B, and an Nth partition unit 220N has a corresponding Nth memory cell 224N. In other embodiments, the number of partition units 220A-220N may not be equal to the number of memory devices.

[0071] Memory cells 224A-224N may include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. Optionally, memory cells 224A-224N may also include 3D stacked memory, including but not limited to high bandwidth memory (HBM). Those skilled in the art will appreciate that specific implementations of memory cells 224A-224N may vary and can be selected from a variety of conventional designs. Rendering targets, such as frame buffers or texture maps, may be stored across memory cells 224A-224N, allowing partitioning cells 220A-220N to write portions of each rendering target in parallel to efficiently utilize the available bandwidth of parallel processor memory 222. In some embodiments, local instances of parallel processor memory 222 may be excluded to facilitate a unified memory design that utilizes system memory combined with local cache memory.

[0072] Optionally, any of the clusters 214A-214N in the processing cluster array 212 is capable of processing data to be written to any of the memory cells 224A-224N within the parallel processor memory 222. The memory crossbar switch 216 can be configured to transfer the output of each cluster 214A-214N to any partition cell 220A-220N or to another cluster 214A-214N, which can perform additional processing operations on the output. Each cluster 214A-214N can communicate with the memory interface 218 via the memory crossbar switch 216 to read from or write to various external memory devices. In one embodiment with memory crossbar switch 216, the memory crossbar switch 216 has a connection to memory interface 218 for communication with I / O unit 204, and a connection to a local instance of parallel processor memory 222, thereby enabling processing units within different processing clusters 214A-214N to communicate with system memory or other memory that is not local to parallel processing unit 202. Generally, the memory crossbar switch 216 may, for example, be able to use virtual channels to separate traffic streams between clusters 214A-214N and partition units 220A-220N.

[0073] Although a single instance of the parallel processing unit 202 is illustrated within the parallel processor 200, any number of instances of the parallel processing unit 202 may be included. For example, multiple instances of the parallel processing unit 202 may be housed on a single plug-in card, or multiple plug-in cards may be interconnected. Different instances of the parallel processing unit 202 may be configured for interoperability, even if the different instances have different numbers of processing cores, different amounts of local parallel processor memory, and / or other configuration differences. Optionally, some instances of the parallel processing unit 202 may include higher precision floating-point units relative to other instances. Systems containing one or more instances of the parallel processing unit 202 or the parallel processor 200 can be implemented in a variety of configurations and form factors, including but not limited to desktop computers, laptop computers, or handheld personal computers, servers, workstations, game consoles, and / or embedded systems.

[0074] Figure 2B This is a block diagram of partition unit 220. Partition unit 220 can be... Figure 2A An example of a partition unit in partition units 220A-220N is shown. As illustrated, partition unit 220 includes an L2 cache 221, a frame buffer interface 225, and a ROP 226 (Raster Operation Unit). The L2 cache 221 is a read / write cache configured to perform load and store operations received from memory crossbar switch 216 and ROP 226. Read misses and urgent write-back requests are output from the L2 cache 221 to the frame buffer interface 225 for processing. Updates can also be sent to the frame buffer for processing via the frame buffer interface 225. In one embodiment, the frame buffer interface 225 interfaces with one of the memory cells in the parallel processor memory, such as... Figure 2A The memory cells 224A-224N (e.g., within the parallel processor memory 222). The staging unit 220 may also additionally or alternatively interface with one of the memory cells in the parallel processor memory via a memory controller (not shown).

[0075] In graphics applications, ROP 226 is a processing unit that performs raster operations such as stenciling, z-testing, blending, etc. ROP 226 then outputs processed graphics data, which is stored in graphics memory. In some embodiments, ROP 226 includes compression logic for compressing depth or color data written to memory and decompressing depth or color data read from memory. The compression logic can be lossless compression logic utilizing one or more of a variety of compression algorithms. The type of compression performed by ROP 226 can vary based on the statistical characteristics of the data to be compressed. For example, in one embodiment, Δ color compression is performed on the depth and color data tile-by-tile.

[0076] ROP 226 can be included in each processing cluster (e.g., Figure 2A The data is contained within clusters 214A-214N rather than within partition unit 220. In such embodiments, read and write requests for pixel data, rather than pixel fragment data, are transmitted via memory crossbar switch 216. The processed graphics data can be displayed on a display device (such as a display device, e.g., a memory crossbar switch 216). Figure 1 On one or more display devices 110, it can be routed for further processing by processor(s) 102, or it can be routed for... Figure 2A One of the processing entities within the parallel processor 200 further processes the data.

[0077] Figure 2C This is a block diagram of the processing cluster 214 within a parallel processing unit. For example, the processing cluster is... Figure 2A An instance of one of the processing clusters 214A-214N. Processing cluster 214 can be configured to execute multiple threads in parallel, where the term "thread" refers to an instance of a specific program executing on a particular set of input data. Optionally, Single Instruction Multiple Data (SIMD) instruction issuing techniques can be used to support the parallel execution of a large number of threads without providing multiple independent instruction units. Alternatively, Single Instruction Multiple Threading (SIMT) techniques can be used to support the parallel execution of a large number of generally synchronous threads using a common instruction unit configured to issue instructions to a set of processing engines within each processing cluster. Unlike SIMD execution mechanisms, where all processing engines typically execute the same instructions, SIMT execution allows different threads to more easily follow divergent execution paths through a given thread program. Those skilled in the art will understand that SIMD processing mechanisms represent a subset of the functionality of SIMT processing mechanisms.

[0078] The operation of the processing cluster 214 can be controlled via the pipeline manager 232, which distributes processing tasks to the SIMT parallel processors. The pipeline manager 232... Figure 2A The scheduler 210 receives instructions and manages the execution of those instructions via the graphics multiprocessor 234 and / or texture unit 236. The illustrated graphics multiprocessor 234 is an exemplary instance of a SIMT parallel processor. However, various types of SIMT parallel processors with different architectures can be included within the processing cluster 214. One or more instances of the graphics multiprocessor 234 can be included within the processing cluster 214. The graphics multiprocessor 234 can process data, and the data crossover switch 240 can be used to distribute the processed data to one of several possible destinations, including other shader units. The pipeline manager 232 can facilitate the distribution of processed data by specifying a destination for the processed data to be distributed via the data crossover switch 240.

[0079] Each graphics multiprocessor 234 within the processing cluster 214 may include the same set of functional execution logic (e.g., arithmetic logic units, load-memory units, etc.). The functional execution logic can be configured in a pipelined manner, in which new instructions can be issued before the completion of previous instructions. The functional execution logic supports a variety of operations, including integer and floating-point arithmetic, comparison operations, Boolean operations, bit shifting, and computation of various algebraic functions. Different operations can be performed using the same functional unit hardware, and any combination of functional units is possible.

[0080] Instructions sent to processing cluster 214 constitute threads. A set of threads executed across a set of parallel processing engines is a thread group. A thread group executes the same program on different input data. Each thread within a thread group can be assigned to a different processing engine within the graphics multiprocessor 234. A thread group may include fewer threads than the number of processing engines within the graphics multiprocessor 234. When a thread group includes fewer threads than the number of processing engines, one or more of the processing engines may be idle during the cycle in which the thread group is processed. A thread group may also include more threads than the number of processing engines within the graphics multiprocessor 234. When a thread group includes more threads than the number of processing engines within the graphics multiprocessor 234, processing can be performed on consecutive clock cycles. Optionally, multiple thread groups can be executed simultaneously on the graphics multiprocessor 234.

[0081] The graphics multiprocessor 234 may include an internal cache memory to perform load and store operations. Optionally, the graphics multiprocessor 234 may forgo the internal cache and use a cache memory within the processing cluster 214 (e.g., L1 cache 248). Each graphics multiprocessor 234 also has a partition unit (e.g., Figure 2AAccess to the L2 cache within the partition units 220A-220N, which are shared across all processing clusters 214 and can be used to transfer data between threads. The graphics multiprocessor 234 can also access off-chip global memory, which may include one or more of local parallel processor memory and / or system memory. Any memory outside the parallel processing unit 202 can be used as global memory. Embodiments where the processing cluster 214 includes multiple instances of the graphics multiprocessor 234 can share common instructions and data, which can be stored in the L1 cache 248.

[0082] Each processing cluster 214 may include an MMU 245 (Memory Management Unit) configured to map virtual addresses to physical addresses. In other embodiments, one or more instances of the MMU 245 may reside in Figure 2A The memory interface 218 is located within the MMU 245. The MMU 245 includes a set of page table entries (PTEs) for mapping virtual addresses to physical addresses in a slice and optionally includes cache line indexes. The MMU 245 may include an address translation lookahead buffer (TLB) or cache that may reside within the graphics multiprocessor 234 or the L1 cache or processing cluster 214. Physical addresses are processed to distribute surface data access locality, thereby allowing efficient request interleaving between partition units. The cache line index can be used to determine whether a request for a cache line has been hit or missed.

[0083] In graphics and computing applications, processing cluster 214 can be configured such that each graphics multiprocessor 234 is coupled to texture unit 236 to perform texture mapping operations, such as determining texture sample locations, reading texture data, and filtering texture data. Texture data is read from an internal texture L1 cache (not shown), or in some embodiments, from an L1 cache within the graphics multiprocessor 234, and fetched as needed from an L2 cache, local parallel processor memory, or system memory. Each graphics multiprocessor 234 outputs processed tasks to data crossbar switch 240 to provide processed tasks to another processing cluster 214 for further processing, or stores processed tasks in an L2 cache, local parallel processor memory, or system memory via memory crossbar switch 216. Pre-ROP 242 (pre-raster operation unit) is configured to receive data from graphics multiprocessor 234, direct data to ROP units, which may be associated with partitioning units as described herein (e.g., Figure 2A The partition units 220A-220N are located together. The preROP 242 unit can perform optimization for color mixing, organize pixel color data, and perform address translation.

[0084] It will be understood that the core architecture described herein is illustrative, and variations and modifications are possible. Any number of processing units (e.g., graphics multiprocessors 234, texture units 236, preROP 242, etc.) can be included within processing cluster 214. Furthermore, although only one processing cluster 214 is shown, parallel processing units as described herein can include any number of instances of processing cluster 214. Optionally, each processing cluster 214 can be configured to operate independently of other processing clusters 214 using separate and distinct processing units, L1 caches, etc.

[0085] Figure 2D An example of a graphics multiprocessor 234 is shown, wherein the graphics multiprocessor 234 is coupled to a pipeline manager 232 of a processing cluster 214. The graphics multiprocessor 234 has an execution pipeline including, but not limited to, an instruction cache 252, an instruction unit 254, an address mapping unit 256, a register file 258, one or more general-purpose graphics processing unit (GPGPU) cores 262, and one or more load / store units 266. The GPGPU cores 262 and the load / store units 266 are coupled to a cache memory 272 and a shared memory 270 via a memory and cache interconnect 268. The graphics multiprocessor 234 may additionally include tensor and / or ray tracing cores 263, which include hardware logic for accelerating matrix and / or ray tracing operations.

[0086] Instruction cache 252 receives a stream of instructions to be executed from pipeline manager 232. Instructions are cached in instruction cache 252 and dispatched to instruction unit 254 for execution. Instruction unit 254 can dispatch instructions as thread groups (e.g., threads), where each thread in the thread group is assigned to a different execution unit within GPGPU core 262. Instructions can access either the local, shared, or global address space by specifying an address within a unified address space. Address mapping unit 256 can be used to translate addresses in the unified address space into different memory addresses that can be accessed by load / store unit 266.

[0087] Register file 258 provides a set of registers for the functional units of graphics multiprocessor 234. Register file 258 provides temporary storage for operands on data paths connected to functional units of graphics multiprocessor 234 (e.g., GPGPU core 262, load / store unit 266). Register file 258 can be partitioned among each functional unit to allocate a dedicated portion of register file 258 to each functional unit. For example, register file 258 can be partitioned among different meridians executed by graphics multiprocessor 234.

[0088] Each GPGPU core 262 may include a floating-point unit (FPU) and / or an integer arithmetic logic unit (ALU) for executing instructions of the graphics multiprocessor 234. In some implementations, the GPGPU core 262 may include hardware logic that would otherwise reside within the tensor and / or ray tracing core 263. The GPGPU cores 262 may be architecturally similar or architecturally distinct. For example, and in one embodiment, a first portion of the GPGPU core 262 includes a single-precision FPU and an integer ALU, while a second portion of the GPGPU core includes a double-precision FPU. Optionally, the FPU may implement the IEEE 754-2008 standard for floating-point arithmetic or implement variable-precision floating-point arithmetic. The graphics multiprocessor 234 may additionally include one or more fixed-function units or special-function units for performing specific functions, such as copying rectangles or pixel blending operations. One or more of the GPGPU cores may also include fixed-function or special-function logic.

[0089] GPGPU core 262 may include SIMD logic capable of executing a single instruction on multiple data sets. Optionally, GPGPU core 262 may physically execute SIMD4, SIMD8, and SIMD16 instructions, and logically execute SIMD1, SIMD2, and SIMD32 instructions. SIMD instructions for the GPGPU core may be generated by a shader compiler at compile time, or automatically generated when executing a program written and compiled for a Single Program Multiple Data (SPMD) or SIMT architecture. Multiple threads of a program configured for a SIMT execution model can be executed via a single SIMD instruction. For example, and in one embodiment, eight SIMD threads performing the same or similar operations can be executed in parallel via a single SIMD8 logic unit.

[0090] The memory and cache interconnect 268 is an interconnect network that connects each functional unit in the graphics multiprocessor 234 to the register file 258 and to the shared memory 270. For example, the memory and cache interconnect 268 is a cross-switch interconnect that allows the load / store unit 266 to perform load and store operations between the shared memory 270 and the register file 258. The register file 258 can operate at the same frequency as the GPGPU core 262, so data transfer between the GPGPU core 262 and the register file 258 has very low latency. The shared memory 270 can be used to implement communication between threads executing on functional units within the graphics multiprocessor 234. The cache memory 272 can be used as a data cache, for example, for caching texture data transferred between functional units and texture units 236. The shared memory 270 can also be used as a cached managed program. In addition to automatically cached data stored in the cache memory 272, threads executing on the GPGPU core 262 can programmatically store data in the shared memory.

[0091] Figures 3A-3C The figure illustrates an additional graphics multiprocessor according to an embodiment. Figures 3A-3B The diagram illustrates the 325 and 350 graphics multiprocessors, which are related to... Figure 2C The graphics multiprocessor 234 is related to and may be used in place of one of those graphics multiprocessors. Therefore, the disclosure of any feature in combination with graphics multiprocessor 234 herein also discloses corresponding combinations with graphics multiprocessors(s) 325, 350, but is not limited thereto. Figure 3C The illustration shows a graphics processing unit (GPU) 380, which includes a dedicated set of graphics processing resources arranged as multi-core groups 365A-365N, which may correspond to graphics multiprocessors 325 and 350. The illustrated graphics multiprocessors 325 and 350 and multi-core groups 365A-365N may be streaming multiprocessors (SMs) capable of executing a large number of execution threads simultaneously.

[0092] Figure 3A The graphics multiprocessor 325 includes, relative to Figure 2DThe graphics multiprocessor 234 includes multiple additional instances of execution resource units. For example, the graphics multiprocessor 325 may include multiple instances of instruction units 332A-332B, register files 334A-334B, and texture units(s) 344A-344B. The graphics multiprocessor 325 also includes multiple sets of graphics or compute execution units (e.g., GPGPU cores 336A-336B, tensor cores 337A-337B, ray tracing cores 338A-338B) and multiple sets of load / store units 340A-340B. The execution resource units have a common instruction cache 330, a texture and / or data cache memory 342, and shared memory 346.

[0093] The components can communicate via interconnect fabric 327. Interconnect fabric 327 may include one or more crossbar switches to enable communication between the components of the graphics multiprocessor 325. Interconnect fabric 327 may be a separate, high-speed network layer on which each component of the graphics multiprocessor 325 is stacked. Components of the graphics multiprocessor 325 communicate with remote components via interconnect fabric 327. For example, GPGPU cores 336A-336B, 337A-337B, and 338A-338B can each communicate with shared memory 346 via interconnect fabric 327. Interconnect fabric 327 can arbitrate communication within the graphics multiprocessor 325 to ensure fair bandwidth allocation among components.

[0094] Figure 3B The graphics processor 350 includes multiple execution resource sets 356A-356D, among which, such as Figure 2D and Figure 3A As illustrated, each set of execution resources includes multiple instruction units, register files, GPGPU cores, and load memory units. Execution resources 356A-356D can work in conjunction with multiple texture units 360A-360D for texture operations, while sharing instruction cache 354 and shared memory 353. For example, execution resources 356A-356D can share instruction cache 354, shared memory 353, and multiple instances of texture and / or data cache memories 358A-358B. The components can be connected via a similar... Figure 3A The interconnection structure 327 communicates with the interconnection structure 352.

[0095] Those skilled in the art will understand that Figure 1 , Figures 2A-2D as well as Figures 3A-3BThe architecture described herein is descriptive and does not limit the scope of this embodiment. Therefore, the techniques described herein can be implemented on any properly configured processing unit, including but not limited to one or more mobile application processors, one or more desktop or server central processing units (CPUs) (including multi-core CPUs), and one or more parallel processing units (such as…). Figure 2A Parallel processing unit 202), and one or more graphics processors or dedicated processing units.

[0096] The parallel processors or GPGPUs described herein can be communicatively coupled to a host / processor core to accelerate graphics operations, machine learning operations, pattern analysis operations, and various general-purpose GPU (GPGPU) functions. The GPU can be communicatively coupled to the host processor / core via a bus or other interconnect (e.g., high-speed interconnects such as PCIe or NVLink). In other embodiments, the GPU may be integrated with the core on the same package or chip and communicatively coupled to the core via an internal processor bus / interconnect (i.e., within the package or chip). Regardless of the method used to connect the GPU, the processor core can assign work to the GPU in the form of a sequence of commands / instructions contained in a job descriptor. The GPU then uses dedicated circuitry / logic to efficiently process these commands / instructions.

[0097] Figure 3C The illustration shows a graphics processing unit (GPU) 380, which includes a set of dedicated graphics processing resources arranged as multi-core groups 365A-365N. While details of only a single multi-core group 365A are provided, it will be understood that other multi-core groups 365B-365N may be equipped with the same or a similar set of graphics processing resources. The details described with reference to multi-core groups 365A-365N also apply to any graphics multiprocessors 234, 325, 350 described herein.

[0098] As illustrated, the multi-core group 365A may include a set of graphics cores 370, a set of tensor cores 371, and a set of ray tracing cores 372. A scheduler / dispatcher 368 schedules and dispatches graphics threads for execution on the individual cores 370, 371, and 372. A set of register files 369 stores operand values ​​used by the cores 370, 371, and 372 when executing graphics threads. These register files may include, for example, integer registers for storing integer values, floating-point registers for storing floating-point values, vector registers for storing compressed data elements (integer and / or floating-point data elements), and operand matrix registers for storing tensor / matrix values. The operand matrix registers may be implemented as a combined set of vector registers.

[0099] One or more combined Level 1 (L1) cache and shared memory units 373 locally store graphics data such as texture data, vertex data, pixel data, ray data, bounding volume data, etc., within each multi-core group 365A. One or more texture units 374 can also be used to perform texture operations such as texture mapping and sampling. A Level 2 (L2) cache 375, shared by all multi-core groups 365A-365N or a subset of multi-core groups 365A-365N, stores graphics data and / or instructions for multiple concurrent graphics threads. As illustrated, the L2 cache 375 can be shared across multiple multi-core groups 365A-365N. One or more memory controllers 367 couple the GPU 380 to memory 366, which can be system memory (e.g., DRAM) and / or dedicated graphics memory (e.g., GDDR6 memory).

[0100] Input / output (I / O) circuitry 363 couples GPU 380 to one or more I / O devices 362, such as digital signal processors (DSPs), network controllers, or user input devices. On-chip interconnects are used to couple I / O devices 362 to GPU 380 and memory 366. One or more memory management units (IOMMUs) 364 of I / O circuitry 363 directly couple I / O devices 362 to system memory 366. Optionally, IOMMU 364 manages page tables for mapping virtual addresses to physical addresses in system memory 366. I / O devices 362, CPU(s) 361, and GPU(s) 380 may then share the same virtual address space.

[0101] In one implementation of IOMMU 364, IOMMU 364 supports virtualization. In this case, IOMMU 364 manages page tables for a first set of guest / graphics virtual addresses to guest / graphics physical addresses and page tables for a second set of guest / graphics physical addresses to (e.g., within system memory 366) system / host physical addresses. The base address of each of the first and second set of page tables can be stored in a control register and swapped out during context switching (e.g., so that the new context is given access to the page tables of the relevant set). Although not in Figure 3C As illustrated in the diagram, each of the cores 370, 371, 372 and / or the multi-core groups 365A-365N may include a Translation Backing Buffer (TLB) for caching guest virtual to guest physical translations, guest physical to host physical translations, and guest virtual to host physical translations.

[0102] The CPU 361, GPU 380, and I / O devices 362 may be integrated on a single semiconductor chip and / or chip package. The illustrated memory 366 may be integrated on the same chip or may be coupled to the memory controller 367 via an off-chip interface. In one implementation, memory 366 includes GDDR6 memory that shares the same virtual address space with other physical system-level memories, but the fundamental principles described herein are not limited to this particular implementation.

[0103] Tensor kernel 371 may include multiple execution units specifically designed to perform matrix operations, which are fundamental computational operations used for performing deep learning operations. For example, synchronous matrix multiplication operations can be used for neural network training and inference. Tensor kernel 371 can perform matrix processing using various operand precisions, including single-precision floating-point (e.g., 32-bit), half-precision floating-point (e.g., 16-bit), integer (16-bit), byte (8-bit), and half-byte (4-bit). For example, neural network implementations extract features from each rendered scene, potentially combining details from multiple frames to construct a high-quality final image.

[0104] In deep learning implementations, schedulable parallel matrix multiplication operations are performed on the tensor kernel 371. Training neural networks, in particular, requires a large number of matrix dot product operations. To handle the inner product formulation of N x N x N matrix multiplications, the tensor kernel 371 may include at least N dot product processing elements. Before matrix multiplication begins, a complete matrix is ​​loaded into the operand matrix register, and for each of the N loops, at least one column of the second matrix is ​​loaded. For each loop, there are N dot products to be processed.

[0105] Depending on the specific implementation, matrix elements can be stored with different precisions, including 16-bit words, 8-bit bytes (e.g., INT8), and 4-bit nibbles (e.g., INT4). Different precision modes can be specified for the tensor kernel 371 to ensure that the most efficient precision is used for different workloads (e.g., inference workloads that tolerate byte and nibble quantization).

[0106] Ray tracing core 372 can accelerate ray tracing operations for both real-time and non-real-time ray tracing implementations. Specifically, ray tracing core 372 may include ray traversal / intersection circuitry for performing ray traversal using a bounding volume hierarchy (BVH) and identifying intersections between rays enclosed within the BVH and primitives. Ray tracing core 372 may also include circuitry for performing depth testing and culling (e.g., using a Z-buffer or similar arrangement). In one implementation, ray tracing core 372 performs traversal and intersection operations consistent with the image denoising techniques described herein, at least partially of which can be performed on tensor core 371. For example, tensor core 371 may implement a deep learning neural network to perform denoising on frames generated by ray tracing core 372. However, CPU(s), graphics core 370, and / or ray tracing core 372 may also implement all or part of the denoising and / or deep learning algorithms.

[0107] Furthermore, as described above, a distributed approach to noise reduction can be employed, in which the GPU 380 is coupled to other computing devices via a network or high-speed interconnect. In this distributed approach, interconnected computing devices can share neural network learning / training data to improve the speed at which the entire system learns to perform noise reduction for different types of image frames and / or different graphics applications.

[0108] Ray tracing core 372 can handle all BVH traversals and / or ray-primitive intersections, thus saving graphics core 370 from being overloaded by thousands of instructions per ray. For example, each ray tracing core 372 includes a first set of specialized circuitry for performing bounding box tests (e.g., for traversal operations) and / or a second set of specialized circuitry for performing ray-triangle intersection tests (e.g., intersecting traversed rays). Thus, for example, a multi-core group 365A can simply initiate ray probing, and ray tracing core 372 independently performs ray traversal and intersection, returning hit data (e.g., hit, miss, multiple hits, etc.) to the thread context. While ray tracing core 370 performs traversal and intersection operations, other cores 371, 372 are freed up to perform other graphics or computational work.

[0109] Optionally, each ray tracing core 372 may include a traversal unit for performing BVH test operations and / or an intersection unit for performing ray-primitive intersection tests. The intersection unit generates "hit," "miss," or "multiple hits" responses, which are then provided to the appropriate threads. During traversal and intersection operations, execution resources of other cores (e.g., graphics core 370 and tensor core 371) are freed up to perform other forms of graphics work.

[0110] In one optional embodiment described below, a hybrid rasterization / ray tracing method is used, in which the operation is distributed between the graphics kernel 370 and the ray tracing kernel 372.

[0111] Raytracing core 372 (and / or other cores 370, 371) may include hardware support for ray tracing instruction sets such as Microsoft's DirectX Raytracing (DXR), which includes the DispatchRays command; and ray-generating shaders, recently hit shaders, any-hit shaders, and miss shaders, which enable the assignment of a unique set of shaders and textures to each object. Another ray tracing platform that can be supported by ray tracing core 372, graphics core 370, and tensor core 371 is Vulkan 1.1.85. However, it should be noted that the fundamental principles described herein are not limited to any particular ray tracing instruction set architecture ISA.

[0112] Generally, each core 372, 371, 370 may support a ray tracing instruction set including instructions / functions for one or more of the following: ray generation, recent hit, any hit, ray-primitive intersection, primitive-by-primitive and hierarchical bounding box construction, miss, visit, and exception. More specifically, preferred embodiments include ray tracing instructions for performing one or more of the following functions:

[0113] Light generation — It can execute ray generation instructions for each pixel, sample, or user-defined job assignment.

[0114] Recently hit — It can execute the nearest hit command to locate the nearest intersection point between light rays and primitives in the scene.

[0115] Any hit —Any hit instruction identifies multiple intersections between light rays and primitives within the scene, thereby potentially identifying new nearest intersections.

[0116] intersect The intersection command performs a ray-primitive intersection test and outputs the result.

[0117] Primitive bounding box construction — This instruction builds a bounding box around a given primitive or group of primitives (e.g., when building a new BVH or other accelerated data structure).

[0118] miss — Indicates all geometry within a specified area of ​​the scene that the light ray missed.

[0119] visit —Indicates the sub-containers that the light ray will traverse.

[0120] abnormal—Including various types of exception handlers (e.g., calls for various error conditions).

[0121] Technologies for GPU-to-host processor interconnects

[0122] Figure 4A The diagram illustrates several GPUs 410-413 (e.g., such as...). Figure 2A The parallel processor 200 shown herein is communicatively coupled to multiple multi-core processors 405-406 via high-speed links 440A-440D (e.g., bus, point-to-point interconnect, etc.). Depending on the implementation, the high-speed links 440A-440D may support communication throughput of 4GB / s, 30GB / s, 80GB / s, or higher. Various interconnect protocols can be used, including but not limited to PCIe 4.0 or 5.0 and NVLink 2.0. However, the basic principles described herein are not limited to any particular communication protocol or throughput.

[0123] Two or more GPUs 410-413 can be interconnected via high-speed links 442A-442B, which can be implemented using the same or different protocols / links as those used for high-speed links 440A-440D. Similarly, two or more multi-core processors 405-406 can be connected via high-speed link 443, which can be a symmetric multiprocessor (SMP) bus operating at speeds of 20GB / s, 30GB / s, 120GB / s, or higher. Alternatively, Figure 4A All communication between the various system components shown can be accomplished using the same protocol / link (e.g., via a common interconnect structure). As mentioned, however, the basic principles described herein are not limited to any particular type of interconnect technology.

[0124] Each multi-core processor 405-406 may be communicatively coupled to processor memories 401-402 via memory interconnects 430A-430B, and each GPU 410-413 may be communicatively coupled to GPU memories 420-423 via GPU memory interconnects 450A-450D. Memory interconnects 430A-430B and 450A-450D may utilize the same or different memory access technologies. By way of example and not limitation, processor memories 401-402 and GPU memories 420-423 may be volatile memories such as dynamic random access memory (DRAM) (including stacked DRAM), graphics DDR SDRAM (GDDR) (e.g., GDDR5, GDDR6), or high-bandwidth memory (HBM), and / or may be non-volatile memories such as 3D Xpoint or Nano-Ram. For example, one portion of these memories may be volatile memory and another portion may be non-volatile memory (e.g., using a two-level memory (2LM) hierarchy).

[0125] As described below, although each processor 405-406 and GPU 410-413 can be physically coupled to specific memories 401-402 and 420-423 respectively, a unified memory architecture can be implemented in which the same virtual system address space (also referred to as the "effective address" space) is distributed across all the various physical memories. For example, processor memories 401-402 can each include 64GB of system memory address space, and GPU memories 420-423 can each include 32GB of system memory address space (in this example, a total of 256GB of addressable memory is obtained).

[0126] Figure 4B The diagram illustrates additional optional details of the interconnect between the multi-core processor 407 and the graphics acceleration module 446. The graphics acceleration module 446 may include one or more GPU chips integrated on a line card coupled to the processor 407 via a high-speed link 440. Alternatively, the graphics acceleration module 446 may be integrated on the same package or chip as the processor 407.

[0127] The illustrated processor 407 includes multiple cores 460A-460D, each having translation-backup buffers 461A-461D and one or more caches 462A-462D. Cores may include various other components for executing instructions and processing data; these components are not illustrated to avoid obscuring the fundamental principles of the components described herein (e.g., instruction fetch units, branch prediction units, decoders, execution units, reordering buffers, etc.). Caches 462A-462D may include a Level 1 (L1) cache and a Level 2 (L2) cache. Additionally, one or more shared caches 456 may be included in the cache hierarchy and shared by the core set 460A-460D. For example, one embodiment of processor 407 includes 24 cores, each with its own L1 cache, twelve shared L2 caches, and twelve shared L3 caches. In this embodiment, one of the L2 and L3 caches is shared by two adjacent cores. The processor 407 and graphics accelerator integrated module 446 are connected to the system memory 441, which may include processor memories 401-402.

[0128] Data and instructions stored in the various caches 462A-462D, 456 and system memory 441 are maintained in consistency via inter-core communication through the consistency bus 464. For example, each cache may have associated cache consistency logic / circuit to communicate via the consistency bus 464 in response to a detected read or write to a particular cache line. In one implementation, a cache snooping protocol is implemented via the consistency bus 464 to listen for cache accesses. Cache snooping / consistency techniques are well understood by those skilled in the art and will not be described in detail herein to avoid obscuring the basic principles described herein.

[0129] A proxy circuit 425 may be provided, which communicatively couples the graphics acceleration module 446 to the coherence bus 464, thereby allowing the graphics acceleration module 446 to participate in the cache coherence protocol as a peer of the core. Specifically, interface 435 provides connectivity to the proxy circuit 425 via a high-speed link 440 (e.g., PCIe bus, NVLink, etc.), and interface 437 connects the graphics acceleration module 446 to the high-speed link 440.

[0130] In one implementation, the accelerator integrated circuit 436 provides cache management, memory access, context management, and interrupt management services for multiple graphics processing engines 431, 432, and N of the graphics acceleration module 446. Each of the graphics processing engines 431, 432, and N may comprise a separate graphics processing unit (GPU). Alternatively, the graphics processing engines 431, 432, and N may comprise different types of graphics processing engines within the GPU, such as graphics execution units, media processing engines (e.g., video encoders / decoders), samplers, and blit engines. In other words, the graphics acceleration module may be a GPU with multiple graphics processing engines 431-432, and N, or the graphics processing engines 431-432, and N may be separate GPUs integrated into a common package, line card, or chip.

[0131] Accelerator integrated circuit 436 may include a memory management unit (MMU) 439 for performing various memory management functions such as virtual-to-physical memory translation (also known as effective-to-real memory translation) and memory access protocols for accessing system memory 441. MMU 439 may also include a translation lookup buffer (TLB) (not shown) for caching virtual / effective-to-physical / real address translations. In one implementation, cache 438 stores commands and data for efficient access by graphics processing engines 431-432, N. Data stored in cache 438 and graphics memories 433-434, M is kept consistent with core caches 462A-462D, 456 and system memory 441. As mentioned, this can be accomplished via proxy circuit 425, which represents cache 438 and memories 433-434, M in participating in cache coherency mechanisms (e.g., sending updates to cache 438 related to modifications / accesses to cache lines on processor caches 462A-462D, 456 and receiving updates from cache 438).

[0132] Register set 445 stores context data for threads executed by graphics processing engines 431-432, N, and context management circuitry 448 manages these thread contexts. For example, context management circuitry 448 may perform save and restore operations during context switching to save and restore the context of each thread (e.g., where a first thread is saved and a second thread is saved so that the second thread can be executed by the graphics processing engine). For example, during context switching, context management circuitry 448 may store the current register value in a designated area of ​​memory (e.g., identified by a context pointer). Upon returning to that context, it may subsequently restore the register value. Interrupt management circuitry 447 may, for example, receive and process interrupts received from system devices.

[0133] In one implementation, the MMU 439 translates the virtual / effective address from the graphics processing engine 431 into a physical / actual address in system memory 441. Optionally, the accelerator integrated circuit 436 supports multiple (e.g., 4, 8, 16) graphics accelerator modules 446 and / or other accelerator devices. The graphics accelerator modules 446 may be dedicated to a single application executing on processor 407, or may be shared among multiple applications. Optionally, a virtualized graphics execution environment is provided, wherein the resources of graphics processing engines 431-432, N are shared with multiple applications or virtual machines (VMs). These resources may be subdivided into "slices," which are allocated to different VMs and / or applications based on processing requirements and priorities associated with the VMs and / or applications.

[0134] Therefore, the accelerator integrated circuit 436 acts as a bridge to the system for the graphics acceleration module 446 and provides address translation and system memory caching services. In one embodiment, to facilitate bridging functionality, the accelerator integrated circuit 436 may also include shared I / O 497 (e.g., PCIe, USB) and hardware for enabling system control over voltage, clock timing, performance, thermal, and safety. The shared I / O 497 may utilize a separate physical connection or may span a high-speed link 440. Additionally, the accelerator integrated circuit 436 may provide virtualization facilities for the host processor to manage virtualization of the graphics processing engine, interrupts, and memory management.

[0135] Because the hardware resources of the graphics processing engines 431-432, N are explicitly mapped to the actual address space viewed by the host processor 407, any host processor can directly address these resources using valid address values. An optional feature of the accelerator integrated circuit 436 is the physical separation of the graphics processing engines 431-432, N, so that they appear as independent units to the system.

[0136] One or more graphics memories 433-434, M may be coupled to each of the graphics processing engines 431-432, N, respectively. Graphics memories 433-434, M store instructions and data processed by each of the graphics processing engines 431-432, N. Graphics memories 433-434, M may be volatile memories such as DRAM (including stacked DRAM), GDDR memory (e.g., GDDR5, GDDR6), or HBM, and / or may be non-volatile memories such as 3D XPoint or Nano-RAM.

[0137] To reduce data traffic on the high-speed link 440, a biasing technique can be used to ensure that the data stored in the graphics memories 433-434, M is the data that will be used most frequently by the graphics processing engines 431-432, N and preferably not used (or at least infrequently used) by the cores 460A-460D. Similarly, the biasing mechanism attempts to keep the data needed by the cores (and preferably not the graphics processing engines 431-432, N) within the caches 462A-462D, 456 of these cores and the system memory 411.

[0138] according to Figure 4C In the variant shown, accelerator integrated circuit 436 is integrated within processor 407. Graphics processing engines 431-432, N communicate directly with accelerator integrated circuit 436 via high-speed link 440 through interfaces 437 and 435 (which can also utilize any form of bus or interface protocol). Figure 4B The operations described are the same, but given the close proximity of the accelerator integrated circuit 436 to the coherence bus 464 and caches 462A-462D, 456, it can potentially perform operations at higher throughput.

[0139] The described embodiments can support different programming models, including a dedicated process programming model (without graphics acceleration module virtualization) and a shared programming model (with virtualization). The latter may include a programming model controlled by accelerator integrated circuit 436 and a programming model controlled by graphics acceleration module 446.

[0140] In a dedicated process model embodiment, graphics processing engines 431-432, N can be dedicated to a single application or process within a single operating system. This single application can aggregate requests from other applications to graphics engines 431-432, N, thereby providing virtualization within a VM / partition.

[0141] In a dedicated process programming model, the graphics processing engines 431-432,N can be shared by multiple VM / application partitions. This shared model requires the hypervisor to virtualize the graphics processing engines 431-432,N to allow access by each operating system. For single-partition systems without a hypervisor, the graphics processing engines 431-432,N are owned by the operating system. In both cases, the operating system can virtualize the graphics processing engines 431-432,N to provide access to each process or application.

[0142] For the shared programming model, the graphics acceleration module 446 or the individual graphics processing engines 431-432, N use process handles to select process elements. Process elements may be stored in system memory 441 and may be addressable using the effective address to actual address translation techniques described herein. The process handle may be an implementation-specific value provided to the host process when registering its context with the graphics processing engines 431-432, N (i.e., invoking system software to add process elements to the process element list). The lower 16 bits of the process handle may be the offset of the process element within the process element list.

[0143] Figure 4D The illustration shows an exemplary accelerator integration slice 490. As used herein, a “slice” refers to a designated portion of the processing resources of the accelerator integrated circuit 436. The application-effective address space 482 within system memory 441 stores process elements 483. Process elements 483 may be stored in response to a GPU call 481 from an application 480 executing on processor 407. Process elements 483 contain the process state corresponding to the application 480. A job descriptor (WD) 484 contained in process elements 483 may be a single job requested by the application, or may contain a pointer to a job queue. In the latter case, WD 484 is a pointer to a job request queue in the application's address space 482.

[0144] The graphics acceleration module 446 and / or the various graphics processing engines 431-432, N can be shared by all or a subset of the processes in the system. For example, the techniques described herein can be used to establish the infrastructure for process states and sending WD 484 to the graphics acceleration module 446 to begin operations in a virtualized environment.

[0145] In one implementation, a dedicated process programming model is implementation-specific. In this model, a single process owns either the graphics acceleration module 446 or a separate graphics processing engine 431. Because the graphics acceleration module 446 is owned by a single process, when assigning the graphics acceleration module 446, the hypervisor initializes the accelerator integrated circuit 436 for the owned partition, and the operating system initializes the accelerator integrated circuit 436 for the owned process.

[0146] In operation, the WD fetch unit 491 in the accelerator integrated slice 490 fetches the next WD 484, which includes instructions for the work to be completed by one of the graphics processing engines in the graphics acceleration module 446. As illustrated, data from the WD 484 may be stored in register 445 and used by the MMU 439, interrupt management circuitry 447, and / or context management circuitry 448. For example, the MMU 439 may include segment / page walkthrough circuitry for accessing segment tables / page tables 486 within the OS virtual address space 485. The interrupt management circuitry 447 may handle interrupt events 492 received from the graphics acceleration module 446. When a graphics operation is performed, the effective address 493 generated by the graphics processing engines 431-432, N is translated into an actual address by the MMU 439.

[0147] The same set of registers 445 may be copied for each graphics processing engine 431-432, N and / or graphics acceleration module 446 and may be initialized by the hypervisor or operating system. Each of these copied registers may be included in the accelerator integration slice 490. Exemplary registers that may be initialized by the hypervisor are shown in Table 1.

[0148] Table 1 - Registers initialized by the management program

[0149] 2 Process region pointers scheduled by physical address (RA) 3 Permission mask overwrite register 4 Interruption vector table entry offset 5 Interrupt vector table entry limit 6 Status Register 7 Logical partition ID 8 The Real Address (RA) management accelerator utilizes record pointers 9 Storage description register

[0150] Table 2 shows exemplary registers that can be initialized by the operating system.

[0151] Table 2 - Registers for Operating System Initialization

[0152]

[0153]

[0154] Each WD 484 can be specific to a particular graphics acceleration module 446 and / or graphics processing engine 431-432, N. It contains all the information required by the graphics processing engine 431-432, N to perform its work, or it can be a pointer to the memory location of the command queue that the application has set up to perform the work to be done.

[0155] Figure 4E The diagram illustrates additional optional details of the shared model. It includes a hypervisor physical address space 498 where a list of process elements 499 is stored. The hypervisor physical address space 498 is accessible via a hypervisor 496, which is used by the graphics acceleration module engine virtualization of the operating system 495.

[0156] The shared programming model allows all or a subset of processes from all or a subset of partitions in the system to use the graphics acceleration module 446. Two programming models exist in which the graphics acceleration module 446 is shared by multiple processes and partitions: time-division sharing and graphics-oriented sharing.

[0157] In this model, the hypervisor 496 owns the graphics acceleration module 446 and makes its functionality available to all operating systems 495. To enable the graphics acceleration module 446 to support virtualization performed by the hypervisor 496, the graphics acceleration module 446 must meet the following requirements: 1) Application job requests must be autonomous (i.e., no state maintenance is required between jobs), or the graphics acceleration module 446 must provide context saving and restoration mechanisms. 2) The graphics acceleration module 446 must guarantee the completion of application job requests within a specified timeframe, including any translation errors, or the graphics acceleration module 446 must provide the ability to preemptively process jobs. 3) When operating under a directed shared programming model, fairness of the graphics acceleration module 446 among processes must be guaranteed.

[0158] For the shared model, application 480 may be required to make an operating system call 495 using the graphics acceleration module 446 type, working descriptor (WD), authority mask register (AMR) value, and context save / restore region pointer (CSRP). The graphics acceleration module 446 type describes the target acceleration function of the system call. The graphics acceleration module 446 type can be a system-specific value. The WD is formatted specifically for the graphics acceleration module 446, and the WD can take the form of a graphics acceleration module 446 command, a valid address pointer to a user-defined structure, a valid address pointer to a command queue, or any other data structure describing the work to be performed by the graphics acceleration module 446. In one embodiment, the AMR value is the AMR state for the current process. The value passed to the operating system is similar to the application setting the AMR. If the accelerator integrated circuit 436 and the graphics acceleration module 446 implementation do not support the User Authority Mask Override Register (UAMOR), the operating system may apply the current UAMOR value to the AMR value before passing the AMR in the hypervisor call. The hypervisor 496 may optionally apply the Current Privilege Mask Override Register (AMOR) value before placing the AMR into the process element 483. The CSRP may be a register 445 containing the effective address of a region in the application address space 482 used by the graphics acceleration module 446 to save and restore the context state. This pointer is optional if saving the state between jobs or during job preemption is not required. The context save / restore region may be fixed system memory.

[0159] Upon receiving a system call, the operating system 495 verifies that application 480 has been registered and has been granted permission to use the graphics acceleration module 446. The operating system 495 then uses the information shown in Table 3 to invoke the hypervisor 496.

[0160] Table 3 - OS calls to the management program

[0161] 2 Authority Mask Register (AMR) value (potentially masked). 3 Valid Address (EA) Context Save / Restore Region Pointer (CSRP) 4 Process ID (PID) and optional thread ID (TID) 5 Virtual address (VA) accelerators utilize record pointers (AURP). 6 Virtual address of the segment table pointer (SSTP) 7 Logical Interrupt Service Number (LISN)

[0162] Upon receiving a call from the hypervisor, hypervisor 496 verifies that operating system 495 is registered and has been granted permission to use graphics acceleration module 446. Hypervisor 496 then places process element 483 in a linked list of process elements corresponding to the type of graphics acceleration module 446. Process elements may include the information shown in Table 4.

[0163] Table 4 - Process Element Information

[0164]

[0165]

[0166] The management program can initialize the 490 register 445 of the multiple accelerator integrated slice.

[0167] like Figure 4F As illustrated, in one optional implementation, a unified memory addressable via a common virtual memory address space is employed, which is used to access physical processor memories 401-402 and GPU memories 420-423. In this implementation, operations performed on GPUs 410-413 utilize the same virtual / effective memory address space to access processor memories 401-402 and vice versa, thereby simplifying programmability. A first portion of the virtual / effective address space can be allocated to processor memory 401, a second portion to processor memory 402, a third portion to GPU memory 420, and so on. The entire virtual / effective memory space (sometimes referred to as the effective address space) can thus be distributed across each of processor memories 401-402 and GPU memories 420-423, allowing any processor or GPU to access any physical memory using virtual addresses mapped to that memory.

[0168] One or more of the MMUs 439A-439E may be provided with bias / coherence management circuitry 494A-494E, which ensures cache coherence between the host processor's (e.g., 405) cache and the GPU's (410-413) cache, and implements biasing techniques that indicate the physical memory in which certain types of data should be stored. Although in Figure 4F The middle figure shows several instances of bias / coherence management circuits 494A-494E, but the bias / coherence circuits may be implemented within the MMU of one or more host processors 405 and / or within the accelerator integrated circuit 436.

[0169] The GPU-attached memories 420-423 can be mapped as part of system memory and accessed using shared virtual memory (SVM) technology, but without suffering the typical performance drawbacks associated with full system cache coherence. The ability of the GPU-attached memories 420-423 to be accessed as system memory without the heavy overhead of cache coherence provides a beneficial operating environment for GPU migration. This arrangement allows the host processor 405 software to set operands and access computation results without the overhead of traditional I / O DMA data copying. Such traditional copying involves driver calls, interrupts, and memory-mapped I / O (MMIO) accesses, all of which are inefficient compared to simple memory access. Meanwhile, the ability to access the GPU-attached memories 420-423 without cache coherence overhead can be critical to the execution time of the migrated computation. For example, in scenarios with heavy streaming write memory traffic, cache coherence overhead can significantly reduce the effective write bandwidth seen by the GPUs 410-413. The efficiency of operand setting, the efficiency of result access, and the efficiency of GPU computation all play a role in determining the effectiveness of GPU migration.

[0170] The choice between GPU bias and host processor bias can be driven by a bias tracer data structure. For example, a bias table can be used, which can be a page-granular structure (i.e., controlled at the memory page level) comprising one or two bits of each memory page attached to the GPU. The bias table can be implemented in a stolen memory range of one or more GPU-attached memories 420-423, with or without a bias cache in GPUs 410-413 (e.g., for caching frequently / recently used bias table entries). Alternatively, the entire bias table can be maintained within the GPU.

[0171] In one implementation, prior to the actual access to the GPU memory, the bias table entries associated with each access to the GPU-attached memories 420-423 are accessed, resulting in the following operations: First, local requests from GPUs 410-413 that find their pages in the GPU bias are directly forwarded to the corresponding GPU memories 420-423. Local requests from GPUs that find their pages in the host bias are forwarded to processor 405 (e.g., via a high-speed link, as discussed above). Optionally, requests from processor 405 that find the requested page in the host processor bias are performed as normal memory reads. Alternatively, requests involving GPU bias pages may be forwarded to GPUs 410-413. If the GPU is not currently using the page, the GPU may subsequently transfer the page to the host processor bias.

[0172] The page bias state can be changed through software-based mechanisms, hardware-assisted software-based mechanisms, or, for a limited set of cases, purely hardware-based mechanisms.

[0173] One mechanism for changing the bias state employs API calls (e.g., OpenCL), which in turn invoke the GPU's device driver. This device driver then sends a message to the GPU (or enqueues a command descriptor) instructing it to change the bias state and perform a cache dump clearing operation on the host for some transitions. The cache dump clearing operation is necessary for transitions from host processor 405 bias to GPU bias, but not for the reverse transition.

[0174] Cache coherence can be maintained by temporarily rendering GPU-biased pages that cannot be cached by the host processor 405. To access these pages, the processor 405 may request access from the GPU 410, which, depending on the implementation, may or may not immediately grant access. Therefore, to reduce communication between the host processor 405 and the GPU 410, it is beneficial to ensure that the GPU-biased pages are those requested by the GPU, not those requested by the host processor 405, and vice versa.

[0175] Graphics processing pipeline

[0176] Figure 5 The diagram illustrates a graphics processing pipeline 500. Graphics multiprocessors (such as...) Figure 2D The graphics multiprocessor 234 in the middle Figure 3A Graphics multiprocessor 325, Figure 3B The graphics multiprocessor 350 can implement the illustrated graphics processing pipeline 500. The graphics multiprocessor can be included within a parallel processing subsystem as described herein, such as... Figure 2A In one embodiment, the parallel processor 200 can be coupled with... Figure 1 This relates to (multiple) parallel processors 112, and can replace one of those parallel processors being used. Various parallel processing systems can be achieved through the parallel processing units described herein (e.g., Figure 2A One or more instances of parallel processing units 202) implement the graphics processing pipeline 500. For example, shader units (e.g., Figure 2C The graphics multiprocessor 234 can be configured to perform the functions of one or more of the vertex processing unit 504, the tessellation control processing unit 508, the tessellation evaluation processing unit 512, the geometry processing unit 516, and the fragment / pixel processing unit 524. The functions of the data assembler 502, primitive assemblers 506, 514, 518, the tessellation unit 510, the rasterizer 522, and the raster operation unit 526 can also be handled by a processing cluster (e.g., Figure 2A Other processing engines and corresponding partition units (e.g., within the processing cluster 214) Figure 2A The graphics processing pipeline 500 is executed by partitioning units 220A-220N. The graphics processing pipeline 500 can also be implemented using dedicated processing units for one or more functions. It is also possible for one or more portions of the graphics processing pipeline 500 to be executed by parallel processing logic within a general-purpose processor (e.g., CPU). Optionally, one or more portions of the graphics processing pipeline 500 can access on-chip memory (e.g., such as memory interface 528) via memory interface 528. Figure 2A The parallel processor memory 222 in the memory interface 528 can be a parallel processor memory 222 in the memory interface 528. Figure 2A An example of memory interface 218. The graphics processor pipeline 500 can also be accessed via, for example... Figure 3C It is achieved through the multi-core 365A.

[0177] Data assembler 502 is a processing unit that collects vertex data from surfaces and primitives. Data assembler 502 then outputs vertex data, including vertex attributes, to vertex processing unit 504. Vertex processing unit 504 is a programmable execution unit that executes vertex shader programs to illuminate and transform vertex data as specified by the vertex shader programs. Vertex processing unit 504 reads data stored in cache, local, or system memory for use in processing vertex data and can be programmed to transform vertex data from object-based coordinate representation to world space coordinate space or normalized device coordinate space.

[0178] The first instance of the primitive assembler 506 receives vertex attributes from the vertex processing unit 504. The primitive assembler 506 reads the stored vertex attributes as needed and constructs graphics primitives for processing by the tessellation control processing unit 508. Graphics primitives include triangles, line segments, points, patches, etc., supported by various graphics processing application programming interfaces (APIs).

[0179] The tessellation control processing unit 508 treats input vertices as control points for a geometric patch. It transforms the control points from the input representation of the patch (e.g., the patch's basis) into a representation suitable for use in the surface evaluation performed by the tessellation evaluation processing unit 512. The tessellation control processing unit 508 can also calculate tessellation factors for the edges of the geometric patch. The tessellation factor is applied to a single edge and quantifies the view-dependent level of detail associated with that edge. The tessellation unit 510 is configured to receive the tessellation factors for the edges of the patch and to subdivide the patch surface into multiple geometric primitives (such as line, triangle, or quadrilateral primitives), which are then passed to the tessellation evaluation processing unit 512. The tessellation evaluation processing unit 512 operates on the parameterized coordinates of the subdivided patch to generate a surface representation and vertex attributes associated with each vertex of the geometric primitive.

[0180] A second instance of the primitive assembler 514 receives vertex attributes from the tessellation evaluation processing unit 512, reads the stored vertex attributes as needed, and constructs graphical primitives for processing by the geometry processing unit 516. The geometry processing unit 516 is a programmable execution unit that executes a geometry shader program to transform the graphical primitives received from the primitive assembler 514 as specified by the geometry shader program. The geometry processing unit 516 can be programmed to subdivide the graphical primitives into one or more new graphical primitives and calculate parameters used to rasterize these new graphical primitives.

[0181] The geometry processing unit 516 can add or delete elements in the geometry flow. The geometry processing unit 516 outputs parameters and vertices specifying new graphic primitives to the primitive assembler 518. The primitive assembler 518 receives parameters and vertices from the geometry processing unit 516 and constructs graphic primitives for processing by the viewport scaling, culling, and clipping unit 520. The geometry processing unit 516 reads data stored in the parallel processor memory or system memory for use when processing geometric data. The viewport scaling, culling, and clipping unit 520 performs clipping, culling, and viewport scaling, and outputs the processed graphic primitives to the rasterizer 522.

[0182] Rasterizer 522 can perform depth culling and other depth-based optimizations. Rasterizer 522 also performs scan transformations on new graphics primitives to generate fragments and outputs those fragments and associated overlay data to fragment / pixel processing unit 524. Fragment / pixel processing unit 524 is a programmable execution unit configured to execute fragment shader programs or pixel shader programs. Fragment / pixel processing unit 524 transforms fragments or pixels received from rasterizer 522 according to the fragment or pixel shader program specified. For example, fragment / pixel processing unit 524 can be programmed to perform operations including but not limited to texture mapping, shading, blending, texture correction, and perspective correction to produce shaded fragments or pixels output to raster operation unit 526. Fragment / pixel processing unit 524 can read data stored in parallel processor memory or system memory for use when processing fragment data. Fragment or pixel shader programs can be configured to shade at the sample, pixel, slice, or other granularity depending on the sampling rate configured for the processing unit.

[0183] Raster operation unit 526 is a processing unit that performs raster operations and outputs pixel data to be stored in graphics memory (e.g., ...). Figure 2A The parallel processor memory 222 and / or such Figure 1 The raster operation unit 526 stores processed graphic data in system memory 104, which is to be displayed on one or more display devices 110 or used for further processing by one or more processors 102 or parallel processors 112. These raster operations include, but are not limited to, stencil printing, z-testing, blending, etc. The raster operation unit 526 can be configured to compress z or color data written to memory and decompress z or color data read from memory.

[0184] Machine Learning Overview

[0185] The architecture described above can be applied to perform training and inference operations using machine learning models. Machine learning has already achieved success in solving many kinds of tasks. The computations generated when training and using machine learning algorithms (e.g., neural networks) are naturally suited for efficient parallel implementation. Therefore, parallel processors such as general-purpose graphics processing units (GPGPUs) have played an important role in the practical implementation of deep neural networks. Parallel graphics processors with a single-instruction, multi-threaded (SIMT) architecture are designed to maximize the amount of parallel processing in the graphics pipeline. In the SIMT architecture, groups of parallel threads attempt to execute program instructions together synchronously as frequently as possible to improve processing efficiency. The efficiency provided by the parallel machine learning algorithm implementation allows for the use of high-capacity networks and makes it possible to train those networks on large datasets.

[0186] Machine learning algorithms are algorithms that can learn from datasets. For example, machine learning algorithms can be designed to model high-level abstractions within a dataset. For instance, image recognition algorithms can be used to determine which category a given input belongs to; given input, regression algorithms can output numerical values; and pattern recognition algorithms can be used to generate converted text or perform text-to-speech and / or speech recognition.

[0187] An exemplary type of machine learning algorithm is a neural network. Many types of neural networks exist; a simple type is the feedforward network. A feedforward network can be implemented as a non-cyclic graph in which nodes are arranged in layers. Typically, a feedforward network topology includes an input layer and an output layer separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation useful for generating the output in the output layer. Network nodes are fully connected to nodes in adjacent layers via edges, but there are no edges between nodes within a single layer. Data received at the nodes in the input layer of the feedforward network is propagated (i.e., “feedforward”) to the nodes in the output layer via an activation function that computes the state of nodes in each successive layer of the network based on coefficients (“weights”) associated with each of the edges connecting these layers. Depending on the specific model represented by the algorithm being executed, the output from the neural network algorithm can take various forms.

[0188] Before a machine learning algorithm can be used to model a specific problem, it is trained using a training dataset. Training a neural network involves: choosing a network topology; using a set of training data representing the problem being modeled by the network; and adjusting the weights until the network model performs with minimum error for all instances in the training dataset. For example, during supervised learning training of a neural network, the output generated by the network in response to an input representing an instance in the training dataset is compared to the “correct” labeled output for that instance, an error signal representing the difference between the output and the labeled output is calculated, and as the error signal is propagated back through the layers of the network, the weights associated with the connections are adjusted to minimize that error. The network is considered “trained” when the error of each output generated based on instances in the training dataset is minimized.

[0189] The accuracy of machine learning algorithms is significantly affected by the quality of the dataset used to train them. The training process can be computationally intensive and may take a considerable amount of time on a conventional general-purpose processor. Therefore, parallel processing hardware is used to train many types of machine learning algorithms. This is particularly useful for optimizing the training of neural networks, as the computations performed when adjusting the coefficients in a neural network are naturally suited to parallel implementation. Specifically, many machine learning algorithms and software applications have been adapted to utilize the parallel processing hardware within general-purpose graphics processing devices.

[0190] Figure 6 This is a generalized diagram of the machine learning software stack 600. Machine learning application 602 can be configured to train neural networks using training datasets or to achieve machine intelligence using trained deep neural networks. Machine learning application 602 may include training and inference functions for neural networks and / or specialized software that can be used to train neural networks prior to deployment. Machine learning application 602 can achieve any type of machine intelligence, including but not limited to: image recognition, mapping and localization, autonomous navigation, speech synthesis, medical imaging, or language translation.

[0191] Hardware acceleration for machine learning applications 604 can be enabled via machine learning framework 602. Machine learning framework 604 provides a library of machine learning primitives. Machine learning primitives are fundamental operations frequently performed by machine learning algorithms. Without machine learning framework 604, developers of machine learning algorithms would need to create and optimize the main computational logic associated with their algorithms, and then re-optimize that logic when developing new parallel processors. Instead, machine learning applications can be configured to use the primitives provided by machine learning framework 604 to perform the necessary computations. Exemplary primitives include tensor convolution, activation functions, and pooling, which are computational operations performed when training convolutional neural networks (CNNs). Machine learning framework 604 can also provide primitives to implement basic linear algebra subroutines performed by many machine learning algorithms, such as matrix and vector operations.

[0192] The machine learning framework 604 can process input data received from the machine learning application 602 and generate appropriate input for the computing framework 606. The computing framework 606 can abstract the low-level instructions provided to the GPGPU driver 608, enabling the machine learning framework 604 to utilize hardware acceleration via the GPGPU hardware 610 without the machine learning framework 604 being very familiar with the architecture of the GPGPU hardware 610. Furthermore, the computing framework 606 can enable hardware acceleration for the machine learning framework 604 across various types and generations of GPGPU hardware 610.

[0193] GPGPU Machine Learning Acceleration

[0194] Figure 7 The diagram illustrates a general-purpose graphics processing unit 700, which can be... Figure 2A Parallel processor 200 or Figure 1The system includes (multiple) parallel processors 112. The General Purpose Processing Unit (GPGPU) 700 can be configured to be particularly efficient at processing computational workloads associated with training deep neural networks. Furthermore, the GPGPU 700 can be directly linked to other instances of GPGPUs to create multi-GPU clusters, thereby improving the training speed, especially for deep neural networks.

[0195] The GPGPU 700 includes a host interface 702 for enabling connectivity with a host processor. The host interface 702 may be a PCI Express interface. However, the host interface may also be a vendor-specific communication interface or communication structure. The GPGPU 700 receives commands from the host processor and uses a global scheduler 704 to distribute the execution threads associated with those commands to sets of processing clusters 706A-706H. Processing clusters 706A-706H share a cache memory 708. The cache memory 708 may act as a higher-level cache for cache memories within the processing clusters 706A-706H. The illustrated processing clusters 706A-706H may be connected to… Figure 2A The processing clusters 214A-214H correspond to this.

[0196] The GPGPU 700 includes memory 714A-714B coupled to processing clusters 706A-706H via a set of memory controllers 712A-712B. Memory 714A-714B may include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. Memory 714A-714B may also include 3D stacked memory, including but not limited to high bandwidth memory (HBM).

[0197] Each processing cluster in the 706A-706H may include a collection of graphics multiprocessors, such as... Figure 2D Graphics multiprocessor 234, Figure 3A Graphics multiprocessor 325, Figure 3B The graphics multiprocessor 350, or may include Figure 3C The 365A-365N series of multi-core processors in these computing clusters. These computing clusters' graphics multiprocessors include various types of integer logic units (ILUs) and floating-point logic units (Floating-Point ILUs) that can perform computational operations suitable for machine learning within a certain precision range. For example, at least a subset of the floating-point units in each of the 706A-706H processing clusters can be configured to perform 16-bit or 32-bit floating-point operations, while different subsets of the floating-point units can be configured to perform 64-bit floating-point operations.

[0198] Multiple instances of GPGPU 700 can be configured to operate as a computing cluster. The communication mechanisms used by the computing cluster for synchronization and data exchange vary across embodiments. For example, multiple instances of GPGPU 700 communicate via host interface 702. In one embodiment, GPGPU 700 includes an I / O hub 709 that couples GPGPU 700 to GPU link 710, which enables direct connections to other instances of the GPGPU. GPU link 710 may be coupled to a dedicated GPU-GPU bridge that enables communication and synchronization between multiple instances of GPGPU 700. Optionally, GPU link 710 is coupled to a high-speed interconnect to transmit and receive data to other GPGPUs or parallel processors. Multiple instances of GPGPU 700 may reside in a separate data processing system and communicate via a network device accessible via host interface 702. Attached to or replacing host interface 702, GPU link 710 may be configured to enable connections to a host processor.

[0199] While the illustrated configuration of the GPGPU 700 can be configured to train a neural network, alternative configurations of the GPGPU 700 can be configured for deployment within high-performance or low-power inference platforms. In the inference configuration, the GPGPU 700 includes fewer processing clusters in the processing clusters 706A-706H compared to the training configuration. Additionally, the memory technology associated with the memories 714A-714B can differ between the inference and training configurations. In one embodiment, the inference configuration of the GPGPU 700 can support inference-specific instructions. For example, the inference configuration can provide support for one or more 8-bit integer dot product instructions, which are typically used during inference operations for a deployed neural network.

[0200] Figure 8 The diagram illustrates a multi-GPU computing system 800. The multi-GPU computing system 800 may include a processor 802 coupled to a plurality of GPGPUs 806A-806D via a host interface switch 804. The host interface switch 804 may be a PCI Express switch device that couples the processor 802 to a PCI Express bus, through which the processor 802 can communicate with the aggregate of GPGPUs 806A-806D. Each of the plurality of GPGPUs 806A-806D may be... Figure 7An example of the GPGPU 700. GPGPU 806A-806D can be interconnected via a set 816 of high-speed point-to-point GPU-GPU links. The high-speed GPU-GPU links can be connected to each of the GPGPU 806A-806D via dedicated GPU links, such as... Figure 7 GPU link 710. P2P GPU link 816 enables direct communication between each GPGPU in the GPGPU 806A-806D, without communication via the host interface bus to which processor 802 is connected. Utilizing GPU-GPU traffic directed to the P2P GPU link, the host interface bus remains available for system memory access, or for communication with other instances of the multi-GPU computing system 800, for example, via one or more network devices. While in Figure 8 The GPGPU806A-806D is connected to the processor 802 via the host interface switch 804, but the processor 802 may alternatively include direct support for the P2P GPU link 816 and can be directly connected to the GPGPU806A-806D.

[0201] Machine learning neural network implementation methods

[0202] The computational architectures described herein can be configured to perform these types of parallel processing, which are particularly well-suited for training and deploying neural networks for machine learning. Neural networks can be generalized as functional networks with graph relationships. As is well known in the art, there are various types of neural network implementations used in machine learning. One exemplary type of neural network is the feedforward network as previously described.

[0203] The second exemplary type of neural network is the Convolutional Neural Network (CNN). CNNs are specialized feedforward neural networks designed for processing data with known, grid-like topologies, such as image data. Therefore, CNNs are commonly used in computational vision and image recognition applications, but they can also be used in other types of pattern recognition, such as speech and language processing. Nodes in the input layer of a CNN are organized as sets of “filters” (feature detectors excited by receptive fields found in the retina), and the output of each set of filters is propagated to nodes in successive layers of the network. The computation for a CNN involves applying convolutional mathematics to each filter to produce the output of that filter. Convolution is a specialized mathematical operation performed by two functions to produce a third function, which is a modified version of one of the two original functions. In convolutional network terminology, the first function of the convolution can be called the input, and the second function can be called the convolution kernel. The output can be called a feature map. For example, the input to a convolutional layer could be a multidimensional array of data defining various color components of an input image. The convolution kernel could be a multidimensional array of parameters, which are adapted through a training process for the neural network.

[0204] Recurrent Neural Networks (RNNs) are a class of feedforward neural networks that include feedback connections between layers. RNNs enable modeling of sequential data by sharing parameter data across different parts of the neural network. Architectures used for RNNs include loops. These loops represent the effect of the current value of a variable on its own value at future moments, because at least a portion of the output data from the RNN is used as feedback to process subsequent inputs in the sequence. This characteristic makes RNNs particularly useful for language processing due to the variable nature of language data.

[0205] The diagrams described below illustrate exemplary feedforward networks, CNN networks, and RNN networks, and describe the general process for training and deploying each of those types of networks, respectively. It will be understood that these descriptions are exemplary and non-limiting with respect to any particular embodiment described herein, and that the concepts illustrated in general are applicable to deep neural networks and machine learning techniques.

[0206] The exemplary neural network described above can be used to perform deep learning. Deep learning is machine learning that uses deep neural networks. In contrast to shallow neural networks that contain only a single hidden layer, the deep neural networks used in deep learning are artificial neural networks composed of multiple hidden layers. Deeper neural networks are generally more computationally intensive to train. However, the additional hidden layers of the network enable multi-step pattern recognition, which produces a reduced output error compared to shallow machine learning techniques.

[0207] Deep neural networks used in deep learning typically consist of a front-end network that performs feature recognition, coupled to a back-end network that represents a mathematical model that can perform operations based on feature representations provided to the model (e.g., object classification, speech recognition, etc.). Deep learning enables machine learning without requiring manual feature engineering on the model. Instead, deep neural networks can learn features based on statistical structure or correlations within the input data. The learned features are then fed to a mathematical model that maps the detected features to the output. The mathematical model used by the network is typically specialized for the specific task to be performed, and different models will be used to perform different tasks.

[0208] Once a neural network is structured, a learning model can be applied to it to train it to perform a specific task. The learning model describes how to adjust the weights within the model to reduce the network's output error. Backpropagation of error is a common method used to train neural networks. An input vector is presented to the network for processing. The network's output is compared to the expected output using a loss function, and an error value is calculated for each neuron in the output layer. The error values ​​are then backpropagated until each neuron has an associated error value that roughly represents its contribution to the original output. The network can then learn from those errors using algorithms such as stochastic gradient descent to update the neural network's weights.

[0209] Figures 9A-9B The diagram illustrates an exemplary convolutional neural network. Figure 9A The diagram illustrates the various layers within a CNN. (Example:) Figure 9A As shown, an exemplary CNN for modeling image processing may receive input 902, which describes the red, green, and blue (RGB) components of an input image. Input 902 may be processed by multiple convolutional layers (e.g., convolutional layer 904, convolutional layer 906). The output from the multiple convolutional layers may optionally be processed by a set of fully connected layers 908. As previously described for feedforward networks, neurons in a fully connected layer have full connections to all activations in the previous layer. The output from the fully connected layer 908 can be used to generate an output from the network. Matrix multiplication can be used instead of convolution to compute activations within the fully connected layer 908. Not all CNN implementations utilize the fully connected layer 908. For example, in some implementations, convolutional layer 906 may generate the CNN output.

[0210] Convolutional layers are sparsely connected, unlike the traditional neural network configuration found in fully connected layers (908). Traditional neural network layers are fully connected, such that each output unit interacts with each input unit. However, as illustrated, convolutional layers are sparsely connected because the output of the convolution of the receptive field (rather than the corresponding state value of each node in the receptive field) is fed into nodes of subsequent layers. The kernels associated with the convolutional layers perform convolution operations, the output of which is sent to the next layer. Dimensionality reduction performed within convolutional layers is one aspect that enables CNNs to scale to process large images.

[0211] Figure 9B The diagram illustrates an exemplary computational stage within a convolutional layer of a CNN. The input 912 to the convolutional layer of the CNN can be processed in three stages of convolutional layer 914. These three stages may include a convolutional stage 916, a detector stage 918, and a pooling stage 920. Subsequently, convolutional layer 914 can output data to successive convolutional layers. The final convolutional layer of the network can generate output feature map data or provide input to fully connected layers, for example, to generate classification values ​​for the input to the CNN.

[0212] In convolutional stage 916, several convolutions are performed in parallel to produce a set of linear activations. Convolutional stage 916 may include affine transformations, which are any transformations that can be specified as a linear transformation plus translation. Affine transformations include rotation, translation, scaling, and combinations of these transformations. The convolutional stage computes the output of a function (e.g., a neuron) connected to a specific region in the input, which can be identified as a local region associated with the neuron. The neuron computes the dot product between the neuron's weights and the weights of the regions in the local input to which the neuron is connected. The output from convolutional stage 916 defines the set of linear activations processed by successive stages of convolutional layers 914.

[0213] Linear activations can be handled by the detector-level 918. In the detector-level 918, each linear activation is handled by a nonlinear activation function. Nonlinear activation functions add nonlinearity to the overall network without affecting the receptive field of the convolutional layers. Several types of nonlinear activation functions can be used. One specific type is the Modified Linear Unit (ReLU), which uses an activation function defined as f(x) = max(0,x) with its threshold set to zero.

[0214] Pooling stage 920 uses a pooling function that replaces the output of convolutional layer 906 with a generalized statistic of the nearby outputs. Pooling functions can be used to introduce translation invariance into neural networks, ensuring that small translations to the input do not alter the pooled output. Local translation invariance can be useful in scenarios where the presence of a feature in the input data is more important than the precise location of that feature. Various types of pooling functions can be used during pooling stage 920, including max pooling, average pooling, and L2-norm pooling. Additionally, some CNN implementations do not include a pooling stage. Instead, such implementations use an additional convolutional stage with increased strides relative to the previous convolutional stage.

[0215] The output from convolutional layer 914 can then be processed by the next layer 922. The next layer 922 can be an additional convolutional layer or one of the fully connected layers 908. For example, Figure 9A The first convolutional layer 904 can output to the second convolutional layer 906, and the second convolutional layer can output to the first layer in the fully connected layer 908.

[0216] Figure 10 The diagram illustrates an exemplary recurrent neural network 1000. In a recurrent neural network (RNN), the network's previous state influences the output of the network's current state. RNNs can be constructed in various ways using various functions. The use of RNNs typically revolves around using mathematical models to predict the future based on previous input sequences. For example, an RNN can be used to perform statistical language modeling to predict an upcoming word given a previous sequence of words. The illustrated RNN 1000 can be described as having the following components: an input layer 1002 that receives an input vector; a hidden layer 1004 for implementing the recurrent function; a feedback mechanism 1005 for enabling the "memory" of previous states; and an output layer 1006 for outputting the result. The RNN 1000 operates based on time steps. The state of the RNN at a given time step is influenced by the feedback mechanism 1005 based on previous time steps. For a given time step, the state of the hidden layer 1004 is defined by the previous state and the input at the current time step. The initial input (x1) at the first time step can be processed by the hidden layer 1004. The second input (x2) can be processed by the hidden layer 1004 using the state information determined during the processing of the initial input (x1). The given state can be computed as s. t =f(Ux t +Ws (t-1) ), where U and W are parameter matrices. The function f is generally nonlinear, such as a variant of the hyperbolic tangent function (Tanh) or the correction function f(x) = max(0, x). However, the specific mathematical function used in hidden layer 1004 can vary depending on the specific implementation details of the RNN 1000.

[0217] In addition to the basic CNN and RNN networks described, variants of those networks can be implemented. An example RNN variant is the Long Short-Term Memory (LSTM) RNN. LSTM RNNs are capable of learning long-term dependencies that may be necessary for processing longer language sequences. A CNN variant is the Convolutional Deep Belief Network (DBN), which has a similar structure to a CNN and is trained in a similar manner to a Deep Belief Network. A Deep Belief Network (DBN) is a generative neural network consisting of multiple layers of stochastic (random) variables. A DBN can be trained layer by layer using greedy unsupervised learning. Subsequently, the learned weights of the DBN can be used to provide a pre-trained neural network by determining the optimal initial set of weights for the neural network.

[0218] Figure 11 The diagram illustrates the training and deployment of a deep neural network. Once the given network has been structured for the task, it is trained using a training dataset 1102. Various training frameworks 1104 have been developed to enable hardware acceleration of the training process. For example, Figure 6 The machine learning framework 604 can be configured as a training framework 604. The training framework 604 can be hooked up with an untrained neural network 1106 and enables the use of the parallel processing resources described herein to train the untrained neural network to generate a trained neural network 1108.

[0219] To begin the training process, initial weights can be selected randomly or by pre-training using a deep belief network. Training loops are then performed in a supervised or unsupervised manner.

[0220] Supervised learning is a learning method in which training is performed as an intermediate operation, such as when the training dataset 1102 includes the input paired with the expected output, or when the training dataset includes the input with known output and the output of the neural network is manually graded. The network processes the input and compares the resulting output with the expected output or the set of expected outputs. The error is then backpropagated through the system. The training framework 1104 can be tuned to adjust the weights controlling the untrained neural network 1106. The training framework 1104 can provide tools to monitor how well the untrained neural network 1106 converges to a model suitable for generating correct answers based on known input data. The training process occurs repeatedly as the network weights are adjusted to improve the output generated by the neural network. The training process can continue until the neural network reaches the statistically expected accuracy associated with the trained neural network 1108. The trained neural network 1108 can then be deployed to perform any number of machine learning operations to generate inference results 1114 based on the input of new data 1112.

[0221] Unsupervised learning is a learning method in which a network attempts to train itself using unlabeled data. Therefore, for unsupervised learning, the training dataset 1102 would include input data without any associated output data. An untrained neural network 1106 can learn groupings within the unlabeled inputs and can determine how individual inputs relate to the entire dataset. Unsupervised training can be used to generate self-organizing graphs, which are a class of trained neural networks 1108 capable of performing operations that facilitate dimensionality reduction of the data. Unsupervised training can also be used to perform anomaly detection, which allows the identification of data points in the input dataset that deviate from the normal pattern of the data.

[0222] Variations of supervised and unsupervised training can also be employed. Semi-supervised learning is a technique in which a mixture of labeled and unlabeled data with the same distribution is included in the training dataset 1102. Progressive learning is a variation of supervised learning in which the input data is used continuously to further train the model. Progressive learning enables the trained neural network 1108 to adapt to new data 1112 without forgetting the knowledge embedded in the network during the initial training.

[0223] Whether supervised or unsupervised, training very deep neural networks can be computationally too intensive for a single computing node. A distributed network of computing nodes can be used instead of a single node to accelerate the training process.

[0224] Figure 12 This is a block diagram illustrating distributed learning. Distributed learning uses multiple distributed computing nodes to perform supervised or unsupervised training of neural network models. Each distributed computing node can include one or more host processors and one or more general-purpose processing nodes, such as... Figure 7 The system features a highly parallel general-purpose graphics processing unit 700. As illustrated, distributed learning can be performed using model parallelism 1202, data parallelism 1204, or a combination of model parallelism and data parallelism 1206.

[0225] In model parallelism (1202), different computing nodes in a distributed system can perform training computations on different parts of a single network. For example, each layer of a neural network can be trained by different processing nodes in a distributed system. Benefits of model parallelism include the ability to scale to particularly large models. Separating the computations associated with different layers of a neural network makes it possible to train very large neural networks where the weights of all layers will not be fitted into the memory of a single computing node. In some instances, model parallelism can be particularly useful in performing unsupervised training of large neural networks.

[0226] In data parallelism 1204, different nodes in a distributed network have complete instances of the model, and each node receives a different portion of the data. The results from the different nodes are then combined. While different methods for data parallelism are possible, all data-parallel training methods require techniques to combine the results and synchronize the model parameters between each node. Exemplary methods for combining data include parameter averaging and update-based data parallelism. Parameter averaging trains each node on a subset of the training data and sets global parameters (e.g., weights, biases) to the average of the parameters from each node. Parameter averaging uses a central parameter server to maintain the parameter data. Update-based data parallelism is similar to parameter averaging, except that updates to the model are passed instead of parameters from the nodes being sent to the parameter server. Furthermore, update-based data parallelism can be performed in a distributed manner, where updates are compressed and transmitted between nodes.

[0227] Combinatorial model and data parallelism can be implemented, for example, in a distributed system where each compute node includes multiple GPUs

[1206] . Each node can have a complete instance of the model, with multiple separate GPUs within each node used to train different parts of the model.

[0228] Distributed training incurs increased overhead compared to training on a single machine. However, the parallel processors and GPGPUs described in this paper can each implement various techniques to reduce the overhead of distributed training, including techniques for enabling and accelerating remote data synchronization for high-bandwidth GPU-to-GPU data transfer.

[0229] Exemplary machine learning applications

[0230] Machine learning can be applied to solve a wide range of technical problems, including but not limited to computer vision, autonomous driving and navigation, speech recognition, and language processing. Computer vision has traditionally been one of the most active research areas for machine learning applications. Applications of computer vision range from reproducing human visual abilities (such as recognizing faces) to creating new categories of visual abilities. For example, a computer vision application can be configured to identify sound waves from vibrations induced in objects visible in a video. Parallel processor-accelerated machine learning enables the training of computer vision applications using training datasets that are significantly larger than previously feasible, and allows the deployment of inference systems using low-power parallel processors.

[0231] Parallel processor-accelerated machine learning has applications in autonomous driving, including lane and road sign recognition, obstacle avoidance, navigation, and driving control. Accelerated machine learning techniques can be used to train driving models based on datasets that define appropriate responses to specific training inputs. The parallel processors described in this paper enable rapid training of increasingly sophisticated neural networks for autonomous driving solutions and allow the deployment of low-power inference processors in mobile platforms suitable for integration into autonomous vehicles.

[0232] Parallel processor-accelerated deep neural networks have been implemented as machine learning methods for Automatic Speech Recognition (ASR). ASR involves creating functions that compute the most probable language sequence given an input speech sequence. Accelerated machine learning using deep neural networks has replaced previous Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs) used for ASR.

[0233] Parallel processor-accelerated machine learning can also be used to accelerate natural language processing. Automated learning programs can leverage statistical inference algorithms to generate robust models for error-prone or unfamiliar inputs. Exemplary natural language processor applications include automated machine translation between human languages.

[0234] Parallel processing platforms for machine learning can be divided into training platforms and deployment platforms. Training platforms are typically highly parallel and include optimizations for accelerating single-node multi-GPU training and multi-node multi-GPU training. Exemplary parallel processors suitable for training include... Figure 7 General purpose graphics processing unit 700 and Figure 8 The multi-GPU computing system 800. In contrast, deployed machine learning platforms typically include low-power parallel processors suitable for use in products such as cameras, autonomous robots, and autonomous vehicles.

[0235] Figure 13The illustration depicts an exemplary inference system-on-a-chip (SOC) 1300 suitable for performing inference using a trained model. SOC 1300 may integrate multiple processing components, including a media processor 1302, a vision processor 1304, a GPGPU 1306, and a multi-core processor 1308. GPGPU 1306 may be a GPGPU as described herein, such as GPGPU 700, and multi-core processor 1308 may be a multi-core processor as described herein, such as multi-core processors 405-406. SOC 1300 may additionally include on-chip memory 1305, which enables a shared on-chip data pool accessible by each of the processing components. The processing components may be optimized for low-power operation to enable deployment on various machine learning platforms, including autonomous vehicles and autonomous robots. For example, one implementation of SOC 1300 may be used as part of a main control system for an autonomous vehicle. When the SOC1300 is configured for use in autonomous vehicles, the SOC is designed and configured to comply with the relevant functional safety standards of the deployment jurisdiction.

[0236] During operation, the media processor 1302 and the vision processor 1304 can work in concert to accelerate computer vision operations. The media processor 1302 enables low-latency decoding of multiple high-resolution (e.g., 4K, 8K) video streams. The decoded video streams can be written to a buffer in on-chip memory 1305. Subsequently, the vision processor 1304 can parse the decoded video and perform preliminary processing operations on the frames of the decoded video to prepare them for processing using a trained image recognition model. For example, the vision processor 1304 can accelerate convolutional operations for a CNN used to perform image recognition on high-resolution video data, while the back-end model computation is performed by the GPGPU 1306.

[0237] The multi-core processor 1308 may include control logic for assisting in the sequencing and synchronization of data transfers and shared memory operations performed by the media processor 1302 and the vision processor 1304. The multi-core processor 1308 may also act as an application processor to execute software applications that can utilize the inference computing power of the GPGPU 1306. For example, at least a portion of navigation and driving logic may be implemented in software executing on the multi-core processor 1308. Such software may directly offload computational workloads to the GPGPU 1306, or it may offload computational workloads to the multi-core processor 1308, which may offload at least a portion of those operations to the GPGPU 1306.

[0238] The GPGPU 1306 may include compute clusters, such as a low-power configuration of the processing clusters 706A-706H within the general-purpose graphics processing unit 700. The compute clusters within the GPGPU 1306 may support instructions specifically optimized for performing inference computations on trained neural networks. For example, the GPGPU 1306 may support instructions for performing low-precision computations, such as 8-bit and 4-bit integer vector operations.

[0239] Additional System Overview

[0240] Figure 14 This is a block diagram of the processing system 1400. Figure 14 Elements having the same or similar names as elements in any other figure herein are described as the same elements in other figures, and can operate or function in a manner similar to those described elsewhere herein. They may include, but are not limited to, components identical to those described elsewhere herein and may be linked to other entities as described elsewhere herein. System 1400 can be used in: single-processor desktop systems, multi-processor workstation systems, or server systems having a large number of processors 1402 or processor cores 1407. System 1400 may be a processing platform incorporated within a system-on-a-chip (SoC) integrated circuit for use in mobile devices, handheld devices, or embedded devices, such as for use in Internet of Things (IoT) devices with wired or wireless connectivity to a local area network or wide area network.

[0241] System 1400 can be equipped with... Figure 1 The processing system corresponding to those components. For example, in different configurations, (multiple) processors 1402 or (multiple) processor cores 1407 can be... Figure 1 The (multiple) processors 102 correspond to the (multiple) graphics processors 1408. Figure 1 The (multiple) parallel processors 112 correspond to this. The external graphics processor 1418 can be... Figure 1 One of the (multiple) plug-in devices 120.

[0242] System 1400 may include, be coupled to, or be incorporated into: a server-based gaming platform, a game console including a game and media console, a mobile game console, a handheld game console, or an online game console. System 1400 may be part of a mobile phone, smartphone, tablet computing device, or mobile internet-connected device (such as a laptop with low internal storage). Processing system 1400 may also include, be coupled to, or be integrated into: wearable devices, such as smartwatch wearable devices; smart glasses or clothing that utilize augmented reality (AR) or virtual reality (VR) features to enhance visual, audio, or haptic output to supplement real-world visual, audio, or haptic experiences or otherwise provide text, audio, graphics, video, holograms or video, or haptic feedback; other augmented reality (AR) devices; or other virtual reality (VR) devices. Processing system 1400 may include, or be part of a television or set-top box device. System 1400 may include, be coupled to, or be integrated into an autonomous vehicle, such as a bus, tractor-trailer, automobile, electric motor or electric cycle, airplane or glider (or any combination thereof). The autonomous vehicle may use system 1400 to process the environment sensed around it.

[0243] One or more processors 1402 may include one or more processor cores 1407 for processor instructions that, when executed, perform operations for system or user software. At least one of the one or more processor cores 1407 may be configured to process a particular instruction set 1409. The instruction set 1409 may facilitate Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or computation via Very Long Instruction Word (VLIW). One or more processor cores 1407 may process different instruction sets 1409, and the different instruction sets 1409 may include instructions for facilitating emulation of other instruction sets. Processor cores 1407 may also include other processing devices, such as digital signal processors (DSPs).

[0244] Processor 1402 may include cache memory 1404. Depending on the architecture, processor 1402 may have a single internal cache or multiple levels of internal caches. In some embodiments, the cache memory is shared among various components of processor 1402. In some embodiments, processor 1402 also uses an external cache (e.g., a Level 3 (L3) cache or a Last Level Cache (LLC)) (not shown), which may be shared among processor cores 1407 using known cache coherence techniques. Register file 1406 may be additionally included in processor 1402, and register file 1406 may include different types of registers for storing different types of data (e.g., integer registers, floating-point registers, status registers, and instruction pointer registers). Some registers may be general-purpose registers, while others may be dedicated to the design of processor 1402.

[0245] One or more processors 1402 may be coupled to one or more interface buses 1410 to transmit communication signals, such as address, data, or control signals, between the processors 1402 and other components in the system 1400. In one embodiment of these embodiments, the interface bus 1410 may be a processor bus, such as a version of a Direct Media Interface (DMI) bus. However, the processor bus is not limited to a DMI bus and may include one or more peripheral component interconnect buses (e.g., PCI, PCI express), memory buses, or other types of interface buses. For example, the processors 1402 may include an integrated memory controller 1416 and a platform controller hub 1430. The memory controller 1416 facilitates communication between memory devices and other components of the system 1400, while the platform controller hub (PCH) 1430 provides connectivity to I / O devices via a local I / O bus.

[0246] Memory device 1420 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase-change memory device, or some other memory device with suitable performance to act as process memory. Memory device 1420 may operate, for example, as system memory for system 1400 to store data 1422 and instructions 1421 for use when one or more processors 1402 execute an application or process. Memory controller 1416 is also coupled to an optional external graphics processor 1418, which may communicate with one or more graphics processors 1408 in processor 1402 to perform graphics and media operations. In some embodiments, graphics, media, or computational operations may be assisted by an accelerator 1412, which is a coprocessor that can be configured to perform a set of specialized graphics, media, or computational operations. For example, accelerator 1412 may be a matrix multiplication accelerator for optimizing machine learning or computational operations. Accelerator 1412 may be a ray tracing accelerator that can be used to perform ray tracing operations in accordance with graphics processor 1408. In one embodiment, an external accelerator 1419 may be used instead of accelerator 1412, or an external accelerator 1419 may be used in accordance with accelerator 1412.

[0247] A display device 1411 may be provided, which can be connected to processor(s) 1402. The display device 1411 may be one or more of the following: an internal display device, such as in a mobile electronic device or a laptop device; or an external display device attached via a display interface (e.g., a display port, etc.). The display device 1411 may be a head-mounted display (HMD), such as a stereoscopic display device for use in virtual reality (VR) or augmented reality (AR) applications.

[0248] The platform controller hub 1430 enables peripheral devices to connect to the memory device 1420 and the processor 1402 via a high-speed I / O bus. I / O peripheral devices include, but are not limited to, an audio controller 1446, a network controller 1434, a firmware interface 1428, a wireless transceiver 1426, a touch sensor 1425, and a data storage device 1424 (e.g., non-volatile memory, volatile memory, hard disk drive, flash memory, NAND, 3D NAND, 3D Xpoint, etc.). The data storage device 1424 can be connected via a storage interface (e.g., SATA) or via a peripheral bus such as a peripheral component interconnect bus (e.g., PCI, PCI express). The touch sensor 1425 may include a touchscreen sensor, a pressure sensor, or a fingerprint sensor. The wireless transceiver 1426 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, 5G, or LTE transceiver. Firmware interface 1428 enables communication with system firmware and may be, for example, a Unified Extensible Firmware Interface (UEFI). Network controller 1434 enables network connectivity to a wired network. In some embodiments, a high-performance network controller (not shown) is coupled to interface bus 1410. Audio controller 1446 may be a multi-channel high-definition audio controller. In some of these embodiments, system 1400 includes an optional legacy I / O controller 1440 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to the system. Platform controller hub 1430 may also be connected to one or more Universal Serial Bus (USB) controllers 1442 to connect input devices, such as a keyboard and mouse combination 1443, a camera 1444, or other USB input devices.

[0249] It will be understood that the illustrated system 1400 is exemplary and not limiting, as other types of data processing systems configured differently may also be used. For example, instances of memory controller 1416 and platform controller hub 1430 may be integrated into a discrete external graphics processor, such as external graphics processor 1418. Platform controller hub 1430 and / or memory controller 1416 may be external to one or more processors 1402. For example, system 1400 may include external memory controller 1416 and platform controller hub 1430, which may be configured as a memory controller hub and peripheral controller hub within a system chipset communicating with processor(s) 1402.

[0250] For example, a circuit board (“sled”) can be used on which components (such as CPUs, memory, and other components) are placed to achieve improved thermal performance. Processing components, such as processors, may be located on the top side of the sled, while nearby memory, such as DIMMs, may be located on the bottom side. As a result of the enhanced airflow provided by this design, components can operate at higher frequencies and power levels than typical systems, thereby increasing performance. Furthermore, the sled is configured to blind-mate power and data communication cables in a rack, thereby enhancing their ability to be quickly removed, upgraded, reinstalled, and / or replaced. Similarly, the individual components located on the sled (such as processors, accelerators, memory, and data storage devices) are configured for easy upgrades due to their increased spacing relative to each other. In the illustrative embodiment, the components additionally include hardware authentication features to verify their authenticity.

[0251] Data centers can utilize a single network fabric (“fabrication”) that supports multiple other network architectures, including Ethernet and omnidirectional paths. Skids can be coupled to switches via fiber optic cables, providing higher bandwidth and lower latency than typical twisted-pair cabling (e.g., Category 5, Category 5e, Category 6, etc.). Due to the high-bandwidth, low-latency interconnects and network architecture, data centers can centralize physically decentralized resources such as memory, accelerators (e.g., GPUs, graphics accelerators, FPGAs, ASICs, neural network and / or AI accelerators, etc.) and data storage drives, and provide them to computing resources (e.g., processors) as needed, allowing computing resources to access these centralized resources as if they were local.

[0252] A power supply or power source can provide voltage and / or current to System 1400 or any component or system described herein. In one example, the power supply includes an AC-DC (alternating current to direct current) adapter for plugging into a wall outlet. Such AC power can be a renewable energy (e.g., solar) power source. In one example, the power source includes a DC power source, such as an external AC-DC converter. The power source or power supply may also include wireless charging hardware for charging via proximity charging. The power source may include an internal battery, AC supply, motion-based power supply, solar power supply, or fuel cell source.

[0253] Figures 15A-15C Illustration of computing systems and graphics processors. Figures 15A-15CThose elements having the same or similar names as elements in any other figure herein describe the same elements in other figures, operate or function in a manner similar to those described elsewhere herein, may include the same components as those described elsewhere herein, and may be linked to other entities as described elsewhere herein, but are not limited thereto.

[0254] Figure 15A This is a block diagram of an embodiment of processor 1500, which may be a variant of one of the processors 1402 and may be used in place of one of those processors. Therefore, the disclosure of any features combined with processor 1500 herein also discloses corresponding combinations with processor(s) 1402(s), but is not limited thereto. Processor 1500 has one or more processor cores 1502A-1502N, an integrated memory controller 1514, and an integrated graphics processor 1508. In the absence of the integrated graphics processor 1508, a system including this processor will include a graphics processor device within a system chipset or coupled via a system bus. Processor 1500 may include additional cores, up to and including the additional core 1502N indicated by the dashed box. Each of processor cores 1502A-1502N includes one or more internal cache units 1504A-1504N. In some embodiments, each processor core 1502A-1502N also has access to one or more shared cache units 1506. Internal cache units 1504A-1504N and shared cache units 1506 represent the cache memory hierarchy within the processor 1500. The cache memory hierarchy may include at least one level of instruction and data cache within each processor core, and one or more levels of shared intermediate caches, such as Level 2 (L2), Level 3 (L3), Level 4 (L4), or other levels of cache, wherein the highest level of cache preceding external memory is classified as LLC. In some embodiments, cache coherence logic maintains coherence between each cache unit 1506 and 1504A-1504N.

[0255] The processor 1500 may also include a collection 1516 of one or more bus controller units and a system agent core 1510. The one or more bus controller units 1516 manage a collection of peripheral buses, such as one or more PCI buses or PCI Express buses. The system agent core 1510 provides management functions for each processor component. The system agent core 1510 may include one or more integrated memory controllers 1514 for managing access to various external memory devices (not shown).

[0256] For example, one or more processor cores among processor cores 1502A-1502N may include support for simultaneous multithreading. System agent core 1510 includes components for coordinating and operating cores 1502A-1502N during multithreaded processing. System agent core 1510 may additionally include a power control unit (PCU) that includes logic and components for regulating the power states of processor cores 1502A-1502N and graphics processor 1508.

[0257] Processor 1500 may additionally include a graphics processor 1508 for performing graphics processing operations. In some of these embodiments, graphics processor 1508 is coupled to a set 1506 of shared cache units and to a system proxy core 1510, which includes one or more integrated memory controllers 1514. System proxy core 1510 may also include a display controller 1511 for driving graphics processor output to one or more coupled displays. Display controller 1511 may also be a separate module coupled to graphics processor via at least one interconnect, or it may be integrated within graphics processor 1508.

[0258] Ring-based interconnect unit 1512 can be used to couple internal components of processor 1500. However, alternative interconnect units, such as point-to-point interconnects, switched interconnects, or other technologies, including those known in the art, can be used. In some embodiments of these embodiments having ring-based interconnect 1512, graphics processor 1508 is coupled to ring-based interconnect 1512 via I / O link 1513.

[0259] Exemplary I / O link 1513 represents at least one of a variety of I / O interconnects, including on-package I / O interconnects that facilitate communication between various processor components and high-performance embedded memory modules 1518 (such as eDRAM modules). Optionally, each processor core in processor cores 1502A-1502N and graphics processor 1508 may use embedded memory module 1518 as a shared final-level cache.

[0260] Processor cores 1502A-1502N can be, for example, homogeneous cores executing the same instruction set architecture. Alternatively, processor cores 1502A-1502N can be heterogeneous in terms of instruction set architecture (ISA), wherein one or more of processor cores 1502A-1502N execute a first instruction set, while at least one of the other cores executes a subset of the first instruction set or a different instruction set. Processor cores 1502A-1502N can be heterogeneous in terms of microarchitecture, wherein one or more cores with relatively high power consumption are coupled to one or more power cores with lower power consumption. As another example, processor cores 1502A-1502N can be heterogeneous in terms of computing power. Furthermore, processor 1500 can be implemented on one or more chips, or implemented as a SoC integrated circuit having the components illustrated, among other components.

[0261] Figure 15B This is a block diagram of the hardware logic of a graphics processor core 1519 according to some embodiments described herein. A graphics processor core 1519 (sometimes referred to as a core slice) can be one or more graphics cores within a modular graphics processor. An example of a graphics processor core 1519 is a graphics core slice, and a graphics processor as described herein can include multiple graphics core slices based on a target power envelope and a performance envelope. Each graphics processor core 1519 may include a fixed-function block 1530 coupled to multiple sub-cores 1521A-1521F (also referred to as sub-slices), which include blocks of modular general-purpose and fixed-function logic.

[0262] Fixed-function block 1530 may include a geometry / fixed-function pipeline 1531, which may be shared by all sub-cores of graphics processor core 1519, for example, in a lower-performance and / or lower-power graphics processor implementation. Geometry / fixed-function pipeline 1531 may include a 3D fixed-function pipeline (e.g., as described below). Figure 16A The 3D pipeline (1612) includes a video front-end unit, a thread generator and a thread dispatcher, and a unified return buffer manager, which uses a unified return buffer (e.g., as described below in...) Figure 17 (Unified return buffer 1718 in the middle).

[0263] Fixed function block 1530 may also include a graphics SoC interface 1532, a graphics microcontroller 1533, and a media pipeline 1534. The graphics SoC interface 1532 provides an interface between the graphics processor core 1519 and other processor cores within the system-on-a-chip integrated circuit. The graphics microcontroller 1533 is a programmable subprocessor configurable to manage various functions of the graphics processor core 1519, including thread dispatch, scheduling, and preemption. The media pipeline 1534 (e.g., Figure 16A and Figure 17 The media pipeline 1616 includes logic for facilitating the decoding, encoding, preprocessing, and / or post-processing of multimedia data, including image and video data. The media pipeline 1534 performs media operations via requests to computation or sampling logic within subcores 1521A-1521F.

[0264] The SoC interface 1532 enables the graphics processor core 1519 to communicate with a general-purpose application processor core (e.g., a CPU) and / or other components within the SoC, including memory hierarchy elements such as shared final-level cache memory, system RAM, and / or embedded on-chip or packaged DRAM. The SoC interface 1532 also enables communication with fixed-function devices within the SoC, such as camera imaging pipelines, and enables and / or implements global memory atomicity, which can be shared between the graphics processor core 1519 and the CPU within the SoC. The SoC interface 1532 also implements power management control for the graphics processor core 1519 and enables interfacing between the clock domain of the graphics core 1519 and other clock domains within the SoC. Optionally, the SoC interface 1532 enables the receipt of command buffers from a command stream converter and a global thread dispatcher configured to provide commands and instructions to each of one or more graphics cores within the graphics processor. When a media operation is about to be executed, these commands and instructions can be dispatched to the media pipeline 1534, or when a graphics processing operation is about to be executed, these commands and instructions can be dispatched to the geometry and fixed-function pipelines (e.g., geometry and fixed-function pipeline 1531, geometry and fixed-function pipeline 1537).

[0265] The graphics microcontroller 1533 can be configured to perform various scheduling and management tasks for the graphics processor core 1519. In one configuration, the graphics microcontroller 1533 can, for example, perform graphics and / or compute workload scheduling for individual graphics parallel engines within the execution unit (EU) arrays 1522A-1522F and 1524A-1524F of the sub-cores 1521A-1521F. In this workload scheduling, host software executing on the CPU core of the SoC including the graphics processor core 1519 can submit workloads via one of a plurality of graphics processor doorbells, which invokes scheduling operations for the appropriate graphics engine. The scheduling operations include: determining which workload should run next, submitting the workload to the command stream converter, preempting existing workloads running on the engine, monitoring the progress of the workload, and notifying the host software when the workload is complete. Optionally, the graphics microcontroller 1533 may also facilitate low-power or idle states of the graphics processor core 1519, thereby providing the graphics processor core 1519 with the ability to save and restore registers within the graphics processor core 1519 across low-power state transitions, independent of the operating system and / or the graphics driver software on the system.

[0266] The graphics processor core 1519 may have more or fewer sub-cores 1521A-1521F as illustrated, up to N modular sub-cores. For each group of N sub-cores, the graphics processor core 1519 may also include shared function logic 1535, shared and / or cache memory 1536, geometry / fixed-function pipeline 1537, and additional fixed-function logic 1538 for accelerating various graphics and computational processing operations. The shared function logic 1535 may include features that can be shared by each of the N sub-cores within the graphics processor core 1519, and... Figure 17 The shared functional logic 1720 (e.g., sampler logic, mathematical logic, and / or inter-thread communication logic) is associated with the logic unit. The shared and / or cache memory 1536 may be the final-level cache for a set 1521A-1521F of N sub-cores within the graphics processor core 1519, and may also act as shared memory accessible by multiple sub-cores. A geometry / fixed-function pipeline 1537, instead of geometry / fixed-function pipeline 1531, may be included within the fixed-function block 1530, and geometry / fixed-function pipeline 1537 may include the same or similar logic units.

[0267] The graphics processing unit (GPU) core 1519 may include additional fixed-function logic 1538, which may include various fixed-function acceleration logics for use by the GPU core 1519. Optionally, the additional fixed-function logic 1538 includes an additional geometry pipeline for use in position-only shading. In position-only shading, there are two geometry pipelines: a full geometry pipeline within the geometry / fixed-function pipeline 1538, 1531; and a culling pipeline, which is an additional geometry pipeline that may be included within the additional fixed-function logic 1538. For example, the culling pipeline may be a simplified version of the full geometry pipeline. The full pipeline and the culling pipeline may execute different instances of the same application, each with a separate context. Position-only shading may hide long culling runs of discarded triangles, thereby enabling earlier shading completion in some instances. For example, the culling pipeline logic within the additional fixed-function logic 1538 can execute the position shader in parallel with the main application and typically generates key results faster than the full pipeline because the culling pipeline only extracts the position attributes of the vertices and shades those attributes without performing rasterization and rendering of pixels to the frame buffer. The culling pipeline can use the generated key results to compute visibility information for all triangles, regardless of whether those triangles were culled. The full pipeline (which can be referred to as the replay pipeline in this example) can consume this visibility information to skip culled triangles, thus shading only the visible triangles that are ultimately passed to the rasterization stage.

[0268] Optionally, the additional fixed-function logic 1538 may also include machine learning acceleration logic, such as fixed-function matrix multiplication logic, which is used to include an implementation optimized for machine learning training or inference.

[0269] Each graphics subcore 1521A-1521F includes a set of execution resources that can be used to perform graphics operations, media operations, and computational operations in response to requests made by the graphics pipeline, media pipeline, or shader program. The graphics subcores 1521A-1521F include: multiple EU arrays 1522A-1522F, 1524A-1524F; thread dispatch and inter-thread communication (TD / IC) logic 1523A-1523F; 3D (e.g., texture) samplers 1525A-1525F; media samplers 1506A-1506F; shader processors 1527A-1527F; and shared local memory (SLM) 1528A-1528F. The EU arrays 1522A-1522F and 1524A-1524F each include multiple execution units, which are general-purpose graphics processing units capable of performing floating-point and integer / fixed-point logic operations to serve graphics operations, media operations, or computational operations (including graphics programs, media programs, or computational shader programs). The TD / IC logic 1523A-1523F performs local thread dispatch and thread control operations for execution units within the sub-core and facilitates communication between threads executing on the execution units of the sub-core. The 3D samplers 1525A-1525F can read textures or other 3D graphics-related data into memory. The 3D samplers can read texture data in different ways based on the configured sample state and the texture format associated with a given texture. The media samplers 1506A-1506F can perform similar read operations based on the type and format associated with the media data. For example, each graphics sub-core 1521A-1521F may alternately include unified 3D and media samplers. Threads executing on execution units within each of the subcores 1521A-1521F can utilize the shared local memory 1528A-1528F within each subcore, enabling threads executing within a thread group to use a common pool of on-chip memory for execution.

[0270] Figure 15CThis is a block diagram of a general-purpose graphics processing unit (GPGPU) 1570 according to embodiments described herein, which may be configured as a graphics processor (e.g., graphics processor 1508) and / or a compute accelerator. The GPGPU 1570 may interconnect with a host processor (e.g., one or more CPUs 1546) and memories 1571, 1572 via one or more system and / or memory buses. Memory 1571 may be system memory that can be shared with one or more CPUs 1546, while memory 1572 is device memory dedicated to the GPGPU 1570. For example, components within the GPGPU 1570 and device memory 1572 may be mapped to memory addresses accessible by one or more CPUs 1546. Access to memories 1571 and 1572 may be facilitated via a memory controller 1568. The memory controller 1568 may include an internal direct memory access (DMA) controller 1569, or may include logic for performing operations that would otherwise be performed by the DMA controller.

[0271] The GPGPU 1570 includes multiple cache memories, including an L2 cache 1553, an L1 cache 1554, an instruction cache 1555, and shared memory 1556, at least a portion of which can also be partitioned as cache memory. The GPGPU 1570 also includes multiple compute units 1560A-1560N. Each compute unit 1560A-1560N includes a set of vector registers 1561, a set of scalar registers 1562, a set of vector logic units 1563, and a set of scalar logic units 1564. Compute units 1560A-1560N may also include local shared memory 1565 and a program counter 1566. Compute units 1560A-1560N may be coupled to a constant cache 1567, which can be used to store constant data that will not change during the execution of the kernel program or shader program executed on the GPGPU 1570. Constant cache 1567 can be a scalar data cache, and cached data can be directly retrieved into scalar register 1562.

[0272] During operation, one or more CPUs 1546 may write commands to registers in the GPGPU 1570 or to memory in the GPGPU 1570 that has been mapped into the accessible address space. The command processor 1557 may read commands from registers or memory and determine how those commands will be processed within the GPGPU 1570. The thread dispatcher 1558 may then be used to dispatch threads to compute units 1560A-1560N to execute those commands. Each compute unit 1560A-1560N can execute threads independently of other compute units. Furthermore, each compute unit 1560A-1560N can be independently configured for conditional computation and can conditionally output the results of the computation to memory. When a submitted command completes, the command processor 1557 may interrupt one or more CPUs 1546.

[0273] Figures 16A-16C The illustrations are from embodiments described herein (e.g., according to...). Figures 15A-15C Block diagrams of the additional graphics processor and computing accelerator architectures provided. Figures 16A-16C Elements having the same or similar names as elements in any other figure herein describe the same elements as those in those figures, are capable of operating or functioning in a manner similar to those described elsewhere herein, may include the same components as those described elsewhere herein, and may be linked to other entities as described elsewhere herein, but are not limited thereto.

[0274] Figure 16A This is a block diagram of a graphics processor 1600, which may be a discrete graphics processing unit or a graphics processor integrated with multiple processing cores or other semiconductor devices such as, but not limited to, memory devices or network interfaces. The graphics processor 1600 may be a variant of the graphics processor 1508 and may be used in place of the graphics processor 1508. Therefore, the disclosure of any features combined with the graphics processor 1508 herein also discloses corresponding combinations with the graphics processor 1600, but is not limited thereto. The graphics processor may communicate via a memory-mapped I / O interface to registers on the graphics processor and using commands placed in processor memory. The graphics processor 1600 may include a memory interface 1614 for accessing memory. The memory interface 1614 may be an interface to local memory, one or more internal caches, one or more shared external caches, and / or system memory.

[0275] Optionally, the graphics processor 1600 also includes a display controller 1602 for driving display output data to the display device 1618. The display controller 1602 includes hardware for combining one or more overlay planes of the display and multiple layers of video or user interface elements. The display device 1618 can be an internal or external display device. In one embodiment, the display device 1618 is a head-mounted display device, such as a virtual reality (VR) display device or an augmented reality (AR) display device. The graphics processor 1600 may include a video codec engine 1606 for encoding media to one or more media encoding formats, decoding media from one or more media encoding formats, or transcoding media between one or more media encoding formats, including but not limited to: Moving Picture Experts Group (MPEG) formats (such as MPEG-2), Advanced Video Decoding (AVC) formats (such as H.264 / MPEG-4 AVC, H.265 / HEVC, Alliance for Open Media (AOMedia) VP8, VP9), and SMPTE 421M / VC-1, and Joint Picture Experts Group (JPEG) formats (such as JPEG, and Motion JPEG (MJPEG)).

[0276] The graphics processor 1600 may include a block image transfer (BLIT) engine 1603 for performing two-dimensional (2D) rasterizer operations, including, for example, bit boundary block transfer. Alternatively, however, one or more components of the graphics processing engine (GPE) 1610 may be used to perform 2D graphics operations. In some embodiments, the GPE 1610 is a computational engine for performing graphics operations, including three-dimensional (3D) graphics operations and media operations.

[0277] GPE 1610 may include a 3D pipeline 1612 for performing 3D operations, such as rendering 3D images and scenes using processing functions acting on 3D primitive shapes (e.g., rectangles, triangles, etc.). The 3D pipeline 1612 includes programmable and fixed-function elements that perform various tasks within the elements of the 3D / media subsystem 1615 and / or generated execution threads. While the 3D pipeline 1612 can be used to perform media operations, embodiments of GPE 1610 also include a media pipeline 1616 specifically designed for performing media operations, such as video post-processing and image enhancement.

[0278] The media pipeline 1616 may include fixed-function or programmable logic units to perform one or more specialized media operations, such as video decoding acceleration, video deinterlacing, and video encoding acceleration, in place of or on behalf of the video codec engine 1606. The media pipeline 1616 may additionally include a thread generation unit to generate threads for execution on the 3D / media subsystem 1615. The generated threads perform calculations on the media operations on one or more graphics execution units included in the 3D / media subsystem 1615.

[0279] The 3D / media subsystem 1615 may include logic for executing threads generated by the 3D pipeline 1612 and the media pipeline 1616. The pipelines may send thread execution requests to the 3D / media subsystem 1615, which includes thread dispatch logic for arbitrating and dispatching various requests for available thread execution resources. Execution resources include an array of graphics execution units for processing 3D and media threads. The 3D / media subsystem 1615 may include one or more internal caches for thread instructions and data. Additionally, the 3D / media subsystem 1615 may also include shared memory for sharing data between threads and for storing output data, including registers and addressable memory.

[0280] Figure 16B The illustration shows a graphics processor 1620, which is a variant of graphics processor 1600 and can be used in place of graphics processor 1600 and vice versa. Therefore, the disclosure of any features combined with graphics processor 1600 herein also discloses corresponding combinations with graphics processor 1620, but is not limited thereto. According to the embodiments described herein, graphics processor 1620 has a sharded architecture. Graphics processor 1620 may include a graphics processing engine cluster 1622, which has within graphics engine shards 1610A-1610D... Figure 16AMultiple instances of the graphics processor engine 1610 are included. Each graphics engine chip 1610A-1610D can be interconnected via a set of chip interconnects 1623A-1623F. Each graphics engine chip 1610A-1610D can also be connected to a memory module or memory device 1626A-1626D via a memory interconnect 1625A-1625D. The memory devices 1626A-1626D can use any graphics memory technology. For example, the memory devices 1626A-1626D can be graphics double data rate (GDDR) memory. The memory devices 1626A-1626D can be high bandwidth memory (HBM) modules, which can be on the die together with their respective graphics engine chips 1610A-1610D. The memory devices 1626A-1626D can be stacked memory devices, which can be stacked on top of their respective graphics engine dies 1610A-1610D. Each graphics engine die 1610A-1610D and its associated memory 1626A-1626D can reside on separate chiplets, which are bonded to a base die or substrate, as shown in... Figures 24B-24D The text describes this in further detail.

[0281] The graphics processor 1620 may be configured with a non-uniform memory access (NUMA) system in which memory devices 1626A-1626D are coupled to associated graphics engine chips 1610A-1610D. A given memory device may be accessed by a different chip than the graphics engine chip to which it is directly connected. However, when accessing a local chip, the access latency to memory devices 1626A-1626D can be minimized. In one embodiment, a cache-coherent NUMA (ccNUMA) system is enabled, which uses chip interconnects 1623A-1623F to enable communication between cache controllers within the graphics engine chips 1610A-1610D to maintain a consistent memory mirror when more than one cache stores the same memory location.

[0282] The graphics processing engine cluster 1622 may be connected to an on-chip or package-based interconnect 1624. The interconnect 1624 enables communication between the graphics engine chips 1610A-1610D and components such as a video codec 1606 and one or more copy engines 1604. The copy engines 1604 can be used to move data out of memory devices 1626A-1626D and memory external to the graphics processor 1620 (e.g., system memory), move data into memory devices 1626A-1626D and memory external to the graphics processor 1620 (e.g., system memory), and move data between memory devices 1626A-1626D and memory external to the graphics processor 1620 (e.g., system memory). The interconnect 1624 can also be used to interconnect the graphics engine chips 1610A-1610D. The graphics processor 1620 may optionally include a display controller 1602 to enable connectivity with an external display device 1618. The graphics processor can also be configured as a graphics accelerator or a computing accelerator. In an accelerator configuration, the display controller 1602 and the display device 1618 can be omitted.

[0283] The graphics processor 1620 can be connected to a host system via a host interface 1628. The host interface 1628 enables communication between the graphics processor 1620, system memory, and / or other system components. The host interface 1628 can be, for example, a PCI express bus or another type of host system interface.

[0284] Figure 16C The figure illustrates a computing accelerator 1630 according to an embodiment described herein. The computing accelerator 1630 may include... Figure 16BThe architecture is similar to that of the graphics processor 1620, and is optimized for computational acceleration. The compute engine cluster 1632 may include a set of compute engine chips 1640A-1640D, which includes execution logic optimized for parallel or vector-based general-purpose computational operations. The compute engine chips 1640A-1640D may not include fixed-function graphics processing logic, but in some embodiments, one or more of the compute engine chips 1640A-1640D may include logic for performing media acceleration. The compute engine chips 1640A-1640D may be connected to memories 1626A-1626D via memory interconnects 1625A-1625D. The memories 1626A-1626D and the memory interconnects 1625A-1625D may be of similar technology to those in the graphics processor 1620, or they may be different technologies. The graphics computing engine chips 1640A-1640D can also be interconnected via chip interconnects 1623A-1623F and can be connected to and / or interconnected by the structure interconnect 1624. In one embodiment, the computing accelerator 1630 includes a large L3 cache 1636 that can be configured as a device-wide cache. The computing accelerator 1630 can also be connected to... Figure 16B The graphics processor 1620 in the system is connected to the host processor and memory via the host interface 1628 in a similar manner.

[0285] Graphics processing engine

[0286] Figure 17 This is a block diagram of a graphics processing engine 1710 of a graphics processor according to some embodiments. The graphics processing engine (GPE) 1710 may be... Figure 16A The image shows a version of GPE 1610, and can also indicate... Figure 16B The graphics engine chip in the chip is 1610A-1610D. Figure 17 Elements having the same or similar names as elements in any other figure herein describe the same elements in other figures, operate or function in a manner similar to those described elsewhere herein, may include the same components as those described elsewhere herein, and may be linked to other entities as described elsewhere herein, but are not limited thereto. For example, also Figure 17 The middle picture shows Figure 16A The 3D pipeline 1612 and the media pipeline 1616. The media pipeline 1616 is optional in some embodiments of the GPE 1710 and may not be explicitly included within the GPE 1710. For example, and in at least one embodiment, a separate media and / or image processor is coupled to the GPE 1710.

[0287] The GPE 1710 may be coupled to or include a command stream converter 1703 that provides a command stream to the 3D pipeline 1612 and / or the media pipeline 1616. Alternatively or additionally, the command stream converter 1703 may be directly coupled to a unified return buffer 1718. The unified return buffer 1718 may be communicatively coupled to the graphics core array 1714. Optionally, the command stream converter 1703 may be coupled to memory, which may be system memory, or one or more of internal cache memory and shared cache memory. The command stream converter 1703 may receive commands from memory and send these commands to the 3D pipeline 1612 and / or the media pipeline 1616. These commands are instructions fetched from a ring buffer that stores commands for the 3D pipeline 1612 and the media pipeline 1616. The ring buffer may additionally include a batch command buffer that stores multiple commands in batches. Commands for the 3D pipeline 1612 may also include references to data stored in memory, such as, but not limited to, vertex and geometry data for the 3D pipeline 1612 and / or image data and memory objects for the media pipeline 1616. The 3D pipeline 1612 and the media pipeline 1616 process commands and data by performing operations via logic within their respective pipelines or by dispatching one or more execution threads to the graphics core array 1714. The graphics core array 1714 may include blocks of one or more graphics cores (e.g., multiple graphics cores 1715A, multiple graphics cores 1715B), each block comprising one or more graphics cores. Each graphics core includes a set of graphics execution resources, including: general-purpose execution logic and graphics-specific execution logic for performing graphics and computational operations; and fixed-function texture processing logic and / or machine learning and artificial intelligence acceleration logic.

[0288] In various embodiments, the 3D pipeline 1612 may include fixed functions and programmable logic for processing one or more shader programs, such as vertex shaders, geometry shaders, pixel shaders, fragment shaders, compute shaders, or other shader programs, by processing instructions and dispatching execution threads to the graphics core array 1714. The graphics core array 1714 provides a unified block of execution resources for use in processing these shader programs. The versatile execution logic (e.g., execution units) within the graphics core(s)(s) 1715A-1715B of the graphics core array 1714 includes support for various 3D API shader languages ​​and can execute multiple synchronous execution threads associated with multiple shaders.

[0289] The graphics core array 1714 may include execution logic for performing media functions such as video and / or image processing. The execution unit may include general-purpose logic, which is programmable to perform parallel general-purpose computational operations in addition to graphics processing operations. The general-purpose logic may be coupled with… Figure 14 (Multiple) processor cores 1407 or Figure 15A The general logic within the cores 1502A-1502N performs processing operations in parallel or in combination.

[0290] Output data generated by a thread executing on the graphics core array 1714 can be output to memory in a uniform return buffer (URB) 1718. URB 1718 can store data for multiple threads. URB 1718 can be used to send data between different threads executing on the graphics core array 1714. URB 1718 can also be used for synchronization between threads on the graphics core array 1714 and fixed-function logic within shared-function logic 1720.

[0291] Optionally, the graphics core array 1714 can be scalable, such that the array includes a variable number of graphics cores, each with a variable number of execution units based on the target power and performance level of the GPE 1710. Execution resources can be dynamically scalable, allowing execution resources to be enabled or disabled as needed.

[0292] The graphics core array 1714 is coupled to shared function logic 1720, which includes multiple resources shared among the graphics cores in the graphics core array. The shared functions within the shared function logic 1720 are hardware logic units that provide specialized supplementary functions to the graphics core array 1714. In various embodiments, the shared function logic 1720 includes, but is not limited to, sampler logic 1721, mathematical logic 1722, and inter-thread communication (ITC) logic 1723. Additionally, one or more caches 1725 may be implemented within the shared function logic 1720.

[0293] At least in cases where the requirement for a given specialized function is insufficient to include it in the graphics core array 1714, shared functionality is implemented. Instead, a single instantiation of that specialized function is implemented as a separate entity in shared function logic 1720 and shared among execution resources within the graphics core array 1714. The exact set of functions shared between and included within the graphics core array 1714 varies depending on the embodiment. Specific shared functions widely used by the graphics core array 1714 within shared function logic 1720 may be included within shared function logic 1716 within the graphics core array 1714. Optionally, shared function logic 1716 within the graphics core array 1714 may include some or all of the logic within shared function logic 1720. In one embodiment, all logic elements within shared function logic 1720 may be replicated within shared function logic 1716 of the graphics core array 1714. Alternatively, shared function logic 1720 is excluded to favor shared function logic 1716 within the graphics core array 1714.

[0294] Execution unit

[0295] Figures 18A-18B The illustration shows thread execution logic 1800 according to an embodiment described herein, which includes an array of processing elements employed in a graphics processor core. Figures 18A-18B Elements having the same or similar names as elements in any other figure described herein are described as the same elements in other figures, and can operate or function in a manner similar to those described elsewhere herein. They may include the same components as those described elsewhere herein and may be linked to other entities as described elsewhere herein, but are not limited thereto. Figures 18A-18B The diagram illustrates an overview of thread execution logic 1800, which can be represented as... Figure 15B The hardware logic of each sub-core 1521A-1521F is illustrated in the diagram. Figure 18A This refers to the execution unit within a general-purpose graphics processing unit, while Figure 18B This indicates an execution unit that can be used within a computing accelerator.

[0296] As in Figure 18AAs illustrated, thread execution logic 1800 may include a shader processor 1802, a thread dispatcher 1804, an instruction cache 1806, a scalable execution unit array including multiple execution units 1808A-1808N, a sampler 1810, shared local memory 1811, a data cache 1812, and a data port 1814. Optionally, the scalable execution unit array can be dynamically scaled by enabling or disabling one or more execution units (e.g., any one of execution units 1808A, 1808B, 1808C, 1808D, up to 1808N-1 and 1808N) based on workload computational requirements. The included components may be interconnected via an interconnect structure linking each component within the array. The thread execution logic 1800 may include one or more connections to memory (such as system memory or cache memory) via instruction cache 1806, data port 1814, sampler 1810, and one or more of execution units 1808A-1808N. Each execution unit (e.g., 1808A) may be an independent, programmable, general-purpose computing unit capable of executing multiple synchronous hardware threads while processing multiple data elements in parallel for each thread. In various embodiments, the array of execution units 1808A-1808N is scalable to include any number of individual execution units.

[0297] Execution units 1808A-1808N can primarily be used to execute shader programs. Shader processor 1802 can handle various shader programs and can dispatch execution threads associated with shader programs via thread dispatcher 1804. The thread dispatcher may include logic for arbitrating thread initiation requests from the graphics pipeline and media pipeline and instantiating the requested thread on one or more execution units 1808A-1808N. For example, a geometry pipeline may dispatch vertex shaders, tessellation shaders, or geometry shaders to thread execution logic for processing. Optionally, thread dispatcher 1804 may also handle runtime thread generation requests from executing shader programs.

[0298] The 1808A-1808N execution units support instruction sets including native support for many standard 3D graphics shader instructions, enabling the execution of shader programs from graphics libraries (e.g., Direct3D and OpenGL) with minimal translation. These execution units support vertex and geometry processing (e.g., vertex programs, geometry programs, vertex shaders), pixel processing (e.g., pixel shaders, fragment shaders), and general-purpose processing (e.g., computation and media shaders). Each execution unit in the 1808A-1808N is capable of multi-issue single-instruction multiple-data (SIMD) execution, and multithreaded operation enables an efficient execution environment in the face of high-latency memory accesses. Each hardware thread within each execution unit has a dedicated high-bandwidth register file and associated independent thread state. For pipelines capable of integer operations, single-precision floating-point operations, and double-precision floating-point operations, capable of SIMD branching, capable of logical operations, capable of transcendental operations, and capable of other miscellaneous operations, execution is multi-issued per clock cycle. While waiting for data from memory or a shared function, the dependency logic within execution units 1808A-1808N causes the waiting thread to sleep until the requested data has been returned. While the waiting thread is sleeping, hardware resources can be dedicated to processing other threads. For example, during the latency associated with vertex shader operations, the execution unit can execute operations for pixel shaders, fragment shaders, or operations involving different vertex shaders (such as…). Figure 21 The diagram illustrates the operation of another type of shader program (vertex shader 2107). Various embodiments can be applied to use execution utilizing Single Instruction Multithreading (SIMT) as an alternative to SIMD use cases, or as an addition to SIMD use cases. References to SIMD kernels or operations can also be applied to SIMD, or in combination with SIMD.

[0299] Each execution unit in the 1808A-1808N operates on an array of data elements. The number of data elements is the "execution size," or the number of channels used for instructions. An execution channel is a logical unit used for data element access, masking, and flow control within instructions. The number of channels can be independent of the number of physical arithmetic logic units (ALUs), floating-point units (FPUs), or other logic units (e.g., tensor cores, ray tracing cores, etc.) used for a particular graphics processor. Furthermore, the 1808A-1808N execution units support both integer and floating-point data types.

[0300] The execution unit instruction set includes SIMD instructions. Various data elements can be stored in registers as compact data types, and the execution unit will process each element based on its data size. For example, when operating on a 256-bit wide vector, the 256 bits of the vector are stored in registers, and the execution unit will operate on the vector as four separate 64-bit compact data elements (quad-word (QW) size data elements), eight separate 32-bit compact data elements (double-word (DW) size data elements), sixteen separate 16-bit compact data elements (word (W) size data elements), or thirty-two separate 8-bit data elements (byte (B) size data elements). However, different vector widths and register sizes are possible.

[0301] Optionally, one or more execution units can be combined into fused execution units 1809A-1809N, which have common thread control logic (1807A-1807N) for the fused EUs. Multiple EUs can be fused into EU groups. Each EU in the fused EU group can be configured to execute a separate SIMD hardware thread. The number of EUs in the fused EU group can vary depending on the embodiment. Additionally, various SIMD widths, including but not limited to SIMD8, SIMD16, and SIMD32, can be executed on an EU-by-EU basis. Each fused graphics execution unit 1809A-1809N includes at least two execution units. For example, fused execution unit 1809A includes a first EU 1808A, a second EU 1808B, and common thread control logic 1807A for the first EU 1808A and the second EU 1808B. The thread control logic 1807A controls the threads executing on the fused graphics execution unit 1809A, thereby allowing each EU within the fused execution units 1809A-1809N to use a common instruction pointer register for execution.

[0302] One or more internal instruction caches (e.g., 1806) are included in thread execution logic 1800 to cache thread instructions for the execution unit. One or more data caches (e.g., 1812) may be included in thread execution logic 1800 to cache thread data during thread execution. Threads executing on execution logic 1800 may also store explicitly managed data in shared local memory 1811. Sampler 1810 may be included to provide texture sampling for 3D operations and media sampling for media operations. Sampler 1810 may include specialized texture or media sampling functions to process texture data or media data during the sampling process before providing sampled data to the execution unit.

[0303] During execution, the graphics pipeline and media pipeline send thread initiation requests to thread execution logic 1800 via thread generation and dispatch logic. Once a set of geometric objects has been processed and rasterized into pixel data, pixel processor logic (e.g., pixel shader logic, fragment shader logic, etc.) within shader processor 1802 is invoked to further compute output information and write the results to output surfaces (e.g., color buffer, depth buffer, stencil buffer, etc.). The pixel shader or fragment shader computes values ​​for individual vertex attributes, which are interpolated across the rasterized objects. The pixel processor logic within shader processor 1802 can then execute pixel shader or fragment shader programs provided by an application programming interface (API). To execute shader programs, shader processor 1802 dispatches threads to execution units (e.g., 1808A) via thread dispatcher 1804. Shader processor 1802 can use texture sampling logic in sampler 1810 to access texture data in a texture map stored in memory. Arithmetic operations on texture data and input geometry data compute pixel color data for each geometric fragment, or discard one or more pixels without further processing.

[0304] In addition, data port 1814 provides a memory access mechanism for thread execution logic 1800 to output processed data to memory for further processing on the graphics processor output pipeline. Data port 1814 may include or be coupled to one or more cache memories (e.g., data cache 1812) to cache data for memory access via data port 1814.

[0305] Optionally, the execution logic 1800 may also include a ray tracer 1805 that provides ray tracing acceleration. The ray tracer 1805 may support a ray tracing instruction set, which includes instructions / functions for ray generation. The ray tracing instruction set can be used with... Figure 3C The ray tracing instruction set supported by the ray tracing core 372 in the ray tracing kernel may be similar to or different from that of the ray tracing kernel.

[0306] Figure 18BThe illustration shows exemplary internal details of the graphics execution unit 1808. The graphics execution unit 1808 may include an instruction fetch unit 1837, a general-purpose register file array (GRF) 1824, an architecture register file array (ARF) 1826, a thread arbiter 1822, a send unit 1830, a branch unit 1832, a set of SIMD floating-point units (FPUs) 1834, and optionally a set of dedicated integer SIMD ALUs 1835. The GRF 1824 and ARF 1826 include sets of general-purpose register files and architecture register files associated with each synchronous hardware thread active in the graphics execution unit 1808. Per-thread architecture state may be maintained in the ARF 1826, while data used during thread execution is stored in the GRF 1824. The execution state of each thread, including the instruction pointer for each thread, may be maintained in thread-specific registers in the ARF 1826.

[0307] The graphics execution unit 1808 may have an architecture that is a combination of synchronous multithreading (SMT) and fine-grained interleaved multithreading (IMT). This architecture may have a modular configuration that can be fine-tuned at design time based on the target number of synchronous threads and the number of registers per execution unit, where execution unit resources are partitioned across logic used to execute multiple synchronous threads. The number of logical threads that can be executed by the graphics execution unit 1808 is not limited to the number of hardware threads, and multiple logical threads can be assigned to each hardware thread.

[0308] Optionally, the graphics execution unit 1808 can collaboratively issue multiple instructions, which can each be different instructions. The thread arbiter 1822 of the graphics execution unit thread 1808 can dispatch instructions to one of the following for execution: sending unit 1830, branching unit 1832, or (multiple) SIMD FPUs 1834. Each execution thread can access 128 general-purpose registers within the GRF 1824, where each register can store 32 bytes accessible as a SIMD 8-element vector with 32-bit data elements. Each execution unit thread can have access to 4 kilobytes within the GRF 1824, but embodiments are not limited thereto, and more or fewer register resources may be provided in other embodiments. The graphics execution unit 1808 can be partitioned into seven hardware threads capable of independently performing computational operations, but the number of threads per execution unit may also vary depending on the embodiment; for example, up to 16 hardware threads may be supported. In an exemplary embodiment where seven threads can access 4 kilobytes, the GRF 1824 can store a total of 28 kilobytes. In another exemplary embodiment where 16 threads can access 4 kilobytes, the GRF 1824 can store a total of 64 kilobytes. However, the number of threads per execution unit is not limited to those examples and can be greater or less than the given number. Flexible addressing modes can allow addressing of multiple registers together, thereby creating practically wider registers or representing straddle-shaped rectangular block data structures.

[0309] Additionally or alternatively, memory operations, sampler operations, and other long-latency system communications can be dispatched via "send" instructions executed through message sending unit 1830. Branch instructions can be dispatched to dedicated branch unit 1832 to facilitate SIMD divergence and eventual convergence.

[0310] The graphics execution unit 1808 may include one or more SIMD floating-point units (FPUs) 1834 for performing floating-point operations. The multiple FPUs 1834 may also support integer computation. In some instances, the multiple FPUs 1834 may perform up to M 32-bit floating-point (or integer) operations in SIMD, or up to 2M 16-bit integer or 16-bit floating-point operations in SIMD. Optionally, at least one of the multiple FPUs may provide extended mathematical capabilities supporting high throughput beyond mathematical functions and double-precision 64-bit floating-point. A set 1835 of 8-bit integer SIMD ALUs may also be present and may be specifically optimized to perform operations associated with machine learning computations.

[0311] Optionally, an array of multiple instances of the graphics execution unit 1808 can be instantiated within a graphics subcore group (e.g., a subslice). For scalability, the product architect can choose the exact number of execution units per subcore group. The execution unit 1808 can execute instructions across multiple execution channels. Furthermore, each thread executing on the graphics execution unit 1808 can be executed on a different channel.

[0312] Figure 19 The figure shows a further exemplary execution unit 1900. Figure 19 Elements having the same or similar names as elements in any other figure herein describe the same elements in other figures, operate or function in a manner similar to those described elsewhere herein, may include the same components as those described elsewhere herein, and may be linked to other entities as described elsewhere herein, but are not limited thereto. Execution unit 1900 may be used for, for example... Figure 16C The computationally optimized execution unit used in the computation engine chips 340A-340D, but not limited to this, is also present. Execution unit 1900 can also be used in, for example... Figure 16B The graphics engine chip 310A-310D is used in the system. The execution unit 1900 may include a thread control unit 1901, a thread status unit 1902, an instruction fetch / prefetch unit 1903, and an instruction decoding unit 1904. The execution unit 1900 may additionally include a register file 1906, which stores registers that can be assigned to hardware threads within the execution unit. The execution unit 1900 may additionally include a send unit 1907 and a branch unit 1908. The send unit 1907 and the branch unit 1908 can communicate with… Figure 18B The sending unit 1830 and branching unit 1832 of the graphics execution unit 1808 operate in a similar manner.

[0313] The execution unit 1900 may further include a computation unit 1910, which includes multiple functional units of different types. The computation unit 1910 may also include an ALU unit 1911, which includes an array of arithmetic logic units. The ALU unit 1911 can be configured to perform 64-bit, 32-bit, and 16-bit integer and floating-point operations. Integer and floating-point operations can be performed simultaneously. The computation unit 1910 may also include a systolic array 1912 and a math unit 1913. The systolic array 1912 includes a wide-W-deep-D network of data processing units, which can be used to perform vector or other data-parallel operations in a systolic manner. The systolic array 1912 can be configured to perform matrix operations, such as matrix dot product operations. The systolic array 1912 can support 16-bit floating-point operations as well as 8-bit and 4-bit integer operations. The systolic array 1912 can be configured to accelerate machine learning operations. The systolic array 1912 can be configured to support the bfloat16 16-bit floating-point format. Mathematics unit 1913 may be included to perform mathematical operations in a more efficient and lower-powered manner than ALU unit 1911. Mathematics unit 1913 may include mathematical logic found in the shared functional logic of graphics processing engines provided by other embodiments described, for example, Figure 17 The shared functional logic 1720 contains mathematical logic 1722. Mathematical unit 1913 can be configured to perform 32-bit and 64-bit floating-point operations.

[0314] The thread control unit 1901 includes logic for controlling the execution of threads within the execution unit. The thread control unit 1901 may include thread arbitration logic for starting, stopping, and preempting the execution of threads within the execution unit 1900. The thread state unit 1902 can be used to store the thread states of threads assigned to execution on the execution unit 1900. Storing the thread states in the execution unit 1900 enables the rapid preemption of threads when they become locked or idle. The instruction fetch / prefetch unit 1903 can retrieve instructions from a higher-level execution logic cache (e.g., such as...). Figure 18A The instruction fetch / prefetch unit 1903 fetches instructions from the instruction cache 1806. The instruction fetch / prefetch unit 1903 also issues prefetch requests for instructions to be loaded into the execution cache based on analysis of the currently executing thread. The instruction decoding unit 1904 can be used to decode instructions to be executed by the computation unit. The instruction decoding unit 1904 can be used as a secondary decoder to decode complex instructions into constituent micro-operations.

[0315] Execution unit 1900 additionally includes register file 1906, which can be used by hardware threads executing on execution unit 1900. Registers in register file 1906 can be partitioned across logic used for executing multiple synchronized threads within computing unit 1910 of execution unit 1900. The number of logical threads that can be executed by graphics execution unit 1900 is not limited to the number of hardware threads, and multiple logical threads can be assigned to each hardware thread. The size of register file 1906 may vary depending on the embodiment, based on the number of supported hardware threads. Register renaming can be used to dynamically allocate registers to hardware threads.

[0316] Figure 20 This is a block diagram illustrating the graphics processor instruction format 2000. The graphics processor execution unit supports instruction sets with various formats. Solid lines represent components typically included in the execution unit instructions, while dashed lines represent optional components or those only included in a subset of the instructions. The instruction format 2000 described and illustrated are macro instructions because they are instructions supplied to the execution unit, as opposed to micro instructions generated from instruction decoding performed once the instruction is processed.

[0317] The graphics processing unit execution unit described herein natively supports instructions in 128-bit instruction format 2010. A 64-bit compact instruction format 2030 is available for some instructions, depending on the selected instructions, instruction options, and number of operands. The native 128-bit instruction format 2010 provides access to all instruction options, while some options and operations are limited in the 64-bit format 2030. The native instructions available in the 64-bit format 2030 vary depending on the embodiment. Instructions are partially compressed using a set of index values ​​in the index field 2013. The execution unit hardware references a set of compression tables based on the index values ​​and uses the output of the compression tables to reconstruct the native instructions of the 128-bit instruction format 2010. Other sizes and formats of instructions may be used.

[0318] For each format, the instruction opcode 2012 defines the operation to be performed by the execution unit. The execution unit executes each instruction in parallel across multiple data elements of each operand. For example, in response to an addition instruction, the execution unit performs a synchronous addition operation across each color channel representing a texture element or image element. By default, the execution unit executes each instruction across all data channels of the operand. The instruction control field 2014 enables control over certain execution options, such as channel selection (e.g., assertion) and data channel order (e.g., blending). For instructions in the 128-bit instruction format 2010, the execution size field 2016 limits the number of data channels that will be executed in parallel. The execution size field 2016 may not be available for the 64-bit compact instruction format 2030.

[0319] Some execution unit instructions have up to three operands, including two source operands (src0 2020, src1 2022) and a destination operand (2018). The execution unit may support dual-destination instructions, where one of the two destinations is implicit. Data manipulation instructions may have a third source operand (e.g., SRC2 2024), where the instruction opcode 2012 determines the number of source operands. The last source operand of an instruction may be an immediate value (e.g., hard-coded) passed with the instruction.

[0320] The 128-bit instruction format 2010 may include an access / addressing mode field 2026, which, for example, specifies whether to use direct register addressing mode or indirect register addressing mode. When using direct register addressing mode, the register addresses of one or more operands are directly provided by bits in the instruction.

[0321] The 128-bit instruction format 2010 may also include an access / addressing mode field 2026, which specifies the addressing mode and / or access mode of the instruction. The access mode can be used to define the data access alignment for the instruction. Access modes including 16-byte aligned access modes and 1-byte aligned access modes are supported, where the byte alignment of the access mode determines the access alignment of the instruction operands. For example, in the first mode, the instruction can use byte-aligned addressing for both the source and destination operands, and in the second mode, the instruction can use 16-byte aligned addressing for all source and destination operands.

[0322] The addressing mode portion of the access / addressing mode field 2026 determines whether the instruction should use direct or indirect addressing. When using direct register addressing mode, bits in the instruction directly provide the register addresses of one or more operands. When using indirect register addressing mode, the register addresses of one or more operands can be calculated based on the address register value and the address immediate field in the instruction.

[0323] Instructions can be grouped based on the 2012-bit opcode field to simplify opcode decoding 2040. For an 8-bit opcode, bits 4, 5, and 6 allow the execution unit to determine the opcode type. The exact opcode groupings shown are merely examples. Move and logic opcode group 2042 can include data move and logic instructions (e.g., move (mov), compare (cmp)). Move and logic group 2042 can share five most significant bits (MSB), where move (mov) instructions are in the form of 0000xxxxb, and logic instructions are in the form of 0001xxxxb. Flow control instruction group 2044 (e.g., call, jump (jmp)) includes instructions in the form of 0010xxxxb (e.g., 0x20). Miscellaneous instruction group 2046 includes a mixture of instructions, including synchronous instructions in the form of 0011xxxxb (e.g., 0x30) (e.g., wait, send)). Parallel math group 2048 includes component-wise arithmetic instructions (e.g., addition, multiplication (mul)) in the form of 0100xxxxb (e.g., 0x40). Parallel math instruction group 2048 performs arithmetic operations in parallel across data channels. Vector math group 2050 includes arithmetic instructions (e.g., dp4) in the form of 0101xxxxb (e.g., 0x50). Vector math group performs arithmetic on vector operands, such as dot product calculations. In one embodiment, the illustrated opcode decoder 2040 can be used to determine which part of the execution unit will be used to execute the decoded instructions. For example, some instructions may be specified as systolic instructions to be executed by the systolic array. Other instructions, such as ray tracing instructions (not shown), may be routed to a ray tracing kernel or ray tracing logic within a slice or partition of the execution logic.

[0324] Graphics Pipeline

[0325] Figure 21 This is a block diagram of a graphics processor 2100 according to another embodiment. Figure 21 Elements having the same or similar names as elements in any other figure described herein are described as the same elements in other figures, and can operate or function in a manner similar to those described elsewhere herein. They may include the same components as those described elsewhere herein and may be linked to other entities as described elsewhere herein, but are not limited thereto.

[0326] The graphics processor 2100 may include different types of graphics processing pipelines, such as a geometry pipeline 2120, a media pipeline 2130, a display engine 2140, thread execution logic 2150, and a rendering output pipeline 2170. The graphics processor 2100 may be a graphics processor within a multi-core processing system including one or more general-purpose processing cores. The graphics processor can be controlled by writing to registers in one or more control registers (not shown) or by commands issued to the graphics processor 2100 via a ring interconnect 2102. The ring interconnect 2102 may couple the graphics processor 2100 to other processing components, such as other graphics processors or general-purpose processors. Commands from the ring interconnect 2102 are interpreted by a command stream converter 2103, which supplies instructions to the various components of the geometry pipeline 2120 or the media pipeline 2130.

[0327] Command stream converter 2103 guides the operation of vertex extractor 2105, which reads vertex data from memory and executes vertex processing commands provided by command stream converter 2103. Vertex extractor 2105 can provide vertex data to vertex shader 2107, which performs coordinate space transformation and lighting operations on each vertex. Vertex extractor 2105 and vertex shader 2107 can execute vertex processing instructions by dispatching execution threads to execution units 2152A-2152B via thread dispatcher 2131.

[0328] Execution units 2152A-2152B may be arrays of vector processors having instruction sets for performing graphics and media operations. Execution units 2152A-2152B may have attached L1 caches 2151 dedicated to each array or shared between arrays. The caches may be configured as data caches, instruction caches, or partitioned into single caches containing data and instructions in different partitions.

[0329] The geometry pipeline 2120 may include a tessellation component for performing hardware-accelerated tessellation of 3D objects. A programmable shell shader 2111 is configurable for tessellation operations. A programmable domain shader 2117 provides back-end evaluation of the tessellation output. A tessellation 2113 operates under the direction of the shell shader 2111 and includes dedicated logic for generating a detailed set of geometric objects based on a coarse geometric model provided as input to the geometry pipeline 2120. Furthermore, the tessellation components (e.g., shell shader 2111, tessellation 2113, and domain shader 2117) can be bypassed if tessellation is not used.

[0330] The complete geometry object can be processed by geometry shader 2119 via one or more threads dispatched to execution units 2152A-2152B, or it can proceed directly to trimmer 2129. The geometry shader can operate on the entire geometry object rather than on vertices or vertex patches as in previous stages of the graphics pipeline. If tessellation is disabled, geometry shader 2119 receives input from vertex shader 2107. Geometry shader 2119 can be programmable by the geometry shader program to perform geometric tessellation even when the tessellation unit is disabled.

[0331] Prior to rasterization, clipper 2129 processes vertex data. Clipper 2129 can be a fixed-function clipper or a programmable clipper with clipping and geometry shader capabilities. The rasterizer and depth test component 2173 in the rendering output pipeline 2170 can dispatch pixel shaders to convert geometric objects into a per-pixel representation. Pixel shader logic can be included in thread execution logic 2150. Optionally, the application can bypass the rasterizer and depth test component 2173 and access unerasterized vertex data via outgoing unit 2123.

[0332] The graphics processor 2100 has an interconnect bus, interconnect structure, or some other interconnect mechanism that allows data and messages to be passed between the main components of the processor. In some embodiments, execution units 2152A-2152B and associated logic units (e.g., L1 cache 2151, sampler 2154, texture cache 2158, etc.) are interconnected via data port 2156 to perform memory accesses and communicate with the processor's rendering output pipeline components. Sampler 2154, caches 2151, 2158, and execution units 2152A-2152B may each have a separate memory access path. Optionally, texture cache 2158 may also be configured as a sampler cache.

[0333] The rendering output pipeline 2170 may include a rasterizer and a depth testing component 2173, which converts vertex-based objects into associated pixel-based representations. The rasterizer logic may include windower / masker units for performing fixed-function triangle and line rasterization. Associated rendering cache 2178 and depth cache 2179 are also available in some embodiments. Pixel manipulation component 2177 performs pixel-based operations on the data; however, in some instances, pixel operations associated with 2D operations (e.g., using mixed bit-block image transfer) are performed by the 2D engine 2141, or alternatively by the display controller 2143 using an overlay display plane during display. A shared L3 cache 2175 may be available for all graphics components, allowing data to be shared without using main system memory.

[0334] The graphics processor media pipeline 2130 may include a media engine 2137 and a video front-end 2134. The video front-end 2134 may receive pipeline commands from a command stream converter 2103. The media pipeline 2130 may include a separate command stream converter. The video front-end 2134 may process the media command before sending it to the media engine 2137. The media engine 2137 may include thread generation functionality for generating threads for dispatch to thread execution logic 2150 via a thread dispatcher 2131.

[0335] The graphics processor 2100 may include a display engine 2140. The display engine 2140 may be external to the processor 2100 and may be coupled to the graphics processor via a ring interconnect 2102 or some other interconnect bus or structure. The display engine 2140 may include a 2D engine 2141 and a display controller 2143. The display engine 2140 may contain dedicated logic capable of operating independently of the 3D pipeline. The display controller 2143 may be coupled to a display device (not shown), which may be a system-integrated display device (such as in a laptop computer) or an external display device attached via a display device connector.

[0336] The geometry pipeline 2120 and media pipeline 2130 may be configured to perform operations based on multiple graphics and media programming interfaces (APIs) and are not dedicated to any single application programming interface (API). The graphics processor's driver software can translate API calls specific to a particular graphics or media library into commands that can be processed by the graphics processor. Support is available for all OpenGL, OpenCL, and / or Vulkan graphics and computing APIs from the Khronos Group. Support is also available for Microsoft's Direct3D library. Combinations of these libraries are supported. Support is also available for the open-source computer vision library OpenCV. Future APIs with 3D-compatible pipelines will also be supported if a pipeline mapping from future APIs to the graphics processor's pipeline is possible.

[0337] Graphical Pipeline Programming

[0338] Figure 22A This diagram illustrates the use of graphics processing pipelines (such as, for example, those incorporated herein). Figure 16A , Figure 17 , Figure 21 A block diagram describing the command format 2200 for pipelined programming of graphics processors. Figure 22B This is a block diagram illustrating a graphics processor command sequence 2210 according to an embodiment. Figure 22ASolid-line diagrams are generally included as components in the drawing command, while dashed lines include optional components or components that are only included in a subset of the drawing command. Figure 22A An exemplary graphics processor command format 2200 includes a client 2202 for identifying a command, a command opcode (opcode) 2204, and data 2206. Sub-opcodes 2205 and command dimensions 2208 are also included in some commands.

[0339] Client 2202 can specify the client unit for processing command data of the graphics device. The graphics processor command parser can examine the client field of each command to adjust further processing of the command and route the command data to the appropriate client unit. The graphics processor client unit may include a memory interface unit, a rendering unit, a 2D unit, a 3D unit, and a media unit. Each client unit may have a corresponding processing pipeline for processing commands. Once a command is received by a client unit, the client unit reads the opcode 2204 and the sub-opcode 2205 (if present) to determine the operation to be performed. The client unit uses the information in the data field 2206 to execute the command. For some commands, the explicit command size 2208 is expected to specify the size of the command. The command parser can automatically determine the size of at least some commands in the command based on the command opcode. Commands can be aligned via multiples of double words. Other command formats can also be used.

[0340] Figure 22B The flowchart illustrates an exemplary graphics processor command sequence 2210. Software or firmware of a data processing system characterized by an exemplary graphics processor may use a version of the illustrated command sequence to establish, execute, and terminate a set of graphics operations. The sample command sequence is shown and described for illustrative purposes only and is not limited to these specific commands or the command sequence. Moreover, commands may be issued as a batch of commands in a command sequence, such that the graphics processor will process the command sequence in a manner that is at least partially simultaneous.

[0341] The graphics processor command sequence 2210 may begin with a pipeline dump clearing command 2212 to allow any active graphics pipeline to complete its currently pending commands. Optionally, the 3D pipeline 2222 and the media pipeline 2224 may not be able to operate concurrently. Pipeline dump clearing is performed to allow any pending commands from active graphics pipelines to complete. In response to pipeline dump clearing, the command parser for the graphics processor will suspend command processing until the active painting engine completes its pending operations and the associated read cache is invalidated. Optionally, any data marked as "dirty" in the render cache may be dumped and cleared to memory. Pipeline dump clearing command 2212 may be used for pipeline synchronization or before placing the graphics processor in a low-power state.

[0342] Pipeline selection command 2213 can be used when a sequence of commands requires the graphics processor to explicitly switch between pipelines. Only one pipeline selection command 2213 is needed in the execution context before issuing a pipeline command, unless the context will issue commands for two pipelines. A pipeline dump clearing command 2212 may be required immediately preceding the pipeline switch via pipeline selection command 2213.

[0343] Pipeline control command 2214 can configure the graphics pipeline for operation and can be used to program the 3D pipeline 2222 and the media pipeline 2224. Pipeline control command 2214 can configure the pipeline state of the active pipeline. Pipeline control command 2214 can be used for pipeline synchronization and to clear data from one or more cache memories within the active pipeline before processing a batch of commands.

[0344] The Return Buffer Status command 2216 can be used to configure the set of return buffers used for a given pipeline to write data. Some pipelined operations require allocating, selecting, or configuring one or more return buffers, in which the operation writes intermediate data during processing. The graphics processor may also use one or more return buffers to store output data and perform cross-thread communication. Return Buffer Status 2216 may include selecting the size and number of return buffers for the set to be used in the pipelined operation.

[0345] The remaining commands in the command sequence differ based on the active pipeline used for the operation. Based on pipeline determination 2220, the command sequence is tailored for either the 3D pipeline 2222 starting at 3D pipeline state 2230, or the media pipeline 2224 starting at media pipeline state 2240.

[0346] Commands for configuring 3D pipeline states 2230 include 3D state setting commands for vertex buffer states, vertex element states, constant color states, depth buffer states, and other state variables that will be configured before processing 3D primitive commands. The values ​​of these commands are determined at least in part based on the specific 3D API in use. In some embodiments, the 3D pipeline state 2230 commands can also selectively disable or bypass certain pipeline elements if they will not be used.

[0347] The 3D Primitive 2232 command can be used to submit 3D primitives for processing by the 3D pipeline. The commands and associated parameters passed to the graphics processor via the 3D Primitive 2232 command are forwarded to the vertex fetch function in the graphics pipeline. The vertex fetch function uses the 3D Primitive 2232 command data to generate multiple vertex data structures. The vertex data structures are stored in one or more return buffers. The 3D Primitive 2232 command can be used to perform vertex operations on the 3D primitives via the vertex shader. To process the vertex shader, the 3D pipeline 2222 dispatches shader execution threads to the graphics processor execution unit.

[0348] The 3D pipeline 2222 can be triggered by executing command 2234 or an event. Registers can be written to trigger command execution. Execution can be triggered via the "go" or "kick" command in a command sequence. Pipeline synchronization commands can be used to trigger command execution so that the command sequence can be dumped and cleared through the graphics pipeline. The 3D pipeline will perform geometry processing on 3D primitives. Once the operation is complete, the resulting geometry is rasterized, and the pixel engine colors the resulting pixels. Additional commands for controlling pixel coloring and pixel backend operations can also be included for those operations.

[0349] When performing media operations, the graphics processor command sequence 2210 may follow the media pipeline 2224 path. Generally, the specific purpose and manner in which the media pipeline 2224 is programmed depends on the media or computational operation to be performed. During media decoding, specific media decoding operations may be diverted to the media pipeline. In some embodiments, the media pipeline may also be bypassed, and media decoding may be performed wholly or partially using resources provided by one or more general-purpose processing cores. The media pipeline may also include elements for general-purpose graphics processing unit (GPGPU) operations, wherein the graphics processor is used to perform SIMD vector operations using computational shader programs that are not explicitly associated with the rendering of graphics primitives.

[0350] The media pipeline 2224 can be configured in a similar manner to the 3D pipeline 2222. A set of commands for configuring media pipeline states 2240 is dispatched or placed in a command queue, preceding the media object commands 2242. Commands 2240 for media pipeline states may include data for configuring media pipeline elements that will be used to process media objects. This includes data for configuring video decoding and video encoding logic within the media pipeline, such as encoding or decoding formats. Commands 2240 for media pipeline states may also support the use of one or more pointers to "indirect" state elements containing batch state settings.

[0351] Media object command 2242 provides a pointer to a media object to be processed by the media pipeline. The media object includes a memory buffer containing video data to be processed. Optionally, all media pipeline states must be valid before issuing media object command 2242. Once the pipeline states are configured and media object command 2242 is queued, media pipeline 2224 is triggered via execution command 2244 or an equivalent execution event (e.g., register write). Post-processing of the output from media pipeline 2224 can then be performed via operations provided by 3D pipeline 2222 or media pipeline 2224. GPGPU operations can be configured and executed in a manner similar to media operations.

[0352] Graphical software architecture

[0353] Figure 23 The illustration depicts an exemplary graphics software architecture for a data processing system 2300. Such a software architecture may include a 3D graphics application 2310, an operating system 2320, and at least one processor 2330. The processor 2330 may include a graphics processor 2332 and one or more general-purpose processor cores 2334. The processor 2330 may be a variant of processor 1402, or any other processor described herein. The processor 2330 may be used in place of processor 1402 or any other processor described herein. Therefore, the disclosure of any feature in combination with processor 1402 or any other processor described herein also discloses corresponding combinations with graphics processor 1330, but is not limited thereto. Furthermore, Figure 23 Elements having the same or similar names as elements in any other figure herein are described as the same elements in other figures, and can operate or function in a manner similar to those described elsewhere herein. They may include, but are not limited to, components identical to those described elsewhere herein, and may be linked to other entities such as those described elsewhere herein. The graphics application 2310 and the operating system 2320 are each executed in the system memory 2350 of the data processing system.

[0354] The 3D graphics application 2310 may include one or more shader programs, which include shader instructions 2312. The shader language instructions may be in a high-level shader language, such as DirectD's High-Level Shading Language (HLSL), OpenGL Shading Language (GLSL), etc. The application may also include executable instructions 2314 in machine language suitable for execution by a general-purpose processor core 2334. The application may also include graphics objects 2316 defined by vertex data.

[0355] The operating system 2320 could be from Microsoft. The operating system 2320 can be a proprietary Unix-like operating system or an open-source Unix-like operating system using a variant of the Linux kernel. The operating system 2320 may support graphics APIs 2322, such as the Direct3D API, OpenGL API, or Vulkan API. When the Direct3D API is in use, the operating system 2320 uses a front-end shader compiler 2324 to compile any shader instructions 2312 employing HLSL into a lower-level shader language. Compilation can be just-in-time (JIT) compilation or pre-compilation of the application executable shader. During the compilation of the 3D graphics application 2310, high-level shaders can be compiled into low-level shaders. Shader instructions 2312 can be provided in an intermediate form, such as a version of the standard portable intermediate representation (SPIR) used by the Vulkan API.

[0356] User-mode graphics driver 2326 may include back-end shader compiler 2327, which translates shader instructions 2312 into a hardware-specific representation. When the OpenGL API is in use, shader instructions 2312 in the GLSL high-level language are passed to user-mode graphics driver 2326 for compilation. User-mode graphics driver 2326 may use operating system kernel-mode functionality 2328 to communicate with kernel-mode graphics driver 2329. Kernel-mode graphics driver 2329 may communicate with graphics processor 2332 to dispatch commands and instructions.

[0357] IP core implementation

[0358] One or more aspects may be implemented by representative code stored on a machine-readable medium that represents and / or defines logic within an integrated circuit (such as a processor). For example, the machine-readable medium may include instructions representing various logics within a processor. When read by a machine, the instructions can cause the machine to manufacture logic for performing the techniques described herein. Such representations (referred to as “IP cores”) are reusable units of logic for an integrated circuit that may be stored on a tangible, machine-readable medium as a hardware model describing the structure of the integrated circuit. The hardware model may be supplied to various consumers or manufacturing facilities that load the hardware model onto a manufacturing machine for manufacturing the integrated circuit. Integrated circuits may be manufactured such that the circuit performs the operations described in association with any of the embodiments described herein.

[0359] Figure 24AThis is a block diagram illustrating an IP core development system 2400 according to an embodiment, which can be used to manufacture integrated circuits to perform operations. The IP core development system 2400 can be used to generate modular, reusable designs that can be incorporated into larger designs or used to build entire integrated circuits (e.g., SOC integrated circuits). Design facility 2430 can generate software simulation 2410 of the IP core design using a high-level programming language (e.g., C / C++). Software simulation 2410 can be used to design, test, and verify the behavior of the IP core using simulation model 2412. Simulation model 2412 can include functional simulation, behavioral simulation, and / or timing simulation. Register transfer level (RTL) design 2415 can then be created or synthesized from simulation model 2412. RTL design 2415 is an abstraction of the behavior of an integrated circuit (including associated logic performed using the modeled digital signals) that models the flow of digital signals between hardware registers. In addition to RTL design 2415, lower-level designs at the logic level or transistor level can also be created, designed, or synthesized. Thus, the specific details of the initial design and simulation can differ.

[0360] The RTL design 2415 or an equivalent can be further synthesized into a hardware model 2420 by the design facility. This hardware model 2420 may employ a hardware description language (HDL) or some other representation of the physical design data. The HDL can be further simulated or tested to validate the IP core design. The IP core design can be stored in non-volatile memory 2440 (e.g., hard disk, flash memory, or any non-volatile storage medium) for delivery to a third-party manufacturing facility 2465. Alternatively, the IP core design can be transmitted via a wired connection 2450 or a wireless connection 2460 (e.g., via the Internet). The manufacturing facility 2465 can then fabricate an integrated circuit at least partially based on the IP core design. The fabricated integrated circuit can be configured to perform operations according to at least one embodiment described herein.

[0361] Figure 24BThe figure shows a cross-sectional side view of integrated circuit package assembly 2270. Integrated circuit package assembly 2470 illustrates an implementation of one or more processor or accelerator devices as described herein. Package assembly 2470 includes a plurality of hardware logic units 2472, 2474 connected to substrate 2480. Logic units 2472, 2474 may be at least partially implemented in configurable logic or fixed-function logic hardware and may include any one or more portions of the processor core(s), graphics processor(s), or other accelerator devices described herein. Each logic unit 2472, 2474 may be implemented within a semiconductor die and coupled to substrate 2480 via interconnect structure 2473. Interconnect structure 2473 may be configured to route electrical signals between logic units 2472, 2474 and substrate 2480 and may include interconnects such as, but not limited to, bumps or pillars. Interconnect structure 2473 can be configured to route electrical signals, such as, for example, input / output (I / O) signals and / or power or ground signals associated with the operation of logic 2472, 2474. Optionally, substrate 2480 can be an epoxy-based laminated substrate. Package substrate 2480 may also include other suitable types of substrates. Package assembly 2470 can be connected to other electrical devices via package interconnect 2483. Package interconnect 2483 can be coupled to the surface of substrate 2480 to route electrical signals to other electrical devices, such as motherboards, other chipsets, or multi-chip modules.

[0362] Logic cells 2472 and 2474 may be electrically coupled to bridge 2482, which is configured to route electrical signals between logic cells 2472 and 2474. Bridge 2482 may be a dense interconnect structure that provides routing for electrical signals. Bridge 2482 may include a bridge substrate made of glass or a suitable semiconductor material. Circuitry may be formed on the bridge substrate to provide chip-to-chip connectivity between logic cells 2472 and 2474.

[0363] Although two logic units 2472, 2474 and bridge 2482 are illustrated, the embodiments described herein may include more or fewer logic units on one or more dies. These one or more dies may be connected by zero or more bridges, as bridge 2482 can be excluded when logic is included on a single die. Alternatively, multiple dies or logic units may be connected by one or more bridges. Furthermore, in other possible configurations (including three-dimensional configurations), multiple logic units, dies, and bridges may be connected together.

[0364] Figure 24CThe illustration shows a package assembly 2490 that includes hardware logic chiplets (e.g., base dies) connected to multiple units of a substrate 2480. Graphics processing units, parallel processors, and / or computing accelerators, as described herein, can be composed of various silicon chiplets manufactured separately. In this context, a chiplet is an integrated circuit that is at least partially packaged, comprising different logic units that can be assembled together with other chiplets into a larger package. Various collections of chiplets with different IP core logic can be assembled into a single device. Furthermore, active interposer technologies can be used to integrate chiplets into base dies or base chiplets. The concepts described herein enable interconnection and communication between different forms of IP within a GPU. IP cores can be fabricated using different process technologies and assembled during manufacturing, avoiding the complexity of aggregating multiple IPs into the same manufacturing process, especially for large SoCs with several styles of IP. Allowing the use of multiple process technologies improves time-to-market and provides a cost-effective approach to creating multiple product SKUs. Furthermore, the decomposed IP is better suited for independent power gating, which can shut down components that are not used in a given workload, thereby reducing overall power consumption.

[0365] The hardware logic chiplet may include a dedicated hardware logic chiplet 2472, a logic or I / O chiplet 2474, and / or a memory chiplet 2475. The hardware logic chiplet 2472 and the logic or I / O chiplet 2474 may be at least partially implemented in configurable logic or fixed-function logic hardware, and may include one or more portions of any of the processor cores(multiple) processor cores(multiple) graphics processors(multiple) parallel processors(multiple) or other accelerator devices described herein. The memory chiplet 2475 may be DRAM (e.g., GDDR, HBM) memory or SRAM cache memory.

[0366] Each chiplet may be fabricated as an individual semiconductor die and coupled to a substrate 2480 via an interconnect structure 2473. The interconnect structure 2473 may be configured to route electrical signals between the various chiplets and logic within the substrate 2480. The interconnect structure 2473 may include interconnects, such as, but not limited to, bumps or pillars. In some embodiments, the interconnect structure 2473 may be configured to route electrical signals, such as, for example, input / output (I / O) signals and / or power signals or ground signals associated with the operation of logic chiplets, I / O chiplets, and memory chiplets.

[0367] Substrate 2480 may be an epoxy-based laminated substrate; however, it is not limited to this, and substrate 2480 may also include other suitable types of substrates. Package assembly 2490 may be connected to other electrical devices via package interconnect 2483. Package interconnect 2483 may be coupled to the surface of substrate 2480 to route electrical signals to other electrical devices, such as motherboards, other chipsets, or multi-chip modules.

[0368] Logic or I / O chiplet 2474 and memory chiplet 2475 may be electrically coupled via bridge 2487, which is configured to route electrical signals between the logic or I / O chiplet 2474 and the memory chiplet 2475. Bridge 2487 may be a dense interconnect structure that provides routing for electrical signals. Bridge 2487 may include a bridge substrate made of glass or a suitable semiconductor material. Circuitry may be formed on the bridge substrate to provide chip-to-chip connectivity between the logic or I / O chiplet 2474 and the memory chiplet 2475. Bridge 2487 may also be referred to as a silicon bridge or an interconnect bridge. For example, bridge 2487 is an embedded multi-die interconnect bridge (EMIB). Alternatively, bridge 2487 may simply be a direct connection from one chiplet to another.

[0369] Substrate 2480 may include hardware components for I / O 2491, cache memory 2492, and other hardware logic 2493. Structure 2485 may be embedded in substrate 2480 to enable communication between various logic chips within substrate 2480 and logic 2491, 2493. Optionally, I / O 2491, structure 2485, cache, bridge, and other hardware logic 2493 may be integrated in a base die stacked on top of substrate 2480. Structure 2485 may be an on-chip network interconnect for exchanging data packets between components of a packaged assembly or another form of packet-switched structure.

[0370] In addition, package assembly 2490 may also include fewer or more components and chiplets interconnected by structure 2485 or one or more bridges 2487. The chiplets within package assembly 2490 can be arranged in a 3D or 2.5D configuration. Generally, bridge structure 2487 can be used to facilitate point-to-point interconnections, for example, between logic or I / O chiplets and memory chiplets. Structure 2485 can be used to interconnect various logic and / or I / O chiplets (e.g., chiplets 2472, 2474, 2491, 2493) with other logic and / or I / O chiplets. Cache memory 2492 within the substrate can act as a global cache for package assembly 2490, as part of a distributed global cache, or as a dedicated cache for structure 2485.

[0371] Figure 24D The illustration shows a package assembly 2494 including an interchangeable chiplet 2495 according to an embodiment. The interchangeable chiplet 2495 can be assembled into a standardized slot on one or more base chipsets 2496, 2498. The base chipsets 2496, 2498 can be coupled via a bridge interconnect 2497, which can be similar to other bridge interconnects described herein and can be, for example, EMIB. Memory chipsets can also be connected to logic or I / O chipsets via bridge interconnects. I / O and logic chipsets can communicate via interconnect structures. Each base chiplet can support one or more slots in a standardized format for logic or I / O or memory / cache.

[0372] SRAM and power delivery circuitry can be fabricated in one or more of base chips 2496 and 2498, which can be fabricated using different process technologies than the interchangeable chiplet 2495, which is stacked on top of the base chips. For example, the base chips 2496 and 2498 can be fabricated using a larger process technology, while the interchangeable chips can be fabricated using a smaller process technology. One or more of the interchangeable chips 2495 can be memory (e.g., DRAM) chips. Different memory densities can be selected for the package assembly 2494 based on the power and / or performance requirements of the product using the package assembly 2494. Furthermore, logic chips with different numbers of different types of functional units can be selected during assembly based on the power and / or performance requirements of the product. Additionally, chips containing different types of IP logic cores can be inserted into the interchangeable chiplet slots, enabling hybrid memory designs that can mix and match IP blocks of different technologies.

[0373] Exemplary System-on-Chip Integrated Circuit

[0374] Figures 25-26B The illustrations depict exemplary integrated circuits and associated graphics processors that can be fabricated using one or more IP cores. In addition to what is illustrated, other logic and circuitry may be included, including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores. Figures 25-26B Elements having the same or similar names as elements in any other figure herein are described as the same elements in other figures, and can operate or function in a manner similar to those described elsewhere herein. They may include the same components as those described elsewhere herein and may be linked to other entities as those described elsewhere herein, but are not limited thereto.

[0375] Figure 25This is a block diagram illustrating an exemplary system-on-a-chip integrated circuit 2500 that can be fabricated using one or more IP cores. The exemplary integrated circuit 2500 includes one or more application processors 2505 (e.g., CPUs), and at least one graphics processor 2510, which may be variations of graphics processors 1408, 1508, 2510, or any other graphics processor described herein, and may be used in lieu of any described graphics processor. Therefore, the disclosure of any features combined with a graphics processor herein also discloses corresponding combinations with graphics processor 2510, but is not limited thereto. The integrated circuit 2500 may additionally include an image processor 2515 and / or a video processor 2520, either of which may be a modular IP core from the same design facility or multiple different design facilities. The integrated circuit 2500 may include peripheral or bus logic, including a USB controller 2525, a UART controller 2530, an SPI / SDIO controller 2535, and an I2S / I2C controller 2540. In addition, the integrated circuit may include a display device 2545 coupled to one or more of a High Definition Multimedia Interface (HDMI) controller 2550 and a Mobile Industry Processor Interface (MIPI) display interface 2555. Storage may be provided by a flash memory subsystem 2560 (including flash memory and a flash memory controller). A memory interface may be provided via a memory controller 2565 to obtain access to an SDRAM or SRAM memory device. Some integrated circuits additionally include an embedded security engine 2570.

[0376] Figures 26A-26B This is a block diagram illustrating an exemplary graphics processor for use within a SoC according to embodiments described herein. The graphics processor may be a variant of graphics processor 1408, 1508, 2510, or any other graphics processor described herein. The graphics processor may be used in place of graphics processor 1408, 1508, 2510, or any other graphics processor described herein. Therefore, the disclosure herein of any feature combined with graphics processor 1408, 1508, 2510, or any other graphics processor described herein also discloses... Figures 26A-26B The corresponding combination of graphics processors, but not limited to this. Figure 26A The illustration shows an exemplary graphics processor 2610, which is a system-on-a-chip integrated circuit that can be manufactured using one or more IP cores according to an embodiment. Figure 26B The illustration shows an additional exemplary graphics processor 2640 of a system-on-a-chip integrated circuit that can be manufactured using one or more IP cores according to an embodiment. Figure 26A The 2610 graphics processor is an example of a low-power graphics processor core. Figure 26BThe Graphics Processor 2640 is an example of a higher-performance graphics processor core. For example, as mentioned at the beginning of this paragraph, each of the Graphics Processor 2610 and Graphics Processor 2640 could be... Figure 25 A variant of the 2610 graphics processor.

[0377] like Figure 26A As shown, the graphics processor 2610 includes a vertex processor 2605 and one or more fragment processors 2615A-2615N (e.g., 2615A, 2615B, 2615C, 2615D, up to 2615N-1 and 2615N). The graphics processor 2610 can execute different shader programs via separate logic, such that the vertex processor 2605 is optimized to perform operations for the vertex shader program, while the one or more fragment processors 2615A-2615N perform fragment (e.g., pixel) shading operations for fragment or pixel shader programs. The vertex processor 2605 executes the vertex processing stage of the 3D graphics pipeline and generates primitive data and vertex data. The fragment processors (multiple) 2615A-2615N use the primitive data and vertex data generated by the vertex processor 2605 to produce frame buffers that are displayed on the display device. Multiple fragment processors 2615A-2615N can be optimized to execute fragment shader programs as provided in the OpenGL API, which can be used to perform operations similar to those of pixel shader programs provided in the Direct 3D API.

[0378] The graphics processor 2610 additionally includes one or more memory management units (MMUs) 2620A-2620B, multiple caches 2625A-2625B, and multiple interconnects 2630A-2630B. These one or more MMUs 2620A-2620B provide virtual-to-physical address mapping for the graphics processor 2610 (including for the vertex processor 2605 and / or multiple fragment processors 2615A-2615N). In addition to referencing vertex data or image / texture data stored in the one or more caches 2625A-2625B, this virtual-to-physical address mapping can also reference vertex data or image / texture data stored in memory. The one or more MMUs 2620A-2620B can be synchronized with other MMUs within the system, enabling each processor 2505-2520 to participate in a shared or unified virtual memory system. Other MMUs within the system include those related to... Figure 25 One or more MMUs associated with one or more application processors 2505, graphics processors 2515, and / or video processors 2520. Components of graphics processor 2610 may correspond to components of other graphics processors described herein. One or more MMUs 2620A-2620B may be associated with... Figure 2C The MMU 245 corresponds to this. The vertex processor 2605 and the chip processors 2615A-2615N can correspond to the graphics multiprocessor 234. According to an embodiment, one or more circuit interconnects 2630A-2630B enable the graphics processor 2610 to interface with other IP cores within the SoC via the SoC's internal bus or via a direct connection. One or more circuit interconnects 2630A-2630B can correspond to... Figure 2C The data cross switch 240 corresponds to this. Further correspondences can be found between the graphics processor 2610 described herein and similar components of various graphics processor architectures.

[0379] like Figure 26B As shown, the graphics processor 2640 includes Figure 26A The graphics processor 2610 includes one or more MMUs 2620A-2620B, caches 2625A-2625B, and circuit interconnects 2630A-2630B. The graphics processor 2640 includes one or more shader cores 2655A-2655N (e.g., 2655A, 2655B, 2655C, 2655D, 2655E, 2655F, up to 2655N-1 and 2655N), which provide a unified shader core architecture in which a single core or type or core can execute all types of programmable shader code, including shader program code for implementing vertex shaders, fragment shaders, and / or compute shaders. The exact number of shader cores present may vary depending on the embodiment and implementation. Additionally, the graphics processor 2640 includes an inter-core task manager 2645, which acts as a thread dispatcher for assigning execution threads to one or more shader cores 2655A-2655N and a fragmentation unit 2658 for accelerating fragmentation operations on slice-based rendering, in which rendering operations for a scene are subdivided in image space, for example to utilize local spatial consistency within the scene or optimize the use of internal caches. The shader cores 2655A-2655N may, for example, be coupled with... Figure 2D The graphics multiprocessor 234 corresponds to, or respectively in Figure 3A and Figure 3B The graphics multiprocessors 325 and 350 correspond to, or are related to Figure 3C The multi-core group 365A corresponds to this.

[0380] The architecture of conventional L3 caches used in new GPU technologies suffers from problems such as inefficient cache utilization. Inefficient cache utilization in GPUs leads to lower performance and higher power consumption.

[0381] In a conventional sense, the L1 cache is virtually indexed (pre-translated addresses), so no cache control based on page table attributes (PTE) can be used.

[0382] Instruction-based memory properties

[0383] In one embodiment, this design includes instruction set architecture (ISA) control, compaction, and other optimizations for processing resources. The design controls cache attributes at multiple levels (e.g., L1, L2, L3) of the processing resources by attaching cache attributes when buffers are created (surface states are created by the driver), by application code directly controlling cache attributes from instructions (ISA enhancement), and by the kernel-mode driver setting memory-side cache policies via page table entries (PTEs).

[0384] As described herein, processing resources refer to processing elements (e.g., GPGPU cores, ray tracing cores, tensor cores, execution resources, execution units (EUs), stream processors, streaming multiprocessors (SMs), graphics multiprocessors) associated with a graphics processor architecture (e.g., parallel processing units, graphics processing engines, multi-core groups, compute units, compute units of subsequent graphics cores) in a graphics processor or GPU. For example, the processing resources may be one of the GPGPU core 262 or tensor / ray tracing core 263 of the graphics multiprocessor 234; the ray tracing cores 338A-338B, tensor cores 337A-337B or GPGPU cores 336A-336B of the graphics multiprocessor 325; the execution resources 356A-356D of the graphics multiprocessor 350; one of the GFX cores 370, tensor cores 371 or ray tracing cores 372 of the multi-core groups 365A-365N; one of the vector logic units 1563 or scalar logic units 1564 of the computing units 1506A-1506N; an execution unit having EU arrays 1522A-1522F or EU arrays 1524A-1524F; execution units 1808A-1808N of the execution logic 1800; and / or execution unit 1900. Processing resources can also be, for example, execution resources within graphics processing engines 431-432, processing clusters 706A-706H, GPGPUs 806A-806D, GPGPU 1306, graphics processing engine 1610, graphics processing engine cluster 1622, and / or graphics processing engine 1710. Processing resources can also be processing resources within graphics processors 2510, 2610, and / or 2640.

[0385] As discussed earlier, the L1 cache is typically indexed virtually (pre-translated addresses), therefore no PTE-based cache control can be used. L1 cacheability control will be implemented through the following two mechanisms.

[0386] For the first mechanism, surface states (or "resource states") are programmed by a user-mode driver to have a default Level 1 cache (e.g., L1 cache) attribute for stateful message access to the surface (or resource), and the Level 1 cache settings for stateless messages are in a non-pipelined state (e.g., programmed by the user-mode driver via state base address commands). Stateful messages are persistently backed up by a table or backed up in memory. Stateless messages are not backed up by a table or memory. Surface states are stored as individual elements, each surface state having its own pointer in a binding table or its own entry in the memory heap. Surface states define the state required by an object (e.g., texture maps read by the sampling engine, buffers read by the sampling engine, render targets read / written by the render cache, starting vertex buffers written by the render cache, media surfaces read from the texture cache or render cache, and media surfaces written to the render cache).

[0387] When both STATE_BASE_ADDRESS (state base address) / Surface_state (surface state) and the sent message instruct L1 / L3 cache control, the LSC (Load-Store Cache) unit and the L3 node need to resolve what should be the final cache attributes. The following table (for example, Figure 27 and Figure 28 The diagram shows the cache attributes obtained in this case (the "default" value is the value specified in STATE_BASE_ADDRESS for stateless messages, or the value specified in Surface_State for stateful messages):

[0388] For L1, the send command attribute can override the cacheability attribute specified by STATE_BASE_ADDRESS or Surface_state.

[0389] For the second mechanism, the sending message descriptor will have optional first-level caching (e.g., L1 cache) caching properties. This is primarily used for stateless messages (but also for stateful messages) when there is no "surface state" to give per-buffer flexibility to the cacheability knob (knob).

[0390] The first-level cacheability attribute supported in the hardware

[0391] Figure 27The illustration, according to one embodiment, shows a table 2700 of the first-level cache attributes supported for each message type. Instruction 2702 provides cache attributes 2704 (or cache properties), and description 2706 for the associated attribute 2704 is provided in... Figure 27 As shown in the image.

[0392] For WB, WT, and WS, any partial writes smaller than an L1 sub-sector (e.g., 8 bytes) will be considered uncacheable. Incompressible writes sent to a compressed surface will also be considered uncacheable.

[0393] For storage instructions with WS (Write-Stream) attributes, data in streamed storage is cached in the L1 cache with low priority due to the prediction that data will be used quickly (e.g., streaming applications). In one example, the Least Recently Used (LRU) location is used to merge partial writes in the L1 cache until a complete cache line is generated based on multiple partial writes. After a complete cache line has been formed, the cache line can be backed up in the L2 / L3 cache or in memory, depending on the L2 / L3 cache attributes. For example, if L2 / L3 is UC, the complete cache line will be stored in memory instead of in the L2 / L3 cache. If L2 / L3 is WB, the complete cache line is stored in the L2 / L3 cache.

[0394] For load instructions with the IAR (Invalidate After Read) attribute, if the data for a cache hit in the L1 cache originates from private memory, that data is invalidated after the read operation. In one example, the data for a cache hit is read from the L1 cache and then discarded. Data can be pushed onto a stack or queue, popped from a stack or queue, and then discarded because the data (e.g., data used in ray tracing applications) is no longer needed by other resources.

[0395] Figure 28 The illustration, according to one embodiment, shows a table 2800 of the multi-level cache attributes supported for each message type. Instructions 2802 provide cache attributes L1 2804 (or cache properties), cache attributes L2 / L3 2805, and descriptions 2806 for the associated attributes 2804 and 2805. Figure 28 As shown in the diagram. Any message pointing to L3 also applies to L2.

[0396] For prefetch instructions with LDS (Load Streaming) attributes such as l1lds_l3uc or l1lds_l3wb, the streamed data is prefetched into L1 only if the L2 / L3 cache is UC (Unified Cache). The streamed data can also be prefetched into the L1 cache and also into the L2 / L3 cache. In one example, the streamed data is prefetched into the L1 cache and given a low priority due to the prediction that the data will be used quickly. The streamed data is then quickly evicted using the LRU (Least Recently Used) position in the L1 cache.

[0397] Loading a cache unit requires decoding the send message MSD to obtain cache attributes and sending it to the L1 block. The L2 / L3 control index is also sent to the L1 block. For miss requests and evictions to the L2 / L3 cache, the L1 block will pass a 2-bit L2 / L3 control index and a 2-bit instruction L2 / L3 control to the L2 / L3 structure.

[0398] Figure 29 The illustration depicts a method 2900 according to one embodiment, having an exemplary sequence of operations for controlling the priority of a first-level cache to improve cache efficiency. According to one embodiment, a graphics processing unit, graphics multiprocessor, or graphics processor having cache units for storing data performs operation 2900. A cache controller (e.g., 248-1, 373-1, 1554-1, 2151-1) of the cache memory (e.g., L1, multiple caches 248, 373, 1554, 2151) determines whether default settings from the driver or instructions from the application will control the cache operation.

[0399] At operation 2902, the cache controller of the cache memory determines default attributes (e.g., UC, WT, WB, WS for store / atomic operations; UC, LDC, LDS for load / prefetch). Drivers (e.g., system drivers, kernel-mode drivers) can set default attributes for the cache memory. At operation 2904, the cache controller of the cache memory determines whether instructions from the application (e.g., store, load, atomic operation, prefetch) have been received. Instructions provide cache attributes (e.g., UC, WT, WB, WS for store / atomic operations; UC, LDC, IAR, LDS for load / prefetch). If the instruction has been received, at operation 2906, the cache controller applies the instruction and associated cache attributes. If the instruction has not yet been received, at operation 2908, the cache controller applies the default cache attributes.

[0400] Figure 30The illustration, according to one embodiment, shows a table 3000 of L1 cache priority for instructions relative to default cache attributes. If an instruction has been received, the cache controller applies the instruction and associated cache attributes 3010 for a store / atomic operation 3002. If the instruction has not yet been received (no instruction), the cache controller applies the default cache attributes 3020.

[0401] Figure 31 The illustration, according to another embodiment, shows table 3100 for L1 cache priority of instructions relative to default cache attributes. If an instruction has been received, the cache controller applies the instruction and associated cache attributes 3110 for load / prefetch 3102. If the instruction has not yet been received (no instruction), the cache controller applies the default cache attributes 3120.

[0402] L2 / L3 Cache Control Overview

[0403] In this design, the L2 / L3 cache can be physically indexed. The L2 / L3 cache capability knob is sourced from the following:

[0404] L2 / L3 Cache Control Index: A 2-bit L2 / L3 cache control index will point to a table entry that will have L2 / L3 cache capability settings. The L2 / L3 cache control will provide four possible indices, but in this example, the design supports two valid states: L2 / L3 cacheable and L2 / L3 non-cacheable. In this example, the other two states will be reserved.

[0405] Surface states will have the default L2 / L3 cache control index for stateful messages, and non-pipelined states (e.g., STATE_BASE_ADDRESS) will have the default L2 / L3 cache control index for stateless messages.

[0406] The send message descriptor will optionally have L2 / L3 cache attributes. In one example, the PTE will also have a 1-bit L3 cache attribute field (e.g., cached / not cached).

[0407] The cache controller of a cache memory (e.g., L2 / L3 cache) arbitrates between instructions as instructed by the application, L2 / L3 cache control of the user-mode driver, and PTE settings as determined by the driver (e.g., system driver, kernel-mode driver). In most cases, the instruction settings will override the L2 / L3 cache control settings for L2 / L3 cache capabilities. However, neither the instructions nor the L2 / L3 cache control can weaken the consistency requirements set by the PTE (page table entry) settings.

[0408] Figure 32 The diagram, according to one embodiment, illustrates a table 3200 showing the L2 / L3 cache priority for instructions relative to the default L2 / L3 cache control. For load, store, or prefetch 3202, if the instruction has been received, the cache controller applies the instruction and its associated cache attributes 3210. If the instruction has not yet been received (no instruction), the cache controller applies the default cache attributes 3220 of the L2 / L3 cache control.

[0409] Figure 33 The illustration, according to another embodiment, shows table 3300, which represents the L2 / L3 cache priority relative to the PTE settings for instructions and default settings. Figure 33 As indicated, for load, store, prefetch, and atomic operations 3302, if the instruction has already been received, the cache controller applies the instruction / default setting 3320 and associated cache attributes, unless the PTE setting and associated cache attribute 3350 override the instruction / default setting. For PTE setting UC, this will override instruction / default setting 3320. For PTE setting WB, it will not override instruction / default setting.

[0410] In one example, the L2 / L3 cache will only support two attributes: cached in L2 / L3 (at the MRU location), or not cached in L3 (UC). UC requests will look up the L2 / L3 cache.

[0411] Memory range attributes provide the visibility and range of data to the cache. The following ranges are defined.

[0412] For private memory scopes, even if mapped to the global memory virtual address space of the context, the data is thread-private (e.g., per-thread scratch space). The application will know whether the data is private or global and can use fields in the instruction to indicate the memory scope. Cache lines used for private data do not need to be consistent with other threads. Therefore, these cache lines do not need to be dumped due to "L1 cache dump sweep" or barrier operations, as these cache lines are not shared with other resources. Dump sweep or barrier operations (or overwrite operations, eviction operations) may have additional fields to indicate the memory scope (e.g., private, local, global). Any message operating on the "SCRATCH (thread-private)" surface type will automatically imply private memory. Note that the read-after-invalidation mode used for loads will only apply to private memory.

[0413] For local memory extents, even if mapped to the context's global virtual address space, data is shared locally only within the first group of the processing resource. These cache lines do not need to be consistent with caches in other second or third groups used for the processing resource, but only within the first group. Therefore, these cache lines do not need to be dumped due to an "L1 cache dump sweep" operation. Applications can provide assertions that data with local memory extents will not be dumped because data in the first group of the processing resource does not need to be consistent with data in other groups of the processing resource. In one example, the "stack" data generated by the ray tracing thread would be in this category. For these surfaces, the default caching policy would typically be "WB" (as programmed in the surface state).

[0414] For the global memory scope, data is globally visible across all threads within a GPU package, across all GPU packages across a node, and within (multiple) host CPUs in a node. These cache lines must be evicted due to the "L1 cache dump sweep" operation. This is the default memory scope.

[0415] Suppress concurrency to increase cache capacity

[0416] Graphics processors, graphics multiprocessors, and GPUs typically have inefficient cache utilization, resulting in lower performance and higher power consumption. Cache misses to the cache cause long latency in retrieving data from more remote memory locations (e.g., HBM).

[0417] Currently, GPUs share a low-level cache between an implicitly managed traditional cache (which loads values ​​into memory and then to the cache) and an explicitly managed (API-visible, temporary, application-managed, explicit) shared local memory (“SLM”) cache. The shared low-level cache can be dynamically partitioned and can be used entirely for traditional caching purposes (if the kernel does not require shared local memory) or entirely for SLM (if SLM requirements are high or many kernel instances are running concurrently).

[0418] Common partitioning schemes for such shared caches may not be ideal. Many partitioning schemes attempt to maximize "occupancy" by scheduling as many kernel instances ("thread groups") as possible. Such partitioning schemes will consume more shared cache for SLM and leave less shared cache for traditional cache uses. Therefore, performance may actually be degraded due to fewer traditional cache hits.

[0419] Shared local memory requirements are determined by kernel attributes and cannot be changed. The size of the shared low-level cache is device-specific and does not change. The only control that can be modified is the number of thread groups executing concurrently.

[0420] Embodiments of the present invention provide systems and methods for increasing control over partitioning schemes used for shared cache purposes, specifically by modifying the number of thread groups that can be executed concurrently. In one example, control is set to more thread groups, which equates to higher occupancy but less conventional cache. In another example, control is set to fewer thread groups, which equates to lower occupancy but more conventional cache.

[0421] This design can curb the number of concurrently executing workgroups (e.g., unit groups) to effectively increase traditional cache capacity. If additional concurrency is not beneficial (e.g., additional threads are stopped on memory accesses or other long-latency operations), reducing occupancy may have little or no impact on real-world performance, while increasing effective cache capacity may be beneficial.

[0422] In one embodiment, control over changing the number of concurrently executing thread groups is explicitly under application control. This control can be fine-grained ("limit to n concurrently executing workgroups") or coarse-grained ("prefer cache over occupancy"), and can be provided as a kernel attribute or runtime API control.

[0423] / / Example kernel properties:

[0424] attribute__((concurrent_work_groups(2)))

[0425] kernel void MyKernel(global uint*dst,global uint*src)

[0426] { ...

[0428] }

[0429] / / Example Host API:

[0430] clSetKernelConcurrentWorkGroups(kernel,2);

[0431] In another embodiment, control over the number of concurrently executing thread groups can be done automatically. For example, the GPU can provide a counter to indicate whether a processor thread is "stopped," meaning the thread resides on the machine but is not making progress. In this case, limiting the amount of concurrency may not affect performance, and it is possible to improve performance by limiting concurrency, especially when cache hit rates are low.

[0432] Figure 34 The illustration shows a flowchart 3400 for controlling a partitioning scheme used for shared cache purposes, according to one embodiment. According to one embodiment, a graphics processing unit, graphics multiprocessor, or graphics processor having partitioning logic or circuitry (e.g., scheduler 210, instruction unit 254, scheduler / dispatcher 368, thread control logic (1807A-1807N), thread arbitrator 1822, thread dispatcher 2131, etc.) performs operation 3400. Control can be set "manually," such as by API hints provided by the application, or "automatically," based on hardware counters (such as cache utilization and / or processor utilization counters).

[0433] At operation 3402, the computer-implemented method includes initiating the dispatch of threads to a group of threads (e.g., a unit group) of a graphics multiprocessor, graphics processor, or GPU. At operation 3404, the computer-implemented method starts N thread groups. At operation 3406, the computer-implemented method includes determining whether threads are stopped at different levels (e.g., a third level including many stopped threads, a second or threshold level including stopped threads, and a first level including a few stopped threads). At operation 3408, given many stopped threads at the third level, the computer-implemented method includes determining whether a cache hit rate has reached a threshold level. If the hit rate reaches the threshold level or higher (e.g., a good hit rate), no changes are needed to the N thread groups, and the method returns to operation 3404. If the hit rate is below the threshold level (e.g., a low hit rate), the computer-implemented method proceeds to operation 3410 to reduce the number of concurrent thread groups (e.g., N-1, N-2, etc.). The computer-implemented method then returns to operation 3404.

[0434] If operation 3406 determines that a thread is stopped at the second level or a threshold level (higher than the first level but less than the third level), then no changes are needed for the N thread groups, and the method returns to operation 3404.

[0435] If operation 3406 determines that the number of threads stopped in the first level (e.g., too few threads are stopped), the method proceeds to operation 3412 to increase the number of concurrent thread groups (e.g., N+1, N+2, etc.). The computer-implemented method then returns to operation 3404.

[0436] Figure 35 The illustration shows how changing the number of workgroups (or cell groups) in one embodiment alters the availability of the traditional cache. Cache 3510 includes a maximum of 8 workgroups (or cell groups), 128KB of shared cache, and 16KB of shared local memory for each workgroup. Cache 3510 has no available space for the traditional cache. In one example, this design reduces the number of workgroups from 8 to 4, leaving 64KB of space for the traditional cache in cache 3520.

[0437] In another embodiment, hardware cache coherence is selected based on static dataflow analysis performed by the front-end compiler over memory. To avoid requiring the software stack to manually dump and clear the cache hierarchy to make the output of the producing kernel visible to the consuming kernel, and to avoid the energy costs of hardware cache coherence (unless manual dumping of the cache hierarchy by the software stack is required), embodiments of hardware cache coherence can implement special load and store instructions that enforce cache coherence. These instructions will differ from normal load and store instructions that do not implement any type of cache coherence. These coherence instructions will not be issued by default by the back-end JIT compiler. Instead, coherence instructions are used if the static dataflow analysis performed by the front-end compiler determines that the output of the producing kernel will be directly consumed by subsequent consuming kernels, and that the output of the producing kernel can reside in on-chip memory (e.g., L1 and L2 caches).

[0438] Otherwise, hardware consistency is of no benefit and should be avoided by using instructions that do not request hardware consistency.

[0439] The selection of the hardware cache coherence system will benefit energy and latency savings for use cases where the output of the generating kernel can be stored on the chip, while providing high-throughput cache without coherence mechanisms when the kernel output must be stored in DRAM or transmitted back to the host.

[0440] Figure 36A The diagram illustrates a flowchart for selecting hardware cache coherency according to one embodiment. According to one embodiment, a graphics processing unit, graphics multiprocessor, or graphics processor with a compiler performs these operations.

[0441] At operation 3602, the computer-implemented method 3600 includes programming using a data-parallel kernel. At operation 3604, the method includes a front-end compiler (high-level language) performing data flow analysis and expressing the data flow between kernel dispatches to a back-end compiler via a low-level intermediate representation (IR). The IR is a data structure or code used internally by the compiler or virtual machine to represent source code. At operation 3606, the computer-implemented method determines whether data should be cached between kernels. As discussed above, the static data flow analysis performed by the front-end compiler determines whether the output of the generating kernel will be directly consumed by subsequent consuming kernels, and whether the output of the generating kernel can reside in on-chip memory (e.g., as...). Figure 36B The memory hierarchy 3650 shown includes L1 caches 3660a, b, c and L2 caches 3670a, b. The memory hierarchy 3650 also includes off-chip memories 3680a, b (e.g., HBM).

[0442] If the data should be cached across kernels, then at operation 3608, the backend compiler (e.g., Just-In-Time (JIT) IR) provides cache-coherent instructions. The computer-implemented approach proceeds to operation 1610: the kernel executes on the GPU.

[0443] If memory buffers should not be cached across kernels, then at operation 1612, the backend compiler (e.g., JIT IR) provides inconsistent instructions.

[0444] Dynamic cache tag / data storage

[0445] Graphics operations frequently utilize compression logic to compress depth or color data written to memory and decompress depth or color data read from memory. The compression logic can be lossless compression logic using one or more of various compression algorithms. Vector compression is frequently performed to save memory space.

[0446] Currently, in L2 or L3 caches, there is a one-to-one correspondence between tags and data storage. Given the increasing cache size to meet future computing demands, area efficiency is crucial. Furthermore, efficient use of data storage (e.g., 64-byte cache lines) is important when dealing with compressible surfaces with clear data and compressed scenarios of 32 bytes. In this design, tags and data storage are completely decoupled, and data allocation can occur only when necessary. This significantly reduces power and area. Moreover, this design improves cache footprint utilization without incurring data storage costs.

[0447] Figure 37A , Figure 37B and Figure 37C The illustration depicts a method according to one embodiment having an exemplary sequence of operations for controlling cache operations to improve cache efficiency. According to one embodiment, a graphics processing unit, graphics multiprocessor, or graphics processor having a cache unit for storing data and a cache controller performs operation 3700.

[0448] At operation 3702, the cache controller (e.g., cache controller 3810) of the cache memory (e.g., private cache, cache line of the road group cache, L2 cache, L3 cache, (multiple) caches 221, 248, 252, 272, 330, 342, 358A-358B, 375, 438, 462A-462D, 706A-706H, 708, 1404, 1504A-1504N, 1506, 1536, 1554, 1567, 1636, 1725, 1806, 1812, 2151, 2158, 2175, 2178, 2179, 2625A-2625B) receives a first memory request (e.g., a first write request). At operation 3704, the cache controller determines control data for the received memory request to determine whether the data is compressed or uncompressed. If the data is uncompressed, then at operation 3705, normal cache operation occurs. For example, tags and data can be coupled in a 1:1 ratio, with 64 bytes of data stored in a 64-byte cache line. In one embodiment, if the data is compressed, the tags and data are decoupled from each other.

[0449] At operation 3706, if compressed data is determined, the cache controller determines whether the cache memory has space for storing the data requested from the memory. If so, at operation 3708, the data is stored in the cache memory, and control data (e.g., a tag field with at least partial address information, a valid field, a modified field, an offset field, and a trace field indicating whether the data is compressed or decompressed) is maintained in a separate, decoupled table in the cache memory. At operation 3709, if sufficient space is not available, the cache controller may evict the data to create sufficient space for the memory request. The control data will indicate the size of the compressed or uncompressed data to be written to the cache memory.

[0450] At operation 3710, the cache controller receives a second memory request (e.g., a second write request). At operation 3712, the cache controller determines control data for the received second request to determine whether the data is compressed or uncompressed. If the data is uncompressed, then at operation 3713, a normal cache operation occurs.

[0451] At operation 3714, the cache controller determines whether the cache memory has space for storing compressed data from the second memory request. If so, at operation 3716, the data is stored in the cache memory, and control data (e.g., a tag field with at least partial address information, a valid field, a modified field, an offset field, and a trace field indicating whether the data is compressed or decompressed) is stored in a separate, decoupled table in the cache memory. If the tags for the first and second requests are the same (e.g., the same address, the same address range), the cache controller can store the data for these requests in the same cache line if there is sufficient space. If the tags for the first and second requests are different, the cache controller can store the data for these requests in different locations in the cache memory.

[0452] If sufficient space is not available, at operation 3818, the cache controller can evict data to create sufficient space for a second memory request.

[0453] If the additional data space is available due to compression, it can be used for a temporary register or a ROP.

[0454] exist Figure 37C At operation 3750, the cache controller receives a read request for data. At operation 3752, the cache controller determines control data for the received read request, and at operation 3753, it performs a tag lookup to determine if the tag of the read request matches any tag in the table of control data for the cache memory. At operation 3754, for a cache tag hit, the cache controller reads the matching tag and marks it as read. A trace field indicates whether the data for the matching tag is compressed, and if so, decompression occurs.

[0455] At operation 3756, for a cache tag miss, the cache controller cannot locate a matching tag and will need to send a request to the memory location with the desired tag.

[0456] Figure 38The diagram illustrates a cache memory architecture 3800 with decoupled control data and data arrays. According to one embodiment, the memory architecture includes decoupled control data 3812 and data arrays 3820 (e.g., L1, L2, L3, and (multiple) caches 221, 248, 252, 272, 330, 342, 358A-358B, 375, 438, 462A-462D, 706A-706H, 708, 1404, 1504A-1504N, 1506, 1536, 1554, 1567, 1636, 1725, 1806, 1812, 2151, 2158, 2175, 2178, 2179, 2625A-2625B) to improve cache and memory efficiency. The graphics processing unit, graphics multiprocessor, or graphics processor includes a cache controller 3210 with control data 3812, a data array 3820 for storing data, and memory 3890 (e.g., high-bandwidth memory, system memory). The operation of method 3700 can be performed using the cache controller 3810. The control data 3812 includes a tag field 3813 with tags A, B, C, ... N, a valid / modified field 3814, an offset field 3816, and a trace field 3818 for indicating compressed or decompressed data. Data is stored in cache lines 3825a-d, etc., of the data array 3820. In one example, each cache line stores 64 bytes of data.

[0457] In one example, 128 bytes of data are compressed into 64 bytes, and then compressed again into 32-byte blocks before being cached in the L2 / L3 cache. For a 64-byte cache line, the 32 bytes are stored in the first region of the cache line containing data array 3820. If the tags of the first and second data blocks match, subsequent 32-byte blocks can be stored in the second region of the same cache line.

[0458] Communication links 3850-1 (e.g., refill, update), 3850-2 (e.g., data block), and 3850-3 (e.g., read, write, ready) provide control and data communication between cache controller 3810, data array 3820, and memory 3890. Cache controller 3810 may include an array of tags for storing address tags for memory requests (read 3841, write 3842). Address information 3830 may include tag 3831, index 3832, and offset information 3833. The cache controller may output a stop message 3843 and receive a dump clear message 3844 for evicting cache lines from data array 3820. Data can be read from data array 3820 using rdata (read data) message 3821 and written to data array using wdata (write data) message 3822. Cache controller 3810 and data array 3820 may be integrated into a cache memory.

[0459] In one example, for compressed data, the tag field is decoupled from the data in the cache memory. In another example, for uncompressed data, the tag field can be coupled to the data (e.g., one tag per data block or cache line).

[0460] In another embodiment, control data with tag address information is coupled to the data. The control data can be extended to include an optional compression field for indicating compression of the data (e.g., nonzero sparsity compression, frequent value compression). Figure 39 The illustration depicts a compression operation based on additional compression for a GPU according to one embodiment. In one example, the compression causes data to be compressed into cache lines using vectorized compression of the data as it is stored.

[0461] Figure 39The diagram illustrates source data field 3910, which has compression field 3926, compressed into fields 3912-3921 of destination data field 3930 using compression and control data 3920. Destination data field 3930 has fields 3937-3941, and compression field 3926 indicates whether compression occurred. Control data 3920 includes tag field 3922, valid field 3923, modified field 3924, offset field 3925, compression field 3926, and optional other control fields 3927-3931. In one example, the source and destination are typically bytes (e.g., 16B, 32B, 64B). Destination 3930 may include a series of zeros for field 3937, a count indicator (e.g., 7) for field 3938, and non-zero data values ​​C, B, and A in subsequent fields 3939, 3940, and 3941. Count indicator 3938 indicates that fields 3912-3918 from the source have zero values ​​and can be compressed. For example, two 64-byte source data blocks can each be compressed to 32 bytes and then stored in a single 64-byte cache line. In another example, four 64-byte source data blocks can each be compressed to 16 bytes and then stored in a single 64-byte cache line. This type of simple compression will have fast decompression for moving cached data to memory (e.g., HBM). In another example, data with frequent values ​​(e.g., 5) can be compressed using fields 3937 and 3938. Special registers can store frequent values.

[0462] Figure 40 The illustration depicts a method 4000 according to one embodiment, having an exemplary sequence of operations for controlling cache operations to improve cache efficiency. According to one embodiment, a graphics processing unit, graphics multiprocessor, or graphics processor having a cache unit for storing data performs operation 4000.

[0463] At operation 4002, the cache controller (e.g., cache controller 3810) of the cache memory (e.g., private cache, cache line of a road group cache, L2 cache, L3 cache, (multiple) caches 221, 248, 252, 272, 330, 342, 358A-358B, 375, 438, 462A-462D, 706A-706H, 708, 1404, 1504A-1504N, 1506, 1536, 1554, 1567, 1636, 1725, 1806, 1812, 2151, 2158, 2175, 2178, 2179, 2625A-2625B) receives a request for data (e.g., a read request, a write request). At operation 4004, the cache controller determines control data for the received request to determine whether the data can be compressed or whether compression would be useless. This may include determining whether a compression field within the control data indicates that compression is available before loading or reading data from the cache memory, or whether the data needs to be decompressed.

[0464] At operation 4006, if the compression field has a first logical value indicating no compression, the cache controller continues normal caching operations. This may include checking the tag field to determine if the cached tag matches the requested tag. If so, the cache controller reads the matching tag (e.g., tag N) from the cache memory to determine the location of the data for the matching tag. The cache controller does not read the previous tag N-1 because the data for tag N will be located in the cache line associated with tag N.

[0465] At operation 4008, if the compression field has a second logical value indicating that compression is available and would be useful, the compressor performs compression for a load request, and the compressed data is loaded into the cache memory. At operation 4009, for a read request, the cache controller checks the tag field to determine if the cached tag matches the tag of the read request. If so, at operation 4010, the cache controller reads the matching tag (e.g., tag N) from the cache memory and also reads the previous tag (e.g., tag N-1) to determine the location of the data for the matching tag. At operation 4012, after locating the data in the cache memory, the cache controller may then need to decompress the data and send the decompressed data to the desired destination. Both tag N and tag N-1 are read because there is a possibility that the desired compressed data may be located in a cache line associated with either the matching tag N or the previous tag N-1.

[0466] In loading scenarios (e.g., when data is not yet in the cache and needs to be loaded into the cache), the current value of the compress field will be changed based on whether compression is worthwhile. For example, if the requested cache line and the next / subsequent cache line in memory can be compressed down to achieve a 2:1 ratio and will not fit in a single cache line, then compression is worthwhile, and both compressed lines will be loaded into the cache and the compress field will be set to true.

[0467] Many of the methods in this method are described in their most basic form, but operations may be added to or removed from any of the methods without departing from the basic scope of the present embodiment. It will be apparent to those skilled in the art that many further modifications and adaptations can be made. Specific embodiments are not provided to limit the concepts but are intended to illustrate them. The scope of the embodiments is not determined by the specific examples provided above, but only by the appended claims.

[0468] If element "A" is coupled to or with element "B", then element A may be directly coupled to element B or indirectly coupled through, for example, element C. When the specification or claims state that component, feature, structure, process, or characteristic A "causes" component, feature, structure, process, or characteristic B, this means that "A" is at least partially caused by "B", but there may also be at least one other component, feature, structure, process, or characteristic that contributes to causing "B". If the specification indicates that a component, feature, structure, process, or characteristic "may", it does not require that particular component, feature, structure, process, or characteristic be included. If the specification or claims refer to an element as "a" or "an", this does not mean that only one described element exists.

[0469] An embodiment is an implementation or example. References to “embodiment,” “one embodiment,” “some embodiments,” or “other embodiments” in the specification mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least some embodiments, but not necessarily in all embodiments. Various appearances of “embodiment,” “one embodiment,” or “some embodiments” do not necessarily all refer to the same embodiment. It should be understood that in the above description of exemplary embodiments, features are sometimes grouped together in a single embodiment, drawing, or description thereof for the purpose of making this disclosure fluent and aiding in understanding one or more of the novel aspects. However, this method of disclosure should not be construed as reflecting an intention that the claimed embodiment requires more features than expressly recited in each claim. Rather, as reflected in the appended claims, the novelty aspect exists in fewer features than all the features of a single foregoing disclosed embodiment. Therefore, the claims are thus expressly incorporated into this specification, with each claim serving as a separate embodiment.

[0470] Some embodiments relate to Example 1, which includes a graphics processor comprising: a processing resource for performing graphics operations; and a cache controller for a cache coupled to the processing resource. The cache controller is configured to control cache priority by determining whether default settings or instructions will control cache operations that control the cache.

[0471] Example 2 includes the subject of Example 1, where the cache controller is configured to determine default properties and whether an instruction has been received.

[0472] Example 3 includes the subject of any one of Examples 1-2, wherein the cache controller is configured to apply the instruction and associated cache attributes if the instruction has been received.

[0473] Example 4 includes the subject of any of Examples 1-3, where the cache controller applies default cache properties if the instruction has not yet been received.

[0474] Example 5 includes the subject of any one of Examples 1-4, where the cache includes a first-level cache that processes resources.

[0475] Example 6 includes the subject of any one of Examples 1-5, wherein cache attributes include non-caching, write-through, write-back, or write-stream for storage operations or atomic operations.

[0476] Example 7 includes the subject of any one of Examples 1-6, wherein the cache attributes include non-caching for load operations or prefetch operations, load cache, or load streaming.

[0477] Example 8 includes the subject of any one of Examples 1-7, wherein a cache controller is configured to receive a storage message with a write-stream attribute for streaming data cached in the cache at a low priority for streaming storage, wherein the least recently used (LRU) position of the cache is used to merge partial writes in the cache until a complete cache line is generated based on multiple partial writes.

[0478] Example 9 includes the subject of any one of Examples 1-8, wherein the cache controller is configured to receive a load message with a read-after-invalidation attribute, which is used to invalidate the data for a cache hit in the cache after the read operation if the data comes from private storage.

[0479] Example 10 includes the subject of any one of Examples 1-9, wherein the cache controller is configured to receive a prefetch message with a load streaming attribute, wherein the streamed data is prefetched into the cache and given a low priority, and wherein the streamed data is then evicted using an LRU position in the cache.

[0480] Some embodiments relate to Example 11, which includes a graphics processor comprising: a processing resource for performing graphics operations; and a cache controller for a second-level cache coupled to the processing resource. The cache controller is configured to arbitrate between instruction settings, second-level cache control settings, and page table entry (PTE) settings to perform second-level cache operations.

[0481] Example 12 includes the subject of Example 11, where instructions are used to override the level 2 cache control settings.

[0482] Example 13 includes the subject of any of Examples 11-12, wherein the cache controller applies the settings of the instruction when the instruction is received.

[0483] Example 14 includes the subject of any one of Examples 11-13, wherein the cache controller is configured to apply the PTE settings with the settings of the overriding instruction and the second-level cache control settings when the PTE settings include no cache.

[0484] Example 15 includes the subject of any one of Examples 1-8, wherein the cache controller is configured to apply the settings of the instruction when the PTE settings include the write-back attribute.

[0485] Some embodiments relate to Example 16, which includes a graphics multiprocessor comprising: a group of processing resources for performing graphics operations; and a cache controller coupled to the group of processing resources. The cache controller is configured to receive a dump clear instruction for data in a cache memory and to determine whether the data has a private memory scope, a local memory scope, or a global memory scope.

[0486] Example 17 includes the subject of Example 16, wherein the cache controller is configured to dump and purge data when the data has a global memory scope.

[0487] Example 18 includes the subject of any one of Examples 16-17, wherein the cache controller is configured to not dump and clear data when data has a private memory range in which the data is thread-private.

[0488] Example 19 includes the subject of any one of Examples 16-18, wherein the cache controller is configured to not perform a dump cleanup when data has a local memory range in which the data is shared locally only within a first group of processing resources and not within a second group of processing resources.

[0489] Example 20 includes the subject of any of Examples 16-19, wherein the assertion indicates that data with local memory ranges should not be dumped and wiped because the data in the first group of processing resources does not need to be consistent with the data in the other groups of processing resources.

[0490] Some embodiments relate to Example 21, which includes a graphics multiprocessor comprising: a scheduler for launching N thread groups; and at least one hardware counter for indicating the number of stopped threads, wherein the scheduler is configured to determine whether a thread is stopped at a first level of some stopped threads, a second level of stopped threads having a threshold range, or a third level of many stopped threads.

[0491] Example 22 includes the subject of Example 21, wherein the scheduler is further configured to determine whether the cache hit rate has reached a threshold level when a large number of stopped threads at the third level are identified.

[0492] Example 23 includes the topic of any of Examples 21-22, wherein the scheduler is further configured to reduce the number of concurrent thread groups when the hit rate is below a threshold level.

[0493] Example 24 includes the subject of any one of Examples 21-23, wherein the scheduler is further configured to not perform changes on the N thread groups when a second level of stopped threads with a threshold range is determined.

[0494] Example 25 includes the subject of any of Examples 21-24, wherein the scheduler is further configured to increase the number of concurrent thread groups when the first level of stopped threads is determined.

[0495] Some embodiments relate to Example 26, which includes a computer-implemented method for a graphics multiprocessor or graphics processor, comprising: starting N thread groups; determining the number of stopped threads using at least one hardware counter; and determining whether the threads are stopped at a first level of some stopped threads, a second level of stopped threads having a threshold range, or a third level of many stopped threads.

[0496] Example 27 includes the subject of Example 26, and further includes: determining whether the cache hit rate has reached a threshold level when a number of stopped threads at the third level are identified.

[0497] Example 28 includes the topics of any of Examples 26-27, and further includes: reducing the number of concurrent thread groups when the hit rate is below a threshold level.

[0498] Example 29 includes the subject of any one of Examples 26-28, and further includes: when a second level of stopped threads with a threshold range is determined, no changes are performed on the N thread groups.

[0499] Example 30 includes the subject of any of Examples 26-29, and further includes: increasing the number of concurrent thread groups when the first level of stopped threads is determined.

[0500] Some embodiments relate to Example 31, which includes a graphics multiprocessor including: on-chip memory; and a front-end compiler for performing static data flow analysis to determine whether the output of the generating kernel will be directly consumed by a subsequent consuming kernel and whether the output of the generating kernel can reside in the on-chip memory.

[0501] Example 32 includes the subject of Example 31, where the front-end compiler provides a low-level intermediate representation (IR).

[0502] Example 33 includes the subject of any of Examples 31-32, in which the front-end compiler determines whether on-chip memory buffers should be cached between kernels based on whether the output of the producing kernel will be directly consumed by subsequent consuming kernels.

[0503] Example 34 includes the subject matter of any one of Examples 31-33, and further includes: a backend compiler for providing cache-consistent instructions when the output based on the producer kernel is directly consumed by subseque...

Claims

1. A graphics processing unit (GPU), comprising: Multiple core groups, each core group including: Multiple cores of the first type; and Multiple kernels of the second type, wherein the multiple kernels of the second type are tensor kernels; Multiple combinations of Level 1 L1 cache and shared memory units, each corresponding to a different core group among the multiple core groups; The second-level L2 cache is shared by the multiple core groups; Multiple memory controllers for coupling the GPU to memory; and A cache controller circuit, associated with the L2 cache, is configured to respond to a load instruction from a first core of the plurality of core sets to: Based on the load instruction, a cache controller is selected from a plurality of stored cache controllers, wherein at least some of the plurality of cache controllers have different cache eviction priorities; and The selected cache control is applied to the data allocated to the L2 cache. The multiple cache controls are used to store data in a data structure.

2. The GPU as claimed in claim 1, wherein, The selected cache control has a streaming cache eviction priority.

3. The GPU as described in claim 1, wherein, The selected cache control is a streaming cache control.

4. The GPU as claimed in claim 1, wherein, The load instruction is used to instruct the data to be used in the global address space.

5. The GPU of claim 1, further comprising: Scheduler / dispatcher circuitry for scheduling and dispatching graphics threads for execution on the multiple core groups; as well as Multiple texture unit groups, each corresponding to a different kernel group among the multiple kernel groups.

6. The GPU of claim 5, further comprising: Input / output (I / O) circuitry for coupling the GPU to one or more I / O devices.

7. The GPU as claimed in claim 1, wherein, Each of the plurality of core groups includes a ray tracing core.

8. A method executed by a graphics processing unit (GPU), the method comprising: Data is processed using multiple core groups, including: Process graphics data using multiple cores of the first type in each of multiple groups; Matrix operations are performed using multiple kernels of the second type in each of multiple groups, wherein the multiple kernels of the second type are tensor kernels; and The data is stored in multiple combinations of Level 1 L1 cache and shared memory units, each of which corresponds to a different core group in the multiple core groups; The multiple core groups share the L2 cache; Data is accessed from memory by multiple memory controllers of the GPU; and Execute the load instruction received from the first core of the plurality of core groups, including: Based on the load instruction, a cache controller is selected from a plurality of stored cache controllers, wherein at least some of the plurality of cache controllers have different cache eviction priorities; and The selected cache control is applied to the data allocated to the L2 cache. The multiple cache controls are stored in a data structure.

9. The method of claim 8, wherein, Selecting the cache control includes selecting a cache control with a streaming cache eviction priority.

10. The method of claim 8, wherein, Selecting the cache control includes selecting streaming cache control.

11. The method of claim 8, further comprising scheduling and dispatching graphics threads for execution on the plurality of core groups.

12. The method of claim 8, further comprising performing ray tracing using a ray tracing core in each of the plurality of core groups.

13. A system for data processing, comprising: Memory; as well as The graphics processing unit (GPU) as described in any one of claims 1 to 7, wherein the GPU is coupled to the memory.

14. A machine-readable medium storing code that, when executed, causes a machine to perform the method as described in any one of claims 8 to 12.

15. An apparatus for data processing, comprising means for performing the method as described in any one of claims 8 to 12.

Images