Adaptive deformable kernel prediction network for image de-noising

JP2021077343A5Inactive Publication Date: 2024-11-05INTEL CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2020150178
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2019-11-07
Filing Date
2020-09-07
Publication Date
2024-11-05
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing graphics processing units (GPUs) face challenges in efficiently handling diverse data processing tasks, particularly in parallel graphics processing, due to limitations in programmability and pipelining techniques, which affect performance and efficiency in operations such as linear interpolation, tessellation, and rasterization.

Method used

The implementation of an adaptive deformable kernel prediction network (ADKPN) within GPUs, which enhances the graphics processing pipeline by dynamically adjusting kernel operations for improved parallel processing and efficiency across various graphics operations.

Benefits of technology

The ADKPN significantly improves the performance and efficiency of GPUs by optimizing kernel operations, allowing for better synchronization and distribution of tasks, thereby enhancing the overall processing capabilities for graphics and machine learning applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2021077343000001
    Figure 2021077343000001
Patent Text Reader

Abstract

To adaptively determine offsets for pixels in de-noising an image by a kernel prediction network so as to realize lower computational cost and good image quality.SOLUTION: A method for de-noising an image by a convolutional neural network, the image including a plurality of pixels, comprises: for each of the pixels of the image, generating a convolutional kernel having a plurality of kernel values for the pixel; generating a plurality of offsets for the pixel respectively corresponding to the kernel values, each of the offsets indicating a deviation from a pixel position of the pixel; determining a plurality of deviated pixel positions based on the pixel position of the pixel and the offsets; and filtering the pixel with the convolutional kernel and pixel values of the deviated pixel positions to obtain a de-noised pixel.SELECTED DRAWING: Figure 29
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This embodiment generally relates to data processing, and more specifically, to data processing by a general-purpose graphics processing unit.

Background Art

[0002] Currently, parallel graphics data processing includes systems and methods developed to perform specific operations on graphics data, such as linear interpolation, tessellation, rasterization, texture mapping, depth testing, etc. So far, graphics processors have used fixed-function computing units for processing graphics data. However, more recently, some parts of graphics processors have been made programmable, and such processors can support a wider variety of operations for processing vertex and fragment data.

[0003] To further improve performance, graphics processors typically implement processing techniques such as pipelining, which attempt to process as much graphics data in parallel as possible across different parts of the graphics pipeline. Parallel graphics processors with a single instruction multiple threads (SIMT) architecture are designed to maximize the amount of parallel processing in the graphics pipeline. In the SIMT architecture, a group of parallel threads attempts to execute program instructions synchronously together as frequently as possible to enhance the efficiency of processing. A general overview of software and hardware related to the SIMT architecture can be found in Shane Cook, CUDA Programming, Chapter 3, pages 37 - 51 (2013).

Brief Description of the Drawings

[0004] Thus, the features described above in this embodiment can be understood in detail in a more specific description of the embodiment summarized above by reference to the embodiments, some of which are illustrated in the accompanying drawings. However, it should be noted that the accompanying drawings only show typical embodiments and should therefore not be considered to limit their scope.

[0005] [Figure 1] This is a block diagram showing a computer system configured to implement one or more embodiments of the embodiments described herein.

[0006] [Figure 2A] An embodiment of a parallel processor component is shown. [Figure 2B] An embodiment of a parallel processor component is shown. [Figure 2C] An embodiment of a parallel processor component is shown. [Figure 2D] An embodiment of a parallel processor component is shown.

[0007] [Figure 3A] This is a block diagram of a graphics multiprocessor and a multiprocessor-based GPU according to an embodiment. [Figure 3B] This is a block diagram of a graphics multiprocessor and a multiprocessor-based GPU according to an embodiment. [Figure 3C] This is a block diagram of a graphics multiprocessor and a multiprocessor-based GPU according to an embodiment.

[0008] [Figure 4A] This illustrates an exemplary architecture in which multiple GPUs are communicatively coupled to multiple multi-core processors. [Figure 4B] This illustrates an exemplary architecture in which multiple GPUs are communicatively coupled to multiple multi-core processors. [Figure 4C] This illustrates an exemplary architecture in which multiple GPUs are communicatively coupled to multiple multi-core processors. [Figure 4D] This illustrates an exemplary architecture in which multiple GPUs are communicatively coupled to multiple multi-core processors. [Figure 4E] This illustrates an exemplary architecture in which multiple GPUs are communicatively coupled to multiple multi-core processors. [Figure 4F] This illustrates an exemplary architecture in which multiple GPUs are communicatively coupled to multiple multi-core processors.

[0009] [Figure 5] This shows a graphics processing pipeline according to an embodiment.

[0010] [Figure 6] This document shows a machine learning software stack according to an embodiment.

[0011] [Figure 7] This shows a general-purpose graphics processing unit according to an embodiment.

[0012] [Figure 8] This document illustrates a multi-GPU computing system according to an embodiment.

[0013] [Figure 9A] The layers of an exemplary deep neural network are shown. [Figure 9B] The layers of an exemplary deep neural network are shown.

[0014] [Figure 10] An exemplary recurrent neural network is shown.

[0015] [Figure 11] This demonstrates the training and deployment of deep neural networks.

[0016] [Figure 12] It is a block diagram showing distributed learning.

[0017]

Figure 13

[0018]

Figure 14

[0019]

Figure 15A

Figure 15B

Figure 15C

[0020]

Figure 16A

Figure 16B

Figure 16C

[0021]

Figure 17

[0022]

Figure 18A

[0023] [Figure 19] An additional execution unit according to the embodiment is shown.

[0024] [Figure 20] This is a block diagram showing graphics processor instruction formats according to several embodiments.

[0025] [Figure 21] This is a block diagram of a graphics processor according to another embodiment.

[0026] [Figure 22A] The graphics processor command format and command sequence according to several embodiments are shown. [Figure 22B] The graphics processor command format and command sequence according to several embodiments are shown.

[0027] [Figure 23] This document presents exemplary graphics software architectures for data processing systems in several embodiments.

[0028] [Figure 24A] This is a block diagram showing an IP core development system according to an embodiment.

[0029] [Figure 24B] A vertical cross-sectional view of an integrated circuit package assembly according to some embodiments described herein is shown.

[0030] [Figure 24C] This shows a package assembly containing multiple units of hardware logic chiplets (e.g., base dies) connected to a circuit board.

[0031] [Figure 24D] This shows a package assembly including a replaceable chiplet according to an embodiment.

[0032] [Figure 25] This is a block diagram showing an exemplary system on a chip integrated circuit according to an embodiment.

[0033] [Figure 26A] This block diagram shows an exemplary graphics processor for use within a SoC according to the embodiments described herein. [Figure 26B] This block diagram shows an exemplary graphics processor for use within a SoC according to the embodiments described herein.

[0034] [Figure 27A] This is a schematic diagram showing a conventional kernel prediction network (KPN).

[0035] [Figure 27B] This is a schematic diagram illustrating an example of pixel filtering using conventional KPN.

[0036] [Figure 28A] This is a schematic diagram showing an adaptive deformable kernel prediction network (ADKPN) according to an embodiment.

[0037] [Figure 28B] This is a schematic diagram showing an example of pixel filtering using ADKPN according to an embodiment.

[0038] [Figure 28C] This graph shows a comparison of training and / or test performance measurements between conventional KPN and ADKPN.

[0039] [Figure 29] This flowchart shows a method for denoising images using ADKPN according to an embodiment.

[0040] [Figure 30A] The input image, reference image, output image generated by a conventional KPN, and output image generated by ADKPN are shown according to the embodiment. [Figure 30B] The input image, reference image, output image generated by a conventional KPN, and output image generated by ADKPN are shown according to the embodiment. [Figure 30C] The input image, reference image, output image generated by a conventional KPN, and output image generated by ADKPN are shown according to the embodiment. [Figure 30D] The input image, reference image, output image generated by a conventional KPN, and output image generated by ADKPN are shown according to the embodiment.

[0041] [Figure 31A] The input image, reference image, output image generated by a conventional KPN, and output image generated by ADKPN are shown according to the embodiment. [Figure 31B] The input image, reference image, output image generated by a conventional KPN, and output image generated by ADKPN are shown according to the embodiment. [Figure 31C] The input image, reference image, output image generated by a conventional KPN, and output image generated by ADKPN are shown according to the embodiment. [Figure 31D] The input image, reference image, output image generated by a conventional KPN, and output image generated by ADKPN are shown according to the embodiment.

[0042] [Figure 32A] The input image, reference image, output image generated by a conventional KPN, and output image generated by ADKPN are shown according to the embodiment. [Figure 32B] The input image, reference image, output image generated by a conventional KPN, and output image generated by ADKPN are shown according to the embodiment. [Figure 32C] The input image, reference image, output image generated by a conventional KPN, and output image generated by ADKPN are shown according to the embodiment. [Figure 32D] The input image, reference image, output image generated by a conventional KPN, and output image generated by ADKPN are shown according to the embodiment.

[0043] [Figure 33A] The input image, reference image, output image generated by a conventional KPN, and output image generated by ADKPN are shown according to the embodiment. [Figure 33B] The input image, reference image, output image generated by a conventional KPN, and output image generated by ADKPN are shown according to the embodiment. [Figure 33C] The input image, reference image, output image generated by a conventional KPN, and output image generated by ADKPN are shown according to the embodiment. [Figure 33D] The input image, reference image, output image generated by a conventional KPN, and output image generated by ADKPN are shown according to the embodiment. [Modes for carrying out the invention]

[0044] In some embodiments, a graphics processing unit (GPU) is communicatively coupled to a host / processor core to accelerate graphics computations, machine learning operations, pattern analysis operations, and various general-purpose GPU (GPGPU) functions. The GPU may be communicatively coupled to the host processor / core over a bus or other interconnect (e.g., a high-speed interconnect such as PCIe or NVLink). In other embodiments, the GPU may be integrated on the same package or chip as the core and communicatively coupled to the core (i.e., inside the package or chip) over an internal processor bus / interconnect. Regardless of how the GPU is connected, the processor core may assign operations to the GPU in the form of a sequence of commands / instructions contained in a work descriptor. The GPU then uses dedicated circuitry / logic to effectively process these commands / instructions.

[0045] In the following description, various specific details are included to provide a more overall understanding. However, it will be apparent to those skilled in the art that the embodiments described herein may be carried out without one or more of these specific details. In other examples, well-known features are omitted to avoid obscuring the details of the embodiments. System Overview

[0046] Figure 1 is a block diagram of a computing system 100 configured to implement one or more embodiments of the embodiments described herein. The computing system 100 includes a processing subsystem 101 having one or more processors 102 and system memory 104. The one or more processors 102 and system memory 104 may communicate via an interconnection path, which may include a memory hub 105. The memory hub 105 may be a separate component within a chipset component or may be integrated within one or more processors 102. The memory hub 105 is coupled to an I / O subsystem 111 via a communication link 106. The I / O subsystem 111 includes an I / O hub 107, which may enable the computing system 100 to receive input from one or more input devices 108. Furthermore, the I / O hub 107 may enable a display controller to provide output to one or more display devices 110A, which may be included within one or more processors 102. In one embodiment, one or more display devices 110A coupled to the I / O hub 107 may include local, internal, or embedded display devices.

[0047] In one embodiment, the processing subsystem 101 includes one or more parallel processors 112 coupled to a memory hub 105 via a bus or other communication link 113. The communication link 113 may be a communication link technology or protocol based on any number of standards, such as PCI Express, but is not limited to, or it may be a vendor-specific communication interface or communication fabric. In one embodiment, one or more parallel processors 112 may form a computation-focused parallel or vector processing system that includes a large number of processing cores and / or processing clusters, such as many integrated core (MIC) processors. In one embodiment, one or more parallel processors 112 may form a graphics processing subsystem that can output pixels to one of one or more display devices 110A coupled via an I / O hub 107. One or more parallel processors 112 may also include a display controller and a display interface (not shown) to enable direct connection to one or more display devices 110B.

[0048] Within the I / O subsystem 111, a system storage unit 114 may be connected to an I / O hub 107 to provide a storage mechanism for the computing system 100. An I / O switch 116 may be used to provide an interface mechanism that enables connections between the I / O hub 107 and other components such as a network adapter 118 and / or a wireless network adapter 119, which may be integrated with the platform, and various other devices that may be added via one or more add-in devices 120. The network adapter 118 may be an Ethernet® adapter or another wired network adapter. The wireless network adapter 119 may include one or more Wi-Fi®, Bluetooth®, Near Field Communication (NFC), or other network devices, including one or more wireless terminals.

[0049] The computing system 100 may include other components not explicitly shown, including USB or other port connections, optical storage drives, video capture devices, and similar devices, which may also be connected to the I / O hub 107. The communication paths interconnecting the various components in Figure 1 may be implemented using any suitable protocol, such as a PCI (Peripheral Components Interconnect) based protocol (e.g., PCI-Express), or any other bus or point-to-point communication interface, and / or a protocol such as an NV-Link high-speed interconnect, or an interconnection protocol known in the art.

[0050] In one embodiment, one or more parallel processors 112 incorporate circuits optimized for graphics and video processing, including, for example, video output circuits, to constitute a graphics processing unit (GPU). In another embodiment, one or more parallel processors 112 incorporate circuits optimized for general-purpose processing, while preserving the underlying computing architecture, which is described in more detail herein. In yet another embodiment, the components of the computing system 100 may be integrated with one or more other system elements on a single integrated circuit. For example, one or more parallel processors 112, a memory hub 105, a processor 102, and an I / O hub 107 may be integrated with a system-on-a-chip (SoC) integrated circuit. Alternatively, the components of the computing system 100 may be integrated into a single package to form a system-in-package (SIP) configuration. In one embodiment, at least some of the components of the computing system 100 may be integrated with a multi-chip module (MCM), which may be interconnected with other multi-chip modules in a modular computing system.

[0051] The computing system 100 shown herein is illustrative and is subject to change and modification. The connectivity topology, including the number and arrangement of bridges, the number of processors 102, and the number of parallel processors 112, may be modified as desired. For example, in some embodiments, system memory 104 is connected to the processors 102 directly rather than via bridges, while other devices communicate with the system memory 104 via memory hubs 105 and processors 102. In other alternative topologies, the parallel processors 112 are connected to I / O hubs 107 rather than memory hubs 105, or directly to one of the one or more processors 102. In other embodiments, the I / O hubs 107 and memory hubs 105 may be integrated into a single chip. Some embodiments may include two or more sets of processors 102 mounted via multiple sockets, which may be coupled with two or more instances of parallel processors 112.

[0052] Some of the specific components shown herein are optional and may not be included in all implementations of the computing system 100. For example, any number of add-in cards or peripherals may be supported, or some components may be omitted. Furthermore, some architectures may use different terminology for components similar to those illustrated in Figure 1. For example, the memory hub 105 may be called the northbridge in some architectures, while the I / O hub 107 may be called the southbridge.

[0053] Figure 2A shows a parallel processor 200 according to an embodiment. Various components of the parallel processor 200 may be implemented using one or more integrated circuit devices, such as a programmable processor, an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). The illustrated parallel processor 200 is an embodiment of one or more parallel processors 112 shown in Figure 1.

[0054] In one embodiment, the parallel processor 200 includes a parallel processing unit 202. The parallel processing unit includes an I / O unit 204 that enables communication with other devices, which includes other examples of the parallel processing unit 202. The I / O unit 204 may be directly connected to other devices. In one embodiment, the I / O unit 204 is connected to other devices via the use of a hub or switch interface, such as a memory hub 105. The connection between the memory hub 105 and the I / O unit 204 forms a communication link 113. Within the parallel processing unit 202, the I / O unit 204 is connected to a host interface 206 and a memory crossbar 216, the host interface 206 receiving commands for the execution of processing operations and the memory crossbar 216 receiving commands for the execution of memory operations.

[0055] When the host interface 206 receives a command buffer via the I / O unit 204, the host interface 206 can instruct the front end 208 to perform work operations to execute these commands. In one embodiment, the front end 208 is coupled with a scheduler 210, which is configured to distribute commands or other operational items to the processing cluster array 212. In one embodiment, the scheduler 210 ensures that the processing cluster array 212 is properly configured and active before tasks are distributed to the processing clusters of the processing cluster array 212. In one embodiment, the scheduler 210 is implemented by the execution of firmware logic on a microcontroller. The microcontroller implementing the scheduler 210 can be configured to perform complex scheduling, as well as work distribution operations at coarse and fine granularity, enabling rapid preemption and context switching of threads running on the processing array 212. In one embodiment, the host software may reveal workloads for scheduling on the processing array 212 via one of several graphics processing doorbells. The workload can then be automatically distributed across the processing array 212 by the scheduler logic 210 within the scheduler microcontroller.

[0056] The processing cluster array 212 may contain up to "N" processing clusters (e.g., cluster 214A, cluster 214B through cluster 214N). Each cluster 214A-214N of the processing cluster array 212 may execute a large number of parallel threads. The scheduler 210 may assign operations to clusters 214A-214N of the processing cluster array 212 using various scheduling and / or work distribution algorithms, which may vary depending on the workload arising for each type of program or computation. Scheduling may be handled dynamically by the scheduler 210 or may be partially assisted by compiler logic during the compilation of program logic configured for execution by the processing cluster array 212. In one embodiment, different clusters 214A-214N of the processing cluster array 212 may be assigned to process different types of programs or to execute different types of computations.

[0057] The processing cluster array 212 can be configured to perform various types of parallel processing operations. In one embodiment, the processing cluster array 212 is configured to perform general-purpose parallel computing operations. For example, the processing cluster array 212 may include logic to perform processing tasks including filtering video and / or audio data, performing modeling operations including physical motion, and performing data transformations.

[0058] In one embodiment, the processing cluster array 212 is configured to perform parallel graphics processing operations. In an embodiment in which the parallel processor 200 is configured to perform graphics processing operations, the processing cluster array 212 may include, but is not limited to, additional logic to support the execution of such graphics processing operations, including texture sampling logic to perform texture operations, as well as tessellation logic and other vertex processing logic. Furthermore, the processing cluster array 212 may be configured to execute graphics processing-related shader programs such as vertex shaders, tessellation shaders, geometry shaders, and pixel shaders. The parallel processing unit 202 may transfer data from system memory via the I / O unit 204 for processing. During processing, the data being transferred may be stored in on-chip memory (e.g., parallel processor memory 222) and then written back to system memory.

[0059] In one embodiment, when the parallel processing unit 202 is used to perform graphics processing, the scheduler 210 may be configured to divide the workload into tasks of approximately equal size, enabling better distribution of graphics processing calculations across multiple clusters 214A-214N of the processing cluster array 212. In some embodiments, parts of the processing cluster array 212 may be configured to perform different types of processing. For example, a first part may be configured to perform vertex shading and topology generation, a second part may be configured to perform tessellation and geometry shading, and a third part may be configured to perform pixel shading or other screen-space operations to produce a rendered image for display. Intermediate data generated by one or more clusters 214A-214N may be stored in a buffer, allowing the intermediate data to be transmitted between clusters 214A-214N for further processing.

[0060] During operation, the processing cluster array 212 may receive processing tasks to be executed via the scheduler 210, which receives commands from the front end 208 defining the processing tasks. With respect to graphics processing operations, processing tasks may include data to be processed, such as surface (patch) data, primitive data, vertex data, and / or pixel data, as well as state parameters and an index of commands that define how the data should be processed (e.g., which program should be executed). The scheduler 210 may be configured to fetch the index corresponding to the task, or to receive the index from the front end 208. The front end 208 may be configured to ensure that the processing cluster array 212 is configured to be active before the workload identified by the incoming command buffers (e.g., batch buffers, push buffers, etc.) is started.

[0061] Each of one or more instances of the parallel processing unit 202 may be coupled to a parallel processor memory 222. The parallel processor memory 222 may be accessed via a memory crossbar 216, which may receive memory requests from the processing cluster array 212 and the I / O unit 204. The memory crossbar 216 may access the parallel processor memory 222 via a memory interface 218. The memory interface 218 may include a plurality of partition units (e.g., partition unit 220A, partition unit 220B to partition unit 220N) which may each be coupled to a portion of the parallel processor memory 222 (e.g., a memory unit). In one implementation example, the number of partition units 220A-220N is equal to the number of memory units, so that the first partition unit 220A has a corresponding first memory unit 224A, the second partition unit 220B has a corresponding memory unit 224B, and the Nth partition unit 220N has a corresponding Nth memory unit 224N. In other embodiments, the number of partition units 220A-220N does not have to be equal to the number of memory devices.

[0062] In various embodiments, the memory units 224A-224N may include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory such as synchronous graphics random access memory (SGRAM) including graphics double data rate (GDDR) memory. In one embodiment, the memory units 224A-224N may also include 3D stack memory, including but not limited to high-bandwidth memory (HBM). Those skilled in the art will understand that the specific embodiments of the memory units 224A-224N may vary and may be selected from one of various conventional designs. Render targets, such as frame buffers or texture maps, may be stored across the memory units 224A-224N, and partition units 220A-220N may write to portions of each render target in parallel, effectively utilizing the available bandwidth of the parallel processor memory 222. In some embodiments, local instances of the parallel processor memory 222 may be excluded in favor of a unified memory design that uses system memory together with local cache memory.

[0063] In one embodiment, any one of the clusters 214A-214N of the processing cluster array 212 may process data that will be written to any memory unit 224A-224N in the parallel processor memory 222. The memory crossbar 216 may be configured to forward the output of each cluster 214A-214N to any partition unit 220A-220N or to another cluster 214A-214N, which may perform additional processing operations on the output. Each cluster 214A-214N may communicate with the memory interface 218 through the memory crossbar 216 to read from or write to various external memory devices. In one embodiment, the memory crossbar 216 has a connection to the memory interface 218 for communicating with the I / O unit 204, as well as a connection to a local instance of the parallel processor memory 222, which allows processing units in different processing clusters 214A-214N to communicate with system memory or other memory that is not local to the parallel processing unit 202. In one embodiment, the memory crossbar 216 may use virtual channels to separate the traffic stream between the clusters 214A-214N and the partition units 220A-220N.

[0064] While a single instance of the parallel processing unit 202 is illustrated within the parallel processor 200, any number of instances of the parallel processing unit 202 may be included. For example, multiple instances of the parallel processing unit 202 may be provided on a single add-in card, or multiple add-in cards may be interconnected. Different instances of the parallel processing unit 202 may be configured to interact with each other, even if the different instances have different numbers of processing cores, different amounts of local parallel processor memory, and / or other configuration differences. For example, in one embodiment, some instances of the parallel processing unit 202 may include higher-precision floating-point units compared to other instances. Systems incorporating one or more instances of the parallel processing unit 202 or parallel processor 200 may be implemented in a variety of configurations and forms, including, but are not limited to, desktop, laptop, or handheld personal computers, servers, workstations, game consoles, and / or embedded systems.

[0065] Figure 2B is a block diagram of a partition unit 220 according to an embodiment. In one embodiment, partition unit 220 is one instance of partition units 220A-220N in Figure 2A. As shown, partition unit 220 includes an L2 cache 221, a framebuffer interface 225, and a ROP 226 (raster operation unit). The L2 cache 221 is a read / write cache configured to perform load-and-store operations received from the memory crossbar 216 and the ROP 226. Read misses and emergency write-back requests are output by the L2 cache 221 to the framebuffer interface 225 for processing. Updates can also be sent to the framebuffer via the framebuffer interface 225 for processing. In one embodiment, the framebuffer interface 225 interfaces with one of the memory units in the parallel processor memory, such as memory units 224A-224N (e.g., in the parallel processor memory 222) in Figure 2A.

[0066] In graphics applications, ROP226 is a processing unit that performs raster operations such as stenciling, z-testing, blending, and similar operations. ROP226 then outputs the processed graphics data, which is stored in graphics memory. In some embodiments, ROP226 includes compression logic that compresses depth or color data written to memory and decompresses depth or color data read from memory. The compression logic may be a lossless compression logic created using one or more of several compression algorithms. The type of compression performed by ROP226 may vary based on the statistical characteristics of the data to be compressed. For example, in one embodiment, delta color compression is performed on depth and color data based on a per-tile criterion.

[0067] In some embodiments, the ROP 226 is contained within each processing cluster (e.g., clusters 214A-214N in Figure 2A) rather than within the partition unit 220. In such embodiments, read and write requests for pixel data, rather than pixel fragment data, are transmitted via the memory crossbar 216. The processed graphics data may be displayed on a display device, such as one of the one or more display devices 110 in Figure 1, and transferred by the processor 102 for further processing, or transferred for further processing by one of the processed entities in the parallel processor 200 in Figure 2A.

[0068] Figure 2C is a block diagram of a processing cluster 214 within a parallel processing unit according to an embodiment. In one embodiment, the processing cluster is one instance of processing clusters 214A-214N in Figure 2A. The processing cluster 214 may be configured to execute many parallel threads, where the term “thread” refers to an instance of a particular program running on a particular set of input data. In some embodiments, single-instruction multiple-data (SIMD) instruction issuing techniques are used to support the parallel execution of many threads without providing multiple independent instruction units. In other embodiments, single-instruction multiple-thread (SIMT) techniques are used to support the parallel execution of many generally synchronized threads using a common instruction unit configured to issue instructions to a set of processing engines within each of the processing clusters. Unlike SIMD execution, where all processing engines typically execute the same instructions, SIMT execution allows different threads to more easily follow a diffuse execution path through a given thread program. Those skilled in the art will understand that SIMD processing represents a functional subset of SIMT processing.

[0069] The operation of the processing cluster 214 can be controlled via a pipeline manager 232 that distributes processing tasks to SIMT parallel processors. The pipeline manager 232 receives instructions from the scheduler 210 in Figure 2A and manages the execution of these instructions via the graphics multiprocessor 234 and / or texture unit 236. The graphics multiprocessor 234 shown is an exemplary instance of a SIMT parallel processor. However, various types of SIMT parallel processors of different architectures may be included within the processing cluster 214. One or more instances of the graphics multiprocessor 234 may be included within the processing cluster 214. The graphics multiprocessor 234 can process data, and the data crossbar 240 may be used to distribute the processed data to one of several possible destinations, including other shader units. The pipeline manager 232 may facilitate the distribution of processed data by identifying the destinations for the processed data to be distributed via the data crossbar 240.

[0070] Each graphics multiprocessor 234 within the processing cluster 214 may contain an identical set of functional execution logic (e.g., arithmetic logic units, load storage units, etc.). The functional execution logic may be configured in a pipelined manner in which new instructions can be issued before previous instructions are completed. The functional execution logic supports a variety of operations, including integer and floating-point arithmetic, comparison operations, Boolean operations, bit shifts, and calculations of various algebraic functions. In one embodiment, the same functional unit hardware may be utilized to perform different operations, and any combination of functional units may be presented.

[0071] Instructions transmitted to the processing cluster 214 constitute threads. A set of threads that execute across a set of parallel processing engines is a thread group. A thread group executes the same program on different input data. Each thread in a thread group may be assigned to a different processing engine within the graphics multiprocessor 234. A thread group may contain fewer threads than the number of processing engines in the graphics multiprocessor 234. When a thread group contains fewer threads than the number of processing engines, one or more processing engines may be idle during the cycle in which the thread group is processed. A thread group may contain more threads than the number of processing engines in the graphics multiprocessor 234. When a thread group contains more threads than the number of processing engines in the graphics multiprocessor 234, processing may be executed over continuous clock cycles. In one embodiment, multiple thread groups may execute simultaneously on the graphics multiprocessor 234.

[0072] In one embodiment, the graphics multiprocessor 234 includes internal cache memory for performing load-and-store operations. In one embodiment, the graphics multiprocessor 234 may have cache memory (e.g., L1 cache 248) prior to the internal cache within the processing cluster 214. Each graphics multiprocessor 234 also has access to an L2 cache in a partition unit (e.g., partition units 220A-220N in Figure 2A) which is shared among all processing clusters 214 and can be used to transfer data between threads. The graphics multiprocessor 234 may also have access to off-chip global memory, which may include one or more local parallel processor memories and / or system memory. Any memory outside the parallel processing unit 202 may be used as global memory. In embodiments where the processing cluster 214 includes multiple instances of the graphics multiprocessor 234, common instructions and data may be shared, which may be stored in the L1 cache 248.

[0073] Each processing cluster 214 may include an MMU 245 (Memory Management Unit) configured to map virtual addresses to physical addresses. In other embodiments, one or more instances of the MMU 245 may reside in the memory interface 218 in Figure 2A. The MMU 245 includes set entries (PTEs) of a page table used to map virtual addresses to the physical addresses of tiles and optionally to cache line indices. The MMU 245 may include an address translation lookaside buffer (TLB), or a cache that may reside within the graphics multiprocessor 234, or an L1 cache, or the processing cluster 214. Physical addresses are processed to distribute surface data access locality, enabling efficient requests for interleaving between partition units. Cache line indices may be used to determine whether a cache line request is a hit or a miss.

[0074] In graphics and computing applications, the processing cluster 214 may be configured such that each graphics multiprocessor 234 is coupled to a texture unit 236 to perform texture mapping operations, such as determining texture sample locations, reading texture data, and filtering texture data. Texture data is read from an internal texture L1 cache (not shown), or, in some embodiments, from an L1 cache within the graphics multiprocessor 234, and fetched from an L2 cache, local parallel processor memory, or system memory as needed. Each graphics multiprocessor 234 outputs processed tasks to a data crossbar 240 to provide the processed tasks to another processing cluster 214 for further processing, or to store the processed tasks in an L2 cache, local parallel processor memory, or system memory via a memory crossbar 216. A preROP 242 (raster operation unit) is configured to receive data from the graphics multiprocessor 234 and direct the data to a ROP unit, which may be located in a partition unit as described herein (e.g., partition units 220A-220N in Figure 2A). The preROP242 unit can perform color blending optimization, pixel color data organization, and address translation.

[0075] It will be understood that the core architecture described herein is illustrative and subject to change and modification. Any number of processing units, such as graphics multiprocessors 234, texture units 236, preROP 242, etc., may be included within the processing cluster 214. Furthermore, while only one processing cluster 214 is shown, the parallel processing units described herein may include any number of instances of the processing cluster 214. In one embodiment, each processing cluster 214 may be configured to operate independently of other processing clusters 214 using isolated and separate processing units, L1 caches, etc.

[0076] Figure 2D shows a graphics multiprocessor 234 according to one embodiment. In such an embodiment, the graphics multiprocessor 234 is coupled with a pipeline manager 232 of a processing cluster 214. The graphics multiprocessor 234 has an execution pipeline that includes, but is not limited to, an instruction cache 252, an instruction unit 254, an address mapping unit 256, a register file 258, one or more general-purpose graphics processing unit (GPGPU) cores 262, and one or more load / store units 266. The GPGPU cores 262 and load / store units 266 are coupled with cache memory 272 and shared memory 270 via a memory and cache interconnect 268. In one embodiment, the graphics multiprocessor 234 further includes a tensor and / or ray tracing core 263 that includes hardware logic for accelerating matrix and / or ray tracing operations.

[0077] In one embodiment, the instruction cache 252 receives a stream of instructions to be executed from the pipeline manager 232. The instructions are cached in the instruction cache 252 and dispatched for execution by the instruction unit 254. The instruction unit 254 may dispatch instructions as a group of threads (e.g., warps), where each thread in the group is assigned to a different execution unit within the GPGPU core 262. Instructions can access any local, shared, or global address space by identifying addresses within a unified address space. An address mapping unit 256 may be used to translate addresses in the unified address space to distinct memory addresses that can be accessed by the load / store unit 266.

[0078] The register file 258 provides a set of registers to the functional units of the graphics multiprocessor 234. The register file 258 provides temporary storage for operands connected to the data paths of the functional units of the graphics multiprocessor 234 (e.g., GPGPU core 262, load / store unit 266). In one embodiment, the register file 258 is divided among the functional units so that each functional unit is allocated a dedicated portion of the register file 258. In one embodiment, the register file 258 is divided among different warps executed by the graphics multiprocessor 234.

[0079] The GPGPU core 262 may each include a floating-point unit (FPU) and / or an integer arithmetic logic unit (ALU) used to execute instructions of the graphics multiprocessor 234. Depending on the embodiment, the GPGPU core 262 may be similar in architecture or different in architecture. For example, and in one embodiment, the first part of the GPGPU core 262 includes a single-precision FPU and an integer ALU, while the second part of the GPGPU core includes a double-precision FPU. In one embodiment, the FPU may implement the IEEE 754-2008 standard for floating-point arithmetic or enable variable-precision floating-point arithmetic. The graphics multiprocessor 234 may further include one or more fixed-function or special-function units that perform specific functions such as rectangular copy or pixel blending operations. In one embodiment, one or more GPGPU cores may also include fixed or special-function logic.

[0080] In one embodiment, the GPGPU core 262 includes SIMD logic capable of executing a single instruction over multiple sets of data. In one embodiment, the GPGPU core 262 may physically execute SIMD4, SIMD8, and SIMD16 instructions and logically execute SIMD1, SIMD2, and SIMD32 instructions. SIMD instructions in the GPGPU core may be generated at compile time by the shader compiler, or may be automatically generated when executing a program written and compiled with respect to a single program multiple data (SPMD) or SIMT architecture. Multiple threads of a program configured for a SIMT execution model may be executed via a single SIMD instruction. For example, and in one embodiment, eight SIMT threads performing the same or similar operation may be executed in parallel via a single SIMD8 logic unit.

[0081] The memory and cache interconnect 268 is an interconnect network connecting each of the functional units of the graphics multiprocessor 234 to the register file 258 and the shared memory 270. In one embodiment, the memory and cache interconnect 268 is a crossbar interconnect that allows the load / store unit 266 to implement load and store operations between the shared memory 270 and the register file 258. The register file 258 may operate at the same frequency as the GPGPU core 262, and therefore data transfer between the GPGPU core 262 and the register file 258 is very low latency. The shared memory 270 may be used to enable communication between threads running on the functional units within the graphics multiprocessor 234. The cache memory 272 may be used as a data cache, for example, to cache texture data communicated between the functional units and the texture unit 236. The shared memory 270 may also be used as a program-managed cache. Threads running on the GPGPU core 262 may programmatically store data in shared memory, in addition to data automatically cached and stored in the cache memory 272.

[0082] Figures 3A to 3C show additional graphics multiprocessors according to an embodiment. Figures 3A to 3B show graphics multiprocessors 325 and 350, which are modifications of the graphics multiprocessor 234 in Figure 2C. Figure 3C shows a graphics processing unit (GPU) 380, which includes a set of graphics processing resources dedicated to graphics processing located in multicore groups 365A-365N. The shown graphics multiprocessors 325 and 350 and multicore groups 365A-365N may be streaming multiprocessors (SMs) capable of simultaneous execution with a large number of execution threads.

[0083] Figure 3A shows a graphics multiprocessor 325 according to an additional embodiment. Compared to the graphics multiprocessor 234 in Figure 2D, the graphics multiprocessor 325 includes multiple additional instances of execution resource units. For example, the graphics multiprocessor 325 may include multiple instances of instruction units 332A-332B, register files 334A-334B, and texture units 344A-344B. The graphics multiprocessor 325 also includes multiple sets of graphics or compute execution units (e.g., GPGPU cores 336A-336B, tensor cores 337A-337B, ray tracing cores 338A-338B) and multiple sets of load / store units 340A-340B. In one embodiment, the execution resource unit has a common instruction cache 330, texture and / or data cache memory 342, and shared memory 346.

[0084] Various components can communicate via the interconnect fabric 327. In one embodiment, the interconnect fabric 327 includes one or more crossbar switches that enable communication between various components of the graphics multiprocessor 325. In one embodiment, the interconnect fabric 327 is a separate high-speed network fabric layer on which each component of the graphics multiprocessor 325 is stacked. Components of the graphics multiprocessor 325 communicate with remote components via the interconnect fabric 327. For example, GPGPU cores 336A-336B, 337A-337B, and 3378A-338B can each communicate with shared memory 346 via the interconnect fabric 327. The interconnect fabric 327 can arbitrate communication within the graphics multiprocessor 325 to ensure fair bandwidth allocation between components.

[0085] Figure 3B shows a graphics multiprocessor 350 according to an additional embodiment. The graphics processor includes multiple sets of execution resources 356A-356D, each set of execution resources including multiple instruction units, register files, GPGPU cores, and load / store units, as illustrated in Figures 2D and 3A. Execution resources 356A-356D may work in conjunction with texture units 360A-360D for texture calculations, while sharing an instruction cache 354 and shared memory 353. In one embodiment, execution resources 356A-356D may share the instruction cache 354 and shared memory 353, as well as multiple instances of textures and / or data cache memories 358A-358B. Various components may communicate via the interconnect fabric 352, as with the interconnect fabric 327 in Figure 3A.

[0086] Those skilled in the art will understand that the architectures described in Figures 1, 2A-2D, and 3A-3B illustrate, but do not limit, the scope of these embodiments. Accordingly, the technologies described herein can be implemented on any appropriately configured processing unit, without departing from the scope of the embodiments described herein, including, but not limited to, one or more mobile application processors, one or more server central processing units (CPUs) including desktop or multicore CPUs, one or more parallel processing units such as the parallel processing unit 202 in Figure 2A, and one or more graphics processors or dedicated processing units.

[0087] In some embodiments, the parallel processors or GPGPUs described herein are communicatively coupled to a host / processor core to accelerate graphics computations, machine learning operations, pattern analysis operations, and various general-purpose GPU (GPGPU) functions. The GPU may be communicatively coupled to the host processor / core over a bus or other interconnect (e.g., a high-speed interconnect such as PCIe or NVLink). In other embodiments, the GPU may be integrated on the same package or chip as the core and communicatively coupled to the core (i.e., inside the package or chip) over an internal processor bus / interconnect. Regardless of how the GPU is connected, the processor core may assign operations to the GPU in the form of a sequence of commands / instructions contained in a work descriptor. The GPU then uses dedicated circuitry / logic to effectively process these commands / instructions.

[0088] Figure 3C shows a graphics processing unit (GPU) 380, including a dedicated set of graphics processing resources located in multicore groups 365A-N. While details are provided only for the single multicore group 365A, it will be understood that the other multicore groups 365B-365N may be equipped with the same or similar sets of graphics processing resources.

[0089] As illustrated, the multicore group 365A may include a set of graphics cores 370, a set of tensor cores 371, and a set of ray tracing cores 372. The scheduler / dispatcher 368 schedules and dispatches graphics threads for execution on various cores 370, 371, and 372. The set of register files 369 stores operand values ​​used by cores 370, 371, and 372 when the graphics threads are executed. These may include, for example, integer registers for storing integer values, floating-point registers for storing floating-point values, vector registers for storing packed data elements (integer and / or floating-point data elements), and tile registers for storing tensor / matrix values. In one embodiment, the tile registers are implemented as a combination of sets of vector registers.

[0090] One or more combinations of Level 1 (L1) caches and shared memory units 373 locally store graphics data such as texture data, vertex data, pixel data, ray data, and boundary volume data within each multicore group 365A. One or more texture units 374 may also be used to perform texturing operations such as texture mapping and sampling. A Level 2 (L2) cache 375 shares all or a subset of multicore groups 365A-365N that store graphics data and / or instructions for multiple parallel graphics threads. As shown in the figure, the L2 cache 375 may be shared across multiple multicore groups 365A-365N. One or more memory controllers 367 connect the GPU 380 to memory 366, which may be system memory (e.g., DRAM) and / or dedicated graphics memory (e.g., GDDR6 memory).

[0091] The input / output (I / O) circuit 363 connects the GPU 380 to one or more I / O devices 362, such as a digital signal processor (DSP), a network controller, or a user input device. An on-chip interconnect may be used to connect the I / O devices 362 to the GPU 380 and memory 366. One or more I / O memory management units (IOMMUs) 364 of the I / O circuit 363 directly connect the I / O devices 362 to system memory 366. In one embodiment, the IOMMU 364 manages multiple sets of page tables to map virtual addresses to physical addresses in system memory 366. In this embodiment, the I / O devices 362, CPU 361, and GPU 380 may share the same virtual address space.

[0092] In one implementation example, IOMMU364 supports virtualization. In this case, it may manage a first set of page tables that maps guest / graphics virtual addresses to guest / graphics physical addresses, and a second set of page tables that maps guest / graphics physical addresses to system / host physical addresses (e.g., within system memory 366). The base addresses of the first and second sets of page tables may be stored in control registers and swapped on a context switch (e.g., this provides a new context through access to the relevant set of page tables). While not shown in Figure 3C, each of the cores 370, 371, 372 and / or multicore groups 365A-365N may include a translation lookaside buffer (TLB) that caches guest virtuals in guest physical translations, guest physicals in host physical translations, and guest virtuals in host physical translations.

[0093] In one embodiment, the CPU 361, GPU 380, and I / O device 362 are integrated on a single semiconductor chip and / or chip package. The illustrated memory 366 may be integrated on the same chip or may be coupled to a memory controller 367 via an off-chip interface. In one implementation example, the memory 366 includes GDDR6 memory that shares the same virtual address space as other physical system-level memory, but the underlying principles of the present invention are not limited to this particular embodiment.

[0094] In one embodiment, the TensorCore 371 includes multiple execution units specifically designed to perform matrix operations, which are fundamental computational operations used to perform deep learning operations. For example, simultaneous matrix multiplication may be used for neural network training and inference. The TensorCore 371 can perform matrix operations with various operand accuracies, including single-precision floating-point (e.g., 32 bits), half-precision floating-point (e.g., 16 bits), integer words (16 bits), bytes (8 bits), and half-bytes (4 bits). In one embodiment, a neural network implementation extracts features from each rendered scene, potentially combining details from multiple frames to construct a high-quality final image.

[0095] In deep learning implementations, parallel matrix multiplication operations are scheduled for execution on the TensorCore 371. Training neural networks, in particular, requires a significant number of matrix dot multiplication operations. To handle the inner product formulation of NxNxN matrix multiplication, the TensorCore 371 may include at least N dot multiplication processing elements. Before matrix multiplication begins, one whole matrix is ​​loaded into a tile register, and at least one column of a second matrix is ​​loaded in each cycle for N cycles. In each cycle, there are N dot products to be processed.

[0096] Matrix elements may be stored with different precisions depending on the specific implementation, including 16-bit words, 8-bit bytes (e.g., INT8), and 4-bit half-bytes (e.g., INT4). Different precision modes may be specified with respect to the tensor core 371 to ensure that the most efficient precision is used for different workloads (e.g., inference workloads that may allow quantization to bytes and half-bytes).

[0097] In one embodiment, the ray tracing core 372 accelerates ray tracing operations with respect to both real-time and non-real-time ray tracing implementations. In particular, the ray tracing core 372 includes ray traversal / intersection circuitry for performing ray traversal using a boundary volume hierarchy (BVH) and identifying intersections between rays and primitives enclosed within a BVH volume. The ray tracing core 372 may also include circuitry for performing deep testing and culling (e.g., using a Z-buffer or similar arrangement). In one implementation example, the ray tracing core 372 performs traversal and intersection operations in conjunction with the image denoising techniques described herein, at least part of which may be performed on the tensor core 371. For example, in one embodiment, the tensor core 371 implements a deep learning neural network and performs denoising of frames generated by the ray tracing core 372. However, the CPU361, graphics core370, and / or ray tracing core372 may also implement all or part of the denoising and / or deep learning algorithms.

[0098] Furthermore, as described above, a distribution method for denoising may be employed in which the GPU380 is located in a computing device coupled to other computing devices on a network or high-speed interconnect. In this embodiment, the interconnected computing devices share neural network training / training data to improve the speed at which the overall system learns to perform denoising with respect to different types of image frames and / or different graphics applications.

[0099] In one embodiment, the ray tracing core 372 handles all BVH traversals and ray primitive intersections, protecting the graphics core 370 from overload with thousands of instructions per ray. In one embodiment, each ray tracing core 372 includes a first set of dedicated circuits for performing bounding box tests (e.g., for traversal operations) and a second set of dedicated circuits for performing ray triangle intersection tests (e.g., for intersecting rays being traversed). Thus, in one embodiment, the multicore group 365A can simply initiate ray probes, while the ray tracing cores 372 can independently perform ray traversals and intersections and return hit data (e.g., hit, no hit, multiple hits, etc.) to the thread context. The other cores 370, 371 are freed to perform other graphics or computation operations, while the ray tracing cores 372 perform traversals and intersection operations.

[0100] In one embodiment, each ray tracing core 372 includes a traversal unit that performs BVH test calculations and an intersection unit that performs ray primitive cross-tests. The intersection unit generates "hit," "no hit," or "multiple hit" responses and provides them to the appropriate thread. During traversal and intersection calculations, the execution resources of other cores (e.g., graphics core 370 and tensor core 371) are freed up to perform other forms of graphics work.

[0101] In one particular embodiment, the hybrid rasterization ray tracing approach described below is used, with the operation distributed between the graphics core 370 and the ray tracing core 372.

[0102] In one embodiment, the ray tracing core 372 (and / or other cores 370, 371) includes hardware support for ray tracing instruction sets such as Microsoft® DirectX Ray Tracing (DXR), including the DispatchRays command, as well as ray generation, nearest hit, arbitrary hit, and miss shaders, enabling the assignment of a unique set of shaders and textures for each object. Another ray tracing platform that may be supported by the ray tracing core 372, graphics core 370, and tensor core 371 is Vulkan 1.1.85. However, it should be noted that the underlying principles of the present invention are not limited to any particular ray tracing ISA.

[0103] Generally, various cores 372, 371, and 370 may support ray tracing instruction sets including instructions / functions related to ray generation, nearest hit, arbitrary hit, ray primitive intersection, per primitive, and hierarchical bounding box construction, misses, visits, and exceptions. More specifically, one embodiment includes ray tracing instructions that perform the following functions:

[0104] Ray Generation - Ray generation instructions can be executed for each pixel, sample, or other user-defined behavioral assignment.

[0105] Closest Hit - The Closest Hit command can perform the positioning of the nearest intersection of rays that have primitives in the scene.

[0106] Any Hit - The Any Hit command identifies multiple intersections between a ray and primitives in the scene and potentially identifies the next nearest intersection.

[0107] The Intersection instruction performs a ray-primitive crossover test and outputs the result.

[0108] Per-primitive Bounding Box Construction - This instruction constructs a bounding box around a given primitive or group of primitives (for example, when constructing a new BVH or other acceleration data structure).

[0109] Miss - Indicates that the ray does not apply to all of the geometry in the scene, or to a specific area of ​​the scene.

[0110] Visit-ray indicates the children's volume that the ray will traverse.

[0111] Exceptions - including various types of exception handlers (e.g., those triggered for various error conditions). Host processor interconnection technology related to GPUs.

[0112] Figure 4A shows an exemplary architecture in which multiple GPUs 410-413 are communicably coupled to multiple multicore processors 405-406 via high-speed links 440A-440D (e.g., bus, point-to-point interconnect). In one embodiment, high-speed links 440A-440D support communication throughput of 4 GB / s, 30 GB / s, 80 GB / s, or higher, depending on the implementation. Various interconnect protocols may be used, but are not limited to PCIe 4.0 or 5.0 and NVLink 2.0. However, the underlying principles of the present invention are not limited to any particular communication protocol or throughput.

[0113] In one embodiment, two or more GPUs 410-413 may be interconnected on high-speed links 442A-442B and implemented using the same or different protocols / links as those used for high-speed links 440A-440D. Similarly, two or more multicore processors 405-406 may be connected via high-speed link 443, which may operate as a symmetric multiprocessor (SMP) bus at speeds of 20 GB / s, 30 GB / s, 120 GB / s, or higher. Alternatively, all communication between the various system components shown in Figure 4A may be achieved using the same protocol / link (e.g., via a common interconnect fabric). As already stated, however, the underlying principles of the present invention are not limited to any particular type of interconnection technology.

[0114] In one embodiment, each multicore processor 405-406 is communicably coupled to processor memory 401-402 via memory interconnect 430A-430B, and each GPU 410-413 is communicably coupled to GPU memory 420-423 via GPU memory interconnect 450A-450D. The memory interconnects 430A-430B and 450A-450D may use the same or different memory access techniques. For illustrative purposes and without limitation, the processor memories 401-402 and GPU memories 420-423 may be volatile memory such as dynamic random access memory (DRAM) (including stacked DRAM), graphics DDR SDRAM (GDDR) (e.g., GDDR5, GDDR6), or high-bandwidth memory (HBM), and / or non-volatile memory such as 3D XPoint or Nano-Ram. In one embodiment, a portion of the memory may be volatile memory and another portion may be non-volatile memory (for example, using a two-level memory (2LM) hierarchy).

[0115] As described below, various processors 405-406 and GPUs 410-413 may be physically coupled to specific memories 401-402 and 420-423, respectively, but a unified memory architecture is implemented in which the same virtual system address space (also called the "effective address" space) is distributed among all of the various physical memories. For example, processor memories 401-402 may each contain 64GB of system memory address space, and GPU memories 420-423 may each contain 32GB of system memory address space (in this example, a total of 256GB of addressable memory is obtained).

[0116] Figure 4B shows additional details regarding the interconnection between a multicore processor 407 and a graphics acceleration module 446 according to one embodiment. The graphics acceleration module 446 may include one or more GPU chips integrated on a line card coupled to the processor 407 via a high-speed link 440. Alternatively, the graphics acceleration module 446 may be integrated on the same package or chip as the processor 407.

[0117] The illustrated processor 407 includes a plurality of cores 460A-460D, each having a translation lookaside buffer 461A-461D and one or more caches 462A-462D. The cores may include various other components for performing instruction and data processing, which are not shown to avoid ambiguity of the underlying principles of the invention (e.g., instruction fetch units, branch prediction units, decoders, execution units, reorder buffers, etc.). The caches 462A-462D may include Level 1 (L1) and Level 2 (L2) caches. Furthermore, one or more shared caches 456 may be included in the cache hierarchy and may be shared by the set of cores 460A-460D. For example, one embodiment of the processor 407 includes 24 cores, each having its own L1 cache, 12 shared L2 caches, and 12 shared L3 caches. In this embodiment, one of the L2 and L3 caches is shared by two adjacent cores. The processor 407 and the graphics accelerator integration module 446 are connected to the system memory 441, which may include processor memory 401-402.

[0118] Coherency is maintained with respect to data and instructions stored in various caches 462A-462D, 456 and system memory 441 via intercore communication over the coherence bus 464. For example, each cache may have associated cache coherency logic / circuitry to communicate over the coherence bus 464 in response to reads or writes to specific cache lines detected. In one implementation example, a cache snooping protocol is implemented over the coherence bus 464 to snoop cache accesses. Cache snooping / coherency techniques are well understood by those skilled in the art and will not be described in detail here to avoid obscuring the underlying principles of the invention.

[0119] In one embodiment, the proxy circuit 425 communicatively couples the graphics acceleration module 446 to the coherence bus 464, enabling the graphics acceleration module 446 to join the cache coherence protocol as a core peer. Specifically, interface 435 provides connectivity to the proxy circuit 425 over a high-speed link 440 (e.g., PCIe bus, NVLink, etc.), and interface 437 connects the graphics acceleration module 446 to the high-speed link 440.

[0120] In one implementation example, the accelerator integrated circuit 436 provides cache management, memory access, context management, and interrupt management services on behalf of the multiple graphics processing engines 431, 432, N of the graphics acceleration module 446. Each of the graphics processing engines 431, 432, N may include a separate graphics processing unit (GPU). Alternatively, the graphics processing engines 431, 432, N may include different types of graphics processing engines within the GPU, such as a graphics execution unit, a media processing engine (e.g., a video encoder / decoder), a sampler, and a bullet engine. In other words, the graphics acceleration module may be a GPU with multiple graphics processing engines 431-432, N, or the graphics processing engines 431-432, N may be individual GPUs integrated on a common package, line card, or chip.

[0121] In one embodiment, the accelerator integrated circuit 436 includes a memory management unit (MMU) 439 for performing various memory management functions, such as virtual-to-physical memory translation (also referred to as effective-to-real memory translation) and a memory access protocol for accessing system memory 441. The MMU 439 may also include a translation lookaside buffer (TLB) (not shown) for caching virtual / effective-to-physical / real address translations. In one implementation example, a cache 438 stores data and commands for efficient access by graphics processing engines 431-432, N. In one embodiment, data stored in the cache 438 and graphics memories 433-434, M are kept coherent with core caches 462A-462D, 456 and system memory 411. As already mentioned, this can be achieved via a proxy circuit 425 that participates in the cache coherency mechanism (e.g., sending updates to cache 438 related to modifying / accessing cache lines on processor caches 462A-462D, 456, and receiving updates from cache 438) on behalf of cache 438 and memory 433-434, M.

[0122] The set of registers 445 stores context data about threads executed by the graphics processing engines 431-432, N, and the context management circuit 448 manages the thread context. For example, the context management circuit 448 may perform save and restore operations to save and restore the contexts of various threads during a context switch (for example, where the first thread is saved and where the second thread is stored, so that a second thread can be executed by the graphics processing engine). For example, on a context switch, the context management circuit 448 may store the values ​​of the current registers in a specified area in memory (for example, identified by a context pointer). Then, it restores the register values ​​when returning to the context. In one embodiment, the interrupt management circuit 447 receives and processes interrupts received from system devices.

[0123] In one implementation example, virtual / effective addresses from the graphics processing engine 431 are translated to real / physical addresses in the system memory 411 by the MMU 439. One embodiment of the accelerator integrated circuit 436 supports multiple (e.g., 4, 8, 16) graphics accelerator modules 446 and / or other accelerator devices. The graphics accelerator modules 446 may be dedicated to a single application running on the processor 407, or they may be shared among multiple applications. In one embodiment, a virtualized graphics execution environment is presented in which the resources of the graphics processing engines 431-432, N are shared among multiple applications or virtual machines (VMs). The resources may be subdivided into “slices” that are allocated to different VMs and / or applications based on processing requests and priorities associated with the VMs and / or applications.

[0124] Therefore, the accelerator integrated circuit acts as a bridge to the system for the graphics acceleration module 446, providing address translation and system memory cache services. Furthermore, the accelerator integrated circuit 436 may provide the host processor with virtualization facilities to manage the virtualization of the graphics processing engine, interrupts, and memory management.

[0125] Since the hardware resources of the graphics processing engines 431-432, N are explicitly mapped to the real address space found in the host processor 407, any host processor can directly address these resources using effective address values. One function of the accelerator integrated circuit 436 is, in one embodiment, to physically isolate the graphics processing engines 431-432, N so that they appear in the system as independent units.

[0126] As already stated, in the embodiment shown, one or more graphics memories 433-434, M are coupled to each of the graphics processing engines 431-432, N. The graphics memories 433-434, M store instructions and data being processed by each of the graphics processing engines 431-432, N. The graphics memories 433-434, M may be volatile memory such as DRAM (including stacked DRAM), GDDR memory (e.g., GDDR5, GDDR6), or HBM, and / or non-volatile memory such as 3D XPoint or Nano-Ram.

[0127] In one embodiment, biasing techniques are used to reduce data traffic over the high-speed link 440, ensuring that the data stored in graphics memories 433-434, M is data that will be most frequently used by the graphics processing engines 431-432, N, and preferably not (or at least not frequently) used by cores 460A-460D. Similarly, the biasing mechanism attempts to hold data required by the cores (and preferably not required by the graphics processing engines 431-432, N) in the core caches 462A-462D, 456 and system memory 411.

[0128] Figure 4C shows another embodiment in which the accelerator integrated circuit 436 is integrated within the processor 407. In this embodiment, the graphics processing engines 431-432, N communicate directly with the accelerator integrated circuit 436 via the high-speed link 440, through interfaces 437 and 435 (and furthermore, any form of bus or interface protocol may be used). The accelerator integrated circuit 436 can perform the same operation as described with respect to Figure 4B, but potentially with higher throughput, closer to the coherence bus 464 and caches 462A-462D, 456.

[0129] One embodiment supports different programming models, including a dedicated processing programming model (without graphics acceleration module virtualization) and a shared programming model (with virtualization). The latter may include a programming model controlled by the accelerator integrated circuit 436 and a programming model controlled by the graphics acceleration module 446.

[0130] In one embodiment of the dedicated processing model, the graphics processing engines 431-432,N are dedicated to a single application or to processing within a single operating system. The single application can funnel other application requests to the graphics engines 431-432,N and provide virtualization within a VM / partition.

[0131] In a dedicated processing programming model, the graphics processing engines 431-432,N may share multiple VM / application partitions. The shared model requires the system hypervisor to virtualize the graphics processing engines 431-432,N, enabling access by each operating system. In a single-partition system without a hypervisor, the graphics processing engines 431-432,N are owned by the operating system. In both cases, the operating system may virtualize the graphics processing engines 431-432,N to provide access to each processing or application.

[0132] With respect to the shared programming model, the graphics acceleration module 446 or the individual graphics processing engines 431-432, N select process elements using process handles. In one embodiment, process elements are stored in system memory 411 and are addressable using the techniques for translating effective addresses to real addresses as described herein. The process handle may be an implementation-specific value provided to host processing when its context is registered by the graphics processing engines 431-432, N (i.e., the calling system software that adds the process elements to a list linked to the process elements). The lower 16 bits of the process handle may be the offset of the process element in the list linked to the process elements.

[0133] Figure 4D shows an exemplary accelerator-integrated slice 490. As used herein, a “slice” includes a specific portion of the processing resources of the accelerator integrated circuit 436. The application effective address space 482 in system memory 411 stores process elements 483. In one embodiment, process elements 483 are stored in response to GPU invocations 481 from an application 480 running on processor 407. Process elements 483 include the processing state of the corresponding application 480. A work descriptor (WD) 484 contained in process element 483 may be a single job requested by the application, or it may contain a pointer to a queue of jobs. In the latter case, WD 484 is a pointer to the job request queue in the application’s address space 482.

[0134] The graphics acceleration module 446 and / or individual graphics processing engines 431-432, N may be shared by all or a subset of the processing in the system. Embodiments of the present invention include infrastructure that sets up the processing state, sends WD484 to the graphics acceleration module 446, and starts a job in a virtualized environment.

[0135] In one implementation example, the dedicated processing programming model is specific to the implementation. In this model, a single process owns either a graphics acceleration module 446 or a separate graphics processing engine 431. Since the graphics acceleration module 446 is owned by a single process, when the graphics acceleration module 446 is allocated, the hypervisor initializes the accelerator integrated circuit 436 with respect to the partition it owns, and the operating system initializes the accelerator integrated circuit 436 with respect to the processing it owns.

[0136] In operation, the WD fetch unit 491 in the accelerator-integrated slice 490 fetches the next WD484, which contains an indicator of the operation performed by one of the graphics processing engines of the graphics acceleration module 446. The data from WD484 may be stored in register 445, as shown in the figure, and may be used by the MMU 439, interrupt management circuit 447, and / or context management circuit 448. For example, one embodiment of the MMU 439 includes a fragment / page walk circuit for accessing the fragment / page table 486 in the OS virtual address space 485. The interrupt management circuit 447 may process interrupt events 492 received from the graphics acceleration module 446. When performing graphics operations, the effective address 493 generated by the graphics processing engines 431-432, N is translated to a real address by the MMU 439.

[0137] In one embodiment, the same set of registers as 445 may be duplicated for each graphics processing engine 431-432, N and / or graphics acceleration module 446 and may be initialized by the hypervisor or operating system. Each of these duplicated registers may be included in the accelerator integration slice 490. Exemplary registers that may be initialized by the hypervisor are shown in Table 1. Table 1 - Hypervisor-initialized registers [Table 1]

[0138] Table 2 shows exemplary registers that can be initialized by the operating system. Table 2 - Operating System Initialized Registers [Table 2]

[0139] In one embodiment, each WD484 is specific to a particular graphics acceleration module 446 and / or graphics processing engines 431-432, N. It contains all the information that graphics processing engines 431-432, N request to perform its operation, or it may be a pointer to a memory location that an application has set up to terminate the command queue for the operation.

[0140] Figure 4E shows additional details relating to one embodiment of the shared model. This embodiment includes a hypervisor real address space 498 in which the process element list 499 is stored. The hypervisor real address space 498 is accessible via a hypervisor 496 that virtualizes the graphics acceleration module engine relating to the operating system 495.

[0141] The shared programming model allows all or a subset of processing from all or a subset of partitions in a system using the graphics acceleration module 446. There are two programming models in which the graphics acceleration module 446 is shared by multiple processing and partitions: time-sliced ​​sharing and graphics-oriented sharing.

[0142] In this model, the system hypervisor 496 owns the graphics acceleration module 446 and makes its functionality available to all operating systems 495. With respect to the graphics acceleration module 446 that supports virtualization by the system hypervisor 496, the graphics acceleration module 446 may comply with the following requirements: 1) Application job requests must be autonomous (i.e., state does not need to be maintained between jobs), or the graphics acceleration module 446 must provide a context saving and restoration mechanism. 2) The graphics acceleration module 446 must guarantee that application job requests will be completed in a certain amount of time, including any translation defects, or the graphics acceleration module 446 must provide the ability to preempt job processing. 3) If the graphics acceleration module 446 operates in the target shared programming model, it must guarantee fairness between processing.

[0143] In one embodiment, with respect to the shared model, application 480 is required to create a system call to operating system 495 having the type of graphics acceleration module 446, a work descriptor (WD), an authorization mask register (AMR) value, and a context save / restore area pointer (CSRP). The type of graphics acceleration module 446 describes the intended acceleration function for the system call. The type of graphics acceleration module 446 may be a system-specific value. The WD is formatted specifically with respect to graphics acceleration module 446 and may be in the form of a graphics acceleration module 446 command, an effective address pointer to a user-defined structure, an effective address pointer to a queue of commands, or any other data structure that describes the operation to be performed by graphics acceleration module 446. In one embodiment, the AMR value is the AMR state used for the current operation. The value passed to the operating system is the same as that of the application that sets the AMR. If the implementations of the accelerator integrated circuit 436 and the graphics acceleration module 446 do not support a User Authority Mask Override Register (UAMOR), the operating system may apply the current UAMOR value to the AMR value before passing the AMR in a hypervisor call. The hypervisor 496 may optionally apply the current Authority Mask Override Register (AMOR) value before placing the AMR in process element 483. In one embodiment, CSRP is one of the registers 445 located at an effective address in the application address space 482 for the graphics acceleration module 446 for saving and restoring context state. This pointer is optional if there is no state required to be saved during a job, or if the job is preempted. The context save / restore area may be pinned system memory.

[0144] Upon receiving the system call, the operating system 495 may verify that application 480 is registered and authorized to use the graphics acceleration module 446. The operating system 495 then invokes the hypervisor 496 with the information shown in Table 3. Table 3 - OS to Hypervisor Invocation Parameters [Table 3]

[0145] Upon receiving a hypervisor call, hypervisor 496 verifies that operating system 495 is registered and authorized to use graphics acceleration module 446. Hypervisor 496 then enters process element 483 into a list linked to process elements relating to the corresponding graphics acceleration module 446 type. The process elements may contain the information shown in Table 4. Table 4 - Process Element Information [Table 4]

[0146] In one embodiment, the hypervisor initializes multiple accelerator-integrated slice 490 registers 445.

[0147] As illustrated in Figure 4F, one embodiment of the present invention uses unified memory addressable via a common virtual memory address space used to access physical processor memories 401-402 and GPU memories 420-423. In this implementation, operations performed on GPUs 410-413 access processor memories 401-402 and vice versa using the same virtual / effective memory address space, thereby simplifying programmability. In one embodiment, a first portion of the virtual / effective address space is allocated to processor memory 401, a second portion to second processor memory 402, a third portion to GPU memory 420, and so on. The entire virtual / effective memory space (sometimes referred to as the effective address space) is thus distributed across processor memories 401-402 and GPU memories 420-423, respectively, allowing any processor or GPU to access any physical memory with virtual addresses mapped to that memory.

[0148] In one embodiment, bias / coherence management circuits 494A-494E within one or more MMUs 439A-439E ensure cache coherence between the cache of the host processor (e.g., 405) and the cache of the GPUs 410-413 and implement biasing techniques that indicate which physical memory should store which type of data. Multiple instances of the bias / coherence management circuits 494A-494E are illustrated in Figure 4F, while the bias / coherence circuits may be implemented within the MMUs of one or more host processors 405 and / or within the accelerator integrated circuit 436.

[0149] One embodiment allows GPU-attached memory 420-423 to be mapped as part of system memory and accessed using shared virtual memory (SVM) technology without suffering the typical performance drawbacks associated with full system cache coherence. The ability of GPU-attached memory 420-423 to be accessed as system memory without cumbersome cache coherence overhead provides a beneficial operating environment for GPU offloading. This arrangement allows software on the host processor 405 to set up operands and access computation results without the overhead of conventional I / O DMA data copying. Such conventional copies include driver calls, interrupts, and memory-mapped I / O (MMIO) access, all of which are inefficient compared to simple memory access. At the same time, the ability to access GPU-attached memory 420-423 without cache coherence overhead can be critical to the execution time of offload computations. For example, in the case of substantial streaming write memory traffic, cache coherence overhead can significantly reduce the effective write bandwidth seen by GPU 410-413. The efficiency of operand setup, the efficiency of result access, and the efficiency of GPU computation all play a role in determining the effectiveness of GPU offloading.

[0150] In one implementation, the choice between GPU bias and host processor bias is made by a bias tracker data structure. For example, a bias table may be used that is page-granular in structure (i.e., controlled at the granularity of memory pages) and contains 1 or 2 bits per GPU-attached memory page. The bias table may be implemented in one or more GPU-attached memory ranges 420-423, with or without a bias cache in GPUs 410-413 (e.g., caching frequently / recently used bias table entries). Alternatively, the entire bias table may be maintained within the GPU.

[0151] In one implementation example, bias table entries associated with each access to GPU-attached memory 420-423 are accessed before the actual access to GPU memory, causing the following actions: Firstly, local requests from GPUs 410-413 to find these pages in the GPU bias are forwarded directly to the corresponding GPU memory 420-423. Local requests from the GPU to find these pages in the host bias are forwarded to processor 405 (for example, over the high-speed link discussed above). In one embodiment, a request to processor 405 to find the requested page in the host processor bias completes a request similar to a standard memory read. Alternatively, a request for a page in the GPU bias may be forwarded to GPU 410-413. The GPU may then move the page to the host processor bias, which is not currently using that page.

[0152] The bias state of a page can be altered by either a software-based mechanism, a hardware-assisted software-based mechanism, or, in limited cases, a set of purely hardware-based mechanisms.

[0153] One mechanism for changing the bias state involves using an API call (e.g., OpenCL), then invoking the GPU's device driver, then sending a message to the GPU facing it (or queuing a command descriptor) to change the bias state, and performing a cache flush on the host for some transitions. The cache flush requires a transition from host processor 405 bias to GPU bias, but not the reverse transition.

[0154] In one embodiment, cache coherency maintains that GPU-biased pages are temporarily uncached by the host processor 405. To access these pages, processor 405 may, depending on the implementation, request access from the GPU 410, which may or may not immediately grant access. Therefore, it is useful to ensure that GPU-biased pages are requested by the GPU but not by the host processor 405, and vice versa, in order to reduce communication between the host processor 405 and the GPU 410. Graphics Processing Pipeline

[0155] Figure 5 shows a graphics processing pipeline 500 according to an embodiment. In one embodiment, a graphics processor may implement the illustrated graphics processing pipeline 500. In one embodiment, the graphics processor may be contained within a parallel processing subsystem described herein, such as the parallel processor 200 in Figure 2A, which is a variation of the parallel processor 112 in Figure 1. Various parallel processing systems may implement the graphics processing pipeline 500 through one or more instances of the parallel processing units described herein (e.g., the parallel processing unit 202 in Figure 2A). For example, a shader unit (e.g., the graphics multiprocessor 234 in Figure 2C) may be configured to perform one or more functions of the vertex processing unit 504, the tessellation control processing unit 508, the tessellation evaluation processing unit 512, the geometry processing unit 516, and the fragment / pixel processing unit 524. The functions of data assembler 502, primitive assemblers 506, 514, 518, tessellation unit 510, rasterizer 522, and raster operation unit 526 may also be performed by other processing engines within a processing cluster (e.g., processing cluster 214 in Figure 2A) and in corresponding partition units (e.g., partition units 220A-220N in Figure 2A). The graphics processing pipeline 500 may also be implemented using dedicated processing units for one or more functions. In one embodiment, one or more parts of the graphics processing pipeline 500 may be performed by parallel processing logic within a general-purpose processor (e.g., CPU). In one embodiment, one or more parts of the graphics processing pipeline 500 may access on-chip memory (e.g., parallel processor memory 222 as shown in Figure 2A) via a memory interface 528, which may be an instance of the memory interface 218 in Figure 2A.

[0156] In one embodiment, the data assembler 502 is a processing unit that collects vertex data relating to surfaces and primitives. The data assembler 502 then outputs the vertex data, including vertex attributes, to the vertex processing unit 504. The vertex processing unit 504 is a programmable execution unit that executes a vertex shader program and performs lighting and transformation of the vertex data identified by the vertex shader program. The vertex processing unit 504 may be programmed to read data stored in a cache, local, or system memory for use in processing the vertex data, and to transform the vertex data from an object-based coordinate representation to world space coordinate space or normalized device coordinate space.

[0157] The first instance of the primitive assembler 506 receives vertex attributes from the vertex processing unit 504. The primitive assembler 506 reads the stored vertex attributes as needed and configures the initial graphics for processing by the tessellation control processing unit 508. Graphics primitives include triangles, line segments, points, patches, etc., which are supported by various graphics processing application programming interfaces (APIs).

[0158] The tessellation control processing unit 508 processes input vertices as control points for the geometric patch. The control points are converted from the input representation from the patch (e.g., the patch base) to a representation suitable for use in surface evaluation by the tessellation evaluation processing unit 512. The tessellation control processing unit 508 may also calculate tessellation factors for the edges of the geometric patch. The tessellation factors are applied to a single edge and quantify the level of detail dependency associated with the edge. The tessellation unit 510 is configured to receive tessellation factors for the edges of the patch, tessellates the patch into multiple geometric primitives such as line, triangle, or quadrilateral primitives, and transmits them to the tessellation evaluation processing unit 512. The tessellation evaluation processing unit 512 operates on the parameterized coordinates of the subdivided patch and generates surface representations and vertex attributes for each vertex associated with the geometric primitives.

[0159] A second instance of the primitive assembler 514 receives vertex attributes from the tessellation evaluation processing unit 512, reads stored vertex attributes as needed, and configures the initial graphics for processing by the geometry processing unit 516. The geometry processing unit 516 is a programmable execution unit that runs a geometry shader program that transforms the graphics primitives received from the primitive assembler 514 as specified by the geometry shader program. In one embodiment, the geometry processing unit 516 is programmed to further divide the graphics primitives into one or more new graphics primitives and to calculate parameters used to rasterize the new graphics primitives.

[0160] In some embodiments, the geometry processing unit 516 may add or remove elements from the geometry stream. The geometry processing unit 516 outputs parameters and vertices that identify a new graphics primitive to the primitive assembler 518. The primitive assembler 518 receives the vertices and parameters from the geometry processing unit 516 and configures the initial graphics for processing by the domain scaling, culling, and clipping unit 520. The geometry processing unit 516 reads data stored in parallel processor memory or system memory for use in processing the geometry data. The domain scaling, culling, and clipping unit 520 performs clipping, culling, and viewpoint scaling and outputs the processed graphics primitive to the rasterizer 522.

[0161] The rasterizer 522 may perform depth culling and other depth-based optimizations. The rasterizer 522 also performs scan transformations on new graphics primitives, generating fragments and outputting these fragments and associated coverage data to the fragment / pixel processing unit 524. The fragment / pixel processing unit 524 is a programmable execution unit configured to run a fragment shader program or a pixel shader program. The fragment / pixel processing unit 524 transforms the fragments or pixels received from the rasterizer 522 as specified by the fragment or pixel shader program. For example, the fragment / pixel processing unit 524 may be programmed to perform operations including, but are not limited to, texture mapping, shading, blending, texture correction, and viewpoint correction, which generate shaded fragments or pixels output to the raster operation unit 526. The fragment / pixel processing unit 524 may read data stored in either parallel processor memory or system memory for use when processing fragment data. A fragment or pixel shader program may be configured to shade at a sample, pixel, tile, or other granularity, depending on the sampling rate configured with respect to the processing unit.

[0162] The raster operation unit 526 is a processing unit that performs raster operations, including but not limited to stenciling, z-testing, blending, and similar operations, and outputs pixel data as processed graphics data to be stored in graphics memory (e.g., parallel processor memory 222 as shown in Figure 2A, and / or system memory 104 as shown in Figure 1) for display on one or more display devices 110, or for further processing by one of one or more processors 102 or parallel processors 112. In some embodiments, the raster operation unit 526 is configured to compress z or color data written to memory and to decompress z or color data read from memory. Machine Learning Overview

[0163] The architecture described above may be applied to train and infer behavior using machine learning models. Machine learning has been successful in solving many types of tasks. The computations that occur when training and using machine learning algorithms (e.g., neural networks) are inherently beneficial for efficient parallel implementation. Therefore, parallel processors such as general-purpose graphics processing units (GPGPUs) have played a significant role in the practical implementation of deep neural networks. Parallel graphics processors with a Single Instruction, Multiple Threads (SIMT) architecture are designed to maximize the amount of parallel processing in the graphics pipeline. In the SIMT architecture, a group of parallel threads attempts to execute program instructions together synchronously as often as possible to increase processing efficiency. The efficiency provided by parallel machine learning algorithm implementations enables the use of high-capacity networks, allowing these networks to be trained on larger datasets.

[0164] Machine learning algorithms are algorithms that can learn based on a set of data. Multiple embodiments of machine learning algorithms can be designed to model high-level extraction within a dataset. For example, an image recognition algorithm can be used to determine which of several categories belongs to which of a given inputs, a regression algorithm can output a number given an input, and a pattern recognition algorithm can be used to generate transformed text or to perform conversation and / or speech recognition on text.

[0165] An exemplary type of machine learning algorithm is the neural network. There are many types of neural networks, and a simple type is the feedforward network. A feedforward network may be implemented as an acyclic graph where nodes are arranged in layers. Typically, a feedforward network topology includes input and output layers separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation useful for generating the output in the output layer. Network nodes are fully connected to nodes in adjacent layers via edges, but there are no edges between nodes within each layer. Data received at the input layer nodes of a feedforward network is propagated (i.e., "feedforward") to the output layer nodes via activation functions that calculate the state of each node in each continuous layer of the network based on coefficients ("weights") associated with each edge connecting the layers. Depending on the specific model represented by the algorithm being performed, the output from a neural network algorithm can take various forms.

[0166] Before a machine learning algorithm can be used to model a particular problem, it is trained using a training dataset. Training a neural network involves selecting a network topology, representing the problem to be modeled by the network using the training data set, and adjusting the weights until the network model runs with minimal error for all instances of the training dataset. For example, during the supervised learning training process for a neural network, the output generated by the network in response to an input representing an instance in the training dataset is compared to an output that is "correctly" classified for that instance. An error signal representing the difference between the output and the classified output is calculated, and the weights associated with the connections are adjusted to minimize the propagation of the error signal backward through the network layers. The network is considered "trained" when the error for each output generated from an instance of the training dataset is minimized.

[0167] The accuracy of machine learning algorithms can be greatly influenced by the quality of the dataset used to train them. The training process can be computationally intensive and require enormous amounts of time on conventional general-purpose processors. Therefore, parallel processing hardware is used to train many types of machine learning algorithms. This is particularly useful for optimizing the training of neural networks, as the computations performed when tuning coefficients within the neural network inherently benefit from parallel implementation. Specifically, many machine learning algorithms and software applications are adapted to the use of parallel processing hardware within general-purpose graphics processing devices.

[0168] Figure 6 is a generalized diagram of the machine learning software stack 600. The machine learning application 602 may be configured to train a neural network using a training dataset, or it may use a trained deep neural network to implement machine intelligence. The machine learning application 602 may include training and inference functions for the neural network and / or dedicated software that can be used to train the neural network before deployment. The machine learning application 602 may implement any type of machine intelligence, including, but is not limited to, image recognition, mapping and localization, autonomous navigation, speech synthesis, medical imaging, or language translation.

[0169] Hardware acceleration for machine learning applications 602 can be enabled through machine learning framework 604. Machine learning framework 604 can provide a library of machine learning primitives. Machine learning primitives are fundamental operations commonly performed by machine learning algorithms. Without machine learning framework 604, developers of machine learning algorithms would need to create and optimize the main computation logic associated with their machine learning algorithms, and then re-optimize that computation logic so that new parallel processors can be developed. Instead, machine learning applications can be configured to perform the necessary computations using primitives provided by machine learning framework 604. Illustrative primitives include tensor convolution, activation functions, and pooling, which are computational operations performed while training a convolutional neural network (CNN). Machine learning framework 604 can also provide primitives to implement fundamental linear algebra subprograms performed by many machine learning algorithms, such as matrix and vector operations.

[0170] The machine learning framework 604 can process input data received from the machine learning application 602 and generate appropriate input to the compute framework 606. The compute framework 606 can summarize the basic instructions provided to the GPGPU driver 608 so that the machine learning framework 604 can leverage hardware acceleration via the GPGPU hardware 610 without requiring the machine learning framework 604 to have detailed knowledge of the GPGPU hardware 610's architecture. Furthermore, the compute framework 606 can enable hardware acceleration for the machine learning framework 604 across various types and generational GPGPU hardware 610. GPGPU Machine Learning Acceleration

[0171] Figure 7 shows a general-purpose graphics processing unit 700 according to an embodiment. In one embodiment, the general-purpose processing unit (GPGPU) 700 may be configured to process computational workloads of a type associated with training deep neural networks particularly efficiently. Furthermore, the GPGPU 700 may be directly related to other examples of GPGPUs, particularly those that create multi-GPU clusters that improve training speed for deep neural networks.

[0172] The GPGPU 700 includes a host interface 702 that enables connection to a host processor. In one embodiment, the host interface 702 is a PCI Express interface. However, the host interface may be a vendor-specific communication interface or communication fabric. The GPGPU 700 receives commands from the host processor and uses a global scheduler 704 to distribute the execution threads associated with these commands to a set of compute clusters 706A-706H. The compute clusters 706A-706H share a cache memory 708. The cache memory 708 may function as a high-level cache within the cache memory of the compute clusters 706A-706H.

[0173] The GPGPU 700 includes memory 714A-B, which is coupled to compute clusters 706A-H via a set of memory controllers 712A-712B. In various embodiments, memory 714A-714B may include various types of memory devices, including graphics random access memory such as dynamic random access memory (DRAM) or synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. In one embodiment, memory 714A-714N may also include 3D stack memory, including but not limited to high-bandwidth memory (HBM).

[0174] In one embodiment, each of the compute clusters 706A-706H includes a set of graphics multiprocessors, such as the graphics multiprocessor 400 in Figure 4A. The graphics multiprocessors in the compute clusters are multiple types of integer and floating-point logic units capable of performing computational operations with a range of precisions, including those suitable for machine learning computations. For example, in one embodiment, at least a subset of the floating-point units in each of the compute clusters 706A-H may be configured to perform 16-bit or 32-bit floating-point operations, while different subsets of the floating-point units may be configured to perform 64-bit floating-point operations.

[0175] Multiple instances of the GPGPU 700 may be configured to operate as a compute cluster. The communication mechanism used by the compute cluster for synchronization and data exchange varies across embodiments. In one embodiment, multiple instances of the GPGPU 700 communicate via a host interface 702. In one embodiment, the GPGPU 700 includes an I / O hub 709 coupled to the GPGPU 700 by a GPU link 710 that enables direct connection to other instances of the GPGPU. In one embodiment, the GPU link 710 is coupled to a dedicated inter-GPU bridge that enables communication and synchronization between multiple instances of the GPGPU 700. In one embodiment, the GPU link 710 is coupled to a high-speed interconnect to send and receive data with other GPGPUs or parallel processors. In one embodiment, multiple instances of the GPGPU 700 are located on separate data processing systems and communicate via network devices accessible via the host interface 702. In one embodiment, the GPU link 710 may be configured to enable connection to a host processor in addition to or as an alternative to the host interface 702.

[0176] On the other hand, the illustrated configuration of the GPGPU700 may be configured to train a neural network, and in one embodiment, an alternative configuration of the GPGPU700 may be provided that can be configured for deployment within a high-performance or low-power inference platform. In the inference configuration, the GPGPU700 includes fewer compute clusters 706A-706H compared to the training configuration. Furthermore, the memory technology associated with memory 714A-714B may differ between the inference and training configurations. In one embodiment, the inference configuration of the GPGPU700 may support inferring specific instructions. For example, the inference configuration may provide support for one or more 8-bit integer dot product instructions, which are commonly used during the inference operation of an unfolded neural network.

[0177] Figure 8 shows a multi-GPU computing system 800 according to an embodiment. The multi-GPU computing system 800 may include a processor 802 coupled to a plurality of GPGPUs 806A-806D via a host interface switch 804. In one embodiment, the host interface switch 804 is a PCI Express switch device that couples the processor 802 to a PCI Express bus through which the processor 802 can communicate with a set of GPGPUs 806A-806D. Each of the plurality of GPGPUs 806A-806D may be an instance of the GPGPU 700 in Figure 7. The GPGPUs 806A-806D can be interconnected via a set of high-speed point-to-point inter-GPU links 816. The high-speed inter-GPU links can connect to each of the GPGPUs 806A-806D via a dedicated GPU link, such as the GPU link 710 shown in Figure 7. The P2P GPU link 816 enables direct communication between each of the GPGPUs 806A-806D without requiring the processor 802 to communicate via the host interface bus to which it is connected. Using the inter-GPU traffic over the P2P GPU link, the host interface bus remains available for system memory access or, for example, for communication with other instances of the multi-GPU computing system 800 via one or more network devices. Alternatively, in the shown embodiment, the GPGPUs 806A-D are connected to the processor 802 via the host interface switch 804, and in one embodiment, the processor 802 can be directly connected to the GPGPUs 806A-806D, including direct support for the P2P GPU link 816. Machine Learning Neural Network Implementation

[0178] The computing architectures provided by the embodiments described herein may be configured, in particular, to perform a type of parallel processing suitable for training and deploying neural networks for machine learning. A neural network can be generalized as a network of functions having graph relationships. As is well known in the art, various types of neural network implementations are used in machine learning. One exemplary type of neural network is the feedforward network, as previously described.

[0179] A second exemplary type of neural network is the convolutional neural network (CNN). A CNN is a dedicated feedforward neural network for processing data with a known grid topology, such as image data. Therefore, while CNNs are commonly used to compute visual and image recognition applications, they may also be used for other types of pattern recognition, such as speech and language processing. Nodes in the CNN input layer are grouped into sets of "filters" (feature detections inspired by receptive fields found in the retina), and the output of each set of filters is propagated to nodes in the network's continuous layers. Computation for a CNN involves applying a convolutional mathematical operation to each filter to produce its output. Convolution is a special mathematical operation performed by two functions to produce a third function, which is a modified version of one of two original functions. In the terminology of convolutional networks, the first function for convolution may be called the input, while the second function may be called the convolutional kernel. The output may be called a feature map. For example, the input to a convolutional layer can be a multidimensional array of data defining various color components of the input image. The convolutional kernel can be a multidimensional array of parameters, which are fitted through a training process for the neural network.

[0180] Recurrent neural networks (RNNs) are a family of feedforward neural networks that include feedback connections between layers. RNNs enable the modeling of sequential data by sharing parameter data across different parts of the neural network. The architecture of RNNs includes cycles. A cycle represents the effect that the current value of a variable has on its own value at a future point in time, so that at least a portion of the output data from the RNN is used as feedback to process subsequent inputs in sequence. This feature makes RNNs particularly useful for language processing due to the variability of how language data can be constructed.

[0181] The diagrams described below present exemplary feedforward, CNN, and RNN networks, and illustrate general processes for training and deploying each of these types of networks, respectively. These descriptions are illustrative and not limited to any specific embodiment described herein; it will be understood that the illustrated concepts are generally applicable to general deep neural network and machine learning techniques.

[0182] The exemplary neural network described above can be used to perform deep learning. Deep learning is machine learning that uses deep neural networks. The deep neural networks used in deep learning are artificial neural networks composed of multiple hidden layers, in contrast to shallow neural networks which contain only a single hidden layer. Deeper neural networks are generally more computationally intensive to train. However, the additional hidden layers in the network enable multi-step pattern recognition, which results in a reduction of output errors compared to shallow machine learning techniques.

[0183] Deep neural networks used in deep learning typically include a front-end network that performs feature recognition, coupled to a back-end network representing a mathematical model that can perform actions (e.g., object classification, speech recognition, etc.) based on feature representations provided to the mathematical model. Deep learning enables machine learning that does not require handcrafted feature engineering performed on the model. Instead, deep neural networks can learn features based on statistical constructions or correlations in the input data. The learned features can be provided to a mathematical model that can map the detected features to an output. The mathematical models used by the network are generally specific to the particular task being performed, and different models will be used to perform different tasks.

[0184] Once a neural network is constructed, a learning model can be applied to it to train the network to perform a specific task. The learning model describes how to adjust the weights in the model to reduce the network's output error. Backpropagation is a common method used to train neural networks. An input vector is presented to the network for processing. The network's output is compared to the desired output using a loss function, and an error value is calculated for each neuron in the output layer. The error values ​​are then propagated backward until each neuron has an associated error value that roughly represents its contribution to the original output. The network can then learn from these errors and update its weights using algorithms such as the stochastic gradient descent algorithm.

[0185] Figures 9A and 9B show an exemplary convolutional neural network. Figure 9A shows the various layers within the CNN. As shown in Figure 9A, an exemplary CNN used to model image processing can receive input 902 describing the red, green, and blue (RGB) components of an input image. Input 902 can be processed by multiple convolutional layers (e.g., convolutional layer 904, convolutional layer 906). The outputs from the multiple convolutional layers can optionally be processed by a set of fully connected layers 908. Neurons in the fully connected layers have full connectivity to all activations in the previous layers, as previously described for feedforward networks. The outputs from the fully connected layers 908 can be used to generate the output result from the network. Activations in the fully connected layers 908 can be computed using matrix multiplication instead of convolution. Not all CNN implementations utilize fully connected layers 908. For example, in some implementations, convolutional layer 906 may generate the output for the CNN.

[0186] The convolutional layers are loosely connected, which differs from the conventional neural network configuration found in fully connected layers 908. Conventional neural network layers are fully connected, and therefore every output unit interacts with every input unit. However, the convolutional layers are loosely connected because the output of the field's convolution is input to the nodes of the subsequent layer as illustrated (instead of each state value of each node in the field). The kernel associated with the convolutional layer performs the convolution operation of the output that is sent to the next layer. Dimensionality reduction performed within the convolutional layer is one aspect that enables the CNN to scale and process larger images.

[0187] Figure 9B shows an exemplary computational stage within a convolutional layer of a CNN. The input to the CNN's convolutional layer 912 may be processed in three stages of convolutional layer 914. The three stages may include a convolutional stage 916, a detection stage 918, and a pooling stage 920. Convolutional layer 914 may then output data to a continuum convolutional layer. The final convolutional layer of the network can generate output feature map data to produce classification values ​​for the input to the CNN, for example, or it can provide input to a fully connected layer.

[0188] The convolution stage 916 performs several convolutions in parallel to generate a set of linear activations. The convolution stage 916 may include affine transformations, which are any transformations that can be identified as linear transformations plus translations. Affine transformations include rotations, translations, scaling, and combinations of these transformations. The convolution stage computes the output of a function (e.g., a neuron) connected to a specific region in the input, which may be determined as the local region associated with the neuron. The neuron computes the dot product between the neuron's weight and the weight of the region in the local input to which the neuron is connected. The output from the convolution stage 916 defines a set of linear activations that are processed by successive stages of the convolution layer 914.

[0189] Linear activations can be processed by detection stage 918. In detection stage 918, each linear activation is processed by a nonlinear activation function. The nonlinear activation function enhances the nonlinear nature of the entire network without affecting the receptive field of the convolutional layer. Several types of nonlinear activation functions can be used. One particular type is the normalized linear unit (ReLU), which uses an activation function defined as f(x) = max(0,x) such that activations are thresholded to zero.

[0190] The pooling stage 920 employs a pooling function that replaces the output of the convolutional layer 906 with a summary statistic of the output of the nearest nearby output. The pooling function can be used to introduce translation invariance into the neural network, so that small transformations to the input do not alter the pooled output. Invariance to local movement can be useful in scenarios where the presence of features in the input data is more important than the exact location of the features. Various types of pooling functions can be used during the pooling stage 920, including max pooling, mean pooling, and l2-norm pooling. Furthermore, some CNN implementations do not include a pooling stage. Instead, such implementations have alternative and additional convolutional stages with increased stride compared to the previous convolutional stages.

[0191] The output from the convolutional layer 914 can then be processed by the next layer 922. The next layer 922 can be an additional convolutional layer or one of the fully connected layers 908. For example, the first convolutional layer 904 in Figure 9A can output to a second convolutional layer 906, while the second convolutional layer can output to the first layer of the fully connected layer 908.

[0192] Figure 10 shows an exemplary recurrent neural network 1000. In a recurrent neural network (RNN), the previous state of the network influences the output of the network's current state. RNNs can be constructed in various ways using various functions. The use of RNNs generally revolves around predicting the future based on a previous set of inputs using mathematical models. For example, an RNN can be used to perform statistical language modeling to predict the next word given a previous set of words. The illustrated RNN 1000 that can be described has an input layer 1002 that receives an input vector, a hidden layer 1004 that implements the recurrent function, a feedback mechanism 1005 that enables a "memory" of the previous state, and an output layer 1006 that outputs the result. The RNN 1000 operates on a timestep basis. The state of the RNN at a given timestep is influenced based on the previous timestep via the feedback mechanism 1005. For a given timestep, the state of the hidden layer 1004 is defined by the previous state and the input at the current timestep. The initial input (x1) of the first time step may be processed by the hidden layer 1004. The second input (x2) may be processed by the hidden layer 1004 using state information determined during the processing of the initial input (x1). Given a state, s t =f(Ux t +Ws t-1 It can be calculated as follows, where U and W are parameter matrices: The function f is generally nonlinear, such as the hyperbolic tangent function (Tanh) or a variation of the normalized function f(x) = max(0,x). However, the specific mathematical function used in the hidden layer 1004 may vary depending on the details of a particular embodiment of the RNN 1000.

[0193] In addition to the basic CNN and RNN networks described, variations on these networks may be possible. One exemplary variation of an RNN is the Long Short-Term Memory (LSTM) RNN. The LSTM RNN enables the learning of long-term dependencies, which may be required to process longer sequences of language. A variation on a CNN is the Convolutional Deep Belief Network, which has a similar structure to a CNN and is trained in a similar manner to a deep belief network. A Deep Belief Network (DBN) is a generative neural network consisting of multiple layers of random variables. A DBN can be trained layer by layer using greedy unsupervised learning. The weights learned in the DBN can then be used to provide a pre-trained neural network by determining the optimal initial set of weights for the neural network.

[0194] Figure 11 illustrates the training and deployment of a deep neural network. Once a given network is built for the task, the neural network is trained using the training dataset 1102. Various training frameworks 1104 have been developed to enable hardware acceleration of the training process. For example, the machine learning framework 604 in Figure 6 can be configured as a training framework 604. The training framework 604 can connect to an untrained neural network 1106 and allow the untrained neural network to be trained using the parallel processing resources described herein in order to produce a trained neural network 1108.

[0195] To begin the training process, the initial weights may be selected randomly or by pre-training using a deep belief network. The training cycle is then performed either supervised or unsupervised.

[0196] Supervised learning is a learning method in which training is performed as an intermediary, for example, when the training dataset 1102 contains inputs that are paired with desired outputs for each input, or when the training dataset contains inputs with known outputs and the outputs of the neural network are manually graded. The network processes the inputs and compares the resulting outputs to a set of expected or desired outputs. Errors are then propagated throughout the system. The training framework 1104 may be tuned to adjust the weights that control the untrained neural network 1106. The training framework 1104 can provide a tool to monitor how well the untrained neural network 1106 is converging toward a model that is suitable for generating correct answers based on known input data. The training process is repeated so that the network's weights are adjusted to improve the outputs produced by the neural network. The training process may continue until the neural network reaches a statistically desirable accuracy associated with the trained neural network 1108. The trained neural network 1108 can then be deployed to implement any number of machine learning operations that generate inference results 1114 based on the input of new data 1112.

[0197] Unsupervised learning is a learning method in which a network attempts to train itself using unclassified data. Therefore, in the context of unsupervised learning, the training dataset 1102 would contain input data without any associated output data. An untrained neural network 1106 can learn groupings within the unclassified inputs and determine how individual inputs relate to the entire dataset. Unsupervised training can be used to generate self-organizing maps, and certain types of trained neural networks 1108 are capable of performing useful operations to reduce the dimensionality of the data. Unsupervised training can also be used to perform anomaly detection, enabling the identification of data points in an input dataset that deviate from the normal pattern of the data.

[0198] Variations of supervised and unsupervised training may be employed. Semi-supervised learning is a technique in which the training dataset 1102 contains a mixture of classified and unclassified data of the same variance. Incremental learning is a variation of supervised learning in which input data is continuously used to further train the model. Incremental learning allows the trained neural network 1108 to apply new data 1112 without forgetting the knowledge that was taught to the network during the initial training.

[0199] Whether supervised or unsupervised, the training process for deep neural networks, in particular, can be quite computationally intensive on a single computing node. Instead of using a single computing node, a distributed network of computing nodes can be used to accelerate the training process.

[0200] Figure 12 is a block diagram of distributed learning. Distributed learning is a training model that uses multiple distributed computing nodes to perform supervised or unsupervised training of a neural network. Each distributed computing node may include one or more host processors and one or more general-purpose processing nodes, such as the highly parallel general-purpose graphics processing unit 700 shown in Figure 7. As illustrated, distributed learning may perform model parallel processing 1202, data parallel processing 1204, or a combination of model and data parallel processing 1204.

[0201] In model parallelism 1202, different computing nodes within a distributed system can perform training computations on different parts of a single network. For example, each layer of a neural network can be trained by different processing nodes in a distributed system. The advantages of model parallelism include the ability to scale to particularly large models. Splitting the computations associated with different layers of a neural network allows for the training of very large neural networks where the weights of all layers do not fit into the memory of a single computing node. In some examples, model parallelism can be particularly useful when performing unsupervised training of large neural networks.

[0202] In data parallelism 1204, different nodes in a distributed network have complete instances of the model, and each node receives a different portion of the data. The results from the different nodes are then combined. While different approaches to data parallelism are possible, all data parallel training approaches require techniques to combine the results and synchronize the model parameters across each node. Exemplary approaches to combining data include parameter averaging and update-based data parallelism. Parameter averaging trains each node with a subset of the training data and sets global parameters (e.g., weights, biases) to the average of the parameters from each node. Parameter averaging uses a central parameter server that holds the parameter data. Update-based data parallelism is similar to parameter averaging except that model updates are transferred instead of parameters being transferred from the nodes to the parameter server. Furthermore, update-based data parallelism can be performed in a distributed manner, and updates are compressed and transferred between nodes.

[0203] The combined model and data parallel processing 1206 can be implemented, for example, in a distributed system where each computing node includes multiple GPUs. Each node may have a complete instance of the model, along with separate GPUs within each node used to train different parts of the model.

[0204] Distributed training has increased overhead compared to training on a single machine. However, the parallel processors and GPGPUs described herein can each implement various techniques to reduce the overhead of distributed training, including techniques that enable high-bandwidth GPU-to-GPU data transfer and accelerated remote data synchronization. Exemplary machine learning applications

[0205] Machine learning can be applied to solve a wide range of technical challenges, and is not limited to computer vision, autonomous driving and navigation, speech recognition, and natural language processing. Computer vision has historically been one of the most active research areas for machine learning applications. Computer vision applications range from recreating human visual abilities, such as face recognition, to creating new categories of visual abilities. For example, a computer vision application may be configured to recognize sound waves from vibrations caused by visible objects in a video. Parallel processor-accelerated machine learning allows computer vision applications to be trained with much larger training datasets than previously possible and enables inference systems to be deployed using low-power parallel processors.

[0206] Parallel processor-accelerated machine learning has applications in autonomous driving, including lane and road sign recognition, obstacle avoidance, navigation, and driving control. Accelerated machine learning techniques can be used to train driving models based on datasets that define appropriate responses to specific training inputs. The parallel processors described herein enable the rapid training of increasingly complex neural networks used in autonomous driving solutions and can enable the deployment of low-power inference processors in mobile platforms suitable for integration into autonomous vehicles.

[0207] Parallel processor-accelerated deep neural networks have enabled machine learning approaches to automatic speech recognition (ASR). ASR involves creating a function that, given an input acoustic sequence, calculates the most likely language sequence. Accelerated machine learning using deep neural networks has enabled the replacement of previously used hidden Markov models (HMMs) and Gaussian mixture models (GMMs) for ASR.

[0208] Parallel processor-accelerated machine learning can also be used to accelerate natural language processing. Automated learning procedures can utilize statistical inference algorithms to generate models that are robust to erroneous or unfamiliar inputs. Exemplary natural language processing applications include automated machine translation between human languages.

[0209] Parallel processing platforms used in machine learning can be divided into training platforms and platform deployments. Training platforms are generally highly parallel and include optimizations to accelerate multi-GPU single-node training and multi-node, multi-GPU training. Exemplary parallel processors suitable for training include the general-purpose graphics processing unit 700 in Figure 7 and / or the multi-GPU computing system 800 in Figure 8. Conversely, machine learning platforms deployed generally include low-power parallel processors suitable for use in products such as cameras, autonomous robots, and autonomous vehicles.

[0210] Figure 13 shows an exemplary inference system-on-chip (SoC) 1300 suitable for performing inference using a trained model. The SoC 1300 may integrate processing components including a media processor 1302, a vision processor 1304, a GPGPU 1306, and a multicore processor 1308. The SoC 1300 may further include on-chip memory 1305, which can enable a shared on-chip data pool accessible by each of the processing components. The processing components may be optimized for low-power operation, enabling deployment to various machine learning platforms, including autonomous vehicles and autonomous robots. For example, one implementation of the SoC 1300 may be used as part of a primary control system for an autonomous vehicle. When the SoC 1300 is configured for use in an autonomous vehicle, the SoC is designed and configured to comply with relevant functional safety standards for deployment authorization.

[0211] During operation, the media processor 1302 and the vision processor 1304 can operate simultaneously to accelerate computer vision operations. The media processor 1302 can enable low-latency decoding of multiple high-resolution (e.g., 4K, 8K) video streams. The decoded video streams can be written to a buffer in on-chip memory 1305. The vision processor 1304 can then analyze the decoded video and perform preliminary operations on the frames of the decoded video in preparation for frame processing using a trained image recognition model. For example, the vision processor 1304 can accelerate CNN convolution operations used to perform image recognition on high-resolution video data, while the backend model calculations are performed by the GPGPU 1306.

[0212] The multicore processor 1308 may include control logic to assist in the sequencing and synchronization of data transfer and shared memory operations performed by the media processor 1302 and the vision processor 1304. The multicore processor 1308 may also function as an application processor running software applications that can utilize the inference compute capabilities of the GPGPU 1306. For example, at least a portion of the navigation and driving logic may be implemented in software running on the multicore processor 1308. Such software may directly issue computational workloads to the GPGPU 1306, or computational workloads may be issued to the multicore processor 1308, which may offload at least a portion of their operations to the GPGPU 1306.

[0213] The GPGPU1306 may include compute clusters, such as a low-power configuration of compute clusters 706A-706H within the general-purpose graphics processing unit 700. The compute clusters within the GPGPU1306 may support instructions specifically optimized for performing inference computations on trained neural networks. For example, the GPGPU1306 can support instructions for performing low-precision calculations, such as 8-bit and 4-bit integer vector operations. System Overview

[0214] Figure 14 is a block diagram of a processing system 1400 according to an embodiment. The system 1400 may be used in a single-processor desktop system, a multi-processor workstation system, or a server system having multiple processors 1402 or processor cores 1407. In one embodiment, the system 1400 is a processing platform embedded in a system-on-a-chip (SoC) integrated circuit for use in mobile, handheld, or embedded devices, such as Internet of Things (IoT) devices with wired or wireless connectivity to a local or wide area network.

[0215] In one embodiment, System 1400 may include, be coupled to, or be integrated into a server-based game platform, i.e., a game console including a game and media console, a mobile game console, a handheld game console, or an online game console. In some embodiments, System 1400 is part of a mobile phone, smartphone, tablet computing device, or mobile internet-connected device, such as a laptop with low internal storage capacity. Processing System 1400 may include, be coupled to, or be integrated into a wearable device, such as a smartwatch wearable device, smart eyewear or clothing enhanced with augmented reality (AR) or virtual reality (VR) features, other augmented reality (AR) devices, or other virtual reality (VR) devices, for providing visual, auditory, and haptic outputs to complement real-world visual, auditory, and haptic experiences, or to provide text, voice, graphics, video, holographic images or videos, or haptic feedback. In some embodiments, Processing System 1400 includes, or is part of, a television or set-top box device. In one embodiment, the system 1400 may include, and may be coupled with or integrated into, an autonomous vehicle such as a bus, tractor trailer, automobile, motor or electric bicycle, airplane or glider (or any combination thereof). The autonomous vehicle may use the system 1400 to process the environment sensed around the vehicle.

[0216] In some embodiments, one or more processors 1402 each include one or more processor cores 1407 that, if executed, process instructions for performing actions for the system or user software. In some embodiments, at least one of the one or more processor cores 1407 is configured to process a particular instruction set 1409. In some embodiments, the instruction set 1409 may facilitate computing via composite instruction set computing (CISC), reduced instruction set computing (RISC), or very long instruction words (VLIW). One or more processor cores 1407 may each process different instruction sets 1409 and may include instructions that facilitate the emulation of other instruction sets. The processor cores 1407 may also include other processing devices, such as digital signal processors (DSPs).

[0217] In some embodiments, the processor 1402 includes cache memory 1404. Depending on the architecture, the processor 1402 may have a single internal cache or multiple levels of internal caches. In some embodiments, the cache memory is shared among various components of the processor 1402. In some embodiments, the processor 1402 also uses an external cache (e.g., a Level 3 (L3) cache or a Last Level Cache (LLC)) (not shown) which may be shared among multiple processor cores 1407 using known cache coherency techniques. The register file 1406 may further include different types of registers (e.g., integer registers, floating-point registers, status registers, and instruction pointer registers) to store different types of data. Some registers may be general-purpose registers, while others may be specific to the design of the processor 1402.

[0218] In some embodiments, one or more processors 1402 are coupled with one or more interface buses 1410 that transmit communication signals, such as addresses, data, or control signals, between the processors 1402 and other components of the system 1400. In one embodiment, the interface bus 1410 may be a processor bus, such as a version of a Direct Media Interface (DMI) bus. However, the processor bus is not limited to a DMI bus and may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In one embodiment, the processor 1402 includes an integrated memory controller 1416 and a platform controller hub 1430. The memory controller 1416 facilitates communication between memory devices and other components of the system 1400, while the platform controller hub (PCH) 1430 provides connectivity to I / O devices via a local I / O bus.

[0219] The memory device 1420 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase-change memory device, or several other memory devices having performance suitable for functioning as processing memory. In one embodiment, the memory device 1420 may act as system memory for the system 1400, storing data 1422 and instructions 1421 for use when one or more processors 1402 execute an application or processing process. The memory controller 1416 may also be coupled with an optional external graphics processor 1418, which may communicate with one or more graphics processors 1408 within processor 1402 to perform graphics and media operations. In some embodiments, graphics, media, and / or computation operations may be assisted by an accelerator 1412, which is a coprocessor that can be configured to perform a dedicated set of graphics, media, or computation operations. For example, in one embodiment, the accelerator 1412 is a matrix multiplication accelerator used to optimize machine learning or computation operations. In one embodiment, accelerator 1412 is a ray tracing accelerator that can be used to perform ray tracing operations in conjunction with graphics processor 1408. In one embodiment, external accelerator 1419 can be used instead of or in conjunction with accelerator 1412.

[0220] In some embodiments, the display device 1411 may be connected to the processor 1402. The display device 1411 may be one or more internal display devices, such as those found in mobile electronic devices or laptop devices, or external display devices (e.g., DisplayPort) attached via a display interface. In one embodiment, the display device 1411 may be a stereoscopic display device for use in a head-mounted display (HMD), such as in virtual reality (VR) or augmented reality (AR) applications.

[0221] In some embodiments, the platform controller hub 1430 allows peripherals to connect to the memory device 1420 and processor 1402 via a high-speed I / O bus. I / O peripherals include, but are not limited to, an audio controller 1446, a network controller 1434, a firmware interface 1428, a wireless transceiver 1426, a touch sensor 1425, and a data storage device 1424 (e.g., non-volatile memory, volatile memory, hard disk drive, flash memory, NAND, 3D NAND, 3D XPoint, etc.). The data storage device 1424 can be connected via a storage interface (e.g., SATA) or via a peripheral bus such as a peripheral component interconnect bus (e.g., PCI, PCI Express). The touch sensor 1425 may include a touchscreen sensor, a pressure sensor, or a fingerprint sensor. The wireless transceiver 1426 may be a Wi-Fi® transceiver, a Bluetooth® transceiver, or a mobile network transceiver such as a 3G, 4G, 5G, or Long-Term Evolution (LTE) transceiver. The firmware interface 1428 enables communication with the system firmware, which may be, for example, a Unified Extensible Firmware Interface (UEFI). The network controller 1434 may enable network connectivity to a wired network. In some embodiments, a high-performance network controller (not shown) is coupled to the interface bus 1410. The audio controller 1446 is, in one embodiment, a multi-channel high-resolution audio controller. In one embodiment, the system 1400 includes an optional legacy I / O controller 1440 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to the system. The platform controller hub 1430 can also connect to one or more Universal Serial Bus (USB) controller 1442-connected input devices, such as a keyboard and mouse combination 1443, a camera 1444, or other USB input devices.

[0222] It will be understood that the system 1400 shown is illustrative and not limiting, as other types of data processing systems configured differently may also be used. For example, instances of the memory controller 1416 and platform controller hub 1430 may be integrated into a separate external graphics processor, such as an external graphics processor 1418. In one embodiment, the platform controller hub 1430 and / or the memory controller 1416 may be external to one or more processors 1402. For example, system 1400 may include an external memory controller 1416 and platform controller hub 1430, which may be configured as a memory controller hub and peripheral controller hub within a system chipset that communicate with the processor 1402.

[0223] For example, a circuit board ("SLED") may be used in which components such as the CPU, memory, and other components are arranged and designed to increase thermal performance. In some examples, processing components such as the processor are located on the upper side of the SLED, while on the other hand, they may be located near memory such as DIMMs on the lower side of the SLED. As a result of the increased airflow provided by this design, the components can operate at higher frequencies and power levels than a typical system, thereby increasing performance. Furthermore, the SLED is configured in a rack to unconditionally mesh with power and data communication cables, thereby extending their ability to be quickly removed, upgraded, reinstalled, and / or replaced. Similarly, individual components located on the SLED, such as the processor, accelerator, memory, and data storage devices, are configured to be easily upgraded due to the increased space between them. In exemplary embodiments, the components further include hardware proof features that reveal their authenticity.

[0224] A data center may use a single network architecture ("fabric") that supports multiple other network architectures, including Ethernet® and OmniPath. The SLED may be coupled to switches via optical fiber, which provides higher bandwidth and lower latency than typical twisted-pair cabling (e.g., Category 5, Category 5e, Category 6, etc.). Due to the high bandwidth, low latency interconnect and network architecture, a data center may pool resources such as memory, accelerators (e.g., GPUs, graphics accelerators, FPGAs, ASICs, neural networks and / or artificial intelligence accelerators, etc.), and data storage devices that are not physically aggregated, in use, provide them to computing resources (e.g., processors) as needed, and allow computing resources to access the pooled resources as if they were locally located.

[0225] A power supply or power source may provide voltage and / or current to System 1400 or any component or system as described herein. In one example, the power supply includes an AC-to-DC (alternating current to direct current) adapter for plugging into a wall outlet. Such AC power may be a renewable energy (e.g., solar power) source. In one example, the power source includes a DC power source, such as an external AC-to-DC converter. In one example, the power source or power supply includes wireless charging hardware that charges via a charging field in proximity. In one example, the power source may include an internal battery, an AC power supply, a motion-based power supply, a solar power supply, or a fuel cell power supply.

[0226] Figures 15A to 15C show computing systems and graphics processors provided by embodiments described herein. Elements of Figures 15A to 15C having the same reference numeral (or name) as elements of any other drawings herein may, but are not limited to, operate or function in any manner similar to those described elsewhere herein.

[0227] Figure 15A is a block diagram of an embodiment of a processor 1500 having one or more processor cores 1502A-1502N, an integrated memory controller 1514, and an integrated graphics processor 1508. The processor 1500 may include additional cores, up to an additional core 1502N represented by a dashed box. Each of the processor cores 1502A-1502N includes one or more internal cache units 1504A-1504N. In some embodiments, each processor core also has access to one or more shared cache units 1506. The internal cache units 1504A-1504N and the shared cache units 1506 represent a cache memory hierarchy within the processor 1500. The cache memory hierarchy may include at least one level of instruction and data cache within each processor core, and one or more levels of shared intermediate cache levels such as cache level 2 (L2), level 3 (L3), level 4 (L4), or other levels that are the highest level of cache before external memory is classified as LLC. In some embodiments, cache coherency logic maintains coherency between various cache units 1506 and 1504A-1504N.

[0228] In some embodiments, the processor 1500 may also include a set of one or more bus controller units 1616 and a system agent core 1510. The one or more bus controller units 1616 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. The system agent core 1510 provides management functions for various processor components. In some embodiments, the system agent core 1510 includes one or more integrated memory controllers 1514 that manage access to various external memory devices (not shown).

[0229] In some embodiments, one or more processor cores 1502A-1502N include support for simultaneous multithreading. In such embodiments, a system agent core 1510 includes components for coordinating and operating the cores 1502A-1502N during multithreaded processing. The system agent core 1510 may further include a power control unit (PCU), which includes logic and components for adjusting the power states of the processor cores 1502A-1502N and the graphics processor 1508.

[0230] In some embodiments, the processor 1500 further includes a graphics processor 1508 that performs graphics processing operations. In some embodiments, the graphics processor 1508 is coupled with a set of shared cache units 1506 and a system agent core 1510 which includes one or more integrated memory controllers 1514. In some embodiments, the system agent core 1510 also includes a display controller 1511 that drives the graphics processor output to one or more coupled displays. In some embodiments, the display controller 1511 may be a separate module coupled with the graphics processor via at least one interconnection, or it may be integrated within the graphics processor 1508.

[0231] In some embodiments, a ring-based interconnect unit 1512 is used to connect the internal components of the processor 1500. However, alternative interconnect units may be used, such as point-to-point interconnects, switch-type interconnects, or other technologies, including those well known in the art. In some embodiments, a graphics processor 1508 is connected to the ring interconnect 1512 via an I / O link 1513.

[0232] An exemplary I / O link 1513 represents at least one of several diverse I / O interconnects, including an on-package I / O interconnect that facilitates communication between various processor components and a high-performance embedded memory module 1518, such as an eDRAM module. In some embodiments, each of the processor cores 1502A-1502N and the graphics processor 1508 use the embedded memory module 1518 as a shared last-level cache.

[0233] In some embodiments, the processor cores 1502A-1502N are homogeneous cores that execute the same instruction set architecture. In other embodiments, the processor cores 1502A-1502N are heterogeneous in terms of instruction set architecture (ISA), where one or more of the processor cores 1502A-1502N execute a first instruction set, while at least one of the other cores executes a subset of the first instruction set or a different instruction set. In one embodiment, the processor cores 1502A-1502N are heterogeneous in terms of microarchitecture, where one or more relatively high-power cores are coupled with one or more low-power power cores. In one embodiment, the processor cores 1502A-1502N are heterogeneous in terms of computing capabilities. Furthermore, the processor 1500 may be implemented on one or more chips, or as an SoC integrated circuit having the illustrated components in addition to other components.

[0234] Figure 15B is a block diagram of the hardware logic of a graphics processor core 1519 according to some embodiments described herein. Elements in Figure 15B having the same reference numeral (or name) as elements in any other drawings herein may, but are not limited to, operate or function in any manner similar to those described elsewhere herein. The graphics processor core 1519 may be referred to as a core slice and may be one or more graphics cores in a modular graphics processor. The graphics processor core 1519 exemplifies one graphics core slice, and the graphics processor may include multiple graphics core slices based on target power and performance limits, as described herein. Each graphics processor core 1519 may include a fixed-function block 1530 coupled with a number of subcores 1521A-1521F, also referred to as sub-slices, which include modular blocks of general-purpose and fixed-function logic.

[0235] In some embodiments, the fixed-function block 1530 includes a geometry / fixed-function pipeline 1531 which can be shared by all subcores within a graphics processor core 1519, for example, in a low-performance and / or low-power graphics processor implementation. In various embodiments, the geometry / fixed-function pipeline 1531 includes a 3D fixed-function pipeline (e.g., a 3D pipeline 1612 as shown in Figure 16A below), a video front-end unit, a thread spawner and thread dispatcher, and an integrated return buffer manager that manages an integrated return buffer, for example, an integrated return buffer 1718 as shown in Figure 17 below.

[0236] In one embodiment, the fixed-function block 1530 also includes a graphics SoC interface 1532, a graphics microcontroller 1533, and a media pipeline 1534. The graphics SoC interface 1532 provides an interface between the graphics processor core 1519 and other processor cores in the system on the chip integrated circuit. The graphics microcontroller 1533 is a programmable subprocessor configurable to manage various functions of the graphics processor core 1519, including thread dispatch, scheduling, and preemption. The media pipeline 1534 (e.g., media pipeline 1616 in Figures 16A and 17) includes logic to facilitate decoding, encoding, pre-processing, and / or post-processing of multimedia data, including image and video data. The media pipeline 1534 implements media operations via requests to compute or sampling logic within subcores 1521A-1521F.

[0237] In one embodiment, the SoC interface 1532 enables the graphics processor core 1519 to communicate with general-purpose application processor cores (e.g., CPU) and / or other components within the SoC, including memory hierarchy elements such as shared last-level cache memory, system RAM, and / or embedded on-chip or on-package DRAM. The SoC interface 1532 can also enable communication with fixed-function devices within the SoC, such as a camera imaging pipeline, and enable and / or implement the use of a global memory atomic that may be shared between the graphics processor core 1519 and the CPU within the SoC. The SoC interface 1532 can also implement power management controls for the graphics processor core 1519 and enable facilitating interfaces between the clock domain of the graphics core 1519 and other clock domains within the SoC. In one embodiment, the SoC interface 1532 enables the reception of command buffers from a command streamer and a global thread dispatcher configured to provide commands and instructions to each of the one or more graphics cores within the graphics processor. Commands and instructions can be dispatched to the media pipeline 1534 when media operations are performed, or to the geometry and fixed-function pipelines (e.g., geometry and fixed-function pipeline 1531, geometry and fixed-function pipeline 1537) when graphics processing operations are performed.

[0238] The graphics microcontroller 1533 may be configured to perform various scheduling and management tasks for the graphics processor core 1519. In one embodiment, the graphics microcontroller 1533 may perform graphics and / or compute workload scheduling on various graphics parallel engines in the execution unit (EU) arrays 1522A-1522F and 1524A-1524F within the subcores 1521A-1521F. In this scheduling model, host software running on the CPU core of the SoC, including the graphics processor core 1519, can present workloads to one of several graphics processor doorbells and invoke scheduling operations to the appropriate graphics engine. Scheduling operations include determining which workload to run next, presenting the workload to the command streamer, preempting existing workloads running on the engine, monitoring the progress of the workloads, and notifying the host software when the workloads are complete. In one embodiment, the graphics microcontroller 1533 may also facilitate low-power or idle states for the graphics processor core 1519, providing the graphics processor core 1519 with the ability to save and restore registers within the graphics processor core 1519 over low-power state transitions, independently of the operating system and / or graphics driver software on the system.

[0239] The graphics processor core 1519 may have up to N modular subcores, more or less than the subcores 1521A-1521F shown in the figure. For each set of N subcores, the graphics processor core 1519 may also include shared function logic 1535, shared and / or cache memory 1536, geometry / fixed function pipeline 1537, and further fixed function logic 1538 that accelerates various graphics and computes processing operations. The shared function logic 1535 may include logic units associated with the shared function logic 1720 in Figure 17 (e.g., sampler, mathematical, and / or inter-thread communication logic), which can be shared by each of the N subcores in the graphics processor core 1519. The shared and / or cache memory 1536 may be a last-level cache for the set of N subcores 1521A-1521F in the graphics processor core 1519 and may also function as shared memory accessible by multiple subcores. The geometry / fixed function pipeline 1537 may be included in place of the geometry / fixed function pipeline 1531 within the fixed function block 1530, and may contain the same or similar logic units.

[0240] In one embodiment, the graphics processor core 1519 includes a further fixed-function logic 1538 which may include various fixed-function acceleration logic for use by the graphics processor core 1519. In one embodiment, the further fixed-function logic 1538 includes a further geometry pipeline for use in position-only shading. In position-only shading, there are two geometry pipelines: a full geometry pipeline within geometry / fixed-function pipelines 1538 and 1531, and a culling pipeline which is a further geometry pipeline that may be included within the further fixed-function logic 1538. In one embodiment, the culling pipeline is a reduced version of the full geometry pipeline. The full pipeline and the culling pipeline can run different instances of the same application, each instance having a separate context. Position-only shading can hide long culling runs of discarded triangles and, in some cases, allow shading to complete more quickly. For example, and in one embodiment, the culling pipeline fetches and shades only the vertex position characteristics without performing pixel rasterization and rendering to the frame buffer, so that the culling pipeline logic within further fixed-function logic 1538 can run the position shader in parallel with the main application and generally produce important results faster than the full pipeline. The culling pipeline can use the important results produced to compute visibility information about all triangles, regardless of whether all triangles are culled or not. The full pipeline (which in this example may be called the replay pipeline) can skip the culled triangles and consume the visibility information to shade only the visible triangles that will ultimately be passed to the rasterization phase.

[0241] In one embodiment, the additional fixed-function logic 1538 may also include machine learning acceleration logic, such as fixed-function matrix multiplication logic, for implementations that include optimization of machine learning training or inference.

[0242] Each graphics subcore 1521A-1521F contains a set of execution resources that can be used to perform graphics, media, and computational operations in response to requests from the graphics pipeline, media pipeline, or shader program. The graphics subcore 1521A-1521F includes multiple EU arrays 1522A-1522F, 1524A-1524F, thread dispatch and inter-thread communication (TD / IC) logic 1523A-1523F, 3D (e.g., texture) sampler 1525A-1525F, media sampler 1526A-1526F, shader processor 1527A-1527F, and shared local memory (SLM) 1528A-1528F. Each EU array 1522A-1522F, 1524A-1524F contains multiple execution units, which are general-purpose graphics processing units capable of performing floating-point and integer / fixed-point logical operations while graphics, media, or compute operations, including graphics, media, or compute shader programs, are running. The TD / IC logic 1523A-1523F performs local thread dispatch and thread control operations for execution units within the subcore and facilitates communication between threads running on the subcore's execution units. The 3D sampler 1525A-1525F can read textures or other 3D graphics-related data into memory. The 3D sampler can read texture data separately based on the configured sample state and texture format associated with a given texture. The media sampler 1526A-1526F can perform similar read operations based on the type and format associated with media data. In one embodiment, each graphics subcore 1521A-1521F may alternately include an integrated 3D and media sampler. Threads running on the execution units within each subcore 1521A-1521F utilize the shared local memory 1528A-1528F within each subcore, enabling thread execution within a thread group that runs using a common pool of on-chip memory.

[0243] Figure 15C is a block diagram of a general-purpose graphics processing unit (GPGPU) 1570, which may be configured as a graphics processor and / or computing accelerator according to embodiments described herein. The GPGPU 1570 may be interconnected with a host processor (e.g., one or more CPUs 1546) and memories 1571, 1572 via one or more system and / or memory buses. In one embodiment, memory 1571 is system memory that may be shared with one or more CPUs 1546, while memory 1572 is device memory dedicated to the GPGPU 1570. In one embodiment, components in the GPGPU 1570 and device memory 1572 may be mapped to memory addresses accessible by one or more CPUs 1546. Access to memories 1571 and 1572 may be facilitated via a memory controller 1568. In one embodiment, the memory controller 1568 may include an internal direct memory access (DMA) controller 1569, or include logic that performs operations that would otherwise be performed by a DMA controller.

[0244] The GPGPU1570 includes multiple cache memories, including an L2 cache 1553, an L1 cache 1554, an instruction cache 1555, and shared memory 1556, at least a portion of which may also be partitioned as cache memory. The GPGPU1570 also includes multiple compute units 1560A-1560N. Each compute unit 1560A-1560N includes a set of vector registers 1561, scalar registers 1562, a vector logic unit 1563, and a scalar logic unit 1564. The compute units 1560A-1560N may also include local shared memory 1565 and a program counter 1566. The compute units 1560A-1560N may be coupled with a constant cache 1567, which may be used to store constant data, which is data that will not change during the execution of a kernel or shader program running on the GPGPU1570. In one embodiment, the constant cache 1567 is a scalar data cache, and the cached data can be directly fetched to the scalar register 1562.

[0245] During operation, one or more CPUs 1546 may write commands to registers or memory of the GPGPU 1570 mapped to an accessible address space. The command processor 1557 may read commands from registers or memory and determine how these commands will be processed by the GPGPU 1570. The thread dispatcher 1558 may then be used to dispatch threads to the compute units 1560A-1560N to execute these commands. Each compute unit 1560A-1560N may execute threads independently of the other compute units. Furthermore, each compute unit 1560A-1560N may be configured independently for conditional calculations and conditionally output the results of those calculations to memory. When an initiated command is completed, the command processor 1557 may interrupt one or more CPUs 1546.

[0246] Figures 16A to 16C show block diagrams of additional graphics processor and compute accelerator architectures provided by embodiments described herein. Elements in Figures 16A to 16C having the same reference numerals (or names) as elements in any other drawings herein may, but are not limited to, operate or function in any manner similar to those described elsewhere herein.

[0247] Figure 16A is a block diagram of a graphics processor 1600, which may be a discrete graphics processing unit, or a graphics processor integrated with multiple processing cores, or, but is not limited to, other semiconductor devices such as memory devices or network interfaces. In some embodiments, the graphics processor communicates using commands placed in processor memory via a memory-mapping I / O interface to registers on the graphics processor. In some embodiments, the graphics processor 1600 includes a memory interface 1614 for accessing memory. The memory interface 1614 may interface to local memory, one or more internal caches, one or more shared external caches, and / or to system memory.

[0248] In some embodiments, the graphics processor 1600 also includes a display controller 1602 that drives display output data to a display device 1618. The display controller 1602 includes hardware for one or more overlay planes relating to the display and configuration of multiple layers of video or user interface elements. The display device 1618 may be an internal or external display device. In one embodiment, the display device 1618 is a head-mounted display device, such as a virtual reality (VR) display device or an augmented reality (AR) display device. In some embodiments, the graphics processor 1600 includes a video codec engine 1606 for encoding, decoding, or transcoding media to, from, or between media encoding formats, including, but not limited to, Moving Picture Experts Group (MPEG) formats such as MPEG-2, Advanced Video Coating (AVC) formats such as H.264 / MPEG-4 AVC, H.265 / HEVC, Open Media Alliance (AOMedia) VP8, VP9, ​​and one or more media encoding formats, including Society of Motion Picture and Television Engineers (SMPTE) 421M / VC-1, Joint Photographic Experts Group (JPEG) formats such as JPEG, and Motion JPEG (MJPEG) format.

[0249] In some embodiments, the graphics processor 1600 includes a block image transfer (BLIT) engine 1604 that performs two-dimensional (2D) rasterizer operations, such as bit boundary block transfers. However, in one embodiment, 2D graphics operations are performed using one or more components of the graphics processing engine (GPE) 1610. In some embodiments, the GPE 1610 is a compute engine for performing graphics operations, including three-dimensional (3D) graphics operations and media operations.

[0250] In some embodiments, the GPE 1610 includes a 3D pipeline 1612 for performing 3D operations, such as rendering 3D images and scenes using processing functions that act on the shapes of 3D primitives (e.g., rectangles, triangles, etc.). The 3D pipeline 1612 includes programmable fixed functional elements that perform various tasks within the element and / or generate execution threads in the 3D / media subsystem 1615. While the 3D pipeline 1612 may be used to perform media operations, embodiments of the GPE 1610 also include a media pipeline 1616, specifically used to perform media operations such as video upload processing and image processing.

[0251] In some embodiments, the media pipeline 1616 includes a fixed function or programmable logic unit that performs one or more dedicated media operations, such as video decoding acceleration, video deinterlacing, and video encoding acceleration, on behalf of or on behalf of the video codec engine 1606. In some embodiments, the media pipeline 1616 further includes a thread spawning unit that spawns threads for execution on the 3D / media subsystem 1615. The spawned threads perform calculations related to media operations on one or more graphics execution units included in the 3D / media subsystem 1615.

[0252] In some embodiments, the 3D / media subsystem 1615 includes logic for executing threads spawned by the 3D pipeline 1612 and the media pipeline 1616. In one embodiment, these pipelines include thread dispatch logic for sending thread execution requests to the 3D / media subsystem 1615 and for arranging and dispatching various requests to available thread execution resources. The execution resources include an array of graphics execution units for handling 3D and media threads. In some embodiments, the 3D / media subsystem 1615 includes one or more internal caches for thread instructions and data. In some embodiments, the subsystem also includes shared memory, including registers and addressable memory for sharing data between threads and storing output data.

[0253] Figure 16B shows a graphics processor 1620 having a tiled architecture according to an embodiment described herein. In one embodiment, the graphics processor 1620 includes a graphics processing engine cluster 1622 having multiple instances of the graphics processing engine 1610 of Figure 16A within graphics engine tiles 1610A-1610D. Each graphics engine tile 1610A-1610D may be interconnected via a set of tile interconnects 1623A-1623F. Each graphics engine tile 1610A-1610D may also be connected to a memory module or memory device 1626A-1626D via a memory interconnect 1625A-1625D. The memory device 1626A-1626D may use any graphics memory technology. For example, the memory device 1626A-1626D may be graphics double data rate (GDDR) memory. In one embodiment, memory devices 1626A-1626D are high-bandwidth memory (HBM) modules that may be on-die with each of these graphics engine tiles 1610A-1610D. In one embodiment, memory devices 1626A-1626D are stacked memory devices that may be stacked on top of each of these graphics engine tiles 1610A-1610D. In one embodiment, each graphics engine tile 1610A-1610D and its associated memory 1626A-1626D reside on separate chiplets and are bonded to a base die or base substrate, as described in more detail in Figures 24B to 24D.

[0254] The graphics processing engine cluster 1622 may be connected to an on-chip or on-package fabric interconnect 1624. The fabric interconnect 1624 may enable communication between the graphics engine tiles 1610A-1610D and components such as the video codec 1606 and one or more copy engines 1604. The copy engine 1604 may be used to move data to and from memory devices 1626A-1626D and memory outside the graphics processor 1620 (e.g., system memory). The fabric interconnect 1624 may also be used to interconnect the graphics engine tiles 1610A-1610D. The graphics processor 1620 may optionally include a display controller 1602 to enable connection of the display device 1618 to an external display device 1618. The graphics processor may also be configured as a graphics or computing accelerator. In the accelerator configuration, the display controller 1602 and the display device 1618 may be omitted.

[0255] The graphics processor 1620 may be connected to a host system via a host interface 1628. The host interface 1628 may enable communication between the graphics processor 1620, system memory, and / or other system components. The host interface 1628 may be, for example, a PCI Express bus or another type of host system interface.

[0256] Figure 16C shows a compute accelerator 1630 according to an embodiment described herein. The compute accelerator 1630 may include architectural similarities with the graphics processor 1620 in Figure 16B and is optimized for compute acceleration. The compute engine cluster 1632 may include a set of compute engine tiles 1640A-1640D that include execution logic for optimizing parallel or vector-based general-purpose compute operations. In one embodiment, one or more compute engine tiles 1640A-1640D may include logic for performing media acceleration, but in some embodiments, the compute engine tiles 1640A-1640D do not include fixed functional graphics processing logic. The compute engine tiles 1640A-1640D may be connected to memory 1626A-1626D via memory interconnects 1625A-1625D. The memories 1626A-1626D and memory interconnects 1625A-1625D may be of similar or different technology as those in the graphics processor 1620. The graphics compute engine tiles 1640A-1640D may also be interconnected via a set of tile interconnects 1623A-1623F, connected to and / or interconnected with the fabric interconnect 1624. In one embodiment, the compute accelerator 1630 includes a large L3 cache 1636 which may be configured as a device-wide cache. The compute accelerator 1630 may also be connected to the host processor and memory via the host interface 1628 in a similar manner to the graphics processor 1620 in Figure 16B. Graphics Processing Engine

[0257] Figure 17 is a block diagram of a graphics processing engine 1710 of a graphics processor according to several embodiments. In one embodiment, the graphics processing engine (GPE) 1710 is a version of the GPE 1510 shown in Figure 15A, and may also represent the graphics engine tiles 1510A-1510D in Figure 15B. Elements of Figure 17 having the same reference number (or name) as elements of any other drawings herein may operate or function in any manner similar to those described elsewhere herein, but are not limited to such manners. For example, the 3D pipeline 1612 and media pipeline 1616 are illustrated in Figure 15A. In some embodiments of the GPE 1710, the media pipeline 1616 is optional and may not be explicitly included within the GPE 1710. For example, in at least one embodiment, a separate media and / or image processor is coupled to the GPE 1710.

[0258] In some embodiments, the GPE 1710 is coupled to or includes a command streamer 1703, which provides a command stream to the 3D pipeline 1612 and / or media pipeline 1616. In some embodiments, the command streamer 1703 is coupled to system memory, or memory which may be one or more of internal cache memory and shared cache memory. In some embodiments, the command streamer 1703 receives commands from memory and sends these commands to the 3D pipeline 1612 and / or media pipeline 1616. Commands are instructions fetched from a ring buffer, which stores commands for the 3D pipeline 1612 and media pipeline 1616. In one embodiment, the ring buffer may further include a batch command buffer that stores batches of multiple commands. Commands relating to the 3D pipeline 1612 may also include, but are not limited to, references to data stored in memory, such as vertex and geometry data relating to the 3D pipeline 1612 and / or image data and memory objects relating to the media pipeline 316. The 3D pipeline 1612 and the media pipeline 1616 process commands and data by performing multiple operations through logic within each pipeline, or by dispatching one or more execution threads to the graphics core array 1714. In one embodiment, the graphics core array 1714 includes one or more blocks of graphics cores (e.g., graphics core 1715A, graphics core 1715B), each block containing one or more graphics cores. Each graphics core includes a set of graphics execution resources, including graphics and computational operations, as well as general-purpose and graphics-specific execution logic that performs fixed-function texture processing, and / or machine learning and artificial intelligence acceleration logic.

[0259] In various embodiments, the 3D pipeline 1612 may include fixed function and programmable logic for processing one or more shader programs, such as vertex shaders, geometry shaders, pixel shaders, fragment shaders, compute shaders, or other shader programs, by processing instructions and dispatching execution threads to the graphics core array 1714. The graphics core array 1714 provides an integrated block of execution resources for use in the processing of these shader programs. The multi-purpose execution logic (e.g., execution units) within the graphics cores 1715A - 1714B of the graphics core array 1714 includes support for various 3D API shader languages and can execute multiple simultaneous execution threads associated with multiple shaders.

[0260] In some embodiments, the graphics core array 1714 also includes media functions, such as execution logic for performing video and / or image processing. In one embodiment, the execution units include general-purpose logic programmable to execute parallel general-purpose computing operations in addition to graphics processing operations. The general-purpose logic can execute processing operations in parallel with, or in conjunction with, the general-purpose logic within the processor core 1407 of FIG. 14 or cores 1502A - 1502N as in FIG. 15A.

[0261] Output data generated by threads executing on the graphics core array 1714 can output data to memory within the unified return buffer (URB) 1718. The URB 1718 can store data for multiple threads. In some embodiments, the URB 1718 may be used to transmit data between different threads executing on the graphics core array 1714. In some embodiments, the URB 1718 may be further used for synchronization between threads on the graphics core array and fixed function logic within the shared function logic 1720.

[0262] In some embodiments, the graphics core array 1714 is scalable, and thus the array includes a variable number of graphics cores, each having a variable number of execution units based on the target power and performance levels of the GPE 1710. In one embodiment, the execution resources are dynamically scalable, and thus the execution resources may be enabled or disabled as needed.

[0263] The graphics core array 1714 is coupled to shared function logic 1720 that includes a plurality of resources shared among the graphics cores within the graphics core array. The shared functions within the shared function logic 172 are hardware logic units that provide complementary functions dedicated to the graphics core array 1714. In various embodiments, the shared function logic 1720 includes, but is not limited to, sampler 1721, math 1722, and inter-thread communication (ITC) 1723 logic. Further, in some embodiments, one or more caches 1725 are implemented within the shared function logic 1720.

[0264] Shared functions are implemented at least when the demand for a given dedicated function is insufficient to include within the graphics core array 1714. Instead, a single instantiation of that dedicated function is implemented as a standalone entity of the shared function logic 1720 and shared among the execution resources within the graphics core array 1714. The exact set of functions shared among and included within the graphics core array 1714 varies across embodiments. In some embodiments, certain shared functions in the shared function logic 1720 that are more widely used by the graphics core array 1714 may be included in the shared function logic 1716 within the graphics core array 1714. In various embodiments, the shared function logic 1716 within the graphics core array 1714 may include some or all of the logic in the shared function logic 1720. In one embodiment, all logic elements within the shared function logic 1720 may overlap within the shared function logic 1716 of the graphics core array 1714. In one embodiment, shared function logic 1720 is excluded in favor of shared function logic 1716 within the graphics core array 1714. Execution Unit

[0265] Figures 18A to 18B show thread execution logic 1800, which includes an array of processing elements employed in a graphics processor core according to embodiments described herein. Elements in Figures 18A to 18B having the same reference numbers (or names) as elements in any other drawings herein may, but are not limited to, operate or function in any manner similar to those described elsewhere herein. Figures 18A to 18B provide an overview of thread execution logic 1800, which may be represented in the hardware logic illustrated in each subcore 221A-221F of Figure 2B. Figure 18A represents an execution unit in a general-purpose graphics processor, while Figure 18B represents an execution unit that may be used in a compute accelerator.

[0266] As illustrated in Figure 18A, in some embodiments, the thread execution logic 1800 includes a shader processor 1802, a thread dispatcher 1804, an instruction cache 1806, a scalable execution unit array including multiple execution units 1808A-1808N, a sampler 1810, shared local memory 1811, a data cache 1812, and a data port 1814. In one embodiment, the scalable execution unit array can be dynamically scaled by enabling or disabling one or more execution units (e.g., any of execution units 1808A, 1808B, 1808C, 1808D through 1808N-1 and 1808N) based on the computational demands of the workload. In one embodiment, the included components are interconnected via an interconnect fabric connected to each component. In some embodiments, the thread execution logic 1800 includes one or more connections to memory, such as system memory or cache memory, through one or more of the instruction cache 1806, data port 1814, sampler 1810, and execution units 1808A-1808N. In some embodiments, each execution unit (e.g., 1808A) is a standalone programmable general-purpose computing unit capable of executing multiple concurrent hardware threads while processing multiple data elements in parallel for each thread. In various embodiments, the array of execution units 1808A-1808N is scalable to include any number of individual execution units.

[0267] In some embodiments, execution units 1808A-1808N are primarily used to execute shader programs. The shader processor 1802 can process various shader programs and dispatch execution threads associated with those shader programs via the thread dispatcher 1804. In one embodiment, the thread dispatcher includes logic that coordinates multiple thread start requests from the graphics and media pipelines and instantiates the requested threads on one or more execution units of the execution units 1808A-1808N. For example, the geometry pipeline can dispatch vertex, tessellation, or geometry shaders to thread execution logic for processing. In some embodiments, the thread dispatcher 1804 can also process runtime thread spawning requests from the running shader program.

[0268] In some embodiments, the execution units 1808A-1808N support an instruction set that includes native support for many standard 3D graphics shader instructions, thus allowing shader programs from graphics libraries (e.g., Direct3D and OpenGL) to run with minimal migration. The execution units support vertex and geometry processing (e.g., vertex programs, geometry programs, vertex shaders), pixel processing (e.g., pixel shaders, fragment shaders), and general-purpose processing (e.g., compute and media shaders). Each of the execution units 1808A-1808N enables multi-issue single-instruction multiple-data (SIMD) execution, and multi-threaded operation enables an efficient execution environment despite higher latency memory access. Each hardware thread within each execution unit has its own dedicated high-bandwidth register file and associated independent thread state. Execution is multi-issue per clock to a pipeline capable of integer, single-precision and double-precision floating-point arithmetic, SIMD branch performance, logical operations, transcendental operations, and other aggregate operations. While waiting for data from memory or from one of the shared functions, dependency logic within execution units 1808A-1808N puts the waiting thread to sleep until the requested data returns. While the waiting thread is sleeping, hardware resources may be dedicated to processing other threads. For example, during the delay associated with a vertex shader operation, the execution unit can perform operations on a pixel shader, a fragment shader, or another type of shader program containing different vertex shaders. Various embodiments may be adapted to use execution using single-instruction multiple-threaded (SIMT) as an alternative to or in addition to the use of SIMD. References to or behaviors of SIMD cores may also apply to SIMT, or to SIMD in combination with SIMT.

[0269] Each execution unit 1808A-1808N operates on an array of data elements. The number of data elements is the "execution size" or the number of channels for an instruction. An execution channel is a logical unit for execution for data element access, masking, and flow control within an instruction. The number of channels may be independent of the number of physical arithmetic logic units (ALUs) or floating-point units (FPUs) for a particular graphics processor. In some embodiments, execution units 1808A-1808N support integer and floating-point data types.

[0270] The execution unit instruction set includes SIMD instructions. Various data elements can be stored as packed data types in registers, and the execution unit will process various elements based on their data size. For example, when manipulating a 256-bit wide vector, 256 bits of the vector are stored in registers, and the execution unit will manipulate the vector as four distinct 64-bit packed data elements (quad-word (QW) sized data elements), eight distinct 32-bit packed data elements (double-word (DW) sized data elements), sixteen distinct 16-bit packed data elements (word (W) sized data elements), or thirty-two distinct 8-bit data elements (byte (B) sized data elements). However, different vector widths and register sizes are possible.

[0271] In one embodiment, one or more execution units may be combined with fused execution units 1809A-1809N having thread control logic (1807A-1807N) common to the fused EUs. Multiple EUs may be fused into an EU group. Each EU within a fused EU group may be configured to execute a separate SIMD hardware thread. The number of EUs in a fused EU group may vary depending on the embodiment. Furthermore, various SIMD widths may be executed per EU, and are not limited to, but include SIMD8, SIMD16, and SIMD32. Each fused graphics execution unit 1809A-1809N includes at least two execution units. For example, fused execution unit 1809A includes a first EU 1808A, a second EU 1808B, and thread control logic 1807A common to the first EU 1808A and the second EU 1808B. The thread control logic 1807A controls threads running on the fused graphics execution unit 1809A, enabling each EU within the fused execution units 1809A-1809N to execute using a common instruction pointer register.

[0272] One or more internal instruction caches (e.g., 1806) are included in the thread execution logic 1800 for caching thread instructions for the execution unit. In some embodiments, one or more data caches (e.g., 1812) are included for caching thread data during thread execution. Thread execution on the execution logic 1800 may also store data explicitly managed in shared local memory 1811. In some embodiments, a sampler 1810 is included to provide texture sampling for 3D calculations and media sampling for media calculations. In some embodiments, the sampler 1810 includes dedicated texture or media sampling functions for processing texture or media data during the sampling process before providing the sampled data to the execution unit.

[0273] During execution, the graphics and media pipelines send thread start requests to the thread execution logic 1800 via thread spawning and dispatch logic. Once a group of geometric objects has been processed and rasterized into pixel data, the pixel processor logic (e.g., pixel shader logic, fragment shader logic, etc.) within the shader processor 1802 is invoked to further compute output information and write the results to output surfaces (e.g., color buffer, depth buffer, stencil buffer, etc.). In some embodiments, the pixel shader or fragment shader computes values ​​for various vertex properties to be interpolated across the rasterized object. In some embodiments, the pixel processor logic within the shader processor 1802 then executes an application programming interface (API) supplied pixel or fragment shader program. To execute the shader program, the shader processor 1802 dispatches multiple threads to execution units (e.g., 1808A) via the thread dispatcher 1804. In some embodiments, the shader processor 1802 uses texture sampling logic in the sampler 1810 to access texture data in multiple texture maps stored in memory. Arithmetic operations on the texture data and input geometry data calculate pixel color data for each geometric fragment, or discard one or more pixels from further processing.

[0274] In some embodiments, the data port 1814 provides the thread execution logic 1800 with a memory access mechanism for outputting processed data to memory for further processing on the graphics processor's output pipeline. In some embodiments, the data port 1814 includes, or is coupled to, one or more cache memories (e.g., data cache 1812) for caching data for memory access through the data port.

[0275] In one embodiment, the execution logic 1800 may also include a ray tracer 1805 that can provide ray tracing acceleration functionality. The ray tracer 1805 may support a ray tracing instruction set that includes instructions / functions related to ray generation. The ray tracing instruction set may be the same as, or different from, the ray tracing instruction set supported by the ray tracing core 245 in Figure 2C.

[0276] Figure 18B shows exemplary internal details of an execution unit 1808 according to an embodiment. The graphics execution unit 1808 may include an instruction fetch unit 1837, a general-purpose register file array (GRF) 1824, an architecture register file array (ARF) 1826, a thread arbiter 1822, a transmit unit 1830, a branch unit 1832, a set of SIMD floating-point units (FPUs) 1834, and, in one embodiment, a set of dedicated integer SIMD ALUs 1835. The GRF 1824 and ARF 1826 include sets of general-purpose register files and architecture register files associated with each concurrent hardware thread that may be active in the graphics execution unit 1808. In one embodiment, for each thread, the architecture state is maintained in the ARF 1826, while data used during thread execution is stored in the GRF 1824. The execution state of each thread, including the instruction pointer for each thread, may be held in thread-specific registers in the ARF 1826.

[0277] In one embodiment, the graphics execution unit 1808 has an architecture that combines simultaneous multithreading (SMT) and fine-grained interleaved multithreading (IMT). The architecture has a modular configuration that can be finely tuned at design time based on the target number of concurrent threads and the number of registers per execution unit, and the resources of the execution unit are divided across the logic used to execute multiple concurrent threads. The number of logical threads that can be executed by the graphics execution unit 1808 is not limited to the number of hardware threads, and multiple logical threads can be assigned to each hardware thread.

[0278] In one embodiment, the graphics execution unit 1808 can jointly issue multiple instructions, each of which may be a different instruction. The thread arbiter 1822 of the graphics execution unit thread 1808 can dispatch instructions to one of the transmit unit 1830, branch unit 1832, or SIMD FPU 1834 for execution. Each execution thread can access 128 general-purpose registers in the GRF1824, each register capable of storing 32 bytes accessible as a SIMD8-element vector of 32-bit data elements. In one embodiment, each execution unit thread has access to 4K bytes in the GRF1824, but embodiments are not limited thereto, and other embodiments may provide more or fewer register resources. In one embodiment, the graphics execution unit 1808 is partitioned into seven hardware threads that can independently perform computational operations, but the number of threads per execution unit can also vary depending on the embodiment. For example, in one embodiment, up to 16 hardware threads are supported. In an embodiment where seven threads can access 4KB, the GRF1824 can store a total of 28KB. If sixteen threads can access 4KB, the GRF1824 can store a total of 64KB. The flexible addressing mode can effectively construct wider registers, or allow registers to be addressed together to represent stride rectangular block data structures.

[0279] In one embodiment, memory operations, sampler operations, and other longer-latency system communications are dispatched via "transmit" instructions executed by a message-passing transmission unit 1830. In one embodiment, branch instructions are dispatched to a dedicated branching unit 1832 to facilitate SIMD divergence and final convergence.

[0280] In one embodiment, the graphics execution unit 1808 includes one or more SIMD floating-point units (FPUs) 1834 that perform floating-point operations. In one embodiment, the FPU 1834 also supports integer operations. In one embodiment, the FPU 1834 can perform SIMD operations on up to M numbers of 32-bit floating-point (or integer) operations, or on up to 2M numbers of 16-bit integer or 16-bit floating-point operations. In one embodiment, at least one of the FPUs provides extended mathematical capabilities that support high-throughput, superior mathematical functions and double-precision 64-bit floating-point. In some embodiments, a set of 8-bit integer SIMD ALUs 1835 also exist and may be specifically optimized to perform operations associated with machine learning computations.

[0281] In one embodiment, an array of multiple instances of the graphics execution unit 1808 may be instantiated in a graphics subcore group (e.g., a subslice). For scalability, the product designer can select the exact number of execution units for each subcore group. In one embodiment, the execution unit 1808 can execute instructions across multiple execution channels. In a further embodiment, each thread running on the graphics execution unit 1808 runs on a different channel.

[0282] Figure 19 shows an additional execution unit 1900 according to an embodiment. The execution unit 1900 may, but is not limited to, an execution unit optimized for use in computation engine tiles 1540A-1540D, for example, as shown in Figure 15C. Modifications of the execution unit 1900 may also be used in graphics engine tiles 1510A-1510D, as shown in Figure 15B. In one embodiment, the execution unit 1900 includes a thread control unit 1901, a thread state unit 1902, an instruction fetch / prefetch unit 1903, and an instruction decode unit 1904. The execution unit 1900 further includes a register file 1906 that stores registers that can be allocated to hardware threads within the execution unit. The execution unit 1900 further includes a transmit unit 1907 and a branch unit 1908. In one embodiment, the transmission unit 1907 and the branching unit 1908 may operate similarly to the transmission unit 1830 and branching unit 1832 of the graphics execution unit 1808 in Figure 18B.

[0283] The execution unit 1900 also includes a compute unit 1910 which includes several different types of functional units. In one embodiment, the compute unit 1910 includes an ALU unit 1911 which includes an array of arithmetic logic units. The ALU unit 1911 may be configured to perform 64-bit, 32-bit, and 16-bit integer arithmetic and floating-point arithmetic. Integer and floating-point arithmetic may be performed simultaneously. The compute unit 1910 may also include a systolic array 1912 and a mathematical unit 1913. The systolic array 1912 includes a network of data processing units of width W and depth D which can be used to perform vector or other data parallel operations in a systolic manner. In one embodiment, the systolic array 1912 may be configured to perform matrix operations, such as matrix dot multiplication operations. In one embodiment, the systolic array 1912 supports 16-bit floating-point arithmetic, as well as 8-bit and 4-bit integer arithmetic. In one embodiment, the systolic array 1912 may be configured to accelerate machine learning operations. In such an embodiment, the systolic array 1912 may be configured to support the bfloat 16-bit floating-point format. In one embodiment, a mathematical unit 1913 may be included to perform a specific subset of mathematical operations in a more efficient and low-power manner than the ALU unit 1911. The mathematical unit 1913 may include variations of mathematical logic that may be found in the shared function logic of the graphics processing engine provided by other embodiments (e.g., mathematical logic 1722 in the shared function logic 1720 in Figure 17). In one embodiment, the mathematical unit 1913 may be configured to perform 32-bit and 64-bit floating-point operations.

[0284] The thread control unit 1901 includes logic for controlling the execution of threads within the execution unit. The thread control unit 1901 may include thread scheduling logic for starting, stopping, and preempting the execution of threads within the execution unit 1900. The thread state unit 1902 may be used to store the thread state for threads assigned to execute on the execution unit 1900. Storing the thread state within the execution unit 1900 enables rapid preemption of these threads when the threads are blocked or become idle. The instruction fetch / prefetch unit 1903 may fetch instructions from the instruction cache of a higher-level execution logic (e.g., instruction cache 1806 as shown in FIG. 18A). The instruction fetch / prefetch unit 1903 may also issue a prefetch request for instructions to be loaded into the instruction cache based on the analysis of the currently executing thread. The instruction decode unit 1904 may be used to decode the instructions to be executed by the computing unit. In one embodiment, the instruction decode unit 1904 may be used as a secondary decoder for decoding complex instructions into component micro-operations.

[0285] The execution unit 1900 further includes a register file 1906 that may be used by the hardware threads executing on the execution unit 1900. The registers in the register file 1906 may be partitioned across the logic used to execute multiple concurrent threads within the computing unit 1910 of the execution unit 1900. The number of logical threads that may be executed by the graphics execution unit 1900 is not limited to the number of hardware threads, and multiple logical threads may be assigned to each hardware thread. The size of the register file 1906 may vary across embodiments based on the number of supported hardware threads. In one embodiment, register renaming may be used to dynamically assign registers to hardware threads.

[0286] Figure 20 is a block diagram showing graphics processor instruction formats 2000 in several embodiments. In one or more embodiments, the graphics processor execution unit supports an instruction set having instructions in multiple formats. Solid lines indicate components commonly included in execution unit instructions, while dashed lines indicate components that are optional or included only in a subset of instructions. In some embodiments, the instruction formats 2000 described and illustrated are macro instructions in the sense that they are instructions supplied to the execution unit, as opposed to microoperations by instruction decoding once the instructions have been processed.

[0287] In some embodiments, the graphics processor execution unit natively supports multiple instructions in the 128-bit instruction format 2010. The 64-bit compressed instruction format 2030 offers several instructions based on the number of selected instructions, instruction options, and operands. The native 128-bit instruction format 2010 provides access to all instruction options, while some options and operations are restricted to the 64-bit format 2030. The native instructions available in the 64-bit format 2030 vary by embodiment. In some embodiments, instructions are partially compressed using a set of multiple index values ​​in the index field 2013. The execution unit hardware references a set of compression tables based on the index values ​​and uses the compression table output to reconstruct native instructions in the 128-bit instruction format 2010. Instructions of other sizes and formats may be used.

[0288] For each format, instruction opcode 2012 defines the operations performed by the execution unit. The execution unit executes each instruction in parallel across multiple data elements of each operand. For example, depending on additional instructions, the execution unit simultaneously performs additional operations across each color channel representing texture or picture components. By default, the execution unit executes each instruction across all data channels of the operand. In some embodiments, instruction control field 2014 allows control over specific execution options such as channel selection (e.g., predication) and data channel ordering (e.g., swizzle). For instructions in the 128-bit instruction format 2010, the execution size field 2016 will limit the number of data channels executed in parallel. In some embodiments, the execution size field 2016 is not available for use in the 64-bit compact instruction format 2030.

[0289] Some execution unit instructions have up to three operands, including two source operands src0 2020, src1 2022, and one destination 2018. In some embodiments, the execution unit supports dual-destination instructions, where one of the destinations is implicitly indicated. Data manipulation instructions may have a third source operand (e.g., SRC2 2024), and the instruction opcode 2012 determines the number of source operands. The last source operand of an instruction may be an immediate value passed with the instruction (e.g., hardcoded).

[0290] In some embodiments, the 128-bit instruction format 2010 includes an access / address mode field 2026 that specifies, for example, whether direct register addressing mode or indirect register addressing mode is used. When direct register addressing mode is used, the register addresses of one or more operands are provided directly by bits in the instruction.

[0291] In some embodiments, the 128-bit instruction format 2010 includes an access / address mode field 2026 that defines the address mode and / or access mode of the instruction. In one embodiment, the access mode is used to define the data access alignment of the instruction. In some embodiments, access modes are supported, including 16-byte aligned access modes and 1-byte aligned access modes, where the byte alignment of the access mode determines the access alignment of the instruction operands. For example, in the first mode, the instruction may use byte-aligned addressing for the source and destination operands, and in the second mode, the instruction may use 16-byte aligned addressing for all source and destination operands.

[0292] In one embodiment, a portion of the address mode in the access / address mode field 2026 determines whether the instruction uses direct or indirect addressing. When direct register addressing mode is used, bits in the instruction directly provide the register addresses of one or more operands. When indirect register addressing mode is used, the register addresses of one or more operands may be calculated based on the values ​​of the address registers and the address immediate field in the instruction.

[0293] In some embodiments, multiple instructions are grouped based on the opcode 2012 bit field to simplify opcode decoding 2040. For 8-bit opcodes, bits 4, 5, and 6 allow the execution unit to determine the type of opcode. The exact opcode groupings shown are examples only. In some embodiments, the move and logic opcode group 2042 includes data move and logic instructions (e.g., move (mov), compare (cmp)). In some embodiments, the move and logic group 2042 shares five most significant bits (MSB), with move (mov) instructions in the form of 0000xxxxb and logic instructions in the form of 0001xxxxb. The flow control instruction group 2044 (e.g., call, jump (jmp)) includes instructions in the form of 0010xxxxb (e.g., 0x20). The mixed instruction group 2046 includes a mixture of instructions, including synchronous instructions of the form 0011xxxxb (e.g., 0x30) (e.g., wait, send). The parallel math instruction group 2048 includes arithmetic instructions on components of the form 0100xxxxb (e.g., 0x40) (e.g., add, multiply (mul)). The parallel math group 2048 performs arithmetic operations in parallel on the data channel. The vector math group 2050 includes arithmetic instructions of the form 0101xxxxb (e.g., 0x50) (e.g., dp4). The vector math group performs arithmetic operations such as dot product calculations on vector operands. The illustrated opcode decode 2040 may, in one embodiment, be used to determine which parts of the execution unit will be used to execute the decoded instructions. For example, several instructions may be designated as systolic instructions that will be executed by the systolic array. Other instructions, such as ray tracing instructions (not shown), may be forwarded to the ray tracing core or ray tracing logic within a slice or partition of the execution logic. Graphics pipeline

[0294] Figure 21 is a block diagram of a graphics processor 2100 according to another embodiment. Elements of Figure 21 having the same reference numeral (or name) as elements of any other drawings herein may, but are not limited to, operate or function in any manner similar to those described elsewhere herein.

[0295] In some embodiments, the graphics processor 2100 includes a geometry pipeline 2120, a media pipeline 2130, a display engine 2140, thread execution logic 2150, and a rendering output pipeline 2170. In some embodiments, the graphics processor 2100 is a graphics processor within a multicore processing system that includes one or more general-purpose processing cores. The graphics processor is controlled by register writes to one or more control registers (not shown) or by commands issued to the graphics processor 2100 via a ring interconnect 2102. In some embodiments, the ring interconnect 2102 connects the graphics processor 2100 to other processing components, such as other graphics processors or general-purpose processors. Commands from the ring interconnect 2102 are interpreted by a command streamer 2103, which supplies instructions to individual components of the geometry pipeline 2120 or the media pipeline 2130.

[0296] In some embodiments, the command streamer 2103 reads vertex data from memory and instructs the vertex fetcher 2105 to operate a component that executes a plurality of vertex processing commands provided by the command streamer 2103. In some embodiments, the vertex fetcher 2105 provides vertex data to the vertex shader 2107, which performs coordinate space transformations and lighting operations for each vertex. In some embodiments, the vertex fetcher 2105 and the vertex shader 2107 execute vertex processing instructions by dispatching execution threads to execution units 2152A-2152B via the thread dispatcher 2131.

[0297] In some embodiments, the execution units 2152A-2152B are arrays of vector processors having instruction sets for performing graphics and media operations. In some embodiments, the execution units 2152A-2152B have an attached L1 cache 2151 which is specific to each array or shared between arrays. The cache may be configured as a data cache, an instruction cache, or a single cache partitioned to contain data and instructions in different partitionings.

[0298] In some embodiments, the geometry pipeline 2120 includes a tessellation component that performs hardware-accelerated tessellation of 3D objects. In some embodiments, a programmable hull shader 2111 constitutes the tessellation operation. A programmable domain shader 2117 provides backend evaluation of the tessellation output. A tessellator 2113 includes dedicated logic that operates at the direction of the hull shader 2111 and generates a set of detailed geometric objects based on a coarse geometric model provided to the geometry pipeline 2120 as input. In some embodiments, if tessellation is not used, the tessellation component (e.g., hull shader 2111, tessellator 2113, and domain shader 2117) can be bypassed.

[0299] In some embodiments, a complete set of geometric objects may be processed by the geometry shader 2119 through one or more threads dispatched to execution units 2152A, 2152B, or proceed directly to the clipper 2129. In some embodiments, the geometry shader operates on the entire geometric object rather than on vertices or patches of vertices, as is the case in earlier stages of the graphics pipeline. When tessellation is disabled, the geometry shader 2119 receives input from the vertex shader 2107. In some embodiments, the geometry shader 2119 is programmable by the geometry shader program to perform geometry tessellation when the tessellation unit is disabled.

[0300] Before rasterization, the clipper 2129 processes the vertex data. The clipper 2129 may be a fixed-function clipper or a programmable clipper having clipping and geometry shader capabilities. In some embodiments, the rasterizer and depth test component 2173 in the rendering output pipeline 2170 dispatches a pixel shader to convert the geometric object into a pixel-by-pixel representation. In some embodiments, the pixel shader logic is contained within the thread execution logic 2150. In some embodiments, the application may bypass the rasterizer and depth test component 2173 and access the unrasterized vertex data via the stream-out unit 2123.

[0301] The graphics processor 2100 has an interconnection bus, an interconnection fabric, or several other interconnection mechanisms that enable data and message passing between the processor's main components. In some embodiments, the execution units 2152A-2152B and associated logic units (e.g., L1 cache 2151, sampler 2154, texture cache 2158, etc.) interconnect via data port 2156 to perform memory access and communicate with the processor's rendering output pipeline components. In some embodiments, the sampler 2154, caches 2151 and 2158, and execution units 2152A-2152B each have separate memory access paths. In one embodiment, the texture cache 2158 may also be configured as a sampler cache.

[0302] In some embodiments, the rendering output pipeline 2170 includes a rasterizer and depth test component 2173 that converts vertex-based objects to their associated pixel-based representations. In some embodiments, the rasterizer logic includes a windower / masker unit that performs fixed-function triangle and line rasterization. In some embodiments, associated rendering caches 2178 and depth caches 2179 are also available. The pixel calculation component 2177 performs pixel-based calculations on the data, although in some examples, pixel calculations associated with 2D calculations (e.g., bitblock image transfer with blending) are performed by the 2D engine 2141 or replaced by the display controller 2143 using an overlay display plane at display time. In some embodiments, a shared L3 cache 2175 is available to all graphics components, enabling data sharing without using main system memory.

[0303] In some embodiments, the graphics processor's media pipeline 2130 includes a media engine 2137 and a video frontend 2134. In some embodiments, the video frontend 2134 receives pipeline commands from a command streamer 2103. In some embodiments, the media pipeline 2130 includes a separate command streamer. In some embodiments, the video frontend 2134 processes multiple media commands before sending them to the media engine 2137. In some embodiments, the media engine 2137 includes a thread spawning function that spawns multiple threads for dispatch to thread execution logic 2150 via a thread dispatcher 2131.

[0304] In some embodiments, the graphics processor 2100 includes a display engine 2140. In some embodiments, the display engine 2140 is external to the processor 2100 and coupled to the graphics processor via a ring interconnect 2102 or some other interconnect bus or fabric. In some embodiments, the display engine 2140 includes a 2D engine 2141 and a display controller 2143. In some embodiments, the display engine 2140 includes dedicated logic that can operate independently of the 3D pipeline. In some embodiments, the display controller 2143 is coupled to a display device (not shown), which may be a system-integrated display device, such as one found in a laptop computer, or an external display device attached via a display device connector.

[0305] In some embodiments, the geometry pipeline 2120 and media pipeline 2130 can be configured to perform operations based on multiple graphics and media programming interfaces and are not specific to any one application programming interface (API). In some embodiments, driver software for the graphics processor translates API calls specific to a particular graphics or media library into commands that can be processed by the graphics processor. In some embodiments, support is provided for all of the Khronos Group's open graphics library (OpenGL), open computing language (OpenCL), and / or Vulkan graphics and compute APIs. In some embodiments, support may also be provided for Microsoft Corporation's Direct3D library. In some embodiments, combinations of these libraries may be supported. Support may also be provided for the open-source computer vision library (OpenCV). Future APIs with compatible 3D pipelines will also be supported if mapping from the future API's pipeline to the graphics processor's pipeline can be performed. Graphics pipeline programming

[0306] Figure 22A is a block diagram showing graphics processor command format 2200 according to several embodiments. Figure 22B is a block diagram showing graphics processor command sequence 2210 according to an embodiment. In Figure 22A, solid lines indicate components commonly included in graphics commands, while dashed lines indicate components that are optional or included only in a subset of graphics commands. The exemplary graphics processor command format 2200 in Figure 22A includes data fields that identify the client 2202, command operation code (opcode) 2204, and data 2206 for the command. Sub-opcodes 2205 and command size 2208 are also included in some commands.

[0307] In some embodiments, client 2202 defines a client unit of a graphics device that processes command data. In some embodiments, a graphics processor command parser examines the client field of each command to coordinate further processing of the command and route the command data to the appropriate client unit. In some embodiments, the graphics processor client unit includes a memory interface unit, a rendering unit, a 2D unit, a 3D unit, and a media unit. Each client unit has a corresponding processing pipeline for processing commands. Once a command is received by a client unit, the client unit reads the opcode 2204 and, if present, the sub-opcode 2205 to determine the operation to perform. The client unit executes the command using the information in the data field 2206. For some commands, an explicit command size 2208 is expected to specify the size of the command. In some embodiments, the command parser automatically determines at least some size of the command based on the command opcode. In some embodiments, commands are aligned using multiples of double words. Other command formats may also be used.

[0308] The flowchart in Figure 22B shows an exemplary graphics processor command sequence 2210. In some embodiments, the software or firmware of the data processing system characterizing the graphics processor embodiment uses the shown version of the command sequence to set up, execute, and terminate a set of graphics operations. Embodiments are shown and described for illustrative purposes only, but are not limited to these specific commands or command sequences. Furthermore, the commands may be issued as a batch of commands in the command sequence, so that the graphics processor will process the sequence of commands in at least partially coherent terms.

[0309] In some embodiments, the graphics processor command sequence 2210 may begin with a pipeline flush command 2212 that causes any active graphics pipeline to complete any currently pending commands to the pipeline. In some embodiments, the 3D pipeline 2222 and the media pipeline 2224 do not operate simultaneously. The pipeline flush is performed so that the active graphics pipeline completes any pending commands. In response to the pipeline flush, the command parser to the graphics processor will suspend command processing until the active drawing engine completes any pending operations and the associated read cache is invalidated. Optionally, any data in the rendering cache marked as "dirty" may be flushed to memory. In some embodiments, the pipeline flush command 2212 may be used during pipeline synchronization or before putting the graphics processor into a low-power state.

[0310] In some embodiments, the pipeline selection command 2213 is used when the command sequence requires the graphics processor to explicitly switch between pipelines. In some embodiments, the pipeline selection command 2213 is required only once within the execution context before issuing pipeline commands unless the context issues commands to both pipelines. In some embodiments, the pipeline flush command 2212 is required immediately before the pipeline switch via the pipeline selection command 2213.

[0311] In some embodiments, pipeline control commands 2214 are used to configure the graphics pipeline for operation and to program the 3D pipeline 2222 and the media pipeline 2224. In some embodiments, pipeline control commands 2214 configure the pipeline state for the active pipeline. In one embodiment, pipeline control commands 2214 are used during pipeline synchronization or to clear data from one or more cache memories in the active pipeline before processing a batch of commands.

[0312] In some embodiments, the return buffer state command 2216 is used to configure a set of return buffers for each pipeline to write data. Some pipeline operations require the allocation, selection, or configuration of one or more return buffers to which intermediate data is written during processing. In some embodiments, the graphics processor also uses one or more return buffers to store output data and to perform cross-thread communication. In some embodiments, the return buffer state 2216 includes selecting the size and number of return buffers to use with a set of pipeline operations.

[0313] The remaining commands in the command sequence differ based on the active pipeline for operation. Based on pipeline determination 2220, the command sequence is adjusted to either 3D pipeline 2222, which starts in 3D pipeline state 2230, or media pipeline 2224, which starts in media pipeline state 2240.

[0314] The commands that constitute the 3D pipeline state 2230 include 3D state setting commands for vertex buffer state, vertex element state, constant color state, depth buffer state, and other state variables configured before the 3D primitive command is processed. The values ​​of these commands are determined at least partially based on the specific 3D API being used. In some embodiments, the commands for the 3D pipeline state 2230 may also selectively disable or bypass specific pipeline elements if those elements would not be used.

[0315] In some embodiments, the 3D primitive 2232 command is used to present a 3D primitive to be processed by the 3D pipeline. The command and associated parameters passed to the graphics processor via the 3D primitive 2232 command are forwarded to a vertex fetch function in the graphics pipeline. The vertex fetch function generates a vertex data structure using the 3D primitive 2232 command data. The vertex data structure is stored in one or more return buffers. In some embodiments, the 3D primitive 2232 command is used to perform vertex operations on the 3D primitive via a vertex shader. To process the vertex shader, the 3D pipeline 2222 dispatches a shader execution thread to the graphics processor execution unit.

[0316] In some embodiments, the 3D pipeline 2222 is triggered via a command or event in the execution 2234. In some embodiments, a register write triggers a command execution. In some embodiments, execution is triggered via a "go" or "kick" command in a command sequence. In one embodiment, command execution is triggered using a pipeline synchronization command to flush the command sequence through the graphics pipeline. The 3D pipeline will perform geometry processing on 3D primitives. Once the operation is complete, the resulting geometric object is rasterized, and the pixel engine colors the resulting pixels. Additional commands to control pixel shading and pixel backend operation may be included for these operations.

[0317] In some embodiments, the graphics processor command sequence 2210 follows a path through the media pipeline 2224 when performing media operations. Generally, specific uses and schemes for programming the media pipeline 2224 depend on the media or computational operation being performed. Certain media decoding operations may be offloaded to the media pipeline during media decoding. In some embodiments, the media pipeline may also be bypassed, and media decoding may be performed using resources provided by one or more general-purpose processing cores, either entirely or partially. In one embodiment, the media pipeline also includes elements for general-purpose graphics processor unit (GPGPU) operations, where the graphics processor is used to perform SIMD vector operations using computational shader programs not explicitly related to rendering graphics primitives.

[0318] In some embodiments, the media pipeline 2224 is configured in a manner similar to the 3D pipeline 2222. A set of commands constituting the media pipeline state 2240 is dispatched to or placed in a command queue before the media object commands 2242. In some embodiments, the commands for the media pipeline state 2240 will include data that constitutes the media pipeline elements used to process media objects. This includes data that constitutes the video decoding and video encoding logic, such as the encoding or decoding format within the media pipeline. In some embodiments, the commands for the media pipeline state 2240 also support the use of one or more pointers to “indirect” state elements that include a batch of state settings.

[0319] In some embodiments, the media object command 2242 provides a pointer to a media object for processing by the media pipeline. The media object includes a memory buffer containing the video data to be processed. In some embodiments, all media pipeline states must be valid before issuing the media object command 2242. Once the pipeline states are configured and the media object command 2242 is queued, the media pipeline 2224 is triggered via an execution command 2244 or an equivalent execution event (e.g., register write). The output from the media pipeline 2224 may then be post-processed by the 3D pipeline 2222 or by the operations provided by the media pipeline 2224. In some embodiments, GPGPU operations are configured and executed in a similar manner to media operations. Graphics Software Architecture

[0320] Figure 23 shows exemplary graphics software architectures for a data processing system 2300 according to several embodiments. In some embodiments, the software architecture includes a 3D graphics application 2310, an operating system 2320, and at least one processor 2330. In some embodiments, the processor 2330 includes a graphics processor 2332 and one or more general-purpose processor cores 2334. The graphics application 2310 and the operating system 2320 each run in the system memory 2350 of the data processing system.

[0321] In some embodiments, the 3D graphics application 2310 includes one or more shader programs, including shader instructions 2312. The shader language instructions may be in a high-level shader language, such as Direct3D High-Level Shader Language (HLSL) or OpenGL Shader Language (GLSL). The application also includes executable machine code instructions 2314 suitable for execution by a general-purpose processor core 2334. The application also includes graphics objects 2316 defined by vertex data.

[0322] In some embodiments, the operating system 2320 is a Microsoft Corporation Microsoft® Windows® operating system, a proprietary UNIX®-style operating system, or an open-source UNIX®-style operating system using a modification of the Linux® kernel. The operating system 2320 can support graphics APIs 2322 such as the Direct3D API, OpenGL API, or Vulkan API. When the Direct3D API is used, the operating system 2320 uses a front-end shader compiler 2324 to compile any shader instructions 2312 in HLSL into a lower-level shader language. The compilation may be just-in-time (JIT) compilation, or the application may perform shader pre-compilation. In some embodiments, high-level shaders are compiled into lower-level shaders during the compilation of the 3D graphics application 2310. In some embodiments, shader instructions 2312 are provided in an intermediate form, such as a version of the Standard Portable Intermediate Representation (SPIR) used by the Vulkan API.

[0323] In some embodiments, the user-mode graphics driver 2326 includes a backend shader compiler 2327 that translates shader instructions 2312 into hardware-specific representations. When the OpenGL API is used, shader instructions 2312 in the GLSL high-level language are passed to the user-mode graphics driver 2326 for compilation. In some embodiments, the user-mode graphics driver 2326 communicates with the kernel-mode graphics driver 2329 using operating system kernel-mode functions 2328. In some embodiments, the kernel-mode graphics driver 2329 communicates with the graphics processor 2332 to dispatch commands and instructions. IP Core Implementation

[0324] One or more aspects of at least one embodiment may be implemented by representative code stored in a machine-readable medium that represents and / or defines the logic within an integrated circuit, such as a processor. For example, the machine-readable medium may include instructions that represent various logic within the processor. When read by a machine, the instructions cause the machine to assemble the logic to perform the techniques described herein. Such representations, known as "IP cores," are reusable units of logic for an integrated circuit that may be stored in a tangible machine-readable medium as hardware models describing the structure of the integrated circuit. Hardware models may be supplied to various customers or manufacturing facilities that mount the hardware models on assembly machines that manufacture integrated circuits. Integrated circuits may be manufactured such that the circuits perform the operations described in relation to any of the embodiments described herein.

[0325] Figure 24A is a block diagram showing an IP core development system 2400 that may be used to manufacture an integrated circuit that performs the operation according to an embodiment. The IP core development system 2400 may be used to produce a modular, reusable design that can be incorporated into a larger design, or it may be used to build an entire integrated circuit (e.g., a SoC integrated circuit). The design facility 2430 can generate a software simulation 2410 of the IP core design in a high-level programming language (e.g., C / C++). The software simulation 2410 may be used to design, test, and verify the behavior of the IP core using a simulation model 2412. The simulation model 2412 may include functional, behavioral, and / or timing simulations. Next, a register transfer level (RTL) design 2415 may be created or synthesized from the simulation model 2412. The RTL design 2415 extracts the behavior of the integrated circuit that models the flow of digital signals between hardware registers and includes the associated logic that is executed using the modeled digital signals. In addition to the RTL design 2415, lower-level designs at the logic level or transistor level may be created, designed, or synthesized. Therefore, certain details of the initial design and simulation may change.

[0326] The RTL design 2415 or equivalent may be further synthesized by the design facility into a hardware model 2420, which may be a hardware description language (HDL) or some other representation of the physical design data. The HDL may be further simulated or tested to verify the IP core design. The IP core design may be stored in non-volatile memory 2440 (e.g., a hard disk, flash memory, or any non-volatile storage medium) for delivery to a third-party manufacturing facility 2465. Alternatively, the IP core design may be transmitted via a wired connection 2450 or a wireless connection 2460 (e.g., via the Internet). The manufacturing facility 2465 then assembles an integrated circuit that is at least partially based on the IP core design. The manufactured integrated circuit may be configured to perform operation according to at least one embodiment described herein.

[0327] Figure 24B shows a vertical cross-sectional view of an integrated circuit package assembly 2470 according to several embodiments described herein. The integrated circuit package assembly 2470 represents an implementation of one or more processors or accelerator devices described herein. The package assembly 2470 includes several units of hardware logic 2472, 2474 connected to a substrate 2480. The logic 2472, 2474 may be at least partially implemented in logic hardware of configurable logic or fixed function and may include one or more parts of any of the processor cores, graphics processors or other accelerator devices described herein. Each unit of logic 2472, 2474 may be implemented in a semiconductor die and coupled to the substrate 2480 via an interconnection structure 2473. The interconnection structure 2473 may be configured to route electrical signals between the logic 2472, 2474 and the substrate 2480 and may include, but is not limited to, interconnections such as bumps or pillars. In some embodiments, the interconnect structure 2473 may be configured to route electrical signals, such as input / output (I / O) signals and / or power or ground signals associated with the operation of logic 2472, 2474. In some embodiments, the substrate 2480 is an epoxy-based laminate. In other embodiments, the package substrate 2480 may include other suitable types of substrates. The package assembly 2470 may be connected to other electrical devices via the package interconnect 2483. The package interconnect 2483 may be bonded to the surface of the substrate 2480 to route electrical signals to other electrical devices such as a motherboard, other chipsets, or multi-chip modules.

[0328] In some embodiments, units of logic 2472 and 2474 are electrically coupled to a bridge 2482 configured to route electrical signals between logic 2472 and 2474. The bridge 2482 may be a high-density interconnection structure that provides the routing of electrical signals. The bridge 2482 may include a bridge substrate made of glass or a suitable semiconductor material. The electrical routing mechanism may be formed on the bridge substrate that provides the inter-chip connection between logic 2472 and 2474.

[0329] Although two logic units, 2472 and 2474, and a bridge, 2482, are illustrated, in the embodiments described herein, more or fewer logic units may be included on one or more dies. Since bridge 2482 may be omitted when the logic is contained on a single die, one or more dies may be connected by zero or fewer bridges. Alternatively, multiple dies or units of logic may be connected by one or more bridges. Furthermore, multiple logic units, dies, and bridges may be connected together with other possible configurations, including a three-dimensional configuration.

[0330] Figure 24C shows a package assembly 2490 containing multiple units of hardware logic chiplets connected to a substrate 2480 (e.g., a base die). The graphics processing units, parallel processors, and / or compute accelerators described herein may consist of a variety of separately manufactured silicon chiplets. In this context, a chiplet is at least partially packaged integrated circuit containing separate units of logic that can be assembled with other chiplets to form a larger package. A variety of chiplets with different IP core logic may be assembled into a single device. Furthermore, chiplets may be integrated with a base die or base chiplet using active interposer technology. The concepts described herein enable interconnection and communication between different forms of IP within a GPU. IP cores may be manufactured using different process technologies and configured during manufacturing, which avoids the complexity of converging multiple IPs, especially on a large SoC with several flavor IPs, into a single manufacturing process. Enabling the use of multiple processing technologies improves time to market, provides a cost-effective way, and allows for the creation of multiple product SKUs. Furthermore, the segmented IP is suitable for independent power gates, allowing components not used in a given workload to be powered off, potentially reducing overall power consumption.

[0331] The hardware logic chiplets may include a dedicated hardware logic chiplet 2472, a logic or I / O chiplet 2474, and / or a memory chiplet 2475. The hardware logic chiplets 2472 and 2474 may be at least partially implemented in logic hardware of configurable logic or fixed functionality and may include one or more parts of any of the processor cores, graphics processors, parallel processors, or other accelerator devices described herein. The memory chiplet 2475 may be DRAM (e.g., GDDR, HBM) memory or cache (SRAM) memory.

[0332] Each chiplet may be manufactured as a separate semiconductor die and coupled to a substrate 2480 via an interconnection structure 2473. The interconnection structure 2473 may be configured to route electrical signals between the various chiplets and the logic within the substrate 2480. The interconnection structure 2473 may include interconnections such as protrusions or columns, but is not limited to these. In some embodiments, the interconnection structure 2473 may be configured to route electrical signals such as input / output (I / O) signals associated with the operation of logic, I / O, and memory chiplets, and / or power or ground signals.

[0333] In some embodiments, the substrate 2480 is an epoxy-based laminated substrate. In other embodiments, the substrate 2480 may include other suitable types of substrates. The package assembly 2490 may be connected to other electrical devices via a package interconnect 2483. The package interconnect 2483 may be bonded to the surface of the substrate 2480 to route electrical signals to other electrical devices such as a motherboard, other chipsets, or multi-chip modules.

[0334] In some embodiments, a logic or I / O chiplet 2474 and a memory chiplet 2475 may be electrically coupled via a bridge 2487 configured to route electrical signals between the logic or I / O chiplet 2474 and the memory chiplet 2475. The bridge 2487 may be a high-density interconnect structure that provides the routing of electrical signals. The bridge 2487 may include a bridge substrate made of glass or a suitable semiconductor material. The electrical routing mechanism may be formed on the bridge substrate that provides the inter-chip connection between the logic or I / O chiplet 2474 and the memory chiplet 2475. The bridge 2487 may also be referred to as a silicon bridge or interconnect bridge. For example, in some embodiments, the bridge 2487 is an embedded multi-die interconnect bridge (EMIB). In some embodiments, the bridge 2487 may simply be a direct connection from one chiplet to another.

[0335] The board 2480 may include hardware components relating to I / O 2491, cache memory 2492, and other hardware logic 2493. The fabric 2485 may be embedded in the board 2480 to enable communication between various logic chiplets and logic 2491, 2493 within the board 2480. In one embodiment, I / O 2491, fabric 2485, cache, bridge, and other hardware logic 2493 may be integrated into a base die that is layered on top of the board 2480.

[0336] In various embodiments, the package assembly 2490 may contain fewer or more components and chiplets than the number interconnected by the fabric 2485 or one or more bridges 2487. The chiplets within the package assembly 2490 may be arranged in a 3D or 2.5D configuration. Generally, the bridge structure 2487 may be used, for example, to facilitate point-to-point interconnects between logic or I / O chiplets and memory chiplets. The fabric 2485 may be used to interconnect various logic and / or I / O chiplets (e.g., chiplets 2472, 2474, 2491, 2493) with other logic and / or I / O chiplets. In one embodiment, the onboard cache memory 2492 may act as a global cache for the package assembly 2490, a portion of an allocated global cache, or a dedicated cache for the fabric 2485.

[0337] Figure 24D shows a package assembly 2494 including a replaceable chiplet 2495 according to an embodiment. The replaceable chiplet 2495 can be assembled with standard slots on one or more base chiplets 2496, 2498. The base chiplets 2496, 2498 can be coupled via a bridge interconnect 2497, which can be similar to other bridge interconnects described herein, such as an EMIB. Memory chiplets can also be connected to logic or I / O chiplets via the bridge interconnect. I / O and logic chiplets can communicate via an interconnect fabric. Each base chiplet may support one or more slots in one of the standard formats of logic or I / O or memory / cache.

[0338] In one embodiment, the SRAM and power supply circuitry may be manufactured into one or more base chiplets 2496, 2498, which may be manufactured using different process technologies compared to a replaceable chiplet 2495 stacked on top of the base chiplet. For example, the base chiplets 2496, 2498 may be manufactured using a larger processing technology, while the replaceable chiplet may be manufactured using a smaller processing technology. One or more replaceable chiplets 2495 may be memory (e.g., DRAM) chiplets. Different memory densities may be selected with respect to the package assembly 2494 based on the target power and / or performance of the product using the package assembly 2494. Furthermore, logic chiplets having different numbers of types of functional units may be selected at the time of assembly based on the target power and / or performance of the product. Furthermore, chiplets containing different types of IP logic cores may be inserted into replaceable chiplet slots, enabling hybrid processor designs in which different technology IP blocks can be mixed and matched. Exemplary system on a chip integrated circuit

[0339] Figures 25 to 26B illustrate exemplary integrated circuits and associated graphics processors that may be manufactured using one or more IP cores according to various embodiments described herein. In addition to those shown, other logic and circuitry may be included, including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores.

[0340] Figure 25 is a block diagram showing an exemplary system on a chip integrated circuit 2500 that may be manufactured using one or more IP cores according to an embodiment. The exemplary integrated circuit 2500 includes one or more application processors 2505 (e.g., CPUs), at least one graphics processor 2510, and may further include an image processor 2515 and / or a video processor 2520, any of which may be modular IP cores from the same or multiple different design facilities. The integrated circuit 2500 includes a USB controller 2525, a UART controller 2530, an SPI / SDIO controller 2535, and I 2 S / I 2 The integrated circuit includes peripherals or bus logic, including a C controller 2540. Furthermore, the integrated circuit may include a display device 2545 coupled with one or more High-Definition Multimedia Interface (HDMI®) controllers 2550 and a Mobile Industry Processor Interface (MIPI) display interface 2555. Storage may be provided by a flash memory subsystem 2560, including flash memory and a flash memory controller. A memory interface may be provided via a memory controller 2565 for access to SDRAM or SRAM memory devices. Some integrated circuits further include an embedded security engine 2570.

[0341] Figures 26A and 26B are block diagrams illustrating exemplary graphics processors for use in a SoC according to embodiments described herein. Figure 26A shows an exemplary graphics processor 2610 of a system on a chip integrated circuit that may be manufactured using one or more IP cores according to an embodiment. Figure 26B shows a further exemplary graphics processor 2640 of a system on a chip integrated circuit that may be manufactured using one or more IP cores according to an embodiment. Graphics processor 2610 in Figure 26A is an example of a low-power graphics processor core. Graphics processor 2640 in Figure 26B is an example of a higher-performance graphics processor core. Graphics processors 2610 and 2640, respectively, may be variations of graphics processor 2510 in Figure 25.

[0342] As shown in Figure 26A, the graphics processor 2610 includes a vertex processor 2605 and one or more fragment processors 2615A-2615N (e.g., 2615A, 2615B, 2615C, 2615D through 2615N-1 and 2615N). The graphics processor 2610 can execute different shader programs through separate logic; therefore, the vertex processor 2605 is optimized to perform operations for vertex shader programs, while one or more fragment processors 2615A-2615N perform fragment (e.g., pixel) shading operations for fragment or pixel shader programs. The vertex processor 2605 executes the vertex processing stage of the 3D graphics pipeline and generates primitive and vertex data. The fragment processors 2615A-2615N use the primitive and vertex data generated by the vertex processor 2605 to generate a frame buffer that is displayed on the display device. In one embodiment, the fragment processors 2615A-2615N are optimized to execute fragment shader programs such as those provided in the OpenGL API and may be used to perform operations similar to those of pixel shader programs directly provided in the 3D API.

[0343] The graphics processor 2610 further includes one or more memory management units (MMUs) 2620A-2620B, caches 2625A-2625B, and circuit interconnects 2630A-2630B. One or more MMUs 2620A-2620B provide virtual-physical address mappings to the graphics processor 2610, including vertex processors 2605 and / or fragment processors 2615A-2615N, and may include reference vertex or image / texture data stored in memory, in addition to vertex or image / texture data stored in one or more caches 2625A-2625B. In one embodiment, one or more MMUs 2620A-2620B may be synchronized with other MMUs in the system and include one or more MMUs associated with one or more application processors 2505, image processors 2515 and / or video processors 2520 in Figure 25, so that each processor 2505-2520 can join a shared or integrated virtual memory system. One or more circuit interconnects 2630A-2630B, according to the embodiment, allow the graphics processor 2610 to interface with other IP cores in the SoC, either via the SoC's internal bus or via direct connection.

[0344] As shown in Figure 26B, the graphics processor 2640 includes one or more MMUs 2620A-2620B, caches 2625A-2625B, and circuit interconnects 2630A-2630B of the graphics processor 2610 in Figure 26A. The graphics processor 2640 includes one or more shader cores 2655A-2655N (e.g., 2655A, 2655B, 2655C, 2655D, 2655E, 2655F through 2655N-1 and 2655N), and the graphics processor 2640 provides a single-core or single-core integrated shader core architecture in which a single core can execute all types of programmable shader code, and the integrated shader core architecture includes shader program code that implements vertex shaders, fragment shaders, and / or compute shaders. The exact number of shader cores presented may vary between embodiments and implementations. Furthermore, the graphics processor 2640 includes an inter-core task manager 2645, which functions as a thread dispatcher that dispatches execution threads to one or more shader cores 2655A-2655N, and as a tiling unit 2658 that accelerates tiling operations for tile-based rendering, in which rendering operations on the scene are subdivided in image space, for example, to take advantage of coherence in local space within the scene or to optimize the use of the internal cache.

[0345] Convolutional neural networks (CNNs) are used in a wide range of image processing tasks, such as denoising machine-rendered images. Kernel prediction networks (KPNs) are a deep learning solution specifically for ray tracing denoising, but they use CNNs to estimate local weighting kernels that are used to calculate denoised pixels from neighboring pixels.

[0346] Figure 27A is a schematic diagram showing a conventional kernel prediction network (KPN). The kernel prediction network will be referred to as "KPN" below. The KPN2700 shown may be a deep and complete convolutional neural network implemented or deployed on a suitable computing engine or any computing hardware for convolutional neural networks or similar deep learning neural networks.

[0347] During training, the KPN2700 is trained with a dataset of pre-collected noisy images and ground validation images of each noisy image. The KPN can be trained and tuned with datasets of numerous iterations to learn its network parameters, such as batch size, node information, and bias. The kernel size of the KPN's convolutional kernel is assigned before the KPN is trained. For example, the kernel size of the KPN's convolutional kernel may be assigned as k × k, where the value of k is selected from one of the following: 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, or similar.

[0348] During runtime estimation of image denoising, the KPN2700 may receive an image, denoise it, and produce a denoised image as output. The image may have multiple pixels. The KPN2700 may include kernel generation logic 2710 and filtering logic 2730. After the image is received, the kernel generation logic 2710 of the KPN2700 may generate a convolution kernel for each of the multiple pixels of the image. The convolution kernel may have multiple kernel values. For each pixel of the image, the filtering logic 2730 of the KPN2700 may use the multiple kernel values ​​of the corresponding convolution kernel generated by the kernel generation logic 2710 to filter the pixel and obtain a denoised pixel. After each of the multiple pixels of the image has been filtered, the KPN2700 may produce a denoised image as output. Details of the filtering described above are explained below with reference to Figure 27B.

[0349] Figure 27B is a schematic diagram showing an example of pixel filtering using a conventional KPN. As previously discussed with respect to Figure 27A, for each pixel of the image, the filtering logic 2730 of the KPN 2700 can filter the pixel and produce a denoised pixel using multiple kernel values ​​of the corresponding convolutional kernel generated by the kernel generation logic 2710.

[0350] More specifically, in the example in Figure 27B, a convolution kernel 2770 with a kernel size of 3x3 is generated with respect to pixel P0 at pixel position 2760-0 of the image, and the convolution kernel 2770 has 9 kernel values ​​from W0 to W8. As shown in Figure 27B, with respect to pixel P0, the KPN2700 can filter pixel P0 with the convolution kernel 2770 having 9 kernel values ​​W0 to W8 in a 3x3 square window centered at pixel position 2760-0 of the image, and the pixel values ​​at 9 pixel positions 2760-0, 2760-1, 2760-2, 2760-3, 2760-4, 2760-5, 2760-6, 2760-7, and 2760-8, and obtain the pixel value of the denoised pixel. More specifically, filtering of pixel P0 is performed to obtain the pixel value of the denoised pixel [Math 1] This can be achieved by calculating a weighted average of the pixel values ​​at the nine pixel positions 2760-0, 2760-1, 2760-2, 2760-3, 2760-4, 2760-5, 2760-6, 2760-7, and 2760-8. In other words, the pixel values ​​of the denoised pixels. [Math 2] However, it can be calculated by the following formula. [Math 3]

[0351] Typically, the sum of kernel values ​​W0 through W8 for a given number 9 can be equal to that obtained in image processing tasks such as image denoising.

[0352] Conventional KPNs have shown potential in image processing tasks such as ray tracing denoising, but this is due to the use of large convolutional kernels, such as the commonly used 21x21 kernel size, which sometimes fails to provide a good balance between image quality and processing cost. To achieve better or similar image quality at a much lower processing cost, an offset can be introduced so that the pixel positions used in KPN image denoising are adaptively determined. Further details will be discussed below.

[0353] In some embodiments, adaptive deformation kernel prediction networks (ADKPNs) are used instead of KPNs in image processing tasks such as image denoising. Generally, ADKPNs can not only generate convolution kernels for each pixel in an image, but also generate offsets that determine the pixel positions used for image denoising. ADKPNs can use pixel values ​​for pixel positions based on offsets in image pixel denoising, rather than using fixed pixel values ​​for pixel positions in a square window centered on the pixel position of pixel P0, as shown in Figure 27B. Such a smarter generalization and filtering framework for convolution kernels may offer significantly better denoising performance compared to convolutional KPNs.

[0354] Figure 28A is a schematic diagram showing an adaptive deformable kernel prediction network (ADKPN) according to an embodiment. The adaptive deformable kernel prediction network will be referred to as "ADKPN" below. The ADKPN2800 shown may be a convolutional neural network implemented or deployed on a computing engine or any computing hardware suitable for convolutional neural networks or similar deep learning neural networks, including, but not limited to, graphics processing units (GPUs), central processing units (CPUs), application-specific integrated circuits (ASICs), etc.

[0355] During training, the ADKPN2800 is trained with a training dataset of representative images, such as pre-collected noisy images and ground verification images of each of those noisy images. For example, the training dataset may include a number of representative images, such as machine-generated noisy images produced by Monte Carlo rendering or any other image rendering solution. In further examples, the training dataset may include test images rendered with 128 samples (spp) per pixel, as well as reference images rendered with 8192 spp or higher. The number of samples per pixel (spp) is known in the art as the number of samples used for rendering pixels by random sampling rendering algorithms such as Monte Carlo rendering. A larger spp results in better image quality.

[0356] Training procedures for ADKPN2800 can be performed using typical training approaches for deep neural networks, such as stochastic gradient descent (SGD), Momentum, AdaGrad, and Adam. ADKPN2800 can be trained and tuned with training datasets for a large number of iterations, learning its network parameters such as batch size, node information, and bias.

[0357] During runtime estimation of image denoising, the ADKPN2800 receives an image, denoises it, and produces a denoised image as output. The image can be any image, such as a machine-generated image produced by Monte Carlo rendering or any other image rendering solution. The image may contain multiple pixels.

[0358] The ADKPN2800 may include kernel generation logic 2810, offset generation logic 2820, and filtering logic 2830. After an image is received, for each of several pixels in the image, the kernel generation logic 2810 of the ADKPN2800 may generate a convolution kernel having several kernel values ​​for the pixel, and the offset generation logic 2820 of the ADKPN2800 may generate several offsets for the pixel corresponding to each of the several kernel values ​​of the convolution kernel, each of which indicates a deviation of the pixel from its pixel position. Next, for each of several pixels in the image, the filtering logic 2830 of the ADKPN2800 may determine several deviated pixel positions and several offsets generated for the pixel by the offset generation logic 2820 based on the pixel position of the pixel, and filters the pixel using the convolution kernel generated for the pixel by the kernel generation logic 2810, as well as the pixel values ​​of the several deviated pixel positions, to obtain a denoised pixel. After each of the multiple pixels in the image has been filtered, the ADKPN2800 can produce a denoised image as output. Details of the filtering described above are explained below with reference to Figure 28B.

[0359] Figure 28B is a schematic diagram showing an example of pixel filtering by ADKPN according to an embodiment. As previously discussed with respect to Figure 28A, the filtering logic 2830 of ADKPN2800 can determine a plurality of deviated pixel positions and a plurality of offsets generated with respect to the pixel based on the pixel position of the pixel, and uses the convolution kernel generated with respect to the pixel, as well as the pixel values ​​of the plurality of deviated pixel positions, to filter the pixel and obtain a denoised pixel.

[0360] More specifically, in the example in Figure 28B, a convolution kernel 2870 with a kernel size of 3x3 is generated with respect to pixel P0 at pixel position 2840 of the image, and the convolution kernel 2870 has 9 kernel values ​​from W0 to W8. The 9 offsets 2850-0, 2850-1, 2850-2, 2850-3, 2850-4, 2850-5, 2850-6, 2850-7, and 2850-8 are also generated with respect to pixel P0 and correspond to the 9 kernel values ​​W0 to W8, respectively. As shown in Figure 28B, with respect to pixel P0, the filtering logic 2830 of ADKPN2800 can determine the deviated pixel positions 2860-0, 2860-1, 2860-2, 2860-3, 2860-4, 2850-5, 2850-6, 2850-7, and 2860-8 of the image marked as P0'-P8' in Figure 28B, based on the pixel position 2840 of pixel P0 and the offsets 2850-0, 2850-1, 2850-2, 2850-3, 2850-4, 2860-5, 2860-6, 2860-7, and 2860-8 of the image. The filtering logic 2830 of the ADKPN2800 can then filter pixel P0 with a convolution kernel 2870 having 9 kernel values ​​W0 to W8 and the pixel values ​​of 9 deviated pixel positions 2860-0, 2860-1, 2860-2, 2860-3, 2860-4, 2860-5, 2860-6, 2860-7 and 2860-8, and the denoised pixel values [Math 4] This is obtained. More specifically, the filtering of pixel P0 is the pixel value of the denoised pixel. [Math 5] This can be done by calculating a weighted average of the pixel values ​​at the nine deviated pixel positions of the image: 2860-0, 2860-1, 2860-2, 2860-3, 2860-4, 2860-5, 2860-6, 2860-7, and 2860-8. In other words, the pixel values ​​of the denoised pixels. [Math 6] However, it can be calculated by the following formula. [Number 7]

[0361] In some embodiments, the sum of multiple kernel values ​​of the convolution kernel generated with respect to the pixels of the image may be 1. In the example of Figure 28B, the sum of the 9 kernel values ​​W0 to W8 of the convolution kernel 2870 may be 1.

[0362] In some embodiments, each of the multiple offsets generated with respect to a pixel in an image may include a position value indicating a deviation of the pixel from its pixel position. For example, each of the offsets 2850-0, 2850-1, 2850-2, 2850-3, 2850-4, 2850-5, 2850-6, 2850-7, and 2850-8 of 9 may include a position value indicating a deviation from pixel position 2840, as shown in Figure 28B. For example, offset 2850-0 may include a position value (u0, v0) indicating a deviation from pixel position 2840. If pixel position 2840 is represented by a position value (x, y), then pixel position 2860-0 may be determined by a position value (x+u0, y+v0) based on pixel position 2840 and offset 2850-0.

[0363] In some embodiments, the position value may include a floating-point value. For example, the aforementioned offset 2850-0 may include a floating-point value (u0, v0). Thus, in a non-limiting example, the pixel value of pixel position 2860-0 with position value (x+u0, y+v0) may be determined by pixel interpolation known in the art, such as nearest neighbor interpolation or bilinear interpolation.

[0364] In some embodiments, multiple kernel values ​​of the convolution kernel may differ with respect to at least two pixels of the image.

[0365] In some embodiments, multiple offsets may differ with respect to at least two pixels of the image.

[0366] In some embodiments, the plurality of offsets can be generated before or simultaneously with the generation of the convolution kernel. For example, after an image is received, the plurality of offsets can first be generated for each of the plurality of pixels of the image, and then the convolution kernel can be generated for each of the plurality of pixels of the image. Alternatively, the plurality of offsets can be generated simultaneously with the convolution kernel for each of the plurality of pixels of the image.

[0367] In some embodiments, the aforementioned upper limit of deviation can be predetermined, for example, before the training of the convolutional neural network. The upper limit of deviation can be predetermined so as to control the computational processing cost of the training / learning procedure of ADKPN2800. For example, for the convolution kernel with a kernel size of 3×3 of ADKPN2800, the upper limit of deviation can be predetermined so as not to be greater than 11 pixels.

[0368] In some embodiments, the kernel size of the convolution kernel can be pre-assigned, for example, before the training of the convolutional neural network. In some embodiments, the kernel size can be pre-assigned as one of 3×3, 5×5, 7×7, …, (2n - 1)×(2n - 1), where n is a positive integer, n >= 2 and n < min(floor(W / 2), floor(H / 2)), and W and H are the width and height of the normalized image used for training, or the width and height of the image to be denoised in runtime estimation. The above mathematical function "floor()" refers to the floor operation known in the art.

[0369] Figure 28C is a graph showing a comparison of training and / or test performance measurements between conventional KPN and ADKPN. In particular, graphs like the one shown in Figure 28C show the loss (vertical axis) against computational iterations (horizontal axis) for conventional KPN and ADKPN. It can be clearly seen that ADKPN produces better results than conventional KPN, i.e., lower losses for similar computational iterations. Alternatively, ADKPN can achieve similar results with lower computational cost than conventional KPN.

[0370] Figure 29 is a flowchart showing a method for denoising images using ADKPN according to an embodiment.

[0371] The ADKPN-based image denoising method 2900 may be implemented on a computing engine or any computing hardware suitable for convolutional neural networks or similar deep learning neural networks, but is not limited to graphics processing units (GPUs), central processing units (CPUs), application-specific integrated circuits (ASICs), etc. The image may contain multiple pixels. In some embodiments, the image may be any image, such as a machine-generated image produced by Monte Carlo rendering or any other image rendering solution.

[0372] Method 2900 may include operations 2910-2940 on each of multiple pixels of the image.

[0373] In block 2910, a convolution kernel having multiple kernel values ​​can be generated for each pixel.

[0374] In block 2920, multiple offsets corresponding to multiple kernel values ​​may be generated with respect to a pixel, and each of these offsets indicates a deviation of the pixel from its pixel position.

[0375] In block 2930, multiple deviated pixel positions can be determined based on the pixel position of the pixel and multiple offsets.

[0376] In block 2940, pixels can be filtered by a convolution kernel and pixel values ​​of multiple deviated pixel positions to obtain denoised pixels.

[0377] In some embodiments, each of the multiple offsets may include a position value indicating a deviation of the pixel from its pixel position.

[0378] In some embodiments, the position value may include a floating-point value.

[0379] In some embodiments, multiple kernel values ​​may differ with respect to at least two pixels of the image.

[0380] In some embodiments, multiple offsets may differ with respect to at least two pixels of the image.

[0381] In some embodiments, multiple offsets may be generated before or simultaneously with the generation of the convolution kernel.

[0382] In some embodiments, filtering pixels having convolution kernels and pixel values ​​for multiple deviant pixel locations may include applying multiple kernel values ​​of the convolution kernel to the pixel values ​​of multiple deviant pixel locations to obtain a weighted average of the pixel values ​​for the multiple pixel locations.

[0383] In some embodiments, the upper limit of deviation may be predetermined, for example, before training the convolutional neural network.

[0384] In some embodiments, the kernel size of the convolutional kernel may be pre-assigned as one of 3×3, 5×5, 7×7, … (2n−1)×(2n−1) before, for example, training a convolutional neural network, where n is a positive integer, n >= 2 and n < min(floor(W / 2), floor(H / 2)), and W and H are the width and height of the image. The mathematical function "floor()" refers to the floor operation, which is known in the art.

[0385] Figures 30A to 30D respectively show an input image, a reference image, an output image generated by a conventional KPN, and an output image generated by an ADKPN according to an embodiment.

[0386] Figure 30A shows an input image synthetically generated by a machine with 128 samples per pixel (spp). Figure 30B shows a reference image synthetically generated by a machine with 8192 spp. The number of samples per pixel (spp) is known in the art as the number of samples used for rendering pixels by a random sampling rendering algorithm such as Monte Carlo rendering. As the spp increases, the image quality becomes better. It can be seen that the input image in Figure 30A has noise, while the reference image in Figure 30B is of high quality.

[0387] Figure 30C shows an output image generated by a conventional KPN with a kernel size of 9×9 by removing noise from the input image in Figure 30A. It can be seen that there are obvious artifacts in the area within the ellipse 3002. Figure 30D shows an output image generated by an ADKPN with a kernel size of 5×5 by removing noise from the input image in Figure 30A according to an embodiment. In contrast to the output image in Figure 30C, it can be seen that the image in Figure 30D exhibits a quality similar to that of the reference image in Figure 30B without artifacts.

[0388] Similarly, Figure 31A shows a noisy input image machine-synthesized with 128 samples (spp) per pixel. Figure 31B shows a high-quality reference image machine-synthesized with 8192 spp. Figure 31C shows the output image generated by a conventional KPN with a 9x9 kernel size after denoising the input image of Figure 31A. Obvious artifacts can be seen in the regions within ellipses 3102, 3104, and 3106. Figure 31D shows the output image generated by an ADKPN with a 5x5 kernel size after denoising the input image of Figure 31A according to this embodiment. In contrast to the output image of Figure 31C, the image of Figure 31D can be seen to represent a similar quality to the artifact-free reference image of Figure 31B.

[0389] Similarly, Figure 32A shows a noisy input image machine-synthesized with 128 samples (spp) per pixel. Figure 32B shows a high-quality reference image machine-synthesized with 8192 spp. Figure 32C shows the output image generated by a conventional KPN with a 9x9 kernel size after denoising the input image of Figure 32A. Obvious artifacts can be seen in the regions within ellipses 3202 and 3204. Figure 32D shows the output image generated by an ADKPN with a 5x5 kernel size after denoising the input image of Figure 32A according to this embodiment. In contrast to the output image of Figure 32C, the image of Figure 32D can be seen to represent a similar quality to the artifact-free reference image of Figure 32B.

[0390] Similarly, Figure 33A shows a noisy input image machine-synthesized with 128 samples (spp) per pixel. Figure 33B shows a high-quality reference image machine-synthesized with 8192 spp. Figure 33C shows the output image generated by a conventional KPN with a 9x9 kernel size after denoising the input image of Figure 33A. Obvious artifacts can be seen in the region within ellipse 3302. Figure 33D shows the output image generated by an ADKPN with a 5x5 kernel size after denoising the input image of Figure 33A according to this embodiment. In contrast to the output image of Figure 33C, the image of Figure 33D can be seen to represent a similar quality to the artifact-free reference image of Figure 33B.

[0391] Furthermore, Figures 30C to 30D, 31C to 31D, 32C to 32D, and 33C to 33D also show the Structural Differences in Identification (DSSIM) index of the output images, respectively. DSSIM, as is known in the art, refers to structural differences and is a variation of Structural Similarities in Identification (SSIM), serving as an index of image differences. The DSSIM index is used to compare the structural differences between the output image and the ground verification image. A smaller DSSIM index indicates a higher quality output image. For example, the DSSIM index of the output image by KPN in Figure 30C may be 0.0672, while the DSSIM index of the output image by ADKPN in Figure 30D may be 0.0610. As can be seen from Figures 30C to 30D, 31C to 31D, 32C to 32D, and 33C to 33D, all output images generated by ADKPN have a smaller DSSIM index than those generated by conventional KPN. In other words, all output images generated by ADKPN are of higher quality than those generated by conventional KPN.

[0392] Therefore, it can be clearly seen that ADKPN with a smaller kernel size performs significantly better than KPN with a larger kernel size.

[0393] Some of the various embodiments may be provided as a computer program product, which may include a computer-readable medium having computer program instructions stored therein, which may be used to program a computer (or other electronic device) for execution by one or more processors to perform processing according to a particular embodiment. The computer-readable medium may include, but is not limited to, magnetic disks, optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic or optical cards, flash memory, or other types of computer-readable medium suitable for storing electronic instructions. Furthermore, embodiments may also be downloaded as a computer program product, and the program may be transferred from a remote computer to a requesting computer. In some embodiments, a non-temporary computer-readable storage medium has data stored therein representing a sequence of instructions, which, when executed by a processor, causes the processor to perform a particular operation.

[0394] Many of these methods are described in their most basic forms, but processing may be added to or removed from any of these methods without departing from the basic scope of these embodiments, and information may be added to or removed from any of the described messages. It will be apparent to those skilled in the art that many further modifications and adaptations are possible. The specific embodiments described above are provided for illustrative purposes only, not to limit the concept. The scope of the embodiments is not determined by the specific examples provided above, but solely by the following claims.

[0395] If it is stated that element "A" is combined with or together with element "B", element A may be directly combined with element B, or indirectly combined through, for example, element C. If the specification or claim states that component, feature, structure, process, or characteristic A "causes" component, feature, structure, process, or characteristic B, it means that "A" is at least a partial cause of "B", but there may also be at least one other component, feature, structure, process, or characteristic that helps bring about the occurrence of "B". If the specification indicates that a component, feature, structure, process, or characteristic "may", "might", or "could", that particular component, feature, structure, process, or characteristic does not have to be included. If the specification or claim refers to one ("a" or "an") element, it does not mean that there is only one element described.

[0396] Embodiments are implementations or examples. References in the specification to “an embodiment,” “one embodiment,” “some embodiments,” or “other embodiments” mean that certain features, structures, or characteristics described in relation to that embodiment are included in at least some embodiments, but not necessarily in all embodiments. The various expressions “embodiment,” “one embodiment,” or “some embodiments” do not all necessarily refer to the same embodiment. Naturally, it should be understood that in the above descriptions of exemplary embodiments, various features may be grouped together in these single embodiments, figures, or descriptions for the purpose of simplifying the disclosure and aiding in the understanding of one or more of the various novel embodiments. However, this method of disclosure should not be interpreted as reflecting an intention that the embodiments described in the claims require more features than are explicitly described in each claim. Rather, as reflected in the following claims, the novel embodiments have fewer features than all of the single embodiments disclosed above. Therefore, the claims are explicitly incorporated in this specification, and each claim stands alone as a separate embodiment.

[0397] The following sections and / or examples relate to further embodiments or examples. Specific features in such examples may be used in any one or more embodiments. Various features of different embodiments or examples may be combined in diverse ways with some of the features included, and other features may be excluded to suit various different applications. Examples may include subject matter such as methods, means for performing the actions of the methods, and at least one machine-readable medium containing instructions for causing a machine to perform the actions of a method, apparatus, or system, which, when performed by a machine, facilitates hybrid communication according to the embodiments and examples described herein.

[0398] Further examples

[0399] Example 1 includes a method for denoising an image using a convolutional neural network implemented on a compute engine, wherein the image comprises multiple pixels, and the method comprises the steps of: generating a convolutional kernel having multiple kernel values ​​for each of the multiple pixels of the image; generating multiple offsets for each of the multiple kernel values, each of which indicates a deviation of the pixel from its pixel position; determining multiple deviated pixel positions based on the pixel position and the multiple offsets; and filtering the pixels by the convolutional kernels and pixel values ​​of the multiple deviated pixel positions to obtain denoised pixels.

[0400] Example 2 includes the subject matter of Example 1, and each of the multiple offsets includes a position value indicating the deviation of the pixel from its pixel position.

[0401] Example 3 includes the theme of Example 2, and the positional values ​​include floating-point values.

[0402] Example 4 includes the subject matter of Example 1, and the multiple kernel values ​​differ with respect to at least two pixels of the image.

[0403] Example 5 includes the subject from Example 1, and the multiple offsets differ with respect to at least two pixels of the image.

[0404] Example 6 includes the subject matter of Example 1, where multiple offsets are generated before or simultaneously with the generation of the convolution kernel.

[0405] Example 7 includes the subject matter of Example 1, and the step of filtering pixels by convolution kernels and pixel values ​​for multiple deviant pixel locations includes the step of applying multiple kernel values ​​of the convolution kernel to the pixel values ​​of multiple deviant pixel locations to obtain a weighted average of the pixel values.

[0406] Example 8 includes the subject matter of Example 1, and the upper limit of deviation is predetermined.

[0407] Example 9 includes the subject matter of Example 1, and the kernel size of the convolutional kernel is pre-assigned as one of 3×3, 5×5, 7×7, …, (2n - 1)×(2n - 1), where n is a positive integer, n >= 2 and n < min(floor(W / 2), floor(H / 2)), and W and H are the width and height of the image.

[0408] Example 10 includes a data storage for storing data including an image, the image including a plurality of pixels, the data storage, and a compute engine coupled to the data storage, the compute engine removing noise from the image by means of a convolutional neural network, the compute engine generating, for each of the plurality of pixels of the image, a convolutional kernel having a plurality of kernel values for the pixel, generating a plurality of offsets for the pixel corresponding respectively to the plurality of kernel values, each of the plurality of offsets indicating a deviation from the pixel position of the pixel, determining, based on the pixel position of the pixel and the plurality of offsets, a plurality of deviated pixel positions, and filtering the pixel by means of the convolutional kernel and pixel values of the plurality of deviated pixel positions to obtain a noise-removed pixel, and includes an apparatus for noise removal of an image.

[0409] Example 11 includes the subject matter of Example 10, and each of the plurality of offsets includes a position value indicating a deviation from the pixel position of the pixel.

[0410] Example 12 includes the subject matter of Example 11, and the position value includes a floating-point value.

[0411] Example 13 includes the subject matter of Example 10, and the plurality of kernel values are different for at least two pixels of the image.

[0412] Example 14 includes the subject matter of Example 10, and the plurality of offsets are different for at least two pixels of the image.

[0413] Example 15 includes the subject matter of Example 10, and the plurality of offsets are generated before or simultaneously with the generation of the convolutional kernel.

[0414] Example 16 includes the subject matter of Example 10 and filtering a pixel by a convolution kernel and pixel values of a plurality of deviated pixel positions includes applying a plurality of kernel values of the convolution kernel to the pixel values of the plurality of deviated pixel positions to obtain a weighted average of the pixel values.

[0415] Example 17 includes the subject matter of Example 10 and an upper limit of deviation is predetermined.

[0416] Example 18 includes the subject matter of Example 10 and a kernel size of the convolution kernel is pre-assigned as one of 3×3, 5×5, 7×7, …, (2n−1)×(2n−1), where n is a positive integer, n >= 2 and n < min(floor(W / 2), floor(H / 2)), and W and H are the width and height of an image.

[0417] Example 19 includes an apparatus comprising means for performing the method claimed in any of Examples 1-9.

[0418] Example 20 includes a machine-readable medium storing a plurality of instructions that, when executed on a computing device, cause the computing device to perform the method claimed in any of Examples 1-9.

[0419] The above description and drawings should be regarded as illustrative rather than restrictive. Those skilled in the art will understand that various modifications and changes can be made to the embodiments described herein without departing from the broader spirit and scope of the invention as set forth in the appended claims.

Claims

1. 1. A method of denoising an image by a convolutional neural network implemented on a compute engine, the image comprising a plurality of pixels, the method comprising: For each of the plurality of pixels of the image, generating a convolution kernel having a plurality of kernel values ​​for the pixel; generating a plurality of offsets for the pixel corresponding respectively to the plurality of kernel values, each offset indicating a deviation of the pixel from a pixel location; determining a plurality of deviant pixel locations based on the pixel location of the pixel and the plurality of offsets; filtering the pixels with the convolution kernel and pixel values ​​of the plurality of deviant pixel locations to obtain denoised pixels; Equipped with the offsets are generated contemporaneously with the generation of the convolution kernel. method.

2. The method of claim 1 , wherein each of the plurality of offsets comprises a position value indicative of the deviation of the pixel from the pixel location.

3. The method of claim 2 , wherein the position value comprises a floating point value.

4. The method of claim 1 , wherein the kernel values ​​are different for at least two pixels of the image.

5. The method of claim 1 , wherein the offsets are different for at least two pixels of the image.

6. The step of filtering the pixels by the convolution kernel and pixel values ​​of the plurality of deviant pixel locations, comprising: applying the kernel values ​​of the convolution kernel to the pixel values ​​of the deviant pixel locations to obtain a weighted average of the pixel values.

6. The method according to any one of claims 1 to 5.

7. The method according to claim 1 , wherein the upper deviation limit is predetermined.

8. 8. The method of claim 1, wherein a kernel size of the convolution kernel is pre-assigned as (2n-1)x(2n-1), where n is a positive integer, n>=2 and n<min(floor(W / 2), floor(H / 2)), and W and H are the width and height of the image.

9. a data storage for storing data including an image, the image including a plurality of pixels; a compute engine coupled to the data storage, the compute engine denoising the image with a convolutional neural network, the compute engine comprising: For each of a plurality of pixels of the image, generating a convolution kernel having a plurality of kernel values ​​for the pixel; generating a plurality of offsets for the pixel corresponding respectively to the plurality of kernel values, each of the plurality of offsets indicating a deviation of the pixel from a pixel location; determining a plurality of deviant pixel locations based on the pixel locations of the pixels and the plurality of offsets; filtering the pixels with the convolution kernel and pixel values ​​of the plurality of deviant pixel locations to obtain denoised pixels; the offsets are generated contemporaneously with the generation of the convolution kernel. A device for image denoising.

10. The apparatus of claim 9 , wherein each of the plurality of offsets comprises a position value indicative of the deviation of the pixel from the pixel location.

11. The apparatus of claim 10 , wherein the position value comprises a floating point value.

12. The apparatus of claim 9 , wherein the kernel values ​​are different for at least two pixels of the image.

13. The apparatus of claim 9 , wherein the offsets are different for at least two pixels of the image.

14. The filtering of the pixels by the convolution kernel and pixel values ​​of the plurality of deviant pixel locations, comprising: applying the plurality of kernel values ​​of the convolution kernel to the pixel values ​​of the plurality of deviant pixel locations to obtain a weighted average of the pixel values.

14. Apparatus according to any one of claims 9 to 13.

15. 15. Apparatus according to any one of claims 9 to 14, wherein the upper limit of deviation is predetermined.

16. 16. The apparatus of claim 9, wherein a kernel size of the convolution kernel is pre-assigned as (2n-1)x(2n-1), where n is a positive integer, n>=2 and n<min(floor(W / 2), floor(H / 2)), and W and H are the width and height of the image.

17. Apparatus comprising means for carrying out the method claimed in any one of claims 1 to 8.

18. A computer program which, when executed on a computing device, causes said computing device to carry out a method as claimed in any one of claims 1 to 8.

19. 20. A machine readable medium storing the computer program of claim 18.