Effective convolution in machine learning environments

Abstract

A mechanism for facilitating intelligent convolution in machine learning environments is described. An apparatus as described in embodiments herein includes: one or more processors, including one or more graphics processors; and detection and selection logic for detecting and selecting an input image having a plurality of geometries associated with an object to which a neural network will be trained. The apparatus further includes filter generation and storage logic (“filter logic”) for generating weighted filters based on the plurality of geometries, wherein the filter logic is further configured to classify filters in a filter bank based on a common geometry among the plurality of geometries, and wherein the filter logic further stores the filter bank in bins based on the common geometry, wherein each bin corresponds to a geometry.

Description

Technical Field

[0001] The embodiments described herein generally relate to computers. More specifically, embodiments for facilitating efficient convolutions in machine learning environments are described. Technical Background

[0002] Neural network models can be implemented as a collection of nodes (neurons) connected in a non-cyclic graph. One type of neural network, the "feedforward network," receives input (a single vector) at the input layer of its nodes and maps the input to values ​​in the output layer of the nodes through a series of hidden layers. Each layer in the network consists of a set of neurons, where each neuron is fully connected to all neurons in the (multiple) adjacent layers, and neurons within a layer do not share any connections. If the network is modeling classification, each node in the output layer can represent one of the possible categories to which an entity belongs, and the value of each output node can represent the probability that the input entity belongs to that category. Convolutional Neural Networks (CNNs) are another type of neural network model that models feature detection by performing convolution operations on input data. For example, in image processing, a CNN can take raw image data as input and then output a set of feature maps that represent an abstraction of the raw data.

[0003] Neural networks are increasingly used in various applications; however, traditional neural network models require significant amounts of memory and computational power. In some cases, deriving a neural network model may involve multiple training iterations, or even consecutive training, before its performance meets specified requirements. (Reference) Figure 15-19 Further description of neural network modeling.

[0004] Current parallel graphics data processing includes systems and methods developed to perform specific operations on graphics data, such as linear interpolation, tessellation, rasterization, texture mapping, depth testing, etc. Furthermore, programmable general-purpose graphics processors can be configured to perform a wide variety of operations for processing vertex and fragment data, as well as for performing general-purpose parallel processing operations. For example, parallel graphics processors with a Single Instruction Multiple Thread (SIMT) architecture are designed to maximize the amount of parallel processing in the graphics pipeline. In a SIMT architecture, multiple sets of parallel threads attempt to execute program instructions together synchronously as often as possible to improve processing efficiency. A general overview of the software and hardware used for SIMT architectures can be found in Shane Cook's *CUDA Programming*, Chapter 3, pp. 37–51 (2013) and / or Nicholas Wilt's *CUDA Handbook, A Comprehensive Guide to GPU Programming*, Chapters 2.6.2–3.1.2 (June 2013). Attached Figure Description

[0005] In the accompanying drawings, embodiments are illustrated by way of example rather than limitation, wherein the same reference numerals refer to similar elements.

[0006] Figure 1 This is a block diagram of a processing system according to an embodiment.

[0007] Figure 2 This is a block diagram of an embodiment of a processor, which has one or more processor cores, an integrated memory controller, and an integrated graphics processor.

[0008] Figure 3 This is a block diagram of a graphics processor, which may be a discrete graphics processing unit or a graphics processor integrated with multiple processing cores.

[0009] Figure 4 This is a block diagram of a graphics processing engine for a graphics processor according to some embodiments.

[0010] Figure 5 This is a block diagram of the hardware logic of a graphics processor core according to some embodiments.

[0011] Figures 6A-6B The diagram illustrates thread execution logic including an array of processing elements employed in a graphics processor core, according to some embodiments.

[0012] Figure 7 This is a block diagram illustrating a graphics processor instruction format according to some embodiments.

[0013] Figure 8 This is a block diagram of another embodiment of a graphics processor.

[0014] Figure 9A This is a block diagram illustrating the graphics processor command format according to an embodiment.

[0015] Figure 9B This is a block diagram illustrating a sequence of graphics processor commands according to an embodiment.

[0016] Figure 10 An exemplary graphical software architecture for a data processing system is shown according to some embodiments.

[0017] Figure 11A This is a block diagram illustrating an IP core development system according to an embodiment that can be used to manufacture integrated circuits to perform operations.

[0018] Figure 11B A cross-sectional side view of an integrated circuit package assembly according to some embodiments is shown.

[0019] Figure 12 This is a block diagram illustrating an exemplary system-on-a-chip integrated circuit that can be fabricated using one or more IP cores according to an embodiment.

[0020] Figures 13A-13B This is a block diagram illustrating an exemplary graphics processor for use within a system-on-a-chip (SoC) according to embodiments described herein.

[0021] Figures 14A-14B Additional exemplary graphics processor logic according to embodiments described herein is shown.

[0022] Figure 15 It is a generalized graph of the machine learning software stack.

[0023] Figure 16A Various layers with a convolutional neural network (CNN) are shown.

[0024] Figure 16B This illustrates the computational stages within the convolutional layers of a CNN.

[0025] Figure 17 An exemplary recurrent neural network is shown.

[0026] Figure 18 The training and deployment of a deep neural network are illustrated.

[0027] Figure 19 This is a block diagram illustrating distributed learning.

[0028] Figure 20A computing device for a host-based intelligent deep learning convolution mechanism according to one embodiment is shown.

[0029] Figure 21 An intelligent deep learning convolution mechanism according to one embodiment is illustrated.

[0030] Figure 22A A method for facilitating the generation and storage of geometry-based filters for use in intelligent deep learning convolutions, according to one embodiment, is illustrated.

[0031] Figure 22B A geometry-based training method for filters and neural networks according to one embodiment is shown.

[0032] Figure 22C A method for facilitating the separation of convolutional filters for intelligent convolution in deep learning, according to one embodiment, is shown.

[0033] Figure 23A A sequence of transactions for geometry-based intelligent convolution in deep learning is shown according to one embodiment.

[0034] Figure 23B A sequence of transactions for separable convolutions for intelligent convolutions in deep learning, according to one embodiment, is shown.

[0035] Figure 24 A structural setup for performing separable convolutions is shown according to one embodiment.

[0036] Figure 25 This is a block diagram illustrating a computer system configured to implement one or more aspects of the embodiments described herein.

[0037] Figures 26A-26D A parallel processor component according to an embodiment is shown.

[0038] Figures 27A-27B This is a block diagram of a graphics multiprocessor according to an embodiment.

[0039] Figure 28A-28G An exemplary architecture is shown in which multiple graphics processing units (GPUs) are communicatively coupled to multiple multi-core processors.

[0040] Figure 29 A graphics processing pipeline according to an embodiment is shown. Detailed Implementation

[0041] Numerous specific details are set forth in the following description. However, many embodiments as described above may be practiced without these specific details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of this description.

[0042] This embodiment provides a novel technique for reducing the time spent by a neural network (NN) model to achieve convergence by providing class-specific weight fillers to the lower convolutional layers. Furthermore, regarding the complexity of the classes to be identified, the neural network can be retrained where the initial layer of two-dimensional (2D) spatial convolutions is modified into two separate one-dimensional (1D) convolutions. This novel technique reduces the complexity of the overall operation and improves overall speed without compromising or losing any accuracy.

[0043] As can be anticipated, throughout this document, terms such as “request,” “query,” “job,” “work,” “work item,” and “workload” can be used interchangeably. Similarly, “application” or “agent” can refer to or include applications that use free-rendering application programming interfaces (APIs) such as open graphics libraries. 11, The APIs provided by 12, etc., are computer programs, software applications, games, workstation applications, etc., where “dispatch” can be interchangeably referred to as “unit of work” or “drawing”, and similarly, “application” can be interchangeably referred to as “workflow” or simply “agent”. For example, a workload such as a 3D game workload can include and dispatch any number and type of “frames”, where each frame can represent an image (e.g., a sailboat, a face). Furthermore, each frame can include and provide any number and type of unit of work, where each unit of work can represent a portion of the image (e.g., a sailboat, a face) represented by its corresponding frame (e.g., the mast of a sailboat, the forehead of a face). However, for consistency, throughout this document, each item can be referred to by a single term (e.g., “dispatch”, “agent”, etc.).

[0044] In some embodiments, terms such as "display" and "display surface" may be used interchangeably to refer to the visible portion of a display device, while the remainder of the display device may be embedded in a computing device such as a smartphone or wearable device. It is foreseeable and should be noted that the embodiments are not limited to any particular computing device, software application, hardware component, display device, display screen or surface, protocol, standard, etc. For example, the embodiments can be applied to and used with any number and type of real-time applications on any number and type of computers, such as desktop computers, laptop computers, tablet computers, smartphones, head-mounted displays, and other wearable devices. Furthermore, for example, rendering high-performance scenarios using this novel technology can be used in a range of scenarios, from simple scenarios such as desktop compositing to complex scenarios such as 3D games and augmented reality applications.

[0045] It should be noted that throughout this document, terms or acronyms such as Convolutional Neural Network (CNN), CNN, Neural Network (NN), NN, Deep Neural Network (DNN), DNN, Recurrent Neural Network (RNN), RNN, etc., can be used interchangeably. Furthermore, throughout this document, terms such as "autonomous machine" or simply "machine," "autonomous vehicle" or simply "vehicle," "autonomous agent" or simply "agent," "autonomous device" or "computing device," "robot," etc., can be used interchangeably.

[0046] System Overview

[0047] Figure 1 This is a block diagram of a processing system 100 according to an embodiment. In various embodiments, system 100 includes one or more processors 102 and one or more graphics processors 108, and may be a single-processor desktop system, a multi-processor workstation system, or a server system having a large number of processors 102 or processor cores 107. In one embodiment, system 100 is a processing platform incorporated within a system-on-a-chip (SoC) integrated circuit for use in mobile devices, handheld devices, or embedded devices.

[0048] In one embodiment, system 100 may include or incorporate a server-based gaming platform, a game console, including a game and media console, a mobile game console, a handheld game console, or an online game console. In some embodiments, system 100 is a mobile phone, smartphone, tablet computing device, or mobile internet device. Processing system 100 may also include a wearable device (such as a smartwatch, smart glasses, augmented reality, or virtual reality device), coupled to or integrated into the wearable device. In some embodiments, processing system 100 is a television or set-top box device having one or more processors 102 and a graphical interface generated by one or more graphics processors 108.

[0049] In some embodiments, one or more processors 102 each include one or more processor cores 107 for processing instructions that, when executed, perform operations on the system and user software. In some embodiments, each of the one or more processor cores 107 is configured to process a particular instruction set 109. In some embodiments, the instruction set 109 may facilitate Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or computation via Very Long Instruction Word (VLIW). Multiple processor cores 107 may each process different instruction sets 109, which may include instructions for facilitating emulation of other instruction sets. Processor cores 107 may also include other processing devices, such as digital signal processors (DSPs).

[0050] In some embodiments, processor 102 includes cache memory 104. Depending on the architecture, processor 102 may have a single internal cache or multiple levels of internal caches. In some embodiments, cache memory is shared among components of processor 102. In some embodiments, processor 102 also uses an external cache (e.g., a Level 3 (L3) cache or a Last Level Cache (LLC)) (not shown), which can be shared among processor core 107 using known cache coherence techniques. Additionally, register file 106 is included in processor 102, which may include different types of registers for storing different types of data (e.g., integer registers, floating-point registers, status registers, and instruction pointer registers). Some registers may be general-purpose registers, while others may be specific to the design of processor 102.

[0051] In some embodiments, one or more processors 102 are coupled to one or more interface buses 110 for transmitting communication signals, such as address, data, or control signals, between the processors 102 and other components in the system 100. In one embodiment, the interface bus 110 may be a processor bus, such as a version of the Direct Media Interface (DMI) bus. However, the processor bus is not limited to the DMI bus and may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In one embodiment, the processor(s) 102 include an integrated memory controller 116 and a platform controller hub (PCH) 130. The memory controller 116 facilitates communication between memory devices and other components of the system 100, while the platform controller hub (PCH) 130 provides connectivity to I / O devices via a local I / O bus.

[0052] Memory device 120 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase-change memory device, or some other memory device with suitable performance for use as processing memory. In one embodiment, memory device 120 may operate as system memory of system 100 for storing data 122 and instructions 121 for use when the one or more processors 102 execute an application or process. Memory controller 116 is also coupled to an optional external graphics processor 112, which may communicate with the one or more graphics processors 108 in processor 102 to perform graphics and media operations. In some embodiments, display device 111 may be connected to processor(s) 102. Display device 111 may be one or more of the following: an internal display device, such as in a mobile electronic device or laptop device; or an external display device attached via a display interface (e.g., a display port, etc.). In one embodiment, display device 111 may be a head-mounted display (HMD), such as a stereoscopic display device used in virtual reality (VR) or augmented reality (AR) applications.

[0053] In some embodiments, the platform controller hub 130 enables peripheral devices to connect to the memory device 120 and the processor 102 via a high-speed I / O bus. I / O peripheral devices include, but are not limited to, an audio controller 146, a network controller 134, a firmware interface 128, a wireless transceiver 126, a touch sensor 125, and a data storage device 124 (e.g., a hard disk drive, flash memory, etc.). The data storage device 124 may be connected via a storage interface (e.g., SATA) or via a peripheral bus such as a peripheral component interconnect bus (e.g., PCI, PCI Express). The touch sensor 125 may include a touchscreen sensor, a pressure sensor, or a fingerprint sensor. The wireless transceiver 126 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver, such as a 3G, 4G, or LTE transceiver. The firmware interface 128 enables communication with system firmware and may be, for example, a Unified Extensible Firmware Interface (UEFI). The network controller 134 enables network connectivity to a wired network. In some embodiments, a high-performance network controller (not shown) is coupled to the interface bus 110. In one embodiment, the audio controller 146 is a multi-channel high-definition audio controller. In one embodiment, the system 100 includes an optional legacy I / O controller 140 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to the system. The platform controller hub 130 may also be connected to one or more Universal Serial Bus (USB) controllers 142 to connect input devices, such as a keyboard and mouse combination 143, a camera 144, or other USB input devices.

[0054] It will be appreciated that the illustrated system 100 is exemplary and not limiting, as other types of data processing systems configured differently may also be used. For example, instances of the memory controller 116 and platform controller hub 130 may be integrated into a discrete external graphics processor, such as external graphics processor 112. In one embodiment, the platform controller hub 130 and / or memory controller 160 may be external to the one or more processors 102. For example, system 100 may include an external memory controller 116 and a platform controller hub 130, which may be configured as a memory controller hub and a peripheral controller hub within a system chipset communicating with the processor(s) 102.

[0055] Figure 2 This is a block diagram of an embodiment of processor 200, which has one or more processor cores 202A to 202N, an integrated memory controller 214, and an integrated graphics processor 208. Figure 2Those elements having the same reference numerals (or names) as elements in any other figure herein may operate or function in any manner similar to, but not limited to, those described elsewhere herein. Processor 200 may include, and include, additional cores 202N, indicated by dashed boxes. Each processor core 202A to 202N includes one or more internal cache units 204A to 204N. In some embodiments, each processor core may also access one or more shared cache units 206.

[0056] Internal cache units 204A to 204N and shared cache unit 206 represent the cache memory hierarchy within processor 200. The cache memory hierarchy may include at least one level of instruction and data cache within each processor core and one or more levels of shared intermediate cache, such as Level 2 (L2), Level 3 (L3), Level 4 (L4), or other levels of cache, wherein the highest-level cache is classified as LLC before external memory. In some embodiments, cache coherence logic maintains coherence between each cache unit 206 and 204A to 204N.

[0057] In some embodiments, the processor 200 may further include a set of one or more bus controller units 216 and a system agent core 210. The one or more bus controller units 216 manage a set of peripheral buses, such as one or more PCI or PCI Fast buses. The system agent core 210 provides management functions for each processor component. In some embodiments, the system agent core 210 includes one or more integrated memory controllers 214 for managing access to various external memory devices (not shown).

[0058] In some embodiments, one or more of processor cores 202A to 202N include support for simultaneous multithreading. In this embodiment, system agent core 210 includes components for coordinating and operating cores 202A to 202N during multithreaded processing. Additionally, system agent core 210 may also include a power control unit (PCU) including logic and components for regulating the power states of processor cores 202A to 202N and a graphics processor 208.

[0059] In some embodiments, processor 200 further includes a graphics processor 208 for performing graphics processing operations. In some embodiments, graphics processor 208 is coupled to a shared cache unit 206 and a system proxy core 210, the system proxy core including one or more integrated memory controllers 214. In some embodiments, system proxy core 210 further includes a display controller 211 to drive graphics processor output to one or more coupled displays. In some embodiments, display controller 211 may also be a separate module coupled to the graphics processor via at least one interconnect, or it may be integrated within graphics processor 208.

[0060] In some embodiments, ring-based interconnect units 212 are used to couple internal components of processor 200. However, alternative interconnect units, such as point-to-point interconnects, switched interconnects, or other technologies, including those well known in the art, may be used. In some embodiments, graphics processor 208 is coupled to ring interconnect 212 via I / O link 213.

[0061] Exemplary I / O link 213 represents at least one of a variety of I / O interconnects, including packaged I / O interconnects that facilitate communication between various processor components and a high-performance embedded memory module 218 (such as an eDRAM module). In some embodiments, each of the processor cores 202A to 202N and the graphics processor 208 uses the embedded memory module 218 as a shared final-level cache.

[0062] In some embodiments, processor cores 202A to 202N are homogeneous cores executing the same instruction set architecture. In another embodiment, processor cores 202A to 202N are heterogeneous in terms of instruction set architecture (ISA), wherein one or more of processor cores 202A to 202N execute a first instruction set, while at least one of the other cores executes a subset of the first instruction set or a different instruction set. In one embodiment, processor cores 202A to 202N are homogeneous in terms of microarchitecture, wherein one or more cores with relatively high power consumption are coupled to one or more power cores with lower power consumption. Additionally, processor 200 can be implemented on one or more chips or implemented as a SoC integrated circuit having, among other components, the components shown.

[0063] Figure 3This is a block diagram of a graphics processing unit 300, which may be a discrete graphics processing unit or a graphics processing unit integrated with multiple processing cores. In some embodiments, the graphics processing unit communicates with memory via a mapped I / O interface to registers on the graphics processing unit and using commands placed in processor memory. In some embodiments, the graphics processing unit 300 includes a memory interface 314 for accessing memory. The memory interface 314 may be an interface to local memory, one or more internal caches, one or more shared external caches, and / or to system memory.

[0064] In some embodiments, the graphics processor 300 further includes a display controller 302 for driving display output data to a display device 320. The display controller 302 includes hardware for one or more overlapping planes of the display and a multi-layer video or user interface element. The display device 320 may be an internal or external display device. In one embodiment, the display device 320 is a head-mounted display device, such as a virtual reality (VR) display device or an augmented reality (AR) display device. In some embodiments, the graphics processor 300 includes a video codec engine 306 for encoding, decoding, or converting media codes to, from, or between one or more media encoding formats, including but not limited to: Moving Picture Experts Group (MPEG) formats (such as MPEG-2), Advanced Video Decoding (AVC) formats (such as H.264 / MPEG-4 AVC), and Society of Motion Picture & Television Engineers (SMPTE) 421M / VC-1, and Joint Group of Picture Experts Group (JPEG) formats (such as JPEG and Motion JPEG (MJPEG)).

[0065] In some embodiments, the graphics processor 300 includes a block image transfer (BLIT) engine 304 for performing two-dimensional (2D) rasterizer operations, including, for example, bit boundary block transfer. However, in one embodiment, 2D graphics operations are performed using one or more components of a graphics processing engine (GPE) 310. In some embodiments, the GPE 310 is a computational engine for performing graphics operations, including three-dimensional (3D) graphics operations and media operations.

[0066] In some embodiments, GPE 310 includes a 3D pipeline 312 for performing 3D operations, such as rendering 3D images and scenes using processing functions acting on 3D primitive shapes (e.g., rectangles, triangles, etc.). The 3D pipeline 312 includes programmable and fixed functional elements that perform various tasks within elements and / or generated execution threads of the 3D / media subsystem 315. While the 3D pipeline 312 can be used to perform media operations, embodiments of GPE 310 also include a media pipeline 316 specifically for performing media operations such as video post-processing and image enhancement.

[0067] In some embodiments, the media pipeline 316 includes fixed-function or programmable logic units to perform one or more specialized media operations, such as video decoding acceleration, video deinterleaving, and video encoding acceleration, in place of or on behalf of the video codec engine 306. In some embodiments, the media pipeline 316 further includes a thread generation unit to generate threads for execution on the 3D / media subsystem 315. The generated threads perform calculations on media operations on one or more graphics execution units included in the 3D / media subsystem 315.

[0068] In some embodiments, the 3D / media subsystem 315 includes logic for executing threads generated by the 3D pipeline 312 and the media pipeline 316. In one embodiment, the pipelines send thread execution requests to the 3D / media subsystem 315, the 3D / media subsystem including thread dispatch logic for arbitrating and dispatching requests to available thread execution resources. Execution resources include an array of graphics execution units for processing 3D and media threads. In some embodiments, the 3D / media subsystem 315 includes one or more internal caches for thread instructions and data. In some embodiments, the subsystem also includes shared memory (including registers and addressable memory) for sharing data between threads and for storing output data.

[0069] Graphics processing engine

[0070] Figure 4 This is a block diagram of a graphics processing engine 410 of a graphics processor according to some embodiments. In one embodiment, the graphics processing engine (GPE) 410 is... Figure 3 The image shows a version of GPE 310. Figure 4 Those elements having the same reference numerals (or names) as elements in any other figure herein may operate or function in any manner similar to, but not limited to, those described elsewhere herein. For example, shown Figure 3The 3D pipeline 312 and media pipeline 316 are included. The media pipeline 316 is optional in some embodiments of the GPE 410 and may not be explicitly included within the GPE 410. For example, and in at least one embodiment, a separate media and / or image processor is coupled to the GPE 410.

[0071] In some embodiments, GPE 410 is coupled to or includes command stream converter 403, which provides command streams to 3D pipeline 312 and / or media pipeline 316. In some embodiments, command stream converter 403 is coupled to memory, which may be system memory, or one or more of internal cache memory and shared cache memory. In some embodiments, command stream converter 403 receives commands from memory and sends these commands to 3D pipeline 312 and / or media pipeline 316. The commands are instructions obtained from a ring buffer storing instructions for 3D pipeline 312 and media pipeline 316. In one embodiment, the ring buffer may additionally include a batch command buffer storing multiple batches of commands. Commands for 3D pipeline 312 may also include references to data stored in memory, such as, but not limited to, vertex data and geometry data for 3D pipeline 312 and / or image data and memory objects for media pipeline 316. The 3D pipeline 312 and the media pipeline 316 process the commands and data by performing operations via logic within their respective pipelines or by dispatching one or more execution threads to the execution graphics core array 414. In one embodiment, the graphics core array 414 includes one or more graphics core blocks (e.g., multiple graphics cores 415A, multiple graphics cores 415B), each block including one or more graphics cores. Each graphics core includes a set of graphics execution resources, which includes: general-purpose execution logic and graphics-specific execution logic for performing graphics and computational operations; and fixed-function texture processing logic and / or machine learning and artificial intelligence acceleration logic.

[0072] In various embodiments, the 3D pipeline 312 includes fixed-function logic and programmable logic for processing one or more shader programs, such as vertex shaders, geometry shaders, pixel shaders, fragment shaders, compute shaders, or other shader programs, by processing instructions and dispatching execution threads to the graphics core array 414. The graphics core array 414 provides a unified block of execution resources for use in processing these shader programs. The multipurpose execution logic (e.g., execution units) within the graphics core(s)(s)415A to 414B of the graphics core array 414 includes support for various 3D API shader languages ​​and can execute multiple synchronous execution threads associated with multiple shaders.

[0073] In some embodiments, the graphics core array 414 further includes execution logic for performing media functions such as video and / or image processing. In one embodiment, in addition to graphics processing operations, the execution unit also includes general-purpose logic programmable to perform parallel general-purpose computing operations. The general-purpose logic can be... Figure 1 (Multiple) processor cores 107 or Figure 2 The general logic within the cores 202A to 202N performs processing operations in parallel or in combination.

[0074] Output data generated by threads executing on the graphics core array 414 can be output to memory in a uniform return buffer (URB) 418. URB 418 can store data from multiple threads. In some embodiments, URB 418 can be used to send data between different threads executing on the graphics core array 414. In some embodiments, URB 418 can also be used for synchronization between threads on the graphics core array and fixed-function logic within shared-function logic 420.

[0075] In some embodiments, the graphics core array 414 is scalable, such that the array includes a variable number of graphics cores, each having a variable number of execution units based on the target power and performance level of the GPE 410. In one embodiment, the execution resources are dynamically scalable, allowing them to be enabled or disabled as needed.

[0076] The graphics core array 414 is coupled to shared function logic 420, which includes multiple resources shared among the graphics cores in the graphics core array. The shared functions within the shared function logic 420 are hardware logic units that provide dedicated supplementary functions to the graphics core array 414. In various embodiments, the shared function logic 420 includes, but is not limited to, sampler 421, math 422, and inter-thread communication (ITC) 423 logic. Additionally, some embodiments implement one or more caches 425 within the shared function logic 420.

[0077] Shared functionality is implemented where the demand for a given dedicated function is insufficient to be contained within the graphics core array 414. Instead, a single instance of the dedicated function is implemented as a separate entity within shared function logic 420 and shared among execution resources within the graphics core array 414. The exact set of functions shared and included within the graphics core array 414 varies across embodiments. In some embodiments, a specific shared function widely used by the graphics core array 414 within shared function logic 420 may be included within shared function logic 416 within the graphics core array 414. In various embodiments, shared function logic 416 within the graphics core array 414 may include some or all of the logic within shared function logic 420. In one embodiment, all logic elements within shared function logic 420 may be repeated within shared function logic 416 of the graphics core array 414. In one embodiment, shared function logic 420 is executed to support shared function logic 416 within the graphics core array 414.

[0078] Figure 5 This is a block diagram of the hardware logic of a graphics processor core 500 according to some embodiments described herein. Figure 5 Those elements having the same reference numerals (or names) as elements in any other figure herein may operate or function in any manner similar to, but not limited to, those described elsewhere herein. In some embodiments, the illustrated graphics processor core 500 includes Figure 4 Within the graphics core array 414. A graphics processor core 500—sometimes referred to as a core slice—can be one or more graphics cores within a modular graphics processor. An example of a graphics processor core 500 is a graphics core slice, and, based on target power envelopes and performance envelopes, a graphics processor as described herein may include multiple graphics core slices. Each graphics core 500 may include a fixed-function block 530 coupled to multiple sub-cores 501A to 501F (also referred to as sub-slices), which include modular general-purpose logic blocks and fixed-function logic blocks.

[0079] In some embodiments, the fixed-function block 530 includes a geometry / fixed-function pipeline 536, which may be shared by all sub-cores of the graphics processor 500, for example, in low-performance and / or low-power graphics processor implementations. In various embodiments, the geometry / fixed-function pipeline 536 includes a 3D fixed-function pipeline (e.g., as in...). Figure 3 and Figure 4 The 3D pipeline (312), video front-end unit, thread deriver and thread dispatcher, and management such as Figure 4 The unified return buffer manager includes unified return buffers such as the unified return buffer 418.

[0080] In one embodiment, fixed function block 530 further includes a graphics SoC interface 537, a graphics microcontroller 538, and a media pipeline 539. The graphics SoC interface 537 provides an interface between the graphics core 500 and other processor cores within the system-on-a-chip integrated circuit. The graphics microcontroller 538 is a programmable subprocessor configurable to manage various functions of the graphics processor 500, including thread dispatch, scheduling, and pre-emption. The media pipeline 539 (e.g., Figure 3 and Figure 4 The media pipeline 316 includes logic for facilitating the decoding, encoding, preprocessing, and / or post-processing of multimedia data, including image and video data. The media pipeline 539 performs media operations via requests for computation or sampling logic within subcores 501 to 501F.

[0081] In one embodiment, SoC interface 537 enables graphics core 500 to communicate with a general-purpose application processor core (e.g., CPU) and / or other components within the SoC, including memory-level architecture elements such as shared final-level cache memory, system RAM, and / or embedded on-chip or package-based DRAM. SoC interface 537 may also enable communication with fixed-function devices within the SoC, such as camera imaging pipelines, and enable the use and / or implementation of global memory atoms that can be shared between graphics core 500 and the CPU within the SoC. SoC interface 537 may also implement power management control for graphics core 500 and enable interfacing between the clock domain of graphics core 500 and other clock domains within the SoC. In one embodiment, SoC interface 537 enables the receipt of command buffers from a command stream converter and a global thread dispatcher configured to provide commands and instructions to each of one or more graphics cores within the graphics processor. When a media operation is about to be executed, these commands and instructions can be dispatched to the media pipeline 539, or when a graphics processing operation is about to be executed, these commands and instructions can be dispatched to the geometry and fixed-function pipelines (e.g., geometry and fixed-function pipeline 536, geometry and fixed-function pipeline 514).

[0082] The graphics microcontroller 538 can be configured to perform various scheduling and management tasks for the graphics core 500. In one embodiment, the graphics microcontroller 538 can perform graphics and / or computational workload scheduling for the various parallel graphics engines within the execution unit (EU) arrays 502A to 502F and 504A to 504F of the sub-cores 501A to 501F. In this scheduling model, host software executing on the CPU core of the SoC including the graphics core 500 can submit workloads via one of a plurality of graphics processor doorbells, which invokes scheduling operations for the appropriate graphics engine. The scheduling operations include: determining which workload to run next, submitting the workload to the command stream converter, preempting existing workloads running on the engine, monitoring the progress of the workload, and notifying the host software when the workload is complete. In one embodiment, the graphics microcontroller 538 may also facilitate a low-power or idle state of the graphics core 500, thereby providing the graphics core 500 with the ability to save and restore registers within the graphics core 500 across low-power state transitions, independent of the operating system and / or the graphics driver software on the system.

[0083] The graphics core 500 may have more or fewer sub-cores 501A to 501F shown, up to N modular sub-cores. For each group of N sub-cores, the graphics core 500 may also include shared function logic 510, shared memory and / or cache memory 512, geometry / fixed function pipeline 514, and additional fixed function logic 516 for accelerating various graphics and computational processing operations. Figure 4 The shared functional logic 420 is associated with logic units (e.g., sampler logic, mathematical logic, and / or inter-thread communication logic). Shared memory and / or cache memory 512 can be the final-level cache for the set of N sub-cores 501A to 501F within the graphics core 500, and can also act as shared memory accessible by multiple sub-cores. A geometry / fixed-function pipeline 514 can be included within the fixed-function block 530 instead of the geometry / fixed-function pipeline 536, and can include the same or similar logic units.

[0084] In one embodiment, the graphics core 500 includes additional fixed-function logic 516, which may include various fixed-function acceleration logics for use by the graphics core 500. In one embodiment, the additional fixed-function logic 516 includes an additional geometry pipeline for use in position-only shading. In position-only shading, there are two geometry pipelines: a full geometry pipeline within geometry / fixed-function pipelines 516 and 536; and a picking pipeline, which is an additional geometry pipeline that may be included within the additional fixed-function logic 516. In one embodiment, the picking pipeline is a simplified version of the full geometry pipeline. The full pipeline and the picking pipeline can execute different instances of the same application, each with a separate context. Position-only shading can hide longer picking runs of discarded triangles, thereby enabling earlier shading completion in some instances. For example, and in one embodiment, the picking pipeline logic within the attached fixed-function logic 516 can execute the position shader in parallel with the main application and typically generates key results faster than a full pipeline, because a full pipeline only extracts and shades the position attributes of vertices without performing rasterization and rendering of pixels to the frame buffer. The picking pipeline can use the generated key results to compute visibility information for all triangles, regardless of whether those triangles were picked. A full pipeline (which may be referred to as the replay pipeline in this example) can consume visibility information to skip picked triangles and shade only the visible triangles that are ultimately passed to the rasterization stage.

[0085] In one embodiment, the additional fixed-function logic 516 may also include machine learning acceleration logic, such as fixed-function matrix multiplication logic, for implementations including machine learning training or inference.

[0086] Each graphics subcore 501A to 501F includes a set of execution resources that can be used to perform graphics operations, media operations, and computational operations in response to requests from the graphics pipeline, media pipeline, or shader program. The graphics subcores 501A to 501F include: multiple EU arrays 502A to 502F, 504A to 504F; thread dispatch and inter-thread communication (TD / IC) logic 503A to 503F; 3D (e.g., texture) samplers 505A to 505F; media samplers 506A to 506F; shader processors 507A to 507F; and shared local memory (SLM) 508A to 508F. EU arrays 502A to 502F and 504A to 504F each include multiple execution units, which are general-purpose graphics processing units capable of performing floating-point and integer / fixed-point logic operations to serve graphics operations, media operations, or computational operations, including graphics programs, media programs, or computational shader programs. TD / IC logic 503A to 503F performs local thread dispatch and thread control operations for execution units within the subcore and facilitates communication between threads executing on the execution units of the subcore. 3D samplers 505A to 505F can read textures or other 3D graphics-related data into memory. The 3D samplers can read texture data in different ways based on the configured sample state and the texture format associated with a given texture. Media samplers 506A to 506F can perform similar read operations based on the type and format associated with media data. In one embodiment, each graphics subcore 501A to 501F may alternately include unified 3D and media samplers. Threads executing on execution units within each of subcores 501A to 501F can utilize shared local memory 508A to 508F within each subcore, so that threads executing within a thread group can use a common on-chip memory pool for execution.

[0087] Execution unit

[0088] Figures 6A to 6B Thread execution logic 600, including an array of processing elements employed in a graphics processor core, is illustrated according to embodiments described herein. Figures 6A to 6B Those elements having the same reference numerals (or names) as those in any other figure herein may operate or function in any manner similar to, but not limited to, those described elsewhere herein. Figure 6A An overview of thread execution logic 600 is shown, which may include what is shown as having Figure 5 Variants of the hardware logic for each of the 501A to 501F sub-cores. Figure 6B Exemplary internal details of the execution unit are shown.

[0089] like Figure 6A As shown, in some embodiments, thread execution logic 600 includes a shader processor 602, a thread dispatcher 604, an instruction cache 606, a scalable execution unit array including multiple execution units 608A to 608N, a sampler 610, a data cache 612, and a data port 614. In one embodiment, the scalable execution unit array can be dynamically scaled by enabling or disabling one or more execution units (e.g., any one of execution units 608A, 608B, 608C, 608D, up to 608N-1 and 608N) based on workload computational needs. In one embodiment, the included components are interconnected via an interconnect structure linking to each component. In some embodiments, thread execution logic 600 includes one or more connections to memory (such as system memory or cache memory) via one or more of the instruction cache 606, data port 614, sampler 610, and execution unit arrays 608A to 608N. In some embodiments, each execution unit (e.g., 608A) is an independent programmable general-purpose computing unit capable of executing multiple synchronous hardware threads while processing multiple data elements in parallel for each thread. In various embodiments, the array of execution units 608A to 608N is scalable to include any number of individual execution units.

[0090] In some embodiments, execution units 608A to 608N are primarily used to execute shader programs. Shader processor 602 can handle various shader programs and dispatch execution threads associated with the shader programs via thread dispatcher 604. In one embodiment, the thread dispatcher includes logic for arbitrating thread initiation requests from the graphics and media pipeline and instantiating the requested threads on one or more execution units 608A to 608N. For example, a geometry pipeline can dispatch vertex processing, tessellation, or geometry processing threads to thread execution logic for processing. In some embodiments, thread dispatcher 604 can also handle runtime thread generation requests from executing shader programs.

[0091] In some embodiments, execution units 608A to 608N support instruction sets that include native support for many standard 3D graphics shader instructions, enabling minimal conversion to execute shader programs from graphics libraries (e.g., Direct3D and OpenGL). These execution units support vertex and geometry processing (e.g., vertex programs, geometry programs, vertex shaders), pixel processing (e.g., pixel shaders, fragment shaders), and general-purpose processing (e.g., computation and media shaders). Each of the execution units 608A to 608N is capable of executing multiple-issue single-instruction multiple-data (SIMD), and multithreaded operation enables an efficient execution environment in the face of high-latency memory accesses. Each hardware thread within each execution unit has a dedicated high-bandwidth register file and associated independent thread state. For pipelines with integer, single-precision floating-point and double-precision floating-point operations, SIMD branching capabilities, logical operations, transcendental operations, and other hybrid operations, execution is multiple-issue per clock cycle. While waiting for data from memory or a shared function, dependency logic within execution units 608A to 608N causes the waiting thread to sleep until the requested data has been returned. While the waiting thread is sleeping, hardware resources may be dedicated to processing other threads. For example, during the latency associated with vertex shader operations, the execution unit may perform operations on a pixel shader, a fragment shader, or another type of shader program that includes different vertex shaders.

[0092] Each execution unit in the execution units 608A to 608N operates on an array of data elements. The number of data elements is the "execution size," or the number of instruction channels. An execution channel is a logical unit that performs data element access, masking, and flow control within instructions. The number of channels may be independent of the number of physical arithmetic logic units (ALUs) or floating-point units (FPUs) for a particular graphics processor. In some embodiments, the execution units 608A to 608N support both integer and floating-point data types.

[0093] The execution unit instruction set includes SIMD instructions. Various data elements can be stored in registers as compressed data types, and the execution unit will process these elements based on their data size. For example, when operating on a 256-bit wide vector, the 256-bit vector is stored in registers, and the execution unit operates on the vector as four individual 64-bit compressed data elements (four times the word length (QW) size), eight individual 32-bit compressed data elements (double the word length (DW) size), sixteen individual 16-bit compressed data elements (word length (W) size), or thirty-two individual 8-bit data elements (byte (B) size). However, different vector widths and register sizes are possible.

[0094] In one embodiment, one or more execution units can be combined into fused execution units 609A to 609N, which have common thread control logic (607A to 607N) for fused EUs. Multiple EUs can be fused into a group of EUs. Each EU in the fused EU group can be configured to execute a separate SIMD hardware thread. The number of EUs in the fused EU group can vary depending on the embodiment. Additionally, different SIMD widths, including but not limited to SIMD8, SIMD16, and SIMD32, can be executed for each EU. Each fused graphics execution unit 609A to 609N includes at least two execution units. For example, fused execution unit 609A includes a first EU 608A, a second EU 608B, and common thread control logic 607A for the first EU 608A and the second EU 608B. Thread control logic 607A controls the threads executing on the fused graphics execution unit 609A, thereby allowing each EU within the fused execution units 609A to 609N to execute using a common instruction pointer register.

[0095] One or more internal instruction caches (e.g., 606) are included in the thread execution logic 600 to cache thread instructions of the execution unit. In some embodiments, one or more data caches (e.g., 612) are included for caching thread data during thread execution. In some embodiments, sampler 610 is included for providing texture sampling for 3D operations and media sampling for media operations. In some embodiments, sampler 610 includes dedicated texture or media sampling functions to process texture or media data during the sampling process before providing sampled data to the execution unit.

[0096] During execution, the graphics and media pipeline sends thread initiation requests to thread execution logic 600 via thread generation and dispatch logic. Once a set of geometric objects has been processed and rasterized into pixel data, pixel processor logic within shader processor 602 (e.g., pixel shader logic, fragment shader logic, etc.) is invoked to further compute output information and write the results to output surfaces (e.g., color buffers, depth buffers, stencil buffers, etc.). In some embodiments, the pixel shader or fragment shader computes values ​​for vertex attributes interpolated across the rasterized object. In some embodiments, the pixel processor logic within shader processor 602 then executes a pixel or fragment shader program provided by an application programming interface (API). To execute the shader program, shader processor 602 dispatches threads to execution units (e.g., 608A) via thread dispatcher 604. In some embodiments, shader processor 602 uses texture sampling logic in sampler 610 to access texture data in a texture map stored in memory. Arithmetic operations are performed on the texture data and the input geometry data to calculate the pixel color data of each geometric fragment, or to discard one or more pixels without further processing.

[0097] In some embodiments, data port 614 provides a memory access mechanism for thread execution logic 600 to output processed data to memory for further processing on the graphics processor output pipeline. In some embodiments, data port 614 includes or is coupled to one or more cache memories (e.g., data cache 612) to cache data via the data port for memory access.

[0098] like Figure 6B As shown, the graphics execution unit 608 may include an instruction fetch unit 637, a general-purpose register file array (GRF) 624, an architecture register file array (ARF) 626, a thread arbiter 622, a send unit 630, a branch unit 632, a set of SIMD floating-point units (FPUs) 634, and a set of dedicated integer SIMD ALUs 635 in one embodiment. The GRF 624 and ARF 626 include the set of general-purpose register files and architecture register files associated with each synchronized hardware thread that may be active in the graphics execution unit 608. In one embodiment, per-thread architecture state is maintained in the ARF 626, while data used during thread execution is stored in the GRF 624. The execution state of each thread, including the instruction pointer of each thread, may be maintained in thread-specific registers in the ARF 626.

[0099] In one embodiment, the graphics execution unit 608 has an architecture that is a combination of synchronous multithreading (SMT) and fine-grained interleaved multithreading (IMT). The architecture has a modular configuration that can be fine-tuned at design time based on the target number of synchronous threads and the target number of registers per execution unit, in which execution unit resources are partitioned across logic used to execute multiple synchronous threads.

[0100] In one embodiment, the graphics execution unit 608 can issue multiple instructions, which can each be different instructions. The thread arbiter 622 of the graphics execution unit thread 608 can dispatch instructions to one of the following for execution: sending unit 630, branching unit 642, or (multiple) SIMD FPUs 634. Each execution thread can access 128 general-purpose registers within the GRF 624, where each register can store 32 bytes accessible as a SIMD 8-element vector with 32-bit data elements. In one embodiment, each execution unit thread accesses 4 kilobytes within the GRF 624, but the embodiment is not limited to this, and more or fewer register resources may be provided in other embodiments. In one embodiment, up to seven threads can execute synchronously, but the number of threads per execution unit may also vary depending on the embodiment. In an embodiment where seven threads can access 4 kilobytes, the GRF 624 can store a total of 28 kilobytes. Flexible addressing modes allow multiple registers to be addressed simultaneously, thereby efficiently constructing wider registers or representing straddle rectangular block data structures.

[0101] In one embodiment, memory operations, sampler operations, and other long-latency system communications are dispatched via a "send" instruction executed by message sending unit 630. In one embodiment, branch instructions are dispatched to dedicated branch unit 632 to facilitate SIMD divergence and eventual convergence.

[0102] In one embodiment, the graphics execution unit 608 includes one or more SIMD floating-point units (FPUs) 634 for performing floating-point operations. In one embodiment, the FPU(s) 634 also support integer computation. In one embodiment, the FPU(s) 634 can perform up to M 32-bit floating-point (or integer) operations in SIMD, or up to 2M 16-bit integer or 16-bit floating-point operations in SIMD. In one embodiment, at least one of the FPUs provides extended mathematical capabilities that support high throughput beyond mathematical functions and double-precision 64-bit floating-point. In some embodiments, a set of 8-bit integer SIMD ALUs 635 also represents and can be specifically optimized to perform operations associated with machine learning computations.

[0103] In one embodiment, an array of multiple instances of the graphics execution unit 608 can be instantiated when graphics subcores are grouped (e.g., sub-slices). For scalability, the product architecture can select the exact number of execution units per subcore group. In one embodiment, the execution unit 608 can execute instructions across multiple execution channels. In a further embodiment, each thread executed on the graphics execution unit 608 is executed on a different channel.

[0104] Figure 7 This is a block diagram illustrating a graphics processor instruction format 700 according to some embodiments. In one or more embodiments, the graphics processor execution unit supports an instruction set having multiple instruction formats. Solid lines represent components typically included in the execution unit instructions, while dashed lines represent optional components or components included only in subsets of the instructions. In some embodiments, the instruction format 700 described and illustrated are macro instructions, as they are instructions supplied to the execution unit, as opposed to micro-operations generated from instruction decoding (once the instruction is processed).

[0105] In some embodiments, the graphics processor execution unit natively supports instructions using a 128-bit instruction format 710. A 64-bit compact instruction format 730 can be used for some instructions based on the selected instruction, multiple instruction options, and the number of operands. The native 128-bit instruction format 710 provides access to all instruction options, while some options and operations are restricted to the 64-bit format 730. The native instructions available in the 64-bit format 730 vary depending on the embodiment. In some embodiments, instructions are partially compressed using a set of index values ​​in an index field 713. The execution unit hardware references a set of compression tables based on the index values ​​and uses the output of the compression tables to reconstruct the native instructions using the 128-bit instruction format 710.

[0106] For each format, the instruction opcode 712 defines the operation to be performed by the execution unit. The execution unit executes each instruction in parallel across multiple data elements of each operand. For example, in response to an add instruction, the execution unit performs a synchronous add operation across each color channel, which represents a texture element or a picture element. By default, the execution unit executes each instruction across all data channels of the operand. In some embodiments, the instruction control field 714 enables control over certain execution options, such as channel selection (e.g., prediction) and data channel ordering (e.g., blending). For instructions using the 128-bit instruction format 710, the execution size field 716 limits the number of data channels that will be executed in parallel. In some embodiments, the execution size field 716 is not available for the 64-bit compact instruction format 730.

[0107] Some execution unit instructions have up to three operands, including two source operands (src0 720, src1 722) and a destination 718. In some embodiments, the execution unit supports dual-destination instructions, where one of these destinations is implicit. Data manipulation instructions may have a third source operand (e.g., SRC2 724), where the instruction opcode 712 determines the number of source operands. The last source operand of the instruction may be an on-the-fly (e.g., hard-coded) value passed using the instruction.

[0108] In some embodiments, the 128-bit instruction format 710 includes an access / address mode field 726, which specifies, for example, whether direct register addressing mode or indirect register addressing mode is used. When direct register addressing mode is used, the register addresses of one or more operands are provided directly by bits in the instruction.

[0109] In some embodiments, the 128-bit instruction format 710 includes an access / address mode field 726 that specifies the address mode and / or access mode of the instruction. In one embodiment, the access mode is used to define the data access alignment for the instruction. Some embodiments support access modes including a 16-byte aligned access mode and a 1-byte aligned access mode, wherein the byte alignment of the access mode determines the access alignment of the instruction operands. For example, in a first mode, the instruction can use byte-aligned addressing for both source and destination operands, and in a second mode, the instruction can use 16-byte aligned addressing for both source and destination operands.

[0110] In one embodiment, the address mode portion of the access / address mode field 726 determines whether the instruction uses direct or indirect addressing. When using direct register addressing mode, bits in the instruction directly provide the register addresses of one or more operands. When using indirect register addressing mode, the register addresses of one or more operands can be calculated based on the address register value and the address immediate number field in the instruction.

[0111] In some embodiments, instructions are grouped based on the 712-bit opcode field to simplify opcode decoding 740. For an 8-bit opcode, bits 4, 5, and 6 allow the execution unit to determine the type of opcode. The precise opcode grouping shown is merely exemplary. In some embodiments, the move and logic opcode group 742 includes data move and logic instructions (e.g., move (mov), compare (cmp)). In some embodiments, the move and logic group 742 shares five most significant bits (MSB), where move (mov) instructions are in the form of 0000xxxxb, and logic instructions are in the form of 0001xxxxb. The flow control instruction group 744 (e.g., call, jump (jmp)) includes instructions in the form of 0010xxxxb (e.g., 0x20). The promiscuous instruction group 746 includes a mixture of instructions, including synchronous instructions (e.g., wait, send) in the form of 0011xxxxb (e.g., 0x30). Parallel math instruction set 748 includes component-based arithmetic instructions (e.g., add, multiply) in the form 0100xxxxb (e.g., 0x40). Parallel math set 748 performs arithmetic operations in parallel across data channels. Vector math set 750 includes arithmetic instructions (e.g., dp4) in the form 0101xxxxb (e.g., 0x50). Vector math set performs arithmetic operations on vector operands, such as dot product.

[0112] Graphics Pipeline

[0113] Figure 8 This is a block diagram of another embodiment of the graphics processor 800. Figure 8 Those elements having the same reference numerals (or names) as those in any other figure herein may operate or function in any manner similar to, but not limited to, those described elsewhere herein.

[0114] In some embodiments, the graphics processor 800 includes a geometry pipeline 820, a media pipeline 830, a display engine 840, thread execution logic 850, and a rendering output pipeline 870. In some embodiments, the graphics processor 800 is a graphics processor within a multi-core processing system including one or more general-purpose processing cores. The graphics processor is controlled by register writes to one or more control registers (not shown) or by commands issued to the graphics processor 800 via a ring interconnect 802. In some embodiments, the ring interconnect 802 couples the graphics processor 800 to other processing components, such as other graphics processors or general-purpose processors. Commands from the ring interconnect 802 are interpreted by a command stream converter 803, which supplies instructions to individual components of the geometry pipeline 820 or the media pipeline 830.

[0115] In some embodiments, a command stream converter 803 directs the operation of a vertex acquirer 805, which reads vertex data from memory and executes vertex processing commands provided by the command stream converter 803. In some embodiments, the vertex acquirer 805 provides vertex data to a vertex shader 807, which performs coordinate space transformation and lighting operations on each vertex. In some embodiments, the vertex acquirer 805 and the vertex shader 807 execute vertex processing instructions by dispatching execution threads to execution units 852A to 852B via a thread dispatcher 831.

[0116] In some embodiments, execution units 852A to 852B are vector processor arrays having an instruction set for performing graphics and media operations. In some embodiments, execution units 852A to 852B have an attached L1 cache 851, which is dedicated to each array or shared between arrays. The cache may be configured as a data cache, an instruction cache, or a single cache partitioned to contain data and instructions in different partitions.

[0117] In some embodiments, the geometry pipeline 820 includes a tessellation component for performing hardware-accelerated tessellation of 3D objects. In some embodiments, a programmable shell shader 811 configures the tessellation operation. A programmable domain shader 817 provides back-end evaluation of the tessellation output. A tessellation unit 813 operates in the direction of the shell shader 811 and includes dedicated logic for generating a detailed set of geometric objects based on a rough geometry model that is provided as input to the geometry pipeline 820. In some embodiments, if tessellation is not used, the tessellation components (e.g., shell shader 811, tessellation unit 813, domain shader 817) can be bypassed.

[0118] In some embodiments, the complete geometry object may be processed by the geometry shader 819 via one or more threads dispatched to the execution units 852A to 852B, or it may proceed directly to the clipper 829. In some embodiments, the geometry shader operates on the entire geometry object (rather than vertices or vertex patches such as those in previous stages of the graphics pipeline). If tessellation is disabled, the geometry shader 819 receives input from the vertex shader 807. In some embodiments, the geometry shader 819 may be programmed by a geometry shader program to perform geometric tessellation when the tessellation unit is disabled.

[0119] Prior to rasterization, clipper 829 processes vertex data. Clipper 829 can be a fixed-function clipper or a programmable clipper with clipping and geometry shader capabilities. In some embodiments, the rasterizer and depth testing unit 873 in the render output pipeline 870 dispatch pixel shaders to convert geometry objects into a per-pixel representation. In some embodiments, pixel shader logic is included in thread execution logic 850. In some embodiments, the application can bypass the rasterizer and depth testing unit 873 and access the unrasterized vertex data via outgoing unit 823.

[0120] The graphics processor 800 has an interconnect bus, interconnect structure, or some other interconnect mechanism that allows data and messages to be transferred among the main components of the graphics processor. In some embodiments, execution units 852A to 852B and associated logic units (e.g., L1 cache 851, sampler 854, texture cache 858, etc.) are interconnected via data port 856 to perform memory accesses and communicate with the processor's rendering output pipeline components. In some embodiments, sampler 854, caches 851, 858, and execution units 852A to 852B each have a separate memory access path. In one embodiment, texture cache 858 may also be configured as a sampler cache.

[0121] In some embodiments, the rendering output pipeline 870 includes a rasterizer and a depth testing unit 873 that converts vertex-based objects into associated pixel-based representations. In some embodiments, the rasterizer logic includes a windower / mask unit for performing fixed-function triangle and line rasterization. Associated rendering cache 878 and depth cache 879 are also available in some embodiments. Pixel manipulation unit 877 performs pixel-based operations on the data; however, in some instances, pixel operations associated with 2D operations (e.g., using mixed bit-block image passing) are performed by the 2D engine 841, or alternatively by the display controller 843 using an overlay display plane at display time. In some embodiments, a shared L3 cache 875 is available for all graphics components, allowing data to be shared without using main system memory.

[0122] In some embodiments, the graphics processor media pipeline 830 includes a media engine 837 and a video front-end 834. In some embodiments, the video front-end 834 receives pipeline commands from a command stream converter 803. In some embodiments, the media pipeline 830 includes a separate command stream converter. In some embodiments, the video front-end 834 processes media commands before sending them to the media engine 837. In some embodiments, the media engine 837 includes a thread generation function for generating threads for dispatch to thread execution logic 850 via a thread dispatcher 831.

[0123] In some embodiments, the graphics processor 800 includes a display engine 840. In some embodiments, the display engine 840 is external to the processor 800 and coupled to the graphics processor via a ring interconnect 802, or some other interconnect bus or mechanism. In some embodiments, the display engine 840 includes a 2D engine 841 and a display controller 843. In some embodiments, the display engine 840 includes dedicated logic capable of operating independently of the 3D pipeline. In some embodiments, the display controller 843 is coupled to a display device (not shown), which may be a system-integrated display device (such as in a laptop computer) or an external display device attached via a display device connector.

[0124] In some embodiments, the geometry pipeline 820 and media pipeline 830 may be configured to perform operations based on multiple graphics and media programming interfaces and are not specific to any one application programming interface (API). In some embodiments, the graphics processor's driver software translates API schedules specific to a particular graphics or media library into commands that can be processed by the graphics processor. In some embodiments, support is provided for all Open Graphics Library (OpenGL), Open Computing Language (OpenCL), and / or Vulkan graphics and computing APIs from the Khronos Group. In some embodiments, support may also be provided for Microsoft's Direct3D library. In some embodiments, combinations of these libraries may be supported. Support may also be provided for the open-source computer vision library (OpenCV). Future APIs with compatible 3D pipelines will also be supported if a pipeline mapping from future APIs to the graphics processor's pipeline can be made.

[0125] Graphical Pipeline Programming

[0126] Figure 9A This is a block diagram illustrating a graphics processor command format 900 according to some embodiments. Figure 9B This is a block diagram illustrating a graphics processor command sequence 910 according to an embodiment. Figure 9ASolid lines in the diagram represent components that are typically included in the drawing command, while dashed lines represent components that are optional or included only in a subset of the drawing command. Figure 9A An exemplary graphics processor command format 900 includes data fields for identifying the client 902, a command operation code (opcode) 904, and data 906 for the command. Some commands also include a sub-opcode 905 and a command size 908.

[0127] In some embodiments, client 902 specifies a client unit of a graphics device that processes command data. In some embodiments, a graphics processor command parser examines the client field of each command to adjust further processing of the command and route command data to the appropriate client unit. In some embodiments, the graphics processor client unit includes a memory interface unit, a rendering unit, a 2D unit, a 3D unit, and a media unit. Each client unit has a corresponding processing pipeline for processing commands. Once a command is received by a client unit, the client unit reads opcode 904 and sub-opcode 905 (if present) to determine the operation to be performed. The client unit uses information within data field 906 to execute the command. For some commands, an explicit command size 908 is expected to specify the size of the command. In some embodiments, the command parser automatically determines the size of at least some commands in the command based on the command opcode. In some embodiments, commands are aligned via multiples of double word length.

[0128] Figure 9B The flowchart illustrates an exemplary graphics processor command sequence 910. In some embodiments, software or firmware of a data processing system characterized by an embodiment of a graphics processor uses a version of the illustrated command sequence to initiate, execute, and terminate a set of graphics operations. Sample command sequences are shown and described for illustrative purposes only, and embodiments are not limited to these specific commands or this command sequence. Moreover, the commands may be issued as a batch of commands in a command sequence, such that the graphics processor will process the command sequence in a manner that is at least partially simultaneous.

[0129] In some embodiments, the graphics processor command sequence 910 may begin with a pipeline dump clearing command 912 to cause any active graphics pipeline to complete its current pending commands. In some embodiments, the 3D pipeline 922 and the media pipeline 924 do not operate simultaneously. Pipeline dump clearing is performed to cause the active graphics pipeline to complete any pending commands. In response to pipeline dump clearing, the command parser for the graphics processor will stop command processing until the active rendering engine completes its pending operations and invalidates the associated read cache. Optionally, any data marked as 'dirty' in the render cache may be dumped and cleared into memory. In some embodiments, pipeline dump clearing command 912 may be used for pipeline synchronization or before placing the graphics processor into a low-power state.

[0130] In some embodiments, a pipeline selection command 913 is used when a sequence of commands requires the graphics processor to explicitly switch between pipelines. In some embodiments, only one pipeline selection command 913 is required in an execution context before a pipeline command is issued, unless the context requires issuing commands for two pipelines. In some embodiments, a pipeline dump clearing command 912 is required exactly before the pipeline switch via pipeline selection command 913.

[0131] In some embodiments, pipeline control command 914 configures a graphics pipeline for operation and programs the 3D pipeline 922 and the media pipeline 924. In some embodiments, pipeline control command 914 configures the pipeline state of an active pipeline. In one embodiment, pipeline control command 914 is used for pipeline synchronization and for clearing data from one or more cache memories within an active pipeline before processing a batch of commands.

[0132] In some embodiments, the return buffer state command 916 is used to configure a set of return buffers for corresponding pipelined write data. Some pipelined operations require allocating, selecting, or configuring one or more return buffers, in which intermediate data is written during processing. In some embodiments, the graphics processor also uses one or more return buffers to store output data and perform cross-thread communication. In some embodiments, the return buffer state 916 includes selecting the size and number of return buffers for a set of pipelined operations.

[0133] The remaining commands in the command sequence vary based on the active pipeline used for the operation. Based on pipeline determination 920, the command sequence is tailored for either the 3D pipeline 922 starting at 3D pipeline state 930, or the media pipeline 924 starting at media pipeline state 940.

[0134] Commands for configuring 3D pipeline states 930 include 3D state setting commands for vertex buffer states, vertex element states, constant color states, depth buffer states, and other state variables to be configured before processing 3D primitive commands. The values ​​of these commands are determined at least in part based on the specific 3D API in use. In some embodiments, the 3D pipeline state 930 commands can also selectively disable or bypass specific pipeline components (if those components will not be used).

[0135] In some embodiments, the 3D primitive 932 command is used to submit 3D primitives to be processed by the 3D pipeline. The command and associated parameters passed to the graphics processor via the 3D primitive 932 command are forwarded to the vertex acquisition function in the graphics pipeline. The vertex acquisition function uses the 3D primitive 932 command data to generate multiple vertex data structures. These vertex data structures are stored in one or more return buffers. In some embodiments, the 3D primitive 932 command is used to perform vertex operations on the 3D primitives via a vertex shader. To process the vertex shader, the 3D pipeline 922 dispatches shader execution threads to the graphics processor execution unit.

[0136] In some embodiments, the 3D pipeline 922 is triggered by executing command 934 or an event. In some embodiments, register writing triggers command execution. In some embodiments, execution is triggered via a 'go' or 'kick' command in a command sequence. In one embodiment, pipeline synchronization commands are used to trigger command execution so that the command sequence is cleared via a graphics pipeline dump. The 3D pipeline performs geometry processing on 3D primitives. Once the operation is complete, the resulting geometry is rasterized, and the pixel engine shades the resulting pixels. Additional commands for controlling pixel shading and pixel backend operations may also be included for these operations.

[0137] In some embodiments, when performing media operations, a sequence of graphics processor commands 910 follows the media pipeline 924 path. Generally, the specific purpose and manner of programming the media pipeline 924 depends on the media or computational operation to be performed. During media decoding, specific media decoding operations can be offloaded to the media pipeline. In some embodiments, the media pipeline can also be bypassed, and media decoding can be performed wholly or partially using resources provided by one or more general-purpose processing cores. In one embodiment, the media pipeline also includes elements for general-purpose graphics processing unit (GPGPU) operations, wherein the graphics processor is used to perform SIMD vector operations using computation shader programs that are not explicitly associated with rendering graphics primitives.

[0138] In some embodiments, the media pipeline 924 is configured in a manner similar to that of the 3D pipeline 922. A set of commands for configuring media pipeline states 940 is dispatched or placed in a command queue before the media object commands 942. In some embodiments, the commands 940 for media pipeline states include data for configuring media pipeline elements that will be used to process media objects. This includes data for configuring video decoding and video encoding logic within the media pipeline, such as encoding or decoding formats. In some embodiments, the commands 940 for media pipeline states also support the use of one or more pointers to "indirect" state elements that contain a batch of state settings.

[0139] In some embodiments, media object command 942 supplies pointers to a media object for processing by the media pipeline. The media object includes a memory buffer containing video data to be processed. In some embodiments, all media pipeline states must be valid before issuing media object command 942. Once the pipeline states are configured and media object command 942 is queued, media pipeline 924 is triggered via execution command 944 or an equivalent execution event (e.g., register write). The output from media pipeline 924 can then be post-processed by operations provided by 3D pipeline 922 or media pipeline 924. In some embodiments, GPGPU operations are configured and executed in a manner similar to media operations.

[0140] Graphical software architecture

[0141] Figure 10 An exemplary graphics software architecture of a data processing system 1000 according to some embodiments is illustrated. In some embodiments, the software architecture includes a 3D graphics application 1010, an operating system 1020, and at least one processor 1030. In some embodiments, the processor 1030 includes a graphics processor 1032 and one or more general-purpose processor cores 1034. The graphics application 1010 and the operating system 1020 each execute in the system memory 1050 of the data processing system.

[0142] In some embodiments, the 3D graphics application 1010 includes one or more shader programs, which include shader instructions 1012. The shader language instructions may employ a high-level shader language, such as High-Level Shading Language (HLSL) or OpenGL Shading Language (GLSL). The application also includes executable instructions 1014, which employ a machine language suitable for execution by a general-purpose processor core 1034. The application also includes graphics objects 1016 defined by vertex data.

[0143] In some embodiments, the operating system 1020 is from Microsoft Corporation. The operating system 1020 may be a dedicated UNIX-like operating system or an open-source UNIX-like operating system using a variant of the Linux kernel. The operating system 1020 may support graphics APIs 1022, such as the Direct3D API, OpenGL API, or Vulkan API. When the Direct3D API is in use, the operating system 1020 uses a front-end shader compiler 1024 to compile any shader instructions 1012 in HLSL into a lower-level shader language. This compilation may be just-in-time (JIT) compilation or pre-compilation of the application-executable shaders. In some embodiments, high-level shaders are compiled into low-level shaders during the compilation of the 3D graphics application 1010. In some embodiments, the shader instructions 1012 are provided in an intermediate form, such as a version of the standard Portable Intermediate Representation (SPIR) used by the Vulkan API.

[0144] In some embodiments, the user-mode graphics driver 1026 includes a back-end shader compiler 1027 for translating shader instructions 1012 into a hardware-specific representation. When using the OpenGL API, shader instructions 1012 in the GLSL high-level language are passed to the user-mode graphics driver 1026 for compilation. In some embodiments, the user-mode graphics driver 1026 uses operating system kernel-mode functionality 1028 to communicate with the kernel-mode graphics driver 1029. In some embodiments, the kernel-mode graphics driver 1029 communicates with the graphics processor 1032 to dispatch commands and instructions.

[0145] IP core implementation

[0146] One or more aspects of at least one embodiment can be implemented by representative code stored on a machine-readable medium that represents and / or defines logic within an integrated circuit, such as a processor. For example, the machine-readable medium may include instructions representing various logic within a processor. When read by a machine, these instructions can cause the machine to manufacture logic for performing the techniques described herein. Such representations (referred to as “IP cores”) are reusable units of logic for an integrated circuit, which can be stored on a tangible, machine-readable medium as a hardware model describing the structure of the integrated circuit. The hardware model can be supplied to various consumers or manufacturing facilities that load the hardware model onto manufacturing machines that manufacture integrated circuits. Integrated circuits can be manufactured such that the circuits perform the operations described in association with any of the embodiments described herein.

[0147] Figure 11A This is a block diagram illustrating an IP core development system 1100 that can be used to manufacture integrated circuits to perform operations, according to an embodiment. The IP core development system 1100 can be used to generate modular, reusable designs that can be incorporated into larger designs or used to build entire integrated circuits (e.g., SOC integrated circuits). Design facility 1130 can generate software simulations 1110 of the IP core design using a high-level programming language (e.g., C / C++). Software simulation 1110 can be used to design, test, and verify the behavior of the IP core using simulation model 1112. Simulation model 1112 can include functional, behavioral, and / or timing simulations. Register transfer level (RTL) designs 1115 can then be created or synthesized from simulation model 1112. RTL design 1115 is an abstraction of the behavior of an integrated circuit (including associated logic performed using the modeled digital signals) that models the flow of digital signals between hardware registers. In addition to RTL design 1115, lower-level designs at logic or transistor levels can also be created, designed, or synthesized. Thus, the specific details of the initial design and simulation can vary.

[0148] The RTL design 1115 or an equivalent can be further synthesized into a hardware model 1120 by the design facility. This hardware model may employ a Hardware Description Language (HDL) or some other representation of the physical design data. The HDL can be further simulated or tested to validate the IP core design. The IP core design can be stored in non-volatile memory 1140 (e.g., hard disk, flash memory, or any non-volatile storage medium) for delivery to a third-party manufacturing facility 1165. Alternatively, the IP core design can be transmitted (e.g., via the Internet) through a wired connection 1150 or a wireless connection 1160. The manufacturing facility 1165 can then fabricate an integrated circuit at least partially based on the IP core design. The fabricated integrated circuit can be configured to perform operations according to at least one embodiment described herein.

[0149] Figure 11BA cross-sectional side view of an integrated circuit package assembly 1170 according to some embodiments described herein is shown. The integrated circuit package assembly 1170 illustrates an implementation of one or more processor or accelerator devices as described herein. The package assembly 1170 includes a plurality of hardware logic units 1172, 1174 connected to a substrate 1180. The logic units 1172, 1174 may be implemented at least partially in configurable logic or fixed-function logic hardware and may include one or more portions of a processor core(s), a graphics processor(s), or any other accelerator device described herein. Each logic unit 1172, 1174 may be implemented within a semiconductor die and coupled to the substrate 1180 via an interconnect structure 1173. The interconnect structure 1173 may be configured to route electrical signals between the logic units 1172, 1174 and the substrate 1180 and may include interconnects such as, but not limited to, bumps or pillars. In some embodiments, interconnect structure 1173 may be configured to route electrical signals, such as, for example, input / output (I / O) signals and / or power or ground signals associated with the operation of logic 1172, 1174. In some embodiments, substrate 1180 is an epoxy-based laminated substrate. In other embodiments, package substrate 1180 may include other suitable types of substrates. Package assembly 1170 may be connected to other electrical devices via package interconnect 1183. Package interconnect 1183 may be coupled to the surface of substrate 1180 to route electrical signals to other electrical devices, such as a motherboard, other chipsets, or multi-chip modules.

[0150] In some embodiments, logic cells 1172, 1174 are electrically coupled to bridge 1182, which is configured to route electrical signals between logic cells 1172, 1174. Bridge 1182 may be a dense interconnect structure that provides routing for electrical signals. Bridge 1182 may include a bridge substrate made of glass or a suitable semiconductor material. Circuitry features may be formed on the bridge substrate to provide chip-to-chip connections between logic cells 1172, 1174.

[0151] Although two logic units 1172 and 1174 and a bridge 1182 are shown, the embodiments described herein may include more or fewer logic units on one or more dies. The one or more dies may be connected by zero or more bridges, since bridge 1182 can be excluded when logic is included on a single die. Alternatively, multiple dies or logic units may be connected by one or more bridges. Furthermore, multiple logic units, dies, and bridges may be connected together in other possible configurations, including a three-dimensional configuration.

[0152] Exemplary System-on-Chip Integrated Circuit

[0153] Figures 12 to 1 Section 4 illustrates exemplary integrated circuits and associated graphics processors that can be fabricated using one or more IP cores according to various embodiments described herein. In addition to those shown, other logic and circuitry may be included, including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores.

[0154] Figure 12 This is a block diagram illustrating an exemplary system-on-a-chip integrated circuit 1200 that can be fabricated using one or more IP cores according to an embodiment. The exemplary integrated circuit 1200 includes one or more application processors 1205 (e.g., CPU), at least one graphics processor 1210, and may additionally include an image processor 1215 and / or a video processor 1220, any of which can be modular IP cores from the same or multiple different design facilities. The integrated circuit 1200 includes peripheral or bus logic, including a USB controller 1225, a UART controller 1230, an SPI / SDIO controller 1235, and an I / O controller 1220. 2 S / I 2 C controller 1240. Additionally, the integrated circuit may include a display device 1245 coupled to one or more of a High Definition Multimedia Interface (HDMI) controller 1250 and a Mobile Industry Processor Interface (MIPI) display interface 1255. Storage may be provided by a flash memory subsystem 1260 (including flash memory and a flash memory controller). A memory interface may be provided via a memory controller 1265 to access SDRAM or SRAM memory devices. Furthermore, some integrated circuits also include an embedded security engine 1270.

[0155] Figures 13A to 13B This is a block diagram illustrating an exemplary graphics processor for use within a SoC according to embodiments described herein. Figure 13A An exemplary graphics processor 1310, which can be fabricated using one or more IP cores according to an embodiment, is shown. Figure 13B An additional exemplary graphics processor 1340 of a system-on-a-chip integrated circuit, which can be fabricated using one or more IP cores according to an embodiment, is shown. Figure 13A The graphics processor 1310 is an example of a low-power graphics processor core. Figure 13B The graphics processor 1340 is an example of a higher-performance graphics processor core. Each of the graphics processors 1310 and 1340 can be... Figure 12 A variant of the 1210 graphics processor.

[0156] like Figure 13AAs shown, the graphics processor 1310 includes a vertex processor 1305 and one or more fragment processors 1315A to 1315N (e.g., 1315A, 1315B, 1315C, 1315D, up to 1315N-1 and 1315N). The graphics processor 1310 can execute different shader programs via separate logic, such that the vertex processor 1305 is optimized to perform vertex shader program operations, while the one or more fragment processors 1315A to 1315N perform fragment (e.g., pixel) shading operations for use in fragment or pixel shader programs. The vertex processor 1305 performs the vertex processing stage of the 3D graphics pipeline and generates primitive and vertex data. The fragment processors (multiple) 1315A to 1315N use the primitive and vertex data generated by the vertex processor 1305 to produce frame buffers displayed on a display device. In one embodiment, fragment processors (multiple) 1315A to 1315N are optimized to execute fragment shader programs provided in the OpenGL API, which can be used to perform operations similar to those of pixel shader programs provided in the Direct 3D API.

[0157] Additionally, the graphics processor 1310 includes one or more memory management units (MMUs) 1320A to 1320B, one or more caches 1325A to 1325B, and one or more circuit interconnects 1330A to 1330B. The one or more MMUs 1320A to 1320B provide virtual-to-physical address mappings for the graphics processor 1310, including vertex processors 1305 and / or (multiple) fragment processors 1315A to 1315N. Besides vertex or image / texture data stored in the one or more caches 1325A to 1325B, the virtual-to-physical address mappings may also reference vertex or image / texture data stored in memory. In one embodiment, the one or more MMUs 1320A to 1320B may interact with system interconnects including those within the same memory. Figure 12 The synchronization of one or more MMUs, including one or more MMUs associated with the one or more application processors 1205, image processor 1215, and / or video processor 1220, enables each processor 1205 to 1220 to participate in a shared or unified virtual memory system. According to an embodiment, the one or more circuit interconnects 1330A to 1330B enable the graphics processor 1310 to interact with other IP cores within the SoC via the SoC's internal bus or via a direct connection.

[0158] like Figure 13B As shown, the graphics processor 1340 includes Figure 13AThe graphics processor 1310 includes one or more MMUs 1320A to 1320B, caches 1325A to 1325B, and circuit interconnects 1330A to 1330B. The graphics processor 1340 includes one or more shader cores 1355A to 1355N (e.g., 1455A, 1355B, 1355C, 1355D, 1355E, 1355F, up to 1355N-1 and 1355N), which provide a unified shader core architecture in which a single core or type or core can execute all types of programmable shader code, including shader program code to implement vertex shaders, fragment shaders, and / or compute shaders. The exact number of shader cores present may vary in embodiments and implementations. Additionally, the graphics processor 1340 includes an inter-core task manager 1345, which acts as a thread dispatcher for assigning execution threads to one or more shader cores 1355A to 1355N and a chunking unit 1358 for accelerating chunked operations for chunked rendering, in which rendering operations for a particular scene are subdivided in the image space, for example to take advantage of local spatial consistency within the scene or to optimize the use of internal caches.

[0159] Figures 14A to 14B Additional exemplary graphics processor logic according to embodiments described herein is illustrated. Figure 14A A graphics core 1400 is shown, which can be included in... Figure 12 The graphics processor 1210 can be as follows Figure 13B The unified shader cores in the 1355A to 1355N. Figure 14B The 1430 is a highly parallel general-purpose graphics processing unit suitable for deployment on multi-chip modules.

[0160] like Figure 14AAs shown, the graphics core 1400 includes a shared instruction cache 1402, texture units 1418, and cache memory / shared memory 1420 common to the execution resources within the graphics core 1400. The graphics core 1400 may include multiple slices 1401A to 1401N or per core partition, and the graphics processor may include multiple instances of the graphics core 1400. Slices 1401A to 1401N may include supporting logic, including local instruction caches 1404A to 1404N, thread schedulers 1406A to 1406N, thread dispatchers 1408A to 1408N, and a set of registers 1410A. To perform logical operations, slices 1401A to 1401N may include a set of additional functional units (AFU1412A to 1412N), floating-point units (FPU 1414A to 1414N), integer arithmetic logic units (ALU1416 to 1416N), addressing calculation units (ACU 1413A to 1413N), double-precision floating-point units (DPFPU 1415A to 1415N), and matrix processing units (MPU 1417A to 1417N).

[0161] Some of these compute units operate with specific precision. For example, FPUs 1414A to 1414N can perform single-precision (32-bit) and half-precision (16-bit) floating-point operations, while DPFPUs 1415A to 1415N perform double-precision (64-bit) floating-point operations. ALUs 1416A to 1416N can perform variable-precision integer operations with 8-bit, 16-bit, and 32-bit precision, and can be configured for mixed-precision operations. MPUs 1417A to 1417N can also be configured for mixed-precision matrix operations, including half-precision floating-point operations and 8-bit integer operations. MPUs 1417 to 1417N can perform a wide variety of matrix operations to accelerate machine learning application frameworks, including enabling accelerated Generalized Matrix-to-Matrix Multiplication (GEMM). AFUs 1412A to 1412N can perform additional logical operations not supported by floating-point or integer units, including trigonometric function operations (e.g., sine, cosine, etc.).

[0162] like Figure 14BAs shown, the General Purpose Processing Unit (GPGPU) 1430 can be configured to perform highly parallel computational operations by the graphics processing unit array. Additionally, the GPGPU 1430 can be directly linked to other instances of GPGPUs to create multi-GPU clusters, thereby improving the training speed, particularly for deep neural networks. The GPGPU 1430 includes a host interface 1432 for implementing connectivity with a host processor. In one embodiment, the host interface 1432 is a PCI Express interface. However, the host interface can also be a provider-specific communication interface or communication structure. The GPGPU 1430 receives commands from the host processor and uses a global scheduler 1434 to distribute the execution threads associated with those commands to a group of compute clusters 1436A to 1436H. Compute clusters 1436A to 1436H share a cache memory 1438. The cache memory 1438 can act as a higher-level cache of the cache memory within the compute clusters 1436A to 1436H.

[0163] The GPGPU 1430 includes memories 1434A to 1434B coupled to computing clusters 1436A to 1436H via a set of memory controllers 1442A to 1442B. In various embodiments, memories 1434A to 1434B may include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory.

[0164] In one embodiment, computing clusters 1436A to 1436H each include a set of graphics cores, such as Figure 14A The graphics core 1400 may include various types of integer logic units and floating-point logic units, which can perform computational operations suitable for machine learning within a certain precision range. For example, in one embodiment, at least a subset of the floating-point units in each of the computing clusters 1436A to 1436H may be configured to perform 16-bit or 32-bit floating-point operations, while different subsets of the floating-point units may be configured to perform 64-bit floating-point operations.

[0165] Multiple instances of GPGPU 1430 can be configured to operate as a computing cluster. The computing mechanisms used by the computing cluster for synchronization and data exchange vary across embodiments. In one embodiment, multiple instances of GPGPU 1430 communicate via host interface 1432. In one embodiment, GPGPU 1430 includes an I / O hub 1439 that couples GPGPU 1430 to GPU links 1440 that implement direct connections to other instances of GPGPU. In one embodiment, GPU link 1440 is coupled to a dedicated GPU-to-GPU bridge that implements communication and synchronization between multiple instances of GPGPU 1430. In one embodiment, GPU link 1440 is coupled to a high-speed interconnect for transmitting and receiving data to and from other GPGPUs or parallel processors. In one embodiment, multiple instances of GPGPU 1430 reside in a separate data processing system and communicate via a network device accessible via host interface 1432. In one embodiment, in addition to or as an alternative to host interface 1432, GPU link 1440 can be configured to implement a connection to a host processor.

[0166] While the illustrated configuration of the GPGPU 1430 can be configured to train neural networks, one embodiment provides an alternative configuration of the GPGPU 1430 that can be deployed within a high-performance or low-power inference platform. In the inference configuration, the GPGPU 1430 includes fewer compute clusters from compute clusters 1436A to 1436H associated with the training configuration. Additionally, the memory technology associated with memories 1434A to 1434B can differ between the inference and training configurations, with higher-bandwidth memory technology dedicated to the training configuration. In one embodiment, the inference configuration of the GPGPU 1430 can support inference-specific instructions. For example, the inference configuration can provide support for one or more 8-bit integer dot product instructions, which are typically used during the inference operations of the deployed neural network.

[0167] Machine Learning Overview

[0168] Machine learning algorithms are algorithms that can learn from a set of data. Implementations of machine learning algorithms can be designed to model high-level abstractions within a dataset. For example, image recognition algorithms can be used to determine which of several categories a given input belongs to; regression algorithms can output numerical values ​​given input; and pattern recognition algorithms can be used to generate translated text or perform text-to-speech and / or speech recognition.

[0169] One example type of machine learning algorithm is a neural network. Many types of neural networks exist; a simple type is the feedforward network. A feedforward network can be implemented as an acyclic graph, where nodes are arranged in layers. Typically, a feedforward network topology consists of an input layer and an output layer, separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation useful for generating the output in the output layer. Network nodes are fully connected to nodes in adjacent layers via edges, but there are no edges between nodes within a single layer. Data received at the nodes in the input layer of the feedforward network is propagated (i.e., “feedforward”) to the nodes in the output layer via activation functions that compute the state of nodes in each consecutive layer of the network based on coefficients (“weights”) associated with each of the edges connecting these layers. Depending on the specific model represented by the algorithm being executed, the output from a neural network algorithm can take various forms.

[0170] Before a machine learning algorithm can be used to model a specific problem, it is trained using a training dataset. Training a neural network involves: selecting a network topology; using a set of training data representing the problem being modeled by the network; and adjusting the weights until the network model exhibits minimum error for all instances in the training dataset. For example, during supervised learning training for a neural network, the output generated by the network in response to inputs representing instances in the training dataset is compared to the “correct” labeled output of those instances; an error signal representing the difference between the output and the labeled output is calculated; and the weights associated with the connections are adjusted to minimize the error as the error signal is backpropagated through the layers of the network. The network is considered “trained” when the error of each output generated from the instances in the training dataset is minimized.

[0171] The accuracy of machine learning algorithms is greatly affected by the quality of the dataset used to train them. The training process can be computationally intensive and may take a significant amount of time on a conventional general-purpose processor. Therefore, parallel processing hardware is used to train many types of machine learning algorithms. This is particularly useful for optimizing the training of neural networks, as the computations performed when adjusting the coefficients in a neural network are naturally suited to parallel implementation. Specifically, many machine learning algorithms and software applications have been adapted to use parallel processing hardware within general-purpose graphics processing devices.

[0172] Figure 15This is a generalized graph of machine learning software stack 1500. Machine learning application 1502 can be configured to train a neural network using a training dataset or to achieve machine intelligence using a trained deep neural network. Machine learning application 1502 may include training and inference capabilities for the neural network and / or dedicated software, which can be used to train the neural network prior to deployment. Machine learning application 1502 can achieve any type of machine intelligence, including but not limited to: image recognition, mapping and localization, autonomous navigation, speech synthesis, medical imaging, or language translation.

[0173] Hardware acceleration for machine learning applications 1502 can be achieved via machine learning framework 1504. Machine learning framework 1504 provides a library of machine learning primitives. Machine learning primitives are the fundamental operations typically performed by machine learning algorithms. Without machine learning framework 1504, developers of machine learning algorithms would need to create and optimize the main computational logic associated with their algorithms, and then re-optimize that computational logic when a new parallel processor is developed. Instead, machine learning applications can be configured to use primitives provided by machine learning framework 1504 to perform the necessary computations. Exemplary primitives include tensor convolution, activation functions, and pooling, which are computational operations performed when training convolutional neural networks (CNNs). Machine learning framework 1504 can also provide primitives for implementing basic linear algebra subroutines performed by many machine learning algorithms, such as matrix and vector operations.

[0174] Machine learning framework 1504 can process input data received from machine learning application 1502 and generate appropriate input for computing framework 1506. Computing framework 1506 can abstract the low-level instructions provided to GPGPU driver 1508, enabling machine learning framework 1504 to utilize hardware acceleration via GPGPU hardware 1510 without requiring machine learning framework 1504 to be very familiar with the architecture of GPGPU hardware 1510. Furthermore, computing framework 1506 can implement hardware acceleration for machine learning framework 1504 across various types and generations of GPGPU hardware 1510.

[0175] Machine learning neural network implementation methods

[0176] The computational architectures provided by the embodiments described herein can be configured to perform these types of parallel processing, which are particularly well-suited for training and deploying neural networks for machine learning. Neural networks can be generalized as networks of functions with graph relationships. As is known in the art, there are various types of neural network implementations used in machine learning. One exemplary type of neural network is the feedforward network as previously described.

[0177] The second exemplary type of neural network is the Convolutional Neural Network (CNN). A CNN is a specialized feedforward neural network designed for processing data with a known, grid-like topology (e.g., image data). Therefore, CNNs are commonly used in computer vision and image recognition applications, but they can also be used in other types of pattern recognition, such as speech and language processing. Nodes in the input layer of a CNN are organized as a set of “filters” (feature detectors inspired by receptive fields found in the retina), and the output of each set of filters is propagated to nodes in successive layers of the network. The computation for a CNN involves applying convolutional mathematics to each filter to produce the output of that filter. Convolution is a specialized mathematical operation performed by two functions to produce a third function, which is a modified version of one of the two original functions. In convolutional network terminology, the first function related to convolution can be referred to as the input, and the second function can be referred to as the convolution kernel. The output can be referred to as a feature map. For example, the input to a convolutional layer can be a multidimensional array of data that defines various color components of the input image. The convolution kernel can be a multidimensional array of parameters, which are adapted through a training process for the neural network.

[0178] Recurrent Neural Networks (RNNs) are a type of feedforward neural network that includes feedback connections between layers. RNNs enable the modeling of sequential data by sharing parameter data across different parts of the neural network. The architecture of an RNN includes loops. These loops represent the effect of the current value of a variable on its own value at future times, because at least a portion of the output data from the RNN is used as feedback to process subsequent inputs in the sequence. This variable nature of language data makes RNNs particularly useful for language processing.

[0179] The diagrams described below illustrate exemplary feedforward, CNN, and RNN networks, and describe the general process for training and deploying each of those types of networks respectively. It will be understood that these descriptions are exemplary and non-limiting with respect to any particular embodiment described herein, and that the concepts shown can generally be applied to deep neural networks and machine learning techniques.

[0180] The exemplary neural network described above can be used to perform deep learning. Deep learning is machine learning performed using deep neural networks. In contrast to shallow neural networks that contain only a single hidden layer, the deep neural networks used in deep learning are artificial neural networks composed of multiple hidden layers. Deeper neural networks are generally more computationally intensive to train. However, the additional hidden layers of the network enable multi-step pattern recognition, which results in reduced output error compared to shallow machine learning techniques.

[0181] Deep neural networks used in deep learning typically include a front-end network for performing feature recognition coupled to a back-end network representing a mathematical model, which can then perform operations (e.g., object classification, speech recognition, etc.) based on the feature representations provided to the model. Deep learning enables machine learning to be performed without requiring manual feature engineering on the model. Instead, deep neural networks can learn features based on statistical structure or correlations within the input data. The learned features can be provided to a mathematical model, which can then map the detected features to the output. The mathematical model used by the network is typically specialized for a specific task to be performed, and different models will be used to perform different tasks.

[0182] Once a neural network is structured, a learning model can be applied to it to train it to perform a specific task. The learning model describes how weights are adjusted within the model to reduce the network's output error. Backpropagation of error is a common method used to train neural networks. An input vector is presented to the network for processing. The network's output is compared to the expected output using a loss function, and an error value is calculated for each neuron in the output layer. These error values ​​are then backpropagated until each neuron has an associated error value that roughly represents its contribution to the original output. The network can then learn from those errors using an algorithm (e.g., stochastic gradient descent) to update the neural network's weights.

[0183] Figures 16A-16B This demonstrates an exemplary convolutional neural network. Figure 16A Show the individual layers within a CNN. For example... Figure 16A As shown, an exemplary CNN for modeling image processing can receive input 1602, which describes the red, green, and blue (RGB) components of an input image. Input 1602 can be processed by multiple convolutional layers (e.g., a first convolutional layer 1604, a second convolutional layer 1606). Optionally, the output from the multiple convolutional layers can be processed by a set of fully connected layers 1608. Neurons in a fully connected layer have full connections to all activation functions in the previous layer, as previously described for feedforward networks. The output from the fully connected layer 1608 can be used to generate an output from the network. Matrix multiplication can be used instead of convolution to compute the activation functions within the fully connected layer 1608. Not all CNN implementations use fully connected layers 1608. For example, in some implementations, a second convolutional layer 1606 can generate the CNN output.

[0184] Convolutional layers are sparsely connected, unlike the traditional neural network configuration found in fully connected layers (1608). Traditional neural network layers are fully connected, such that each output unit interacts with each input unit. However, convolutional layers are sparsely connected because the output of the convolution of the receptive field (rather than the corresponding state value of each node in the receptive field) is fed to nodes in subsequent layers, as illustrated. The kernels associated with the convolutional layers perform convolution operations, the output of which is sent to the next layer. Dimensionality reduction performed within convolutional layers is one aspect that enables CNNs to scale to handle large images.

[0185] Figure 16B This illustrates an exemplary computational phase within a convolutional layer of a CNN. The input 1612 to the convolutional layer of the CNN can be processed in three phases within convolutional layer 1614. These three phases may include a convolutional phase 1616, a detector phase 1618, and a pooling phase 1620. Convolutional layer 1614 can then output the data to successive convolutional layers. The last convolutional layer of the network can generate output feature map data or provide input to fully connected layers, for example, to generate classification values ​​for the input to the CNN.

[0186] Several convolutions are performed in parallel within convolutional stage 1616 to produce a set of linear activation functions. Convolutional stage 1616 may include affine transformations, which are any transformations that can be specified as a linear transformation plus a translation. Affine transformations include rotation, translation, scaling, and combinations of these transformations. The convolutional stage computes the output of a function (e.g., a neuron) connected to a specific region in the input, which can be determined as a local region associated with the neuron. The neuron computes the dot product between the neuron's weights and a region in the local input to which the neuron is connected. The output from convolutional stage 1616 defines a set of linear activation functions processed by successive stages of convolutional layer 1614.

[0187] Linear activation functions can be processed by detector stage 1618. In detector stage 1618, each linear activation function is processed by a nonlinear activation function. Nonlinear activation functions add nonlinearity to the overall network without affecting the receptive field of the convolutional layers. Several types of nonlinear activation functions can be used. One specific type is the Modified Linear Unit (ReLU), which uses an activation function defined as f(x) = max(0, x) such that the activation function is thresholded to zero.

[0188] Pooling stage 1620 uses a pooling function that replaces the output of the second convolutional layer 1606 with a generalized statistical value of the nearby output. The pooling function can be used to introduce translation invariance into the neural network, such that slight translations to the input do not change the pooling output. Local translation invariance can be useful when the presence of features in the input data is more important than the precise location of the features. Various types of pooling functions can be used during pooling stage 1620, including max pooling, average pooling, and L2-norm pooling. Additionally, some CNN implementations do not include a pooling stage. Instead, such implementations use an additional convolutional stage with a larger stride relative to the previous convolutional stage.

[0189] The output from convolutional layer 1614 can then be processed by the next layer 1622. The next layer 1622 can be either an additional convolutional layer or a fully connected layer 1608. For example, Figure 16A The first convolutional layer 1604 can output to the second convolutional layer 1606, and the second convolutional layer can output to the first layer in the fully connected layer 1608.

[0190] Figure 17 An exemplary recurrent neural network is shown. In a recurrent neural network (RNN), the network's previous state influences the output of the network's current state. RNNs can be constructed in a wide variety of ways using a variety of functions. The use of RNNs often revolves around using mathematical models to predict the future based on previous input sequences. For example, RNNs can be used to perform statistical language modeling to predict an upcoming word given a previous sequence of words. The RNN 1700 shown can be described as having the following components: an input layer 1702 that receives an input vector; a hidden layer 1704 for implementing the recurrent function; a feedback mechanism 1705 for implementing a 'memory' of previous states; and an output layer 1706 for outputting the result. RNN 1700 operates based on time steps. The state of the RNN at a given time step is influenced by the feedback mechanism 1705 based on previous time steps. For a given time step, the state of the hidden layer 1704 is defined by the previous state and the input at the current time step. The initial input (x1) at the first time step can be processed by the hidden layer 1704. The second input (x2) can be processed by hidden layer 1704 using the state information determined during the processing of the initial input (x1). The given state can be computed as s. t =f(Ux t +Ws t-1), where U and W are parameter matrices. The function f is typically nonlinear, such as a variant of the hyperbolic tangent function (Tanh) or the correction function f(x) = max(0, x). However, the specific mathematical function used in hidden layer 1704 can vary depending on the specific implementation details of the RNN 1700.

[0191] In addition to the basic CNN and RNN networks described, variations of those networks can be implemented. An example RNN variant is the Long Short-Term Memory (LSTM) RNN. LSTM RNNs are capable of learning long-term dependencies necessary for processing longer language sequences. A CNN variant is the Convolutional Deep Belief Network, which has a similar structure to a CNN and is trained in a similar manner to a deep belief network. A deep belief network (DBN) is a generative neural network consisting of multiple layers of stochastic (random) variables. A DBN can be trained layer by layer using greedy unsupervised learning. The learned weights of the DBN can then be used to provide a pre-trained neural network by determining an optimal set of initial weights for the neural network.

[0192] Figure 18 This demonstrates the training and deployment of deep neural networks. Once a given network has been structured for a task, it is trained using the training dataset 1802. Various training frameworks have been developed to accelerate the training process using hardware. For example, Figure 15 The machine learning framework 1504 can be configured as a training framework 1804. The training framework 1804 can be hooked into an untrained neural network 1806, enabling the use of the parallel processing resources described herein to train the untrained neural network to generate a trained neural network 1808. To begin the training process, initial weights can be selected randomly or through pre-training using a deep belief network. The training loop is then performed in a supervised or unsupervised manner.

[0193] Supervised learning is a learning method in which training is performed as an arbitration operation, such as when the training dataset 1802 includes inputs (which are paired with the expected outputs of said inputs), or when the training dataset includes inputs with known outputs and the outputs of the neural network are manually graded. The network processes the inputs and compares the resulting outputs with a set of expected or desired outputs. The error is then backpropagated through the system. The training framework 1804 can be tuned to adjust the weights controlling the untrained neural network 1806. The training framework 1804 can provide tools for monitoring the extent to which the untrained neural network 1806 converges to a model suitable for generating correct answers based on known input data. The training process occurs repeatedly as the network weights are adjusted to improve the outputs generated by the neural network. The training process can continue until the neural network reaches the statistically expected accuracy associated with the trained neural network 1808. The trained neural network 1808 can then be deployed to implement any number of machine learning operations.

[0194] Unsupervised learning is a learning method in which a network attempts to train itself using unlabeled data. Therefore, for unsupervised learning, the training dataset 1802 would include input data without any associated output data. The untrained neural network 1806 can learn groupings within the unlabeled inputs and can determine how individual inputs relate to the overall dataset. Unsupervised training can be used to generate self-organizing maps, which are a type of trained neural network 1807 capable of performing operations useful in data dimensionality reduction. Unsupervised training can also be used to perform anomaly detection, which allows the identification of data points in the input dataset that deviate from the normal pattern of the data.

[0195] Variations in supervised and unsupervised training can also be employed. Semi-supervised learning is a technique where the training dataset 1802 comprises a mixture of labeled and unlabeled data with the same distribution. Incremental learning is a variant of supervised learning where the input data is continuously used for further training of the model. Incremental learning enables the trained neural network 1808 to adapt to new data 1812 without forgetting the knowledge embedded within the network during the initial training.

[0196] Whether supervised or unsupervised, training very deep neural networks can be computationally too intensive for a single computing node. A distributed network of computing nodes can be used instead of a single node to accelerate the training process.

[0197] Figure 19This is a block diagram illustrating distributed learning. Distributed learning trains a model using multiple distributed computing nodes to perform supervised or unsupervised training of a neural network. Each of these distributed computing nodes may include one or more host processors and general-purpose processing nodes. As shown, distributed learning can perform model parallelization 1902, data parallelization 1904, or a combination of model and data parallelization 1904.

[0198] In Model Parallelism 1902, different computing nodes in a distributed system can perform training computations on different parts of a single network. For example, each layer of a neural network can be trained by different processing nodes in a distributed system. Benefits of model parallelism include the ability to scale to exceptionally large models. Splitting the computations associated with different layers of a neural network makes it possible to train very large neural networks, where the weights of all layers are not fitted into the memory of a single computing node. In some instances, model parallelism can be particularly useful in performing unsupervised training of large neural networks.

[0199] In data parallelization 1904, different nodes in a distributed network have complete instances of the model, and each node receives a different portion of the data. The results from the different nodes are then combined. While different approaches to data parallelization are possible, all data-parallel training methods require a technique for combining the results and synchronizing the model parameters across each node. Exemplary methods for combining data include parameter averaging and update-based data parallelization. Parameter averaging trains each node on a subset of the training data and sets global parameters (e.g., weights, biases) to the average of the parameters from each node. Parameter averaging uses a central parameter server that maintains the parameter data. Update-based data parallelization is similar to parameter averaging, except that model updates are passed instead of parameters from nodes to a parameter server. Additionally, update-based data parallelization can be performed in a distributed manner, where updates are compressed and passed between nodes.

[0200] For example, combined model and data parallelism can be implemented in a distributed system, where each computing node includes multiple GPUs. Each node can have a complete instance of the model, with individual GPUs within each node used to train different parts of the model.

[0201] Distributed training incurs increased overhead compared to training on a single machine. However, the parallel processors and GPGPUs described in this paper can each implement techniques to reduce the overhead of distributed training, including techniques for enabling high-bandwidth GPU-to-GPU data transfer and accelerating remote data synchronization.

[0202] Exemplary machine learning applications

[0203] Machine learning can be applied to solve a wide range of technical problems, including but not limited to computer vision, autonomous driving and navigation, speech recognition, and language processing. Computer vision has traditionally been one of the most active research areas for machine learning applications. Applications of computer vision range from reproducing human visual abilities (e.g., recognizing faces) to creating new categories of visual abilities. For example, a computer vision application can be configured to identify sound waves from vibrations induced in objects visible in a video. Parallel processor-accelerated machine learning enables the training of computer vision applications using training datasets significantly larger than previously feasible ones, and allows the deployment of inference systems using low-power parallel processors.

[0204] Parallel processor-accelerated machine learning has applications in autonomous driving, including lane and road sign recognition, obstacle avoidance, navigation, and driving control. Accelerated machine learning techniques can be used to train driving models based on datasets that define appropriate responses to specific training inputs. The parallel processors described in this paper enable the rapid training of increasingly sophisticated neural networks for autonomous driving solutions and allow the deployment of low-power inference processors in mobile platforms suitable for integration into autonomous vehicles.

[0205] Parallel processor-accelerated deep neural networks have been implemented as machine learning methods for Automatic Speech Recognition (ASR). ASR involves creating functions that compute the most probable language sequence given an input speech sequence. Accelerated machine learning using deep neural networks has replaced previous Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs) used for ASR.

[0206] Parallel processor-accelerated machine learning can also be used to accelerate natural language processing. Automated learning programs can use statistical inference algorithms to generate models that are robust to errors or unfamiliar inputs. Exemplary natural language processor applications include automated machine translation between human languages.

[0207] Parallel processing platforms for machine learning can be divided into training platforms and deployment platforms. Training platforms are typically highly parallel and include optimizations to accelerate multi-GPU single-node training and multi-node multi-GPU training, while deployment machine learning (e.g., inference) platforms typically include lower-power parallel processors suitable for use in products such as cameras, autonomous robots, and autonomous vehicles.

[0208] Figure 20A computing device 2000 is illustrated, comprising a master-controlled intelligent deep learning convolutional mechanism (“convolutional mechanism”) 2010 according to one embodiment. The computing device 2000 represents a communication and data processing device (including, but not limited to, smart wearable devices, smartphones, virtual reality (VR) devices, head-mounted displays (HMDs), mobile computers, Internet of Things (IoT) devices, laptop computers, desktop computers, server computers, etc.), and can be connected to… Figure 1 The processing equipment 100 is similar to or the same as that used in the above references; therefore, for the sake of brevity, clarity and ease of understanding, it will not be discussed or repeated further hereafter. Figure 1-19 Many details of the statement.

[0209] The computing device 2000 may also include (but is not limited to) autonomous machines or artificial intelligence agents, such as mechanical agents or machines, electronic agents or machines, virtual agents or machines, electromechanical agents or machines, etc. Examples of autonomous machines or artificial intelligence agents may include (but are not limited to) robots, autonomous vehicles (e.g., self-driving cars, self-driving aircraft, self-navigating sailboats, etc.), autonomous devices (self-operating engineering vehicles, self-operating medical devices, etc.). Throughout this document, "computing device" may be used interchangeably as "autonomous machine" or "artificial intelligence agent" or simply "robot".

[0210] It is foreseeable that although the terms "autonomous vehicle" and "autonomous driving" are used throughout this document, the embodiments are not limited thereto. For example, "autonomous vehicle" is not limited to automobiles, but can include any number and type of autonomous machines, such as robots, autonomous devices, home autonomous devices, etc., and any one or more tasks or operations associated with such autonomous machines can be referred to interchangeably with autonomous driving.

[0211] The computing device 2000 may further include (but is not limited to) large computing devices, such as server computers, desktop computers, etc., and may further include set-top boxes (e.g., internet-based cable TV set-top boxes, etc.), GPS-based devices, etc. The computing device 2000 may include mobile computing devices used as communication devices, such as cellular phones including smartphones, personal digital assistants (PDAs), tablet computers, laptop computers, e-readers, smart TVs, television platforms, wearable devices (e.g., glasses, watches, bracelets, smart cards, jewelry, clothing, etc.), media players, etc. For example, in one embodiment, the computing device 600 may include a mobile computing device employing a computer platform that integrates various hardware and / or software components of the computing device 2000 onto a single chip, such as an integrated circuit (“IC”) like a system-on-a-chip (“SoC” or “SOC”).

[0212] As shown, in one embodiment, computing device 2000 may include any number and type of hardware and / or software components, such as (but not limited to) a graphics processing unit (“GPU” or simply a “graphics processor”) 2014, a graphics driver (also referred to as a “GPU driver,” “graphics driver logic,” “driver logic,” user-mode driver (UMD), UMD, user-mode driver framework (UMDF), UMDF, or simply a “driver”) 2016, a central processing unit (“CPU” or simply an “application processor”) 2012, memory 2008, network devices, drivers, etc., and input / output (I / O) sources 2004 such as a touchscreen, touch panel, touchpad, virtual or conventional keyboard, virtual or conventional mouse, ports, connectors, etc. Computing device 2000 may include an operating system (OS) 2006 that serves as an interface between the hardware and / or physical resources of computing device 2000 and the user. It is foreseeable that the graphics processor 2014 and application processor 2012 may be... Figure 1 One or more of the processors 102.

[0213] It should be understood that, for some implementations, a system equipped with fewer or more features than the examples described above may be preferred. Therefore, the configuration of the computing device 2000 may vary depending on the implementation, depending on numerous factors such as price constraints, performance requirements, technological improvements, or other circumstances.

[0214] Implementations may be carried out as any one or a combination of the following: one or more microchips or integrated circuits interconnected using a motherboard, hard-wired logic, software stored in a memory device and executed by a microprocessor, firmware, application-specific integrated circuits (ASICs) and / or field-programmable gate arrays (FPGAs). Throughout this document, the terms “logic,” “module,” “component,” “engine,” “mechanism,” “tool,” “circuit,” and “circuit system” are used interchangeably and, by way of example, include software, hardware, firmware, or any combination thereof.

[0215] In one embodiment, as shown, the convolution mechanism 2010 may be controlled by the memory 2008 of the computing device 2000. In another embodiment, the convolution mechanism 2010 may be controlled by the operating system 2010 or the graphics driver 2016. In yet another embodiment, the convolution mechanism 2010 may be controlled by or part of a graphics processing unit (“GPU” or “graphics processor”) 2014 or the firmware of the graphics processor 2014. For example, the convolution mechanism 2010 may be embedded or implemented as part of the processing hardware of the graphics processor 2014. Similarly, in yet another embodiment, the convolution mechanism 2010 may be controlled by or part of a central processing unit (“CPU” or “application processor”) 2012. For example, the convolution mechanism 2010 may be embedded or implemented as part of the processing hardware of the application processor 2012.

[0216] In another embodiment, the convolution mechanism 2010 may be controlled by any number and type of components of the computing device 2000, or a portion thereof. For example, one part of the convolution mechanism 2010 may be controlled by the operating system 2006, or a portion thereof; another part may be controlled by the graphics processor 2014, or a portion thereof; and another part may be controlled by the application processor 2012, or a portion thereof. One or more parts of the convolution mechanism 2010 may be controlled by the operating system 2006 and / or any number and type of computing device 2000, or a portion thereof. It is foreseeable that the embodiments are not limited to any implementation of the convolution mechanism 2010 or any method of controlling it, and one or more parts or components of the convolution mechanism 2010 may be adopted or implemented as hardware, software, or any combination thereof, such as firmware.

[0217] The computing device 2000 can control multiple network interfaces to provide access to networks such as LANs, wide area networks (WANs), metropolitan area networks (MANs), personal area networks (PANs), Bluetooth, cloud networks, mobile networks (e.g., 3G, 4G, etc.), intranets, the Internet, etc. The multiple network interfaces may include, for example, a wireless network interface with an antenna, which may represent one or more antennas. The multiple network interfaces may also include, for example, a wired network interface for communicating with remote devices via a network cable, which may be, for example, an Ethernet cable, coaxial cable, optical fiber, serial cable, or parallel cable.

[0218] The embodiments may be provided as, for example, computer program products, which may include one or more machine-readable media on which machine-executable instructions are stored, which, when executed by one or more machines (such as a computer, computer network, or other electronic device), cause the one or more machines to perform operations according to the embodiments described herein. Machine-readable media may include, but are not limited to: floppy disks, optical disks, CD-ROMs (compact disc read-only memory), magneto-optical disks, ROMs, RAMs, EPROMs (erasable programmable read-only memory), EEPROMs (electrically erasable programmable read-only memory), magnetic cards or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions.

[0219] Furthermore, the embodiments can be downloaded as a computer program product, wherein the program can be transmitted from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a communication link (e.g., a modem and / or a network connection) by means of one or more data signals embodied in and / or modulated by a carrier or other propagation medium.

[0220] Throughout this document, the term "user" may be used interchangeably with "viewer," "observer," "person," "individual," "end-user," etc. It should be noted that throughout this document, terms such as "graphics domain" may be used interchangeably with "graphics processing unit," "graphics processor," or simply "GPU," and similarly, "CPU domain" or "host domain" may be used interchangeably with "computer processing unit," "application processor," or simply "CPU."

[0221] It should be noted that terms such as "node," "computing node," "server," "server device," "cloud computer," "cloud server," "cloud server computer," "machine," "host," "device," "computing device," "computer," and "computing system" are used interchangeably throughout this document. Furthermore, it should be noted that terms such as "application," "software application," "program," "software program," "package," and "software package" are used interchangeably throughout this document. Also, terms such as "job," "input," "request," and "message" are used interchangeably throughout this document.

[0222] Figure 21 An embodiment is shown. Figure 20 The convolution mechanism (2010). For the sake of brevity, references will not be repeated or discussed hereafter. Figure 1-20Many details are discussed. In one embodiment, the convolution mechanism 2010 may include any number and type of components, such as (but not limited to): detection and selection logic 2101; filter generation and storage logic (“filter logic”) 2103; communication / compatibility logic 2107; separable convolution logic 2109; and training and fine-tuning logic 2111.

[0223] As mentioned earlier, traditional deep learning-based recognition techniques often require and consume significant amounts of memory and computing power, making them unsuitable for low-end, small, or embedded devices such as tablets, smartphones, and HMDs. For example, in visual recognition, there may be over 60 million parameters (requiring a network model size of 250MB), and each image performs over 1.5 million floating-point operations (measured using a normalized size of 224x224). Some deeper networks require even larger network model sizes (e.g., over 500MB) with even greater computational requirements (e.g., over 16 million floating-point operations). At least due to these limitations, most current training and scoring operations are performed using larger or high-end computing devices and are prevented from being deployed on smaller or low-end computing devices.

[0224] For example, in machine and / or deep learning environments across various computing devices, neural networks like CNNs often have relatively high computational or processing resource requirements, such as requirements for model size (which can be as large as 100MB), runtime memory, and the number of FLOPs. This level of requirement often hinders the development and training of neural networks; particularly for resource-constrained applications (e.g., in IoT scenarios), real-time applications in autonomous driving, and low-end and / or smaller computing devices. Traditional technologies cannot handle such operations, although additional hardware / software accelerators are required.

[0225] Today, developers are striving to develop novel neural networks for a variety of use cases that require continuous network training, often multiple times, to meet their detection requirements. This training is performed, for example, by changing filter parameters and several upper-layer configurations. Conventional techniques like Constant, Gaussian, and Xavier are used as weight fillers with several limitations and drawbacks.

[0226] For example, some of these drawbacks include: 1) generalized weight fillers often converge in later iterations; 2) hit-and-test techniques are required to find good parameter functions, such as Gaussian standard deviation; and 3) sometimes the default function does not converge at all, which can happen if the developers are working on a new neural network.

[0227] In N-dimensional space, the categories to be detected are arranged in separable clusters. Training a neural network is essentially the task of finding the parameters of a function in N-dimensional space that allows the function to distinguish all categories. Setting default values ​​for the weights before training means assigning default values ​​to these parameters of the function. The embodiment provides a novel technique for using custom feature layers that essentially bring any parameters closer to their desired values ​​for efficient category separation.

[0228] The embodiments provide a novel technique for geometry-based convolution to reduce the training time of neural networks. For example, this new technique can reduce the time it takes for a neural network to achieve convergence by providing geometry-specific filters associated with the lower convolutional layers of the neural network. Further, for example, instead of using generalized or random filters, the filters are generated and trained to be geometry-specific, and then stored in a repository such as database(s)2130, resulting in efficient and fast training of the neural network. This provides less trial and error because the selection of lower-level filters is supported by visual inspection, which is relatively more intuitive. Furthermore, in this way, training can start at a lower learning rate and subsequently convergence is achieved somewhat more easily.

[0229] In one embodiment, such as regarding Figure 22A and 23A To further illustrate, the detection and selection logic 2101 is used to detect any number and type of images from various sources (e.g., stored images, websites, etc.), and subsequently select customized training input images that contain primitive geometries (e.g., circles, triangles, rectangles, etc.) associated with the objects being trained by the neural network. Because the objects have geometric shapes, in one embodiment, image selection is performed based on the geometry contained within these images, making it possible to generate and store a robust library of geometry-based filters for subsequent use in training purposes.

[0230] During the detection and selection of the input image, as facilitated by the training and fine-tuning logic 2111, the filter logic 2103 is triggered to generate weights to provide filters based on the selected customized input image and geometry. These filters, while based on geometry, are not yet trained, but are chosen to represent the shape of the object being trained on the neural network. For example, if the object is a widescreen television, it is expected to be rectangular. Similarly, other objects can have other geometries, such as circles, squares, etc.

[0231] As previously mentioned, these filters are typically small lower-level filters because they are easily adaptable to changes in the lower layers of the neural network, enabling them to process and manipulate even the smallest or least effective information. Once these filters are obtained through filter logic 2103, they are subsequently analyzed using one or more visual inspection tools based on a common geometry. In one embodiment, analyzing the filters includes observing filters associated with a common geometry, such that certain filters can be grouped together based on their corresponding geometries.

[0232] In one embodiment, once the filters are analyzed based on a common geometry, filter logic 2103 is triggered to classify these filters by their corresponding common geometry, and then store them in bins representing those geometries. For example, circle-based filters are grouped and stored in bins representing circles, and rectangle-based filters are grouped and stored in bins representing rectangles, etc.

[0233] In one embodiment, such as regarding Figure 22B and 23A To further illustrate, these stored filters are then used for geometry-specific training of a neural network for recognizing objects (e.g., televisions) from a custom input image. In one embodiment, detection and selection logic 2101 selects a bin with filters that closely represent the shape of the object. Continuing with the example of a television as the object, detection and selection logic 2101 would select a bin labeled as rectangular, as it is expected to contain rectangle-based filters.

[0234] In one embodiment, when selecting a relevant bin and its filter, training and fine-tuning logic 2111 then initiates training of the selected filter based on values ​​or corresponding information from that bin. For example, not all rectangles are the same; a rectangular credit card is expected to be much smaller than a rectangular television. Similarly, there are different shapes of objects, such as a rectangular rug being softer than a rectangular dining table. Any such values ​​or data obtained from the bin can be used by training and fine-tuning logic 2111 to initialize the relevant filter to match the shape of the object of interest (e.g., a television).

[0235] Once the filters are trained to be geometry-based, the training and fine-tuning logic 2111 then initiates geometry-specific training of the neural network based on the geometry-based filters. This geometry-specific training of the neural network using geometry-based filters provides targeted and efficient training of the neural network, allowing it to converge faster than training via conventional random initialization.

[0236] As mentioned above, convolution is considered one of the most computationally intensive processes in neural networks due to frequent and irregular memory accesses. Some traditional techniques, such as GEMM-based convolution, are known to separate memory access tasks from purely computational tasks to improve efficiency; however, these traditional techniques consume too much time in preparation (e.g., aligning the memory in a column matrix format) before the separation can be performed.

[0237] For reference Figure 22C , 23B As shown in Figure 24, the embodiment further provides a novel technique for identifying and separating filters associated with lower-level filters of a neural network for training purposes. In one embodiment, detection and selection logic 2101 is used to detect and select an input image of an object for training the neural network. It is foreseeable that all layers of the neural network can be identified, but lower layers and their filters are selected and used for separation because the lower layers are where training begins, and therefore these layers are more adapted to the newly learned features.

[0238] In some embodiments, such as those facilitated by training and fine-tuning logic 2111, filter logic 2103 is used to generate weights to provide filters based on selected input images and their content, wherein the images are images of one or more objects for which the neural network is being trained. Once the filters are obtained, in one embodiment, separable convolution logic 2109 is triggered to identify the individual layers of the neural network, including upper and lower layers, wherein any filters associated with these lower layers are also identified when identifying the lower layers. Upon completion of this identification, separable convolution logic 2019 is subsequently triggered to separate or divide the original-sized lower-layer filters into multiple smaller-sized filters and reset their values ​​to random initializations, while keeping the upper-layer filters in their current form. For example, a typical small 2D lower-layer convolutional filter (e.g., a 3×3 filter or a 5×5 filter) can be divided into two even smaller 1D lower-layer convolutional filters (e.g., 1×3 and 3×1 filters or 1×5 and 5×1 filters).

[0239] In one embodiment, once the lower-level filters are separated, the training and fine-tuning logic 2111 then trains the resulting neural network using the separated pairs of lower-level filters, where each pair of separated lower-level filters is followed by a single rectified linear unit (ReLU). Predictably, this type of training of the neural network is performed for those objects identified earlier in the selected input image. This novel technique using smaller lower-level convolutional filters reduces the overall complexity of neural network training, such as from O(n⁴) to O(n³), without affecting training accuracy, while improving training speed by providing simpler and faster operation with 1D filters.

[0240] The computing device 2000 is further shown to communicate with one or more repositories, datasets and / or databases (such as multiple databases 2130, e.g., cloud storage, non-cloud storage, etc.), wherein the multiple databases 2130 may reside in local or remote storage via multiple communication media 2125, such as one or more networks (e.g., cloud networks, proximity networks, mobile networks, intranets, the Internet, etc.).

[0241] It is foreseeable that software applications running on computing device 2000 can be responsible for using one or more components of computing device 2000 (e.g., GPU 2014, graphics driver 2016, CPU 2012, etc.) to perform or facilitate the execution of any number and type of tasks. When performing tasks as defined by the software application, one or more components such as GPU 2014, graphics driver 2016, CPU 2012, etc., can communicate with each other to ensure that those tasks are processed and completed accurately and in a timely manner.

[0242] The communication / compatibility logic 2107 can be used to facilitate the required communication and compatibility between any number of devices of the computing device 2000 and various components of the convolution mechanism 2010.

[0243] Communication / compatibility logic 2107 can be used to facilitate dynamic communication and compatibility between computing device 2000 and the following: any number and type of other computing devices (such as mobile computing devices, desktop computers, server computing devices, etc.); processing devices or components (such as CPUs, GPUs, etc.); capture / sensing / detection devices (such as capture / sensing components, including cameras, depth-sensing cameras, camera sensors, red-green-blue (RGB) sensors, microphones, etc.); display devices (such as output components, including displays, display areas, display projectors, etc.); user / context-aware components and / or identification / authentication sensors / devices (such as...). Object feature sensors / detectors, scanners, etc.); (multiple) databases 2130, such as memory or storage devices, databases and / or data sources (such as data storage devices, hard disk drives, solid-state drives, hard disks, memory cards or devices, memory circuits, etc.); (multiple) communication media 2125, such as one or more communication channels or networks (e.g., cloud networks, the Internet, intranets, cellular networks, proximity networks, such as Bluetooth, Bluetooth Low Energy (BLE), Bluetooth Smart, Wi-Fi proximity, radio frequency identification (RFID), near field communication (NFC), body area network (BAN), etc.); wireless or wired communication and related protocols (e.g., WiMAX, Ethernet, etc.); connectivity and location management technologies; software applications / websites (e.g., social and / or business network websites, business applications, games and other entertainment applications); and programming languages, while ensuring compatibility with changing technologies, parameters, protocols, standards, etc.

[0244] Throughout this document, terms such as “logic,” “component,” “module,” “framework,” “engine,” “mechanism,” “circuit,” and “circuit system” are used interchangeably and, by way of example, may include software, hardware, firmware, or any combination thereof. In one example, “logic” may refer to or include a software component capable of working with one or more of a computing device such as computing device 2000, such as an operating system (e.g., operating system 2006), a graphics driver (e.g., graphics driver 2016), etc. In another example, “logic” may refer to or include a hardware component capable of being physically installed with or as part of one or more system hardware elements of a computing device such as computing device 2000, such as an application processor (e.g., CPU 2012), a graphics processor (e.g., GPU 214), etc.). In yet another embodiment, “logic” may refer to or include a firmware component capable of being part of the system firmware of a computing device such as computing device 2000, such as the firmware of an application processor (e.g., CPU 2012) or a graphics processor (e.g., GPU 2014)).

[0245] In addition, specific brands, words, terms, phrases, names, and / or acronyms (such as "neural network," "convolutional layer," "convolutional kernel," "training weights," "2D convolution," "1D convolution," "separable convolution," "weight matrix," "deep neural network," "additional feature layer," "scaling factor," "sparse channel," "network structure," "FC layer or row," "narrow or shrink network," "input," "add," "learn," "prune," "computation," "get," "shrink," "training," "fine-tuning," "output," "GPU," "GPU domain," "GPGPU," "CPU," "CPU domain," "graphics driver," "workload," "application," "graphics pipeline," "pipeline process," "register," "register file," "RF," "extended register file," "ERF," "execution unit," "EU," "instruction," "API," "3D API") are also included. Any use of terms such as “fragment shader,” “YUV texture,” “shader execution,” “existing UAV functionality,” “existing backend,” “hardware,” “software,” “agent,” “graphics driver,” “kernel-mode graphics driver,” “user-mode driver,” “user-mode driver framework,” “buffer,” “graphics buffer,” “task,” “process,” “operation,” “software application,” “game,” etc. should not be construed as limiting the embodiments to software or devices bearing that label in products or documents outside of this document.

[0246] It is foreseeable that any number and type of components can be added to and / or removed from the convolution mechanism 2010 to facilitate various embodiments including the addition, removal, and / or enhancement of certain features. For the sake of simplicity, clarity, and ease of understanding of the convolution mechanism 2010, many standard and / or known components, such as components of computing devices, are not shown or discussed herein. It is foreseeable that the embodiments described herein are not limited to any technology, topology, system, architecture, and / or standard, and are dynamic enough to adopt and adapt to any future changes.

[0247] Figure 22A A method 2200 for facilitating the generation and storage of geometry-based filters for use in intelligent deep learning convolutions, according to one embodiment, is illustrated. For the sake of brevity, previous references may not be discussed or repeated herein. Figure 1-21 Many details are discussed. Any process related to method 2200 can be executed by processing logic, which may include, for example, [details omitted]. Figure 20 The convolution mechanism 2010 facilitates hardware (such as circuit systems, special-purpose logic, programmable logic, etc.), software (such as instructions running on a processing device), or a combination thereof. For the sake of brevity and clarity, the processes associated with method 2200 are illustrated or described in a linear sequence; however, it is foreseeable that any number of processes may be executed in parallel, asynchronously, or in a different order.

[0248] For reference Figure 21 As described, method 2200 begins at box 2201, as... Figure 21 The detection and selection logic 2101 facilitates the detection and selection of images containing primitive geometric shapes as customized training inputs, where the images are images of objects (e.g., people, animals, trees, mountains, oceans, machines, rain, etc.) to which the neural network is to be trained. As by Figure 21 The filter logic 2103 and the training and fine-tuning logic 2111 facilitate the generation of neural network weights at box 2203 to provide filters based on a customized training input image and the geometry revealed by that image.

[0249] As by Figure 21Facilitated by filter logic 2103, at block 2205, one or more visual inspection tools are used to analyze these filters. This analysis of the filters may include identifying common geometries shared by multiple filters, so that groups of filters can subsequently be classified based on their corresponding common geometries. At block 2207, as... Figure 21 The filter logic 2103 facilitates the subsequent classification of analyzed filters based on their common geometry, storing them in boxes corresponding to their common geometry. For example, all filters associated with the geometry of a circle are grouped together as a library and stored in boxes labeled or named "circle". These boxes are stored... Figure 21 In one or more databases 2130.

[0250] Figure 22B A geometry-based training method 2250 for filters and neural networks according to one embodiment is shown. For the sake of brevity, previous references may not be discussed or repeated herein. Figure 1-22A Many details are discussed. Any process related to method 2250 can be executed by processing logic, which may include, for example, [details omitted]. Figure 20 The convolution mechanism 2010 facilitates hardware (such as circuit systems, special-purpose logic, programmable logic, etc.), software (such as instructions running on a processing device), or a combination thereof. For the sake of brevity and clarity, the processes associated with method 2250 are illustrated or described in a linear sequence; however, it is foreseeable that any number of processes may be executed in parallel, asynchronously, or in a different order.

[0251] For reference Figure 21 As described, method 2250 begins at box 2251, wherein, as by Figure 21 The detection and selection logic 2101 facilitates the training of a detection and selection neural network on an input image of an object (e.g., a television). As facilitated by... Figure 21 The detection and selection logic 2101 further facilitates, at box 2253, the selection of filters from bins that match the geometry of the object. For example, if the object being trained on by the neural network is a widescreen television, a bin labeled “rectangular” is selected because any filters contained in that bin may involve a rectangular shape.

[0252] In one embodiment, at block 2255, as by Figure 21 The training and fine-tuning logic 2111 facilitates the subsequent training of the selected filter based on the corresponding geometry using any values ​​or data obtained from the bin. At box 2257, as facilitated by... Figure 21The training and fine-tuning logic 2111 further facilitates the use of geometry-trained filters to initiate geometry-specific training of the neural network of an object, allowing for faster, more efficient, and targeted training of the neural network based on the object's geometry.

[0253] Figure 22C A method 2270 for facilitating the separation of convolutional filters for intelligent convolution in deep learning, according to one embodiment, is shown. For the sake of brevity, previous references may not be discussed or repeated herein. Figure 1-22B Many details are discussed. Any process related to method 2270 can be performed by processing logic, which may include, for example, [details omitted]. Figure 20 The convolution mechanism 2010 facilitates hardware (such as circuit systems, special-purpose logic, programmable logic, etc.), software (such as instructions running on a processing device), or a combination thereof. For the sake of brevity and clarity, the processes associated with method 2270 are illustrated or described in a linear sequence; however, it is foreseeable that any number of processes may be executed in parallel, asynchronously, or in a different order.

[0254] For reference Figure 21 As described, method 2250 begins at box 2271, as by Figure 21 The detection and selection logic 2101 facilitates this, where the detection and selection neural network is trained on an input image of an object (e.g., a television). At box 2272, as facilitated by... Figure 21 The detection and selection logic 2101 further facilitates the identification of the various layers of the neural network, including upper and lower layers, as well as any filters associated with the lower layers.

[0255] At box 2273, as shown by Figure 21 The separable convolutional logic 2109 facilitates the subsequent separation or division of any lower-level filters into smaller lower-level filters, while any upper-level filters remain in their current form. For example, a small lower-level 2D filter can be divided into two smaller lower-level 1D filters. At block 2275, as... Figure 21 The training and fine-tuning logic 2111 facilitates the training of the resulting neural network of the object using pairs of separated filters (e.g., two 1D filters), where each pair of separated filters is followed by a dedicated ReLU unit.

[0256] Figure 23A A transaction sequence 2300 for geometry-based intelligent convolution in deep learning is illustrated according to one embodiment. For the sake of brevity, previous references may not be discussed or repeated herein. Figure 1-22C Many details are discussed. It is foreseeable that the embodiments are not limited to this illustration. For the sake of brevity, previous references may be omitted or repeated thereafter. Figure 1-22CMany details are discussed. Any process related to transaction sequence 2300 can be executed by processing logic, which may include, for example, [details omitted]. Figure 20 The convolution mechanism 2010 facilitates hardware (such as circuit systems, special-purpose logic, programmable logic, etc.), software (such as instructions running on a processing device), or a combination thereof. For the sake of brevity and clarity, the processes associated with the transaction sequence 2300 are illustrated or described in a linear sequence manner; however, it is foreseeable that any number of processes may be executed in parallel, asynchronously, or in a different order.

[0257] Transaction sequence 2300 begins with receiving an input image 2301 of an object, wherein the input image 2301 contains primitive geometry 2303 associated with the object, and weights are generated to provide filters A 2305 and N 2307 based on the geometry 2303. For example, a neural network is trained for the object identified in the input image 2301. In one embodiment, visual tools are used to analyze filters A 2305 and N 2307 to find common shapes in the geometry 2303. For example, any filters AN 2305-2307 associated with common geometry of geometry 2303 are categorized into geometry-based groups and stored in their respective bin(s) 2311. For example, bin(s) 2311 are associated with geometry(s) 2303 such that each bin(s) 2311 is assigned a geometry from geometry(s) 2303.

[0258] In one embodiment, filters AN 2305-2307 (2313) are initialized based on values ​​obtained from the corresponding bins 2311 and any user input 2319. When filters A-N 2305-2307 have been trained based on the geometries 2303, geometry-specific training of the neural network is initiated (2315), resulting in the output 2317 of the geometry-specific trained neural network.

[0259] Figure 23B A transaction sequence 2350 for intelligent convolution in deep learning is illustrated according to one embodiment. For the sake of brevity, previous references may not be discussed or repeated herein. Figure 1-23A Many details are discussed. It is foreseeable that the embodiments are not limited to this illustration. For the sake of brevity, previous references may be omitted or repeated thereafter. Figure 1-23A Many details are discussed. Any process related to transaction sequence 2350 can be executed by processing logic, which may include, for example, [details omitted]. Figure 20The convolution mechanism 2010 facilitates hardware (such as circuit systems, special-purpose logic, programmable logic, etc.), software (such as instructions running on a processing device), or a combination thereof. For the sake of brevity and clarity, the processes associated with the transaction sequence 2350 are illustrated or described in a linear sequence manner; however, it is foreseeable that any number of processes may be executed in parallel, asynchronously, or in different orders.

[0260] Transaction sequence 2350 begins by detecting and selecting lower-level 2D convolutional filters A 2351 and N 2353 for the neural network being trained on the target. In one embodiment, the 2D convolutional filters A 2351 and N 2353 are separated or divided into 1D convolutional filter pairs A1 2361, A2 2363 and N1 2365, N2 2367, respectively. These separated 1D convolutional filters 2361-2367 are then used to train the neural network (2371), thereby producing a trained neural network 2373.

[0261] Figure 24 A structural setup for performing separable convolutions according to one embodiment is shown. For the sake of brevity, previous references may not be discussed or repeated herein. Figure 1-23B Many details are discussed. It is foreseeable that the embodiments are not limited to this example.

[0262] As previously referenced Figure 21 As described, in one embodiment, the separation of convolutions can be performed in two stages, stage 1 2401 and stage 2 2451, such as training a neural network in both stages 2401 and 2451. For example, as shown, in stage 1 2401, the neural network can be trained using the original structure and dataset, such as a data layer 2403 including 2D convolutional filters (such as 2D convolutional filter 2411). As shown and previously discussed, this lower data layer 2403 of the neural network may include basic extractors and classifiers, such as feature extractor 2405 and feature classifier 2407.

[0263] In one embodiment, stage 2 2451 begins with the detection and identification of feature extractor 2405, and feature classifier 2407 continues retraining at this lower layer of the neural network (2453). In one embodiment, in stage 2 2451, any lower-layer convolutional filters are partitioned or split into pairs of smaller lower-layer convolutional filters; for example, the 2D convolutional filter 2411 from stage 2401 is now shown as split into two 1D convolutional filters 2461A, 2461B. Further, the learning rate of a portion of this lower layer (including the two 1D convolutional filters 2411) is set to positive (2457), while the learning rate of the remainder of the lower layer is set to zero (2459).

[0264] Additional example graphics processing systems

[0265] The details of the embodiments described above can be incorporated into the graphics processing system and device described below. Figures 25 to 29 The graphics processing systems and devices shown illustrate systems and graphics processing hardware capable of implementing any and all of the techniques described above.

[0266] Figure 25 This is a block diagram illustrating a computer system 2500 configured to implement one or more aspects of the embodiments described herein. The computing system 2500 includes a processing subsystem 2501 having one or more processors 2502 and a system memory 2504, the one or more processors and the system memory communicating via an interconnect path, the interconnect path including a memory hub 2505. The memory hub 2505 may be a separate component within a chipset assembly or integrated within one or more processors 2502. The memory hub 2505 is coupled to an I / O subsystem 2511 via a communication link 2506. The I / O subsystem 2511 includes an I / O hub 2507 that enables the computing system 2500 to receive input from one or more input devices 2508. Additionally, the I / O hub 2507 enables a display controller (which may be included in one or more processors 2502) to provide output to one or more display devices 2510A. In one embodiment, one or more display devices 2510A coupled to the I / O hub 2507 may include local display devices, internal display devices, or embedded display devices.

[0267] In one embodiment, the processing subsystem 2501 includes one or more parallel processors 2512 coupled to a memory hub 2505 via a bus or other communication link 2513. The communication link 2513 can be one of any number of standards-based communication link technologies or protocols (such as, but not limited to, PCI Express), or a vendor-specific communication interface or communication architecture. In one embodiment, the one or more parallel processors 2512 form a computation-centric parallel or vector processing system including a large number of processing cores and / or processing clusters such as integrated many-core (MIC) processors. In one embodiment, the one or more parallel processors 2512 form a graphics processing subsystem that can output pixels to one of one or more display devices 2510A coupled via an I / O hub 2507. The one or more parallel processors 2512 may also include a display controller and a display interface (not shown) to enable direct connection to one or more display devices 2510B.

[0268] Within the I / O subsystem 2511, system storage unit 2514 can be connected to I / O hub 2507 to provide storage for computing system 2500. I / O switch 2516 can be used to provide an interface mechanism to enable connections between I / O hub 2507 and other components that can be integrated into the platform, such as network adapter 2518 and / or wireless network adapter 2519, as well as various other devices that can be added via one or more plug-in devices 2520. Network adapter 2518 can be an Ethernet adapter or another wired network adapter. Wireless network adapter 2519 can include one or more of Wi-Fi, Bluetooth, Near Field Communication (NFC), or other network devices including one or more radio devices.

[0269] The computing system 2500 may include other components not explicitly shown, such as USB or other port connectors, optical storage drives, video capture devices, etc., which may also be connected to the I / O hub 2507. Figure 25 The communication paths for interconnecting various components can be implemented using any suitable protocol, such as a PCI (Peripheral Component Interconnect) based protocol (e.g., PCI-Express), or (multiple) other bus or point-to-point communication interfaces and / or protocols such as NV-Link high-speed interconnect or interconnect protocols known in the art.

[0270] In one embodiment, one or more parallel processors 2512 incorporate circuitry optimized for graphics and video processing, including, for example, video output circuitry, and said circuitry constitutes a graphics processing unit (GPU). In another embodiment, one or more parallel processors 2512 incorporate circuitry optimized for general-purpose processing while retaining the underlying computing architecture described in more detail herein. In yet another embodiment, components of the computing system 2500 may be integrated with one or more other system elements on a single integrated circuit. For example, one or more parallel processors 2512, a memory hub 2505, processor(s)2502, and an I / O hub 2507 may be integrated into a system-on-a-chip (SoC) integrated circuit. Alternatively, components of the computing system 2500 may be integrated into a single package to form a system-in-package (SIP) configuration. In other embodiments, at least a portion of the components of the computing system 2500 may be integrated into a multi-chip module (MCM) that may interconnect with other MCMs to form a modular computing system.

[0271] It should be understood that the computing system 2500 shown herein is exemplary and variations and modifications are possible. The connection topology can be modified as needed, including the number and arrangement of bridges, the number of processors(multiple) 2502, and the number of parallel processors(multiple) 2512. For example, in some embodiments, system memory 2504 is connected directly to processors(multiple) 2502 instead of via bridges, while other devices communicate with system memory 2504 via memory hub 2505 and processors(multiple) 2502. In other alternative topologies, parallel processors(multiple) 2512 are connected to I / O hub 2507 or directly to one or more processors 2502, instead of to memory hub 2505. In other embodiments, I / O hub 2507 and memory hub 2505 may be integrated into a single chip. Some embodiments may include two or more groups of processors(multiple) 2502 attached via multiple sockets, which may be coupled to two or more instances of parallel processors(multiple) 2512.

[0272] Some specific components shown in this document are optional and may not be included in all implementations of the computing system 2500. For example, any number of plug-in cards or peripheral devices may be supported, or some components may be omitted. Furthermore, some architectures may be described using different terminology. Figure 25 Similar components are shown. For example, in some architectures, the memory hub 2505 may be referred to as the Northbridge, while the I / O hub 2507 may be referred to as the Southbridge.

[0273] Figure 26A A parallel processor 2600 according to an embodiment is illustrated. Various components of the parallel processor 2600 can be implemented using one or more integrated circuit devices such as a programmable processor, an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). According to an embodiment, the illustrated parallel processor 2600 is... Figure 25 The illustration shows one or more variants of the parallel processor 2512.

[0274] In one embodiment, the parallel processor 2600 includes a parallel processing unit 2602. The parallel processing unit includes an I / O unit 2604 that enables communication with other devices, including other instances of the parallel processing unit 2602. The I / O unit 2604 may be directly connected to other devices. In one embodiment, the I / O unit 2604 is connected to other devices via a hub or switch interface, such as a memory hub 2505. The connection between the memory hub 2505 and the I / O unit 2604 forms a communication link 2513. Within the parallel processing unit 2602, the I / O unit 2604 is connected to a host interface 2606 and a memory crossbar switch 2616, wherein the host interface 2606 receives commands relating to performing processing operations, and the memory crossbar switch 2616 receives commands relating to performing memory operations.

[0275] When host interface 2606 receives a command buffer via I / O unit 2604, host interface 2606 can route work operations for executing those commands to front end 2608. In one embodiment, front end 2608 is coupled to scheduler 2610, which is configured to assign commands or other work items to processing cluster array 2612. In one embodiment, scheduler 2610 ensures that processing cluster array 2612 is correctly configured and in an active state before assigning tasks to processing clusters in processing cluster array 2612. In one embodiment, scheduler 2610 is implemented via firmware logic executed on a microcontroller. Microcontroller-implemented scheduler 2610 can be configured to perform complex scheduling and work assignment operations at both coarse-grained and fine-grained levels, thereby enabling fast preemption and context switching of threads executing on processing array 2612. In one embodiment, host software can validate workloads scheduled on processing array 2612 via one of a plurality of graphics processing doorbell mechanisms. These workloads can then be automatically distributed by the scheduler 2612 logic within the scheduler microcontroller across the processing array 2610.

[0276] The processing cluster array 2612 may include up to "N" processing clusters (e.g., cluster 2614A, cluster 2614B, up to cluster 2614N). Each cluster 2614A through 2614N of the processing cluster array 2612 can execute a large number of concurrent threads. The scheduler 2610 may use various scheduling and / or work distribution algorithms to allocate work to the clusters 2614A through 2614N of the processing cluster array 2612, and these algorithms may vary depending on the workload caused by each type of program or computation. Scheduling may be handled dynamically by the scheduler 2610, or it may be partially assisted by compiler logic during the compilation of program logic configured to be executed by the processing cluster array 2612. In one embodiment, the different clusters 2614A through 2614N of the processing cluster array 2612 may be assigned to process different types of programs or to perform different types of computations.

[0277] The processing cluster array 2612 can be configured to perform various types of parallel processing operations. In one embodiment, the processing cluster array 2612 is configured to perform general-purpose parallel computing operations. For example, the processing cluster array 2612 may include logic for performing processing tasks including filtering video and / or audio data, performing modeling operations including physical operations, and performing data transformations.

[0278] In one embodiment, the processing cluster array 2612 is configured to perform parallel graphics processing operations. In embodiments where the parallel processor 2600 is configured to perform graphics processing operations, the processing cluster array 2612 may include additional logic for supporting the execution of such graphics processing operations, including but not limited to texture sampling logic, tessellation logic, and other vertex processing logic for performing texture operations. Additionally, the processing cluster array 2612 may be configured to execute shader programs related to graphics processing, such as, but not limited to, vertex shaders, tessellation shaders, geometry shaders, and pixel shaders. The parallel processing unit 2602 may transfer data from system memory via I / O unit 2604 for processing. During processing, the transferred data may be stored in on-chip memory (e.g., parallel processor memory 2622) and then written back to system memory.

[0279] In one embodiment, when the parallel processing unit 2602 is used to perform graphics processing, the scheduler 2610 can be configured to divide the processing workload into tasks of approximately equal size to better distribute graphics processing operations across multiple clusters 2614A to 2614N of the processing cluster array 2612. In some embodiments, portions of the processing cluster array 2612 can be configured to perform different types of processing. For example, a first portion can be configured to perform vertex shading and topology generation, a second portion can be configured to perform tessellation and geometry shading, and a third portion can be configured to perform pixel shading or other screen-space operations to produce a rendered image for display. Intermediate data generated by one or more of clusters 2614A to 2614N can be stored in a buffer to allow intermediate data to be transferred between clusters 2614A to 2614N for further processing.

[0280] During operation, the processing cluster array 2612 may receive processing tasks to be executed via scheduler 2610, which receives commands defining the processing tasks from front-end 2608. For graphics processing operations, processing tasks may include data to be processed, such as surface (patch) data, graph data, vertex data, and / or pixel data, as well as state parameters defining how the data is processed and indices of commands (e.g., which program to execute). Scheduler 2610 may be configured to retrieve indices corresponding to tasks or may receive indices from front-end 2608. Front-end 2608 may be configured to ensure that processing cluster array 2612 is configured to be active before a workload specified by an incoming command buffer (e.g., a batch buffer, a stack buffer, etc.) is initiated.

[0281] Each of one or more instances of the parallel processing unit 2602 may be coupled to the parallel processor memory 2622. The parallel processor memory 2622 may be accessed via a memory crossbar switch 2616, which receives memory requests from the processing cluster array 2612 and the I / O unit 2604. The memory crossbar switch 2616 may access the parallel processor memory 2622 via a memory interface 2618. The memory interface 2618 may include a plurality of partition units (e.g., partition units 2620A, 2620B, up to partition units 2620N), each of which may be coupled to a portion (e.g., a memory cell) of the parallel processor memory 2622. In one implementation, the number of partition units 2620A to 2620N is configured to be equal to the number of memory cells, such that a first partition unit 2620A has a corresponding first memory cell 2624A, a second partition unit 2620B has a corresponding memory cell 2624B, and an Nth partition unit 2620N has a corresponding Nth memory cell 2624N. In other embodiments, the number of partition units 2620A to 2620N may not be equal to the number of memory devices.

[0282] In various embodiments, memory cells 2624A to 2624N may include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. In one embodiment, memory cells 2624A to 2624N may also include 3D stacked memory, including but not limited to high bandwidth memory (HBM). Those skilled in the art will understand that the specific implementation of memory cells 2624A to 2624N can vary and can be selected from one of a variety of conventional designs. Render targets, such as frame buffers or texture maps, may be stored on memory cells 2624A to 2624N, thereby allowing partitioning cells 2620A to 2620N to write portions of each render target in parallel to efficiently utilize the available bandwidth of parallel processor memory 2622. In some embodiments, to support a unified memory design utilizing system memory along with local cache memory, local instances of parallel processor memory 2622 may be excluded.

[0283] In one embodiment, any one of clusters 2614A to 2614N of the processing cluster array 2612 can process data to be written to any one of the memory cells 2624A to 2624N within the parallel processor memory 2622. The memory crossbar switch 2616 can be configured to pass the output of each cluster 2614A to 2614N to any partition cell 2620A to 2620N or another cluster 2614A to 2614N, which can perform additional processing operations on the output. Each cluster 2614A to 2614N can communicate with the memory interface 2618 via the memory crossbar switch 2616 to perform read or write operations for various external memory devices. In one embodiment, the memory crossbar switch 2616 may be connected to the memory interface 2618 to communicate with the I / O unit 2604, and may be connected to a local instance of the parallel processor memory 2622, thereby enabling processing units within different processing clusters 2614A to 2614N to communicate with system memory or other memory that is not local to the parallel processing unit 2602. In one embodiment, the memory crossbar switch 2616 may use virtual channels to separate traffic flows between clusters 2614A to 2614N and partition units 2620A to 2620N.

[0284] While a single instance of the parallel processing unit 2602 is shown within the parallel processor 2600, any number of instances of the parallel processing unit 2602 can also be included. For example, multiple instances of the parallel processing unit 2602 can be provided on a single plug-in card, or multiple plug-in cards can be interconnected. Even if different instances have different numbers of processing cores, different amounts of local parallel processor storage, and / or other configuration differences, different instances of the parallel processing unit 2602 can be configured to operate interactively. For example, in one embodiment, some instances of the parallel processing unit 2602 may include higher precision floating-point units relative to other instances. Systems incorporating one or more instances of the parallel processing unit 2602 or the parallel processor 2600 can be implemented in various configurations and form factors, including but not limited to desktop computers, laptop or handheld personal computers, servers, workstations, game consoles, and / or embedded systems.

[0285] Figure 26B This is a block diagram of a partitioning system 2620 according to an embodiment. In one embodiment, the partitioning system 2620 is... Figure 26AAn example of one of partition units 2620A to 2620N. As shown, partition unit 2620 includes an L2 cache 2621, a frame buffer interface 2625, and a ROP 2626 (raster operation unit). The L2 cache 2621 is a read / write cache configured to perform load and store operations received from memory crossbar switch 2616 and ROP 2626. Read miss and urgent write-back requests are output from the L2 cache 2621 to the frame buffer interface 2625 for processing. Updates can also be sent to the frame buffer via the frame buffer interface 2625 for processing. In one embodiment, the frame buffer interface 2625 interacts with one of the memory cells in the parallel processor memory, such as memory cells 2624A to 2624N of FIG. 26 (e.g., within parallel processor memory 2622).

[0286] In graphics applications, the ROP 2626 is a processing unit that performs raster operations such as stencil printing, z-testing, blending, etc. The ROP 2626 then outputs processed graphics data, which is stored in graphics memory. In some embodiments, the ROP 2626 includes compression logic for compressing depth or color data written to memory and decompressing depth or color data read from memory. The compression logic may be lossless compression logic utilizing one or more of a variety of compression algorithms. The type of compression performed by the ROP 2626 can vary based on the statistical characteristics of the data to be compressed. For example, in one embodiment, Δ color compression is performed on the depth and color data on a tile-by-tile basis.

[0287] In some embodiments, ROP 2626 is included within each processing cluster (e.g., clusters 2614A to 2614N of FIG. 26) instead of within partition unit 2620. In this embodiment, read and write requests for pixel data are transmitted via memory crossbar switch 2616 instead of pixel fragment data. The processed graphics data can be displayed on a display device such as... Figure 25 On one or more display devices 2510, routed by processor(s) 2502 for further processing, or by... Figure 26A One of the processing entities within the parallel processor 2600 is routed for further processing.

[0288] Figure 26CThis is a block diagram of a processing cluster 2614 within a parallel processing unit according to an embodiment. In one embodiment, the processing cluster is an instance of one of the processing clusters 2614A to 2614N of FIG. 26. The processing cluster 2614 can be configured to execute multiple threads in parallel, where the term "thread" refers to an instance of a specific program executing on a specific input dataset. In some embodiments, a Single Instruction Multiple Data (SIMD) instruction issuing technique is used to support the parallel execution of a large number of threads without providing multiple independent instruction units. In other embodiments, a Single Instruction Multiple Threading (SIMT) technique is used to support the parallel execution of a large number of substantially synchronous threads using a common instruction unit configured to issue instructions to a set of processing engines within each of the processing cluster. Unlike the SIMD execution mechanism, where all processing engines typically execute the same instructions, SIMT execution allows different threads to more easily follow divergent execution paths through a given thread program. Those skilled in the art will understand that the SIMD processing mechanism represents a subset of the functionality of the SIMT processing mechanism.

[0289] The operation of processing cluster 2614 can be controlled via pipeline manager 2632, which distributes processing tasks to SIMT parallel processors. Pipeline manager 2632 receives instructions from scheduler 2610 of Figure 26 and manages the execution of those instructions via graphics multiprocessor 2634 and / or texture unit 2636. The graphics multiprocessor 2634 shown is an exemplary instance of a SIMT parallel processor. However, various types of SIMT parallel processors with different architectures can be included within processing cluster 2614. One or more instances of graphics multiprocessor 2634 can be included within processing cluster 2614. Graphics multiprocessor 2634 can process data, and data cross switch 2640 can be used to distribute processed data to one of several possible destinations, including other shading units. Pipeline manager 2632 can facilitate the distribution of processed data by specifying destinations for data to be distributed via data cross switch 2640.

[0290] Each graphics multiprocessor 2634 within the processing cluster 2614 may include the same group of functional execution logic (e.g., arithmetic logic units, load-memory units, etc.). The functional execution logic can be configured in a pipelined manner, where new instructions can be issued before completing previous instructions. The functional execution logic supports various operations, including integer and floating-point arithmetic, comparison operations, Boolean operations, bit shifting, and calculations of various algebraic functions. In one embodiment, the same functional unit hardware can be used to perform different operations, and any combination of functional units can exist.

[0291] Instructions transmitted to the processing cluster 2614 constitute threads. A group of threads executing on a set of parallel processing engines is a thread group. Thread groups execute the same program on different input data. Each thread within a thread group can be assigned to a different processing engine within the graphics multiprocessor 2634. A thread group may include fewer threads than the number of processing engines within the graphics multiprocessor 2634. When a thread group includes fewer threads than the number of processing engines, one or more of the processing engines may be idle during the cycle of processing the thread group. A thread group may also include more threads than the number of processing engines within the graphics multiprocessor 2634. When a thread group includes more threads than the number of processing engines within the graphics multiprocessor 2634, processing can be performed on consecutive clock cycles. In one embodiment, multiple thread groups can be executed simultaneously on the graphics multiprocessor 2634.

[0292] In one embodiment, the graphics multiprocessor 2634 includes an internal cache memory for performing load and store operations. In one embodiment, the graphics multiprocessor 2634 may forgo the internal cache and instead use a cache memory (e.g., L1 cache 2608) within the processing cluster 2614. Each graphics multiprocessor 2634 may also access an L2 cache within a partition unit (e.g., partition units 2620A to 2620N of FIG. 26) shared across all processing clusters 2614, and this cache can be used to transfer data between threads. The graphics multiprocessor 2634 may also access off-chip global memory, which may include one or more of local parallel processor memory and / or system memory. Any memory outside of the parallel processing unit 2602 may be used as global memory. Embodiments where the processing cluster 2614 includes multiple instances of the graphics multiprocessor 2634 may share common instructions and data that can be stored in the L1 cache 2608.

[0293] Each processing cluster 2614 may include an MMU 2645 (Memory Management Unit) configured to map virtual addresses to physical addresses. In other embodiments, one or more instances of the MMU 2645 may reside within the memory interface 2618 of FIG. 26. The MMU 2645 includes a set of page table entries (PTEs) for mapping virtual addresses to physical addresses and optionally cache line indexes. The MMU 2645 may include an address translation lookahead buffer (TLB) or cache that may reside within the graphics multiprocessor 2634 or the L1 cache or processing cluster 2614. Physical addresses are processed to distribute surface data access locality to achieve efficient request interleaving between partition units. Cache line indexes can be used to determine whether a request for a cache line is a hit or a miss.

[0294] In graphics and computing applications, processing cluster 2614 can be configured such that each graphics multiprocessor 2634 is coupled to texture unit 2636 to perform texture mapping operations, such as determining texture sample locations, reading texture data, and filtering texture data. Texture data is read from an internal texture L1 cache (not shown) or, in some embodiments, from an L1 cache within the graphics multiprocessor 2634, and is retrieved as needed from an L2 cache, local parallel processor memory, or system memory. Each graphics multiprocessor 2634 outputs a processed task to data crossover switch 2640 to provide the processed task to another processing cluster 2614 for further processing or to store the processed task in an L2 cache, local parallel processor memory, or system memory via memory crossover switch 2616. Pre-ROP 2642 (Pre-Raster Operation Unit) is configured to receive data from graphics multiprocessor 2634 and direct the data to ROP units, which can be located using partitioned units (e.g., partitioned units 2620A to 2620N of FIG. 26) as described herein. The preROP 2642 unit can optimize color mixing, organize pixel color data, and perform address translation.

[0295] It should be understood that the core architecture described herein is exemplary and variations and modifications are possible. For example, any number of processing units such as the graphics multiprocessor 2634, texture unit 2636, preROP 2642, etc., can be included within the processing cluster 2614. Furthermore, although only one processing cluster 2614 is shown, the parallel processing units as described herein can include any number of instances of the processing cluster 2614. In one embodiment, each processing cluster 2614 can be configured to operate independently of other processing clusters 2614 using separate and different processing units, L1 cache, etc.

[0296] Figure 26D A graphics multiprocessor 2634 according to one embodiment is illustrated. In such an embodiment, the graphics multiprocessor 2634 is coupled to a pipeline manager 2632 of a processing cluster 2614. The graphics multiprocessor 2634 has an execution pipeline including, but not limited to, an instruction cache 2652, an instruction unit 2654, an address mapping unit 2656, a register file 2658, one or more general-purpose graphics processing unit (GPGPU) cores 2662, and one or more load / store units 2666. The GPGPU cores 2662 and the load / store units 2666 are coupled to a cache memory 2672 and a shared memory 2670 via a memory and cache interconnect 2668.

[0297] In one embodiment, instruction cache 2652 receives a stream of instructions to be executed from pipeline manager 2632. These instructions are cached in instruction cache 2652 and dispatched for execution by instruction unit 2654. Instruction unit 2654 can dispatch instructions as thread groups (e.g., threads), with each thread in the thread group assigned to a different execution unit within GPGPU core 2662. Instructions can access any of the local, shared, or global address spaces by specifying an address within a unified address space. Address mapping unit 2656 can be used to translate addresses in the unified address space into different memory addresses accessible by load / store unit 2666.

[0298] Register file 2658 provides a set of registers for the functional units of graphics multiprocessor 2624. Register file 2658 provides temporary storage for operands on data paths connected to functional units of graphics multiprocessor 2624 (e.g., GPGPU core 2662, load / store unit 2666). In one embodiment, register file 2658 is partitioned among each functional unit such that each functional unit is allocated a dedicated portion of register file 2658. In one embodiment, register file 2658 is partitioned between different meridians being executed by graphics multiprocessor 2624.

[0299] Each GPGPU core 2662 may include a floating-point unit (FPU) and / or an integer arithmetic logic unit (ALU) for executing instructions of the graphics multiprocessor 2624. According to embodiments, the architecture of the GPGPU core 2662 may be similar or different. For example, in one embodiment, a first portion of the GPGPU core 2662 includes a single-precision FPU and an integer ALU, while a second portion of the GPGPU core includes a double-precision FPU. In one embodiment, the FPU may implement the IEEE 754-2008 floating-point arithmetic standard or enable variable-precision floating-point arithmetic. Additionally, the graphics multiprocessor 2624 may also include one or more fixed-function or special-function units for performing specific functions such as copying rectangles or pixel blending operations. In one embodiment, one or more of the GPGPU cores may also contain fixed-function or special-function logic.

[0300] In one embodiment, the GPGPU core 2662 includes SIMD logic capable of executing a single instruction on multiple sets of data. In one embodiment, the GPGPU core 2662 can physically execute SIMD4, SIMD8, and SIMD16 instructions, and logically execute SIMD1, SIMD2, and SIMD32 instructions. SIMD instructions for the GPGPU core can be generated by a shader compiler at compile time, or automatically generated when executing a program written and compiled for a Single Program Multiple Data (SPMD) or SIMT architecture. Multiple threads of a program configured for a SIMT execution model can be executed via a single SIMD instruction. For example, in one embodiment, eight SIMD threads performing the same or similar operations can be executed in parallel via a single SIMD8 logic unit.

[0301] The memory and cache interconnect 2668 is an interconnect network that connects each functional unit of the graphics multiprocessor 2624 to the register file 2658 and shared memory 2670. In one embodiment, the memory and cache interconnect 2668 is a cross-switch interconnect that allows the load / store unit 2666 to perform load and store operations between the shared memory 2670 and the register file 2658. The register file 2658 can operate at the same frequency as the GPGPU core 2662, thus data transfer between the GPGPU core 2662 and the register file 2658 has very low latency. The shared memory 2670 can be used to implement communication between threads executing on functional units within the graphics multiprocessor 2634. For example, the cache memory 2672 can be used as a data cache to cache texture data communicated between functional units and texture units 2636. The shared memory 2670 can also be used as a cached, managed program. In addition to the automatically cached data stored in cache memory 2672, threads executing on GPGPU core 2662 can also programmatically store data in shared memory.

[0302] Figures 27A to 27B An additional graphics multiprocessor according to an embodiment is shown. The graphics multiprocessors 2725 and 2750 shown are... Figure 26C Variants of the 2634 graphics multiprocessor. The 2725 and 2750 graphics multiprocessors shown can be configured as streaming multiprocessors (SM) capable of executing a large number of execution threads simultaneously.

[0303] Figure 27A A graphics multiprocessor 2725 according to an additional embodiment is shown. The graphics multiprocessor 2725 includes, relative to... Figure 26DThe graphics multiprocessor 2634 may include multiple additional instances of its execution resource units. For example, the graphics multiprocessor 2725 may include multiple instances of instruction units 2732A to 2732B, register files 2734A to 2734B, and multiple texture units 2744A to 2744B. The graphics multiprocessor 2725 may also include multiple sets of graphics or compute execution units (e.g., GPGPU cores 2736A to 2736B, GPGPU cores 2737A to 2737B, GPGPU cores 2738A to 2738B) and multiple sets of load / store units 2740A to 2740B. In one embodiment, the execution resource units have a common instruction cache 2730, a texture and / or data cache memory 2742, and a shared memory 2746.

[0304] Various components can communicate via interconnect fabric 2727. In one embodiment, interconnect fabric 2727 includes one or more crossbar switches to enable communication between components of the graphics multiprocessor 2725. In another embodiment, interconnect fabric 2727 is a separate, high-speed network structure layer on which each component of the graphics multiprocessor 2725 is stacked. Components of the graphics multiprocessor 2725 communicate with remote components via interconnect fabric 2727. For example, GPGPU cores 2736A-2736B, 2737A-2737B, and 27378A-2737B can each communicate with shared memory 2746 via interconnect fabric 2727. Interconnect fabric 2727 can arbitrate communication within the graphics multiprocessor 2725 to ensure fair bandwidth allocation among components.

[0305] Figure 27B A graphics multiprocessor 2750 according to an additional embodiment is shown. Figure 26D and Figure 27A As shown, the graphics processor includes multiple sets of execution resources 2756A to 2756D, each set of execution resources including multiple instruction units, register files, GPGPU cores, and load memory units. Execution resources 2756A to 2756D can work with (multiple) texture units 2760A to 2760D to perform texture operations, while sharing instruction cache 2754 and shared memory 2762. In one embodiment, execution resources 2756A to 2756D can share instruction cache 2754, shared memory 2762, and multiple instances of texture and / or data cache memories 2758A to 2758B. Various components can be connected via... Figure 27A The interconnect structure 2727 communicates with the similar interconnect structure 2752.

[0306] Those skilled in the art will understand that Figure 25 , Figures 26A to 26Dand Figures 27A to 27B The architecture described herein is descriptive and does not limit the scope of the embodiments of the invention. Therefore, the techniques described herein can be implemented on any appropriately configured processing unit, including but not limited to: one or more mobile application processors; one or more desktop computer or server central processing units (CPUs), including multi-core CPUs; one or more parallel processing units such as the parallel processing unit 2602 of FIG. 26; and one or more graphics processors or dedicated processing units, without departing from the scope of the embodiments described herein.

[0307] In some embodiments, a parallel processor or GPGPU, as described herein, is communicatively coupled to a host / processor core to accelerate graphics operations, machine learning operations, pattern analysis operations, and various general-purpose GPU (GPGPU) functions. The GPU may be communicatively coupled to the host processor / core via a bus or other interconnect (e.g., high-speed interconnects such as PCIe or NVLink). In other embodiments, the GPU may be integrated on the same package or chip as the core and communicatively coupled to the core via an internal processor bus / interconnect (i.e., within the package or chip). Regardless of how the GPU is connected, the processor core can assign work to the GPU in the form of a sequence of commands / instructions contained in a job descriptor. The GPU then uses dedicated circuitry / logic to efficiently process these commands / instructions.

[0308] Technologies for GPU-to-host processor interconnects

[0309] Figure 28A An exemplary architecture is shown in which multiple GPUs 2810 to 2813 are communicatively coupled to multiple multi-core processors 2805 to 2806 via high-speed links 2840 to 2843 (e.g., bus, point-to-point interconnect, etc.). In one embodiment, the high-speed links 2840 to 2843 support communication throughput of 4GB / s, 30GB / s, 80GB / s, or higher, depending on the implementation. Various interconnect protocols can be used, including but not limited to PCIe 4.0 or 5.0 and NVLink 2.0. However, the basic principles of the invention are not limited to any particular communication protocol or throughput.

[0310] Furthermore, in one embodiment, two or more of GPUs 2810 to 2813 are interconnected via high-speed links 2844 to 2845, which can be implemented using the same or different protocols / links as those used for high-speed links 2840 to 2843. Similarly, two or more of multi-core processors 2805 to 2806 can be connected via high-speed link 2833, which can be a symmetric multiprocessor (SMP) bus operating at speeds of 20GB / s, 30GB / s, 120GB / s, or higher. Alternatively, Figure 28AAll communication between the various system components shown can be accomplished using the same protocol / link (e.g., via a common interconnect structure). However, as mentioned, the basic principles of the invention are not limited to any particular type of interconnect technology.

[0311] In one embodiment, each multi-core processor 2805 to 2806 is communicatively coupled to processor memories 2801 to 2802 via memory interconnects 2830 to 2831, and each GPU 2810 to 2813 is communicatively coupled to GPU memories 2820 to 2823 via GPU memory interconnects 2850 to 2853. Memory interconnects 2830 to 2831 and 2850 to 2853 may utilize the same or different memory access technologies. By way of example and not limitation, processor memories 2801 to 2802 and GPU memories 2820 to 2823 may be volatile memories such as dynamic random access memory (DRAM) (including stacked DRAM), graphics DDR SDRAM (GDDR) (e.g., GDDR5, GDDR6), or high-bandwidth memory (HBM), and / or may be non-volatile memories such as 3D XPoint or Nano-RAM. In one embodiment, one portion of the memory may be volatile memory, while another portion may be non-volatile memory (e.g., using a two-level memory (2LM) hierarchy).

[0312] As described below, although the various processors 2805 to 2806 and GPUs 2810 to 2813 can each be physically coupled to specific memories 2801 to 2802 and 2820 to 2823 respectively, a unified memory architecture can be implemented, in which the same virtual system address space (also known as the “effective address” space) is distributed across all the various physical memories. For example, processor memories 2801 to 2802 can each include 64GB of system memory address space, and GPU memories 2820 to 2823 can each include 32GB of system memory address space (resulting in a total of 256GB of addressable memory space in the example described).

[0313] Figure 28B Additional details are shown regarding the interconnection between a multi-core processor 2807 and a graphics acceleration module 2846 according to one embodiment. The graphics acceleration module 2846 may include one or more GPU chips integrated on a line card coupled to the processor 2807 via a high-speed link 2840. Alternatively, the graphics acceleration module 2846 may be integrated on the same package or chip as the processor 2807.

[0314] The processor 2807 shown in Figure 28 includes multiple cores 2860A to 2860D, each having a translational backstop buffer 2861A to 2861D and one or more caches 2862A to 2862D. These cores may include various other components (e.g., instruction fetch units, branch prediction units, decoders, execution units, reordering buffers, etc.) for executing instructions and processing data not shown to avoid obscuring the basic principles of the invention. Caches 2862A to 2862D may include Level 1 (L1) and Level 2 (L2) caches. Furthermore, one or more shared caches 2826 may be included in the cache hierarchy and shared by the respective groups of cores 2860A to 2860D. For example, one embodiment of the processor 2807 includes 24 cores, each having its own L1 cache, 12 shared L2 caches, and 12 shared L3 caches. In this embodiment, one of the L2 and L3 caches is shared by two adjacent cores. The processor 2807 and graphics accelerator integration module 2846 are connected to the system memory 2841, which may include processor memories 2801 to 2802.

[0315] Consistency is maintained for data and instructions stored in various caches 2862A to 2862D, 2856 and system memory 2841 via inter-core communication through the coherence bus 2864. For example, each cache may have associated cache coherence logic / circuit to communicate via the coherence bus 2864 in response to a detected read or write to a particular cache line. In one implementation, a cache snooping protocol is implemented via the coherence bus 2864 to snoop on cache accesses. Cache snooping / coherence techniques will be well understood by those skilled in the art, and to avoid obscuring the basic principles of the invention, they will not be described in detail here.

[0316] In one embodiment, proxy circuitry 2825 communicatively couples graphics acceleration module 2846 to coherence bus 2864, thereby allowing graphics acceleration module 2846 to participate in cache coherence protocols as a peer of the core. Specifically, interface 2835 provides connectivity to proxy circuitry 2825 via high-speed link 2840 (e.g., PCIe bus, NVLink, etc.), and interface 2837 connects graphics acceleration module 2846 to link 2840.

[0317] In one implementation, the accelerator integrated circuit 2836 provides cache management, memory access, context management, and interrupt management services for multiple graphics processing engines 2831, 2832, and N representing the graphics acceleration module 2846. The graphics processing engines 2831, 2832, and N may each include a separate graphics processing unit (GPU). Alternatively, the graphics processing engines 2831, 2832, and N may include different types of graphics processing engines within the GPU, such as graphics execution units, media processing engines (e.g., video encoders / decoders), samplers, and block image transfer engines. In other words, the graphics acceleration module may be a GPU with multiple graphics processing engines 2831, 2832, and N, or the graphics processing engines 2831 to 2832, and N may be separate GPUs integrated into a common package, line card, or chip.

[0318] In one embodiment, the accelerator integrated circuit 2836 includes a memory management unit (MMU) 2839 for performing various memory management functions such as virtual-to-physical memory translation (also known as effective-to-real memory translation) and memory access protocols for accessing system memory 2841. The MMU 2839 may also include a translation back buffer (TLB) (not shown) for caching virtual / effective-to-physical / real address translations. In one implementation, cache 2838 stores commands and data for efficient access by graphics processing engines 2831 to 2832, N. In one embodiment, the data stored in cache 2838 and graphics memories 2833 to 2834, N is kept consistent with core caches 2862A to 2862D, 2856 and system memory 2811. As mentioned, this can be accomplished via proxy circuitry 2825, which participates in cache coherency mechanisms on behalf of cache 2838 and memories 2833 to 2834, N (e.g., sending updates to cache 2838 related to modifications / accesses to cache lines on processor caches 2862A to 2862D, 2856 and receiving updates from cache 2838).

[0319] A set of registers 2845 stores context data for threads executed by graphics processing engines 2831 to 2832, N, and context management circuitry 2848 manages the thread context. For example, context management circuitry 2848 can perform save and restore operations to save and restore the context of various threads during context switching (e.g., where a first thread is saved and a second thread is stored so that the second thread can be executed by the graphics processing engine). For example, during context switching, context management circuitry 2848 can store the current register value to a designated area in memory (e.g., identified by a context pointer). The context management circuitry can restore the register value upon returning to the context. In one embodiment, interrupt management circuitry 2847 receives and processes interrupts received from system devices.

[0320] In one implementation, the MMU 2839 translates the virtual / effective address from the graphics processing engine 2831 into a physical / actual address in system memory 2811. One embodiment of the accelerator integrated circuit 2836 supports multiple (e.g., 4, 8, 16) graphics accelerator modules 2846 and / or other accelerator devices. The graphics accelerator module 2846 may be dedicated to a single application executing on processor 2807, or it may be shared among multiple applications. In one embodiment, a virtual graphics execution environment is presented, where the resources of graphics processing engines 2831 to 2832, N are shared with multiple applications or virtual machines (VMs). Resources may be subdivided into “shards” allocated to different VMs and / or applications based on processing requirements and priorities associated with the VMs and / or applications.

[0321] Therefore, the accelerator integrated circuit acts as a bridge for the system of the graphics acceleration module 2846, and provides address translation and system memory caching services. Furthermore, the accelerator integrated circuit 2836 can provide virtualization facilities for the host processor to manage the virtualization of the graphics processing engine, interrupts, and memory management.

[0322] Because the hardware resources of the graphics processing engines 2831 to 2832, N are explicitly mapped to the actual address space seen by the host processor 2807, any host processor can directly address these resources using valid address values. In one embodiment, one function of the accelerator integrated circuit 2836 is the physical separation of the graphics processing engines 2831 to 2832, N, so that they appear as independent units on the system.

[0323] As mentioned, in the illustrated embodiment, one or more graphics memories 2833 to 2834, M are coupled to each of graphics processing engines 2831 to 2832, N, respectively. Graphics memories 2833 to 2834, M store instructions and data being processed by each of graphics processing engines 2831 to 2832, N. Graphics memories 2833 to 2834, M can be volatile memories such as DRAM (including stacked DRAM), GDDR memory (e.g., GDDR5, GDDR6), or HBM, and / or can be non-volatile memories such as 3D XPoint or Nano-RAM.

[0324] In one embodiment, to reduce data traffic on link 2840, a biasing technique is used to ensure that the data stored in graphics memories 2833 to 2834, M is the data most frequently used by graphics processing engines 2831 to 2832, N, and preferably not used (or at least infrequently used) by cores 2860A to 2860D. Similarly, the biasing mechanism attempts to keep the data required by the cores (and preferably not graphics processing engines 2831 to 2832, N) within the caches 2862A to 2862D, 2856 of the cores and system memory 2811.

[0325] Figure 28C Another embodiment in which the accelerator integrated circuit 2836 is integrated within the processor 2807 is shown. In this embodiment, graphics processing engines 2831 to 2832, N communicate directly with the accelerator integrated circuit 2836 via high-speed link 2840 through interfaces 2837 and 2835 (this can also utilize any form of bus or interface protocol). The accelerator integrated circuit 2836 can perform operations related to... Figure 28B The operation described is the same, but given its close proximity to the coherence bus 2862 and caches 2862A to 2862D, 2826, it may operate at a higher throughput.

[0326] One embodiment supports different programming models, including a dedicated process programming model (without graphics acceleration module virtualization) and a shared programming model (with virtualization). The shared programming model may include a programming model controlled by the accelerator integrated circuit 2836 and a programming model controlled by the graphics acceleration module 2846.

[0327] In one embodiment of the dedicated process model, graphics processing engines 2831 to 2832, N are dedicated to a single application or process within a single operating system. A single application can centralize requests from other applications to graphics engines 2831 to 2832, N, thereby providing virtualization within a VM / partition.

[0328] In a dedicated process programming model, graphics processing engines 2831 to 2832, N can be shared by multiple VM / application partitions. This shared model requires a hypervisor to virtualize the graphics processing engines 2831 to 2832, N, allowing access by each operating system. For a single-partition system without a hypervisor, the graphics processing engines 2831 to 2832, N are owned by the operating system. In both cases, the operating system can virtualize the graphics processing engines 2831 to 2832, N to provide access to each process or application.

[0329] For a shared programming model, the graphics acceleration module 2846 or the individual graphics processing engines 2831 to 2832, N use a process handle to select process elements. In one embodiment, the process elements are stored in system memory 2811 and can be addressed using the effective address to physical address translation techniques described herein. The process handle may be an implementation-specific value provided to the host process when registering its context with the graphics processing engines 2831 to 2832, N (i.e., invoking system software to add a process element to the process element list). The lower 16 bits of the process handle may be the offset of the process element within the process element list.

[0330] Figure 28D An exemplary accelerator integration slice 2890 is shown. As used herein, a “slice” refers to a designated portion of the processing resources of the accelerator integrated circuit 2836. The application-effective address space 2882 within system memory 2811 stores process elements 2883. In one embodiment, process element 2883 is stored in response to a GPU call 2881 from an application 2880 executing on processor 2807. Process element 2883 contains the processing state of the corresponding application 2880. The job descriptor (WD) 2884 contained in process element 2883 may be a single job requested by the application, or it may contain a pointer to a job queue. In the latter case, WD 2884 is a pointer to a job request queue in the application address space 2882.

[0331] The graphics acceleration module 2846 and / or individual graphics processing engines 2831 to 2832, N can be shared by all or some processes in the system. Embodiments of the invention include infrastructure for establishing a processing state and sending WD2884 to the graphics acceleration module 2846 to begin work in a virtual environment.

[0332] In one implementation, the dedicated process programming model is implementation-specific. In this model, a single process owns either the graphics acceleration module 2846 or a separate graphics processing engine 2831. Since the graphics acceleration module 2846 is owned by a single process, the hypervisor initializes the accelerator integrated circuit 2836 to obtain its assigned partition, and the operating system initializes the accelerator integrated circuit 2836 to obtain its assigned process when the graphics acceleration module 2846 is allocated.

[0333] In operation, the WD acquisition unit 2891 in the accelerator integration slice 2890 acquires the next WD 2884, which includes instructions for work to be performed by one of the graphics processing engines of the graphics acceleration module 2846. As shown, data from the WD 2884 can be stored in register 2845 and used by the MMU 2839, interrupt management circuitry 2847, and / or context management circuitry 2846. For example, one embodiment of the MMU 2839 includes segment / page walk circuitry for accessing segment / page tables 2886 within the OS virtual address space 2885. The interrupt management circuitry 2847 can handle interrupt events 2892 received from the graphics acceleration module 2846. When performing graphics operations, the effective address 2893 generated by the graphics processing engines 2831 to 2832, N is translated into an actual address by the MMU 2839.

[0334] In one embodiment, the same set of registers 2845 is copied for each graphics processing engine 2831 to 2832, N and / or graphics acceleration module 2846, and this set of registers can be initialized by a hypervisor or operating system. Each of these copied registers can be included in the accelerator integration slice 2890. Exemplary registers that can be initialized by a hypervisor are shown in Table 2.

[0335] Table 2 - Management Program Initialization Registers

[0336] 1 Slice Control Register 2 Real Address (RA) Scheduler Region Pointer 3 Authorization mask override register 4 Interruption vector table entry offset 5 Interruption vector table entry limit 6 Status Register 7 Logical partition ID 8 The Real Address (RA) management accelerator utilizes record pointers 9 Storage description register

[0337] Table 3 shows exemplary registers that can be initialized by the operating system.

[0338] Table 3 - Operating System Initialization Registers

[0339]

[0340]

[0341] In one embodiment, each WD 2884 is specific to a particular graphics acceleration module 2846 and / or graphics processing engines 2831 to 2832, N. The WD contains all the information required for the graphics processing engines 2831 to 2832, N to complete their work, or the WD may be a pointer to a memory location where the application has established a queue of work commands to be completed.

[0342] Figure 28E Additional details of one embodiment of the shared model are shown. This embodiment includes a hypervisor physical address space 2898 in which a list of process elements 2899 is stored. The hypervisor physical address space 2898 is accessible via a hypervisor 2896 that virtualizes the graphics acceleration module engine of operating system 2895.

[0343] The shared programming model allows all or some processes from all or some partitions of the system to use the graphics acceleration module 2846. There are two programming models in which the graphics acceleration module 2846 is shared by multiple processes and partitions: time-sliced ​​sharing and direct graphics sharing.

[0344] In this model, the hypervisor 2896 owns the graphics acceleration module 2846 and makes its functionality available to all operating systems 2895. For the graphics acceleration module 2846 to support the virtualization of the hypervisor 2896, the graphics acceleration module 2846 must meet the following requirements: 1) Application job requests must be autonomous (i.e., no need to maintain state between jobs), or the graphics acceleration module 2846 must provide context saving and restoration mechanisms. 2) The graphics acceleration module 2846 guarantees completion of application job requests within a specified time, including any translation errors, or the graphics acceleration module 2846 provides the ability to preempt job processing. 3) When operating in a direct shared programming model, fairness of the graphics acceleration module 2846 within the process must be guaranteed.

[0345] In one embodiment, for the shared model, application 2880 is required to make an operating system 2895 system call using the graphics acceleration module 2846 type, working descriptor (WD), authorization mask register (AMR) value, and context save / restore region pointer (CSRP). The graphics acceleration module 2846 type describes the target acceleration function of the system call. The graphics acceleration module 2846 type can be a system-specific value. The WD is specifically formatted for the graphics acceleration module 2846 and can be in the following forms: graphics acceleration module 2846 command; valid address pointer to a user-defined structure; valid address pointer to a command queue; or any other data structure describing the work to be performed by the graphics acceleration module 2846. In one embodiment, the AMR value is the AMR state for the current process. The value passed to the operating system is similar to that of the application setting the AMR. If the implementation of the accelerator integrated circuit 2836 and the graphics acceleration module 2846 does not support the User Authorization Mask Override Register (UAMOR), the operating system can apply the current UAMOR value to the AMR value before passing the AMR in the hypervisor call. Before placing the AMR in process element 2883, hypervisor 2896 may optionally apply the Current Authorization Mask Override Register (AMOR) value. In one embodiment, CSRP is one of registers 2845 containing the effective address of a region in application address space 2882 for the graphics acceleration module 2846 to save and restore context state. This pointer is optional if saving state between jobs is not required or when a job is preempted. The context save / restore region may be plugged-in system memory.

[0346] Upon receiving a system call, the operating system 2895 can verify that the application 2880 has been registered and authorized to use the graphics acceleration module 2846. The operating system 2895 then uses the information shown in Table 4 to invoke the hypervisor 2896.

[0347] Table 4 - Operating System Call Parameters for the Hypervisor

[0348] 1 Working Descriptor (WD) 2 Authorization Mask Register (AMR) value (may be masked) 3 Valid Address (EA) Context Save / Restore Region Pointer (CSRP) 4 Process ID (PID) and optional thread ID (TID) 5 Virtual address (VA) accelerators utilize record pointers (AURP). 6 Virtual address of the Memory Segment Table Pointer (SSTP) 7 Logical Interrupt Service Number (LISN)

[0349] Upon receiving a call from the management program, the management program 2896 can verify that the operating system 2895 has been registered and authorized to use the graphics acceleration module 2846. The management program 2896 then places the process elements 2883 into a linked list corresponding to the graphics acceleration module 2846 type. The process elements may contain the information shown in Table 5.

[0350] Table 5 - Process Element Information

[0351] 1 Working Descriptor (WD) 2 Authorization Mask Register (AMR) value (may be masked) 3 Valid Address (EA) Context Save / Restore Region Pointer (CSRP) 4 Process ID (PID) and optional thread ID (TID) 5 Virtual address (VA) accelerators utilize record pointers (AURP). 6 Virtual address of the Memory Segment Table Pointer (SSTP) 7 Logical Interrupt Service Number (LISN) 8 Interrupt vector table, exported from hypervisor call parameters 9 Status Register (SR) Value 10 Logical Partition ID (LPID) 11 The Real Address (RA) management accelerator utilizes record pointers 12 Storage Descriptor Register (SDR)

[0352] In one embodiment, the hypervisor initializes multiple registers 2845 of the accelerator integration slice 2890.

[0353] like Figure 28F As shown, one embodiment of the invention employs a unified memory addressable via a common virtual memory address space for accessing physical processor memories 2801-2802 and GPU memories 2820-2823. In this implementation, operations performed on GPUs 2810-2813 utilize the same virtual / effective memory address space to access processor memories 2801-2802 and vice versa, thereby simplifying programmability. In one embodiment, a first portion of the virtual / effective address space is allocated to processor memory 2801, a second portion to second processor memory 2802, a third portion to GPU memory 2820, and so on. The entire virtual / effective memory space (sometimes referred to as the effective address space) is thus distributed across each of processor memories 2801-2802 and GPU memories 2820-2823, thereby allowing any processor or GPU to access any physical memory having a virtual address mapped to said memory.

[0354] In one embodiment, the bias / coherence management circuitry 2894A to 2894E within one or more of the MMUs 2839A to 2839E ensures cache coherence between the host processor (e.g., 2805) and the caches of the GPUs 2810 to 2813, and implements a biasing technique that indicates the physical memory where certain types of data should be stored. Although in Figure 28F Several instances of bias / coherence management circuitry 2894A to 2894E are shown, but bias / coherence circuitry can also be implemented within the MMU of one or more host processors 2805 and / or within the accelerator integrated circuit 2836.

[0355] One embodiment allows the GPU-attached memories 2820 to 2823 to be mapped as part of the system memory and accessed using shared virtual memory (SVM) technology, without suffering the typical performance drawbacks associated with system-wide cache coherence. The ability to access the GPU-attached memories 2820 to 2823 as system memory avoids the heavy cache coherence overhead, providing a favorable operating environment for GPU offloading. This arrangement allows the host processor 2805 software to set operands and access computation results without the overhead of traditional I / O DMA data copies. These traditional copies involve driver calls, interrupts, and memory-mapped I / O (MMIO) accesses, which are inefficient compared to simple memory accesses. Meanwhile, the ability to access the GPU-attached memories 2820 to 2823 without cache coherence overhead can be critical for the execution time of offloading computations. For example, in scenarios with heavy streaming write memory traffic, cache coherence overhead can significantly reduce the effective write bandwidth seen by the GPUs 2810 to 2813. The efficiency of operand setup, the efficiency of result access, and the efficiency of GPU computation all play a crucial role in determining the effectiveness of GPU offloading.

[0356] In one implementation, the choice between GPU bias and host processor bias is driven by a bias tracker data structure. For example, a bias table can be used, which may be a page-granular structure comprising 1 or 2 bits per GPU-attached memory page (i.e., controlled at the memory page level). The bias table can be implemented within the stolen memory range of one or more GPU-attached memories 2820-2823, with or without a bias cache in GPUs 2810-2813 (e.g., caching frequently / recently used entries of the bias table). Alternatively, the entire bias table can be maintained within the GPU.

[0357] In one implementation, the bias table entries associated with each access to GPU-attached memories 2820-2823 are accessed before the actual access to GPU memory, such that: First, local requests from GPUs 2810-2813 to find pages in the GPU bias are directly forwarded to the corresponding GPU memories 2820-2823. Local requests from GPUs to find pages in the host bias are forwarded to processor 2805 (e.g., via a high-speed link as described above). In one embodiment, a request from processor 2805 to find a requested page in the host processor bias completes like a normal memory read. Alternatively, requests for GPU-biased pages can be forwarded to GPUs 2810-2813. If the GPU is not currently using the page, the GPU can convert the page to host processor bias.

[0358] The page bias state can be changed through software-based mechanisms, hardware-assisted software mechanisms, or, for a finite set of cases, hardware-only mechanisms.

[0359] One mechanism for changing the bias state employs an API call (e.g., OpenCL), which in turn invokes the GPU device driver, which then sends a message to the GPU (or enqueues a command descriptor), thereby instructing the GPU to change the bias state. For certain transitions, a cache dump clearing operation is performed on the host machine. This cache dump clearing operation is necessary for transitions from host processor 2805 bias to GPU bias, but not necessary for the reverse transition.

[0360] In one embodiment, cache coherence is maintained by temporarily presenting GPU bias pages that the host processor 2805 cannot cache. To access these pages, the processor 2805 can request access from the GPU 2810, which may grant access immediately or not, depending on its implementation. Therefore, to reduce communication between the processor 2805 and the GPU 2810, it is advantageous to ensure that the GPU bias pages are pages needed by the GPU but not by the host processor 2805, and vice versa.

[0361] Figure 28GA multi-GPU computing system according to an embodiment is illustrated. The multi-GPU computing system may include a processor 2803 coupled to a plurality of GPUs 2814A-2814D via a host interface switch 2804. In one embodiment, the host interface switch 2804 is a PCI Express switch device that couples the processor 2803 to a PCI Express bus, through which the processor 2803 can communicate with the group of GPUs 2814A-2814D. The GPUs 2814A-2814D may be interconnected via a set of high-speed point-to-point GPU-GPU links 2816. The high-speed GPU-to-GPU links may be connected to each of the GPUs 2814A-2814D via dedicated GPU links. The P2P GPU links 2816 enable direct communication between each of the GPUs 2814A-2814D without requiring communication via the host interface bus to which the processor 2803 is connected. Utilizing GPU-to-GPU traffic directed to the P2P GPU link, the host interface bus remains available for system memory access or communication with other instances of the multi-GPU computing system 2800, for example, via one or more network devices. While in the illustrated embodiment, GPUs 2814A-2814D are connected to processor 2803 via host interface switch 2804, in one embodiment, processor 2803 includes direct support for the P2P GPU link 2816 and can be directly connected to GPUs 2814A-2814D.

[0362] Graphics processing pipeline

[0363] Figure 29 A graphics processing pipeline 2900 according to an embodiment is illustrated. In one embodiment, a graphics processor may implement the illustrated graphics processing pipeline 2900. The graphics processor may be included in a parallel processing subsystem such as those described herein. Figure 26A Within the parallel processor 2600, in one embodiment, the parallel processor is Figure 25 Variations of the (multiple) parallel processors 2512. As described herein, various parallel processing systems can be implemented via parallel processing units (e.g., Figure 26A One or more instances of parallel processing units 2602 are used to implement the graphics processing pipeline 2900. For example, shader units (e.g., Figure 27AThe graphics multiprocessor 2634 can be configured to perform the functions of one or more of the vertex processing unit 2904, the tessellation control processing unit 2908, the tessellation evaluation processing unit 2912, the geometry processing unit 2916, and the fragment / pixel processing unit 2924. The functions of the data assembler 2902, primitive assemblers 2906, 2914, 2918, the tessellation unit 2910, the rasterizer 2922, and the raster operation unit 2926 can also be handled by a processing cluster (e.g., Figure 26A Other processing engines and corresponding partition units (e.g., within the processing cluster 2614) Figure 26A The graphics processing pipeline 2900 is executed by partitioning units 260A to 2620N. The graphics processing pipeline 2900 can also be implemented using one or more dedicated processing units. In one embodiment, one or more portions of the graphics processing pipeline 2900 can be executed by parallel processing logic within a general-purpose processor (e.g., a CPU). In one embodiment, one or more portions of the graphics processing pipeline 2900 can access on-chip memory (e.g., parallel processor memory 2622 as shown in FIG. 26) via a memory interface 2928, which can be an instance of memory interface 2618 of FIG. 26.

[0364] In one embodiment, the data assembler 2902 is a processing unit that collects vertex data of surfaces and primitives. The data assembler 2902 then outputs vertex data, including vertex attributes, to the vertex processing unit 2904. The vertex processing unit 2904 is a programmable execution unit that executes a vertex shader program to illuminate and transform the vertex data as specified by the vertex shader program. The vertex processing unit 2904 reads data stored in a cache, local, or system memory for processing vertex data and can be programmed to transform the vertex data from an object-based coordinate representation to world space coordinate space or normalized device coordinate space.

[0365] The first instance of the primitive assembler 2906 receives vertex attributes from the vertex processing unit 2900. The primitive assembler 2906 reads the stored vertex attributes as needed and constructs graphic primitives for processing by the tessellation control processing unit 2908. Graphic primitives include triangles, line segments, points, patches, etc., supported by various graphics processing application programming interfaces (APIs).

[0366] The tessellation control processing unit 2908 treats input vertices as control points for a geometric patch. These control points are transformed from an input representation from the patch (e.g., the patch's basis) into a representation suitable for surface evaluation by the tessellation evaluation processing unit 2912. The tessellation control processing unit 2908 can also calculate tessellation factors for the edges of the geometric patch. The tessellation factor is applied to a single edge and quantifies the view-dependent level of detail associated with the edge. The tessellation unit 2910 is configured to receive the tessellation factors for the edges of the patch and subdivide the patch into multiple geometric primitives, such as lines, triangles, or quadrilaterals, which are then transmitted to the tessellation evaluation processing unit 2912. The tessellation evaluation processing unit 2912 operates on the parameterized coordinates of the subdivided patch to generate a surface representation and vertex attributes associated with each vertex of the geometric primitive.

[0367] A second instance of the primitive assembler 2914 receives vertex attributes from the tessellation evaluation processing unit 2912, reads stored vertex attributes as needed, and constructs graphical primitives for processing by the geometry processing unit 2916. The geometry processing unit 2916 is a programmable execution unit that executes a geometry shader program to transform the graphical primitives received from the primitive assembler 2914 as specified by the geometry shader program. In one embodiment, the geometry processing unit 2916 is programmed to subdivide the graphical primitives into one or more new graphical primitives and calculate parameters for rasterizing the new graphical primitives.

[0368] In some embodiments, the geometry processing unit 2916 can add or remove elements from the geometry flow. The geometry processing unit 2916 outputs parameters and vertices specifying new graphic primitives to the primitive assembler 2918. The primitive assembler 2918 receives the parameters and vertices from the geometry processing unit 2916 and constructs graphic primitives for processing by the viewport scaling, picking, and clipping unit 2920. The geometry processing unit 2916 reads data stored in the parallel processor memory or system memory for processing the geometry data. The viewport scaling, picking, and clipping unit 2920 performs clipping, picking, and viewport scaling, and outputs the processed graphic primitives to the rasterizer 2922.

[0369] Rasterizer 2922 can perform depth picking and other depth-based optimizations. Rasterizer 2922 also performs scan transformations on new graphic primitives to generate segments and outputs these segments and associated overlay data to segment / pixel processing unit 2924. Segment / pixel processing unit 2924 is a programmable execution unit configured to execute segment shader programs or pixel shader programs. Segment / pixel processing unit 2924 transforms segments or pixels received from rasterizer 2922 as specified by the segment or pixel shader program. For example, segment / pixel processing unit 2924 can be programmed to perform operations including but not limited to texture mapping, shading, blending, texture correction, and perspective correction to produce shaded segments or pixels output to raster operation unit 2926. Segment / pixel processing unit 2924 can read data stored in parallel processor memory or system memory for use when processing segment data. Segment or pixel shader programs can be configured to shade at sample, pixel, tile, or other granularities according to a sampling rate configured for the processing unit.

[0370] The raster operation unit 2926 is a processing unit that performs raster operations including but not limited to stencil printing, z-testing, and blending, and outputs pixel data as processed graphic data for storage in a graphics memory (e.g., the parallel processor memory 2622 in FIG. 26, and / or such as...). Figure 25 The system memory 2504 is used for display on one or more display devices 2510 or for further processing by one or more processors 2502 or (a plurality of) parallel processors 2512. In some embodiments, the raster operation unit 2926 is configured to compress z or color data written to memory and decompress z or color data read from memory.

[0371] References to "an embodiment," "an embodiment," "an example embodiment," "various embodiments," etc., indicate that the embodiments(s) thus described may include specific features, structures, or characteristics, but not every embodiment necessarily includes these specific features, structures, or characteristics. Furthermore, some embodiments may have some, all, or none of the features described for other embodiments.

[0372] In the foregoing specification, various embodiments have been described with reference to specific exemplary embodiments. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the embodiments set forth in the appended claims. Therefore, the specification and drawings are to be regarded in an illustrative rather than restrictive sense.

[0373] The term “coupling” and its derivatives may be used in the following description and claims. “Coupling” is used to indicate that two or more elements cooperate or interact with each other, but they may or may not have an intermediate physical or electrical component between them.

[0374] As used in the claims, unless otherwise stated, the use of ordinal adjectives such as “first,” “second,” “third,” etc., to describe common elements merely indicates that different instances of similar elements are mentioned and is not intended to imply that the element so described must be in a given sequence, whether in time, space, hierarchy, or any other way.

[0375] The following terms and / or examples relate to further embodiments or examples. Details from the examples may be used anywhere in one or more embodiments. Various features of different embodiments or examples may be combined in various ways with some of the included features and others that have been excluded to suit a variety of different applications. Examples may include subjects such as: methods; means for performing actions of said methods; at least one machine-readable medium including instructions that, when executed by a machine, cause said machine to perform the actions of said method; or devices or systems for facilitating hybrid communication according to the embodiments and examples described herein.

[0376] Some embodiments relate to Example 1, which includes a device for facilitating intelligent convolution in a machine learning environment, the device comprising: one or more processors, including one or more graphics processors; detection and selection logic for detecting and selecting an input image having a plurality of geometries associated with an object to which the neural network will be trained; filter generation and storage logic (“filter logic”) for generating weights-providing filters based on the plurality of geometries, wherein the filter logic further classifies the filters in the filter group based on a common geometry among the plurality of geometries, and wherein the filter logic further stores the filter group in bins based on the common geometry, wherein each bin corresponds to a geometry.

[0377] Example 2 includes the subject of Example 1, where detection and selection logic is used to select bins based on the geometry of an object, wherein the bins comprise a filter bank associated with the geometry of the object.

[0378] Example 3 includes the topics of Examples 1-2, and further includes training and fine-tuning logic for initializing geometry-based training of the filter bank based on the geometry of the object and the values ​​obtained from the bin.

[0379] Example 4 includes the topics of Examples 1-3, where the training and fine-tuning logic is based on a specific geometry of the neural network trained by a trained filter bank.

[0380] Example 5 includes the topics of Examples 1-4, where detection and selection logic is used to detect layers of a neural network, wherein the layers include upper and lower layers, and the detection and selection logic is further used to detect and identify existing convolutional filters associated with the lower-level layers.

[0381] Example 6 includes the topics of Examples 1-5, and further includes separable convolutional logic for separating existing convolutional filters of a neural network into pairs of new convolutional filters, wherein each new convolutional filter is half the size of the existing convolutional filters, wherein training and fine-tuning logic is used to train the neural network based on the new convolutional filter pairs.

[0382] Example 7 includes the themes of Examples 1-6, in which one or more graphics processors are co-located with one or more application processors on a common semiconductor package.

[0383] Some embodiments relate to Example 8, which includes a method for facilitating intelligent convolution in a machine learning environment, the method comprising: detecting and selecting an input image having a plurality of geometries associated with an object on which a neural network will be trained at a computing device; generating filters that provide weights based on the plurality of geometries; classifying filters in a filter bank based on a common geometry among the plurality of geometries; and storing the filter bank in bins based on the common geometry, wherein each bin corresponds to a geometry.

[0384] Example 9 includes the subject of Example 8, further including selecting bins based on the geometry of an object, wherein the bins include a filter bank associated with the geometry of the object.

[0385] Example 10 includes the topics covered in Examples 8-9, and further includes geometry-based training of the filter bank based on the geometry of the object and values ​​obtained from the bin.

[0386] Example 11 includes the topics covered in Examples 8-10, and further includes geometry shape-specific training of a neural network based on a trained filter bank.

[0387] Example 12 includes the subject of Examples 8-11, and further includes a detection neural network layer, wherein the layer includes an upper layer and a lower layer; and detects and identifies existing convolutional filters associated with the lower-level layer.

[0388] Example 13 includes the subject of Examples 8-12, and further includes separating an existing convolutional filter of a neural network into pairs of new convolutional filters, wherein the size of each new convolutional filter is half that of the existing convolutional filter; and training the neural network based on the pairs of new convolutional filters.

[0389] Example 14 includes the subject matter of Examples 8-13, wherein a computing device includes one or more processors, the processors including one or more graphics processors, the one or more graphics processors being co-located with one or more application processors on a common semiconductor package.

[0390] Some embodiments relate to Example 15, which includes a data processing system comprising a storage device having instructions and a processing device for executing the instructions, the processing device being configured to: detect and select an input image having a plurality of geometries associated with an object on which a neural network will be trained at a computing device; generate filters providing weights based on the plurality of geometries; classify filters in a filter bank based on a common geometry among the plurality of geometries; and store the filter bank in bins based on the common geometry, wherein each bin corresponds to a geometry.

[0391] Example 16 includes the subject of Example 15, wherein the processor device is used to: select bins based on the geometry of an object, wherein the bins include a filter bank associated with the geometry of the object.

[0392] Example 17 includes the topics of Examples 15-16, where the processor device is used to: initialize geometry-based training of the filter bank based on the geometry of the object and the values ​​obtained from the bin.

[0393] Example 18 includes the subject of Examples 15-17, wherein the processor device is used to: initiate geometry-specific training of a neural network based on a trained filter bank.

[0394] Example 19 includes the subject of Examples 15-18, wherein a processor device is used to: separate an existing convolutional filter of a neural network into pairs of new convolutional filters, wherein the size of each new convolutional filter is half that of the existing convolutional filter; and to train the neural network based on the new convolutional filter pairs.

[0395] Example 20 includes the subject of Examples 15-19, wherein the processor device is used to: separate an existing convolutional filter of a neural network into pairs of new convolutional filters, wherein the size of each new convolutional filter is half that of the existing convolutional filter; and to train the neural network based on the new convolutional filter pairs.

[0396] Example 21 includes the subject matter of Examples 8-13, wherein a computing device includes one or more processors, the processors including one or more graphics processors, the one or more graphics processors being co-located with one or more application processors on a common semiconductor package.

[0397] Some embodiments relate to Example 22, which includes an apparatus comprising: means for detecting a lower layer of a convolutional neural network (CNN); means for detecting and selecting an input image having a plurality of geometries associated with an object on which the neural network will be trained at a computing device; means for generating filters that provide weights based on the plurality of geometries; means for classifying filters in a filter bank based on a common geometry among the plurality of geometries; and means for storing the filter bank in bins based on the common geometry, wherein each bin corresponds to a geometry.

[0398] Example 23 includes the subject of Example 22 and further includes means for selecting a box based on the geometry of an object, wherein the box includes a filter bank associated with the geometry of the object.

[0399] Example 24 includes the subject matter of Examples 22-23, and further includes means for initializing geometry-based training of a filter bank based on the geometry of the object and values ​​obtained from the bin.

[0400] Example 25 includes the subject matter of Examples 22-24, and further includes means for geometry-specific training of a neural network based on a trained filter bank.

[0401] Example 26 includes the subject matter of Examples 22-25, and further includes means for: separating an existing convolutional filter of a neural network into pairs of new convolutional filters, wherein the size of each new convolutional filter is half that of the existing convolutional filter; and training the neural network based on the new convolutional filter pairs.

[0402] Example 27 includes the subject matter of Examples 22-26, and further includes means for: separating an existing convolutional filter of a neural network into new pairs of convolutional filters, wherein the size of each new convolutional filter is half that of the existing convolutional filter; and training the neural network based on the new pairs of convolutional filters.

[0403] Example 28 includes the subject matter of Examples 22-27, wherein the device includes one or more processors, the processors including one or more graphics processors, the one or more graphics processors being co-located with one or more application processors on a common semiconductor package.

[0404] Example 29 includes at least one non-transient or tangible machine-readable medium comprising a plurality of instructions which, when executed on a computing device, are used to implement or perform the method of any one of claims or Examples 8-14.

[0405] Example 30 includes at least one machine-readable medium comprising a plurality of instructions which, when executed on a computing device, are used to implement or perform the method as claimed in any one of claims 8-14.

[0406] Example 31 includes a system comprising a mechanism for implementing or performing the method as claimed in any one of claims 8-14.

[0407] Example 32 includes an apparatus comprising means for performing the method as described in any one of claims 8-14.

[0408] Example 33 includes a computing device arranged to implement or perform the method as claimed in any one of claims 8-14.

[0409] Example 34 includes a communication device arranged to implement or perform the method as claimed in any one of claims 8-14.

[0410] Example 35 includes at least one machine-readable medium comprising a plurality of instructions which, when executed on a computing device, are used to implement or perform the method of any of the preceding claims or to implement the device of any of the preceding claims.

[0411] Example 36 includes at least one non-transient or tangible machine-readable medium comprising a plurality of instructions which, when executed on a computing device, are used to implement or perform the method or implement the device described in any of the preceding claims.

[0412] Example 37 includes a system comprising a mechanism for implementing or performing the method of any of the preceding claims or implementing the device of any of the preceding claims.

[0413] Example 38 includes an apparatus comprising means for performing the method described in any of the preceding claims.

[0414] Example 39 includes a computing device arranged to implement or perform the method of any of the preceding claims or to implement the device of any of the preceding claims.

[0415] Example 40 includes a communication device arranged to implement or perform the method described in any of the preceding claims or to implement the device described in any of the preceding claims.

[0416] The accompanying drawings and the foregoing description provide examples of various embodiments. Those skilled in the art will understand that one or more of the described elements may be combined into a single functional element. Alternatively, certain elements may be split into multiple functional elements. Elements from one embodiment may be added to another embodiment. For example, the order of processes described herein may be changed and is not limited to the manner described herein. Furthermore, the actions of any flowchart need not be performed in the order shown; nor is it necessary to perform all of these actions. Moreover, those actions that do not depend on other actions may be performed in parallel with other actions. The scope of the embodiments is by no means limited by these specific examples. Numerous variations (whether or not explicitly given in the specification) are possible, such as differences in structure, scale, and the use of materials. The scope of the embodiments is at least as broad as that given by the appended claims.

Claims

1. A device for facilitating intelligent convolution in machine learning environments, the device comprising: One or more processors, coupled to memory, said one or more processors are used for: Detecting and selecting input images having multiple geometric shapes associated with an object, a neural network is trained for said object; A filter with weights is generated based on the multiple geometries; The filters are classified into filter groups based on a common geometry among the plurality of geometries, wherein the common geometry is identified based on the analysis of lower-level filters in the filters using one or more visual inspection tools, wherein two or more filters in the filters that are shared by the common geometry among the common geometry are grouped into a single filter group in the filter group; as well as The filter banks are stored in bins based on the common geometry, such that bins are selected based on the corresponding geometry of the object, and the bins have filter banks associated with the geometry.

2. The device as described in claim 1, characterized in that, The one or more processors are further configured to initialize geometry-based training of the filter bank based on the geometry of the object and values ​​obtained from the bin.

3. The device as described in claim 1, characterized in that, The one or more processors are further used to initiate geometry-specific training of the neural network based on a trained filter bank.

4. The device as described in claim 1, characterized in that, The one or more processors are further configured to: detect layers of the neural network, wherein the layers include upper and lower layers; and detect and identify existing convolutional filters associated with lower-level layers.

5. The device as described in claim 1, characterized in that, The one or more processors are further configured to: separate existing convolutional filters of the neural network into pairs of new convolutional filters, wherein the size of each new convolutional filter is half that of the existing convolutional filters; and train the neural network based on the pairs of new convolutional filters.

6. The device as described in claim 1, characterized in that, The one or more processors include one or more graphics processors co-located with one or more application processors on a common semiconductor package.

7. A method for facilitating intelligent convolution in machine learning environments, the method comprising: At the computing device, an input image having multiple geometric shapes associated with an object is detected and selected, and a neural network is trained for the object. A filter with weights is generated based on the multiple geometries; The filters are classified into filter groups based on a common geometry among the plurality of geometries, wherein the common geometry is identified based on the analysis of lower-level filters in the filters using one or more visual inspection tools, wherein two or more filters in the filters that are shared by the common geometry among the common geometry are grouped into a single filter group in the filter group; as well as The filter banks are stored in bins based on the common geometry, such that bins are selected based on the corresponding geometry of the object, and the bins have filter banks associated with the geometry.

8. The method as described in claim 7, characterized in that, Further includes: The filter bank is initialized for geometry-based training based on the geometry of the object and the values ​​obtained from the bin.

9. The method as described in claim 7, characterized in that, Further includes: The neural network is then trained based on a trained filter bank, with the training tailored to the geometry.

10. The method as described in claim 7, characterized in that, Further includes: Detecting layers of the neural network, wherein the layers include an upper layer and a lower layer; as well as Detect and identify existing convolutional filters associated with lower-level layers.

11. The method as described in claim 7, characterized in that, Further includes: The existing convolutional filters of the neural network are separated into pairs of new convolutional filters, wherein the size of each new convolutional filter is half that of the existing convolutional filters; and The neural network is trained based on the new convolutional filter pairs.

12. The method as described in claim 7, characterized in that, The computing device includes one or more processors, the one or more processors including one or more graphics processors co-located with one or more application processors on a common semiconductor package.

13. At least one machine-readable medium comprising a plurality of instructions, which, when executed on a computing device, cause the computing device to perform an operation comprising: Detecting and selecting input images having multiple geometric shapes associated with an object, a neural network is trained for said object; A filter with weights is generated based on the multiple geometries; The filters are classified into filter groups based on a common geometry among the plurality of geometries, wherein the common geometry is identified based on the analysis of lower-level filters in the filters using one or more visual inspection tools, wherein two or more filters in the filters that are shared by the common geometry among the common geometry are grouped into a single filter group in the filter group; as well as The filter banks are stored in bins based on the common geometry, such that bins are selected based on the corresponding geometry of the object, and the bins have filter banks associated with the geometry.

14. The machine-readable medium as claimed in claim 13, characterized in that, The operation further includes: initializing geometry-based training of the filter bank based on the geometry of the object and the values ​​obtained from the bin.

15. The machine-readable medium as claimed in claim 13, characterized in that, The operation further includes: initiating geometry-specific training of the neural network based on a trained filter bank.

16. The machine-readable medium as claimed in claim 13, characterized in that, The operation further includes: Detecting layers of the neural network, wherein the layers include an upper layer and a lower layer; and Detect and identify existing convolutional filters associated with lower-level layers.

17. The machine-readable medium as claimed in claim 13, characterized in that, The operation further includes: The existing convolutional filters of the neural network are separated into pairs of new convolutional filters, wherein the size of each new convolutional filter is half that of the existing convolutional filters; and The neural network is trained based on the new convolutional filter pairs, wherein the computing device includes one or more processors, the one or more processors including one or more graphics processors co-located with one or more application processors on a common semiconductor package.

Images