Persistent scratchpad memory for inter-program data exchange

Abstract

The present invention discloses a persistent scratch memory for inter-program data exchange. A technique for sharing data exchange between kernels (per instruction set) executed on a system with multiple processing units is disclosed. In one embodiment, each processing unit includes an on-chip scratch memory that is accessible by the kernels executed on the processing unit. For example, all or part of the scratch memory can be allocated and configured so that the scratch memory can be accessed by multiple kernels in parallel, one or more kernels in serial, or a combination of both.

Description

Technical Field

[0001] The present disclosure relates to sharing memory between programs, and more particularly, to persistent scratchpad memory for inter-program data exchange. Background Art

[0002] Typically, a quantity of on-chip scratch memory (e.g., 16KB, 32KB, or 64KB in size) is allocated from a scratch memory pool and assigned to a set of one or more execution contexts ("SECs") (e.g., an execution context is a thread or process) that execute kernels in a serial and / or parallel manner, where a kernel is an instruction set, such as a program. In one example, a set of one or more execution contexts is a cooperative thread array (CTA). The scratch memory allocated to a SEC is private to the SEC, and once the SEC completes kernel execution, the data stored in the scratch memory is not persistent. In addition, the scratch memory does not have automatic backup memory. Therefore, in conventional systems, data stored in the scratch memory is exchanged between different kernels by having each SEC executing one kernel explicitly copy the data from its assigned scratch pad to global memory and each SEC executing another kernel explicitly retrieve the data from global memory.

[0003] To reach global memory (e.g., DRAM or cache), the memory hierarchy of caches (e.g., L1, L2, etc.) is typically traversed to transfer data through one or more levels of cache. The speed of exchanging data through global memory is at least an order of magnitude slower than the latency of accessing scratch memory. In addition, since the scratch memory is private to each SEC, different SECs executing the same kernel must also exchange data through global memory. Therefore, there is a need to allow SECs and / or kernels to exchange data faster than traditional methods. Summary of the invention

[0004] Fast on-chip memory (e.g., scratch pad) can be shared between different cores executing on the same processing unit by one or more threads or processes, where each processing unit has its own private scratch pad memory pool. In the context of the following specification, a thread (e.g., an execution thread) is an instantiation of an instruction set to be executed. A core can be executed by one or more SECs. Fast on-chip memory can also be shared between different SECs executing the same core on the same processing unit. Shared scratch pad memory can reduce the latency of transferring data between cores or between different SECs compared to transferring data to global memory through a memory hierarchy including one or more levels of cache. Shared scratch pad memory can also reduce the latency of synchronization operations between different SECs and different cores compared to synchronization through a memory hierarchy.

[0005] The present invention discloses a method and system for multiple processing units, the multiple processing units include a first processing unit, the first processing unit includes a first note and a first register file, wherein the first processing unit is capable of executing a first kernel and a second kernel. The multiple processing units also include a second processing unit, the second processing unit includes a second note and a second register file, wherein the second processing unit is capable of executing a third kernel. Scheduler logic is coupled to the multiple processing units, wherein the scheduler circuit assigns kernels to be executed by the multiple processing units, and memory is coupled to the multiple processing units. The parallel processing system is configured to store first data to the first note or the second note so that the first data is not stored in the memory, wherein the first note is configured to allow such a first kernel to store the second data to the first note, to allow such a second kernel to load the second data from the first note, and to prohibit such a third kernel from storing or loading the second data from the first note. The second note is configured to allow such a third kernel to store the third data to the second note, and to prohibit such a first kernel and the second kernel from loading the third data from the second note. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1A is a conceptual diagram illustrating a spatial pipeline for executing kernels A, B, and C, according to one embodiment.

[0007] Figure 1B is a conceptual diagram illustrating a timing pipeline for executing cores A, B, and C according to one embodiment.

[0008] Figure 2 A flow chart of a method for multiple cores to exchange data through a shared memory and a persistent memory according to one embodiment is shown.

[0009] Figure 3 A parallel processing unit according to one embodiment is shown.

[0010] Figure 4A According to an embodiment Figure 3 A general-purpose processing cluster in a parallel processing unit.

[0011] Figure 4B According to an embodiment Figure 3 The memory partitioning unit of the parallel processing unit.

[0012] Figure 5A According to an embodiment Figure 4A Streaming multiprocessor.

[0013] Figure 5B is a conceptual diagram illustrating a processing pipeline that executes cores A, B, and C, according to one embodiment.

[0014] Figure 5C A flowchart of a method for configuring a shared scratchpad memory pool in a processing unit to enable a SEC to exchange data through a shared memory and a persistent memory according to one embodiment is shown.

[0015] Fig. 6A According to one embodiment, the use Figure 3 Conceptual diagram of the processing system implemented by the PPU.

[0016] Figure 6B An exemplary system is shown that can implement the various architectures and / or functionality of the various preceding embodiments. DETAILED DESCRIPTION

[0017] A highly parallel processor, such as a graphics processing unit (GPU), is a computing device capable of executing a very large number of threads in parallel. In the context of the following specification, a thread is a process or execution context. In one embodiment, the highly parallel processor operates as a coprocessor to a main central processing unit (CPU) or host: in other words, the data-parallel, compute-intensive portions of an application running on the host are offloaded to the coprocessor device.

[0018] More precisely, a portion of an application that is executed multiple times but independently on different data can be isolated into a kernel function that is executed on a highly parallel processor as many different threads. To this end, such a function is compiled into a parallel thread execution (PTX) instruction set, and the resulting kernel is converted into the target highly parallel processor instruction set when installed.

[0019] Batches of threads executing a kernel are organized into SEC grids (e.g., CTAs). In one embodiment, SEC implements (Compute Unified Device Architecture) thread block where It is a parallel computing platform and programming model. The PTX programming model is explicitly parallel: a PTX program specifies the execution of a given thread of an array of parallel threads.

[0020] Threads in a SEC execute the same kernel simultaneously and can communicate with each other. To coordinate the communication of threads within a SEC, synchronization points are specified as places where threads wait until all threads in the SEC have reached a specific instruction. Each thread has a unique thread identifier in the SEC. The program uses data parallel decomposition to partition input, work, and results across the threads of the SEC.

[0021] Each SEC thread uses its thread identifier to determine its assigned role, assign specific input and output locations, calculate addresses, and select work to be performed. In one embodiment, the thread identifier is a three-element vector tid (containing elements tid.x, tid.y, and tid.z) that specifies the position of the thread in a one-dimensional, two-dimensional, or three-dimensional SEC. In one embodiment, each thread identifier component ranges from zero to the number of thread IDs in the corresponding SEC dimension. In one embodiment, each SEC has a one-dimensional, two-dimensional, or three-dimensional shape specified by a three-element vector ntid (elements ntid.x, ntid.y, and ntid.z). In one embodiment, the vector ntid specifies the number of threads in each SEC dimension.

[0022] Threads in a SEC are executed in groups called warps in a SIMT (single instruction, multiple threads) fashion. A warp is the largest subset of threads in a single SEC, so threads execute the same instruction at the same time. Threads within a warp are numbered sequentially. The warp size is a machine-dependent constant. In one embodiment, a warp has 32 threads. Some applications can maximize performance by knowing the warp size, so PTX includes a runtime immediate constant, WARP_SZ, which can be used for any instruction that allows an immediate operand.

[0023] There is a maximum number of threads that a SEC can contain, which is limited by the processor logic. However, SECs executing the same kernel can be batched together into a grid of independently executing SECs so that the total number of threads launched in a single kernel call is very large. However, in traditional highly parallel processors, organizing SECs into a grid comes at the expense of reduced thread communication and synchronization, since threads in different SECs cannot communicate and synchronize with each other. Multiple SECs can execute concurrently and in parallel, or sequentially (continuously), depending on the platform.

[0024] Each SEC has a unique SEC identifier (ctaid) in the SEC grid. Each SEC grid has a one-dimensional, two-dimensional, or three-dimensional shape specified by the parameter nctaid. Each grid also has a unique time grid identifier (gridid). In one embodiment, threads can read and use identifiers and parameter values ​​through predefined read-only special registers %tid, %ntid, %ctaid, %nctaid, and %gridid.

[0025] A host issues a series of kernel calls to a highly parallel processing device. Each kernel executes as a batch of threads organized as a SEC grid. The pipeline-based programming model enables control of concurrency and dependencies between different kernels during execution by the highly parallel processor. In one embodiment, the highly parallel processor includes multiple processing units, each of which has a dedicated private scratch memory pool. The pipeline-based programming model allows different kernels being executed by a single processing unit (serial or parallel) to share scratch memory allocated from the private scratch memory pool of the single processing unit.

[0026] Figure 1A is a conceptual diagram illustrating a spatial pipeline 100 for executing kernels A, B, and C according to one embodiment. A highly parallel processor 105 includes a plurality of processing units 110, only one of which is in Figure 1A . The highly parallel processor 105 is coupled to the global memory 120 via a memory hierarchy including one or more memory address mapping, translation and / or cache units (not shown). Unlike loading and storing data from / to the global memory 120 through the memory hierarchy, transferring data to and / or from the shared scratch memory 115 in the processing unit 110 is fast, without copying or moving data between different levels of the memory hierarchy, providing low latency data transfer. A single shared scratch memory 115 is allocated from a private scratch memory pool in the processing unit 110.

[0027] The pipeline-based programming model specifies that kernels A, B, and C execute concurrently, and as kernels A, B, and C are executed, data is exchanged through a single shared scratchpad memory 115. Each kernel is assigned by one or more SECs based on the processing resources required by the kernel. For example, kernel A generates data and stores it to shared scratchpad memory 115, and the data is loaded from shared scratchpad memory 115 and consumed by kernel B and / or kernel C. In one embodiment, before kernel B and / or kernel C loads the data generated and stored by kernel A, the execution of kernel B and / or kernel C is synchronized with kernel A.

[0028] Kernel B may process data generated by kernel A, which is loaded from shared scratch memory 115, and store the processed data back to shared scratch memory 115. Kernel C and / or kernel A may then load the processed data from shared scratch memory 115 for further processing. Finally, the further processed data stored in shared scratch memory 115 by kernel C may be loaded from shared scratch memory 115 by kernel A and / or kernel B. Example applications for which the spatial pipeline 100 may be specified include highly parallel packet processing for networking. When execution of kernels A, B, and C is completed, the shared scratch memory 115 may be deallocated and one or more SECs may be reassigned.

[0029] In addition to supporting low latency data exchange between different cores, the shared scratchpad memory 115 also supports low latency data exchange between different SECs assigned to the same core for SECs executed in each processing unit 110. Example applications can improve performance through low latency data exchange between SECs in the same processing unit 110, including pooling operations and / or reduction operations of neural networks.

[0030] More illustrative information about various optional architectures and features that can implement the above framework according to the needs of the user will now be described. It should be noted that the following information is set forth for illustrative purposes and should not be interpreted as limiting in any way. Any of the following features can be optionally combined with or without excluding other features described.

[0031] Figure 1B is a conceptual diagram illustrating a timing pipeline 130 for executing kernels A, B, and C according to one embodiment. Figure 1A In contrast, the shared scratch memory 115 in the processing unit 110 is replaced with a shared persistent scratch memory 125. Like the shared scratch memory 115, the shared persistent scratch memory 125 also provides low-latency data transfer between cores and / or between SECs without copying or moving data between different levels of the memory hierarchy. A single shared persistent scratch memory 125 is allocated from a private scratch memory pool in the processing unit 110.

[0032] The pipeline-based programming model specifies that kernels A, B, and C execute in series, and data is exchanged through a single shared persistent scratchpad memory 125 as kernels A, B, and C are executed. Each kernel is assigned to one or more SECs based on the processing resources required by the kernel. For example, kernel A generates data and stores it in the shared persistent scratchpad memory 125. The data is then loaded from the shared persistent scratchpad memory 125 and consumed by kernel B. The data stored by kernel A in the shared persistent scratchpad memory 125 is retained even after the execution of kernel A is completed because kernels B and C have not yet completed execution. Therefore, shared memory is not only shared but also persistent. After the execution of kernel A is completed, one or more SECs assigned to kernel A can be reassigned.

[0033] Any SEC executing on a processing unit 110 can access the shared persistent scratchpad memory 125. However, in one embodiment, a SEC executing on a second processing unit 110 within a highly parallel processor 105 cannot access the shared persistent scratchpad memory 125 in the processing unit 110.

[0034] After the execution of kernel A is completed, kernel B processes the data generated by kernel A and stores the processed data in the shared persistent scratch memory 125. The data stored by kernel B in the shared persistent scratch memory 125 is retained even after the execution of kernel B is completed because kernel C has not yet completed execution. However, when the execution of kernel B is completed, one or more SECs assigned to kernel B can be reassigned.

[0035] The processed data stored by kernel B can be loaded from the shared persistent scratch memory 125 by kernel C for further processing. After the execution of kernel C is completed, the shared persistent scratch memory 125 can be deallocated and one or more SECs assigned to kernel C can be reassigned. Example applications that can specify a time pipeline 130 include deep learning inference, which includes multiple highly parallel operations executed sequentially.

[0036] Figure 2 2 shows a flow chart of a method 200 for exchanging data between multiple cores through a shared persistent memory according to one embodiment. In one embodiment, the method 200 may be used to exchange data through a shared scratchpad memory 115 and a shared persistent scratchpad memory 125, respectively. Figure 1A and Figure 1B shown.

[0037] At step 205, processing unit 110 receives a plurality of kernels for execution according to a spatial pipeline, such as spatial pipeline 100. In one embodiment, spatial pipeline 100 may be represented using a graph-based definition to indicate concurrent execution constraints and dependencies. Such a graph-based definition may be converted into one or more kernels including spatial and / or temporal pipelines.

[0038] At step 210, the processing unit determines the processing resources assigned to each core. The processing resources may include one or more functions or operations, registers, per-thread private memory, and a large amount of shared memory. In conventional processing units, the processing resources assigned to any core of the same SEC are equal. In contrast, the processing unit 110 may be configured to allocate different amounts of processing resources to different cores assigned to the same SEC.

[0039] In step 215, processing resources are allocated to each core, and SECs are assigned to satisfy the data exchange constraints defined by the spatial and / or temporal pipelines. In one embodiment, the processing unit 110 obtains the size of the shared memory resources specified for each core. In one embodiment, the size of the shared scratch memory 115 allocated to one or more cores is equal to the maximum shared memory resources specified for one or more cores. In one embodiment, the size of the shared scratch memory 115 allocated to one or more cores is equal to the sum of the shared memory resources specified for one or more cores. In step 220, the SECs assigned to the cores are started.

[0040] At step 225, the processing unit 110 determines that the execution of the kernel has completed, and then deallocates the processing resources in step 230, and the SEC can be assigned to other kernels. In one embodiment, the shared scratch memory 115 allocated to one or more kernels is deallocated when the execution of all kernels in the shared scratch memory 115 is completed.

[0041] The number of SECs that a processing unit 110 can process concurrently depends on the number of registers required by each thread and the size of the per-SEC shared scratch memory required by a given kernel, because the processing resources (e.g., registers and private scratch memory pools) in a processing unit 110 are split among all threads assigned to the kernel. If each processing unit 110 does not have enough registers and / or shared memory available to process at least one SEC, the kernel will fail to start. In one embodiment, the scratch memory can be allocated various sizes (e.g., 16KB, 32KB, 64KB, etc.).

[0042] Although method 200 is described in the context of a processor, method 200 may also be performed by a program, custom logic or circuit, or a combination of custom circuit and program. For example, method 200 may be performed by a GPU or CPU or any logic device or processor capable of providing a shared scratchpad memory. In addition, a person of ordinary skill in the art will appreciate that any system that performs method 200 falls within the scope and spirit of the embodiments of the present disclosure.

[0043] In addition to the shared scratch memory 115 and / or the shared persistent scratch memory 125, each thread also has private local memory. Typically, threads contained in a single SEC share a portion of memory allocated from a private scratch memory pool in the processing unit 110. Typically, the portion of memory allocated from the private scratch memory pool is deallocated when the SEC completes execution. Importantly, this portion of memory is not visible to threads in other SECs, and because this portion of memory is not persistent, it cannot be shared between different SECs that execute sequentially. Therefore, in a conventional memory hierarchy, threads in different SECs exchange data using a high-latency path through the global memory 120 provided by store and load operations.

[0044] In one embodiment, all threads can access additional memory spaces: constant, texture, and surface memory spaces. Constant memory and texture memory are read-only; however, surface memory is readable and writable. Global memory 120, constant memory, texture memory, and surface memory spaces are optimized for different memory usage scenarios. For example, texture memory provides different addressing modes and data filtering for specific data formats. It should be noted that texture memory and surface memory are caches, and the cache is not consistent with global memory writes and surface memory writes in the same kernel call, so any texture fetch or surface read of an address that has been written via a global write or a surface write in the same kernel call returns undefined data. In other words, a thread can safely read certain texture memory locations or surface memory locations only if the memory location has been updated by a previous kernel call or memory copy, but cannot safely read if the memory location has been previously updated by the same thread or another thread from the same kernel call.

[0045] The shared scratch memory 115 and the shared persistent scratch memory 125 are not cached. Data stored in the shared scratch memory 115 and the shared persistent scratch memory 125 are not moved or copied to the backing store. From the thread's perspective, accessing the shared scratch memory 115 and the shared persistent scratch memory 125 is similar to accessing the global memory 120, but much faster.

[0046] In one embodiment, the architecture of the highly parallel processor 105 is built around a scalable array of processing units 110, where each processing unit 110 is a multi-threaded streaming multiprocessor (SM). When a host program calls a kernel grid, the grid's SECs are enumerated and distributed to processing units 110 with available execution capacity. The threads of a SEC execute concurrently on a processing unit 110. When a SEC terminates, a new block is started on the vacated processing unit 110.

[0047] The processing unit 110 includes multiple scalar processor (SP) cores, a multithreaded instruction unit, and an on-chip private scratchpad memory pool. The processing unit 110 creates, manages, and executes concurrent threads in hardware and has no scheduling overhead. It implements single instruction barrier synchronization. Fast barrier synchronization as well as lightweight thread creation and zero-overhead thread scheduling can effectively support very fine-grained parallelism, for example, by assigning a thread to each data element (such as a pixel in an image, a voxel in a volume, a cell in a grid-based calculation), allowing low-granularity decomposition of the problem.

[0048] To manage hundreds of threads running several different programs, processing unit 110 uses an architecture called SIMT (single instruction, multiple threads). Processing unit 110 maps each thread to a scalar processor core, and each scalar thread executes independently using its own instruction address and register state. The multiprocessor scheduling unit creates, manages, schedules, and executes threads in groups of parallel threads called warps (a term derived from weaving, which is the first parallel thread technology). The individual threads that make up the SIMT warp start together at the same program address, but are otherwise free to branch and execute independently.

[0049] When a particular processing unit 110 is given one or more thread blocks to execute, the processing unit 110 splits the thread blocks into warps that are scheduled by the scheduler unit. The way thread blocks are split into warps is always the same; each warp contains consecutive threads with increasing thread IDs, with the first warp containing thread 0.

[0050] At each instruction issue time, the scheduler unit selects a warp that is ready to execute and issues the next instruction to the warp's active threads. Warps execute one common instruction at a time, so full efficiency is achieved when all threads of a warp agree on their execution path. If the threads of a warp diverge via a data-dependent conditional branch, the warp will execute each branch path taken serially and disable threads that are not on that path, and when all paths are completed, the threads will converge back to the same execution path. Branch divergence occurs only within a warp; different warps execute independently, whether the warps are executing a common code path or executing disjoint code paths.

[0051] The SIMT architecture is similar to the SIMD (Single Instruction, Multiple Data) vector organization in that a single instruction controls multiple processing elements. The key difference is that the SIMD vector organization exposes the SIMD width to the software, while the SIMT instructions specify the execution and branching behavior of a single thread. Compared to SIMD vector machines, SIMT enables programmers to write thread-level parallel code for independent scalar threads, as well as data-parallel code for coordinated threads. For correct purposes, programmers can essentially ignore SIMT behavior; however, by noticing that the code rarely requires thread divergence within a warp, a significant performance gain can be achieved. In practice, this is similar to the role of cache lines in traditional code: cache line size can be safely ignored when designing for correctness, but code structure must be considered when designing for optimal performance. On the other hand, vector architectures require software to merge loads into vectors and manually manage divergence. Embodiments of the SIMT architecture are described in detail below.

[0052] Parallel processing architecture

[0053] Figure 3 A parallel processing unit (PPU) 300 according to one embodiment is shown. In one embodiment, the PPU 300 is a multi-threaded processor implemented on one or more integrated circuit devices. In one embodiment, the highly parallel processor 105 is a PPU 300. The PPU 300 is a latency-hidden architecture designed to process many threads in parallel. In one embodiment, the PPU 300 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device such as a liquid crystal display (LCD) device. In other embodiments, the PPU 300 can be used to perform general-purpose calculations. Although an exemplary parallel processor is provided herein for illustrative purposes, it should be noted that such a processor is described for illustrative purposes only and that any processor can be used to supplement and / or replace the processor.

[0054] One or more PPUs 300 can be configured to accelerate thousands of high-performance computing (HPC), data center, and machine learning applications. PPU 300 can be configured to accelerate numerous deep learning systems and applications, including autonomous vehicle platforms, deep learning, high-precision speech, image, and text recognition systems, intelligent video analysis, molecular simulation, drug discovery, disease diagnosis, weather forecasting, big data analysis, astronomy, molecular dynamics simulation, financial modeling, robotics, factory automation, real-time language translation, online search optimization and personalized user recommendations, and similar applications.

[0055] like Figure 3As shown, the PPU 300 includes an input / output (I / O) unit 305, a front-end unit 315, a scheduler unit 320, a work distribution unit 325, a hub 330, a crossbar switch (Xbar) 370, one or more general processing clusters (GPCs) 350, and one or more memory partition units 380. The PPU 300 can be connected to a host processor or other PPUs 300 via one or more high-speed NVLink 310 interconnects. The PPU 300 can be connected to a host processor or other peripheral devices via an interconnect 302. The PPU 300 can also be connected to a local memory 304 including multiple memory devices. In one embodiment, the local memory 304 is a global memory 120. In one embodiment, the local memory may include multiple dynamic random access memory (DRAM) devices. The DRAM devices can be configured as a high bandwidth memory (HBM) subsystem with multiple DRAM dies stacked in each device.

[0056] The NVLink 310 interconnect enables the system to expand and include one or more PPUs 300 in combination with one or more CPUs, supporting cache coherency between the PPU 300, the CPU, and the CPU master. Data and / or commands may be sent by the NVLink 310 through the hub 330 to or from other units of the PPU 300, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). NVLink 310 in combination with Figure 5B Detailed description.

[0057] I / O unit 305 is configured to send and receive communications (e.g., commands, data, etc.) from a host processor (not shown) via interconnect 302. I / O unit 305 may communicate with the host processor directly via interconnect 302 or through one or more intermediate devices (e.g., a memory bridge). In one embodiment, I / O unit 305 may communicate with one or more other processors (such as one or more PPUs 300) via interconnect 302. In one embodiment, I / O unit 305 implements a peripheral component interconnect express (PCIe) interface for communicating over a PCIe bus, and interconnect 302 is a PCIe bus. In alternative embodiments, I / O unit 305 may implement other types of well-known interfaces for communicating with external devices.

[0058] I / O unit 305 decodes packets received via interconnect 302. In one embodiment, the packets represent commands configured to cause PPU 300 to perform various operations. I / O unit 305 sends the decoded commands to various other units of PPU 300, as specified by the commands. For example, certain commands may be sent to front end unit 315. Other commands may be sent to hub 330 or other units of PPU 300, such as one or more replication engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In other words, I / O unit 305 is configured to route communications between and among various logical units of PPU 300.

[0059] In one embodiment, a program or kernel executed by a host processor encodes a command stream in a buffer that provides a workload to the PPU 300 for processing. The workload may include several instructions and data to be processed by those instructions. A buffer is an area in memory that is accessible (e.g., read / write) by both the host processor and the PPU 300. For example, the I / O unit 305 may be configured to access a buffer in system memory connected to the interconnect 302 via a memory request sent through the interconnect 302. In one embodiment, the host processor writes a command stream to the buffer and then sends a pointer to the start of the command stream to the PPU 300. The front end unit 315 receives a pointer to one or more command streams. The front end unit 315 manages one or more streams, reads commands from the streams, and forwards the commands to the various units of the PPU 300.

[0060] The front end unit 315 is coupled to a scheduler unit 320, which configures each GPC 350 to process tasks defined by one or more streams. The scheduler unit 320 is configured to track state information related to various tasks managed by the scheduler unit 320. The state may indicate which GPC 350 the task is assigned to, whether the task is active or inactive, a priority associated with the task, etc. The scheduler unit 320 manages the execution of multiple tasks on one or more GPCs 350.

[0061] Scheduler unit 320 is coupled to work distribution unit 325, which is configured to dispatch tasks for execution on GPC 350. Work distribution unit 325 may keep track of a number of scheduled tasks received from scheduler unit 320. In one embodiment, work distribution unit 325 manages a pending task pool and an active task pool for each GPC 350. The pending task pool may include a number of time slots (e.g., 32 time slots) containing tasks assigned to be processed by a particular GPC 350. The active task pool may include a number of time slots (e.g., 4 time slots) for tasks that are being actively processed by GPC 350. When a GPC 350 completes execution of a task, the task is evicted from the active task pool of GPC 350, and one of the other tasks from the pending task pool is selected and scheduled for execution on GPC 350. If an active task on a GPC 350 has become idle, such as while waiting for a data dependency to be resolved, then the active task may be evicted from the GPC 350 and returned to the pending task pool, while another task in the pending task pool is selected and scheduled for execution on the GPC 350 .

[0062] Work distribution unit 325 communicates with one or more GPCs 350 via XBar 370. XBar 370 is an interconnect network that couples many units of PPU 300 to other units of PPU 300. For example, XBar 370 can be configured to couple work distribution unit 325 to a particular GPC 350. Although not explicitly shown, one or more other units of PPU 300 can also be connected to XBar 370 via hub 330.

[0063] Tasks are managed by a scheduler unit 320 and dispatched to GPCs 350 by a work distribution unit 325. GPCs 350 are configured to process tasks and generate results. Results can be consumed by other tasks within the GPC 350, routed to different GPCs 350 via XBar 370, or stored in memory 304. Results can be written to memory 304 via a memory partition unit 380, which implements a memory interface for reading data from and writing data to memory 304. Results can be sent to another PPU 300 or CPU via NVLink 310. In one embodiment, a PPU 300 includes a number U of memory partition units 380, which is equal to the number of independent and distinct memory devices coupled to the memory 304 of the PPU 300. This will be described below in conjunction with Figure 4B The memory partition unit 380 is described in more detail.

[0064] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API) that enables one or more applications to be executed on the host processor to schedule operations for execution on the PPU 300. In one embodiment, multiple computing applications are executed simultaneously by the PPU 300, and the PPU 300 provides isolation, quality of service (QoS), and independent address spaces for multiple computing applications. The application can generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks to be executed by the PPU 300. The driver kernel outputs the tasks to one or more streams being processed by the PPU 300. Each task can include one or more related thread groups, referred to herein as warps. In one embodiment, a warp includes 32 related threads that can be executed in parallel.

[0065] Figure 4A According to an embodiment Figure 3 PPU 300 GPC 350. Figure 4A As shown, each GPC 350 includes multiple hardware units for processing tasks. In one embodiment, each GPC 350 includes a pipeline manager 410, a pre-raster operation unit (PROP) 415, a raster engine 425, a work distribution crossbar switch (WDX) 480, a memory management unit (MMU) 490, and one or more data processing clusters (DPCs) 420. It should be understood that Figure 4A The GPC 350 may include instead Figure 4A Other hardware units or units other than those shown in Figure 4A Other hardware units than those shown in .

[0066] In one embodiment, the operation of the GPC 350 is controlled by a pipeline manager 410. The pipeline manager 410 manages the configuration of one or more DPCs 420 for processing tasks assigned to the GPC 350. In one embodiment, the pipeline manager 410 may configure at least one of the one or more DPCs 420 to implement at least a portion of a graphics rendering pipeline. For example, a DPC 420 may be configured to execute a vertex shading program on a programmable streaming multiprocessor (SM) 440. The pipeline manager 410 may also be configured to route packets received from the work distribution unit 325 to appropriate logic units in the GPC 350. For example, some packets may be routed to fixed-function hardware units in the PROP 415 and / or the raster engine 425, while other packets may be routed to the DPC 420 for processing by the primitive engine 435 or the SM 440. In one embodiment, the pipeline manager 410 may configure at least one of the one or more DPCs 420 to implement a neural network model and / or a computational pipeline.

[0067] PROP unit 415 is configured to route data generated by raster engine 425 and DPC 420 to a raster operations (ROP) unit, in conjunction with Figure 4B The PROP unit 415 may also be configured to perform optimization of color blending, organize pixel data, perform address translation, etc.

[0068] The raster engine 425 includes several fixed-function hardware units configured to perform various raster operations. In one embodiment, the raster engine 425 includes a setup engine, a coarse raster engine, a culling engine, a clipping engine, a fine raster engine, and a tile aggregation engine. The setup engine receives the transformed vertices and generates plane equations associated with the geometric primitives defined by the vertices. The plane equations are sent to the coarse raster engine to generate coverage information for the primitives (e.g., x, y coverage masks for tiles). The output of the coarse raster engine is sent to the culling engine, where fragments associated with primitives that fail the z-test are culled, and unculled fragments are sent to the clipping engine, where fragments outside the viewing cone are clipped. Those fragments that remain after clipping and culling can be passed to the fine raster engine to generate attributes of pixel fragments based on the plane equations generated by the setup engine. The output of the raster engine 425 includes, for example, fragments to be processed by a fragment shader implemented in the DPC 420.

[0069] Each DPC 420 included in the GPC 350 includes an M-pipeline controller (MPC) 430, a primitive engine 435, and one or more SMs 440. The MPC 430 controls the operation of the DPC 420, routing packets received from the pipeline manager 410 to appropriate units in the DPC 420. For example, packets associated with vertices may be routed to the primitive engine 435, which is configured to fetch vertex attributes associated with the vertices from the memory 304. Conversely, packets associated with the dispatch unit 515 may be sent to the SM 440.

[0070] SM 440 includes a programmable stream processor configured to process tasks represented by multiple threads. Each SM440 is multithreaded and configured to concurrently execute multiple threads (e.g., 32 threads) from a specific thread group. In one embodiment, SM 440 implements a SIMD architecture, wherein each thread in a thread group (e.g., a warp) is configured to process different data sets based on the same instruction set. All threads in a thread group execute the same instruction. In another embodiment, SM 440 implements a SIMT (single instruction, multiple threads) architecture, wherein each thread in a thread group is configured to process different data sets based on the same instruction set, but wherein each thread in a thread group is allowed to diverge during execution. In one embodiment, a program counter, a call stack, and an execution state are maintained for each warp, and when threads in a warp diverge, concurrency between warps and serial execution within a warp are enabled. In another embodiment, a program counter, a call stack, and an execution state are maintained for each individual thread, thereby achieving equal concurrency between all threads within and between warps. When execution state is maintained for each individual thread, threads executing the same instructions can be converged and executed in parallel for maximum efficiency. Figure 5A SM 440 is described in more detail.

[0071] MMU 490 provides an interface between GPC 350 and memory partition unit 380. MMU 490 can provide virtual address to physical address translation, memory protection, and arbitration of memory requests. In one embodiment, MMU 490 provides one or more translation lookaside buffers (TLBs) for performing translation of virtual addresses to physical addresses in memory 304.

[0072] Figure 4B According to an embodiment Figure 3 The memory partition unit 380 of the PPU 300. Figure 4B As shown, the memory partition unit 380 includes a raster operation (ROP) unit 450, a level 2 (L2) cache 460, and a memory interface 470. The memory interface 470 is coupled to the memory 304. The memory interface 470 may implement a 32, 64, 128, 1024 bit data bus, etc., for high-speed data transfer. In one embodiment, the PPU 300 incorporates U memory interfaces 470, one memory interface 470 for each pair of memory partition units 380, wherein each pair of memory partition units 380 is connected to a corresponding memory device of the memory 304. For example, the PPU 300 may be connected to up to Y memory devices 304, such as a high bandwidth memory stack or a graphics double data rate version 5 synchronous dynamic random access memory or other types of persistent memory.

[0073] In one embodiment, memory interface 470 implements an HBMx memory interface and Y is equal to half of U. In one embodiment, memory 304 supports single error correction double error detection (SECDED) error correction code (ECC) to protect data. ECC provides higher reliability for computing applications that are sensitive to data corruption. Reliability is particularly important in large-scale cluster computing environments where PPU 300 processes very large data sets and / or long-running applications.

[0074] In one embodiment, the PPU 300 implements a multi-level memory hierarchy. In one embodiment, the memory partition unit 380 supports unified memory to provide a single unified virtual address space for the CPU and PPU 300 memory, enabling data sharing between virtual memory systems. In one embodiment, the frequency of accesses by the PPU 300 to memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 300 that accesses the page more frequently. In one embodiment, the NVLink 310 supports address translation services that allow the PPU 300 to directly access the CPU's page tables and provide the PPU 300 with full access to the CPU's memory.

[0075] In one embodiment, the copy engine transfers data between multiple PPUs 300 or between a PPU 300 and a CPU. The copy engine can generate a page fault for an address that is not mapped to a page table. The memory partition unit 380 can then service the page fault, map the address into a page table, and then the copy engine can perform the transfer. In conventional systems, fixed memory (e.g., non-pageable) is operated for multiple copy engines between multiple processors, which significantly reduces the available memory. Due to hardware paging faults, addresses can be passed to the copy engine without worrying about whether the memory page is resident, and whether the copy process is transparent.

[0076] Data from memory 304 or other system memory can be retrieved by memory partition unit 380 and stored in L2 cache 460, which is located on the chip and shared between various GPCs 350. As shown, each memory partition unit 380 includes a portion of L2 cache 460 associated with the corresponding memory 304. Lower-level caches can then be implemented in various units within GPC 350. For example, each SM 440 can implement a level 1 (L1) cache. L1 cache is private memory dedicated to a specific SM 440. The L1 cache within a specific SM 440 can be accessed by any SEC executing on SM 440. Data from L2 cache 460 can be retrieved and stored in each L1 cache for processing in the functional units of SM 440. L2 cache 460 is coupled to memory interface 470 and XBar 370.

[0077] The ROP unit 450 performs graphics raster operations related to pixel color such as color compression, pixel blending, etc. The ROP unit 450 also implements depth testing in conjunction with the raster engine 425, receiving the depth of a sample position associated with a pixel fragment from the culling engine of the raster engine 425. The depth of the sample position associated with the fragment is tested relative to the corresponding depth in the depth buffer. If the fragment passes the depth test for the sample position, the ROP unit 450 updates the depth buffer and sends the result of the depth test to the raster engine 425. It will be understood that the number of memory partition units 380 can be different than the number of GPCs 350, and thus each ROP unit 450 can be coupled to each GPC 350. The ROP unit 450 tracks packets received from different GPCs 350 and determines to which GPC 350 the results generated by the ROP unit 450 are routed via the Xbar 370. Although in Figure 4B In the embodiment shown, ROP unit 450 is included within memory partition unit 380, but in other embodiments, ROP unit 450 may be external to memory partition unit 380. For example, ROP unit 450 may reside in GPC 350 or another unit.

[0078] Figure 5A According to an embodiment Figure 4A The streaming multiprocessor 440. Figure 5A As shown, SM 440 includes an instruction cache 505, one or more scheduler units 510, a register file 520, one or more processing cores 550, one or more special function units (SFUs) 552, one or more load / store units (LSUs) 554, an interconnect network 580, a scratch memory pool 525, and an L1 cache 570. The shared scratch memory 115 and / or the shared persistent scratch memory 125 are allocated from the scratch memory 525.

[0079] As described above, the work distribution unit 325 dispatches tasks for execution on the GPC 350 of the PPU 300. Tasks are assigned to specific DPCs 420 within the GPC 350, and if the task is associated with a dispatch unit 515 or a kernel, the task may be assigned to an SM 440. The scheduler unit 510 receives tasks from the work distribution unit 325 and manages the scheduling of instructions for one or more thread blocks assigned to the SM 440. The scheduler unit 510 schedules the thread blocks included in the SEC to execute as warps of parallel threads, where each thread block is assigned at least one warp. In one embodiment, each warp executes 32 threads. The scheduler unit 510 can manage a plurality of different thread blocks and assign warps to different thread blocks, and then dispatch instructions from a plurality of different SECs to various functional units (e.g., core 550, SFU 552, and LSU 554) during each clock cycle.

[0080] SEC provides a programming model to organize groups of communicating threads, which allows developers to express the granularity of thread communication, making the expressions richer and the parallel decomposition more efficient. The cooperative launch API supports synchronization between thread blocks to execute parallel algorithms. The conventional programming model provides a single simple construct for synchronizing cooperative threads: a barrier across all threads of a thread block (e.g., the syncthreads() function). However, programmers often want to define thread groups at a granularity smaller than the thread block granularity and synchronize within the defined groups to achieve higher performance, design flexibility, and software reuse in the form of a collective group-wide function interface.

[0081] SEC enables programmers to explicitly define thread groups at sub-block (e.g., as small as a single thread) and multi-block granularity, and to perform collective operations (such as synchronization) on threads in cooperative groups. The programming model supports clean composition across software boundaries so that libraries and utility functions can safely synchronize in their local contexts without making assumptions about convergence. SEC primitives support new patterns of cooperative parallelism, including producer-consumer parallelism, opportunistic parallelism, and global synchronization over the entire SEC grid.

[0082] When the kernel is annotated to indicate concurrency between different kernels or dependencies between different kernels, the SEC assigned as the execution kernel is correspondingly allocated a shared scratch memory 115 or a shared persistent scratch memory 125. Similarly, when a single kernel is annotated to indicate concurrency and / or dependencies are assigned to multiple SECs, the multiple SECs assigned as the execution kernels are correspondingly allocated a shared scratch memory 115 or a shared persistent scratch memory 125.

[0083] When two kernels exchanging data are assigned to multiple SECs in two or more processing units 110, the SEC in the first processing unit 110 cannot exchange data with the SEC in the second processing unit 110 through the shared scratch memory in the first or second processing unit 110. Therefore, kernels may be assigned to SECs in a sliced ​​manner so that threads assigned to different kernels exchanging data are executed in the same processing unit 110. For example, assuming that four SECs are executed in each processing unit 110, the first pair of SECs executed in the first processing unit 110 are assigned to the first kernel, and the remaining second pair of SECs executed in the first processing unit 110 are assigned to the second kernel. The first pair of SECs executing the first kernel can exchange data with the second pair of SECs executing the second kernel.

[0084] The dispatch unit 515 is configured to send instructions to one or more functional units. In one embodiment, the scheduler unit 510 includes two dispatch units 515, which enable dispatching two different instructions from the same thread warp during each clock cycle. In alternative embodiments, each scheduler unit 510 may include a single dispatch unit 515 or additional dispatch units 515.

[0085] Each SM 440 includes a register file 520 that provides a set of registers for the functional units of the SM 440. In one embodiment, the register file 520 is divided between each functional unit so that each functional unit is assigned a dedicated portion of the register file 520. In another embodiment, the register file 520 is divided between the different warps executed by the SM 440. The register file 520 provides temporary storage for operands for the data paths connected to the functional units.

[0086] Each SM 440 includes L processing cores 550. In one embodiment, SM 440 includes a large number (e.g., 128, etc.) of different processing cores 550. Each core 550 may include a full pipeline, single precision, double precision and / or mixed precision processing unit, the mixed precision processing unit including a floating point arithmetic logic unit and an integer arithmetic logic unit. In one embodiment, the floating point arithmetic logic unit implements the IEEE 754-2008 standard for floating point arithmetic. In one embodiment, core 550 includes 64 single precision (32-bit) floating point cores, 64 integer cores, 32 double precision (64-bit) floating point cores, and 8 tensor cores.

[0087] The tensor core is configured to perform matrix operations, and in one embodiment, one or more tensor cores are included in core 550. In particular, the tensor core is configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inference. In one embodiment, each tensor core operates on a 4x4 matrix and performs a matrix multiplication and accumulation operation D=A×B+C, where A, B, C, and D are 4x4 matrices.

[0088] In one embodiment, the matrix multiplication inputs A and B are 16-bit floating point matrices, while the accumulation matrices C and D can be 16-bit floating point matrices or 32-bit floating point matrices. The tensor cores operate on 16-bit floating point input data and perform 32-bit floating point accumulations. The 16-bit floating point multiplication requires 64 operations, producing full-precision products, which are then accumulated using 32-bit floating point additions with other intermediate products of the 4×4×4 matrix multiplication. In practice, tensor cores are used to perform larger two-dimensional or higher-dimensional matrix operations built from these smaller elements. APIs (such as the CUDA 9 C++ API) expose specialized matrix loads, matrix multiplications and accumulations, and matrix storage operations to efficiently use tensor cores from CUDA-C++ programs. At the CUDA level, the warp-level interface assumes that the 16×16 size matrix spans all 32 threads of the warp.

[0089] Each SM 440 also includes M SFUs 552 that perform special functions (e.g., attribute evaluation, reciprocal square root, etc.). In one embodiment, the SFUs 552 may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, the SFUs 552 may include a texture unit configured to perform texture map filtering operations. In one embodiment, the texture unit is configured to load a texture map (e.g., a 2D array of texture pixels) from the memory 304 and sample the texture map to produce sampled texture values ​​for use in a shader program executed by the SM 440. In one embodiment, the texture map is stored in the L1 cache 470. The texture unit implements texture operations, such as filtering operations using mip maps (i.e., texture maps at different levels of detail). In one embodiment, each SM 440 includes two texture units.

[0090] Each SM 440 also includes N LSUs 554 that implement load and store operations to / from the scratch memory 525 and L1 cache 570 and register file 520. Each SM 440 includes an interconnect network 580 that connects each functional unit to the register file 520 and connects the LSUs 554 to the register file 520, the scratch memory 525, and the L1 cache 570. In one embodiment, the interconnect network 580 is a crossbar switch that can be configured to connect any functional unit to any register in the register file 520 and to connect the LSUs 554 to memory locations in the register file and the scratch memory 525 and / or L1 cache 570.

[0091] The L1 cache 570 is a separate array of on-chip memory that allows data storage and communication between the SM 440 and the primitive engine 435. In one embodiment, the L1 cache 570 includes 128KB of storage capacity and is in the path from the SM 440 to the memory partition unit 380. The L1 cache 570 can be used to cache reads and writes. One or more of the L1 cache 570, L2 cache 460, and memory 304 are backing stores in a multi-level memory hierarchy.

[0092] In one embodiment, a single array of on-chip memory is configured to provide separate allocations to implement scratch memory 525 and L1 cache 570. Combining the data cache and scratch memory functionality into a single block of memory provides the best overall performance for both types of memory accesses. This capacity can be used by the core as a cache that does not use shared memory. For example, if the shared memory is configured to use half of the capacity, texture and load / store operations can use the remaining capacity. Combining the scratch memory 525 and L1 cache 570 allows the scratch memory 525 to function as a high throughput pipeline for exchanging data, while the L1 cache 570 simultaneously provides high bandwidth and low latency access to frequently reused data stored in memory 304.

[0093] When configured for general parallel computing, a simpler configuration can be used compared to graphics processing. Specifically, Figure 3The fixed function graphics processing unit is shown bypassed, creating a simpler programming model. In the general parallel computing configuration, the work distribution unit 325 assigns and distributes thread blocks directly to the DPC 420. The threads in the block execute the same program, use unique thread IDs in the computation to ensure that each thread produces unique results, use SM 440 to execute the program and perform computations, use scratch memory 525 to communicate between threads, and use LSU 554 to read and write global memory through L1 cache 570 and memory partition unit 380. When configured for general parallel computing, SM 440 can also write commands that scheduler unit 320 can use to start new work on DPC 420.

[0094] The PPU 300 may be included in a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, etc. In one embodiment, the PPU 300 is included on a single semiconductor substrate. In another embodiment, the PPU 300 is included on a system on a chip (SoC) along with one or more other devices (such as an additional PPU 300, a memory 304, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), etc.).

[0095] In one embodiment, PPU 300 may be included on a graphics card that includes one or more memory devices. The graphics card may be configured to interface with a PCIe slot on a motherboard of a desktop computer. In yet another embodiment, PPU 300 may be an integrated graphics processing unit (iGPU) or parallel processor included in a chipset of a motherboard.

[0096] Shared persistent note memory

[0097] Figure 5B is a conceptual diagram illustrating a processing pipeline for executing kernels A, B, and C, according to one embodiment. Kernel A is annotated to indicate that kernel A should execute concurrently with kernels B and C. Kernel B and C are annotated to indicate that kernel C should execute after kernel B, and kernel C will consume data generated by kernel A, wherein the data generated by kernel A is generated based at least in part on data received from kernel B. Therefore, kernels A, B, and C are assigned to SECs in the same processing unit 440, and shared persistent scratchpad memories 526 and 527 are allocated to execute the SECs.

[0098] Specifically, SEC1 is assigned to core A, SEC2 is assigned to core B, and SEC3 is assigned to core C. Shared persistent scratch memory 526 is allocated to and shared by SEC1 and SEC2. Shared persistent scratch memory 527 is allocated to and shared by SEC1 and SEC3. Data stored by SEC2 in shared persistent scratch memory 526 is persistent, even after SEC2 completes execution of core B. Therefore, SEC1 can load and generate data concurrently with executing SEC3 based on data generated by SEC2. Importantly, SEC3 cannot access shared persistent scratch memory 526 and overwrite data stored by SEC2. In addition, the amount of memory allocated from the private scratch memory pool in processing unit 110 is not necessarily equal, which is used to provide shared persistent scratch memory 526 and 527. Note that a SEC (such as SEC1) can access multiple shared persistent scratch memories.

[0099] As with shared scratch memory 115, shared persistent scratch memory 125 also provides low latency data transfer between cores and / or between SECs executing on processing unit 440 without copying or moving data between different levels of the memory hierarchy. In addition, one or more shared persistent scratch memories may be assigned to multiple cores, each with different processing resource requirements, for transferring data between each other and / or within cores assigned to two or more SECs.

[0100] Once kernel A and kernel B are launched, they can freely exchange data until the execution of the two kernels is completed. Since the data stored by kernel B in the shared persistent scratchpad memory 526 will be retained even after the execution of kernel B is completed, the kernel launch overhead is reduced. Figure 5B As shown, C comes after B), when one kernel generates data due to execution, the data can be passed to the next kernel through the memory system. To transfer data, the producer (B) performs a "write" operation. During the startup of the consumer (C), the "write" operation is followed by a "read" operation. In a conventional system, a write followed by a read will experience the delay of the entire memory system hierarchy, rather than the fast delay of the persistent scratch memory 526. However, when the persistent scratch memory 526 is available, the producer can keep the data in the persistent scratch memory 526, and the consumer only needs to find the data in the persistent scratch memory 526 at startup. The persistent characteristics of the persistent scratch memory 526 can reduce the shutdown overhead of B and the startup overhead of C.

[0101] Figure 5CA flow chart of a method 500 for configuring a shared memory pool in a processing unit 440 to enable SECs to exchange data through a shared persistent scratch memory is shown according to one embodiment. In one embodiment, the method 500 may be used to exchange data through a shared persistent scratch memory 125, 526, and / or 527.

[0102] In step 510, processing unit 440 receives a kernel for execution. Processing unit 110 may be configured to operate in one of two scratch memory modes. When operating in private scratch memory mode, separate portions of scratch memory 525 are allocated to be accessed only by each individual SEC assigned to execute a kernel within processing unit 440. In other words, private scratch memory mode is compatible with conventional operation of per-SEC scratch memory. When operating in shared scratch mode, a portion of scratch memory 525 is allocated to be accessed by one or more SECs assigned to the same kernel and executing in processing unit 440, and / or by SECs assigned to different kernels and executing in processing unit 440.

[0103] In step 530, scheduler unit 510 determines whether the annotation associated with the kernel indicates that the kernel is running in a shared scratch memory mode or a private scratch memory mode. If the mode specified by the kernel is equal to the current operating mode of the scratch memory 525 in the processing unit 440, scheduler unit 510 proceeds to step 210. Otherwise, the operating mode of the scratch memory 525 cannot satisfy the data exchange constraints specified for the kernel, and in step 535, scheduler unit 530 waits for the processing unit 440 to become idle before changing the scratch memory mode. Steps 210, 215, 220, 225, and 230 are completed, as previously described in conjunction with Figure 2 described.

[0104] In one embodiment, different kernels executing within processing unit 440 may use different operating modes for different portions of scratch memory 525, so that an operating mode is defined for each allocated portion of scratch memory 525 and tracked by scheduler unit 530. When scheduler unit 530 supports a per-allocated portion operating mode, steps 530 and 535 may be omitted.

[0105] When the scratch memory 525 is configured in shared mode, a portion of the scratch memory 525 that was previously available to a single SEC executing a single kernel on a single processing unit 440 can now be shared between multiple SECs executing the same or different kernels, where the SECs can execute the kernels in parallel or serially. The latency of exchanging data through the scratch memory 525 is reduced compared to exchanging data through memory 304. Synchronization of threads can be performed across multiple SECs sharing the same portion of the scratch memory 525, thereby reducing synchronization latency. In one embodiment, synchronization logic is provided to transmit synchronization signals within the processing unit 440 while exchanging data through the scratch memory 525. In one embodiment, synchronization logic is provided for transmitting synchronization signals between processing units 440 while exchanging data through global memory 120.

[0106] Furthermore, when the scratch memory 525 is configured in shared mode, data stored in the scratch memory 525, which was previously available only during the execution of a single kernel, can be preserved between executions of the same or different kernels executed serially in the same processing unit 440. Thus, kernel launch overhead can be reduced because data can be exchanged between different kernels executed serially, as well as within a single kernel executed by multiple SECs.

[0107] Exemplary Computing System

[0108] As developers expose and exploit more parallelism in applications such as artificial intelligence computing, systems with multiple GPUs and CPUs are used in a variety of industries. High-performance GPU-accelerated systems with tens of thousands to more computing nodes are deployed in data centers, research facilities, and supercomputers to solve larger problems. As the number of processing devices in high-performance systems increases, communication and data transmission mechanisms need to scale to support the increased bandwidth.

[0109] Fig. 6A According to one embodiment, the use Figure 3 A conceptual diagram of a processing system 600 implemented by a PPU 300 of FIG. 6A. The exemplary processing system 600 may be configured to implement Figure 2 The method 200 shown in and / or Figure 5C The method 500 shown in FIG. 6 is a processing system 600 including a CPU 630, a switch 610, and a plurality of PPUs 300 and corresponding memories 304. NVLink 310 provides a high-speed communication link between each PPU 300. Fig. 6A302 connections, but the number of connections to each PPU 300 and CPU 630 may vary. Switch 610 interfaces between interconnect 302 and CPU 630. PPU 300, memory 304, and NVLink 310 may be located on a single semiconductor platform to form a parallel processing module 625. In one embodiment, switch 610 supports two or more protocols that interface between various different connections and / or links.

[0110] In another embodiment (not shown), NVLink 310 provides one or more high-speed communication links between each PPU 300 and CPU 630, and switch 610 interfaces between interconnect 302 and each PPU 300. PPU 300, memory 304, and interconnect 302 may be located on a single semiconductor platform to form parallel processing module 625. In yet another embodiment (not shown), interconnect 302 provides one or more communication links between each PPU 300 and CPU 630, and switch 610 interfaces between each PPU 300 using NVLink 310 to provide one or more high-speed communication links between PPU 300. In another embodiment (not shown), NVLink 310 provides one or more high-speed communication links between PPU 300 and CPU 630 through switch 610. In yet another embodiment (not shown), interconnect 302 provides one or more communication links directly between each PPU 300. One or more NVLink 310 high-speed communication links may be implemented as a physical NVLink interconnect or an on-chip or on-die interconnect using the same protocol as NVLink 310.

[0111] In the context of this specification, a single semiconductor platform may refer to a unique single semiconductor-based integrated circuit manufactured on a die or chip. It should be noted that the term single semiconductor platform may also refer to a multi-chip module with increased connectivity that simulates on-chip operations and is substantially improved by utilizing conventional bus implementations. Of course, various circuits or devices may also be placed separately or in various combinations of semiconductor platforms, depending on the needs of the user. Alternatively, the parallel processing module 625 may be implemented as a circuit board substrate, and each of the PPU 300 and / or memory 304 may be a packaged device. In one embodiment, the CPU 630, switch 610, and parallel processing module 625 are located on a single semiconductor platform.

[0112] In one embodiment, the signaling rate of each NVLink 310 is 20 to 25 Gbit / s, and each PPU 300 includes six NVLink 310 interfaces (e.g., Figure 5B Each PPU 300 includes five NVLink 310 interfaces. Each NVLink 310 provides a data transfer rate of 25 GBytes / sec in each direction, with six links providing 300 GBytes / sec. When the CPU 630 also includes one or more NVLink 310 interfaces, the NVLink 310 can be used exclusively for, for example, Figure 5B PPU to PPU communication as shown, or some combination of PPU to PPU and PPU to CPU.

[0113] In one embodiment, NVLink 310 allows direct load / store / atomic access from CPU 630 to memory 304 of each PPU 300. In one embodiment, NVLink 310 supports coherency operations, allowing data read from memory 304 to be stored in the cache hierarchy of CPU 630, reducing cache access latency of CPU 630. In one embodiment, NVLink 310 includes support for address translation services (ATS), allowing PPU 300 to directly access page tables within CPU 630. One or more NVLink 310 can also be configured to operate in a low power mode.

[0114] Figure 6B An exemplary system 665 is shown in which various architectures and / or functions of various previous embodiments may be implemented. The exemplary system 665 may be configured to implement Figure 2 The method 200 shown in and / or Figure 5C 500 as shown in the method 500. As shown, a system 665 is provided, which includes at least one central processing unit 630 connected to a communication bus 575. The communication bus 575 can be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), HyperTransport or any other bus or one or more point-to-point communication protocols. The system 665 also includes a main memory 640. Control logic (software) and data are stored in the main memory 640, which can take the form of a random access memory (RAM).

[0115] The system 665 also includes an input device 660, a parallel processing system 625, and a display device 645, such as a conventional CRT (cathode ray tube), LCD (liquid crystal display), LED (light emitting diode), plasma display, etc. User input can be received from the input device 660 (e.g., keyboard, mouse, touch pad, microphone, etc.). Each of the aforementioned modules and / or devices can even be located on a single semiconductor platform to form the system 665. Alternatively, the various modules can also be placed separately or in various combinations of semiconductor platforms according to the needs of the user.

[0116] Furthermore, system 665 may be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, etc.) through network interface 635 for communication purposes.

[0117] The system 665 may also include auxiliary storage (not shown). The auxiliary storage 610 includes, for example, a hard disk drive and / or a removable storage drive, representative of a floppy disk drive, a tape drive, an optical disk drive, a digital versatile disk (DVD) drive, a recording device, a universal serial bus (USB) flash memory. The removable storage drive reads from and / or writes to a removable storage unit in a well-known manner.

[0118] Computer programs or computer control logic algorithms may be stored in the main memory 640 and / or the secondary storage. These computer programs, when executed, enable the system 665 to perform various functions. The memory 640, storage, and / or any other storage are possible examples of computer-readable media.

[0119] The architecture and / or functionality of the various previous figures may be implemented in the context of a general purpose computer system, a circuit board system, a game console system dedicated to entertainment purposes, a dedicated system, and / or any other desired system. For example, the system 665 may take the form of a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head mounted display, a handheld electronic device, a mobile telephone device, a television, a workstation, a game console, an embedded system, and / or any other type of logic.

[0120] Although various embodiments have been described above, it should be understood that they are presented by way of example only and not limitation. Thus, the breadth and scope of a preferred embodiment should not be limited by any of the above-described exemplary embodiments, but should be limited only in accordance with the following claims and their equivalents.

[0121] Machine Learning

[0122] Deep Neural Networks (DNNs) developed on processors such as the PPU 300 have been used in a variety of use cases: from self-driving cars to faster drug development, from automatic image captioning in online image databases to intelligent real-time language translation in video chat applications. Deep learning is a technology that models the neural learning process of the human brain, constantly learning, constantly getting smarter, and delivering more accurate results faster over time. A child is initially taught by an adult to correctly identify and classify various shapes, and eventually is able to recognize shapes without any coaching. Similarly, deep learning or neural learning systems need to be trained in object recognition and classification in order to become more intelligent and efficient in recognizing basic objects, occluded objects, etc. while also assigning context to objects.

[0123] At the simplest level, neurons in the human brain look at the various inputs they receive, assign a level of importance to each of these inputs, and pass outputs to other neurons for processing. An artificial neuron, or perceptron, is the most basic model of a neural network. In one example, a perceptron can receive one or more inputs that represent various features of an object that the perceptron is being trained to recognize and classify, and each of these features is given a certain weight based on the importance of that feature when defining the shape of the object.

[0124] Deep neural network (DNN) models include multiple layers of many connected nodes (e.g., perceptrons, Boltzmann machines, radial basis functions, convolutional layers, etc.), which can be trained with large amounts of input data to solve complex problems quickly and accurately. In one example, the first layer of a DNN model breaks down an input image of a car into its parts and looks for basic patterns (such as lines and angles). The second layer assembles lines to find higher-level patterns, such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the last few layers generate labels for the input image, identifying the model of a specific car brand.

[0125] Once a DNN is trained, it can be deployed and used to recognize and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include recognizing handwritten numbers on a check deposited into an ATM, identifying images of friends in photos, providing movie recommendations to over 50 million users, recognizing and classifying different types of cars, pedestrians, and road hazards in a self-driving car, or translating human language in real time.

[0126] During training, data flows through the DNN in a forward propagation phase until a prediction is produced, which indicates a label corresponding to the input. If the neural network does not correctly label an input, the error between the correct label and the predicted label is analyzed, and the weights are adjusted for each feature during a backpropagation phase until the DNN correctly labels that input and other inputs in the training data set. Training complex neural networks requires a large amount of parallel computing performance, including floating-point multiplications and additions supported by the PPU 300. Inference, which is less computationally intensive than training, is a latency-sensitive process in which a trained neural network is applied to new inputs it has not seen before to classify images, translate speech, and generally reason about new information.

[0127] Neural networks rely heavily on matrix math operations, and complex multi-layer networks require a lot of floating point performance and bandwidth for efficiency and speed. With thousands of processing cores optimized for matrix math operations and delivering tens to hundreds of TFLOPS of performance, the PPU 300 is a computing platform capable of delivering the performance required for deep neural network-based artificial intelligence and machine learning applications.

[0128] It should be noted that the techniques described herein may be embodied in executable instructions stored in a computer-readable medium for use by or in combination with a processor-based instruction execution machine, system, device or apparatus. It will be appreciated by those skilled in the art that for certain embodiments, various types of computer-readable media may be included to store data. As used herein, "computer-readable media" includes one or more any suitable media for storing executable instructions of a computer program, so that an instruction execution machine, system, device or apparatus can read (or obtain) instructions from a computer-readable medium and execute the instructions for implementing the described embodiments. Suitable storage formats include one or more electrical, magnetic, optical and electromagnetic formats. A non-exhaustive list of conventional exemplary computer-readable media includes: portable computer disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM), flash memory devices and optical storage devices, including portable compact disks (CDs), portable digital video disks (DVDs), etc.

[0129] It should be understood that the arrangement of the components shown in the drawings is for illustrative purposes, and other arrangements are possible. For example, one or more elements described herein may be implemented in whole or in part as electronic hardware components. Other elements may be implemented in software, hardware, or a combination of software and hardware. In addition, while implementing the functions described herein, some or all of these other elements may be combined together, some elements may be completely omitted, and other components may be added. Therefore, the subject matter described herein may be embodied in many different variations, and all such variations are considered to fall within the scope of the claims.

[0130] For ease of understanding the subject matter described herein, many aspects are described with the sequence of actions. It will be appreciated by those skilled in the art that various operations can be performed by dedicated circuits or circuits, and by program instructions performed by one or more processors, or by a combination of the two. The description of the sequence of any action herein is not intended to imply that the specific order described for performing the sequence must be followed. All methods described herein can be performed in any appropriate order, unless otherwise noted herein or context clearly contradicts.

[0131] The use of the terms "a" and "an" and similar references in the context of describing the subject matter (especially in the context of the appended claims) should be interpreted as including the singular and the plural, unless otherwise stated herein or clearly contradicted by the context. The use of the term "at least one" followed by a list of one or more items (e.g., "at least one of A and B") should be interpreted as meaning one item (A or B) selected from the listed items, or any combination of two or more listed items (A and B), unless otherwise stated herein or clearly contradicted by the context. In addition, the above description is for illustrative purposes only and not for limitation, as the scope of protection sought is defined by the claims and any equivalents thereof. The use of any and all examples or exemplary language provided herein (e.g., "for example") is intended only to better illustrate the subject matter and will not limit the scope of the subject matter unless otherwise stated. The use of the word "based on" and other similar phrases in the claims and the specification to indicate the conditions for producing a result is not intended to exclude any other conditions for producing the result. No language in the specification should be interpreted as indicating that any non-claimed element is necessary for the practice of the claimed invention.

Claims

1. A parallel processing system, comprising: Multiple processing units, including: a first processing unit including a first register file, wherein the first processing unit is capable of executing a first kernel and a second kernel, and a second processing unit including a second scratch pad and a second register file, wherein the second processing unit is capable of executing a third kernel; scheduler logic coupled to the plurality of processing units, wherein the scheduler logic assigns kernels to particular ones of the plurality of processing units to be executed; and a memory coupled to the plurality of processing units, wherein the parallel processing system is configured to cause the first data to be stored in the first note or the second note without causing the first data to be stored in the memory; wherein the first note is configured to: allow the first kernel to store second data to the first note, allow the second kernel to load the second data from the first note, and not allow the third kernel to store or load the second data from the first note, and The second note is configured to: allow the third kernel to store the third data into the second note, and not allow the first kernel and the second kernel to load the third data from the second note.

2. The parallel processing system of claim 1, wherein the first kernel is associated with one or more threads.

3. The parallel processing system of claim 1, wherein the first kernel and the second kernel are executed concurrently on the first processing unit for at least one time period. 4 . The parallel processing system of claim 3 , wherein during a first time period, the first kernel and the second kernel are executed synchronously by the first scratch pad.

5. The parallel processing system of claim 1, wherein such first kernel and second kernel are executed serially on the first processing unit.

6. The parallel processing system of claim 1 , wherein the first processing unit comprises a third note, the third note being configured to be accessed by the first kernel and a fourth kernel executing on the first processing unit, wherein the third note is configured to allow the first kernel to store fifth data to the third note, to allow the fourth kernel to load the fifth data from the third note, and to not allow the second kernel and the third kernel to store or load the fifth data from the third note. 7 . The parallel processing system of claim 1 , wherein the parallel processing system allocates the first scratch pad from a scratch pad memory pool included in the first processing unit.

8. The parallel processing system of claim 7, wherein the parallel processing system deallocates the first scratch pad after executing such first kernel and second kernel, thereby returning the first scratch pad to the scratch pad memory pool. 9 . The parallel processing system according to claim 8 , wherein the parallel processing system obtains a size of the first scratch pad and enables the first scratch pad to have sufficient storage locations for the obtained size.

10. The parallel processing system of claim 9, wherein the size is 4 KB, 8 KB, 16 KB, 32 KB, 64 KB, or 128 KB.

11. The parallel processing system of claim 1 , wherein the first processing unit is configured to execute in a first mode and a second mode, wherein during the first mode, only a single core can access the first note, and wherein during the second mode, multiple cores executing in parallel or serially can access the first note.

12. The parallel processing system of claim 11, wherein the parallel processing system causes the single core to complete execution in the first mode before operating in the second mode.

13. The parallel processing system of claim 11, wherein the first mode is a private mode and the second mode is a shared mode.

14. The parallel processing system of claim 1 , further comprising a third processing unit, the third processing unit comprising a third scratch pad and a third register file, wherein the third processing unit is capable of executing the first kernel and the second kernel, and the scheduler logic: assigning a first set of one or more execution contexts to execute the first kernel on the third processing unit; and A second set of one or more execution contexts is assigned to execute the second kernel on the third processing unit, wherein the first set of one or more execution contexts exchange data with the second set of one or more execution contexts through the third scratch pad and complete execution without accessing the first scratch pad.

15. The parallel processing system of claim 1, wherein before loading the second data, execution of the first kernel and the second kernel are synchronized by the first scratch pad.

16. A computer-implemented method comprising: assigning a first kernel and a second kernel to be executed by a first processing unit of a plurality of processing units coupled to a memory, the first processing unit comprising a first scratch pad and a first register file, wherein the first processing unit is capable of executing the first kernel and the second kernel; assigning a third kernel to be executed by a second processing unit of the plurality of processing units, the second processing unit comprising a second scratch pad and a second register file, wherein the second processing unit is capable of executing the third kernel, wherein storing first data to the first scratch pad or the second scratch pad does not cause the first data to be stored in the memory; configuring the first note to: allow such first kernel to store second data to the first note, allow such second kernel to load the second data from the first note, and not allow such third kernel to store or load the second data from the first note; as well as The second note is configured to: allow such third kernel to store third data into the second note, and not allow such first kernel and second kernel to load the third data from the second note.

17. The computer implemented method of claim 16, wherein such first kernel and second kernel execute concurrently on the first processing unit for at least one time period.

18. The computer implemented method of claim 16, wherein such first kernel and second kernel execute serially on the first processing unit.

19. The computer-implemented method of claim 16, wherein the first processing unit is configured to execute in a first mode and a second mode, wherein during the first mode, only a single core is able to access the first note, and wherein during the second mode, multiple cores executing in parallel or serially are able to access the first note.

20. The computer-implemented method of claim 19, further comprising: The single kernel is caused to complete execution in the first mode before operating in the second mode.

21. A non-transitory computer readable medium storing computer instructions which, when executed by one or more processors, cause the one or more processors to perform the following steps: assigning a first kernel and a second kernel to be executed by a first processing unit of a plurality of processing units coupled to a memory, the first processing unit comprising a first scratch pad and a first register file, wherein the first processing unit is capable of executing the first kernel and the second kernel; assigning a third kernel to be executed by a second processing unit of the plurality of processing units, the second processing unit comprising a second scratch pad and a second register file, wherein the second processing unit is capable of executing the third kernel, wherein storing first data to the first scratch pad or the second scratch pad does not cause the first data to be stored in the memory; configuring the first note to allow such first kernel to store second data to the first note, to allow such second kernel to load the second data from the first note, and to not allow such third kernel to store or load the second data from the first note; as well as The second note is configured to allow such a third kernel to store third data to the second note, and to not allow such a first kernel and a second kernel to load the third data from the second note.

Images