Method and apparatus for efficient access to multi-dimensional data structures and / or other large data blocks

Abstract

The present disclosure relates to methods and apparatus for efficient access to multi-dimensional data structures and / or other large data blocks. A parallel processing unit includes a plurality of processors, each coupled to a memory access hardware circuit. Each memory access hardware circuit is configured to receive, from a coupled processor, a memory access request specifying a coordinate of a multi-dimensional data structure, wherein the memory access hardware circuit is one of a plurality of memory access circuits, each coupled to one of the respective processors; and, in response to the memory access request, convert the coordinate of the multi-dimensional data structure to a plurality of memory addresses of the multi-dimensional data structure, and asynchronously transmit at least a portion of the multi-dimensional data structure using the plurality of memory addresses for processing by the coupled processor. The memory locations can be in a shared memory and / or an external memory of the coupled processor.

Description

[0001] Cross-references to related applications This application relates to the following jointly assigned, jointly pending U.S. patent applications, the entire contents of each of which are incorporated by reference: U.S. Patent Application No. 17 / 691,621, entitled “Cooperative Group Arrays,” filed on March 10, 2022; U.S. Patent Application No. 17 / 691,690, entitled "Distributed Shared Memory," filed on March 10, 2022; U.S. Patent Application No. 17 / 691,759, entitled “Virtualizing Hardware Processing Resources in a Processor,” filed on March 10, 2022; U.S. Patent Application No. 17 / 691,288, entitled “Programmatically Controlled Data Multicasting Across Multiple ComputeEngines,” filed on March 10, 2022; U.S. Application No. 17 / 691,296, entitled "Hardware-Accelerated Synchronization with Asynchronous Transaction Support", filed on March 10, 2022; U.S. Patent Application No. 17 / 691,303, entitled "Fast Data Synchronization in Processors and Memory," filed on March 10, 2022; The U.S. patent application No. 17 / 691,406, filed on March 10, 2022, entitled “Efficient Matrix Multiply and Add with a Group of Warps”; U.S. Patent Application No. 17 / 691,872, filed on March 10, 2022, entitled "Techniques for Scalable Load Balancing of Thread Groups in a Processor"; U.S. Patent Application No. 17 / 691,808, filed on March 10, 2022, entitled "Flexible Migration of Executing Software Between Processing Components Without Need For Hardware Reset"; and U.S. Patent Application No. 17 / 691,422, filed on March 10, 2022, entitled “Method and Apparatus For Efficient Access To Multidimensional Data Structures and / or other Large Data Blocks”. Technical Field

[0002] This technology typically involves improving processor processing efficiency and reducing processor power consumption. More specifically, the technology described in this paper relates to dedicated circuitry for processing memory access data blocks by a parallel processor. Background Technology

[0003] Massively parallel high-performance computing systems—systems containing computational processing cores that perform many parallel operations—decompose complex computations into smaller tasks, which can then be executed simultaneously in parallel by multiple processing cores. For example, GEMM (General Matrix Multiplication) is a fundamental building block for many operations in neural networks (such as fully connected layers, recurrent and convolutional layers like RNNs, LSTMs, or GRUs) and scientific applications. GEMM is typically defined as the operation C = αAB + βC, where A and B are matrix inputs, α and β are scalar inputs, and C is a pre-existing matrix covered by the output. In many applications, the matrix can be very large (e.g., 1024x1024 elements)—requiring thousands of separate computations.

[0004] To improve efficiency, modern GPUs divide this matrix input into tiles and compute these tiles in parallel to increase computation speed. This parallel processing allows complex computations to be performed in a small fraction of the time compared to the time required for only one or a few processors to compute the same computation sequentially. For example, the result of multiplying two large matrices can be determined by a set of parallel threads, where each element of the resulting matrix is ​​computed by the corresponding thread in that set of parallel threads.

[0005] Furthermore, the latest GPUs from NVIDIA and other manufacturers have introduced Tensor Cores to maximize the speed of tensor multiplication. These Tensor Cores accelerate matrix multiplication and accumulation operations in machine learning and scientific applications. However, while Tensor Cores significantly improve computational speed, memory access speed has not kept pace.

[0006] Many modern processing systems organize memory in a hierarchical structure (e.g., L1 cache, L2 cache, L3 cache, global memory, etc.). This memory hierarchy stores data currently being processed by the processing cores closer to them, allowing it to be served with lower latency. The cache memory closest to the processing cores, such as the L1 cache, can be partitioned, distributed, or otherwise organized so that each processing core or group of processing cores has exclusive access to its own cache, avoiding latency caused by memory contention with other cores. This cache memory is typically supported by hardware circuitry that maintains tags and is responsible for automatically writing “dirty” (newer) cache lines back to main memory before flushing—eliminating the need for explicit cache management by software programmers. The L1 cache may often be “on-chip,” along with the processing core it serves. In some systems, parallel processing cores can access non-cacheable “shared memory,” which may also be “on-chip” or at least closer to the parallel processing core than the L2 cache. See, for example, U.S. Patent Application No. 11 / 554,552, filed October 30, 2006, entitled "Shared Memory For Concurrent Threads in a Multithreaded Processor Core". This memory is shared between different processing cores to allow them to synchronize and communicate, as well as to increase data location and data reuse.

[0007] Traditionally, retrieving data from global memory (sometimes called "main memory" or "external memory") into shared memory involves a multi-step process. The processor initiates this process by executing a memory load instruction from main memory. This instruction retrieves the addressed data from main memory and stores it in a cache line of the cache memory. In modern GPU architectures, there can be multiple different levels of cache (e.g., L3, L2, L1). Finally, data is retrieved from the cache memory "closest" to the processor (e.g., the L1 cache) and stored in one or more registers of the processor. Such registers can be allocated in a register file (potentially local or another block of "on-chip" memory)—different registers in the register file are assigned to different processors or processor cores.

[0008] In situations where certain common tasks (such as matrix multiplication) require large data transfers, this traditional method of loading data into GPU shared memory can consume a significant amount of registers over extended and often unpredictable periods. During this time (which can last for thousands of cycles in some cases due to long main memory latency or other dependencies), registers may be occupied and unusable for any other purpose. This register binding can prevent the processor in the shared memory from performing other useful work before freeing up the registers.

[0009] Instructions such as the CUDA LDGSTS (Asynchronous Global-to-Shared Memory Copy) instruction, described in U.S. Patent No. 11,080,051, published on August 3, 2021, entitled "Techniques for Efficiently Transfer Data To a Processor," improve latency associated with data movement from global memory to shared memory in NVIDIA's streaming multiprocessors (SMs) by decoupling the L1 cache and / or register files and writing data retrieved from main memory directly to shared memory. However, further improvements are needed in the methods for moving data into and out of shared memory to more effectively manage memory access demands and improve overall data processing efficiency, while still enabling higher mathematical throughput in fields such as artificial intelligence (AI), deep learning (DL), and other applications that can leverage parallel execution. Attached Figure Description

[0010] The following detailed description of exemplary, non-limiting illustrative embodiments should be read in conjunction with the accompanying drawings: Figure 1A GPU architecture including parallel processing units is illustrated according to some example embodiments, wherein each streaming multiprocessor is coupled to a tensor memory access unit (“TMAU”), which provides dedicated hardware circuitry for memory address calculation and moving multidimensional data structures or data blocks into / out of several types of memory.

[0011] Figure 2 The diagram illustrates the interaction between the streaming multiprocessor, the tensor memory access unit circuitry coupled to the streaming multiprocessor, the external memory, and the local shared memory of the streaming multiprocessor when loading data blocks from external memory to shared memory, according to some example embodiments.

[0012] Figure 3A and 3B (Collectively referred to as Figure 3) illustrates tensor parameters that can be applied to tensor addressing stored in external memory and accessed by tensor memory access units according to some example embodiments.

[0013] Figure 4A and 4B (Collectively referred to as Figure 4) illustrates aspects such as out-of-bounds conditions according to some example embodiments, which are out-of-bounds conditions that can be detected by the tensor memory access unit when tensor data is read from external memory.

[0014] Figure 5A and 5B (Collectively referred to as Figure 5) shows example descriptors for accessing data according to some example embodiments.

[0015] Figure 6 This is a schematic diagram of a memory access request processing pipeline in a tensor memory access unit according to some example embodiments.

[0016] Figure 7A Example parameters affecting tensor memory access unit reading tensor data are shown according to some example embodiments.

[0017] Figure 7B Example high-level pseudocode for tensor memory access unit processing according to some embodiments is shown.

[0018] Figure 7C Example high-level pseudocode is shown to represent the use of TMAU by a streaming multiprocessor to load and store tensor data for GEMM (Generalized Matrix Multiplication) computations.

[0019] Figure 8A-8K (Collectively referred to as Figure 8) illustrates the use of example data loading modes according to some example embodiments, particularly the tile mode and the image-to-column mode.

[0020] Figures 9A-9D (Collectively referred to as Figure 9) illustrates an example of data swizzling that can be processed by a tensor memory access unit according to some example embodiments.

[0021] Figure 10 An example parallel processing unit of a GPU according to some embodiments is shown.

[0022] Figure 11A Illustrations are shown according to some embodiments Figure 10 An example of a general-purpose processing cluster (GPC) within a parallel processing unit, where each streaming multiprocessor in the general-purpose processing cluster is coupled to a tensor memory access unit.

[0023] Figure 11B It shows Figure 10 An example memory partition cell for a parallel processing unit.

[0024] Figure 12 It shows Figure 11A An example of streaming multiprocessors.

[0025] Figure 13A Is using Figure 10 An example conceptual diagram of a processing system implemented by a parallel processing unit (PPU).

[0026] Figure 13B This is a block diagram of an exemplary system in which various architectures and / or functions of various prior embodiments can be implemented. Detailed Implementation

[0027] Detailed description of example non-limiting embodiments The example non-limiting techniques described in this disclosure provide streaming multiprocessors (SMs) or other parallel processor cores in parallel processing systems with tightly coupled dedicated hardware circuitry for moving data into and out of memory. For example, the disclosed techniques enable each parallel processor core to be tightly coupled to tensor memory access unit (TMAU) hardware circuitry for moving large blocks of data between shared memory of the parallel processor cores and external memory (e.g., the global memory of the parallel processing system).

[0028] Many computing applications require very large (e.g., megabytes or even gigabytes) data movements between global memory and the computational cores of parallel processors (such as SMs). Typically, data arranged in global memory in a complex, multidimensional structure with non-sequential access patterns must be transferred to shared memory or other memory (SMEM) local to the SM before being used by it. For example, when multiple threads running on one or more SMs perform the multiplication of two very large matrices (e.g., matrices used in DL applications), the data for these two matrices needs to be copied from global memory to the shared memory of those SMs before they can be manipulated.

[0029] Accessing such multidimensional structures in global memory typically incurs significant computational overhead. This overhead can stem from complex address calculations, out-of-bounds condition handling, and resolving SMEM read / write repository conflicts. This type of overhead can negatively impact the performance of kernels running on SM and incur substantial software development costs. This computational overhead is particularly evident in applications like DL, such as convolutional kernels. Typical convolutional kernels access multidimensional data structures (matrices representing tensors or other sets of information) that can be arranged according to standard layouts of different types in global memory. Performance losses associated with address calculations in DL kernels can be attributed to register file (RF) bandwidth consumption, additional RF capacity requirements, out-of-bounds condition handling, limited instruction cache capacity, instruction scheduling challenges, and more. Performance trials on various DL networks have shown average performance losses exceeding 10%. Furthermore, regarding DL software costs, some developers estimate that up to 90% of developer time is spent writing and testing data access code. Developer time is spent on the complexity of instruction scheduling, the challenges of register allocation, and the need to customize the kernel for different tile sizes. The address computation complexity associated with the kernel affects its functional correctness and performance optimization.

[0030] To address the problems outlined, exemplary embodiments of this disclosure provide a dedicated memory access unit coupled to the SM. In some embodiments where the dedicated memory access unit includes the ability to facilitate the movement of data in tensors or other multidimensional data structures, it may also be referred to as a Tensor Memory Access Unit (TMAU). However, the type of data that the TMAU can move is not limited to tensor data, and the target computational core using that data need not be a tensor core, but can be any type of processing core.

[0031] A key design goal of TMAU is to provide an efficient data transfer mechanism for coupled one or more SMs to move large amounts of data between memory locations, such as global memory locations and shared memory locations. By offloading a large number of related data access operations from the kernel running on one or more SMs to TMAU, TMAU can make one or more SMs more computationally efficient. Unlike kernels that rely on load / store instructions per thread (operating relatively small amounts of data), TMAU is configured to accept requests for larger blocks of data or other data structures. Multiple kilobytes or megabytes of data can be transferred for subsequent use by one or more SMs by issuing a single request to TMAU. Furthermore, although a request to TMAU can be issued by a single thread running on a single SM, the acquired data can be used by multiple threads executing on that SM or multiple SMs.

[0032] The apparatus according to the technology described in this disclosure can feed the SM core math unit at a faster rate than techniques that rely on the SM to calculate memory addresses in the data to be copied and track the progress of copying large blocks of data. Example non-limiting embodiments provide block data transfer techniques that result in reduced data transfer and memory access overhead. Reduced data transfer and memory access overhead can significantly reduce the power consumption of multiprocessors (e.g., at the SM level) and improve processing efficiency. For example, consider an assembly line chef responsible for grilling steaks and ribs in a restaurant. This assembly line chef can grill and plate steaks and ribs very quickly. However, in a busy restaurant, the assembly line chef is typically not responsible for leaving their post to retrieve meat from the restaurant's large walk-in refrigerator, cut the meat into pieces, remove fat, etc. Instead, the assembly line chef relies on their apprentice (assistant) chefs to do this work. The assembly line chef can then focus on what only they can do; grilling steaks and ribs to perfection according to customer orders.

[0033] The LDGSTS instruction mentioned above reduces data access latency by moving data from global memory to the shared memory of the SM without intermediate writes to the L1 cache and / or register file. However, using this instruction, moving large blocks of data requires the SM to perform a large number of complex address calculations before issuing a memory access request to the memory system. Compared to the LDGSTS instruction executed by the SM, TMAU enables the SM to asynchronously transfer larger blocks of data with a single instruction and offloads related address calculations from threads on the SM to the TMAU. Furthermore, instead of each parallel execution thread issuing its own instruction to fetch small portions (e.g., tiles) of data from global memory via the LDGSTS instruction or other conventional load / store instructions, TMAU enables a single thread in a thread group, such as a cooperative thread array (“CTA”), to issue an instruction to fetch data for access by all other threads in the group.

[0034] TMAU can be considered similar to a Direct Memory Access (DMA) engine because TMAU can handle reads and writes to global memory independently of the request processor. The key difference is that TMAU can understand and traverse multidimensional data layouts, while DMA typically handles linearly arranged data. Furthermore, in one example embodiment, TMAU does not require the request processor to include memory addresses in its memory access requests. TMAU can instead generate appropriate memory addresses based on the coordinates of the multidimensional structure provided by the request processing core.

[0035] In one embodiment, each TMAU is tightly coupled to an SM, and in some embodiments, each TMAU is coupled to a specific SM in a one-to-one relationship. Tight coupling to a particular SM allows the TMAU to serve memory access requests more efficiently with less contention, rather than having to serve requests from multiple processors. Unlike a DMA engine that receives commands from a driver, each TMAU receives memory access requests from its coupled SM. In some embodiments, compared to a DMA engine that is limited to reading from global memory, the TMAU can copy data from global memory to shared memory, from shared memory to global memory, from a global memory source address to a global memory destination address, and / or from a shared (local) memory source address to a shared (local) memory destination address. During copying within shared memory, the TMAU coupled to the first SM can move data between the shared / local memory of the first SM and the shared / local memory of any other SM in the GPU. For example, in one embodiment, the TMAU can copy data from distributed shared memory local to the first SM to distributed shared memory local to another SM.

[0036] TMAU may also include the ability to detect data reads that exceed tensor boundaries. In some embodiments, in contrast to a technique where each thread on the SM loads a data amount from global memory, TMAU can load data for any number of threads or groups of threads in the coupled SM. Furthermore, in response to a single request for a data block from the requesting SM, TMAU is capable of generating multiple requests, each for a corresponding (different) portion of the requested block.

[0037] In another embodiment, a single TMAU can serve multiple SMs, where each SM can send an independent request to the single TMAU. In this embodiment, a hardware-implemented arbitrator can operate to accept requests from multiple SMs and forward the requests sequentially to the single TMAU. The single TMAU serves requests received from different SMs by transferring data to the local shared memory of the respective requesting SM.

[0038] Parallel processing system including TMAU circuit Figure 1 A parallel processing unit, such as a GPU, is schematically illustrated according to some non-limiting embodiments. Figure 1 As shown, GPU 100 includes multiple processors. In some embodiments, the multiple processors include multi-core processors, such as streaming multiprocessors (SMs), 102a…102n (collectively referred to as 102). Each SM 102 includes multiple processing cores, such as functional units 104a…104m (collectively referred to as 104). In some embodiments, these functional units 104 can perform various types of computations, such as 32-bit floating-point arithmetic, 16-bit floating-point arithmetic, integer arithmetic of different precisions, etc. Furthermore, some of these functional units 104 may include tensor cores, which are designed to carry multiple GEMMs per clock cycle on an N×N matrix, containing floating-point values ​​for floating-point multiplication and addition. The number of SMs in the GPU and the number of functional units in the SMs are not limited. Each functional unit 104 in an SM can access the SM's register file 106, L1 cache 108, and shared / local memory 110. In some embodiments, such as Figure 1 In the embodiment shown, L1 cache 108 may be part of shared / local memory 110. In some other embodiments, L1 cache and shared memory 110 may be separate from each other. Furthermore, in some embodiments, shared memory 110 may be part of a distributed shared memory (DSMEM) arrangement that is also accessible to threads executing on other SMs. U.S. Application No. 17 / 691,690, entitled "Distributed Shared Memory," describes distributed shared memory, the entire contents of which are incorporated herein by reference.

[0039] Multiple SMs 102 can access the global memory 116 external to the GPU 100 through the global memory interface 114. The global memory 116 may include hierarchical cache memory (e.g., L2 cache and / or L3 cache) and dynamic random access memory (DRAM). In some examples, the global memory 116 may include a memory management unit (MMU), an X-Bar or hierarchical crossbar interconnect network, memory partitioning units, and / or references. Figure 10 , 11A The memory described in 11B.

[0040] Multiple cores, such as functional unit 104, within each SM 102 are configured to process multiple threads in parallel. A thread (e.g., an execution thread) is an instantiation of a set of instructions or kernels configured to be executed by functional unit 104 on a specific dataset. Threads within a thread block can execute concurrently, and multiple thread blocks can execute concurrently. In some embodiments, Single Instruction Multiple Data (SIMD) instruction issuance technology is used to support the parallel execution of a large number of threads without providing multiple independent instruction units. In other embodiments, Single Instruction Multiple Threading (SIMT) technology is used to support the parallel execution of a large number of general-purpose synchronous threads by using a common instruction unit configured to issue instructions to a set of cores.

[0041] Each functional unit 104 can be connected via an interconnect network to cache memory 108, shared memory 110, and register file 104, for example, a hierarchical crossbar switch with one or more read and / or write crossbar switches. Shared memory 110 and cache memory 108, which may be an L1 cache, provide low-latency on-chip memory in the vicinity of functional units 104 on the SM 102. Register file 106 may include data registers for different functional units that can be software-assigned to the multiple functional units 104 and / or different thread bundles executed by the SM 102. Register file 106 provides temporary storage for functional units 104 on the SM.

[0042] GPU 100 can support multiple address spaces, including local, shared, and global, to support thread-specific data visibility. Additional read-only address spaces, including constants and textures, can be supported. Each thread has its own thread-local or private memory, which can be controlled through register allocation (see, for example, U.S. Patent Nos. 8,555,035 and 7,634,621, which are incorporated herein by reference as expressly stated).

[0043] Each thread within the same or different thread blocks can access global memory 116 using a hierarchical cache memory. Each thread within the same thread block can access an allocated portion of shared memory 110, which can be considered per-clock shared memory. Each execution thread block can have an allocated portion of shared memory 110. Shared memory 110 is a software-managed cache used to load data from global memory, thereby reducing the number of off-chip memory accesses by the execution threads. Software explicitly allocates and accesses shared memory 110. Threads within a thread block are synchronized (e.g., after cooperatively loading data from global memory into shared memory) to avoid conflicts over critical resource usage.

[0044] When multiple threads in a thread block expect to use the same data from global memory 116, shared memory 110 can be used to store that data, thereby reducing the number of requests for the same data from global memory 116 by each thread. Shared memory 110 can also be used to improve thread access to data by loading and storing data from global memory 116 in a merged mode and then reordering it in shared memory 110, thereby avoiding unmerged memory accesses.

[0045] In some examples Figure 1 In the embodiments shown, where shared memory 110 includes L1 cache 108, the shared memory may be referred to as unified memory or unified cache. The unified cache may be provided in the same on-chip memory (e.g., SRAM) used for both the L1 cache and shared memory, and includes a mechanism for allocating how much unified memory is dedicated to the L1 cache relative to the amount of shared memory used for each kernel call. In some examples, the unified cache may also include a dynamically configurable register file (e.g., register file 106). For more information on unified cache systems and how they are configured, see the following references, which are incorporated herein by reference as expressly stated: U.S. Patent Application Publication No. 2018 / 0322078; and CUDA C Programming Guide, May 2019, PG-02829-001_v10.1.

[0046] Multiple SMs 102a-102n can access global memory 116 through multiple TMAUs 112a-112n (collectively referred to as 112). Each SM 102 is tightly coupled to a corresponding TMAU 112, which is configured to access global memory 116 via global memory interface 114. In some embodiments, the tight coupling between SM 102 and TMAU 112 is one-to-one, and each SM has its own dedicated TMAU 112, but embodiments are not limited thereto. Each TMAU 112 has read / write access to the shared memory 110 and L1 cache 108 of the corresponding tightly coupled SM 102 by issuing requests to the memory subsystem and global memory 116. In some embodiments, in addition to read / write access to the shared memory 110 of its coupled SM, TMAU 112 also has read and / or write access to shared memory on other SMs by issuing requests to the memory subsystem. Distributed shared memory, which can be used by a TMAU of one SM to access shared memory on another SM, is described in U.S. Application No. 17 / 691,690, which is incorporated by reference. Furthermore, a TMAU can transfer multidimensional data structures or other data between a large-capacity global memory and a linear shared global memory accessible by a cooperative group array (CGA) that can be executed on one or more SMs.

[0047] When software running on one or more functional units 104 needs data stored in global memory 116, the software uses a "load" command to start a thread. The load command can load data from global memory 116 and store the data in shared memory 110, making it visible to all threads (e.g., all threads in a thread block). Once the data is stored in shared memory, threads can access the data multiple times.

[0048] Each TMAU 112 enables the circuitry of the processing core within the corresponding SM to continue the mathematical and other processing of the application kernel, while address calculation and memory access operations are outsourced to tightly coupled circuitry dedicated to address calculation and memory access. As described below, the TMAU 112, coupled to the SM 102 and possessing its own hardware circuitry for calculating memory addresses and reading and writing to shared and global memory, can improve the overall application kernel performance by outsourcing any type of data that the TMAU can access. The ability to outsource such data access and perform it asynchronously in cases involving access to large, multidimensional data structures or blocks of data that typically consume hundreds or more clock cycles provides a particularly significant performance improvement.

[0049] Figure 2An example interaction is shown during memory accesses performed by threads running on SM 102, global memory 116, SM 102 coupled to SM 102, shared memory 110, and L2 cache 202 according to some embodiments.

[0050] When a thread running on SM 102 needs to access a data block, the SM determines the access parameters of the data block in global memory and, in operation 204, instructs TMAU 112 to acquire the data block by transmitting a single memory access request. The type of access parameters that need to be provided to TMAU from the SM can vary depending on whether the requested data block is a tensor, as described in detail below. As described in more detail below, requests for non-tensor block data may include, in addition to the global memory address and shared memory address of the requested data, the size of the block to be loaded. Requests for tensor data include a pointer to a tensor descriptor, the location coordinates associated with the requested block, and the shared memory address.

[0051] In some cases, a request from the SM may request data larger than that that can be requested and / or obtained from global memory through a single load / store request. For example, the memory subsystem may only handle requests up to the size of one L2 cache line. Therefore, in response to a single memory access request from the SM requesting a large amount of data (a data structure or block larger than the maximum size allowed for a single request to the memory subsystem), the TMAU 112 forms and issues multiple memory access requests to obtain the full amount of the requested data. The TMAU 112 operates asynchronously with the SM 102 that issued the request and continues to generate multiple memory access requests in operation 206, each with a different address for the corresponding sub-block of the requested data. The multiple memory access requests are transferred from the TMAU 112 to the L2 cache 202.

[0052] Operation 208 represents a response from L2 cache 202 (or global memory) to each of the multiple memory access requests sent by operation 206. Sub-blocks may be written to shared memory 110 in operation 210 and / or by TMAU 112 in operation 212. Operations 212 and 214 may provide the completion status of synchronization request SM 102 and the data request. For example, TMAU may increment a counter as each sub-block is written to shared memory. In some embodiments, each sub-block request generated from TMAU includes a counter address in shared memory, and updating (incrementing) the counter may be performed by the shared memory. SM may monitor the counter to determine when to write the entire requested data block to shared memory. In some embodiments, the request sent from SM includes an address counter address, and SM includes hardware dedicated to monitoring counter synchronization.

[0053] Between operation 206 issuing a memory access request for data and the subsequent synchronization of data writing to shared memory in operation 214, many clock cycles may pass. This interval can be several thousand clock cycles, especially for requests for large amounts of data. However, since SM 102 can request an entire data block from TMAU 112 in a single request 204 and then continue processing instructions, while TMAU 112 asynchronously and independently of SM 112 can obtain data by issuing one or more requests to global memory (e.g., via L2 cache 202), the processing efficiency of SM can be improved. The power consumption of SM can also be reduced by delegating the extensive address calculations required to obtain large amounts of data structures or blocks, as well as the associated coordinates for loading and storing individual sub-blocks of large amounts of data, to the hardware in TMAU.

[0054] Compared to the embodiments of this disclosure, when using the LDGSTS instructions described above, the SM, or more specifically, each thread, calculates the address of each sub-block to be loaded and issues the instructions directly to global memory (e.g., via L2202). The SM must then synchronize itself with the shared memory 110 of the corresponding sub-block. Since each thread issues a corresponding request for each data block, and this request is limited to the maximum size of the request processed by the memory system, a large number of requests can be transferred from the SM to the memory subsystem. The generation of a large number of requests, and the synchronization of the SM and shared memory for each block requested by each thread, incur significant overhead in terms of both processing and power consumption. Compared to the LDGSTS instructions and other prior art methods, the embodiments disclosed herein enable a thread in a thread group on the SM to request all data from all threads in the group from the TMAU, and also enable the thread to continue processing its tasks asynchronously with the TMAU until the TMAU has completed the transfer of the request.

[0055] Access Tensors While TMAU 112 can be used to access any type of data block arrangement, in some embodiments, TMAU includes tensor-specific capabilities. For example, in applications such as deep learning (DL), large amounts of data may be stored in tensors. Tensors can be of any dimension, from one-dimensional tensors such as one-dimensional arrays to n-dimensional tensors such as n-dimensional arrays, where n is a positive number. Although in some embodiments only tensors of dimensions 1-5 are supported, according to some other embodiments, the size and dimension of tensors are limited only by memory, and TMAU 112 does not impose restrictions on the size and / or dimension of tensors that can be requested by SM as blocks.

[0056] The TMAU circuitry enables kernel developers to access sub-blocks within a tensor using coordinates (e.g., (x, y) in a two-dimensional tensor), which are computationally simpler than memory addresses. TMAU translates the coordinates into one or more corresponding memory addresses before making a request to external memory.

[0057] Figures 3A-3B (Collectively referred to as Figure 3) shows the parameters that can be used by SM to access tensor data. Figure 3A A three-dimensional tensor 302 stored in global memory is shown. Tensor 302 can be written to global memory by a process executing on a CPU, GPU, or other processor in a computer system. Some embodiments of this disclosure provide threads executing on one or more SMs of the GPU to read and / or write to tensor 302 located in global memory.

[0058] Tensor 302 is accessed by SM in the form of blocks (such as, for example, box 306) smaller than the entire tensor. Figure 3A The tensor parameters shown include the tensor's dimension, the size of each dimension, the stride of each dimension, and the size of the elements in the tensor. The block to be accessed within the tensor is characterized by the size of each dimension of the block. The dimension of the block is the same as the dimension of the tensor. Tensors may have padding along certain dimensions, as shown by padding the area above and to the right of tensor 302 within tensor 304. Padding can be indicated by the tensor stride in the tensor definition, where the stride of the tensor in a particular dimension is defined as the size of the tensor in that dimension plus the size of the padding in that dimension. Note that the same tensor can be accessed using blocks of different sizes. In this embodiment, for each tensor, all required parameters are defined in a "tensor descriptor," which combines the tensor and access block attributes. The required parameters must be defined in the descriptor before issuing a memory access request to the TMAU.

[0059] A tensor descriptor is a data structure defined in global memory and uniquely identified by its address in global memory. It can be defined on the host side before kernel execution or on the GPU during kernel runtime. Typical tensor access patterns assume loading multiple blocks from the same tensor. Loading the tensor descriptor from global memory for each new TMAU request for a block would be inefficient because global memory latency negatively impacts performance. Therefore, in some embodiments, the TMAU has a dedicated descriptor cache (see [link to documentation]). Figure 6 This allows for the utilization of time tensor access consistency across multiple kernels running on SM.

[0060] Figure 3BA two-dimensional filled tensor 308 is shown. This diagram illustrates the "elements" 310 within the tensor, the block 312 within the tensor, and the padding 314 associated with the dimension shown. The tensor height H and width W, as well as the element size 310, are defined. Tensor 308 is filled with padding 314 in the x-direction. Therefore, the tensor stride in the x-direction includes the width of the padding. Block 312 is the data required by the kernel and also has its own height (block height) and width (block width). The SM can access block 312 by providing the origin 316 of the block solely through its coordinates (x, y) in the tensor coordinate system.

[0061] Figures 4A-4B (Collectively referred to as Figure 4) illustrates some aspects of the operations handled by TMAU when accessing tensors in external memory. For example... Figure 4A The diagram illustrates that a block to be read from tensor 308 (a two-dimensional tensor in this example) can be located at many different positions where the block's anchors are within the tensor. As shown, some anchor positions may result in the box containing memory regions that extend beyond the boundaries of tensor 308.

[0062] Figure 4B The diagram illustrates that out-of-bounds situations can occur in many regions of tensor 308. For example, the diagram shows the corresponding box locations, where the left side of the box, the right side of the block, the top and right sides of the block, the top of the block, or the entire block can extend beyond the boundaries of the tensor in external memory.

[0063] TMAU must properly handle out-of-bounds conditions where the requested block may cross tensor boundaries in global memory. Figure 4B Examples of request blocks extending outside a 2D tensor are shown. If any requested element is outside a tensor, its value may be forced to zero or some other predefined special constant (e.g., a non-numeric (NAN) value).

[0064] The way out-of-bounds access is handled depends on the specific application. In the simplest case, zero is assigned to elements located outside the tensor. A typical example is when convolutional filters are applied to pixels near image boundaries, some of which may be located outside the image.

[0065] In more complex applications, it may be necessary to pad out-of-bounds elements with a dedicated non-zero constant. One example is the fusion of a normalization layer with subsequent convolutional layers in a deep learning neural network. The normalization layer applies a bias and scaling to each element before convolutional processing. Out-of-bounds elements must be set to zero for convolutional filtering to function correctly; however, due to normalization, they are assigned bias values. To handle this, the TMAU can be programmed to assign and identify a special non-numeric (NaN) constant to indicate out-of-bounds access. When tensor data in global memory is written to shared memory, the TMAU may write the special NaN constant to the shared memory location. A kernel might be needed to check if each element in global memory is equal to this special constant. If the special constant is detected, zero is assigned to the element; otherwise, scaling and bias are applied. This processing may only be relevant to floating-point formats during the training phase of deep learning. The special NaN encoding is format-specific and based on the tensor descriptor format settings. See, for example, U.S. Patent Application No. 17 / 497,507, entitled “Neural Network Data Replacement,” filed October 8, 2021, the entire contents of which are incorporated herein by reference.

[0066] Figures 5A-5B (Collectively referred to as Figure 5) This illustrates the parameter groupings used by the TMAU for efficient access to tensors in memory within the context of two-dimensional tensors and their corresponding blocks. The parameters required by the TMAU to uniquely identify blocks within a tensor are divided into three groups: a set of "tensor descriptor" parameters describing the entire tensor, a set of "access descriptor" parameters typically describing blocks within the tensor, and TMAU "instruction parameters" identifying a specific block. The tensor descriptor parameters and access descriptor parameters are as follows... Figure 5A As shown, the TMAU command parameters are as follows: Figure 5B As shown.

[0067] like Figure 5A As shown, in one embodiment, tensor descriptor parameters include tensor height, tensor width, tensor stride, and element size. Tensor stride represents the tensor size (height or width) plus padding in a specific dimension. Access descriptor parameters include block height, block width, and out-of-bounds value. Tensor height, tensor width, tensor stride, block height, and block width are specified per dimension of the tensor. Figure 5B As shown, the TMAU instruction parameters only include the starting coordinates of the block (e.g., (x, y)). Therefore, the starting coordinates of an n-dimensional vector will be an n-dimensional tuple.

[0068] TMAU processing path Figure 6 An example data processing path for TMAU, according to some embodiments, is illustrated schematically. Figure 6In this embodiment, TMAU 612 is included within SM 602 as shown. However, it should be understood that in some embodiments, although TMAU 612 is not physically located within SM 602, it may be tightly coupled to SM 602.

[0069] The Memory Input / Output Controller (MIOC) 604 provides an interface between the request processing pipeline of the SM 602 and the TMAU 612. The TMAU 612 receives memory access requests issued by the SM through the MIOC 604. Received memory access requests are fed into an internal request queue 606. In some embodiments, requests in queue 606 are processed in a first-in, first-out (FIFO) order. However, in other embodiments, requests in the queue may be selected for further processing based on one or more characteristics of the request, such as request type, size of the read or write request, data type of the request, memory space to be accessed, etc.

[0070] Request queue 606 can receive two types of requests: tensors (with tensor descriptors) and non-tensors (linear memory, without tensor descriptors). Requests can be of different request types, such as load, store, reduce, prefetch, etc. For each request for tensor data, the TMAU needs a pointer to a descriptor that provides the necessary information about the tensor to be accessed. In some embodiments, request queue 606 is a single queue that receives both types of requests, while in other embodiments, corresponding queues can service each type of request. In some embodiments, the TMAU may process only requests for tensor data, and in some other embodiments, it may process only requests for non-tensor block data.

[0071] For performance reasons, in some embodiments where the TMAU is configured to receive memory access requests for tensor data, the TMAU maintains a descriptor cache 608 to hold recently used tensor descriptors. Because common access patterns often involve many requests received in close proximity accessing the same tensor descriptor, the descriptor cache can provide reduced latency. The cache can be labeled with the global address of the tensor descriptor. Each received memory access request can specify the global address of the associated tensor descriptor. The cache is connected to a generic cache controller (GCC) 622 via an interface. While processing the current request in the internal request queue 606, the TMAU can check if the descriptor for the next request resides in cache 608. If not (i.e., if it is a miss), a descriptor load request is issued to the GCC to prefetch the descriptor from global memory into cache 608. This parallel processing helps to hide the latency of descriptor prefetching.

[0072] When a request is selected from queue 606 for processing in TMAU 602, if the request is tensor-oriented, the selected request is sent to setup block 610. When a memory access request is received in setup block 610, setup block 610 obtains the corresponding descriptor from descriptor cache 608. Setup block 610 collects and / or computes the necessary parameters for request processing. Although many parameters required for memory access are available (or included) in the descriptor, some additional parameters are received in the memory access request. For example, setup unit circuitry can be configured to perform logic similar to that shown in Table 1 below with reference to FIG8 to calculate the required parameters, such as address calculations based on the tensor descriptor. It also checks the correctness of the request input parameters. As described above, bandwidth utilization for memory access requests from SM to TMAU is optimized by providing the parameters used by multiple memory access requests to be obtained from the corresponding tensor descriptors, and by providing memory access requests from SM that carry only parameters unique to a specific request. Parameters specific to the memory access request, such as block coordinates or addresses, can be carried as direct parameters of the request. The setup block is configured to perform calculations and error checking on the parameters. If the parameters do not meet the predefined TMAU requirements, an error is generated and the request is discarded. The setup block operates in parallel with the request generator 716, providing a pipeline for setting up request generation, thereby reducing latency.

[0073] The request generator 616 is the main TMAU engine. For requests of tensor data, it receives relevant parameters from the setup block and traverses the tensor space by iterating through multidimensional coordinates, mapping coordinates to addresses, checking out-of-bounds conditions, calculating shared memory addresses, calculating global memory addresses, and generating requests to the memory subsystem. The request generator generates as many requests as needed to the memory system to load / store tensor data blocks, while adhering to the maximum size of memory requests processed by the memory subsystem. Typically, the memory subsystem imposes a maximum cache line size (e.g., the size of an L2 cache line) on each request received at the memory subsystem. The request generator optimizes requests to improve the efficiency of the memory subsystem. The processing of the request generator 616 provides automatic generation of access requests for the entire block through dedicated hardware, thereby reducing power consumption. High-level example pseudocode illustrating the processing within the request generator is provided in [link to example]. Figure 7B As shown in the image.

[0074] Data requests are transmitted to the memory subsystem via the Generic Network Interface Controller (GNIC) interface 614, and each request is tracked in the response completion circuit 618. This tracking enables asynchronous processing with the SM. Responses to requests are received at the GNIC response processor 620, which communicates with the request tracking circuit 618 to track the completion status of each request sent from the request generator 716.

[0075] If the memory access request received from the SM is for block data that is not a tensor, in some embodiments, the request can be bypassed by sending the descriptor cache 608 to the request generator 616. For example... Figure 6 As shown, for example, a request for non-tensor block data can be routed from queue 604 to the request generator, bypassing descriptor cache 608 and setup unit 610. However, in some embodiments, such a request can be routed from queue 606 to setup unit 610 before being processed in request generator 616. A request for a large non-tensor data block received from the SM may include the global memory address of the block, the shared memory address of the block, and the size of the block in bytes. For a request for a large non-tensor data block received from the SM, request generator 616 may automatically generate a sequence of requests to the memory subsystem, where each request is for a smaller sub-block of the requested block. The request generator calculates the global memory address of the sub-block based on the global memory address of the block included in the request received from the SM, and the size of the sub-block can be determined based on the maximum size of the requests processed by the memory subsystem. Request completion tracking circuitry 618 tracks memory requests for sub-blocks and responses received from the memory subsystem in the same manner as described above with respect to tensor data blocks.

[0076] Figure 7A and Figure 7B Example parameters are shown. Figure 7B Block 704 shown uses these parameters to track when the TMAU circuitry reads tensor data structure 702. Figure 7A This illustrates a portion of the processing logic implemented in the TMAU hardware. Figure 7B The example high-level pseudocode shown includes parameters for anchor, base, and current element. Figure 7C Example high-level pseudocode is shown, in which SM calls the tensor load operation in TMAU to copy data from global memory to shared memory, and then writes the resulting data back to global memory.

[0077] Figure 7B The pseudocode in the example is a high-level example of some processing steps performed by the TMAU in response to receiving a request from its coupled SM to obtain a block from a tensor in global memory. The pseudocode is arranged in five nested loops, each corresponding to one of the five coordinate axes of the tensor data space. Although this example is for a five-dimensional tensor data space, some embodiments may support N nested loops for an N-dimensional tensor data space, where N can be any positive integer.

[0078] The current element is processed within the innermost loop by specifying its calculated coordinates in each of the five dimensions (coordinates c0, c1, c2, c3, and c4), the address in shared memory where the current element will be loaded, and the global address of the current element. After obtaining the current element, the global memory address and the shared memory address of the next element are obtained by incrementing the global address by the element size of the tensor and the shared memory address by a predefined shared memory address increment (which can be defined in the tensor descriptor and can be based on the element size defined for the tensor). Processing within the innermost loop includes operations performed by TMAU for copying tensor data, such as checking for out-of-bounds conditions.

[0079] The innermost loop provides iterations of elements along dimension 0 (dimension 0-4) to dimension 0 coordinates ("tensorDescriptor.traversalStride[0]") by starting from the coordinates of the requested block in dimension 0 (blockstart0) and incrementing the current coordinate c0 in dimension 0 by the traversal stride of dimension 0. This coordinates exceed the box size in dimension 0 ("blockStart0 + tensorDescriptor.boxSize[0]"; exceeding the block boundary).

[0080] When the innermost loop (which iterates over the tensor elements in dimension 0) exits, the base global address of the next outer dimension (i.e., dimension 1) is increased by the tensor stride defined for dimension 0 ("baseGlobalAddr[1] += tensorDescriptor.tensorStride[0]"). This effectively advances the global address to the next slice. The base global address of each dimension is initially determined based on the global address corresponding to the anchor element of the requested block.

[0081] like Figure 7B As shown, in a manner similar to that described above for dimension 0, each loop provides the number of iterations in the corresponding dimension determined by the starting block coordinates, the traversal stride along that dimension, and the box size for that dimension. It should be noted that the traversal stride and box size for each dimension are defined in the tensor descriptor of the tensor.

[0082] By performing the processing involved in copying data blocks from tensors in the global memory in the hardware, TMAU can significantly reduce the computational load on the SM for data movement, thereby improving the processing efficiency of the SM and also reducing its power consumption.

[0083] Figure 7BThe pseudocode above provides high-level execution logic and omits details related to certain aspects, such as efficient L2 request generation, promiscuity, and handling of out-of-bounds conditions performed by TMAU in reading and / or writing tensors.

[0084] In addition to generating L2 requests (requests to global memory), the TMAU tracks returned data to report TMAU transaction completion. The TMAU must have a dedicated counter to track issued L2 requests. The counter increments each time a request is sent to the L2 cache. It decrements when data is returned from the L2 cache. Once the counter reaches zero, the entire block is loaded into shared memory, and the TMAU can report transaction completion. For efficiency purposes, the TMAU can use a single counter to track a group of multiple back-to-back transactions and report the completion of the last transaction in the group. In some embodiments, the counter may be maintained in a predefined location in shared memory. The SM may include synchronization circuitry to monitor the counter and may implement synchronization barriers, etc., based on the counter.

[0085] Figure 7C Example pseudocode for a convolutional filter with implicit GEMMs, executed by a kernel running on SM, is shown. As mentioned above, a GEMM is generally defined as the operation C = αAB + βC, where A and B are matrix inputs, α and β are scalar inputs, and C is a pre-existing matrix covered by the output. Ordinary matrix multiplication AB is a GEMM where α equals 1 and β equals 0. Many DL applications require such computations. An example of an efficient matrix multiplication and addition implementation that can utilize TMAU is described in U.S. Application No. 17 / 691,406 entitled “Efficient Matrix Multiply and Add with a Groupof Warps,” the entire contents of which are incorporated herein by reference.

[0086] The kernel obtains pointers to tensor descriptors for three tensors: the activation tensor, the weight tensor, and the output tensor, along with the size information for each of these tensors. In GEMM computation, the activation, weight, and output tensors can be represented as matrices A, B, and C, respectively. When the kernel issues a subsequent memory access request (tensorBlockLoad()) to the TMAU, it provides the TMAU with pointers to the tensor descriptors for the activation, weight, and output tensors.

[0087] The logic is organized as a series of nested loops, which replicate the block sequence of each tensor by copying the corresponding block in each iteration of the innermost loop. In each iteration of the innermost loop, the kernel issues a corresponding tensorBlockLoad request to the coupled TMAU to load a block from each of the activation and weight tensors. The tensorBlockLoad request takes the tensor's address in global memory (determined by the SM) and its address in shared memory as parameters; the tensor data in global memory is written to that address. The arrangement of the nested loops allows the outer three loops to iterate vertically, horizontally, and through channels, while the innermost loop iterates via a convolutional filter.

[0088] The NHWC (N (dimensions), height, width, channels) layout is assumed for the activation tensor and the KNWC layout for the weight tensor. The code iterates through the W and H dimensions. It accumulates for each r and s position of the channels (C dimension) and the convolutional filters. For simplicity, iteration through the N and K dimensions is not shown. For a given [c, s, r], TMAU loads the data block from global memory into shared memory. The load is performed for both the activation and weight tensors. After loading the data of both matrices into shared memory, SM may call GEMM computation (computeGEMM()). In some embodiments, GEMM computation is performed by dedicated hardware circuitry, and the result is accumulated into the output matrix. Matrix multiplication is computed in shared memory.

[0089] After performing mathematical operations using tensor data loaded in shared memory, TMAU is powered by the kernel on SM by issuing a request (tensorBlockStore()) and providing the address of the output matrix where the result from GEMM is stored, and the address in the shared memory where the result is to be written, in order to save the result from the shared memory buffer to a tensor in global memory.

[0090] Supports tensor loading mode TMAU supports multiple memory layouts for tensors. For example, a 3D image tensor may have a tensor layout format NDHWC, where the innermost dimension C represents the number of channels (e.g., in an image tensor, each channel can represent a color), the D, H, and W dimensions correspond to the depth, height, and width dimensions, respectively, and N represents the batch size of the tensor.

[0091] In addition to supporting multiple tensor layout formats, TMAU also supports tensors stored in global memory in either non-interleaved or interleaved mode. In interleaved mode, TMAU can support various slice sizes (e.g., 16-byte slices, 32-byte slices, etc.). In some embodiments, the tensor descriptor can specify whether the tensor is in non-interleaved or interleaved mode in global memory, and also specify the size of the slice in interleaved mode.

[0092] Furthermore, in some embodiments, TMAU supports more than one tensor loading mode. For example, tile mode and image-to-column (also known as "im2col") mode can be supported as tensor data loading modes.

[0093] Tile mode is preferred in some cases because it eliminates the need for data copying in implicit universal matrix multiplication (GEMM) implementations, thus providing significant memory bandwidth savings. On the other hand, performance may be penalized in some cases due to tile quantization effects. Tile mode is a general TMAU loading pattern widely used in various deep learning and high-performance computing (HPC) applications. (The above is about...) Figure 7A and Figure 7B An example of tensor traversal for tile patterns is described.

[0094] The im2col mode is primarily used for convolutional kernels based on implicit GEMM. When im2col mode is selected, TMAU performs an image-to-column conversion when loading tensor blocks from global memory. This adds additional complexity to the tensor traversal algorithm.

[0095] In tile mode, the tensor parameter `boxSize[]` uniquely defines the size of the bounding box in the tensor space, which holds all elements that TMAU should load in response to instructions from SM. Each element of `boxSize[]` specifies the size of the bounding box along the corresponding dimension: `boundingBox[i] = boxSize[i]`. The coordinates specified in the TMAU memory access request from SM uniquely define the position of the bounding box in the tensor space.

[0096] In im2col mode, the size and position of the bounding box are defined differently. The number of dimensions of the bounding box is one less than the number of tensor dimensions in the tensor descriptor. Instead of using `boxSize[]`, alternative parameters are provided in the tensor descriptor to support im2col mode. These alternative parameters include: `rangeNDHW`, `rangeC`, `boxBaseCornerDHW`, and `boxFarCornerDHW`. `boxBaseCornerDHW` and `boxFarCornerDHW` define the size and position of the bounding box in the DHW (depth, height, width) space. `boxBaseCornerDHW` specifies the initial coordinates of the bounding box origin, i.e., the top-left corner of the box. `boxFarCornerDHW` specifies the initial position relative to the bottom-right corner. The position of the corner point is defined as a signed offset from the corresponding tensor corner point. Therefore, bounding box corners can be specified both inside and outside the tensor boundary.

[0097] The location of the bounding box corners is affected by the convolution filter size and the selected dilation factor. Angular coordinates can be calculated as half the filter size multiplied by the dilation factor. The precision of the bounding box corners is chosen to provide a wide range of convolution kernel sizes and dilation factors. Based on practical application analysis, smaller-dimensional tensors may require higher precision. For example, speech processing applications using 3D tensors may require dilation factors up to 8K, while image processing applications using 4D or 5D tensors require much smaller dilation factors up to 128.

[0098] The bounding box sizes `boxBaseCornerDHW` and `boxFarCornerDHW` are defined using the following formula: `boundingBox{D,H,W} = tensorSize{D,H,W} - boxBaseCorner{D,H,W} + boxFarCorner{D,H,W}`. For dimension C, the size is defined by the `rangeC` parameter.

[0099] Figure 8A This illustrates how the bounding box depends on the settings of `boxBaseCorner{D,H,W}` and `boxFarCorner{D,H,W}`. This example demonstrates that multiple types of bounding boxes can be used in data structures, and that quantization can be avoided in the `im2col` mode.

[0100] In tile mode, the number of elements to load depends on the `boxSize[]` parameter. When TMAU iterates through a specific dimension, it uses the corresponding value in `boxSize[]` to determine how many elements to load. In im2col mode, `rangeNDHW` is used to determine how many elements to load along the `NDHW` dimension, while `rangeC` is used to determine dimension C. A single TMAU request may require TMAU to traverse multiple images in a batch (N dimensions) to load the requested number of elements. When TMAU switches from the current image to the next image during the traversal of multiple images, it may skip channels outside the range defined by the `rangeC` parameter.

[0101] In tile mode, the TMAU request coordinates specify the bounding box location (origin) in tensor space. In im2col mode, coordinates along the C and N dimensions are used similarly to tile mode; however, coordinates along the W, H, and D dimensions specify the base location (top left corner) of the convolution filter in tensor space. For proper processing, TMAU requires that the base location of the filter is always defined within the bounding box. Furthermore, coordinate offsets for these dimensions must be specified in the TMAU request. Offsets allow for positions relative to the tensor-specified block, thus using only the minimum number of bytes. Offsets are added to the filter base location coordinates to determine the starting position in tensor space from which the loading operation must commence. The same offset is used to position the bounding box relative to the initial coordinates specified in boxBaseCornerDHW. Offsets are applied to a subset of coordinates according to the table defined above. Offsets are defined as unsigned integers with variable precision. The precision depends on the tensor dimensions and is selected based on an earlier adjustment of the bounding box coordinate precision.

[0102] In some embodiments, all offsets are packed into 16 bits within a single register. The number of offsets depends on the tensor dimension; therefore, the precision may vary accordingly. In a typical convolutional kernel, once the filter base is computed, it can be reused for multiple TMAU requests with different coordinate offsets. The number of reuses depends on the size of the convolutional filter. For example, for a 3x3 filter, nine requests would be issued for the same filter base location.

[0103] For staggered layouts, the C-coordinate must be specified based on channel slices rather than individual channels. This applies to both tile and im2col modes.

[0104] Table 1 below shows high-level example pseudocode for the logic implemented in the TMAU, more specifically, in the setup block, to configure tensors and access parameters based on the tensor descriptor identified in the received TMAU request. Table 1: Sample pseudocode for initializing the load tensor (3D-5D dimensions) The following example illustrates the use of the im2col mode. A 3x3 convolution filter is applied to the NHWC tensor (64x14x9x64). Each request loads 64 elements along the N, H, and W dimensions and 8 elements along the C dimension.

[0105] In the first example, such as Figure 8B As shown, the filter can cross tensor boundaries and access surrounding padding (boundaries) that can be defined as zero or constant values. The tensor descriptor parameters are set as follows: tensorSize[0]=64; tensorSize[1]=9; tensorSize[2]=14; tensorSize[4]=64; rangeNDHW = 64; rangeC = 8; boxBaseCornerW= -1; boxBaseCornerH = -1; boxFarCornerW = -1; boxFarCornerH = -1. Figure 8B The example illustrates the processing of requests for coordinates (7, 7, 4, 0) and different coordinate offset values: (0, 0), (1, 1), and (2, 2). This example shows the different boundary regions of the loaded tensor. These are defined as offsets. The requester specifies the boundary regions to TMAU and how many elements need to be loaded (e.g., a series of elements—64 in this example). This can be specified as a parameter in the tensor descriptor. Another parameter, which can be provided at the instruction level, can specify the starting position of the block used for the load request. TMAU knows it must start loading tensor elements from the specified starting position, plus offsets kept within the rectangle shown, and load the specific amount of data.

[0106] In the next example, the filter is configured to remain within the tensor boundaries, so no padding / boundaries are needed on the tensor. The tensor descriptor parameters are set as follows: rangeNDHW = 64; rangeC = 8; boxBaseCornerW = 0; boxBaseCornerH = 0; boxFarCornerW = -2; boxFarCornerH = -2. Figure 8C The processing of requests for coordinates (7,7,4,0) and different coordinate offset values: (0,0), (1,1), and (2,2) is shown.

[0107] For comparison, the handling of a similar convolutional case in tile mode is illustrated in the next example. A single TMAU request might load all the pixels required for convolution computation at all filter locations. To achieve this, additional halo pixels must be loaded. The number of halo pixels depends on the filter size.

[0108] In the next example, a 3x3 convolution filter is applied to the NHWC tensor (64x14x8x64). The filter can extend beyond the tensor boundaries, and access can be defined as the surrounding padding (boundary) with zero or constant values. A single request loads 10x10 tiles along the H and W dimensions and 8 elements along the C dimension. Each loaded 10x10 tile has 2 halo rows and 2 columns. The tensor descriptor parameters are set as follows: tensorSize[0]=64; tensorSize[1]=8; tensorSize[2]=14; tensorSize[4]=64; boxSize[0]=8; boxSize[1]=10; boxSize[2]=10; boxSize[3]=1. For any given filter location, only one 8x8 tile is used for the convolution computation. Figure 8D The processing of a request with coordinates (0, -1, -1, 0) is shown. Negative W and H block coordinates are required to access pixels with tensor out-of-bounds values ​​of zero or constant (padded). The 8x8 tiles shown are used to process different filter locations: (0, 0), (1,1), and (2, 2).

[0109] The following example is similar to the previous one, but the filter must remain within the tensor boundaries and padding / boundaries are not allowed. A single TMAU request loads 8x8 tiles along the H and W dimensions and 8 elements along the C dimension. Each loaded 8x8 tile has 2 aura rows and 2 columns. The tensor descriptor parameters are set as follows: boxSize[0] = 8; boxSize[1] = 8; boxSize[2] = 8; boxSize[3] = 1. For any given filter location, convolution is performed using 6x6 tiles. At any given time, only 36 pixels are used for mathematical operations. This is less than the optimal 64 pixels. This is an example of tile quantization effects that can affect overall performance. Figure 8E The processing of a TMAU request with coordinates (0, 0, 0, 0) is shown. Setting the W and H block coordinates to zero prevents exceeding the tensor boundaries. The 6x6 tiles shown are used to process different filter locations: (0, 0), (1, 1), and (2, 2).

[0110] The tensor descriptor `traversalStride` parameter affects both tile mode and `im2col` mode. In tile mode, a larger `traversalStride` results in a smaller number of tensor locations accessed by the load, thus reducing the total number of loaded elements. In `im2col` mode, for comparison, the number of elements loaded along the NDHW dimensions does not depend on the `traversalStride` along these dimensions: it is equal to the tensor descriptor `rangeNDHW` parameter. However, as with tile mode, the number of elements traversed along the W, H, and D dimensions is affected by the `traversalStride` based on the formula `ceil(boundingBox{D,H,W} / traversalStride{D,H,W})`.

[0111] Figure 8F The traversalStride processing in im2col mode is shown. A 3x3 convolutional filter is applied to the NHWC tensor (64x14x9x64), with a traversalStride of 2. Each request loads 32 elements in the N, H, and W dimensions, and 16 elements in the C dimension. The tensor descriptor parameters are set as follows: tensorSize[0]=64; tensorSize[1]=9; tensorSize[2]=14; tensorSize[4]=64; traversalStride=2; rangeNDHW=32; fangeC=16; boxBaseCornerW=-1; boxBaseCornerH=-1; boxFarCornerW=-1; boxFarCornerH=-1. Figure 8B The processing of a request with coordinates (7, 7, 5, 0) and different coordinate offset values: (0, 0), (1, 1), (2, 2) is shown. Note that in this example, pixels are loaded from the top row of the bounding box, not from the bottom row. They are also loaded from the first and last columns.

[0112] Figure 8G A slightly modified example is shown where the tensor size along the W and H dimensions is reduced by one pixel: NHWC (64x13x8x64). Note that in this example, the pixels are loaded from the top and bottom rows of the bounding box. However, they are not loaded from the last column.

[0113] The next example, such as Figure 8HThe diagram illustrates traversalStride processing in tile mode. A 3x3 convolutional filter is applied to the NHWC tensor (64x14x8x64), resulting in a traversalStride of 2. This is similar to the earlier example where the traversalStride was 1. Figure 8D A single TMAU request can provide pixels for all convolution filter locations by loading additional halo pixels.

[0114] In some embodiments, the TMAU may not have dedicated hardware for convolution dilation processing, and other TMAU circuitry can provide the necessary support for this feature. However, the precision of the im2col coordinate offset and bounding box corner coordinates is chosen to provide a wide range of convolution kernel sizes and dilation factors. Figure 8I This explains how the dilation factor affects the bounding box settings of a 3x3 convolutional filter. Note that dilation affects the position of the box, but not its size.

[0115] Figure 8J This demonstrates how to handle a 2-fold inflation factor in im2col mode. A 3x3 convolutional filter is applied to the NHWC tensor (64x14x9x64). Each request loads 64 elements in the N, H, and W dimensions and 16 elements in the C dimension. The tensor descriptor parameters are set as follows: tensorSize[0]=64; tensorSize[1]=9; tensorSize[2]=14; tensorSize[4]=64; rangeNDHW = 64; rangeC = 16; boxBaseCornerW = -2; boxBaseCornerH = -2; boxFarCornerW = -2; boxFarCornerH = -2. Figure 8J The processing of a request with coordinates (7, 6, 3, 0) and different coordinate offset values: (0, 0), (2, 2), (4, 4) is shown.

[0116] Figure 8K It shows the relationship with Figure 8JA similar example is handled in tile mode. A single TMAU request can provide pixels for all convolution filter locations by loading additional halo pixels. The number of halo pixels depends on the filter size and dilation factor. A 3x3 convolution filter is applied to the NHWC tensor (64x14x8x64). A single request loads 12x12 tiles along the H and W dimensions and 8 elements along the C dimension. Each loaded 12x12 tile has 4 halo rows and 4 columns. The tensor descriptor parameters are set as follows: tensorSize[0]=64; tensorSize[1]=8; tensorSize[2]=14; tensorSize[4]=64; boxSize[0]=8; boxSize[1]=12; boxSize[2]=12; boxSize[3]=1. For any given filter location, only one 8x8 tile is used for convolution computation. Figure 8K The processing of a request with coordinates (0, -2, -2, 0) is shown. Negative W and H block coordinates are required to access pixels that are out of bounds using tensors with zero or constant (padding). The 8x8 tiles shown are used to process different filter locations: (0, 0), (2, 2), and (4, 4).

[0117] Support for mixed tensor data In many applications, TMAU loads data into shared memory in the same order as its layout in global memory. However, some applications require additional data movement to avoid performance degradation. This can be implemented as an application-dependent optimization. TMAU supports non-promiscuous mode, where data is written to shared memory in the same arrangement as in global memory, and promiscuous mode, where data is written to shared memory according to a predefined or configurable promiscuous pattern, resulting in a different arrangement of data than in global memory. When TMAU processes memory access requests, it may generate multiple external memory requests, and for each generated external memory request, it can generate a corresponding target address and promiscuous pattern for the target shared memory. In implementation, two options can be used for tracking the target address and promiscuous pattern—either by sending all information via a memory system containing the request and response, or by storing the information in a trace table in SM and sending the corresponding index to that table via a memory system containing the request and response. In either case, the memory system response can use this information to determine the address and pattern for writing data to the target shared memory.

[0118] In some embodiments, the L2 cache line is organized into four 32B sectors. Shared memory is organized in groups of eight repositories, for a total of four groups. Mapping four sectors in a cache line to specific repository groups offers flexibility: any sector can be mapped to any group, one sector per group. Furthermore, 16B half-sectors can be swapped within a sector. This provides additional flexibility for mapping 16B amounts to subgroups of the four repositories.

[0119] Data is organized in a specific order in global memory; however, this order may not match the order in which applications access data in shared memory. A good example is row-matrix organization versus column-matrix access. This difference in data organization can lead to repository conflicts when accessing shared memory. To avoid this problem, data can be loaded into shared memory through a haphazard arrangement of shared memory repositories. L2 cache row sectors are mapped to shared memory repository groups and subgroups according to a predefined pattern that guarantees the avoidance of read and write repository conflicts. TMAU supports multiple patterns based on specific tensor layouts. In turn, data consumers must understand these patterns and access data accordingly.

[0120] In some embodiments, the TMAU can co-occur data loaded into row-organized shared memory. In the example, the shared memory is organized by row, with each row being 128B (128 bytes) and having a unique address. The shared memory library co-occurrence pattern can be encoded in an 8x8 table, where each entry represents the library subgroup ID of a 16B subblock within a 128B data block. The appropriate row is selected from the table based on the last 3 bits of the target shared memory address (row ID). Note that these bits are taken from a logical address within the CTA shared memory region. It is an offset from the base address of that region. It is not necessarily the same as the shared memory physical address.

[0121] exist Figure 9A The example library allocation table for the swizzle_128B mode is shown in the image.

[0122] Figure 9B-9D It shows according to Figure 9A Example data layout of the library allocation table in global and shared memory for swizzle_128B mode. Figure 9BThis displays a 4D NHWC tensor with dimensions of 1x10x10x64 (i.e., N=1, H=10, W=10, and C=64) in global memory. 2B / channel and 64-channel occupy 128B. Each enumerated unit, sometimes called a pixel, represents 8 channels (16B). Image 902 has a W and H size of 10 and includes a halo pixel 906 to support a 3x3 convolutional filter 904 along an 8x8 image patch. During processing, the convolutional filter moves one pixel iteratively left-right and up-down. The unit is... Figure 9A -D enumerates them in the order they are stored in global memory. Channel ranges are represented with different fill patterns.

[0123] Figure 9C This shows H=0 and 1 in the global memory. Figure 9B A portion of the tensor shown. Figure 9C Each row of cells in the array represents a single 128B L2 cache line. Figure 9D This illustrates how the same data, according to an embodiment, is stored in shared memory. Each row represents 128 bytes of data distributed across the memory repository. The data is mixed according to a table of the swizzle_128B pattern. Figure 9D On the right, from the perspective of the GMMA application, a data view is shown with filter positions R=0 and S=-0. GMMA must know the repository mix and stride to provide the correct data across the 16 8x8 tiles.

[0124] This promiscuous adaptation allows for implementations where the order in which data is stored in global memory differs from the order in which it is stored in shared memory. In some embodiments, the TMAU provides scrambling of the data as it moves from global memory to shared memory because, for some applications, the SM reads data vertically (e.g., in data columns). Furthermore, when writing to shared memory, the TMAU considers the memory repository layout within the shared memory to optimize subsequent read accesses of that data by the SM. In the illustrated example, the shared memory is planned in repositories, specifically in 8 repositories. At any given clock cycle, each repository is read, but only a small portion of the data from any given repository can be read. In the figure, each filled pattern represents data written to different repositories in the shared memory according to the promiscuous pattern of the tensor. If data from H=0 W=0-7 is to be read from shared memory, and if this data in shared memory is arranged in the same way as in global memory, reading this data while avoiding repository conflicts requires 8 clock cycles. Therefore, as... Figure 9DAs shown on the left, the data from H=0 W=0-7 is distributed across all eight banks in the shared memory so that all this data (i.e., the data from H=0 W=0-7) is read in parallel across the eight banks. This increases the data throughput per clock cycle.

[0125] exist Figure 9D On the right, the rightmost column shows the 8x8 tiles for each H when W=0, with arrows indicating the locations in shared memory where tiles H=0, W=0 and H=1, W=0 (enumerating tiles 0 and 80 respectively) are written. Similarly, the second column from the right shows the 8x8 tiles for each H when W=1, with arrows indicating the locations in shared memory where tiles H=0, W=1 and H=1, W=1 (enumerating tiles 0 and 80 respectively) are written. This is based on a pre-configured table (e.g., in TMAU). Figure 9A The table shown is used to perform mixing.

[0126] In some embodiments, the GMMA is a fixed-function hardware unit within the GPU Tensor Core, configured to perform matrix-to-matrix multiplications into an accumulator. For example, two 16x16 matrices can be multiplied into an accumulator matrix using a GMMA. In some embodiments, the GMMA can be limited to matrices smaller than a predefined size. When two matrices are to be multiplied, the GMMA, in an example embodiment, is the user of data fed by the TMAU. When a matrix-to-matrix multiplication is required in a computational kernel running on the SM, the kernel request can request the TMAU to copy the data of each of the two matrices into shared memory, and then issue a matrix-to-matrix multiplication request to the GMMA. In response, the GMMA can perform its multiplication operation using the data already loaded into shared memory by the TMAU. If promiscuous mode is used, the kernel can read data from shared memory based on promiscuous mode information, perform its computation, and then write the result back to shared memory. Promiscuous mode is based on a pre-configured table, such as one in the TMAU. Figure 9A Use the table shown to perform the operation.

[0127] In some embodiments, the GMMA circuit can be configured to, for example, Figure 9D Shared memory data is read from the 8x8 pixel tile shown on the right. To obtain the data at positions R=0 and S=0 (see...) Figure 9B In the global memory, where there are no inconsistencies (R=0 S=0 in the image), all channels 0-63 at position R=0 S=0 need to be read from shared memory. For the first 8x8 pixel tile read by GMMA, such as... Figure 9D As shown in the upper right corner of the image on the right side, for positions R=0, S=0 pixels, channels C=0-7, H=0, and W=0-7 are read. Because... Figure 9DThe data shown is mixed together in the shared memory, and all channels 0-63 of the eight positions, including R=0 and S=0, can be read within eight clock cycles.

[0128] GMMA operations can be performed using a 3x3 convolutional filter (904) by a convolutional kernel in applications such as... Figure 9B The image 902 shown is invoked. For each location, R=0, S=0, etc., the filter needs to perform matrix multiplication on the 3x3 box, where the location is as follows: Figure 9B The top-left position is shown in the bottom right image. However, the GMMA circuit may read the 8x8 block on each read.

[0129] Multicast support TMAU supports programmatic multicast, where a single TMAU generates a load request, but data is delivered to multiple targets (e.g., SMs). For example, in response to a load request from a kernel executing on a first SM, a TMAU request coupled to the first SM originates tensor data or other data blocks from global memory and is written not only to the shared memory of the first SM (in some embodiments, the requesting SM does not need to receive the requested data) but also to the shared memory of one or more other SMs. To support this, the requesting TMAU is characterized by being provided with a list of received CTAs. In some embodiments, the received CTA IDs can be encoded in a 16-bit mask, where each bit corresponds to a specific CTA ID. In some embodiments, a data request with a multicast option initiates a TMAU multicast request. The mask of the target CTAs can be encoded in the target address provided to the instruction.

[0130] Each receiver CTA needs to detect transaction completion. Completion detection can be based on an arrival / wait synchronization mechanism. For example, each received data packet can include a shared memory address corresponding to the arrival / wait structure location, and a counter in the structure can be updated based on the number of data bytes received. Receiver CTAs can implement synchronization based on barriers on the counter, etc.

[0131] To support preemption, the TMAU tracks received data packets to detect transaction completion. Typically, all bookkeeping is arranged locally within the TMAU. However, in a multicast scenario, the requesting TMAU must consider transaction completion for all receivers. Therefore, additional acknowledgment mechanisms can be established across multiple TMAUs. Each time a TMAU receives data, it must relay an event to the requesting TMAU. The requesting TMAU represents the total number of data packets received by all receivers. An example multicast implementation that can be implemented using TMAUs is described in U.S. Application No. 17 / 691,288, entitled “Programmatically Controlled Data Multicasting Across Multiple Compute Engines,” the entire contents of which are incorporated herein by reference.

[0132] Prefetch support In addition to loading tensor data, TMAU also supports data prefetch requests to prefetch data from global memory DRAM to the L2 cache. This provides an opportunity to reduce tensor load latency and improve overall performance. Prefetching can be particularly beneficial for multicast operations where latency impacts the execution of multiple CTAs. Prefetch request processing is similar to other load operations, but TMAU does not need to perform any type of completion tracking, etc. For tensor data, prefetch request processing is somewhat similar to load operations, where the tensor descriptor and coordinates define how the request should be processed. However, for tensor data prefetch requests, TMAU may not be able to handle shared memory / global alignment and requests processed at the sector or cache line granularity.

[0133] Storage and reduction requests TMAU storage requests copy data blocks from shared memory to global memory. Data in shared memory is processed in linear address space order; however, the target memory is treated as a multidimensional tensor. The maximum dimension is the same as that of load requests.

[0134] Similar to TMAU loading, a tensor descriptor pointer, a shared memory base address, and the coordinates of the target block in tensor space are provided to the TMAU storage request. Storage requests can be executed in both tile and im2col modes. Storage requests can also support staggered layouts and can specify a shared memory library promiscuous mode. Storage with traversal strides can be supported. In some embodiments, storage operations can also support handling out-of-bounds conditions using ZFILL / CFILL. Furthermore, in some embodiments, TMAU supports copying data from shared memory to global memory or copying data from shared memory to shared memory with reduction. Supported reduction operations may include, but are not limited to, any one of AND, ADD, XOR, MIN, MAX, DEC, OR, and INC.

[0135] No descriptor request Widespread application execution does not require knowledge of the underlying data layout for memory-to-memory transactions. In this case, data is treated as a sequential array of blocks of a predetermined size. In some embodiments, for example, a default block size of 16B can be configured for TMAU operations. Memory access requests for non-tensor data blocks are much simpler than those for tensors, and in some embodiments, only the source address, destination address, and number of blocks are needed to perform the transfer. All of these parameters can be specified at the instruction level (i.e., provided in the request for TMAU) without requiring associated tensor descriptors stored in global memory. This simplifies the programming model because the steps of tensor descriptor definition can be eliminated for such memory access requests. If the number of blocks to be transferred is zero, these instructions are treated as no-ops (NOPs).

[0136] TMAU supports dedicated instructions for descriptorless data transfers (also known as non-tensor data requests). These instructions can be used to copy data from global memory to shared memory, from shared memory to global memory, and from shared memory to shared memory. In another embodiment, global-to-global copying can be implemented. Furthermore, another instruction performs reduction by copying data from shared memory to global memory or from shared memory to shared memory. Supported reduction operations can include, but are not limited to, any of AND, ADD, XOR, MIN, MAX, DEC, OR, and INC. TMAU supports descriptorless data prefetch requests from DRAM to L2.

[0137] Synchronization and Transaction Completion TMAU supports request completion events. In some embodiments, arrival / waiting barriers are used as a completion detection mechanism. Each TMAU load request requires the shared memory address where the barrier structure resides. TMAU includes this address in each L2 request. The barrier structure is updated accordingly when data arrives at the target SM. TMAU itself does not participate in barrier updates. This mechanism can be used for both unicast and multicast requests.

[0138] In addition, TMAU supports dedicated instructions that can be used to detect the completion of all previously issued TMAU requests.

[0139] TMAU's programming model TMAU is designed to move large blocks of tensors or other data between global and shared memory. A single TMAU load request can bring kilobytes, megabytes, or even larger amounts of data, which can be processed by multiple threads and CTAs. Similarly, large blocks of shared memory data generated by large arrays of threads can be saved to global memory as tensors or other forms through a single TMAU store operation.

[0140] The scalar nature of TMAU requests does not align well with the multithreaded nature of the CUDA programming paradigm. Therefore, some implementations provide an intuitive and uninterrupted programming model that can be integrated with the CUDA environment for using TMAU in applications. This programming model offers flexibility in program development and is intuitive and easy to learn for application developers.

[0141] In typical deep learning applications, TMAU is expected to be used iteratively. Multiple CTAs traverse tensors stored in global memory by accessing different tiles. In each iteration, tensor blocks (tiles) are extracted and processed. For each block, the application determines its location in tensor space by calculating multidimensional coordinates. Furthermore, the application must compute the shared memory addresses used to store these blocks.

[0142] The scalar nature of TMAU instructions makes the Unified Data Path (UDP) and Unified Register File (URF) efficient execution sites. This applies not only to TMAU instructions but also to the surrounding code that generates the necessary instruction parameters. This approach eliminates code execution redundancy, saves register file (RF) size, bandwidth, power consumption, and free vector data paths. Due to the iterative nature of TMAU-related code, it is important to keep the iterative parameters in the URF. Any URF / RF loading / storage will result in performance penalties and additional power consumption.

[0143] In some embodiments, a mechanism is provided to help the compiler identify single-threaded bundle semantics of nearby code blocks and express them via CUDA and PTX (Parallel Thread Execution Instruction Set Architecture). The modification adds the ".one" modifier. In the following code, the suggested modifier forces the selection of a single thread for execution: The execution thread is selected from the set of active threads defined by the mask. It is important to consistently select the same thread each time a code block is executed. Note that `_warpsync.exclusive` causes all threads to synchronize before and after the code block execution. The proposed programming model simplifies code analysis, provides an opportunity to generate TMAU-related code for UDP execution, and retains the relevant data in the URF.

[0144] The CUDA-level model sits atop the PTX architecture, where a single thread is consistently selected for code block execution. In the following code, the `__one_sync(mask)` function provides the necessary functionality: In some embodiments, TMAU-based access is implemented through a set of functions. Four C-style groups are defined to cover the following cases: tile payload with L2 descriptors, tile payload without tensor descriptors, im2col payload with tensor descriptors, and im2col payload without tensor descriptors. These functions can take tensor descriptor pointers, shared memory target addresses, shared memory addresses for reaching / waiting for barriers, tensor coordinate sets of the access block origin, pipeline structures, and optional tensor descriptors as input parameters. The im2col group also expects coordinate offsets within the convolution kernel.

[0145] In an example embodiment, the kernel executing on the SM can issue a memory access request to the TMAU to copy tensors between global and shared memory using tensor copy instructions of the following form: Memory access requests to the TMAU to prefetch tensor data into the L2 cache can be issued via tensor prefetch instructions of the following form: Memory access requests to the TMAU to copy non-tensor data blocks between global and shared memory can be issued via block copy instructions of the following form: Memory access requests to the TMAU to prefetch non-tensor data blocks from global memory into the L2 cache can be issued via block prefetch instructions of the following form: prefetch_block address number_blocks.

[0146] Example of a parallel processing GPU architecture using TMAU An illustrative example architecture of the TMAU disclosed herein will now be described. The following information is for illustrative purposes only and should not be construed as limiting in any way. Any of the following features may be optionally combined, excluding or not excluding other features described.

[0147] Figure 10 A parallel processing unit (PPU) 1000 according to one embodiment is illustrated. In one embodiment, the PPU 1000 is a multi-threaded processor implemented on one or more integrated circuit devices. The PPU 1000 is a latency-hiding architecture designed to process multiple threads in parallel. A thread (e.g., an execution thread) is an instantiation of a set of instructions configured to be executed by the PPU 1000. In one embodiment, the PPU 1000 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline to process three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device such as a liquid crystal display (LCD). In other embodiments, the PPU 1000 can be used to perform general-purpose computing. In some other embodiments, the PPU 1000 is configured to implement large neural networks in deep learning applications or other high-performance computing applications.

[0148] One or more PPU 1000s can be configured to accelerate thousands of high-performance computing (HPC), data center, and machine learning applications. The PPU 1000 can also be configured to accelerate numerous deep learning systems and applications, including autonomous vehicle platforms, deep learning, high-precision speech, image, and text recognition systems, intelligent video analytics, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations.

[0149] like Figure 10As shown, PPU 1000 includes an input / output (I / O) unit 1005, a front-end unit 1015, a scheduler unit 1020, a job allocation unit 1025, a hub 1030, a crossbar (Xbar) 1070, one or more general purpose processing clusters (GPCs) 1050, and one or more partitioning units 1080. PPU 1000 can be connected to a host processor or other PPU 1000 via one or more high-speed NVLink interconnects 1010. PPU 1000 can be connected to a host processor or other peripheral devices via interconnect 1002. PPU 1000 can also be connected to a memory including multiple memory devices 1004. In one embodiment, memory device 1004 may include multiple dynamic random access memory (DRAM) devices. The DRAM devices may be configured as a high-bandwidth memory (HBM) subsystem, and multiple DRAM dies are stacked within each device.

[0150] The NVLink 1010 interconnect enables system expansion and includes one or more PPU 1000s coupled with one or more CPUs, supporting cache coherence between the PPU 1000 and the CPU, as well as CPU master control. The NVLink 1010 transmits data and / or commands to other units of the PPU 1000, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown), via hub 1030. NVLink 1010 coupling... Figure 13A and Figure 13B To describe in more detail.

[0151] I / O unit 1005 is configured to send and receive communications (e.g., commands, data, etc.) from a host processor (not shown) via interconnect 1002. I / O unit 1005 may communicate directly with the host processor via interconnect 1002 or via one or more intermediate devices, such as memory bridges. In one embodiment, I / O unit 1005 may communicate with one or more other processors, such as one or more PPUs 1000, via interconnect 1002. In one embodiment, I / O unit 1005 implements a Peripheral Component Interconnect (PCIe) interface for communication via a PCIe bus, and interconnect 1002 is a PCIe bus. In alternative embodiments, I / O unit 1005 may implement other types of well-known interfaces for communicating with external devices.

[0152] I / O unit 1005 decodes packets received via interconnect 1002. In one embodiment, a packet represents a command configured to cause PPU 1000 to perform various operations. I / O unit 1005 sends the decoded commands to various other units of PPU 1000, as the commands may specify. For example, some commands may be sent to front-end unit 1015. Other commands may be sent to hub 1030 or other units of PPU 1000, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In other words, I / O unit 1005 is configured to route communication between various logical units of PPU 1000.

[0153] In one embodiment, a program executed by the host processor encodes a command stream in a buffer that provides a workload to the PPU 1000 for processing. The workload may include instructions and data to be processed by those instructions. The buffer is an area in memory accessible (e.g., read / write) by both the host processor and the PPU 1000. For example, I / O unit 1005 may be configured to access a buffer in system memory connected to interconnect 1002 via a memory request transmitted over interconnect 1002. In at least one embodiment, the host processor writes the command stream to the buffer and then sends a pointer indicating the start of the command stream to the PPU 1000. Front-end unit 1015 receives pointers to one or more command streams. Front-end unit 1015 manages one or more streams, reads commands from the streams, and forwards the commands to the respective units of the PPU 1000.

[0154] Front-end unit 1015 is coupled to scheduler unit 1020, which configures various GPCs 1050 to process tasks defined by one or more flows. Scheduler unit 1020 is configured to track status information related to the various tasks managed by scheduler unit 1020. Status can indicate which GPC 1050 a task is assigned to, whether the task is active or inactive, the priority associated with the task, etc. Scheduler unit 1020 manages multiple tasks executing on one or more GPCs 1050.

[0155] Scheduler unit 1020 is coupled to job allocation unit 1025, which is configured to dispatch tasks for execution on GPC 1050. Job allocation unit 1025 can track multiple scheduled tasks received from scheduler unit 1020. In one embodiment, job allocation unit 1015 manages a pool of pending tasks and an active task pool for each GPC 1050. The pool of pending tasks may include multiple time slots (e.g., 32 time slots) containing tasks assigned to a particular GPC 1050 for processing. The active task pool may include multiple time slots (e.g., 4 time slots) for tasks actively processed by GPC 1050. As GPC 1050 completes the execution of a task, the task is evicted from the active task pool of GPC 1050, and one of other tasks is selected from the pool of pending tasks and scheduled for execution on GPC 1050. If an active task is already idle on the GPC1050, such as while waiting for data dependencies to be resolved, the active task can be evicted from the GPC1050 and returned to the task pool, while another task in the task pool is selected and scheduled to be executed on the GPC1050.

[0156] The work allocation unit 1025 communicates with one or more GPCs 1050 via an XBar 1070. The XBar 1070 is an interconnect network that couples many units of the PPU 1000 to other units of the PPU 1000. For example, the XBar 1070 can be configured to couple the work allocation unit 1025 to a specific GPC 1050. Although not explicitly shown, one or more other units of the PPU 1000 can also be connected to the XBar 1070 via a hub 1030.

[0157] Tasks are managed by scheduler unit 1020 and assigned to GPC 1050 by job allocation unit 1025. GPC 1050 is configured to process tasks and produce results. Results can be consumed by other tasks in GPC 1050, routed to different GPCs 1050 via XBar 1070, or stored in memory 1004. Results can be written to memory 1004 via partitioning unit 1080, which implements a memory interface for writing data to or reading data from memory 1004. Results can be transferred to another PPU 1000 or CPU via NVLink 1010. In one embodiment, PPU 1000 includes U partitioning units 1080, which is equal to the number of separate and distinct memory devices 1004 coupled to PPU 1000. Partitioning units 1080 are described below. Figure 11B To describe in more detail.

[0158] In one embodiment, the host processor executes a driver core that implements an application programming interface (API) that enables one or more applications executing on the host processor to schedule operations for execution on the PPU 1000. In one embodiment, multiple computing applications are executed concurrently by the PPU 1000, and the PPU 1000 provides isolation, Quality of Service (QoS), and independent address spaces for the multiple computing applications. Applications can generate instructions (e.g., API calls) that cause the driver core to generate one or more tasks for the PPU 1000 to execute. The driver core outputs the tasks to one or more streams processed by the PPU 1000. Each task may include one or more associated thread groups, referred to herein as a warp. In one embodiment, a warp includes 32 associated threads that can execute in parallel. A cooperating thread can refer to multiple threads, including instructions for performing tasks and exchanging data via shared memory. Layered packages of threads, cooperative threads, and threads such as cooperative thread arrays (CTA) and cooperative group arrays (CGA) according to some embodiments are described in more detail in U.S. Application No. 17 / 691,621 entitled “Cooperative Group Arrays”, filed March 10, 2022, the entire contents of which are incorporated herein by reference.

[0159] Figure 11A An embodiment is shown. Figure 10 The PPU 1000 has a GPC 1050. (For example...) Figure 11A As shown, each GPC 1050 includes multiple hardware units for processing tasks. In one embodiment, each GPC 1050 includes a pipeline manager 1110, a pre-raster operation unit (PROP) 1115, a raster engine 1125, a work assignment crossbar switch (WDX) 1180, a memory management unit (MMU) 1190, and one or more data processing clusters (DPCs) 1120. It should be understood that... Figure 11A The GPC1050 may include replacements Figure 11A The units shown or except Figure 11A Other hardware units besides the units shown.

[0160] In one embodiment, the operation of GPC 1050 is controlled by pipeline manager 1110. Pipeline manager 1110 manages the configuration of one or more DPCs 1120 to handle tasks assigned to GPC 1050. In one embodiment, pipeline manager 1110 may configure at least one of one or more DPCs 1120 to implement at least a portion of the graphics rendering pipeline, neural networks, and / or computation pipelines. For example, with respect to the graphics rendering pipeline, DPC 1120 may be configured to execute vertex shader programs on programmable streaming multiprocessor (SM) 1140. Pipeline manager 1110 may also be configured to route packets received from job allocation unit 1025 to appropriate logic units within GPC 1050. For example, some packets may be routed to fixed-function hardware units in PROP 1115 and / or raster engine 1125, while other packets may be routed to DPC 1120 for processing by primitive engine 1135 or SM 1140.

[0161] PROP unit 1115 is configured to route data generated by raster engine 1125 and DPC 1120 to raster operation (ROP) unit, in conjunction with Figure 11B For a more detailed description, PROP unit 1115 can also be configured to perform optimizations for color blending, organize pixel data, perform address translation, and so on.

[0162] Each DPC 1120 included in GPC 1050 includes an M-pipeline controller (MPC) 1130, a primitive engine 1135, and one or more SMs 1140. The MPC 1130 controls the operation of the DPC 1120, routing packets received from the pipeline manager 1110 to the appropriate units within the DPC 1120. For example, packets associated with vertices may be routed to the primitive engine 1135, which is configured to retrieve vertex attributes associated with the vertex from memory 1004. Conversely, packets associated with shader programs may be sent to the SMs 1140.

[0163] The SM 1140 includes a programmable streaming processor configured to process tasks represented by multiple threads. Each SM 1140 is multithreaded and configured to execute multiple threads (e.g., 32 threads) from a specific thread group concurrently. In one embodiment, the SM 1140 implements a SIMD (Single Instruction, Multiple Data) architecture, where each thread in a group of threads (e.g., a thread bundle) is configured to process a different dataset based on the same instruction set. All threads in the thread group execute the same instructions. In another embodiment, the SM 1140 implements a SIMT (Single Instruction, Multiple Threads) architecture, where each thread in a group of threads is configured to process a different dataset based on the same instructions, but where individual threads in the thread group are allowed to diverge during execution. In one embodiment, a program counter, call stack, and execution state are maintained for each thread bundle, thereby achieving concurrency between the thread bundle and serial execution within the thread bundle when threads in the thread bundle diverge. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, thereby ensuring equal concurrency among all threads within and between thread bundles. To maximize efficiency, while maintaining execution state for each individual thread, threads executing the same instructions can be converged and executed in parallel. The following section combines... Figure 12 A more detailed description of SM 1140.

[0164] The MMU 1190 provides an interface between the GPC 1050 and the partition unit 1080. The MMU 1190 can provide virtual-to-physical address translation, memory protection, and arbitration of memory requests. In one embodiment, the MMU 1190 provides one or more Translation Lookaside Buffers (TLBs) for performing virtual-to-physical address translation in memory 1004.

[0165] Figure 11B An embodiment is shown. Figure 10 The PPU 1000's memory partition unit 1080. For example... Figure 11BAs shown, the memory partition unit 1080 includes a raster operation (ROP) unit 1150, a secondary (L2) cache 1160, and a memory interface 1170. The memory interface 1170 is coupled to the memory 1004. The memory interface 1170 can implement a 32, 64, 128, or 1024-bit data bus, or a similar implementation for high-speed data transfer. In one embodiment, the PPU 1000 includes U memory interfaces 1170, one memory interface 1170 per pair of partition units 1080, wherein each pair of partition units 1080 is connected to a corresponding memory device 1004. For example, the PPU 1000 can be connected to up to Y memory devices 1004, such as a high-bandwidth memory stack or a graphics dual data rate version 5 synchronous dynamic random access memory, or other types of persistent storage.

[0166] In one embodiment, memory interface 1170 implements the HBM2 memory interface, and Y is equal to half of U. In one embodiment, the HBM2 memory stack is located on the same physical package as the PPU 1000, providing significant power savings and area savings compared to conventional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory dies, and Y=4, with each die comprising two 128-bit channels for a total of eight channels and a 1024-bit data bus width.

[0167] In one embodiment, the memory 1004 supports Single Error Correction Double Error Detection (SECDED) error correction code (ECC) to protect data. ECC provides higher reliability for computing applications sensitive to data corruption. Reliability is especially important in large-scale cluster computing environments where the PPU 1000 processes very large datasets and / or long-running applications.

[0168] In one embodiment, the PPU 1000 implements a multi-level memory hierarchy. In one embodiment, the memory partitioning unit 1080 supports unified memory to provide a single unified virtual address space for the CPU and PPU 1000 memory, thereby enabling data sharing between virtual memory systems. In one embodiment, the frequency of PPU 1000 accesses to memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 1000 that accesses pages more frequently. In one embodiment, the NVLink 1010 supports an address translation service that allows the PPU 1000 to directly access the CPU's page tables and provides full access to the CPU's memory through the PPU 1000.

[0169] In one embodiment, the replication engine transfers data between multiple PPUs 1000 or between a PPU 1000 and a CPU. The replication engine can generate page faults for addresses not mapped to page tables. Memory partitioning unit 1080 can then service the page faults, mapping the addresses to page tables, after which the replication engine performs the transfer. In traditional systems, multiple replication engines operate on fixed (i.e., non-pageable) memory across multiple processors, substantially reducing available memory. In the event of a hardware page fault, the address can be passed to the replication engine without concern for whether a memory page resides, and the replication process is transparent.

[0170] Data from memory 1004 or other system memory can be retrieved by memory partitioning unit 1080 and stored in L2 cache 1160, which is located on-chip and shared among various GPCs 1050. As shown, each memory partitioning unit 1080 includes a portion of the L2 cache 1160 associated with the corresponding memory device 1004. Lower-level caches can then be implemented in individual cells within the GPC 1050. For example, each in SM1140 can implement a Level 1 (L1) cache. The L1 cache is a private memory dedicated to a specific SM 1140. Data can be retrieved from L2 cache 1160 and stored in each L1 cache for processing in the functional units of SM 1140. L2 cache 1160 is coupled to memory interface 1170 and XBar 1070.

[0171] ROP unit 1150 performs graphic raster operations related to pixel color, such as color compression and pixel blending. ROP unit 1150, in conjunction with raster engine 1125, also performs depth testing, receiving depth from the culling engine of raster engine 1125 for sample locations associated with pixel fragments. It tests the depth for the corresponding depth in the depth buffer at the sample location associated with the fragment. If the fragment passes the depth test for that sample location, ROP unit 1150 updates the depth buffer and sends the result of the depth test to raster engine 1125. It will be recognized that the number of partition units 1080 can differ from the number of GPCs 1050; therefore, each ROP unit 1150 can be coupled to each GPC 1050. ROP unit 1150 tracks packets received from different GPCs 1050 and determines whether the result generated by ROP unit 1150 should be routed to a GPC 1050 via XBar 1070. Although in Figure 11BROP unit 1150 is included within memory partition unit 1080, but in other embodiments, ROP unit 1150 may be located outside memory partition unit 1080. For example, ROP unit 1150 may reside in GPC 1050 or other units.

[0172] Figure 12 An embodiment is shown. Figure 11A The streaming multiprocessor 1140. For example... Figure 12 As shown, SM 1140 includes instruction cache 1205, one or more scheduler units 1210, register file 1220, one or more processing cores 1250, one or more special function units (SFU) 1252, one or more load / store units (LSU) 1254, interconnect network 1280, and shared memory / L1 cache 1270.

[0173] As described above, the work allocation unit 1025 schedules tasks to execute on the GPC 1050 of the PPU 1000. Tasks are assigned to a specific DPC 1120 within the GPC 1050, and if a task is associated with a shader program, it is assigned to an SM 1140. The scheduler unit 1210 receives tasks from the work allocation unit 1025 and manages the instruction scheduling of one or more thread blocks assigned to the SM 1140. The scheduler unit 1210 schedules the thread blocks to execute as thread bundles of parallel threads, wherein each thread block is assigned at least one thread bundle. In one embodiment, each thread bundle executes 32 threads. The scheduler unit 1210 can manage multiple different thread blocks, assign thread bundles to different thread blocks, and then dispatch instructions from multiple different cooperative groups to various functional units (e.g., core 1250, SFU 1252, and LSU 1254) in each clock cycle.

[0174] Collaboration groups are a programming model for organizing groups of communicating threads. They allow developers to express the granularity at which threads are communicating, enabling richer and more efficient parallel decompositions. The Collaboration Startup API supports synchronization between blocks of threads to execute parallel algorithms. The conventional programming model provides a single, simple construct for synchronizing collaborating threads: a barrier across all threads in a block (e.g., the `syncthreads()` function). However, programmers often define thread groups at a smaller granularity than thread blocks and synchronize within those groups to achieve higher performance, design flexibility, and software reuse through a set of group-wide functional interfaces.

[0175] Collaboration groups enable programmers to explicitly define thread groups at the sub-block (i.e., as small as a single thread) and multi-block granularity, and perform collection operations, such as synchronizing threads within a collaboration group. This programming model supports clean composition across software boundaries, allowing library and utility functions to be safely synchronized in their local environment without having to make assumptions about convergence. Collaboration group primitives enable new patterns of collaborative parallelism, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire mesh of thread blocks. Hierarchical thread groups, such as Collaborative Thread Arrays (CTAs) and Collaborative Group Arrays (CGAs) according to some embodiments, are described in more detail in U.S. Application No. 17 / 691,621, which is incorporated herein by reference.

[0176] Scheduling unit 1215 is configured to send instructions to one or more functional units. In this embodiment, scheduler unit 1210 includes two scheduling units 1215, which enable two different instructions from the same thread bundle to be scheduled in each clock cycle. In another embodiment, each scheduler unit 1210 may include a single scheduling unit 1215 or additional scheduling units 1215.

[0177] Each SM 1140 includes a register file 1220 that provides a set of registers for functional units of the SM 1140. In one embodiment, the register file 1220 is partitioned between each functional unit, thereby allocating a dedicated portion of the register file 1220 for each functional unit. In another embodiment, the register file 1220 is partitioned between different thread bundles executed by the SM 1140. The register file 1220 provides temporary storage for operands of data paths connected to functional units.

[0178] Each SM 1140 includes multiple processing cores 1250. In one embodiment, the SM 1140 includes a large number (e.g., 128, etc.) of different processing cores 1250. Each core 1250 may include a fully pipelined, single-precision, double-precision, and / or mixed-precision processing unit, which includes floating-point arithmetic logic units and integer arithmetic logic units. In one embodiment, the floating-point arithmetic logic unit implements the IEEE 754-2008 standard for floating-point arithmetic. In one embodiment, core 1250 includes 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0179] Tensor cores are configured to perform matrix operations, and in one embodiment, one or more tensor cores are included in core 1250. Specifically, tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations used for neural network training and inference. In one embodiment, each tensor core operates on a 4×4 matrix and performs matrix multiplication and accumulation operations D = A×B + C, where A, B, C, and D are 4×4 matrices.

[0180] In one embodiment, matrix multiplication inputs A and B are 16-bit floating-point matrices, while accumulation matrices C and D can be either 16-bit or 32-bit floating-point matrices. The Tensor Core performs 32-bit floating-point accumulation on the 16-bit floating-point input data. The 16-bit floating-point multiplication requires 64 operations and yields a full-precision product, which is then accumulated with other intermediate multiplications using 32-bit floating-point addition to perform a 4×4×4 matrix multiplication. In practice, the Tensor Core is used to perform matrix operations on larger two-dimensional or higher-dimensional matrices composed of these smaller components. APIs (such as the CUDA C++ API) expose specialized matrix loading, matrix multiplication and accumulation, and matrix storage operations to efficiently utilize the Tensor Core from CUDA-C++ programs. At the CUDA level, the thread bundle level interface assumes a 16×16 matrix spanning all 32 thread bundle threads.

[0181] In some embodiments, transpose hardware is included in processing core 1250 or another functional unit (e.g., SFU 1252 or LSU 1254) and is configured to generate matrix data stored diagonally and / or generate the original matrix and / or transpose matrix from the diagonal. The transpose hardware may be provided within shared memory 1270 to the register file 1220 load path of SM 1140.

[0182] In one example, matrix data stored diagonally can be retrieved from DRAM and stored in shared memory 1270. When an instruction to perform processing using the diagonally stored matrix data is processed, transpose hardware located in the path of shared memory 1270 and register file 1220 can provide the original matrix, transpose matrix, compressed original matrix, and / or compressed transpose matrix. Up to the last storage before the instruction, a single matrix data stored diagonally can be retained, and the matrix type specified by the instruction is generated in register file 1220 as needed.

[0183] Each SM 1140 also includes multiple SFUs 1252 that perform special functions (e.g., attribute evaluation, inverse square root, etc.). In one embodiment, an SFU 1252 may include a tree traversal unit (e.g., TTU 1143) configured to traverse a hierarchical tree data structure. In one embodiment, an SFU 1252 may include a texture unit (e.g., texture unit 1142) configured to perform texture map filtering operations. In one embodiment, a texture unit is configured to load a texture map (e.g., a 2D array of texture pixels) from memory 1004 and sample the texture map to produce sampled texture values ​​for use by a shader program executed by the SM 1140. In one embodiment, the texture map is stored in shared memory / L1 cache 1170. Texture units use mip maps (e.g., texture maps with different levels of detail) to implement texture operations (such as filtering operations). In one embodiment, each SM 1140 includes two texture units.

[0184] Each SM 1140 also includes multiple LSUs 1254 that implement load and store operations between the shared memory / L1 cache 1270 and the register file 1220. Each SM 1140 includes an interconnect network 1280 that connects each functional unit to the register file 1220, and the LSUs 1254 are connected to the register file 1220 and the shared memory / L1 cache 1270. In one embodiment, the interconnect network 1280 is a crossbar switch that can be configured to connect any functional unit to any register in the register file 1220 and to connect the LSUs 1254 to memory locations in the register file 1220 and the shared memory / L1 cache 1270. In an example embodiment, the LSU 1254 includes a TMAU 112. However, in some embodiments, the TMAU 112 may be decoupled from the LSU. Each TMAU 112 may be tightly coupled to a single SM or one or more SMs. In an embodiment where the TMAU 112 is tightly coupled with multiple SMs, the arbitrator can receive requests from the SMs and forward them continuously to the TMAU 112.

[0185] Shared memory / L1 cache 1270 is an array of on-chip memory that allows data storage and communication between SM 1140 and primitive engine 1135, as well as between threads within SM 1140. In one embodiment, shared memory / L1 cache 1270 includes a storage capacity of 128KB and is located in the path from SM 1140 to partition unit 1180. Shared memory / L1 cache 1270 can be used for cache reads and writes. One or more of shared memory / L1 cache 1270, L2 cache 1160, and memory 1004 are backup storage.

[0186] Combining data caching and shared memory functionality into a single memory block provides optimal global performance for both types of memory access. The capacity is used by programs that do not use shared memory or as a cache. For example, if shared memory is configured to use half its capacity, texture and load / store operations can use the remaining capacity. Integration within the shared memory / L1 cache 1270 enables it to function as a high-throughput pipeline for streaming data, while providing high-bandwidth and low-latency access to frequently reused data.

[0187] In the context of this disclosure, SM or “streaming multiprocessor” refers to a processor with an architecture as described in USP 7,447,873 granted to Nordquist, including improvements and advancements thereto, and implemented, for example, in multiple generations of NVIDIA GPUs. For example, an SM may include multiple processing engines or cores configured to execute multiple threads concurrently, arranged in multiple Single Instruction Multiple Data (SIMD) groups (e.g., thread bundles), wherein each thread executes the same data processing procedure comprising a series of instructions on different input objects within the same SIMD group, and different threads within the same SIMD group use different processing engines or cores to execute. An SM may also typically provide a local register file with multiple channels, wherein each processing engine or core is configured to access a different subset of the channels; and instruction dispatch logic configured to select one of the SIMD groups and dispatch one of the instructions of the same data processing procedure in parallel to each of the multiple processing engines, wherein each processing engine executes the same instructions in parallel with each of the other processing engines, each of the other processing engines using a subset of the local register file channels accessible to it. SMs typically also include core interface logic configured to initiate the execution of one or more SIMD groups. As shown in the figure, such SMs have been built to provide fast local shared memory, enabling data sharing / reuse and synchronization among all threads of a CTA executing on the SM.

[0188] When configured for general-purpose parallel computing, a simpler configuration can be used compared to graphics processing. Specifically, a simpler programming model is created by bypassing the fixed-function graphics processing unit. In at least one embodiment, in the general-purpose parallel computing configuration, the work allocation unit directly allocates and distributes blocks of threads to the DPC 1120. Threads within a block execute the same program, using unique thread IDs in the computation to ensure each thread produces unique results. The SM 1140 executes the program and performs the computation, the shared memory / L1 cache 1270 facilitates communication between threads, and the LSU 1254 reads and writes global memory via the shared memory / L1 cache 1270 and the memory partitioning unit 1080. When configured for general-purpose parallel computing, the SM 1140 writes commands to the scheduler unit 1020 that can be used to start new jobs on the DPC 1120.

[0189] The PPU 1000 is included in or coupled to desktop computers, laptop computers, tablet computers, servers, supercomputers, smartphones (e.g., wireless, handheld devices), personal digital assistants (PDAs), digital cameras, vehicles, head-mounted displays, handheld electronic devices, etc. In one embodiment, the PPU 1000 is implemented on a single semiconductor substrate. In another embodiment, the PPU 1000 is included in a system-on-a-chip (SoC) along with one or more other devices (e.g., additional PPU 1000, memory 1004, reduced instruction set computer (RISC) CPU, one or more memory management units (MMUs), digital-to-analog converters (DACs), etc.).

[0190] In one embodiment, the PPU 1000 may be included on a graphics card that includes one or more storage devices 1004. The graphics card may be configured to connect to a PCIe slot on a desktop computer motherboard. In another embodiment, the PPU 1000 may be an integrated graphics processing unit (iGPU) or a parallel processor included in a chipset of the motherboard.

[0191] Exemplary computing system As developers expose and leverage more parallelism in applications such as artificial intelligence computing, systems with multiple GPUs and CPUs are being used across a variety of industries. High-performance GPU-accelerated systems with tens of thousands to thousands of compute nodes are being deployed in data centers, research facilities, and supercomputers to tackle larger problems. With the increasing number of processing devices in high-performance systems, communication and data transmission mechanisms need to be scaled to support the increased bandwidth.

[0192] Figure 13A This is based on the use of one embodiment. Figure 10A conceptual diagram of a processing system 1300 implemented by a PPU 1000. The exemplary system 1300 can be configured to implement the methods disclosed in this application (e.g., Figure 1 , 2 (TMAU in 6 or 11A). The processing system 1300 includes a CPU 1330, a switch 1355, and multiple PPUs 1000 and corresponding memory 1004. NVLink 1010 provides a high-speed communication link between each PPU 1000. Although a specific number of NVLink 1010 and interconnect 1002 connections are as follows... Figure 13A As shown, the number of connections to each PPU 1000 and CPU 1330 may vary. A switch 1355 connects between interconnect 1002 and CPU 1330. The PPU 1000, memory 1004, and NVLink 1010 may reside on a single semiconductor platform to form a parallel processing module 1325. In one embodiment, the switch 1355 supports interface protocols between two or more various connections and / or links.

[0193] In another embodiment (not shown), NVLink 1010 provides one or more high-speed communication links between each PPU 1000 and CPU 1330, and switch 1355 provides an interface between interconnect 1002 and each PPU 1000. PPU 1000, memory 1004, and interconnect 1002 may reside on a single semiconductor platform to form parallel processing module 1325. In yet another embodiment (not shown), interconnect 1002 provides one or more communication links between each PPU 1000 and CPU 1330, and switch 1355 uses NVLink 1010 to connect between each PPU 1000 to provide one or more high-speed communication links between PPUs 1000. In another embodiment (not shown), NVLink 1010 provides one or more high-speed communication links between PPUs 1000 and CPU 1330 via switch 1355. In yet another embodiment (not shown), interconnect 1002 directly provides one or more communication links between each PPU 1000. One or more NVLink1010 high-speed communication links can be implemented as physical NVLink interconnects or as on-chip or on-chip interconnects using the same protocol as NVLink 1010.

[0194] In the context of this specification, a single semiconductor platform can refer to a single, unified semiconductor-based integrated circuit fabricated on a die or chip. It should be noted that the term single semiconductor platform can also refer to a multi-chip module with enhanced connectivity, simulating on-chip operation and making substantial improvements over implementations using conventional buses. Of course, various circuits or devices can also be placed individually or in various combinations of semiconductor platforms, depending on the user's needs. Alternatively, the parallel processing module 1325 can be implemented as a circuit board substrate, and each PPU 1000 and / or memory 1004 can be a packaged device. In one embodiment, the CPU 1330, switch 1355, and parallel processing module 1325 reside on a single semiconductor platform.

[0195] In one embodiment, each NVLink 1010 has a signaling rate of 20 to 25 gigabits per second and each PPU1000 includes six NVLink 1010 interfaces (e.g., Figure 13A As shown, each PPU 1000 includes five NVLink 1010 interfaces. Each NVLink 1010 provides a data transfer rate of 25 GB / s in each direction, with six links providing 1000 GB / s. When the CPU 1330 also includes one or more NVLink 1010 interfaces, the NVLink 1010 can be specifically used for applications such as... Figure 13A The example shows PPU-to-PPU communication, or some combination of PPU-to-PPU and PPU-to-CPU communication.

[0196] In one embodiment, NVLink 1010 allows direct load / store / atomic access to memory 1004 of each PPU 1000 from CPU 1330. In one embodiment, NVLink 1010 supports coherent operations, allowing data read from memory 1004 to be stored in the cache hierarchy of CPU 1330, reducing cache access latency of CPU 1330. In one embodiment, NVLink 1010 includes support for Address Translation Service (ATS), allowing PPU 1000 to directly access page tables within CPU 1330. One or more NVLinks 1010 can also be configured to operate in low-power mode.

[0197] Figure 13B An exemplary system 1365 is illustrated, in which various architectures and / or functions of various prior embodiments can be implemented. The exemplary system 1365 can be configured to implement the methods disclosed in this application (e.g., Figure 1 , 2 TMAU in 6 or 11A).

[0198] As shown in the figure, a system 1365 is provided, including at least one central processing unit 1330 connected to a communication bus 1375. The communication bus 1375 can be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol. System 1365 also includes a main memory 1340. Control logic (software) and data are stored in the main memory 1340, which may take the form of random access memory (RAM).

[0199] System 1365 also includes input device 1360, parallel processing system 1325, and display device 1345, such as a conventional CRT (cathode ray tube), LCD (liquid crystal display), LED (light-emitting diode), plasma display, etc. User input, such as keyboard, mouse, touchpad, microphone, etc., can be received from input device 1360. Each of the aforementioned modules and / or devices can even be located on a single semiconductor platform to form system 1365. Alternatively, depending on the user's needs, the various modules can also be placed individually or in various combinations of semiconductor platforms.

[0200] In addition, system 1365 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a wired network, or the like) via network interface 1335 for communication purposes.

[0201] System 1365 may also include auxiliary storage (not shown). Auxiliary storage includes, for example, hard disk drives and / or removable storage drives, representing floppy disk drives, magnetic tape drives, compact disk drives, digital universal disk (DVD) drives, recording devices, universal serial bus (USB) flash memory. The removable storage drive reads from and / or writes to the removable storage unit in a well-known manner.

[0202] Computer programs or computer control logic algorithms may be stored in main memory 1340 and / or auxiliary memory. When executed, such computer programs enable system 1365 to perform various functions. Memory 1340, memory, and / or any other memory are possible examples of computer-readable media.

[0203] The architectures and / or functions of the various previously shown figures can be implemented in the context of general-purpose computer systems, circuit board systems, game console systems dedicated to entertainment purposes, special-purpose systems, and / or any other desired systems. For example, System 1365 can be implemented as a desktop computer, laptop computer, tablet computer, server, supercomputer, smartphone (e.g., wireless handheld device), personal digital assistant (PDA), digital camera, vehicle, head-mounted display, handheld electronic device, mobile phone device, television, workstation, game console, embedded system, and / or any other type of logic.

[0204] Applications can be implemented through applications executed by a main processor such as a CPU. In one embodiment, the device driver may implement an application programming interface (API) that defines various functions that can be used by the application to generate graphical data for display. A device driver is a software program that includes multiple instructions controlling the operation of the PPU 1000. The API provides an abstraction for the programmer, enabling the programmer to utilize dedicated graphics hardware such as the PPU 1000 to generate graphical data without the programmer using the specific instruction set of the PPU 1000. An application may include an API call that is routed to the device driver of the PPU 1000. The device driver interprets the API call and performs various operations in response to the API call. In some cases, the device driver may perform operations by executing instructions on the CPU. In other cases, the device driver may perform operations at least partially by initiating operations on the PPU 1000 using the input / output interface between the CPU and the PPU 1000. In one embodiment, the device driver is configured to implement a graphics processing pipeline 1400 utilizing the hardware of the PPU 1000.

[0205] Various programs can be executed within the PPU 1000 to implement different stages of application processing. For example, a device driver can launch a kernel on the PPU 1000 to perform one stage of processing on an SM 1140 (or one or more SM 1140s). The device driver (or the initial kernel executed by the PPU 1000) can also launch other kernels on the PPU 1000 to perform other stages of processing. If application processing includes a graphics processing pipeline, some stages of the graphics processing pipeline can be implemented on fixed-function hardware, such as a rasterizer or data assembler implemented within the PPU 1000. It should be understood that a kernel can be processed by one or more intervening fixed-function hardware units before being processed by a subsequent kernel on an SM 1140.

[0206] All patents and print publications cited above are incorporated herein by reference as if explicitly stated.

[0207] Although the invention has been described in conjunction with embodiments that are presently considered to be the most practical and preferred, it should be understood that the invention is not limited to the disclosed embodiments, but rather is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A parallel processor, comprising: Memory interface; Multiple processors; as well as Multiple memory access hardware circuits, each memory access hardware circuit being coupled to a processor among the multiple processors and configured to: Receive a memory access request for a multidimensional data structure from the coupled processor, the memory access request specifying the coordinates of the multidimensional data structure; as well as In response to the memory access request, at least the coordinates of the multidimensional data structure are converted into multiple memory addresses of the multidimensional data structure, and using the multiple memory addresses, at least a portion of the multidimensional data structure is asynchronously transferred between a location in external memory accessed through the memory interface and a location in the shared memory of the coupled processor. At least the coordinates define the location of the data block within the multidimensional data structure stored in the external memory; The memory access hardware circuitry coupled to the processor is further configured to determine whether the data block is part of a tensor stored in the external memory or whether the data block includes non-tensor data in the external memory.

2. The parallel processor of claim 1, wherein the memory access hardware circuitry coupled to the processor is configured to read from and write to the processor’s shared memory and the external memory coupled to the memory access hardware circuitry.

3. The parallel processor of claim 1, wherein the memory access hardware circuitry coupled to the processor is further configured to perform the conversion by obtaining the base address of the multidimensional data structure from a descriptor previously stored for the multidimensional data structure.

4. The parallel processor of claim 3, wherein the memory access hardware circuitry coupled to the processor includes a setup circuitry configured to prefetch the previously stored descriptors from the external memory into a descriptor cache in the memory access hardware circuitry before at least converting the coordinates of the multidimensional data structure into a plurality of memory addresses of the multidimensional data structure.

5. The parallel processor of claim 3, wherein the memory access hardware circuitry coupled to the processor is further configured to: detect out-of-bounds memory locations of the multidimensional data structure in the external memory during reading of the multidimensional data structure.

6. The parallel processor of claim 3, wherein the memory access hardware circuitry coupled to the processor is further configured to: during a write to the location in the shared memory of the coupled processor, encode the out-of-bounds memory location of the multidimensional data structure in the external memory with a predetermined value.

7. The parallel processor of claim 1, wherein the memory access hardware circuitry coupled to the processor is further configured to perform the asynchronous transfer by writing the at least portion directly from the external memory to the processor's shared memory, or directly from the processor's shared memory to the external memory.

8. The parallel processor of claim 1, wherein the memory access hardware circuitry coupled to the processor is further configured to perform the asynchronous transfer in response to a memory access request received in a single message, regardless of the size of the requested data block.

9. The parallel processor of claim 1, wherein the memory access hardware circuitry coupled to the processor is further configured to: read at least a portion of the data from the external memory and write the data block in a promiscuous manner to the shared memory.

10. The parallel processor of claim 9, wherein the memory access hardware circuitry coupled to the processor is further configured to write at least a portion of the data in the promiscuous form to the shared memory according to a predetermined promiscuous pattern.

11. The parallel processor of claim 10, wherein the memory access hardware circuitry coupled to the processor is further configured to determine the predetermined promiscuous mode based on information included in the response received for the request, in order to perform the reading of at least a portion of the data from the external memory.

12. The parallel processor of claim 1, wherein the memory access hardware circuitry coupled to the processor is further configured to access the external memory in block mode.

13. The parallel processor of claim 1, wherein the memory access hardware circuitry coupled to the processor is further configured to access the external memory in a picture-to-column mode.

14. The parallel processor of claim 1, wherein the memory access hardware circuitry coupled to the processor includes a request queue, a descriptor cache, a setup circuit, a request generation circuit, and a request completion tracking circuit.

15. The parallel processor of claim 1, wherein the memory access hardware circuitry coupled to the processor is further configured to: update a counter in the shared memory of the coupled respective processor for each sub-block of data in the at least portion of the data, wherein the respective processor includes synchronization circuitry configured to monitor a predetermined value of the counter.

16. The parallel processor of claim 1, wherein the memory access hardware circuitry coupled to the processor is further configured to: read the at least portion of the external memory and write the at least portion to a location in the shared memory of each of the plurality of processors in the group of processors.

17. The parallel processor of claim 1, wherein the memory access request is a load operation from the external memory to the shared memory or a store operation from the shared memory to the external memory.

18. The parallel processor of claim 1, wherein the asynchronous transfer includes a reduction operation.

19. The parallel processor of claim 1, wherein the asynchronous transfer includes a multicast operation.

20. The parallel processor of claim 1, wherein each of the processors comprises a plurality of parallel processing cores having different computing power and / or precision, the plurality of parallel processing cores accessing a common instruction cache memory.

21. The parallel processor of claim 1, further comprising arbitrator hardware circuitry, wherein the memory access hardware circuitry is coupled to the processor and at least one other processor via the arbitrator hardware circuitry.

22. A method executed in a parallel processor comprising multiple processors, the method comprising: The memory access hardware circuitry of the processors coupled to the plurality of processors receives, from the coupled plurality of processors, a memory access request for coordinates of a specified multidimensional data structure, wherein the memory access hardware circuitry is one of a plurality of memory access circuits, each of which is coupled to one of the respective processors. as well as In response to the memory access request, at least the coordinates of the multidimensional data structure are converted into multiple memory addresses of the multidimensional data structure, and using the multiple memory addresses, at least a portion of the multidimensional data structure is asynchronously transferred between a location in external memory accessed through the memory interface and a location in the shared memory of the coupled processor. At least the coordinates define the location of the data block within the multidimensional data structure stored in the external memory; The method further includes determining whether the data block is part of a tensor stored in the external memory or whether the data block includes non-tensor data in the external memory.

23. A memory access hardware circuit, comprising: Interface to external memory; Memory input / output interface, used to receive memory access requests from the processor; At least one memory interface coupled to one or more other processors and their respective shared memory; as well as The processing pipeline is configured as follows: The processor receives a memory access request for coordinates of a specified multidimensional data structure, wherein the memory access hardware circuit is one of a plurality of memory access circuits, each of which is coupled to one of the corresponding processors. as well as In response to the memory access request, at least the coordinates of the multidimensional data structure are converted into multiple memory addresses of the multidimensional data structure, and using the multiple memory addresses, at least a portion of the multidimensional data structure is asynchronously transferred between a location in external memory accessed through the memory interface and a location in the shared memory of the coupled processor. At least the coordinates define the location of the data block within the multidimensional data structure stored in the external memory; The processing pipeline is further configured to determine whether the data block is part of a tensor stored in the external memory or whether the data block includes non-tensor data in the external memory.

Images