System and method for multicast of data

Abstract

The invention relates to a system and method for multicast of data. A first instruction to load first data to a first register file associated with the first sub-partition unit is received. A second instruction to load the first data to a second register file associated with the second sub-partition unit is received. Based on the instruction identifier, the first instruction and the second instruction are merged into a first entry of the requesting merger. The first entry is associated with first data. In response to determining that the first data is available in the cache, the first data is multicast from the cache to the first register file and the second register file.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to parallel processing systems. Specifically, embodiments of the present disclosure relate to parallel processing systems and methods for multicasting data. Background Art

[0002] Parallel processing in high-performance computing (HPC) systems involves executing multiple computing tasks or operations simultaneously. This is accomplished by breaking down larger computations into smaller, independent subtasks that are processed concurrently by multiple processors. In some cases, parallel processing involves distributed computing, where tasks are spread across multiple computing clusters. Each cluster can operate independently and can facilitate communication to share results. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 A parallel processing unit is shown in accordance with at least one embodiment of the present disclosure.

[0004] Figure 2 A conceptual diagram is shown when instructions requesting constant data are coalesced and the constant data is multicast to an associated unified register file, in accordance with at least one embodiment of the present disclosure.

[0005] Figure 3A An example request merger content addressable memory (CAM) table is shown in accordance with at least one embodiment of the present disclosure.

[0006] Figure 3B Illustrated is a method for merging requests into a Figure 3A Example pseudo code for the example request merger CAM table.

[0007] Figure 3C Illustrated is a method for merging requests into a Figure 3A Continuation of the example pseudo-code in the Example Request Combiner CAM table.

[0008] Figure 4 A flowchart of a method for multicasting data according to at least one embodiment of the present disclosure is shown.

[0009] Figure 5 A parallel processing unit according to an embodiment is shown.

[0010] Figure 6A It shows the embodiment Figure 5 A general-purpose processing cluster within a parallel processing unit.

[0011] Figure 6B It shows the embodiment Figure 5 A memory partitioning unit of a parallel processing unit.

[0012] Figure 7A It shows the embodiment Figure 6A Streaming multiprocessor.

[0013] Figure 7B is used according to the embodiment Figure 5 Conceptual diagram of the processing system implemented by the PPU.

[0014] Figure 7C An exemplary system is shown that can implement the various architectures and / or functionality of the various previous embodiments.

[0015] Figure 8 According to the embodiment Figure 5 Conceptual diagram of the graphics processing pipeline implemented by the PPU. DETAILED DESCRIPTION

[0016] HPC systems can use specialized hardware architectures (e.g., parallel processing units (PPUs)) to enhance parallel processing capabilities. PPUs are designed to achieve high performance using a large number of small, parallel execution threads on a dedicated programmable multiprocessor. In a PPU, a group of threads (e.g., warps) can concurrently execute the same instruction (e.g., the same instruction executed by different threads) on a multiprocessor with different input data. This execution model is called single instruction, multiple thread (SIMT) and is commonly used for parallel computing. PPUs are designed to execute programs (e.g., kernels, shader programs, etc.) in parallel by many warps on the PPU, where each thread in the warp typically operates on a different portion of data. In some cases, more than one multiprocessor may request the same data at different points in the execution of the program. To address these situations, some PPUs may include one or more dedicated memory spaces, such as a constant cache and associated unified registers, which are optimized for storing and accessing constant data. A constant cache can be a read-only memory space that is configured to store constant data that remains the same across multiple warps during program execution. Constant data can include mathematical constants, program parameters, transformation matrices, coefficients, and any other data that remain unchanged during program execution. Constant data can be read from a constant cache and written to a unified register (referred to herein as "write back") associated with one or more threads of a warp that crosses the request constant data. However, providing services for multiple threads requesting the same constant data can produce bandwidth issues. For example, a constant cache can have a write-back bandwidth of 8 bytes per clock cycle. The first thread, second thread, third thread, and fourth thread can request the same 32 bytes of constant data. Each request can be serviced individually, with a total write-back bandwidth of 16 clock cycles.

[0017] To mitigate the bandwidth associated with writing back constant data, some conventional techniques can match the addresses of incoming requests for constant data. The matched addresses can be coalesced, and the associated constant data can be multicast to corresponding unified registers. Coalescing is the process of combining multiple memory access requests (e.g., requests to read data from a constant cache) into a single memory access request. In the context of this disclosure, multicasting refers to the process of sending constant data from a constant cache to multiple requestors simultaneously for write-back to a unified register. A unified register can generally refer to an on-chip memory structure used to store variables or values ​​that remain constant across a group of parallel threads. To match addresses between incoming requests, some conventional systems can implement wide address matching logic with an associated area cost. For example, a system can implement address matching logic for matching 64-bit memory addresses. The area allocated for address matching logic can be significant for constant caches that are replicated extensively by multiple processor counts on a PPU. Area is an important consideration in PPU design because it influences factors such as cost efficiency, power efficiency, and heat dissipation.

[0018] In another conventional solution, a single thread can programmatically coordinate the multicasting of data. The PPU can implement software to coordinate between threads before issuing a memory read instruction. Designing such software can be challenging and incurs significant overhead in terms of design complexity.

[0019] Another conventional solution is to implement forced aging requests to write constant data from the constant cache back to the unified register, thereby expanding the window in which writeback requests can be merged. However, using forced aging to merge writeback requests increases the static latency of thread execution. Furthermore, after merging, the data must still be broadcast to the unified registers of the associated multiprocessor cores to achieve the desired writeback bandwidth reduction. Conventional techniques can blindly broadcast the data for writeback to the requesters simultaneously to achieve the desired reduced writeback bandwidth. This solution requires all requesters to have idle cycles for unified register writebacks at the same time, and coordinating the writeback time with each unified register associated with each multiprocessor increases complexity.

[0020] Aspects and embodiments of the present disclosure address the above-mentioned and other deficiencies of conventional PPUs by providing a technique that enables the PPU to merge writeback requests using a reduced number of bits. Constant data can be multicast to corresponding unified register files of disjoint cores of a multiprocessor. The PPU can integrate several independent multiprocessors, such as a streaming multiprocessor (SM), on a single die. Each multiprocessor can include a request merger and a constant cache that is accessible to all subpartition units and associated processing cores provided on the multiprocessor. Each subpartition unit can execute thread warps, thereby allowing thread warps to be executed in parallel. A subpartition unit can issue a writeback request to the request merger to read data from the constant cache and write the data to one or more unified registers associated with the subpartition. In some embodiments, the writeback request is associated with an instruction that performs an operation using the requested data. Therefore, for convenience, the terms "request" and "instruction" may be used interchangeably herein, but it is noted that aspects and implementations of the present disclosure can generally be applied to merging requests to access constant data. In some embodiments, the writeback request can be merged by the request merger based on an instruction identifier. When the data is available in the cache, the request merger can cause the data to be multicast to a unified register associated with requests having the same instruction identifier. In some embodiments, the instruction identifier can include a cache line identifier and a byte offset. Thus, the instruction identifier used to match the writeback request can be reduced from a memory address (e.g., 64 bits) to a cache line identifier (e.g., 9 bits) and a byte offset (e.g., 16 bits).

[0021] In at least one embodiment, to multicast constant data, a request merger may simultaneously write the data to one or more write-back buffers. The write-back buffers may store the data until the corresponding unified register file is available for write-back. For example, a first request and a second request may request the same constant data. The request merger may merge the first request and the second request into the same entry of the request merger. In response to the constant data being available in the constant cache, the request merger may write the constant data to a first write-back buffer associated with the first request and a second write-back buffer associated with the second request. When a first unified register corresponding to the first request is available for write-back, the first write-back buffer may load the constant data into the first unified register. When a second unified register corresponding to the second request is available, the second write-back buffer may load the constant data into the second unified register. Thus, temporarily storing constant data in a universal write-back buffer may allow the constant data to be multicast to separate cores of a multiprocessor without synchronizing unified registers for write-back.

[0022] Advantages of the techniques disclosed herein include, but are not limited to, reduced bandwidth associated with writing constant data back to a unified register file. This can be achieved by merging multiple writeback requests using a minimum number of bits for request / instruction identification and matching. Furthermore, utilizing a common writeback buffer for multicasting data can eliminate the complexity and hardware associated with synchronous unified register writebacks. Thus, aspects and embodiments of the present disclosure can provide flexibility with relatively little complexity when merging writeback requests and multicasting data to a unified register to reduce writeback bandwidth.

[0023] It should be noted that various aspects of the above-described methods and systems are described in detail below by way of example and not limitation. The embodiments and examples provided below may reference write-back requests / instructions associated with a particular sub-partition unit of a multiprocessor. However, the write-back request may generally be associated with a thread executing on any processing core associated with a PPU. Furthermore, the embodiments and examples provided below may reference unified registers associated with a particular sub-partition unit. However, the unified register may be associated with any processing unit of a multiprocessor, a sub-partition unit, an individual thread, or a PPU. Furthermore, it will be understood that thread bundles are used herein as an example and not as a limitation, and it will be noted that aspects and implementations of the present disclosure may generally be applied to a group of threads that are scheduled together. Furthermore, it will be understood that unified registers are used herein as an example and not as a limitation. It should be noted that aspects and implementations of the present disclosure may utilize other registers to multicast data. For example, data may be multicast to a general register file according to the techniques described herein.

[0024] Figure 1 FIG2 shows a parallel processing unit (PPU) 100 according to at least one embodiment of the present disclosure. For example, the PPU 100 may include a graphics processing unit (GPU). Figure 1 As shown, the PPU 100 includes one or more multiprocessors 102. For example, each multiprocessor 102 may be a streaming multiprocessor (SM), a compute unit (CU), an integrated many-core (MIC), etc. Each multiprocessor 102 includes, but is not limited to, a plurality of sub-partition units 104A to 104N (collectively referred to herein as "sub-partition units 104"), a request merger 120, and a constant cache 110.

[0025] The warps assigned to the multiprocessor 102 are distributed among the sub-partition units 104. Each sub-partition unit 104 can be assigned any number of warps, however, a given warp can only be assigned to one sub-partition unit 104. As shown, each sub-partition unit 104 may include, but is not limited to, multiple functional units 106A through 106N (collectively referred to herein as "functional units 106"), unified register files (URFs) 108A through 108N (collectively referred to herein as "URFs 108"), and write-back buffers 109A through 109N (collectively referred to herein as "WB buffers 109"). Each functional unit 106 may include multiple processing cores (e.g., Compute Unified Device Architecture (CUDA) cores) that execute separate threads, thereby enabling parallel processing. In at least one embodiment, the functional units 106 may operate according to a single instruction, multiple thread (SIMT) architecture. For example, all threads within a warp execute the same instruction simultaneously, but may operate on different data.

[0026] In at least one embodiment, the functional unit 106 can perform various calculations, such as floating-point 32-bit precision arithmetic, floating-point 16-bit precision arithmetic, integer arithmetic of different precisions, etc. In at least one embodiment, the functional unit 106 can include one or more tensor cores (not shown) for performing matrix multiplication operations using mixed-precision arithmetic. For example, one or more tensor cores can operate on a low-precision format (e.g., half-precision floating point (FP16)) to accelerate calculations.

[0027] As described above, each sub-partition unit 104 may include a unified register file 108. A unified register file 108 is a collection of registers used to store temporary data, intermediate results, operands during computation, and / or constant data. A unified register file (URF) 108 is a shared memory resource that includes, but is not limited to, any number of unified registers and any number of read and / or write ports. Each unified register in a URF 108 is independently accessible by all threads included in a warp executing on the corresponding sub-partition unit 104. A URF 108 may be implemented in any technically feasible manner. In alternative embodiments, the unified registers included in a URF 108 may be arranged and assigned to threads and / or warps in any technically feasible manner. In at least one embodiment, each thread running on a given functional unit 104 may use a dedicated set of registers separate from the URF 108 associated with that thread. In at least one embodiment, these registers may be thread-private, providing a dedicated space for storing data associated with that thread.

[0028] It is understood that the number of multiprocessors 102 in the PPU 100, the number of sub-partition units 104 in a single multiprocessor 102, and the number of functional units 106 in a sub-partition unit 104 are not limited. Each sub-partition unit 104 in the multiprocessor 102 may include a URF 108 and may access a constant cache 110 specific to the multiprocessor 102. In at least one embodiment, Figure 1 In the illustrated embodiment, the constant cache 110 may be part of a shared / local memory associated with the multiprocessor 102. In some other embodiments, the constant cache 110 may be separate from the shared memory associated with the multiprocessor 102.

[0029] In some embodiments, the constant cache 110 may be used to store constants or values ​​that do not change during the execution of a kernel or shader program. These constants may include, for example, parameters, coefficients, or any other data that remains constant across multiple thread warps. Thus, the constant cache 110 may be a read-only cache located closer to the processing unit / core than global memory, thereby providing faster access times for threads accessing the same constant data. In some embodiments, the values ​​stored in the constant cache 110 may be updated. For example, a kernel / shader program may update the value of a global memory address associated with the constant cache and issue an invalidate to the constant cache 110 to allow the constant cache 110 to observe the updated value. In some embodiments, the constant cache 110 may be located on a primary structural unit of the PPU 100 (e.g., a graphics processing cluster (GPC)) and associated with multiple multiprocessors 102 (e.g., SMs). In some embodiments, the constant cache 110 may be associated with a single multiprocessor 102, such as with respect to Figure 1 For example, each constant cache 110 may be associated with a single multiprocessor 102. Constant cache 110 may generally refer to a caching mechanism associated with constant memory throughout the memory hierarchy of PPU 100.

[0030] In at least one embodiment, multiprocessor 102 can access global memory (not shown) external to PPU 100 through a global memory interface. Global memory can include hierarchical cache memory (e.g., L2 cache and / or L3 cache) and dynamic random access memory (DRAM). In at least one embodiment, global memory can include reference Figure 6A 、 Figure 6B 、 Figure 7A and Figure 7B The described memory management unit (MMU), X-Bar or hierarchical crossbar interconnect network, memory partitioning unit, and / or memory.

[0031] The plurality of functional units 104 in each multiprocessor 102 are configured to process multiple threads in parallel. A thread (e.g., an execution thread) is an instance of an instruction set or kernel that is configured to be executed by a functional unit 104 on a particular data set. Threads in a thread block can be executed concurrently, and multiple thread blocks can be executed concurrently. In some embodiments, single instruction multiple data (SIMD) instruction issuance technology is used to support the parallel execution of a large number of threads without providing multiple independent instruction units. In other embodiments, single instruction multiple thread (SIMT) technology is used to support the parallel execution of a large number of generally synchronized threads using a general-purpose instruction unit that is configured to issue instructions to a group of cores.

[0032] Each functional unit 104 may be connected to a URF 108 and a constant cache 110 via an interconnect network (e.g., a hierarchical crossbar having one or more read and / or write crossbars). The constant cache 110 may provide low-latency on-chip memory close to the sub-partition units of the multiprocessor 102. The unified register file 108 may include data registers that may be assigned to different functional units in the plurality of functional units 104 and / or different warps executed by the multiprocessor 102. The register file 108 provides temporary storage for the functional units 104 on the multiprocessor 102.

[0033] The PPU 100 can support multiple address spaces, including local, shared, and global address spaces, to support data visibility for threads. Additional read-only address spaces, including textures, can be supported. Each thread has its own per-thread local or private memory, which can be controlled through allocation registers.

[0034] The sub-partition unit 104 may request access to constant data stored in the constant cache 110 and / or stored in the constant memory 114 through the request merger 120. Each sub-partition unit 104 in the multiprocessor 102 is coupled to a request merger 120 configured to access the constant memory 114. Each request merger 120 reads access to the constant cache of the corresponding multiprocessor 102 by issuing a request to the memory subsystem (e.g., by issuing a request to a general cache controller (GCC)).

[0035] When one or more sub-partition units 104 need to access constant data stored in the constant cache 110, the software will initiate a thread with a "load" or "read" instruction. The load instruction can cause the constant data to be read from the constant cache 110 into one or more unified registers of the URF 108, making it available to all threads in the corresponding sub-partition unit 104. In at least one embodiment, in the event of a cache miss, the constant data can be written from the constant memory 114 to the constant cache 110.

[0036] The request merger 120 may merge multiple requests / instructions issued by one or more sub-partition units 104 of the multiprocessor 102 to multicast constant data into one or more corresponding URFs 108. In at least one embodiment, the request merger 120 may include a data structure (e.g., a content addressable memory (CAM) table), wherein each entry of the request merger includes one or more requests / instructions for loading the same constant data into the corresponding URF. The request merger 120 may merge multiple requests / instructions into the same entry based on instruction identifiers associated with the instructions. For example, the request merger may merge a first instruction and a second instruction having the same instruction identifier into the same entry of the request merger 120. Matching instruction identifiers may indicate that the first instruction and the second instruction are the same instruction (e.g., the same instruction, but associated with different warps and executed on different sub-partition units 104), and therefore require the same constant data for execution. Request merger 120 may multicast the same constant data from constant cache 110 to URF 108A associated with the first request and URF 108B associated with the second instruction.

[0037] The request merger 120 may initiate the process of multicasting constant data from the constant cache 110 to the corresponding URF 108 by selecting an eligible entry from a plurality of eligible entries. An eligible entry may be defined as an entry in the request merger 120 that has a pending instruction to load constant data into a currently available URF 108 in the constant cache 110. When an eligible entry is selected, the associated constant data may be pushed to the WB buffer 109 of each subpartition unit 104 associated with the selected entry. The WB buffer 109 may temporarily store constant data scheduled to be written back to the associated URF 108. In at least one embodiment, the WB buffer 109 is a first-in, first-out (FIFO) queue. In some cases, the constant data stored in the WB buffer 109 may be written to the associated URF 108 of each subpartition 104 in the order requested.

[0038] Figure 2A conceptual diagram 200 is shown in accordance with at least one embodiment of the present disclosure, wherein instructions requesting constant data are merged and the constant data are multicast to an associated unified register file (URF). Figure 1 Individual threads or warps executing on a sub-partition unit 104 of a multiprocessor may request access to constant data in a constant cache 110 associated with the multiprocessor and load the constant data into one or more registers in a unified register file (e.g., URF 108) associated with the multiprocessor. In at least one embodiment, multiple requests may be received in the same clock cycle. For example, request 202A, request 202B, request 202C, and request 202N (collectively referred to herein as request 202) may be received in the same clock cycle. In some embodiments, each request 202 may be associated with a request using a clock cycle associated with the multiprocessor. Figure 1 The SIMT architecture described herein is associated with warps that execute the same instruction simultaneously. For example, request 202A may be associated with a warp that executes an instruction on sub-partition unit 104A, while request 202B may be associated with a different warp that executes an instruction on sub-partition unit 104B. One or more threads within a warp may request constant data from a constant cache to continue execution. In some cases, threads across multiple warps may request the same constant data. For example, requests 202A and 202B may be associated with corresponding warps that request the same constant data executed on corresponding sub-partition units. In at least one embodiment, requests 202A and 202B may be associated with the same instruction (e.g., the same instruction parameters).

[0039] In at least one embodiment, request 202 may be an instruction associated with a thread to be executed, and may also be referred to herein as "instruction 202." In at least one embodiment, instruction 202 may include parameters that determine the operation to be performed, such as an opcode (opcode), source operands, destination operands, and / or an execution context. For example, an opcode may specify the operation or calculation that the thread is instructed to perform, including arithmetic operations (addition, multiplication, etc.), logical operations (AND, OR, XOR, etc.), memory operations (load, store, etc.), control flow operations (branch, loop, etc.), and other instructions. A source operand is data or the address of data on which the operation is to be performed, including registers, constants, data in global memory, etc. The destination operand may include a location where the result of the operation is to be stored, such as a register, a location in device memory, etc. The execution context may include information related to the current state of the thread, including a program counter (PC), an instruction pointer, register values, etc.

[0040] The N-to-1 arbiter 204 may arbitrate access to the merge pipeline 210 for merging requests 202A to 202N and multicasting associated constant data. The N-to-1 arbiter 204 may be implemented using any suitable arbitration scheme, including but not limited to round-robin arbitration, static priority arbitration, rotating priority arbitration, fair share arbitration, dynamic priority arbitration, token passing arbitration, reservation-based arbitration, contention-based arbitration, etc. In at least one embodiment, a memory input / output controller (MIOC) 201 may transmit instructions / requests to the merge pipeline 210. In at least one embodiment, the arbitration may be performed by the MIOC 201.

[0041] During the tag lookup 212 stage of the merge pipeline 210, a tag lookup may be performed to determine whether the requested constant exists in the constant cache 110. The address included in the request 202 may be divided into a tag, an index, and an offset to identify the location of the requested constant in the constant cache 110. In at least one embodiment, the general cache controller (GCC) 206 may perform the tag lookup by comparing the tag of the address with the tags stored in the constant cache 110, where each cache line in the constant cache 110 includes an associated tag. If the result of the comparison is a match, it is a cache hit. If the result of the comparison is not a match, it is a cache miss. In at least one embodiment, the constant cache 110 is connected to the general cache controller 206 (GCC) via an interface. In the event of a miss on the constant cache 110 , a request may be issued to the GCC 206 to load data from the constant memory 114 into the constant cache 110 , and the GCC 206 may manage the transfer of the constant data from the constant memory to the constant cache 110 .

[0042] Regardless of whether the result of the tag lookup 212 stage is a cache hit or a cache miss, the request 202 may be stored in the request merger 120, awaiting the writeback of the constant data to the URF 108. Depending on the implementation described herein, the writeback of constant data may refer to the process of loading data from the constant cache 110 into one or more URFs 108. The request merger 120 is a mechanism designed to merge or combine multiple requests 202 corresponding to the same constant data / same instruction so that warps issuing the same instruction are not serviced separately. Specifically, the request merger 120 may be a dedicated hardware structure having multiple entries, each corresponding to constant data and an associated instruction / request. Each entry in the request merger 120 may include an instruction to write / load the constant data into the associated URF 108. In at least one embodiment, each entry in the request merger 120 is associated with an instruction identifier, and multiple instructions with the same instruction identifier may be merged into the same entry in the request merger 120, awaiting writeback.

[0043] In an illustrative example, the request merger 120 may be a content addressable memory (CAM). Each entry in the CAM may include a lookup portion and a payload portion. The lookup portion may include an instruction identifier to which the incoming request 202 is to be matched. The instruction identifier may include any combination of instruction parameters associated with the incoming request 202. For example, the instruction parameters may include a cache line identifier (e.g., a set identifier, a way identifier, etc.), a destination unified register, a payload size for the read operation of the request 202, a scoreboard identifier, a byte offset associated with the cache line, an opcode, a cache tag identifier, etc. In general, each entry in the CAM may include sufficient information to ensure that merging two or more requests for constant data is functionally correct and / or acceptable. The payload may include an indication that the request 202 requests access to the same constant data and therefore the warp issues the same instruction. In embodiments where each request 202 is associated with a particular warp, the payload may include a bit mask that indicates that the warp is waiting for the same constant data to be written back to the associated unified register, as described below with respect to Figure 3A shown.

[0044] In response to request 202 being received into request consolidator 120, request consolidator 120 may compare received request 202 with existing entries of request consolidator 120. As described above, each entry of request consolidator 120 may be associated with an instruction identifier (e.g., one or more instruction parameters). If the instruction identifier of request 202 matches an existing entry of request consolidator 120, request 202 may be merged into the existing entry. If the instruction identifier does not match, a new entry may be allocated and associated with the instruction identifier of request 202, and request 202 may be merged into the newly allocated request consolidator 120 entry.

[0045] In response to constant data being available in the constant cache 110, the request merger 120 may initiate a write-back process. To initiate the write-back process, the request merger 120 selects an entry from the entries that are eligible for write-back. An eligible entry is an entry within the request merger that has a pending request 202 associated with constant data available for reading from the constant cache 110. In at least one embodiment, the request merger 120 may select from the plurality of eligible entries using an oldest-age-first approach by comparing an age field associated with each eligible entry. The GCC 206 may facilitate loading the constant data associated with the selected entry into the associated write-back buffer 109.

[0046] The write-back buffer 109 may temporarily hold constant data selected by the request merger 120 so that it can be written back to the associated URF 108. In at least one embodiment, a single write-back buffer 109 may be associated with a single sub-partition unit and associated URF 108, as described above with respect to Figure 1 As shown. In the illustrative example, request 202A and request 202B may be the same instruction associated with sub-partition 104A and sub-partition 104B, respectively, requesting the same constant data. Request merger 120 may merge (e.g., combine) requests 202A and 202B into a given, identical entry of request merger 120. In response to request merger 120 selecting a given entry for writeback, the constant data may be pushed into writeback buffer 109A associated with sub-partition 104A and writeback buffer 109B associated with sub-partition 104B. The constant data may be loaded from writeback buffers 109A and 109B into URF 108A and URF 108B. Thus, the constant data may be multicast from writeback buffer 109 to URF 108 in a single clock cycle. In at least one embodiment, when write-back buffer 109 becomes full, write-back buffer 109 may apply back pressure to request merger 120 to prevent entries from being offloaded into write-back buffer 109 .

[0047] In some embodiments, merged requests associated with the same subpartition 104 can be serialized when they are written from the writeback buffer 109 to the associated URF 108. For example, subpartition 104A may include four warps with constant cache load instructions, which are merged into a single entry of the request merger 120 entry according to the merge pipeline 210 described above. When an entry of the request merger 120 is selected for writeback, the associated constant data can be written to the writeback buffer 109A along with the warp identifier associated with the warp of the request merger entry. Writing data from the writeback buffer 109A back to the associated URF 108A can be serialized over N cycles for each merged warp, where N is the number of merged warps in the writeback buffer 109A. In at least one embodiment, writebacks of warps can be serialized based on warp identifiers. For example, the first warp with warp identifier '0' can be written back to the URF 108A, followed by the second warp with warp identifier '1'. In at least one embodiment, the writebacks of warps may be serialized in the order in which they were merged.

[0048] In at least one embodiment, a given sub-partition unit may include multiple URFs associated with one or more warps executing on the given sub-partition unit. For example, a given sub-partition unit may include a per-warp URF such that each warp executing on the given sub-partition unit uses a dedicated URF. In such an embodiment, the given sub-partition unit may write data back to such a per-warp URF in a single cycle. Typically, the N warps of a given sub-partition unit may be grouped into a set of M-URFs such that write-back serialization will require at least one cycle and at most N cycles to write back the data.

[0049] Figure 3A An example request merger content addressable memory (CAM) table 300 is shown in accordance with at least one embodiment of the present disclosure. In at least one embodiment, the request merger CAM 300 may correspond to Figure 2 1. In at least one embodiment, the request consolidator CAM 300 may be a dedicated memory for consolidating requests to write constant data from a constant cache / constant memory into an associated register file, as described above. In at least one embodiment, the control and management of the request consolidator CAM 300 may be facilitated by hardware components, firmware, and / or one or more software layers.

[0050] The request merger CAM 300 may include a plurality of entries 304A through 304N (collectively referred to herein as "entries 304"). Each entry may include a lookup and a payload. For example, entry 304A may include a lookup portion 302 and a payload portion 322. The lookup portion 320 may include an identifier of the instruction to be matched. The lookup portion 320 may include, but is not limited to, any combination of a destination register 306, a scoreboard identifier 308, a cache line ID 310, a byte offset 311, a payload size 312, an opcode 314, and / or a tag 316. The destination register 306 may be an identifier of a destination register associated with one or more URFs 108. The scoreboard identifier 308 may include an identifier of a scoreboard to be released upon writeback. A scoreboard is a mechanism for out-of-order instruction execution dependency tracking. The cache line ID 310 may include a set identifier and a way identifier associated with the constant cache 110. The byte offset 311 may be an identifier of a specific byte within the cache line where the constant data is located. The load size 312 may indicate the size of the associated constant data (e.g., 32 bits, 64 bits, 96 bits, etc.). The opcode 314 may be an instruction code that indicates a specific operation to be performed by the thread. The opcode 314 may include arithmetic operations (e.g., addition, multiplication, etc.), logical operations (e.g., AND, OR, etc.), memory operations (e.g., load, store, etc.), and control flow operations (e.g., branching, looping, etc.).

[0051] Lookup portion 320 may include various other fields that include information not associated with the instruction identifier (e.g., metadata). In at least one embodiment, lookup portion 320 may include a field to indicate whether entry 304A is valid. In at least one embodiment, portion 302 may include a field to indicate whether destination register 306 is valid. In at least one embodiment, lookup portion 320 may include an indicator of whether constant data associated with entry 304A is available for writeback. In at least one embodiment, lookup portion 320 may include a field to indicate whether the instruction / request in entry 304A has the same instruction identifier as another instruction in request merger CAM 300 associated with the same warp. In at least one embodiment, lookup portion 320 may include a field to indicate the age of entry 304A.

[0052] The payload portion 322 may include a bit mask for identifying warps waiting for the same constant data associated with entry 304A. The bit positions within the bit mask may represent identifiers associated with the warps. If a bit is set to '1', the corresponding warp is indicated in the entry as waiting for constant data; if a bit is set to '0', the corresponding warp is not waiting for constant data. In an illustrative example, the bit mask "1000010000001000" may represent the payload 322 of entry 304A, where the 0th, 5th, and 12th warps are waiting for constant data associated with entry 304A. It should be understood that the request merger CAM 300 is used herein as an example and not as a limitation, and it should be noted that other associative memory solutions and implementations of the request merger may be used to merge requests. For example, the request merger may implement a hash table, a tree structure, or other associative memory.

[0053] Figure 3B Example pseudocode for merging requests into an example request merger CAM table 300 according to at least one embodiment of the present disclosure is shown. The pseudocode is arranged to process requests through a series of if / else statements. The request is processed by a first if statement, where a for loop iterates through each entry of the CAM table 300. If the request matching the existing entry and the exit entry can be merged (mergeable), the request is merged into the existing entry and the process ends. If the existing entry is valid and the destination register, read size, scoreboard ID, byte offset, set ID, and path ID of the request match the destination register, read size, scoreboard ID, byte offset, set ID, and path ID of the existing entry, then the request matches the existing entry. If the existing entry is not locked and the existing entry does not match the thread warp ID of the request, then the existing entry is mergeable.

[0054] Continue to refer Figure 3C Otherwise, if the warp IDs match, the requests may not be merged and the pipeline may stall.

[0055] Otherwise, if a new entry is needed in CAM table 300, a determination is made as to whether CAM table 300 has an unused entry. If CAM table 300 has an unused entry, the new entry may be assigned to the unused entry. Once assigned, the set ID, path ID, destination unified register, read size, scoreboard ID, and byte offset associated with the new entry are set to the corresponding values ​​associated with the request. Furthermore, the new entry is set to valid; the new entry is unlocked; if the tag associated with the request hits, the data ready indicator is set to true; the warp mask of the new entry is set to include the warp associated with the request, and the new entry is set to the youngest entry in CAM table 300.

[0056] Otherwise, a new entry is needed, but there are no unused entries in the CAM table 300, and the pipeline is stalled.

[0057] Figure 4 A flowchart of a method 400 for multicasting data according to at least one embodiment of the present disclosure is shown. Although method 400 is described in the context of a processing unit, method 400 may also be performed by a program, custom circuitry, or a combination of custom circuitry and a program. For example, method 400 may be performed by a GPU (graphics processing unit), a CPU (central processing unit), or any processor capable of multicasting data. Furthermore, one of ordinary skill in the art will appreciate that any system that performs method 400 is within the scope and spirit of embodiments of the present invention.

[0058] We will now provide more illustrative information regarding various optional architectures and features that can be used to implement the above framework, depending on the user's desires. It should be noted that the following information is for illustrative purposes only and should not be construed as limiting in any way. Any of the following features may be optionally combined, with or without the other features described.

[0059] Although shown in a particular order or sequence, the order of operations may be modified unless otherwise indicated. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated operations may be performed in a different order, and some operations may be performed in parallel. In addition, one or more operations may be omitted in various embodiments. Therefore, not all operations are required in every embodiment.

[0060] In operation 402 of method 400, processing logic (such as, Figure 1 The request merger 120 in the embodiment may receive a request for loading the first data into the first sub-partition unit (such as Figure 1 The first instruction in the first unified register file (URF) associated with the sub-partition unit 104A) of the partition unit 104A.

[0061] In operation 404 of method 400, processing logic may receive a command for loading the first data into a second sub-partition unit (eg, Figure 1 In at least one embodiment, the first instruction and the second instruction are the same instruction. In at least one embodiment, the first instruction and the second instruction may be the same instruction because they are the same instruction executed by different threads or warps, rather than the same instance of a single instruction executed by a single thread. In at least one embodiment, the instructions are received into the request merger according to an arbitration scheme (e.g., a round-robin arbitration scheme).

[0062] In operation 406 of method 400, processing logic may merge the first instruction and the second instruction into a first entry of a request merger based on the instruction identifier. The first entry is associated with first data. In at least one embodiment, the first instruction and the second instruction are merged into the first entry of the request merger, and the processing logic is configured to identify the instruction identifier associated with the first entry of the request merger. The processing logic is further configured to determine that the instruction identifier associated with the first entry matches corresponding instruction identifiers associated with the first instruction and the second instruction, and to merge the first instruction and the second instruction into the first entry. In at least one embodiment, the instruction identifier may include at least one of a destination register number, a write scoreboard identifier, a payload size, an opcode, a cache line identifier, a cache tag identifier, or any combination thereof.

[0063] In operation 408 of method 400, processing logic may determine that the first data is available in the cache, e.g. Figure 1 In at least one embodiment, the constant cache 110 is associated with the first partition unit and the second partition unit. In at least one embodiment, the constant cache is configured to store constant data.

[0064] In operation 410 of method 400, in response to determining that the first data is available in the cache, processing logic may multicast the first data to the first unified register file and the second unified register file. In at least one embodiment, in response to multicasting the first data from the cache to the first URF and the second URF, processing logic may remove the first entry from the request merger. In at least one embodiment, the cache is a read-only cache. In at least one embodiment, to multicast the data, processing logic may write the data to a first write-back buffer and a second write-back buffer. The first write-back buffer may be associated with the first unified register file, and processing logic may write the first data to the first write-back buffer. The first write-back buffer may hold the first data until the first unified register file is available for write-back. The second write-back buffer may be associated with the second unified register file, and processing logic may write the first data to the second write-back buffer. The second write-back buffer may hold the first data until the second unified register file is available for write-back.

[0065] Parallel processing architecture

[0066] Figure 5A parallel processing unit (PPU) 500 is shown according to one embodiment. In one embodiment, the PPU 500 is a multi-threaded processor implemented on one or more integrated circuit devices. The PPU 500 is a latency-hiding architecture designed for processing many threads in parallel. A thread (e.g., an execution thread) is an instance of an instruction set configured to be executed by the PPU 500. In one embodiment, the PPU 500 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device (such as a liquid crystal display (LCD) device). In other embodiments, the PPU 500 can be used to perform general-purpose computations. Although an exemplary parallel processor is provided herein for illustrative purposes, it should be specifically noted that the processor is described for illustrative purposes only and any processor may be used in addition to and / or in place of the processor.

[0067] One or more PPUs 500 can be configured to accelerate thousands of high-performance computing (HPC), data center, and machine learning applications. PPUs 500 can be configured to accelerate numerous deep learning systems and applications, including autonomous vehicle platforms, deep learning, high-precision speech, image, and text recognition systems, intelligent video analysis, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations, among others.

[0068] like Figure 5 As shown, the PPU 500 includes an input / output (I / O) unit 505, a front-end unit 515, a scheduler unit 520, a work distribution unit 525, a hub 530, a crossbar switch (Xbar) 570, one or more processing clusters 550 (e.g., a general processing cluster (GPC)), and one or more partition units 580. The PPU 500 can be connected to a host processor or other PPUs 500 via one or more high-speed NVLink 510 interconnects. The PPU 500 can be connected to a host processor or other peripheral devices via interconnect 502. The PPU 500 can also be connected to a local memory including multiple memory devices 504. In one embodiment, the local memory can include multiple dynamic random access memory (DRAM) devices. The DRAM devices can be configured as a high-bandwidth memory (HBM) subsystem, in which multiple DRAM dies are stacked within each device.

[0069] The NVLink 510 interconnect enables the system to scale and include one or more PPUs 500 in conjunction with one or more CPUs, supporting cache coherency between the PPU 500 and the CPU, and CPU mastering. Data and / or commands can be sent by the NVLink 510 through the hub 530 to or from other units of the PPU 500, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). Figure 7B NVLink 510 is described in more detail.

[0070] I / O unit 505 is configured to send and receive communications (i.e., commands, data, etc.) from a host processor (not shown) via interconnect 502. I / O unit 505 can communicate with the host processor directly via interconnect 502, or through one or more intermediary devices (such as a memory bridge). In one embodiment, I / O unit 505 can communicate with one or more other processors (e.g., one or more PPUs 500) via interconnect 502. In one embodiment, I / O unit 505 implements a Peripheral Component Interconnect Express (PCIe) interface for communicating over a PCIe bus, and interconnect 502 is a PCIe bus. In alternative embodiments, I / O unit 505 can implement other types of known interfaces for communicating with external devices.

[0071] The I / O unit 505 decodes data packets received via the interconnect 502. In one embodiment, the data packets represent commands configured to cause the PPU 500 to perform various operations. The I / O unit 505 sends the decoded commands to various other units of the PPU 500 as specified by the commands. For example, some commands may be sent to the front end unit 515. Other commands may be sent to the hub 530 or other units of the PPU 500, such as one or more copy engines, video encoders, video decoders, a power management unit, etc. (not explicitly shown). In other words, the I / O unit 505 is configured to route communications between and among the various logical units of the PPU 500.

[0072] In one embodiment, a program executed by a host processor encodes a command stream in a buffer that provides a workload to the PPU 500 for processing. The workload may include many instructions and data to be processed by those instructions. A buffer is an area in memory that is accessible (i.e., read / write) by both the host processor and the PPU 500. For example, the I / O unit 505 can be configured to access a buffer in system memory connected to the interconnect 502 via a memory request transmitted through the interconnect 502. In one embodiment, the host processor writes a command stream into the buffer and then sends a pointer to the start of the command stream to the PPU 500. The front end unit 515 receives pointers to one or more command streams. The front end unit 515 manages one or more streams, reads commands from the streams, and forwards the commands to the various units of the PPU 500.

[0073] The front-end unit 515 is coupled to a scheduler unit 520, which configures various processing clusters 550 to process tasks defined by one or more flows. The scheduler unit 520 is configured to track status information associated with the various tasks managed by the scheduler unit 520. The status may indicate which processing cluster 550 the task is assigned to, whether the task is active or inactive, the priority associated with the task, and the like. The scheduler unit 520 manages the execution of multiple tasks on one or more processing clusters 550.

[0074] Scheduler unit 520 is coupled to work distribution unit 525, which is configured to dispatch tasks for execution on processing clusters 550. Work distribution unit 525 can track a number of scheduled tasks received from scheduler unit 520. In one embodiment, work distribution unit 525 manages a pending task pool and an active task pool for each processing cluster 550. The pending task pool can include a number of time slots (e.g., 32 time slots) containing tasks assigned to be processed by a particular processing cluster 550. The active task pool can include a number of time slots (e.g., 4 time slots) for tasks being actively processed by processing cluster 550. When a processing cluster 550 completes execution of a task, the task is evicted from the active task pool of processing cluster 550, and one of the other tasks from the pending task pool is selected and scheduled for execution on processing cluster 550. If an active task on processing cluster 550 has become idle, such as while waiting for data dependencies to be resolved, then the active task can be evicted from processing cluster 550 and returned to the pending task pool, while another task in the pending task pool is selected and scheduled for execution on processing cluster 550.

[0075] Work distribution unit 525 communicates with one or more processing clusters 550 via XBar (crossbar) 570. XBar 570 is an interconnect network that couples many units of PPU 500 to other units of PPU 500. For example, XBar 570 can be configured to couple work distribution unit 525 to a specific processing cluster 550. Although not explicitly shown, one or more other units of PPU 500 can also be connected to XBar 570 via hub 530.

[0076] Tasks are managed by the scheduler unit 520 and dispatched to the processing clusters 550 by the work distribution unit 525. The processing clusters 550 are configured to process tasks and generate results. The results can be consumed by other tasks within the processing cluster 550, routed to a different processing cluster 550 via the XBar 570, or stored in the memory 504. The results can be written to the memory 504 via the partition unit 580, which implements a memory interface for reading data from and writing data to the memory 504. The results can be sent to another PPU 504 or CPU via the NVLink 510. In one embodiment, the PPU 500 includes a number U of partition units 580, which is equal to the number of independent and different memory devices 504 coupled to the PPU 500. Figure 6B The partition unit 580 is described in more detail.

[0077] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API) that enables one or more applications to be executed on the host processor to schedule operations for execution on the PPU 500. In one embodiment, multiple computing applications are executed simultaneously by the PPU 500, and the PPU 500 provides isolation, quality of service (QoS), and independent address spaces for the multiple computing applications. The application can generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks to be executed by the PPU 500. The driver kernel outputs the tasks to one or more streams being processed by the PPU 500. Each task can include one or more related groups of threads, referred to herein as warps. In one embodiment, a warp includes 32 related threads that can execute in parallel. Collaborating threads can refer to multiple threads that include instructions to perform tasks and that can exchange data through shared memory. In combination Figure 7A Describes threads and cooperative threads in more detail.

[0078] Figure 6A According to one embodiment, Figure 5 The processing cluster 550 of the PPU 500 is Figure 6AAs shown, each processing cluster 550 includes multiple hardware units for processing tasks. In one embodiment, each processing cluster 550 includes a pipeline manager 610, a pre-raster operation unit (PROP) 615, a raster engine 625, a work distribution crossbar (WDX) 680, a memory management unit (MMU) 690, and one or more data processing clusters (DPCs) 620. It should be understood that Figure 6A The processing cluster 550 may include instead Figure 6A Other hardware units or other than the units shown in Figure 6A Other hardware units besides those shown in .

[0079] In one embodiment, the operation of processing cluster 550 is controlled by pipeline manager 610. Pipeline manager 610 manages the configuration of one or more DPCs 620 for processing tasks assigned to processing cluster 550. In one embodiment, pipeline manager 610 may configure at least one of one or more DPCs 620 to implement at least a portion of a graphics rendering pipeline. For example, DPC 620 may be configured to execute vertex shading programs on programmable multiprocessor 640. Multiprocessor 640 may typically include a streaming multiprocessor, a compute unit, an integrated many-core, etc. Pipeline manager 610 may also be configured to route packets received from work distribution unit 625 to appropriate logic units within processing cluster 550. For example, some packets may be routed to fixed-function hardware units within PROP 615 and / or raster engine 625, while other packets may be routed to DPC 620 for processing by primitive engine 635 or multiprocessor 640. In one embodiment, pipeline manager 610 may configure at least one of one or more DPCs 620 to implement a neural network model and / or a compute pipeline.

[0080] PROP unit 615 is configured to route data generated by raster engine 625 and DPC 620 to the raster operations (ROP) unit, in conjunction with Figure 6B The PROP unit 615 may also be configured to perform optimizations for color blending, organize pixel data, perform address translation, and the like.

[0081] The raster engine 625 includes several fixed-function hardware units configured to perform various raster operations. In one embodiment, the raster engine 625 comprises a setup engine, a coarse raster engine, a culling engine, a clipping engine, a fine raster engine, and a tile aggregation engine. The setup engine receives the transformed vertices and generates plane equations associated with the geometric primitives defined by the vertices. The plane equations are sent to the coarse raster engine to generate coverage information for the primitives (e.g., the x, y coverage mask of the tile). The output of the coarse raster engine is sent to the culling engine, where fragments associated with primitives that fail the z-test are culled, and to the clipping engine, where fragments outside the viewing frustum are clipped. Those fragments remaining after clipping and culling can be passed to the fine raster engine to generate pixel fragment attributes based on the plane equations generated by the setup engine. The raster engine 625 output includes, for example, fragments to be processed by the fragment shader implemented in the DPC 620.

[0082] Each DPC 620 included in the processing cluster 550 includes an M-pipe controller (MPC) 630, a primitive engine 635, and one or more multiprocessors 640. The MPC 630 controls the operation of the DPC 620 and routes packets received from the pipeline manager 610 to appropriate units within the DPC 620. For example, packets associated with vertices may be routed to the primitive engine 635, which is configured to retrieve vertex attributes associated with the vertices from the memory 504. Conversely, packets associated with shading programs may be sent to the multiprocessor 640.

[0083] In some embodiments, multiprocessor 640 includes a programmable multiprocessor, such as a programmable streaming multiprocessor, configured to process tasks represented by multiple threads. Each multiprocessor 640 is multithreaded and configured to concurrently execute multiple threads (e.g., 32 threads) from a particular warp. In one embodiment, multiprocessor 640 implements a SIMD (single instruction, multiple data) architecture, in which each thread in a warp is configured to process a different data set based on the same instruction set. All threads in a warp execute the same instructions. In another embodiment, multiprocessor 640 implements a SIMT (single instruction, multiple thread) architecture, in which each thread in a warp is configured to process a different data set based on the same instruction set, but in which individual threads in a warp are allowed to diverge during execution. In one embodiment, a program counter, call stack, and execution state are maintained for each warp, enabling concurrency between warps and serial execution within warps when threads within a warp diverge. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, thereby achieving equal concurrency between all threads within and between warps. When execution state is maintained for each individual thread, threads executing the same instruction can be converged and executed in parallel for maximum efficiency. Figure 7A Multiprocessor 640 is described in more detail.

[0084] The MMU 690 provides an interface between the processing cluster 550 and the partition unit 580. The MMU 690 can provide virtual address to physical address translation, memory protection, and arbitration of memory requests. In one embodiment, the MMU 690 provides one or more translation lookaside buffers (TLBs) for performing translations from virtual addresses to physical addresses in the memory 504.

[0085] Figure 6B According to one embodiment, Figure 5 The memory partition unit 580 of the PPU 500. Figure 6B As shown, the memory partition unit 580 includes a raster operations (ROP) unit 650, a level 2 (L2) cache 660, and a memory interface 670. The memory interface 670 is coupled to the memory 504. The memory interface 670 can implement a 32-, 64-, 128-, or 1024-bit data bus for high-speed data transfer. In one embodiment, the PPU 500 incorporates U memory interfaces 670, one for each pair of partition units 580, where each pair of partition units 580 is connected to a corresponding memory device 504. For example, the PPU 500 can be connected to up to Y memory devices 504, such as a high-bandwidth memory stack or graphics double data rate version 5 synchronous dynamic random access memory or other types of persistent memory.

[0086] In one embodiment, memory interface 670 implements an HBM2 memory interface, and Y is equal to half of U. In one embodiment, the HBM2 memory stack is located on the same physical package as PPU 500, providing significant power and area savings compared to conventional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory dies and Y is equal to 4, where the HBM2 stack includes two 128-bit channels per die, for a total of 8 channels and a data bus width of 1024 bits.

[0087] In one embodiment, memory 504 supports single-error correction, double-error detection (SECDED) error correction code (ECC) to protect data. ECC provides increased reliability for computing applications sensitive to data corruption. Reliability is particularly important in large cluster computing environments where PPU 500 processes very large data sets and / or long-running applications.

[0088] In one embodiment, the PPU 500 implements a multi-level memory hierarchy. In one embodiment, the memory partitioning unit 580 supports unified memory to provide a single, unified virtual address space for the CPU and PPU 500 memory, enabling data sharing between virtual memory systems. In one embodiment, the frequency of PPU 500 accesses to memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 500 where the pages are accessed more frequently. In one embodiment, NVLink 510 supports address translation services that allow the PPU 500 to directly access the CPU's page tables and provide full access to the CPU's memory by the PPU 500.

[0089] In one embodiment, the copy engine transfers data between multiple PPUs 500 or between a PPU 500 and a CPU. The copy engine can generate a page fault for an address that is not mapped to a page table. The memory partition unit 580 can then service the page fault, map the address into a page table, and then the copy engine can perform the transfer. In conventional systems, multiple copy engines operate on fixed memory (e.g., non-pageable) between multiple processors, which significantly reduces the available memory. Due to hardware page faults, addresses can be passed to the copy engine without worrying about whether the memory page is resident, and the copy process is transparent.

[0090] Data from memory 504 or other system memory can be retrieved by memory partition unit 580 and stored in L2 cache 660, which is located on-chip and shared between the various processing clusters 550. As shown, each memory partition unit 580 includes a portion of L2 cache 660 associated with the corresponding memory device 504. Lower-level caches can then be implemented in multiple units within a processing cluster 550. For example, each multiprocessor 640 can implement a level 1 (L1) cache. The L1 cache is a dedicated memory dedicated to a particular multiprocessor 640. Data from the L2 cache 660 can be retrieved and stored in each L1 cache for processing in the functional units of the multiprocessor 640. The L2 cache 660 is coupled to the memory interface 670 and the XBar 570.

[0091] The ROP unit 650 performs graphics raster operations related to pixel color such as color compression, pixel blending, etc. The ROP unit 650 also implements depth testing in conjunction with the raster engine 625, receiving the depth of the sample position associated with the pixel fragment from the culling engine of the raster engine 625. The depth of the sample position associated with the fragment is tested relative to the corresponding depth in the depth buffer. If the fragment passes the depth test for the sample position, the ROP unit 650 updates the depth buffer and sends the result of the depth test to the raster engine 625. It will be understood that the number of partition units 580 can be different than the number of processing clusters 550, and therefore each ROP unit 650 can be coupled to each processing cluster 550. The ROP unit 650 tracks packets received from the different processing clusters 550 and determines to which processing cluster 550 the results generated by the ROP unit 650 are routed via the Xbar 570. Although in Figure 6B In the embodiment, ROP unit 650 is included within memory partition unit 580, but in other embodiments, ROP unit 650 may be external to memory partition unit 580. For example, ROP unit 650 may reside in processing cluster 550 or another unit.

[0092] Figure 7A According to one embodiment, Figure 6A The multiprocessor 640. Figure 7A As shown, the multiprocessor 640 includes an instruction cache 705, one or more scheduler units 710, a register file 720, one or more processing cores 750, one or more special function units (SFUs) 752, one or more load / store units (LSUs) 754, an interconnection network 780, and a shared memory / L1 cache 770.

[0093] As described above, the work distribution unit 525 schedules tasks for execution on the processing cluster 550 of the PPU 500. Tasks are assigned to specific DPCs 620 within the processing cluster 550, and if the task is associated with a shader program, the task may be assigned to a multiprocessor 640. The scheduler unit 710 receives tasks from the work distribution unit 525 and manages the scheduling of instructions assigned to one or more thread blocks assigned to the multiprocessor 640. The scheduler unit 710 schedules the thread blocks for execution as warps of parallel threads, where each thread block is assigned at least one warp. In one embodiment, each warp executes 32 threads. The scheduler unit 710 can manage multiple different thread blocks, assign warps to different thread blocks, and then dispatch instructions from multiple different cooperative groups to various functional units (e.g., core 750, SFU 752, and LSU 754) during each clock cycle.

[0094] Cooperative Groups is a programming model for organizing groups of communicating threads that allows developers to express the granularity at which threads are communicating, enabling the expression of richer and more efficient decompositions of parallelism. The cooperative launch API supports synchronization between thread blocks to execute parallel algorithms. Conventional programming models provide a single simple construct for synchronizing cooperating threads: a barrier across all threads of a thread block (e.g., the syncthreads() function). However, programmers often want to define thread warps at a granularity smaller than the thread block granularity and synchronize within the defined group, enabling higher performance, design flexibility, and software reuse in the form of a collective group-wide function interface.

[0095] Cooperative Groups enable programmers to explicitly define warps at sub-block (e.g., as small as a single thread) and multi-block granularity and perform collective operations, such as synchronization, on threads in the cooperative group. The programming model supports clean composition across software boundaries so that libraries and utility functions can safely synchronize in their local environment without making assumptions about convergence. Cooperative Group primitives enable new patterns of cooperative parallelism, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire grid of thread blocks.

[0096] The dispatch unit 715 is configured to deliver instructions to one or more functional units. In this embodiment, the scheduler unit 710 includes two dispatch units 715, which enable scheduling two different instructions from the same warp during each clock cycle. In alternative embodiments, each scheduler unit 710 may include a single dispatch unit 715 or additional dispatch units 715.

[0097] Each multiprocessor 640 includes a register file 720 that provides a set of registers for the functional units of the multiprocessor 640. In one embodiment, the register file 720 is divided between each functional unit so that each functional unit is allocated a dedicated portion of the register file 720. In another embodiment, the register file 720 is divided between the different warps being executed by the multiprocessor 640. The register file 720 provides temporary storage for operands connected to the data paths of the functional units.

[0098] Each multiprocessor 640 includes L processing cores 750. In one embodiment, the multiprocessor 640 includes a large number (e.g., 128, etc.) of different processing cores 750. Each core 750 may include a fully pipelined, single-precision, double-precision, and / or mixed-precision processing unit, including a floating-point arithmetic logic unit (FLU) and an integer arithmetic logic unit (ALU). In one embodiment, the FLU implements the IEEE 754-2008 standard for floating-point operations. In one embodiment, the core 750 includes 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0099] Tensor cores are configured to perform matrix operations, and in one embodiment, one or more tensor cores are included in core 750. Specifically, the tensor cores are configured to perform deep learning matrix operations, such as convolution operations for neural network training and inference. In one embodiment, each tensor core operates on a 4×4 matrix and performs a matrix multiplication and accumulation operation D=A×B+C, where A, B, C, and D are 4×4 matrices.

[0100] In one embodiment, the matrix multiplication inputs A and B are 16-bit floating point matrices, while the accumulation matrices C and D can be 16-bit floating point or 32-bit floating point matrices. The tensor cores operate on 16-bit floating point input data as well as 32-bit floating point accumulations. The 16-bit floating point multiplication requires 64 operations to produce a full-precision product, which is then accumulated using 32-bit floating point additions with other intermediate products of the 4×4×4 matrix multiplication. In practice, tensor cores are used to perform larger two-dimensional or higher-dimensional matrix operations built from these smaller elements. APIs (such as the CUDA 9 C++ API) expose specialized matrix load, matrix multiplication and accumulation, and matrix store operations to efficiently use tensor cores from CUDA-C++ programs. At the CUDA level, the warp-level interface assumes that the 16×16 size matrix spans all 32 threads of the warp.

[0101] Each multiprocessor 640 also includes M SFUs 752 that perform special functions (e.g., attribute evaluation, reciprocal square root, etc.). In one embodiment, the SFUs 752 may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, the SFUs 752 may include a texture unit configured to perform texture map filtering operations. In one embodiment, the texture unit is configured to load a texture map (e.g., a 2D array of texels) from memory 504 and sample the texture map to generate sampled texture values ​​for use in shader programs executed by the multiprocessor 640. In one embodiment, the texture map is stored in shared memory / L1 cache 670. The texture unit performs texture operations, such as filtering operations using mip maps (e.g., texture maps of different levels of detail). In one embodiment, each SM 540 includes two texture units.

[0102] Each multiprocessor 640 also includes N LSUs 754 that perform load and store operations between the shared memory / L1 cache 770 and the register file 720. Each multiprocessor 640 includes an interconnect network 780 that connects each functional unit to the register file 720 and connects the LSUs 754 to the register file 720 and the shared memory / L1 cache 770. In one embodiment, the interconnect network 780 is a crossbar switch that can be configured to connect any functional unit to any register in the register file 720 and to connect the LSUs 754 to memory locations in the register file and the shared memory / L1 cache 770.

[0103] The shared memory / L1 cache 770 is an on-chip memory array that allows data storage and communication between the multiprocessor 640 and the primitive engine 635, as well as between threads in the multiprocessor 640. In one embodiment, the shared memory / L1 cache 770 includes 128KB of storage capacity and is in the path from the multiprocessor 640 to the partition unit 580. The shared memory / L1 cache 770 can be used to cache reads and writes. One or more of the shared memory / L1 cache 770, the L2 cache 660, and the memory 504 is a backing store.

[0104] Combining data cache and shared memory functionality into a single memory block provides the best overall performance for both types of memory access. This capacity can be used by programs as a cache that does not utilize the shared memory. For example, if the shared memory is configured to use half its capacity, texture and load / store operations can use the remaining capacity. Integration within shared memory / L1 cache 770 enables shared memory / L1 cache 770 to function as a high-throughput pipeline for streaming data, while providing high-bandwidth and low-latency access to frequently reused data.

[0105] When configured for general parallel computing, a simpler configuration can be used compared to graphics processing. Specifically, Figure 5 The fixed-function graphics processing unit is shown bypassed, creating a simpler programming model. In a general-purpose parallel computing configuration, the work distribution unit 525 assigns and distributes thread blocks directly to the DPC 620. The threads in the block execute the same program, using unique thread IDs in computations to ensure each thread produces unique results, using the multiprocessor 640 to execute the program and perform computations, using the shared memory / L1 cache 770 to communicate between threads, and using the LSU 754 to read and write global memory through the shared memory / L1 cache 770 and the memory partition unit 580. When configured for general-purpose parallel computing, the multiprocessor 640 can also write commands that the scheduler unit 520 can use to start new work on the DPC 620.

[0106] The PPU 500 may be included in a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, etc. In one embodiment, the PPU 500 is included on a single semiconductor substrate. In another embodiment, the PPU 500 is included on a system-on-chip (SoC) along with one or more other devices (such as an additional PPU 500, a memory 504, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), etc.).

[0107] In one embodiment, PPU 500 may be included on a graphics card that includes one or more memory devices 504. The graphics card may be configured to interface with a PCIe slot on a motherboard of a desktop computer. In another embodiment, PPU 500 may be an integrated graphics processing unit (iGPU) or parallel processor included in a chipset of the motherboard.

[0108] Exemplary Computing System

[0109] Systems with multiple GPUs and CPUs are being used across various industries as developers expose and exploit greater parallelism in applications such as artificial intelligence computing. High-performance GPU-accelerated systems with tens to thousands of computing nodes are deployed in data centers, research institutions, and supercomputers to solve larger problems. As the number of processing devices within high-performance systems increases, communication and data transmission mechanisms need to scale to support this increased bandwidth.

[0110] Figure 7B According to one embodiment, the Figure 5 Conceptual diagram of a processing system 700 implemented by a PPU 500. The exemplary system 765 may be configured to implement Figure 4 The processing system 700 includes a CPU 730, a switch 712, and each of the plurality of PPUs 500 and a corresponding memory 504. The NVLink 510 provides a high-speed communication link between each PPU 500. Figure 7B 502 connections, but the number of connections connected to each PPU 500 and CPU 730 may vary. Switch 712 interfaces between interconnect 502 and CPU 730. PPU 500, memory 504, and NVLink 510 may be located on a single semiconductor platform to form parallel processing module 725. In one embodiment, switch 712 supports two or more protocols that interface between various different connections and / or links.

[0111] In another embodiment (not shown), NVLink 510 provides one or more high-speed communication links between each PPU 500 and CPU 730, and switch 712 interfaces between interconnect 502 and each PPU 500. The PPUs 500, memory 504, and interconnect 502 may be located on a single semiconductor platform to form a parallel processing module 725. In yet another embodiment (not shown), interconnect 502 provides one or more communication links between each PPU 500 and CPU 730, and switch 712 interfaces between each PPU 500 using NVLink 510 to provide one or more high-speed communication links between the PPUs 500. In another embodiment (not shown), NVLink 510 provides one or more high-speed communication links between the PPUs 500 and CPU 730 through switch 712. In yet another embodiment (not shown), interconnect 502 provides one or more communication links directly between each PPU 500. One or more NVLink 510 high-speed communication links may be implemented as a physical NVLink interconnect or as an on-chip or on-die interconnect using the same protocol as NVLink 510.

[0112] In the context of this specification, a single semiconductor platform may refer to a unique, single semiconductor-based integrated circuit fabricated on a die or chip. It should be noted that the term single semiconductor platform may also refer to a multi-chip module with increased connectivity that emulates on-chip operation and is substantially improved by utilizing conventional bus implementations. Of course, various circuits or devices may also be placed separately or in various combinations of semiconductor platforms, depending on the needs of the user. Alternatively, the parallel processing module 725 may be implemented as a circuit board substrate, and each of the PPU 500 and / or memory 504 may be a packaged device. In one embodiment, the CPU 730, switch 712, and parallel processing module 725 are located on a single semiconductor platform.

[0113] In one embodiment, the signaling rate of each NVLink 510 is 20 to 25 Gbit / s, and each PPU 500 includes six NVLink 510 interfaces (e.g., Figure 7B As shown, each PPU 500 includes five NVLink 510 interfaces. Each NVLink 510 provides a data transfer rate of 25 Gbit / s in each direction, with six links providing 500 Gbit / s. When the CPU 730 also includes one or more NVLink 510 interfaces, the NVLink 510 can be used specifically for Figure 7B PPU to PPU communication shown, or some combination of PPU to PPU and PPU to CPU.

[0114] In one embodiment, NVLink 510 allows direct load / store / atomic access from the CPU 730 to the memory 504 of each PPU 500. In one embodiment, NVLink 510 supports coherency operations, allowing data read from memory 504 to be stored in the cache hierarchy of the CPU 730, reducing cache access latency for the CPU 730. In one embodiment, NVLink 510 includes support for Address Translation Services (ATS), allowing the PPU 500 to directly access page tables within the CPU 730. One or more NVLinks 510 can also be configured to operate in a low-power mode.

[0115] Figure 7C An exemplary system 765 is shown in which various architectures and / or functionalities of various previous embodiments may be implemented. The exemplary system 765 may be configured to implement Figure 4 The method 400 shown in FIG.

[0116] As shown, a system 765 is provided that includes at least one central processing unit 730 connected to a communication bus 775. The communication bus 775 can be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or one or more point-to-point communication protocols. The system 765 also includes a main memory 740. Control logic (software) and data are stored in the main memory 740, which can take the form of random access memory (RAM).

[0117] System 765 also includes an input device 760, a parallel processing system 725, and a display device 745, such as a conventional CRT (cathode ray tube), an LCD (liquid crystal display), an LED (light emitting diode), a plasma display, etc. User input can be received from input device 760 (e.g., a keyboard, a mouse, a touchpad, a microphone, etc.). Each of the aforementioned modules and / or devices can even be located on a single semiconductor platform to form system 765. Alternatively, the modules can be located separately or in various combinations of semiconductor platforms, depending on the user's needs.

[0118] Furthermore, system 765 may be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, etc.) through network interface 735 for communication purposes.

[0119] System 765 may also include auxiliary storage (not shown). Auxiliary storage 610 includes, for example, a hard drive and / or a removable storage drive, representative of a floppy disk drive, a tape drive, an optical disk drive, a digital versatile disk (DVD) drive, a recording device, or a universal serial bus (USB) flash memory. The removable storage drive reads from and / or writes to a removable storage unit in a well-known manner.

[0120] Computer programs or computer control logic algorithms may be stored in the main memory 740 and / or secondary storage. These computer programs, when executed, enable the system 765 to perform various functions. The memory 740, storage, and / or any other storage are possible examples of computer-readable media.

[0121] The architecture and / or functionality of the various preceding figures can be implemented in the context of a general-purpose computer system, a circuit board system, a game console system dedicated to entertainment purposes, a dedicated system, and / or any other desired system. For example, system 765 can take the form of a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, a mobile telephone device, a television, a workstation, a game console, an embedded system, and / or any other type of logic.

[0122] Although various embodiments have been described above, it should be understood that they are presented by way of example only and not limitation. Therefore, the breadth and scope of the preferred embodiment should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

[0123] Graphics processing pipeline

[0124] In one embodiment, the PPU 500 includes a graphics processing unit (GPU). The PPU 500 is configured to receive commands specifying a shader for processing graphics data. Graphics data can be defined as a set of primitives, such as points, lines, triangles, quadrilaterals, triangle strips, etc. Typically, a primitive includes data specifying a plurality of vertices of the primitive (e.g., in a model space coordinate system) and attributes associated with each vertex of the primitive. The PPU 500 can be configured to process the primitives to generate a frame buffer (e.g., pixel data for each of the pixels of a display).

[0125] An application writes model data for a scene (e.g., a collection of vertices and attributes) into memory (such as system memory or memory 504). The model data defines each of the objects that may be visible on the display. The application then makes an API call to the driver kernel, requesting the model data to be rendered and displayed. The driver kernel reads the model data and writes commands to one or more streams to perform operations to process the model data. These commands may reference different shader programs to be implemented on the multiprocessors 640 of the PPU 500, including one or more of vertex shading, hull shading, domain shading, geometry shading, and pixel shading. For example, one or more of the multiprocessors 640 may be configured to execute a vertex shading program that processes multiple vertices defined by the model data. In one embodiment, different multiprocessors 640 may be configured to execute different shading programs simultaneously. For example, a first subset of the multiprocessors 640 may be configured to execute a vertex shading program, while a second subset of the multiprocessors 640 may be configured to execute a pixel shading program. The first subset of the multiprocessors 640 processes the vertex data to generate processed vertex data and writes the processed vertex data to the L2 cache 660 and / or memory 504. After the processed vertex data is rasterized (e.g., converted from three-dimensional data to two-dimensional data in screen space) to generate fragment data, a second subset of the multiprocessors 640 performs pixel shading to generate processed fragment data, which is then blended with other processed fragment data and written to a frame buffer in the memory 504. The vertex shading program and the pixel shading program can be executed simultaneously, processing different data from the same scene in a pipelined manner, until all model data for the scene has been rendered to the frame buffer. The contents of the frame buffer are then transmitted to the display controller for display on the display device.

[0126] Figure 8 According to one embodiment, Figure 5 500 . The graphics processing pipeline 800 is an abstract flow chart of the processing steps implemented to generate a 2D computer-generated image from 3D geometric data. As is well known, pipeline architectures can perform long-latency operations more efficiently by dividing the operations into multiple stages, where the output of each stage is coupled to the input of the next consecutive stage. Thus, the graphics processing pipeline 800 receives input data 801 that is passed from one stage of the graphics processing pipeline 800 to the next stage to generate output data 802. In one embodiment, the graphics processing pipeline 800 may represent a graphics processing pipeline composed of API-defined graphics processing pipeline. Alternatively, graphics processing pipeline 800 can be implemented within the functional and architectural context of the previous figures and / or one or more of any subsequent figures.

[0127] like Figure 8As shown, the graphics processing pipeline 800 includes a pipeline architecture comprising multiple stages. These stages include, but are not limited to, a data assembly stage 810, a vertex shading stage 820, a primitive assembly stage 830, a geometry shading stage 840, a viewport scale, cull, and clip (VSCC) stage 850, a rasterization stage 860, a fragment shading stage 870, and a raster operation stage 880. In one embodiment, input data 801 includes commands that configure a processing unit to implement the stages of the graphics processing pipeline 800 and configure geometric primitives (e.g., points, lines, triangles, quads, triangle strips, or fans, etc.) to be processed by these stages. Output data 802 may include pixel data (i.e., color data), which is copied to a frame buffer or other type of surface data structure in memory.

[0128] The data assembly stage 810 receives input data 801, which specifies vertex data for high-level surfaces, primitives, etc. The data assembly stage 810 collects the vertex data in temporary storage or queues, such as by receiving a command from the host processor that includes a pointer to a buffer in memory and reading the vertex data from the buffer. The vertex data is then passed to the vertex shading stage 820 for processing.

[0129] The vertex shading stage 820 processes vertex data by executing a set of operations (e.g., a vertex shader or program) on each vertex at a time. A vertex may be specified, for example, as a 4-coordinate vector (e.g., , ) associated with one or more vertex attributes (e.g., color, texture coordinates, surface normal, etc.).<x,y,z,w> ). The vertex shading stage 820 can manipulate various vertex attributes, such as position, color, texture coordinates, etc. In other words, the vertex shading stage 820 performs operations on the vertex coordinates or other vertex attributes associated with the vertex. These operations typically include lighting operations (e.g., modifying the color attribute of the vertex) and transformation operations (e.g., modifying the coordinate space of the vertex). For example, a vertex can be specified using coordinates in object coordinate space, which are transformed by multiplying the coordinates by a matrix that converts the coordinates from the object coordinate space to world space or normalized-device-coordinate (NCD) space. The vertex shading stage 820 generates transformed vertex data that is passed to the primitive assembly stage 830.

[0130] The primitive assembly stage 830 collects the vertices output by the vertex shading stage 820 and groups the vertices into geometric primitives for processing by the geometry shading stage 840. For example, the primitive assembly stage 830 can be configured to group every three consecutive vertices into geometric primitives (e.g., triangles) for transmission to the geometry shading stage 840. In some embodiments, particular vertices can be reused for consecutive geometric primitives (e.g., two consecutive triangles in a triangle strip can share two vertices). The primitive assembly stage 830 transmits the geometric primitives (e.g., a collection of associated vertices) to the geometry shading stage 840.

[0131] The geometry shading stage 840 processes geometric primitives by executing a set of operations (e.g., geometry shaders or programs) on the geometric primitives. A tessellation operation can generate one or more geometric primitives from each geometric primitive. In other words, the geometry shading stage 840 can subdivide each geometric primitive into a finer mesh of two or more geometric primitives for processing by the rest of the graphics processing pipeline 800. The geometry shading stage 840 passes the geometric primitives to the viewport SCC stage 850.

[0132] In one embodiment, the graphics processing pipeline 800 may operate within a multiprocessor and vertex shading stage 820, primitive assembly stage 830, geometry shading stage 840, fragment shading stage 870, and / or hardware / software associated therewith, and may sequentially perform processing operations. Once the sequential processing operations are completed, in one embodiment, the viewport SCC stage 850 may utilize the data. In one embodiment, primitive data processed by one or more stages in the graphics processing pipeline 800 may be written to a cache (e.g., an L1 cache, a vertex cache, etc.). In this case, in one embodiment, the viewport SCC stage 850 may access the data in the cache. In one embodiment, the viewport SCC stage 850 and the rasterization stage 860 are implemented as fixed function circuits.

[0133] The viewport SCC stage 850 performs viewport scaling, culling, and clipping of geometric primitives. Each surface being rendered is associated with an abstract camera position. The camera position represents the position of the viewer viewing the scene and defines a viewing cone that surrounds the objects of the scene. The viewing cone can include a viewing plane, a back plane, and four clipping planes. Any geometric primitives that are completely outside the viewing cone can be culled (e.g., discarded) because they will not contribute to the final rendered scene. Any geometric primitives that are partially inside the viewing cone and partially outside the viewing cone can be clipped (e.g., converted to new geometric primitives that are enclosed within the viewing cone). In addition, each geometric primitive can be scaled based on the depth of the viewing cone. All potentially visible geometric primitives are then transferred to the rasterization stage 860.

[0134] The rasterization stage 860 converts 3D geometric primitives into 2D fragments (e.g., capable of being used for display, etc.). The rasterization stage 860 can be configured to use the vertices of the geometric primitives to set a set of plane equations from which various attributes can be interpolated. The rasterization stage 860 can also calculate a coverage mask for multiple pixels, which indicates whether one or more sample positions of a pixel intercept the geometric primitive. In one embodiment, a z test can also be performed to determine whether the geometric primitive is occluded by other geometric primitives that have been rasterized. The rasterization stage 860 generates fragment data (e.g., interpolated vertex attributes associated with a specific sample position for each covered pixel), which is transmitted to the fragment shading stage 870.

[0135] The fragment shading stage 870 processes the fragment data by executing a set of operations (e.g., a fragment shader or program) on each of the fragments. The fragment shading stage 870 can generate pixel data (e.g., color values) for the fragment, such as by performing lighting operations or sampling a texture map using the fragment's interpolated texture coordinates. The fragment shading stage 870 generates pixel data, which is sent to the raster operations stage 880.

[0136] The raster operations stage 880 may perform various operations on the pixel data, such as performing alpha tests, stencil tests, and blending the pixel data with other pixel data corresponding to other fragments associated with the pixel. When the raster operations stage 880 has completed processing the pixel data (e.g., output data 802), the pixel data may be written to a render target, such as a frame buffer, a color buffer, etc.

[0137] It should be appreciated that one or more additional stages may be included in the graphics processing pipeline 800 in addition to or in place of one or more of the above-described stages. Various implementations of the abstract graphics processing pipeline may implement different stages. Furthermore, in some embodiments, one or more of the above-described stages may be excluded from the graphics processing pipeline (such as the geometry shading stage 840). Other types of graphics processing pipelines are contemplated within the scope of the present disclosure. Furthermore, any stage of the graphics processing pipeline 800 may be implemented by one or more dedicated hardware units within a graphics processor (such as the PPU 500). Other stages of the graphics processing pipeline 800 may be implemented by programmable hardware units (such as the multiprocessor 640 of the PPU 500).

[0138] The graphics processing pipeline 800 can be implemented via an application program executed by a host processor (such as a CPU). In one embodiment, a device driver can implement an application programming interface (API) that defines various functions that can be utilized by the application program to generate graphics data for display. A device driver is a software program that includes multiple instructions that control the operation of the PPU 500. The API provides an abstraction for programmers, allowing them to utilize specialized graphics hardware (such as the PPU 500) to generate graphics data without requiring them to utilize the specific instruction set of the PPU 500. An application program can include API calls that are routed to the device driver of the PPU 500. The device driver interprets the API calls and performs various operations in response to the API calls. In some cases, the device driver can perform operations by executing instructions on the CPU. In other cases, the device driver can perform operations at least in part by initiating operations on the PPU 500 using an input / output interface between the CPU and the PPU 500. In one embodiment, the device driver is configured to implement the graphics processing pipeline 800 using the hardware of the PPU 500.

[0139] Various programs can be executed within the PPU 500 to implement the various stages of the graphics processing pipeline 800. For example, a device driver can launch a kernel on the PPU 500 to execute the vertex shading stage 820 on one multiprocessor 640 (or multiple multiprocessors 640). The device driver (or the initial kernel executed by the PPU 500) can also launch other kernels on the PPU 500 to execute other stages of the graphics processing pipeline 800, such as the geometry shading stage 840 and the fragment shading stage 870. In addition, some of the stages of the graphics processing pipeline 800 can be implemented on fixed unit hardware, such as a rasterizer or data assembler implemented within the PPU 500. It should be appreciated that the results from one kernel can be processed by one or more intermediate fixed-function hardware units before being processed by subsequent kernels on the multiprocessor 640.

[0140] Machine Learning

[0141] Deep neural networks (DNNs) developed on processors such as the PPU 500 are already being used in a variety of use cases: from self-driving cars to faster drug development, from automatic image captioning in online image databases to intelligent real-time language translation in video chat applications. Deep learning is a technology that models the neural learning process of the human brain, constantly learning, getting smarter, and delivering more accurate results faster over time. A child, initially taught by an adult to correctly identify and classify various shapes, eventually becomes able to recognize shapes without any tutoring. Similarly, deep learning or neural learning systems need to be trained in object recognition and classification to become smarter and more efficient at recognizing basic objects, occluded objects, and assigning context to objects.

[0142] At the simplest level, neurons in the human brain examine the various inputs they receive, assign a level of importance to each of these inputs, and pass outputs to other neurons for processing. An artificial neuron, or perceptron, is the most basic model of a neural network. In one example, a perceptron can receive one or more inputs representing various features of the object it is being trained to recognize and classify, and each of these features is assigned a certain weight based on its importance in defining the object's shape.

[0143] Deep neural network (DNN) models consist of multiple layers of connected nodes (e.g., perceptrons, Boltzmann machines, radial basis functions, convolutional layers, etc.), which can be trained with large amounts of input data to solve complex problems quickly and accurately. In one example, the first layer of a DNN model breaks down an input image of a car into its components and looks for basic patterns (such as lines and angles). The second layer assembles the lines to look for higher-level patterns, such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the final layers generate labels for the input image, identifying the model of a specific car brand.

[0144] Once trained, a DNN can be deployed and used to recognize and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include recognizing handwritten numbers on check deposits at an ATM, identifying images of friends in photos, providing movie recommendations to over 50 million users, identifying and classifying different types of cars, pedestrians, and road hazards in self-driving cars, or translating human speech in real time.

[0145] During training, data flows through the DNN in a forward propagation phase until a prediction is produced, which indicates the label corresponding to the input. If the neural network does not correctly label the input, the error between the correct label and the predicted label is analyzed, and the weights are adjusted for each feature during a backward propagation phase until the DNN correctly labels that input and other inputs in the training dataset. Training complex neural networks requires a large amount of parallel computing performance, including floating-point multiplications and additions supported by the PPU 500. Inference is a less computationally intensive and latency-sensitive process than training, in which a trained neural network is applied to new inputs it has not seen before to classify images, translate speech, and generally reason about new information.

[0146] Neural networks rely heavily on matrix math operations, and complex, multi-layer networks require significant floating-point performance and bandwidth for efficiency and speed. With thousands of processing cores optimized for matrix math operations and delivering tens to hundreds of TFLOPS of performance, the PPU 500 is a computing platform capable of delivering the performance required for deep neural network-based artificial intelligence and machine learning applications.

Claims

1. A parallel processing unit (PPU), comprising: a plurality of sub-partition units configured as execution threads; a cache memory coupled to the plurality of sub-partition units; as well as a request merger coupled to the plurality of sub-partition units and the cache, wherein the request merger is configured to: receiving a first instruction to load first data into a first register file associated with a first sub-partition unit of the plurality of sub-partition units; receiving a second instruction to load the first data into a second register file associated with a second sub-partition unit of the plurality of sub-partition units; merging the first instruction and the second instruction into a first entry of the request merger based on an instruction identifier, wherein the first entry is associated with the first data; determining that the first data is available in the cache; as well as In response to determining that the first data is available in the cache, the first data is multicast from the cache to the first register file and the second register file.

2. The PPU according to claim 1, further comprising: a first write-back buffer associated with the first register file, wherein the request merger is to write the first data to the first write-back buffer, wherein the first write-back buffer holds the first data until the first register file is available for write-back; and A second write-back buffer associated with the second register file, wherein the request merger is to write the first data to the first write-back buffer, wherein the second write-back buffer holds the first data until the second register file is available for write-back.

3. The PPU of claim 2 , wherein to multicast the first data from the cache to the first register file and the second register file, the request merger is configured to: The data is written into the first write-back buffer and the second write-back buffer. 4 . The PPU of claim 1 , wherein the first instruction and the second instruction are the same instruction executed by corresponding threads. The PPU of claim 1 , wherein the cache is configured to store constant data.

6. The PPU of claim 1 , wherein to merge the first instruction and the second instruction into the first entry of the request merger based on an instruction identifier, the request merger is configured to: identifying an instruction identifier associated with the first entry of the request merger; determining that the instruction identifier associated with the first entry matches corresponding instruction identifiers associated with the first instruction and the second instruction; and The first instruction and the second instruction are merged into the first entry.

7. The PPU of claim 6, wherein the instruction identifier comprises at least one of a destination register number, a write scoreboard identifier, a load size, an opcode, a cache line identifier, a program counter, or a cache tag identifier.

8. The PPU according to claim 1 , wherein the request merger is further configured to: In response to multicasting the first data from the cache to the first register file and the second register file, removing the first entry from the request merger.

9. The PPU of claim 1, wherein instructions are received into the request merger according to an arbitration scheme.

10. The PPU of claim 1, wherein the cache is a read-only cache.

11. A parallel processor comprising: A plurality of multiprocessors, each multiprocessor having a cache and a request coalescer, wherein the request coalescer is configured to: receiving a first instruction to load first data into a first register file; receiving a second instruction to load the first data into a second register file; merging the first instruction and the second instruction into a first entry of the request merger based on an instruction identifier, wherein the first entry is associated with the first data; determining that the first data is available in the cache; as well as In response to determining that the first data is available in the cache, the first data is multicast from the cache to the first register file and the second register file.

12. The parallel processor according to claim 11, further comprising: a first write-back buffer associated with the first register file, wherein the request merger is to write the first data to the first write-back buffer, wherein the first write-back buffer holds the first data until the first register file is available for write-back; and A second write-back buffer associated with the second register file, wherein the request merger is to write the first data to the second write-back buffer, wherein the second write-back buffer holds the first data until the second register file is available for write-back.

13. The parallel processor according to claim 12, wherein in order to multicast the first data from the cache to the first register file and the second register file, the request merger is configured to: The data is written into the first write-back buffer and the second write-back buffer.

14. The parallel processor of claim 11 , wherein each of the plurality of multiprocessors further comprises a plurality of sub-partition units, wherein the first instruction is associated with a first sub-partition unit of the plurality of sub-partition units, and the second instruction is associated with a second sub-partition unit of the plurality of sub-partition units.

15. The parallel processor of claim 11, wherein the first instruction and the second instruction are the same instruction executed by corresponding threads.

16. The parallel processor of claim 11, wherein the cache is configured to store constant data.

17. The parallel processor according to claim 11, wherein in order to merge the first instruction and the second instruction into the first entry of the request merger, the request merger is configured to: identifying an instruction identifier associated with the first entry of the request merger; determining that the instruction identifier associated with the first entry matches corresponding instruction identifiers associated with the first instruction and the second instruction; and The first instruction and the second instruction are merged into the first entry.

18. The parallel processor of claim 17, wherein the instruction identifier comprises at least one of a destination register number, a write scoreboard identifier, a load size, an opcode, a cache line identifier, a program counter, or a cache tag identifier.

19. The parallel processor according to claim 11, wherein the request merger is further configured to: In response to multicasting the first data from the cache to the first register file and the second register file, removing the first entry from the request merger.

20. A method comprising: receiving a first instruction to load first data into a first register file associated with a first sub-partition unit of the plurality of sub-partition units; receiving a second instruction to load the first data into a second register file associated with a second sub-partition unit of the plurality of sub-partition units; merging the first instruction and the second instruction into a first entry of a request merger based on an instruction identifier, wherein the first entry is associated with the first data; determining that the first data is available in a cache associated with the plurality of sub-partition units; as well as In response to determining that the first data is available in the cache, the first data is multicast from the cache to the first register file and the second register file.

Images