Systems and methods for aperture-specific cache operations

Abstract

The invention discloses a system and method for aperture-specific caching operations. Comprising a first cache is coupled to a system memory and a parallel processing unit (PPU) comprising a second cache. An operation to modify a cache line of a second cache associated with a first aperture of a system memory is received. A first subset of cache lines of a second cache is identified. The first subset of cache lines is associated with a first aperture of the system memory. A second cache line subset of the second cache is identified. The second subset of cache lines is associated with a second aperture of the system memory. The first subset of cache lines is modified in accordance with the specification of the cache operation.

Description

[0001] Related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 566,142, filed on March 15, 2024, which is incorporated herein by reference in its entirety. Technical Field

[0003] Embodiments of the present disclosure generally relate to parallel processing systems. Specifically, embodiments of the present disclosure relate to parallel processing systems and methods for aperture-specific cache operations. Background Art

[0004] Parallel processing in high-performance computing (HPC) systems involves executing multiple computing tasks or operations simultaneously. This is achieved by breaking down larger computations into smaller, independent subtasks that can be processed concurrently by multiple processors. In some cases, parallel processing involves distributed computing, where tasks are spread across multiple computing clusters. Each cluster can operate independently and can facilitate communication to share results. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 is a block diagram illustrating a computer system configured to implement one or more aspects of the present disclosure.

[0006] Figure 2 is a diagram illustrating an aperture-specific cache coherency system in accordance with at least one embodiment of the present disclosure.

[0007] Figure 3 A flowchart illustrating an aperture-specific buffer operation method according to at least one embodiment of the present disclosure is shown.

[0008] Figure 4 A parallel processing unit is shown in accordance with at least one embodiment of the present disclosure.

[0009] Figure 5A According to at least one embodiment of the present disclosure, Figure 4 A general-purpose processing cluster within a parallel processing unit.

[0010] Figure 5B According to at least one embodiment of the present disclosure, Figure 4 A memory partitioning unit of a parallel processing unit.

[0011] Figure 6AAccording to at least one embodiment of the present disclosure, Figure 5A Streaming multiprocessor.

[0012] Figure 6B According to at least one embodiment of the present disclosure, Figure 4 Conceptual diagram of the processing system implemented by the PPU.

[0013] Figure 6C An exemplary system is shown in which the various architecture and / or functionality of the various previous embodiments may be implemented.

[0014] Figure 7 According to at least one embodiment of the present disclosure Figure 4 Conceptual diagram of the graphics processing pipeline implemented by the PPU. DETAILED DESCRIPTION

[0015] High-performance computing (HPC) systems can use specialized hardware architectures, such as parallel processing units (PPUs), to enhance parallel processing capabilities. PPUs are designed to extract high performance using a large number of small parallel execution threads on a dedicated programmable multiprocessor. In a PPU, a group of threads (e.g., a warp) can concurrently execute the same instruction on multiple processors with different input data. This execution model is called single instruction, multiple thread (SIMT) and is commonly used for parallel computing. PPUs are designed to execute programs (e.g., kernels, shader programs, etc.) in parallel by executing many thread groups on the PPU, where each thread in the thread group typically operates on a different portion of data. Because the PPU architecture is typically optimized for executing many parallel threads simultaneously, PPUs (e.g., graphics processing units (GPUs)) can be used to accelerate artificial intelligence (AI), HPC, cloud, and hyperscale workloads. Specifically, as AI models enhance deep recommendation systems that contain large amounts of data (e.g., 10TB or more), their complexity and scale are rapidly increasing.

[0016] System architectures are emerging that tightly couple the central processing unit (CPU) and the PPU via high-bandwidth interconnects (e.g., high-speed buses or dedicated on-chip communication channels) to enable fast memory access. Notably, the bidirectional, high-bandwidth, cache-coherent connection between CPU and PPU memory allows multiple application threads (CPU or PPU) to directly access system-assigned memory. For example, the PPU can cache system memory associated with and owned by the CPU in its internal PPU cache. Similarly, the CPU can cache video memory associated with and owned by the PPU in its internal CPU cache. Some legacy systems can maintain coherency between GPU caches (e.g., L1, L2, L3, etc.) and CPU caches (e.g., L0, L1, L2, system-level cache (SLC)) through software-based explicit cache flushing and invalidation. For example, the CPU can allocate a portion of system memory and designate it as a buffer available to the PPU for processing. When the PPU completes its work on the buffer, it may modify some data associated with addresses in system memory that belong to the CPU but are known to the CPU. Software mechanisms can ensure that the PPU has written all of its data to the CPU cache before control of the buffer is transferred back to the CPU. Cache invalidation can be performed on the entire GPU cache so that every cache line associated with system memory is written back to the CPU cache and invalidated. This process can take many (e.g., thousands) cycles and increase the latency between GPU-CPU handshakes. This latency can impact the performance and reliability of latency-sensitive systems such as advanced driver assistance systems (ADAS) and autonomous driving systems (ADS).

[0017] To avoid the issues associated with software-managed CPU-GPU coherence, some conventional systems can use a coherent memory interconnect with native hardware support to leverage hardware coherence between the GPU and CPU caches. For example, chip-to-chip (C2C) hardware coherence can enable the GPU to cache system memory in cache at a cache line granularity, without requiring page migrations between system memory and video memory. Hardware coherence can simplify and accelerate CPU-GPU communication for data that is closely shared between the CPU and PPU. This hardware coherence can improve the performance of memory accesses to non-local memory, such as CPU threads accessing GPU memory (e.g., video memory) or GPU threads accessing CPU memory (e.g., system memory). Hardware-managed coherence simplifies the programming model because explicit software coherence is not used. For example, if a cache line in the GPU cache is modified, the CPU may receive an indication that the cache line has been modified and can retrieve the relevant data from the GPU when the CPU needs it. Therefore, explicit flushing and invalidation of GPU hardware coherent data may not be required. However, coherently caching system memory buffers that are only accessible to the PPU may incur unnecessary overhead associated with hardware coherence because the CPU may never need to access the data stored in such memory buffers.

[0018] Aspects and embodiments of the present disclosure address the aforementioned deficiencies and other deficiencies of conventional cache coherence systems by providing aperture-specific cache management operations for cache coherence. To enable memory sharing between the CPU and PPU, the CPU can specify an aperture associated with a shared memory device. An aperture is a portion of address space that is persistently associated with a specific peripheral device or memory unit. An aperture can reach an external device, such as a read-only memory (ROM) or random access memory (RAM) chip, or the internal memory of the CPU itself. For example, memory devices included in a system can be addressed starting from zero. However, because a system has more than one memory device with the same addressing scheme (e.g., system memory, video memory, etc.), the system may have ambiguous addressing. To address this issue, the system can specify multiple apertures, each associated with a memory device in the system. Thus, apertures form an address translation layer below the virtual-to-physical mapping level. For example, when a buffer is allocated in system memory for use by the PPU, the buffer can be designated as a system memory aperture. When a buffer is allocated in video memory, the buffer can be designated as a video memory aperture.

[0019] In at least one embodiment, the system may maintain a logical distinction between non-uniform system memory aperture buffers and consistent system memory aperture buffers within the same system memory. Buffers designated as non-uniform system memory aperture buffers may include system memory buffers that the system expects only the PPU to access, such as game buffers, textures, compressible surfaces, etc. Buffers designated as consistent system memory aperture buffers may include system memory buffers associated with workload sharing dispatched to the PPU, which the system expects the CPU to access. When creating a system memory buffer, the system memory buffer may be designated as a non-uniform aperture buffer or a consistent aperture buffer. For example, if the buffer is frequently used for communication between the CPU and the PPU, the buffer may be designated as a consistent system memory aperture buffer. If the buffer is expected to be used by the PPU rather than the CPU, the buffer may be designated as a non-uniform system memory aperture buffer.

[0020] In at least one embodiment, cache coherence can be managed based on aperture designation. Buffers designated as coherent system memory aperture buffers can be cached coherently using hardware-managed coherence techniques. The coherence of buffers designated as non-coherent system memory aperture buffers can be managed using aperture-specific cache operations. In at least one embodiment, the coherence of buffers designated as non-coherent memory aperture buffers can be managed using explicit cache flushes and invalidations. For example, before control of a non-coherent memory buffer is transferred from the PPU back to the CPU, cache operations can be issued to flush all cache lines designated as non-coherent system memory cache lines and / or invalidate all cache lines designated as non-coherent system memory cache lines. The PPU can compare the aperture identifier in the aperture field of each cache line in its cache and invalidate cache lines designated as non-coherent system memory aperture cache lines. Consequently, non-coherent system memory cache lines in one or more caches associated with the PPU can be flushed, written back to system memory, and invalidated. To invalidate a cache line, the PPU can write the associated data back to the CPU cache, mark the cache line as invalid, and discard the cache line. To flush a cache line, the PPU can write the associated data back to the CPU cache and keep the cache line in a clean state for future reuse. It will be understood that aperture-specific invalidate and flush cache operations are used here as an example, not a limitation, and it is noted that other cache operations can target aperture-specific cache lines.

[0021] Advantages of the techniques disclosed herein include, but are not limited to, reducing destructive interference between coherent system memory cache lines and non-coherent system memory cache lines. Specifically, by targeting specific aperture cache lines, the latency associated with invalidating PPU cache lines can be reduced because fewer cache lines are targeted. Furthermore, coherent system memory aperture cache lines that are performance-critical and not associated with applications that issue non-coherent system memory aperture cache invalidations can be maintained in the cache and managed through hardware coherency techniques.

[0022] Figure 1 1 is a block diagram illustrating a computer system 100 configured to implement one or more aspects of the present disclosure. System 100 includes system memory 102 and a central processing unit (CPU) 104, which can communicate via an interconnection path (e.g., a bus, a dedicated memory bridge, or other communication path). A parallel processing unit (PPU) 120 is operatively coupled to CPU 104 via a communication path (e.g., a system bus, a peripheral component interconnect Express (PCIe), a northbridge / southbridge architecture, or other communication path). In at least one embodiment, PPU 120 can be directly integrated onto the CPU 104 die and can handle communications internally within CPU 104. In yet another embodiment, PPU 120 can be integrated with one or more other system elements (e.g., CPU 104 and system memory 102) to form a system on a chip (SoC). In such embodiments, the CPU 104, PPU 120, and other components of the system 100 may communicate using an architecture-specific general-purpose interface, such as an Advanced Reduced Instruction Set Computer (RISC) Machine (ARM) Common Hardware Interface (CHI) or a dedicated chip-to-chip (C2C) interface.

[0023] In at least one embodiment, PPU 120 can incorporate circuitry optimized for graphics and video processing, including, for example, video output circuitry, and constitute a graphics processing unit (GPU). In another embodiment, PPU 120 incorporates circuitry optimized for general-purpose processing while retaining the underlying compute architecture. System 100 also includes video memory (VMEM) 130, which PPU 120 can use to store necessary data, such as textures, frame buffers, shaders, and other graphics elements. In at least one embodiment, VMEM 130 can include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as video random access memory (VRAM) or synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) SGRAM. In at least one embodiment, VMEM 130 can include one or more memory stacks, such as multiple DRAM dies stacked vertically to form high-bandwidth memory (HBM). It will be appreciated that the specific implementation of VMEM 130 can vary and can be selected from a number of available designs.

[0024] The PPU 120 may include one or more multiprocessors 124A to 124N (collectively referred to herein as multiprocessors 124) and a cache 122. For example, each multiprocessor 124 may be a streaming multiprocessor (SM), a compute unit (CU), a multi-integrated core (MIC), etc. Each multiprocessor 124 may be responsible for executing parallel processing tasks that involve performing the same operation on multiple data concurrently. Each multiprocessor 124 may execute a certain number of threads simultaneously, so that the PPU 120 as a whole may execute a large number of threads concurrently across all multiprocessors 124. Each multiprocessor 124 may include an L1 cache (not shown), or use a corresponding L1 cache external to the multiprocessor 124, for performing load and store operations. Each multiprocessor 124 may access a cache 122 shared between all multiprocessors 124 and may be used to transfer data between threads. The cache 122 may be arranged along any level of the cache hierarchy (e.g., L1, L2, L3, etc.). In at least one embodiment, cache 122 may be a shared memory (e.g., local memory) that may act as a programmable cache shared between threads. Each multiprocessor 124 may also have access to a global memory, which may include, for example, system memory 102 and / or VMEM 130.

[0025] The CPU 104 may include one or more processing cores 106A through 106N (collectively referred to herein as "processing cores 106") and a cache 110. Each processing core 106 may be a separate processing unit within the CPU 104 that independently executes instructions. Each processing core 106 may be a complete processing unit, having its own arithmetic logic unit (ALU), control unit, registers, and other components required to execute program instructions. Each processing core 106 may have a dedicated cache (e.g., L1, L2, etc.) and may further access the cache 110. In at least one embodiment, the cache 110 may be a shared cache, such as an L3 cache, an L4 cache, a system-level cache (SLC), etc. Each processing core 160 may also have access to a global memory, such as the system memory 102.

[0026] It will be appreciated that the system 100 shown herein is illustrative and is subject to variation and modification. The connection topology, number of CPUs 104, number of PPUs 120, number of processing cores 106 within a CPU 104, and number of multiprocessors 124 within a PPU 120 may be modified as desired. Furthermore, the specific components shown herein are not intended to be exhaustive; for example, any number of add-in cards, peripherals, switches, network adapters, etc. may be supported but are not described herein.

[0027] In at least one embodiment, PPU 120 is a graphics processor having a rendering pipeline that can be configured to perform various tasks related to pixel data generated from graphics data provided by CPU 104 and / or system memory. In operation, CPU 104 is the central processing unit of computer system 100, controlling and coordinating the operations of other system components. Specifically, CPU 104 can issue commands that control the operation of PPU 120.

[0028] In at least one embodiment, the CPU 104 and the PPU 120 may be tightly coupled such that the CPU 104 and the PPU 120 may share system memory 102 and VMEM 130. The system 100 may include a high-bandwidth interconnect, such as a high-speed bus or an on-chip communication channel, such as the ARM CHI. This design is intended to enable efficient collaboration between the CPU 104 and the PPU 120 to perform parallel processing tasks, such as graphics rendering or general computing.

[0029] In some cases, the CPU may offload certain computing tasks from the CPU to the PPU, for example, to take advantage of the parallel processing capabilities of the GPU, such as graphics rendering, simulation, machine learning, etc. In response to identifying a particular task within a program suitable for execution on the PPU, the CPU may share the relevant data stored in system memory 102 with the PPU 120. For example, the CPU 104 may set aside a block of memory (e.g., a buffer) in the system memory 102 for use by the PPU 120.

[0030] In at least one embodiment, address translation services (ATS) can allow the CPU 104 and PPU 120 to share one or more per-process page tables, which enables both CPU 104 threads and PPU 120 threads to access system-allocated memory residing in physical memory (e.g., system memory 102 or VMEM 130). For example, chip-to-chip (C2C) hardware coherency can enable the PPU 120 to cache system memory 102 buffers within cache 122 at cache line granularity without requiring page migration between system memory 102 and VMEM 130. Hardware coherency can simplify and accelerate CPU-GPU communication for closely shared data between the CPU 104 and PPU 120 (referred to herein as "coherent data" or "coherent buffers"). In at least one embodiment, the system 100 can include a hardware cache hardware interface protocol that uses a directory-based system to achieve coherency between the CPU 104 cache and the PPU 120 cache. For example, a directory-based protocol may include several states for each cache line, such as modified (M), exclusive (E), shared (S), and invalid (I). However, caching certain system memory buffers (referred to herein as "non-coherent data" or "non-coherent buffers") that are accessed only by the PPU 120 in a coherent manner may incur unnecessary overhead associated with hardware coherency. Therefore, the aperture manager 112 of the CPU 104 and / or the aperture manager 121 of the PPU 120 may issue aperture-specific cache operations to manage the sharing of non-coherent system memory buffers.

[0031] Aperture managers 112 and 121 may be software components that manage the software sequence for communication between CPU 104 and PPU 120. This software sequence may include application programming interfaces (APIs), device initialization, memory allocation, data transfer between CPU 104 and PPU 120, kernel compilation and execution, CPU-GPU coherence, and the like. Specifically, aperture manager 112 may enable sharing of non-coherent system memory buffers between CPU 104 and PPU 120. To enable memory sharing, aperture manager 112 may specify an aperture associated with a shared memory device. An aperture is a portion of an address space that is persistently associated with a specific peripheral device or memory unit. An aperture can reach external devices, such as read-only memory (ROM) or random access memory (RAM) chips, or the internal memory of CPU 104 itself. For example, memory devices included in system 100 may be addressable starting from zero. However, because system 100 has more than one memory device (e.g., system memory 102, VMEM 130, etc.), system 100 may have ambiguous addressing. To address this issue, aperture manager 112 can specify one of multiple apertures when allocating a buffer from a memory device. For example, when allocating a buffer in system memory 102, aperture manager 112 can specify the buffer as a system memory aperture. When allocating a buffer in VMEM 130, aperture manager 112 can specify the buffer as a video memory aperture.

[0032] In at least one embodiment, the aperture manager 112 can maintain a distinction between non-uniform system memory apertures and consistent system memory apertures. Buffers designated as non-uniform system memory apertures can include system memory buffers accessible only to the PPU 120, such as game buffers, textures, compressible surfaces, and the like. Buffers designated as consistent system memory aperture buffers can include system memory buffers associated with workload sharing assigned to the PPU 120. When creating system memory buffers, the aperture manager 112 can designate the buffer as either a non-uniform system memory aperture or a consistent system memory aperture. For example, the aperture manager 112 can designate a buffer frequently used for communication between the CPU 104 and the PPU 120 as a consistent system memory aperture buffer. The aperture manager 112 can designate a buffer containing data used by the PPU 120, rather than the CPU 104, as a non-uniform system memory aperture buffer.

[0033] In at least one embodiment, cache coherence can be managed based on aperture designation. Buffers designated as coherent system memory apertures can be cached coherently in cache 122 according to the hardware-managed coherence techniques described above. Aperture-specific cache operations can be used to manage the coherence of buffers designated as non-coherent system memory aperture buffers. In at least one embodiment, the coherence of buffers designated as non-coherent memory aperture buffers can be maintained by aperture manager 121 of PPU 120 through explicit cache flushes and invalidations. For example, before control of a non-coherent memory buffer is transferred from PPU 120 back to CPU 104, aperture manager 121 can issue cache management operations to cache 122 to flush and / or invalidate all cache lines associated with the non-coherent system memory aperture. In response to receiving such cache operations, PPU 120 can compare the fields corresponding to the aperture identifier in each cache line of cache 122 and selectively flush and / or invalidate cache lines associated with the non-coherent system memory aperture. Therefore, the non-coherent system memory cache line in cache 122 may be evicted from cache, written back to system memory 102 and / or one or more CPU 104 caches, and invalidated.

[0034] Figure 2 FIG2 is a diagram illustrating an aperture-specific cache coherence system according to at least one embodiment of the present disclosure. FIG200 may include similar elements shown in computing system 100, such as those shown in FIG200 . Figure 1 It can be noted that Figure 1 Elements can be used here to help describe Figure 2 For the purpose of illustration and not limitation, Figure 2 The operations described are described as being performed serially. Although shown in a particular order or sequence, the order of operations may be modified unless otherwise noted. Therefore, the illustrated embodiments should be understood as examples only, and the operations described may be performed in a different order, and some operations may be performed in parallel. In addition, one or more operations may be omitted in some embodiments. Therefore, not all of the operations described are required in every embodiment, and other process flows are possible. In some embodiments, the same, different, fewer, or more operations may be performed. Schematic 200 illustrates a technique for utilizing memory aperture buffer designation as part of a cache coherence technique. Schematic 200 illustrates a CPU 104 including a cache 122 and an aperture manager 112, system memory associated with the CPU 104, a PPU 120 including a cache 122, an aperture manager 121, and a VMEM 130 associated with the PPU 120.

[0035] Cache 122 can be configured to cache data stored in system memory 102 and / or VMEM 130. In at least one embodiment, CPU 104 can allocate a buffer in system memory 102 to be processed by aperture manager 112. The size of the allocated system memory buffer can depend on the amount of data to be processed by PPU 120. CPU 104 can fill the system memory buffer with the necessary data and transfer the buffer from CPU 104 to PPU 120. In at least one embodiment, aperture manager 112 can specify an aperture associated with the system memory before transferring the system memory buffer to PPU 120. In at least one embodiment, aperture manager 112 can designate the system memory buffer as a non-coherent system memory buffer or a coherent system memory buffer. In at least one embodiment, aperture manager 112 can designate a buffer containing data primarily intended to be accessed and used by PPU 120 as a non-coherent system memory aperture buffer. For example, aperture manager 112 can designate a buffer containing texture data, vertex data, shader programs, compressible surfaces, frame buffer data, game buffers, GPU buffers, etc. as a non-coherent system memory aperture buffer. In at least one embodiment, aperture manager 112 may designate as a coherent system memory aperture buffer a buffer containing data that is intended to be closely shared between PPU 120 and CPU 104 (e.g., a vertex buffer). For example, if a system memory buffer is frequently used for communication between the CPU and PPU, aperture manager 112 may designate the buffer as a coherent system memory aperture buffer.

[0036] When the PPU 120 writes data to the cache 122, the cache 122 may be updated based on the aperture associated with the data as specified by the aperture manager 112. In an illustrative example, the cache 122 may include a plurality of cache lines 208A through 208F (collectively referred to herein as "cache lines 208"), where each cache line 208 includes at least an aperture field 202 and a data field 206. The aperture field 202 may include one or more bits that indicate the aperture associated with the data 206. For example, "01" may indicate that the cached data is associated with a non-coherent system memory aperture buffer, "10" may indicate that the cached data is associated with a coherent system memory aperture buffer, and "11" may indicate that the cached data is associated with a video memory aperture buffer. As such, data stored at cache line 208A, cache line 208C, and cache line 208D may be associated with a non-coherent system memory aperture buffer; data stored at cache line 208B may be associated with a coherent system memory aperture buffer; and data stored at cache line 208F may be associated with a video memory aperture buffer.

[0037] After the PPU 120 completes its work on the allocated buffer, the aperture manager 121 may issue an aperture-specific cache line invalidation operation to the cache 122. For example, the aperture manager 121 may issue a cache operation to invalidate cache lines in the cache 122 associated with the non-coherent system memory aperture. In response to receiving a cache operation to invalidate cache lines of the non-coherent system memory aperture, a cache controller (not shown) of the cache 122 may sequentially access each cache line 208 to determine the identifier stored at the respective aperture field 202 of the cache line 208. The cache controller may also cause the cache lines storing data associated with the non-coherent system memory aperture to be flushed and invalidated. For example, in response to receiving a cache operation to invalidate cache lines associated with the non-coherent memory aperture, cache lines 208A, 208C, and 208D may be flushed and invalidated.

[0038] To flush a cache line associated with a non-uniform memory aperture, data may be written back to one or more caches associated with the system 102 and / or CPU 104, such as cache 122. In at least one embodiment, cache 122 is a write-back cache such that when a cache line is flushed and invalidated due to a cache operation that invalidates a non-uniform system memory cache line, the modified data is written back to the system memory 102. In at least one embodiment, to invalidate a non-uniform system memory aperture cache line, the state of the cache line may be updated to reflect an invalid state. In at least one embodiment, cache 122 may operate according to a Modified, Exclusive, Shared, Invalid (MESI) coherence protocol such that one or more bits of a non-uniform memory aperture cache line are updated to reflect an invalid state. For example, valid bits associated with cache lines 208A, 208C, and 208D may be updated to reflect an invalid state.

[0039] In at least one embodiment, PPU 120 can ensure that a coherent system memory cache line remains in cache 122 and is not invalidated by an operation that invalidates a non-coherent system memory aperture cache line. For example, in response to aperture manager 121 issuing a cache operation to invalidate a non-coherent system memory cache line, a cache controller of cache 122 can ensure that cache line 208B remains in cache 122 and is not invalidated. By using software-managed invalidation of non-coherent system memory aperture cache lines, coherent system memory aperture cache lines can be managed using a hardware coherence protocol without causing destructive interference between coherent system memory cache lines and non-coherent system memory cache lines in cache 122.

[0040] In at least one embodiment, cache line 208 may include one or more cache lines that are not in the cache. Figure 2. For example, cache lines 208 may each include a tag, index, offset, block offset, etc. In at least one embodiment, cache lines 208 may include one or more additional fields to implement software-managed coherence at different levels of abstraction. For example, cache line 208 may include an additional field indicating the process associated with the data. CPU 104 may be a multi-core system with multiple independently running processes. A first process executing on CPU 104 may allocate a first non-coherent data buffer for processing by PPU 120. A second process executing on CPU 104 may allocate a second non-coherent data buffer for processing by PPU 120. Cache 122 may write a portion of the first non-coherent data buffer to cache line 208A and update fields of cache line 208A with an identifier for the first process. Cache 122 may write a portion of the second non-coherent data buffer to cache lines 208C and 208D and update corresponding fields with an identifier for the second process. When CPU 104 is ready to resume control of the first non-coherent buffer, aperture manager 112 may initiate a cache operation to invalidate all non-coherent cache lines associated with the first process. As a result, cache line 208A may be flushed and invalidated, while cache lines 208C and 208D remain valid and in cache 122.

[0041] In at least one embodiment, aperture manager 121 may issue non-uniform aperture cache line invalidations after the associated tasks accelerated by PPU 120 are completed. For example, an application processing on CPU 104 may request (e.g., using the aperture manager, an API, etc.) a non-uniform system memory 102 buffer for the GPU. An operating system (OS) running on CPU 104 may allocate the non-uniform system memory buffer and notify PPU 120. For example, a device driver associated with PPU 120 may interface between the OS and PPU 120 hardware so that the device driver is notified when the OS allocates memory for use by PPU 120. Once the non-uniform system memory buffer is allocated and transferred to PPU 120, PPU 120 may launch a kernel or task to perform computations on PPU 120 using its parallel processing capabilities. After executing the assigned kernel / task, PPU 120 may notify CPU 104 (e.g., using an interrupt, an event, inter-process communication, etc.) that the assigned task has completed. Upon receiving an indication that the PPU 120 has completed the assigned task, the PPU 120 may issue a cache line operation to invalidate the non-coherent system memory aperture cache line.

[0042] Figure 3A flowchart of a method 300 for aperture-specific cache operations according to at least one embodiment of the present disclosure is shown. Although method 300 is described in the context of a processing unit, method 300 may also be performed by a program, custom circuitry, or a combination of custom circuitry and a program. For example, method 300 may be performed by a GPU (graphics processing unit), a CPU (central processing unit), or any processor capable of issuing or receiving aperture-specific cache operations. Furthermore, one of ordinary skill in the art will understand that any system that performs method 300 is within the scope and spirit of embodiments of the present invention.

[0043] More illustrative information will now be provided regarding various optional architectures and features that can implement the above framework, depending on the user's desires. It should be strongly noted that the following information is for illustrative purposes only and should not be construed as limiting in any way. Any of the following features may be optionally combined without excluding the other features described.

[0044] Although shown in a particular order or sequence, the order of operations may be modified unless otherwise indicated. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated operations may be performed in a different order, and some operations may be performed in parallel. In addition, one or more operations may be omitted in various embodiments. Therefore, not all operations are required in every embodiment.

[0045] At operation 302 of method 300, processing circuitry may receive a cache operation at a parallel processing unit (PPU) including a first cache to modify a cache line of the first cache associated with a first aperture of a system memory of the processing device. In at least one embodiment, the processing device may be Figure 1 CPU 104, the first cache may be Figure 1 The cache 122, and the PPU can be Figure 1 PPU 120. In at least one embodiment, the PPU and the processing device are interconnected via an interface using the Common Hardware Interface (CHI) protocol.

[0046] At operation 304 of method 300, the processing circuitry may identify a first cache line subset of a first cache, wherein the first cache line subset is associated with a first aperture of the system memory. In an illustrative example, the first cache line subset may include Figure 2 Cache lines 208A, 208C, and 208D of the system memory. In at least one embodiment, the first aperture is a non-uniform aperture of the system memory.

[0047] At operation 306 of method 300, the processing circuitry may identify a second cache line subset of the first cache, wherein the second cache line subset is associated with a second aperture of the system memory. In an illustrative example, the second cache line subset may include Figure 2In at least one embodiment, the second aperture is a uniform aperture of the system memory. In at least one embodiment, the first subset of cache lines and the second subset of cache lines are identified based on identifiers associated with the non-uniform aperture and the uniform aperture, respectively.

[0048] At operation 308 of method 300, the processing circuitry may modify the first cache line subset as specified by the cache operation. For example, the processing circuitry may modify Figure 2 In at least one embodiment, the cache operation is an invalidation operation, and to modify the first cache line subset, the processing circuit system writes the data stored in the first cache line subset back to the second cache of the processing device and invalidates the first cache line subset. In at least one embodiment, the second cache may be Figure 1 110. In at least one embodiment, the cache operation is a refresh operation, and to modify the first cache line subset, the processing circuitry writes data stored in the first cache line subset back to the second cache of the processing device and maintains a clean state of the first cache line subset.

[0049] In at least one embodiment, the processing circuitry may further cause the second subset of cache lines to be maintained in the second cache. In at least one embodiment, the coherence of the second subset of cache lines is managed by hardware associated with the processing device. For example, coherent cache lines associated with the coherent system memory aperture may be managed using a directory-based approach based on a hardware interface.

[0050] In at least one embodiment, the first cache line subset may be further distinguished and invalidated based on a process identifier indicating one of a plurality of processes associated with the processing device. For example, the processing circuitry may receive an operation to invalidate non-uniform aperture cache lines of a second cache associated with the first process. The process logic may further invalidate one or more cache lines in the first cache line subset associated with the first process.

[0051] Parallel processing architecture

[0052] Figure 4A parallel processing unit (PPU) 400 is shown according to one embodiment. In one embodiment, the PPU 400 is a multi-threaded processor implemented on one or more integrated circuit devices. The PPU 400 is a latency-hiding architecture designed for processing many threads in parallel. A thread (e.g., an execution thread) is an instance of an instruction set configured to be executed by the PPU 400. In one embodiment, the PPU 400 is a graphics processing unit (GPU) that is configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device (such as a liquid crystal display (LCD) device). In other embodiments, the PPU 400 can be used to perform general-purpose computations. Although an exemplary parallel processor is provided herein for illustrative purposes, it should be specifically noted that the processor is described for illustrative purposes only and any processor may be used in addition to and / or in place of the processor.

[0053] One or more PPUs 400 can be configured to accelerate thousands of high-performance computing (HPC), data center, and machine learning applications. PPUs 400 can be configured to accelerate numerous deep learning systems and applications, including autonomous vehicle platforms, deep learning, high-precision speech, image, and text recognition systems, intelligent video analysis, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations, among others.

[0054] like Figure 4 As shown, the PPU 400 includes an input / output (I / O) unit 405, a front-end unit 415, a scheduler unit 420, a work distribution unit 425, a hub 430, a crossbar switch (Xbar) 470, one or more processing clusters 450 (e.g., general processing clusters GPCs), and one or more partition units 480. The PPU 400 can be connected to a host processor or other PPUs 400 via one or more high-speed NVLink 410 interconnects. The PPU 400 can be connected to a host processor or other peripheral devices via interconnect 402. The PPU 400 can also be connected to a local memory including multiple memory devices 404. In one embodiment, the local memory can include multiple dynamic random access memory (DRAM) devices. The DRAM devices can be configured as a high-bandwidth memory (HBM) subsystem, in which multiple DRAM dies are stacked within each device.

[0055] The NVLink 410 interconnect enables the system to scale and include one or more PPUs 400 in conjunction with one or more CPUs, supporting cache coherency between the PPU 400 and the CPU, and CPU mastering. Data and / or commands can be sent by the NVLink 410 through the hub 430 to or from other units of the PPU 400, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). Figure 6B NVLink 410 is described in more detail.

[0056] I / O unit 405 is configured to send and receive communications (e.g., commands, data, etc.) from a host processor (not shown) via interconnect 402. I / O unit 405 can communicate with the host processor directly via interconnect 402, or through one or more intermediate devices (such as a memory bridge). In one embodiment, I / O unit 405 can communicate with one or more other processors (e.g., one or more PPUs 400) via interconnect 402. In one embodiment, I / O unit 405 implements a Peripheral Component Interconnect Express (PCIe) interface for communicating over a PCIe bus, and interconnect 402 is a PCIe bus. In alternative embodiments, I / O unit 405 can implement other types of known interfaces for communicating with external devices.

[0057] I / O unit 405 decodes data packets received via interconnect 402. In one embodiment, the data packets represent commands configured to cause PPU 400 to perform various operations. I / O unit 405 sends the decoded commands to various other units of PPU 400 as specified by the commands. For example, some commands may be sent to front-end unit 415. Other commands may be sent to hub 430 or other units of PPU 400, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In other words, I / O unit 405 is configured to route communications between and among the various logical units of PPU 400.

[0058] In one embodiment, a program executed by a host processor encodes a command stream in a buffer that provides a workload to the PPU 400 for processing. The workload may include many instructions and data to be processed by those instructions. A buffer is an area of ​​memory that is accessible (e.g., read / write) by both the host processor and the PPU 400. For example, the I / O unit 405 may be configured to access a buffer in system memory connected to the interconnect 402 via a memory request transmitted over the interconnect 402. In one embodiment, the host processor writes a command stream into the buffer and then sends a pointer to the start of the command stream to the PPU 400. The front end unit 415 receives pointers to one or more command streams. The front end unit 415 manages the one or more streams, reads commands from the streams, and forwards the commands to the various units of the PPU 400.

[0059] Front-end unit 415 is coupled to scheduler unit 420, which configures various processing clusters 450 to process tasks defined by one or more flows. Scheduler unit 420 is configured to track status information associated with the various tasks managed by scheduler unit 420. The status may indicate which processing cluster 450 a task is assigned to, whether the task is active or inactive, the priority associated with the task, and the like. Scheduler unit 420 manages the execution of multiple tasks on one or more processing clusters 450.

[0060] Scheduler unit 420 is coupled to work distribution unit 425, which is configured to dispatch tasks for execution on processing clusters 450. Work distribution unit 425 can track a number of scheduled tasks received from scheduler unit 420. In one embodiment, work distribution unit 425 manages a pending task pool and an active task pool for each processing cluster 450. The pending task pool can include a number of time slots (e.g., 32 time slots) containing tasks assigned to be processed by a particular processing cluster 450. The active task pool can include a number of time slots (e.g., 4 time slots) for tasks being actively processed by processing cluster 450. When a processing cluster 450 completes execution of a task, the task is evicted from the active task pool of processing cluster 450, and one of the other tasks from the pending task pool is selected and scheduled for execution on processing cluster 450. If an active task on processing cluster 450 has become idle, such as while waiting for a data dependency to be resolved, then the active task may be evicted from processing cluster 450 and returned to the pending task pool, while another task in the pending task pool is selected and scheduled for execution on processing cluster 450 .

[0061] Work distribution unit 425 communicates with one or more processing clusters 450 via XBar (crossbar) 570. XBar 470 is an interconnect network that couples many units of PPU 400 to other units of PPU 400. For example, XBar 470 can be configured to couple work distribution unit 425 to a specific processing cluster 450. Although not explicitly shown, one or more other units of PPU 400 can also be connected to XBar 470 via hub 430.

[0062] Tasks are managed by the scheduler unit 420 and dispatched to the processing clusters 450 by the work distribution unit 425. The processing clusters 450 are configured to process tasks and generate results. The results can be consumed by other tasks within the processing cluster 450, routed to a different processing cluster 450 via the XBar 470, or stored in the memory 404. The results can be written to the memory 404 via the partition unit 480, which implements a memory interface for reading data from and writing data to the memory 404. The results can be sent to another PPU 404 or CPU via the NVLink 410. In one embodiment, the PPU 400 includes a number U of partition units 480, which is equal to the number of independent and different memory devices 404 coupled to the PPU 400. Figure 5B The partition unit 480 is described in more detail.

[0063] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API) that enables one or more applications to be executed on the host processor to schedule operations for execution on the PPU 400. In one embodiment, multiple computing applications are executed simultaneously by the PPU 400, and the PPU 400 provides isolation, quality of service (QoS), and independent address spaces for the multiple computing applications. The application can generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks to be executed by the PPU 400. The driver kernel outputs the tasks to one or more streams being processed by the PPU 400. Each task can include one or more related groups of threads, referred to herein as warps. In one embodiment, a warp includes 32 related threads that can execute in parallel. Collaborating threads can refer to multiple threads that include instructions to perform tasks and that can exchange data through shared memory. In combination Figure 6A Describes threads and cooperative threads in more detail.

[0064] Figure 5A According to one embodiment, Figure 4 The processing cluster 450 of the PPU 400 is Figure 5AAs shown, each processing cluster 450 includes multiple hardware units for processing tasks. In one embodiment, each processing cluster 450 includes a pipeline manager 510, a pre-raster operation unit (PROP) 515, a raster engine 525, a work distribution crossbar (WDX) 580, a memory management unit (MMU) 590, and one or more data processing clusters (DPCs) 520. It should be understood that Figure 5A The processing cluster 450 may include instead Figure 5A Other hardware units or other than the units shown in Figure 5A Other hardware units besides those shown in .

[0065] In one embodiment, the operation of processing cluster 450 is controlled by pipeline manager 510. Pipeline manager 510 manages the configuration of one or more DPCs 520 for processing tasks assigned to processing cluster 450. In one embodiment, pipeline manager 510 may configure at least one of one or more DPCs 520 to implement at least a portion of a graphics rendering pipeline. For example, DPC 520 may be configured to execute vertex shading programs on programmable multiprocessor 540. Multiprocessor 540 may typically include a streaming multiprocessor, a compute unit, a number of integrated cores, and the like. Pipeline manager 510 may also be configured to route packets received from work distribution unit 525 to appropriate logic units within processing cluster 450. For example, some packets may be routed to fixed-function hardware units within PROP 515 and / or raster engine 525, while other packets may be routed to DPC 520 for processing by primitive engine 535 or multiprocessor 540. In one embodiment, pipeline manager 510 may configure at least one of one or more DPCs 520 to implement a neural network model and / or a compute pipeline.

[0066] PROP unit 515 is configured to route data generated by raster engine 525 and DPC 520 to the raster operations (ROP) unit, in conjunction with Figure 5B The PROP unit 515 may also be configured to perform optimization of color blending, organize pixel data, perform address translation, and the like.

[0067] The raster engine 525 includes several fixed-function hardware units configured to perform various raster operations. In one embodiment, the raster engine 525 comprises a setup engine, a coarse raster engine, a culling engine, a clipping engine, a fine raster engine, and a tile aggregation engine. The setup engine receives the transformed vertices and generates plane equations associated with the geometric primitives defined by the vertices. The plane equations are sent to the coarse raster engine to generate coverage information for the primitives (e.g., the x, y coverage mask of the tile). The output of the coarse raster engine is sent to the culling engine, where fragments associated with primitives that fail the z-test are culled, and to the clipping engine, where fragments outside the viewing frustum are clipped. Those fragments remaining after clipping and culling can be passed to the fine raster engine to generate pixel fragment attributes based on the plane equations generated by the setup engine. The raster engine 525 output includes, for example, fragments to be processed by the fragment shader implemented in the DPC 520.

[0068] Each DPC 520 included in the processing cluster 450 includes an M-pipe controller (MPC) 530, a primitive engine 535, and one or more multiprocessors 540. The MPC 530 controls the operation of the DPC 520 and routes packets received from the pipeline manager 510 to appropriate units within the DPC 520. For example, packets associated with vertices may be routed to the primitive engine 535, which is configured to retrieve vertex attributes associated with the vertices from the memory 404. Conversely, packets associated with shading programs may be sent to the multiprocessor 540.

[0069] In some embodiments, the multiprocessor 540 includes a programmable multiprocessor, such as a programmable streaming multiprocessor, configured to process tasks represented by multiple threads. Each multiprocessor 640 is multithreaded and configured to simultaneously execute multiple threads (e.g., 32 threads) from a particular thread group. In one embodiment, the multiprocessor 540 implements a SIMD (single instruction, multiple data) architecture, in which each thread in a thread group (e.g., a warp) is configured to process a different data set based on the same instruction set. All threads in a thread group execute the same instructions. In another embodiment, the multiprocessor 540 implements a SIMT (single instruction, multiple thread) architecture, in which each thread in a thread group is configured to process a different data set based on the same instruction set, but in which individual threads in a thread group are allowed to diverge during execution. In one embodiment, a program counter, call stack, and execution state are maintained for each warp, enabling concurrency between warps and serial execution within warps when threads within the warp diverge. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, thereby achieving equal concurrency between all threads within and between warps. When execution state is maintained for each individual thread, threads executing the same instruction can be converged and executed in parallel for maximum efficiency. Figure 6A Multiprocessor 540 is described in more detail.

[0070] The MMU 590 provides an interface between the processing cluster 450 and the partition unit 480. The MMU 590 can provide virtual address to physical address translation, memory protection, and arbitration of memory requests. In one embodiment, the MMU 590 provides one or more translation lookaside buffers (TLBs) for performing translations from virtual addresses to physical addresses in the memory 404.

[0071] Figure 5B According to one embodiment, Figure 4 The memory partition unit 480 of the PPU 400. Figure 5BAs shown, the memory partition unit 480 includes a raster operations (ROP) unit 550, a level 2 (L2) cache 560, and a memory interface 570. The memory interface 570 is coupled to the memory 404. The memory interface 570 can implement a 32-, 64-, 128-, or 1024-bit data bus for high-speed data transfer. In one embodiment, the PPU 400 incorporates U memory interfaces 570, one for each pair of partition units 480, where each pair of partition units 580 is connected to a corresponding memory device 404. For example, the PPU 400 can be connected to up to Y memory devices 404, such as a high-bandwidth memory stack or graphics double data rate version 5 synchronous dynamic random access memory or other types of persistent memory.

[0072] In one embodiment, memory interface 570 implements an HBM2 memory interface, and Y is equal to half of U. In one embodiment, the HBM2 memory stack is located on the same physical package as PPU 400, providing significant power and area savings compared to conventional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory dies and Y is equal to 4, where the HBM2 stack includes two 128-bit channels per die, for a total of 8 channels and a data bus width of 1024 bits.

[0073] In one embodiment, memory 404 supports single-error correction, double-error detection (SECDED) error correction code (ECC) to protect data. ECC provides increased reliability for computing applications sensitive to data corruption. Reliability is particularly important in large cluster computing environments where PPU 400 processes very large data sets and / or long-running applications.

[0074] In one embodiment, the PPU 400 implements a multi-level memory hierarchy. In one embodiment, the memory partitioning unit 480 supports unified memory to provide a single, unified virtual address space for the CPU and PPU 400 memory, enabling data sharing between virtual memory systems. In one embodiment, the frequency of PPU 400 accesses to memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 400 that accesses them more frequently. In one embodiment, NVLink 410 supports address translation services that allow the PPU 400 to directly access the CPU's page tables and provide full access to the CPU's memory by the PPU 400.

[0075] In one embodiment, the copy engine transfers data between multiple PPUs 400 or between a PPU 400 and a CPU. The copy engine can generate a page fault for an address that is not mapped to a page table. The memory partition unit 480 can then service the page fault, map the address into a page table, and then the copy engine can perform the transfer. In conventional systems, multiple copy engines operate on fixed memory (e.g., non-pageable) between multiple processors, which significantly reduces the available memory. Due to hardware page faults, addresses can be passed to the copy engine without worrying about whether the memory page is resident, and the copy process is transparent.

[0076] Data from memory 404 or other system memory can be retrieved by memory partition unit 480 and stored in L2 cache 560, which is located on-chip and shared between various processing clusters 450. As shown, each memory partition unit 480 includes a portion of L2 cache 560 associated with the corresponding memory device 404. Lower-level caches can then be implemented in multiple units within a processing cluster 450. For example, each multiprocessor 540 can implement a level 1 (L1) cache. The L1 cache is a dedicated memory dedicated to a particular multiprocessor 540. Data from the L2 cache 560 can be retrieved and stored in each L1 cache for processing in the functional units of the multiprocessor 540. The L2 cache 560 is coupled to the memory interface 570 and the XBar 470.

[0077] The ROP unit 550 performs graphics raster operations related to pixel color, such as color compression, pixel blending, etc. The ROP unit 550 also implements depth testing in conjunction with the raster engine 525, receiving the depth of the sampling position associated with the pixel fragment from the culling engine of the raster engine 525. The depth is tested against the corresponding depth in the depth buffer of the sampling position associated with the fragment. If the fragment passes the depth test for the sampling position, the ROP unit 550 updates the depth buffer and transmits the result of the depth test to the raster engine 525. It will be appreciated that the number of partition units 480 may be different than the number of processing clusters 450, and therefore, each ROP unit 550 may be coupled to each processing cluster 450. The ROP unit 550 tracks packets received from different processing clusters 450 and determines to which processing cluster 450 the result generated by the ROP unit 550 is routed via the Xbar 470. Although the ROP unit 550 is included in Figure 5B 4. In other embodiments, ROP unit 550 may be located within memory partition unit 480, but in other embodiments, ROP unit 550 may be located outside memory partition unit 480. For example, ROP unit 550 may be located within processing cluster 450 or another unit.

[0078] Figure 6AAccording to one embodiment, Figure 5A The multiprocessor 540. Figure 6A As shown, the multiprocessor 540 includes an instruction cache 605, one or more scheduler units 610, a register file 620, one or more processing cores 650, one or more special function units (SFUs) 652, one or more load / store units (LSUs) 654, an interconnection network 680, and a shared memory / L1 cache 670.

[0079] As described above, the work distribution unit 425 schedules tasks for execution on the processing cluster 450 of the PPU 400. Tasks are assigned to specific DPCs 520 within the processing cluster 450, and if the task is associated with a shader program, the task may be assigned to the multiprocessor 540. The scheduler unit 610 receives tasks from the work distribution unit 425 and manages the scheduling of instructions assigned to one or more thread blocks assigned to the multiprocessor 540. The scheduler unit 610 schedules the thread blocks for execution as warps of parallel threads, where each thread block is assigned at least one warp. In one embodiment, each warp executes 32 threads. The scheduler unit 610 can manage multiple different thread blocks, assign warps to different thread blocks, and then dispatch instructions from multiple different cooperative groups to various functional units (e.g., core 650, SFU 652, and LSU 654) during each clock cycle.

[0080] Cooperative Groups is a programming model for organizing groups of communicating threads that allows developers to express the granularity at which threads are communicating, enabling the expression of richer and more efficient decompositions of parallelism. The cooperative launch API supports synchronization between thread blocks to execute parallel algorithms. The conventional programming model provides a single simple structure for synchronizing cooperating threads: a barrier across all threads of a thread block (i.e., the syncthreads() function). However, programmers often want to define thread groups at a granularity smaller than the thread block granularity and synchronize within the defined group, enabling higher performance, design flexibility, and software reuse in the form of a collective group-wide function interface.

[0081] Cooperative Groups enable programmers to explicitly define thread groups at sub-block (e.g., as small as a single thread) and multi-block granularity and perform collective operations, such as synchronization, on threads in a cooperative group. The programming model supports clean composition across software boundaries so that libraries and utility functions can safely synchronize in their local environment without making assumptions about convergence. Cooperative Group primitives enable new patterns of cooperative parallelism, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire grid of thread blocks.

[0082] The dispatch unit 615 is configured to deliver instructions to one or more functional units. In this embodiment, the scheduler unit 610 includes two dispatch units 615, which enable scheduling two different instructions from the same warp during each clock cycle. In alternative embodiments, each scheduler unit 610 may include a single dispatch unit 615 or additional dispatch units 615.

[0083] Each multiprocessor 540 includes a register file 620 that provides a set of registers for the functional units of the multiprocessor 540. In one embodiment, the register file 620 is divided between each functional unit so that each functional unit is allocated a dedicated portion of the register file 620. In another embodiment, the register file 620 is divided between the different warps executed by the multiprocessor 540. The register file 620 provides temporary storage for operands connected to the data paths of the functional units.

[0084] Each multiprocessor 540 includes L processing cores 650. In one embodiment, the multiprocessor 540 includes a large number (e.g., 128, etc.) of different processing cores 650. Each core 650 may include a fully pipelined, single-precision, double-precision, and / or mixed-precision processing unit, including a floating-point arithmetic logic unit (FLU) and an integer arithmetic logic unit (ALU). In one embodiment, the FLU implements the IEEE 754-2008 standard for floating-point operations. In one embodiment, the cores 650 include 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0085] Tensor cores are configured to perform matrix operations, and in one embodiment, one or more tensor cores are included in core 650. Specifically, the tensor cores are configured to perform deep learning matrix operations, such as convolution operations for neural network training and inference. In one embodiment, each tensor core operates on a 4×4 matrix and performs a matrix multiplication and accumulation operation D=A×B+C, where A, B, C, and D are 4×4 matrices.

[0086] In one embodiment, the matrix multiplication inputs A and B are 16-bit floating point matrices, while the accumulation matrices C and D can be 16-bit floating point or 32-bit floating point matrices. The tensor cores operate on 16-bit floating point input data as well as 32-bit floating point accumulations. The 16-bit floating point multiplication requires 64 operations to produce a full-precision product, which is then accumulated using 32-bit floating point additions with other intermediate products of the 4×4×4 matrix multiplication. In practice, tensor cores are used to perform larger two-dimensional or higher-dimensional matrix operations built from these smaller elements. APIs (such as the CUDA 9 C++ API) expose specialized matrix load, matrix multiplication and accumulation, and matrix store operations to efficiently use tensor cores from CUDA-C++ programs. At the CUDA level, the warp-level interface assumes that the 16×16 size matrix spans all 32 threads of the warp.

[0087] Each multiprocessor 540 also includes M SFUs 652 that perform special functions (e.g., attribute evaluation, reciprocal square root, etc.). In one embodiment, the SFUs 652 may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, the SFUs 652 may include a texture unit configured to perform texture map filtering operations. In one embodiment, the texture unit is configured to load a texture map (e.g., a 2D array of texels) from memory 404 and sample the texture map to generate sampled texture values ​​for use in shader programs executed by the multiprocessor 540. In one embodiment, the texture map is stored in shared memory / L1 cache 570. The texture unit performs texture operations, such as filtering operations using mip maps (e.g., texture maps of different levels of detail). In one embodiment, each SM 440 includes two texture units.

[0088] Each multiprocessor 540 also includes N LSUs 654 that perform load and store operations between the shared memory / L1 cache 670 and the register file 620. Each multiprocessor 540 includes an interconnect network 680 that connects each functional unit to the register file 620 and the LSUs 654 to the register file 620 and the shared memory / L1 cache 670. In one embodiment, the interconnect network 680 is a crossbar switch that can be configured to connect any functional unit to any register in the register file 620 and to connect the LSUs 654 to memory locations in the register file and the shared memory / L1 cache 670.

[0089] The shared memory / L1 cache 670 is an on-chip memory array that allows data storage and communication between the multiprocessor 540 and the primitive engine 535, as well as between threads in the multiprocessor 540. In one embodiment, the shared memory / L1 cache 670 includes 128KB of storage capacity and is in the path from the multiprocessor 540 to the partition unit 480. The shared memory / L1 cache 670 can be used to cache reads and writes. One or more of the shared memory / L1 cache 670, the L2 cache 560, and the memory 404 is a backing store.

[0090] Combining data cache and shared memory functionality into a single memory block provides the best overall performance for both types of memory access. This capacity can be used by programs as a cache that is not using shared memory. For example, if the shared memory is configured to use half of its capacity, texture and load / store operations can use the remaining capacity. Integration within shared memory / L1 cache 670 enables shared memory / L1 cache 670 to function as a high-throughput pipeline for streaming data, while providing high-bandwidth and low-latency access to frequently reused data.

[0091] When configured for general parallel computing, a simpler configuration can be used compared to graphics processing. Specifically, Figure 4 The fixed-function graphics processing unit is shown bypassed, creating a simpler programming model. In a general-purpose parallel computing configuration, the work distribution unit 425 assigns and distributes thread blocks directly to the DPC 520. The threads in the block execute the same program, using unique thread IDs in the computation to ensure that each thread produces a unique result, using the multiprocessor 540 to execute the program and perform the computation, using the shared memory / L1 cache 670 to communicate between the threads, and using the LSU 654 to read and write to global memory through the shared memory / L1 cache 670 and the memory partition unit 480. When configured for general-purpose parallel computing, the multiprocessor 540 can also write commands that the scheduler unit 420 can use to start new work on the DPC 520.

[0092] The PPU 400 may be included in a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, etc. In one embodiment, the PPU 400 is included on a single semiconductor substrate. In another embodiment, the PPU 400 is included on a system-on-chip (SoC) along with one or more other devices (such as an additional PPU 400, a memory 404, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), etc.).

[0093] In one embodiment, PPU 400 may be included on a graphics card that includes one or more memory devices 404. The graphics card may be configured to interface with a PCIe slot on a motherboard of a desktop computer. In another embodiment, PPU 400 may be an integrated graphics processing unit (iGPU) or parallel processor included in a chipset of the motherboard.

[0094] Exemplary Computing System

[0095] Systems with multiple GPUs and CPUs are being used across various industries as developers expose and exploit greater parallelism in applications such as artificial intelligence computing. High-performance GPU-accelerated systems with tens to thousands of computing nodes are deployed in data centers, research institutions, and supercomputers to solve larger problems. As the number of processing devices within high-performance systems increases, communication and data transmission mechanisms need to scale to support this increased bandwidth.

[0096] Figure 6B According to one embodiment, the Figure 4 A conceptual diagram of a processing system 600 implemented with a PPU 400. The exemplary system 665 may be configured to implement Figure 3 The processing system 600 includes a CPU 630, a switch 612, and each of the plurality of PPUs 400 and a corresponding memory 404. The NVLink 410 provides a high-speed communication link between each PPU 400. Figure 6B 402 connections, but the number of connections connected to each PPU 400 and CPU 630 may vary. Switch 612 interfaces between interconnect 402 and CPU 630. PPU 400, memory 404, and NVLink 410 may be located on a single semiconductor platform to form a parallel processing module 625. In one embodiment, switch 612 supports two or more protocols that interface between various different connections and / or links.

[0097] In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between each PPU 400 and CPU 630, and switch 612 interfaces between interconnect 402 and each PPU 400. PPUs 400, memory 404, and interconnect 402 may be located on a single semiconductor platform to form parallel processing module 625. In yet another embodiment (not shown), interconnect 402 provides one or more communication links between each PPU 400 and CPU 630, and switch 612 interfaces between each PPU 400 using NVLink 410 to provide one or more high-speed communication links between the PPUs 400. In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between PPUs 400 and CPU 630 through switch 612. In yet another embodiment (not shown), interconnect 402 provides one or more communication links directly between each PPU 400. One or more NVLink 410 high-speed communication links may be implemented as a physical NVLink interconnect or as an on-chip or on-die interconnect using the same protocol as NVLink 410 .

[0098] In the context of this specification, a single semiconductor platform may refer to a unique, single semiconductor-based integrated circuit fabricated on a die or chip. It should be noted that the term single semiconductor platform may also refer to a multi-chip module with increased connectivity that emulates on-chip operation and is substantially improved by utilizing conventional bus implementations. Of course, various circuits or devices may also be placed separately or in various combinations of semiconductor platforms, depending on the needs of the user. Alternatively, the parallel processing module 625 may be implemented as a circuit board substrate, and each of the PPU 400 and / or memory 404 may be a packaged device. In one embodiment, the CPU 630, switch 612, and parallel processing module 625 are located on a single semiconductor platform.

[0099] In one embodiment, the signaling rate of each NVLink 410 is 20 to 25 Gbit / s, and each PPU 400 includes six NVLink 410 interfaces (e.g., Figure 6B As shown, each PPU 400 includes five NVLink 410 interfaces. Each NVLink 410 provides a data transfer rate of 25 Gbit / s in each direction, with six links providing 500 Gbit / s. When the CPU 630 also includes one or more NVLink 410 interfaces, the NVLink 410 can be used exclusively for Figure 6B PPU to PPU communication shown, or some combination of PPU to PPU and PPU to CPU.

[0100] In one embodiment, NVLink 410 allows direct load / store / atomic access from the CPU 630 to the memory 404 of each PPU 400. In one embodiment, NVLink 410 supports coherency operations, allowing data read from memory 404 to be stored in the cache hierarchy of the CPU 630, reducing cache access latency for the CPU 630. In one embodiment, NVLink 410 includes support for Address Translation Services (ATS), allowing the PPU 400 to directly access page tables within the CPU 630. One or more NVLinks 410 can also be configured to operate in a low-power mode.

[0101] Figure 6C An exemplary system 665 is shown in which various architectures and / or functionalities of various previous embodiments may be implemented. The exemplary system 665 may be configured to implement Figure 3 The method 300 shown in FIG.

[0102] As shown, a system 665 is provided that includes at least one central processing unit 630 connected to a communication bus 675. The communication bus 675 can be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or one or more point-to-point communication protocols. The system 665 also includes a main memory 640. Control logic (software) and data are stored in the main memory 640, which can take the form of random access memory (RAM).

[0103] System 665 also includes an input device 660, a parallel processing system 625, and a display device 645, such as a conventional CRT (cathode ray tube), an LCD (liquid crystal display), an LED (light emitting diode), a plasma display, etc. User input can be received from input device 660 (e.g., a keyboard, a mouse, a touchpad, a microphone, etc.). Each of the aforementioned modules and / or devices can even be located on a single semiconductor platform to form system 665. Alternatively, the modules can be located separately or in various combinations of semiconductor platforms, depending on the user's needs.

[0104] Furthermore, system 665 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, etc.) through network interface 635 for communication purposes.

[0105] System 665 may also include auxiliary storage (not shown). Auxiliary storage 610 includes, for example, a hard drive and / or a removable storage drive, representative of a floppy disk drive, a tape drive, an optical disk drive, a digital versatile disk (DVD) drive, a recording device, or a universal serial bus (USB) flash memory. The removable storage drive reads from and / or writes to a removable storage unit in a well-known manner.

[0106] Computer programs or computer control logic algorithms may be stored in the main memory 640 and / or secondary storage. These computer programs, when executed, enable the system 665 to perform various functions. The memory 640, storage, and / or any other storage are possible examples of computer-readable media.

[0107] The architecture and / or functionality of the various preceding figures can be implemented in the context of a general-purpose computer system, a circuit board system, a game console system dedicated to entertainment purposes, a dedicated system, and / or any other desired system. For example, system 665 can take the form of a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, a mobile telephone device, a television, a workstation, a game console, an embedded system, and / or any other type of logic.

[0108] Although various embodiments have been described above, it should be understood that they are presented by way of example only, not limitation. Therefore, the breadth and scope of a preferred embodiment should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

[0109] Graphics processing pipeline

[0110] In one embodiment, the PPU 400 includes a graphics processing unit (GPU). The PPU 400 is configured to receive commands specifying a shader for processing graphics data. Graphics data can be defined as a set of primitives, such as points, lines, triangles, quadrilaterals, triangle strips, etc. Typically, a primitive includes data specifying a plurality of vertices of the primitive (e.g., in a model space coordinate system) and attributes associated with each vertex of the primitive. The PPU 400 can be configured to process the primitives to generate a frame buffer (e.g., pixel data for each of the pixels of a display).

[0111] An application writes model data for a scene (e.g., a collection of vertices and attributes) into memory (such as system memory or memory 404). The model data defines each of the objects that may be visible on the display. The application then makes an API call to the driver kernel, requesting the model data to be rendered and displayed. The driver kernel reads the model data and writes commands to one or more streams to perform operations to process the model data. These commands may reference different shader programs to be implemented on the multiprocessors 540 of the PPU 400, including one or more of vertex shading, hull shading, domain shading, geometry shading, and pixel shading. For example, one or more of the multiprocessors 540 may be configured to execute a vertex shading program that processes multiple vertices defined by the model data. In one embodiment, different multiprocessors 540 may be configured to execute different shading programs simultaneously. For example, a first subset of the multiprocessors 540 may be configured to execute a vertex shading program, while a second subset of the multiprocessors 540 may be configured to execute a pixel shading program. The first subset of the multiprocessors 540 processes the vertex data to generate processed vertex data and writes the processed vertex data to the L2 cache 460 and / or memory 404. After the processed vertex data is rasterized (e.g., converted from three-dimensional data to two-dimensional data in screen space) to generate fragment data, a second subset of the multiprocessors 540 performs pixel shading to generate processed fragment data, which is then blended with other processed fragment data and written to a frame buffer in the memory 404. The vertex shading program and the pixel shading program can be executed simultaneously, processing different data from the same scene in a pipelined manner, until all model data for the scene has been rendered to the frame buffer. The contents of the frame buffer are then transmitted to the display controller for display on the display device.

[0112] Figure 7 According to one embodiment, Figure 4 4. A conceptual diagram of a graphics processing pipeline 700 implemented by a PPU 400 of FIG. 7. The graphics processing pipeline 700 is an abstract flow diagram of the processing steps implemented to generate a 2D computer-generated image from 3D geometric data. As is well known, pipeline architectures can perform long-latency operations more efficiently by dividing the operations into multiple stages, where the output of each stage is coupled to the input of the next consecutive stage. Thus, the graphics processing pipeline 700 receives input data 701 that is passed from one stage of the graphics processing pipeline 700 to the next stage to generate output data 702. In one embodiment, the graphics processing pipeline 700 may represent a graphics processing pipeline consisting of API-defined graphics processing pipeline. Alternatively, graphics processing pipeline 700 can be implemented within the functional and architectural context of the previous figures and / or one or more of any subsequent figures.

[0113] like Figure 7As shown, the graphics processing pipeline 700 includes a pipeline architecture comprising multiple stages. These stages include, but are not limited to, a data assembly stage 710, a vertex shading stage 720, a primitive assembly stage 730, a geometry shading stage 740, a viewport scale, cull, and clip (VSCC) stage 750, a rasterization stage 760, a fragment shading stage 770, and a raster operation stage 780. In one embodiment, input data 701 includes commands that configure a processing unit to implement the stages of the graphics processing pipeline 700 and configure geometric primitives (e.g., points, lines, triangles, quads, triangle strips, or fans, etc.) to be processed by these stages. Output data 702 may include pixel data (i.e., color data), which is copied to a frame buffer or other type of surface data structure in memory.

[0114] The data assembly stage 710 receives input data 701, which specifies vertex data for high-level surfaces, primitives, etc. The data assembly stage 710 collects the vertex data in temporary storage or queues, such as by receiving a command from the host processor including a pointer to a buffer in memory and reading the vertex data from the buffer. The vertex data is then passed to the vertex shading stage 720 for processing.

[0115] The vertex shading stage 720 processes vertex data by executing a set of operations (e.g., a vertex shader or program) on each vertex at a time. A vertex may be specified, for example, as a 4-coordinate vector (e.g., ) associated with one or more vertex attributes (e.g., color, texture coordinates, surface normal, etc.).<x,y,z,w> ). The vertex shading stage 720 can manipulate various vertex attributes, such as position, color, texture coordinates, etc. In other words, the vertex shading stage 720 performs operations on the vertex coordinates or other vertex attributes associated with the vertex. These operations typically include lighting operations (e.g., modifying the color attribute of the vertex) and transformation operations (e.g., modifying the coordinate space of the vertex). For example, a vertex can be specified using coordinates in object coordinate space, which are transformed by multiplying the coordinates by a matrix that converts the coordinates from the object coordinate space to world space or normalized-device-coordinate (NCD) space. The vertex shading stage 720 generates transformed vertex data that is passed to the primitive assembly stage 730.

[0116] The primitive assembly stage 730 collects the vertices output by the vertex shading stage 720 and groups the vertices into geometric primitives for processing by the geometry shading stage 740. For example, the primitive assembly stage 730 can be configured to group every three consecutive vertices into geometric primitives (e.g., triangles) for transmission to the geometry shading stage 740. In some embodiments, particular vertices can be reused for consecutive geometric primitives (e.g., two consecutive triangles in a triangle strip can share two vertices). The primitive assembly stage 730 transmits the geometric primitives (e.g., a collection of associated vertices) to the geometry shading stage 740.

[0117] The geometry shading stage 740 processes geometric primitives by executing a set of operations (e.g., geometry shaders or programs) on the geometric primitives. Tessellation operations can generate one or more geometric primitives from each geometric primitive. In other words, the geometry shading stage 740 can subdivide each geometric primitive into a finer mesh of two or more geometric primitives for processing by the rest of the graphics processing pipeline 700. The geometry shading stage 740 passes the geometric primitives to the viewport SCC stage 750.

[0118] In one embodiment, the graphics processing pipeline 700 may operate within a streaming multiprocessor and vertex shading stage 720, primitive assembly stage 730, geometry shading stage 740, fragment shading stage 770, and / or hardware / software associated therewith, and may perform processing operations sequentially. Once the sequential processing operations are completed, in one embodiment, the viewport SCC stage 750 may utilize the data. In one embodiment, primitive data processed by one or more stages in the graphics processing pipeline 700 may be written to a cache (e.g., an L1 cache, a vertex cache, etc.). In this case, in one embodiment, the viewport SCC stage 750 may access the data in the cache. In one embodiment, the viewport SCC stage 750 and the rasterization stage 760 are implemented as fixed function circuits.

[0119] The viewport SCC stage 750 performs viewport scaling, culling, and clipping of geometric primitives. Each surface being rendered is associated with an abstract camera position. The camera position represents the position of the viewer viewing the scene and defines a viewing cone that surrounds the objects of the scene. The viewing cone can include a viewing plane, a back plane, and four clipping planes. Any geometric primitives that are completely outside the viewing cone can be culled (e.g., discarded) because they will not contribute to the final rendered scene. Any geometric primitives that are partially inside the viewing cone and partially outside the viewing cone can be clipped (e.g., converted to new geometric primitives that are enclosed within the viewing cone). In addition, each geometric primitive can be scaled based on the depth of the viewing cone. All potentially visible geometric primitives are then transferred to the rasterization stage 760.

[0120] The rasterization stage 760 converts 3D geometric primitives into 2D fragments (e.g., capable of being used for display, etc.). The rasterization stage 760 can be configured to use the vertices of the geometric primitives to set a set of plane equations from which various attributes can be interpolated. The rasterization stage 760 can also calculate a coverage mask for multiple pixels, which indicates whether one or more sample positions of the pixel intercept the geometric primitive. In one embodiment, a z test can also be performed to determine whether the geometric primitive is occluded by other geometric primitives that have already been rasterized. The rasterization stage 760 generates fragment data (e.g., interpolated vertex attributes associated with a specific sample position for each covered pixel), which is passed to the fragment shading stage 770.

[0121] The fragment shading stage 770 processes the fragment data by executing a set of operations (e.g., a fragment shader or program) on each of the fragments. The fragment shading stage 770 can generate pixel data (e.g., color values) for the fragment, such as by performing lighting operations or sampling a texture map using the fragment's interpolated texture coordinates. The fragment shading stage 770 generates pixel data, which is sent to the raster operations stage 780.

[0122] The raster operations stage 780 may perform various operations on the pixel data, such as performing alpha tests, stencil tests, and blending the pixel data with other pixel data corresponding to other fragments associated with the pixel. When the raster operations stage 780 has completed processing the pixel data (e.g., output data 702), the pixel data may be written to a render target, such as a frame buffer, a color buffer, etc.

[0123] It should be appreciated that one or more additional stages may be included in the graphics processing pipeline 700 in addition to or in place of one or more of the above-described stages. Various implementations of the abstract graphics processing pipeline may implement different stages. Furthermore, in some embodiments, one or more of the above-described stages may be excluded from the graphics processing pipeline (such as the geometry shading stage 740). Other types of graphics processing pipelines are contemplated within the scope of the present disclosure. Furthermore, any stage of the graphics processing pipeline 700 may be implemented by one or more dedicated hardware units within a graphics processor (such as the PPU 400). Other stages of the graphics processing pipeline 700 may be implemented by programmable hardware units (such as the multiprocessor 540 of the PPU 400).

[0124] The graphics processing pipeline 700 can be implemented via an application program executed by a host processor (such as a CPU). In one embodiment, a device driver can implement an application programming interface (API) that defines various functions that can be utilized by the application program to generate graphics data for display. A device driver is a software program that includes multiple instructions that control the operation of the PPU 400. The API provides an abstraction for programmers, allowing them to utilize specialized graphics hardware (such as the PPU 400) to generate graphics data without requiring them to utilize the specific instruction set of the PPU 400. An application program can include API calls that are routed to the device driver of the PPU 400. The device driver interprets the API calls and performs various operations in response to the API calls. In some cases, the device driver can perform operations by executing instructions on the CPU. In other cases, the device driver can perform operations at least in part by initiating operations on the PPU 400 using an input / output interface between the CPU and the PPU 400. In one embodiment, the device driver is configured to implement the graphics processing pipeline 700 using the hardware of the PPU 400.

[0125] Various programs can be executed within the PPU 400 to implement the various stages of the graphics processing pipeline 700. For example, a device driver can launch a kernel on the PPU 400 to execute the vertex shading stage 720 on one multiprocessor 540 (or multiple multiprocessors 540). The device driver (or the initial kernel executed by the PPU 400) can also launch other kernels on the PPU 400 to execute other stages of the graphics processing pipeline 700, such as the geometry shading stage 740 and the fragment shading stage 770. In addition, some of the stages of the graphics processing pipeline 700 can be implemented on fixed unit hardware, such as a rasterizer or data assembler implemented within the PPU 400. It should be appreciated that the results from one kernel may be processed by one or more intermediate fixed-function hardware units before being processed by subsequent kernels on the multiprocessor 540.

[0126] Machine Learning

[0127] Deep neural networks (DNNs) developed on processors such as the PPU 400 are already being used in a variety of use cases: from self-driving cars to faster drug development, from automatic image captioning in online image databases to intelligent real-time language translation in video chat applications. Deep learning is a technology that models the neural learning process of the human brain, constantly learning, getting smarter, and delivering more accurate results faster over time. A child, initially taught by an adult to correctly identify and classify various shapes, eventually becomes able to recognize shapes without any tutoring. Similarly, deep learning or neural learning systems need to be trained in object recognition and classification to become smarter and more efficient at recognizing basic objects, occluded objects, and assigning context to objects.

[0128] At the simplest level, neurons in the human brain examine the various inputs they receive, assign a level of importance to each of these inputs, and pass outputs to other neurons for processing. An artificial neuron, or perceptron, is the most basic model of a neural network. In one example, a perceptron can receive one or more inputs representing various features of the object it is being trained to recognize and classify, and each of these features is assigned a certain weight based on its importance in defining the object's shape.

[0129] Deep neural network (DNN) models consist of multiple layers of connected nodes (e.g., perceptrons, Boltzmann machines, radial basis functions, convolutional layers, etc.), which can be trained with large amounts of input data to solve complex problems quickly and accurately. In one example, the first layer of a DNN model breaks down an input image of a car into its components and looks for basic patterns (such as lines and angles). The second layer assembles the lines to find higher-level patterns, such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the final layers generate labels for the input image, identifying the model of a specific car brand.

[0130] Once trained, a DNN can be deployed and used to recognize and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include recognizing handwritten numbers on check deposits at an ATM, identifying images of friends in photos, providing movie recommendations to over 50 million users, identifying and classifying different types of cars, pedestrians, and road hazards in self-driving cars, or translating human speech in real time.

[0131] During training, data flows through the DNN in a forward propagation phase until a prediction is produced, which indicates the label corresponding to the input. If the neural network does not correctly label the input, the error between the correct label and the predicted label is analyzed, and the weights are adjusted for each feature during a backward propagation phase until the DNN correctly labels that input and other inputs in the training dataset. Training complex neural networks requires a large amount of parallel computing performance, including floating-point multiplications and additions supported by the PPU 400. Inference is a less computationally intensive and latency-sensitive process than training, in which a trained neural network is applied to new inputs it has not seen before to classify images, translate speech, and generally reason about new information.

[0132] Neural networks rely heavily on matrix math operations, and complex, multi-layer networks require significant floating-point performance and bandwidth for efficiency and speed. With thousands of processing cores optimized for matrix math operations and delivering tens to hundreds of TFLOPS of performance, the PPU 300 is a computing platform capable of delivering the performance required for deep neural network-based artificial intelligence and machine learning applications.

Claims

1. A system comprising: a processing device comprising a first cache; a system memory operatively coupled to the processing device; as well as a parallel processing unit (PPU) operatively coupled to the processing device, wherein the PPU includes a second cache, and wherein the PPU is configured to: receiving a cache operation to modify a cache line of the second cache associated with a first aperture of the system memory; as well as identifying a first subset of cache lines of the second cache, wherein the first subset of cache lines is associated with the first aperture of the system memory; identifying a second subset of cache lines of the second cache, wherein the second subset of cache lines is associated with a second aperture of the system memory; as well as The first subset of cache lines is modified as specified by the cache operation.

2. The system of claim 1 , wherein the cache operation is an invalidate operation, and wherein to modify the first cache line subset, the PPU is to: writing data stored in the first cache line subset back to the first cache; and The first subset of cache lines is invalidated.

3. The system of claim 1 , wherein the cache operation is a refresh operation, and wherein to modify the first cache line subset, the PPU is to: writing data stored in the first cache line subset back to the first cache; and The first subset of cache lines is maintained in a clean state. 4 . The system of claim 1 , wherein the first aperture is a non-uniform aperture of the system memory and the second aperture is a uniform aperture of the system memory.

5. The system of claim 1 , wherein the PPU is further configured to: The second subset of cache lines is maintained within the second cache. 6 . The system of claim 1 , wherein the first subset of cache lines and the second subset of cache lines are identified based on an identifier associated with a non-coherent system memory aperture and an identifier associated with a coherent system memory aperture, respectively. 7 . The system according to claim 1 , wherein the PPU and the processing device are interconnected through an interface using a common hardware interface (CHI) protocol. 8 . The system of claim 1 , wherein the first cache line subset is further distinguishable and invalidated based on a process identifier indicating one of a plurality of processes associated with the processing device.

9. The system of claim 1, wherein coherency of the second subset of cache lines is managed by hardware associated with the processing device.

10. A method comprising: receiving, at a parallel processing unit (PPU) comprising a first cache, a cache operation to modify a cache line of the first cache associated with a first aperture of a system memory of the processing device; identifying a first cache line subset of the first cache, wherein the first cache line subset is associated with the first aperture of the system memory; identifying a second subset of cache lines of the first cache, wherein the second subset of cache lines is associated with a second aperture of the system memory; as well as The first subset of cache lines is modified as specified by the cache operation.

11. The method of claim 10, wherein the cache operation is an invalidate operation, and wherein modifying the first subset of cache lines comprises: writing data stored in the first subset of cache lines back to a second cache of the processing device; as well as The first subset of cache lines is invalidated.

12. The method of claim 10, wherein the cache operation is a refresh operation, and wherein modifying the first subset of cache lines comprises: writing data stored in the first subset of cache lines back to a second cache of the processing device; as well as The first subset of cache lines is maintained in a clean state.

13. The method of claim 10, wherein the first aperture is a non-uniform aperture of the system memory and the second aperture is a uniform aperture of the system memory.

14. The method according to claim 10, further comprising: The second subset of cache lines is maintained within the first cache.

15. The method of claim 10, wherein the first and second cache line subsets are identified based on an identifier associated with a non-uniform aperture and an identifier associated with a uniform aperture, respectively. 16 . The method according to claim 10 , wherein the PPU and the processing device are interconnected through an interface using a common hardware interface (CHI) protocol. 17 . The method of claim 10 , wherein the first cache line subset is distinguishable and invalidated further based on a process identifier indicative of one of a plurality of processes associated with the processing device.

18. The method of claim 10, wherein coherency of the second subset of cache lines is managed by hardware associated with the processing device.

19. One or more processors comprising processing circuitry configured to: receiving, at a parallel processing unit (PPU) comprising a first cache, a cache operation to modify a cache line of the first cache associated with a first aperture of a system memory of the processing device; identifying a first cache line subset of the first cache, wherein the first cache line subset is associated with the first aperture of the system memory; identifying a second subset of cache lines of the first cache, wherein the second subset of cache lines is associated with a second aperture of the system memory; as well as The first subset of cache lines is modified as specified by the cache operation.

20. One or more processors according to claim 19, wherein the cache operation is an invalidate operation, and wherein to modify the first subset of cache lines, the processing circuitry is to: writing data stored in the first subset of cache lines back to a second cache of the processing device; and The first subset of cache lines is invalidated as invalid.

Images