Address translation service for implementing memory coherency

Abstract

The invention relates to address translation services for implementing memory coherency. The first virtual address is translated to a first physical address using a first translation agent associated with a first I / O device of the system. The first physical address is associated with an address space of the first I / O device. The first address translation request is sent to a second translation agent associated with a CPU of the system. The first address translation request includes a first physical address. A first address translation response is received from the second translation agent. The second address translation response includes a second physical address. The second physical address is associated with an address space of the system.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 566,139, filed on March 15, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] Embodiments of the present disclosure generally relate to Address Translation Services (ATS). Specifically, embodiments of the present disclosure relate to systems and methods for implementing memory coherence in ATS. Background Art

[0004] Parallel processing in high-performance computing (HPC) systems involves executing multiple computing tasks or operations simultaneously. This is achieved by breaking down larger computations into smaller, independent subtasks that can be processed concurrently by multiple processors. In some cases, parallel processing involves distributed computing, where tasks are spread across multiple computing clusters. Each cluster can operate independently and can facilitate communication to share results. HPC systems often utilize virtual memory systems to provide applications with larger, more flexible address spaces. Virtual addresses can be mapped using an address translation service (ATS). Address translation services include the mechanisms and processes used in a computer system to translate virtual addresses into physical addresses. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 is a block diagram illustrating a computer system configured to implement one or more aspects of the present disclosure.

[0006] Figure 2 is an example diagram depicting address translation operations to implement memory consistency in accordance with at least one embodiment of the present disclosure.

[0007] Figure 3 Examples of page table entries (PTEs) for implementing the address translation operations described herein are shown, in accordance with at least one embodiment of the present disclosure.

[0008] Figure 4 A flowchart of a method for implementing a memory consistency address translation service according to at least one embodiment of the present disclosure is shown.

[0009] Figure 5 A parallel processing unit is shown in accordance with at least one embodiment of the present disclosure.

[0010] Figure 6AAccording to at least one embodiment of the present disclosure, Figure 5 A general-purpose processing cluster within a parallel processing unit.

[0011] Figure 6B According to at least one embodiment of the present disclosure, Figure 5 A memory partitioning unit of a parallel processing unit.

[0012] Figure 7A According to at least one embodiment of the present disclosure, Figure 6A Streaming multiprocessor.

[0013] Figure 7B According to at least one embodiment of the present disclosure, Figure 5 Conceptual diagram of the processing system implemented by the PPU.

[0014] Figure 7C An exemplary system is shown in which the various architecture and / or functionality of the various previous embodiments may be implemented.

[0015] Figure 8 According to at least one embodiment of the present disclosure Figure 5 Conceptual diagram of the graphics processing pipeline implemented by the PPU. Summary of the Invention

[0016] In various computer systems, input / output (I / O) devices communicate over network fabrics such as the Peripheral Component Interconnect (PCI) or PCI-Express (PCIE). Such I / O devices can include, for example, field programmable gate array (FPGA) accelerators, network interface cards (NICs), and graphics processing units (GPUs).

[0017] To support memory consistency between the host central processing unit (CPU) subsystem and one or more physical I / O devices (such as one or more GPUs of the system), the CPU subsystem and one or more GPUs can operate within the same system physical address (SPA) space. SPA space refers to the physical address range that the system can use to access main memory, such as system memory associated with the CPU subsystem, memory on peripheral devices, etc. An address translation service (ATS) can be implemented to ensure that the CPU subsystem, GPU, and other input / output (I / O) devices have an unambiguous view of the SPA. As used herein, SPA can generally refer to the final, translated physical address. In some conventional systems, I / O devices (such as GPUs) obtain SPAs using a request / response process. For example, a GPU can provide a virtual address (VA) as part of an address translation request (ATR) packet to obtain an associated SPA. The CPU subsystem can obtain the associated SPA using a translation agent (such as an input / output memory mapping unit (IOMMU) within the CPU subsystem), which contains a mapping of VAs (also referred to herein as input / output virtual addresses (IOVAs)) to SPAs. The CPU subsystem may provide SPA as part of its response to the GPU, and the GPU cache may be tagged with SPA. In some cases, an identifier such as a Process Address Space ID (PASID) is used to specify an address to be translated into a virtual address space belonging to a particular process, such as (in the case of virtualization) a virtualized I / O device or a virtual machine (VM). Therefore, this address translation scheme is referred to herein as PASID ATS.

[0018] In addition to one or more other features not described herein, PASID ATS can also implement hardware cache coherence between the CPU and GPU. For example, the GPU can use SPA as a tag through its cache to ensure that memory visibility between the GPU cache and the CPU cache is consistent. Figure 1 , thereby achieving CPU-GPU coherence. However, PASID ATS cannot enable GPU-specific optimizations provided by GPU page tables because VA is directly converted to SPA through the IOMMU without using GPU page tables, such as those provided by the GMMU. As a result, PASID ATS may not be able to provide support for memory compression, GPU work creation, various surface formats, and various address types (e.g., Fabric Linear Addresses (FLAs), Memory Mapped I / O (MMIO) addresses, etc.) because the attributes that enable such support are stored in the GMMU page tables.

[0019] In some conventional systems, an I / O device (such as a GPU) may use the GMMU within the GPU to translate between VA and its own internal addresses, such as GPU physical addresses (GPA). In this translation scheme, the GPA associated with the GPU may also be the SPA associated with the CPU, since there is only one address translation stage. While this address translation scheme can enable the aforementioned GPU-specific optimizations associated with the GMMU properties, it may not be able to maintain memory safety because the memory is completely managed by the GPU. As used herein, GPA generally refers to the address space resulting from the GMMU translation. Based on this definition, GPA can be equivalent to SPA or IOVA in some cases, as described herein.

[0020] Various aspects and implementations of the present disclosure address the aforementioned and other deficiencies of conventional address translation services by performing address translation serially. To perform address translation serially, the system may provide I / O devices with the ability to translate VA to GPU physical addresses / guest physical addresses (GPA) and a translation agent, such as an IOMMU associated with the system CPU, to translate GPA to SPA. GPA may also refer to GPU physical addresses, or, from the perspective of the CPU subsystem, IOVA.

[0021] For example, the GPU of the system can use a conversion agent (such as GMMU) associated with the GPU to convert VA into GPA. In some embodiments, the GPU can use the received page directory base (PDB) identifier to perform address translation. A unique PDB identifier can be assigned (for example, by a hypervisor, an operating system (OS), etc.) and can be used as a context identifier. The GPU can use the PDB identifier to determine the top level of the page table to convert VA into GPA. In some embodiments, the GPU can determine additional attributes stored in the GMMU page table that implement GPU-specific functions. In response to determining that the GPA is associated with system memory, the GPU can send an ATS request to the CPU to convert the GPA into SPA. Since the process information associated with the PDB identifier has been used by the GMMU, the ATS request can be sent to the IOMMU associated with the CPU without the associated PASID. Therefore, the address translation scheme may also be referred to as non-PASID ATS in this article.

[0022] In response to receiving an ATS request with a GPA from the GMMU, the CPU subsystem may use a translation agent (such as an IOMMU) to convert the GPA (IOVA from the CPU subsystem's perspective) to an SPA. The IOMMU may send an ATS response including the SPA to the GPU. The GPU may cache the SPA in one or more of its TLBs for subsequent memory requests and / or use the SPA to mark the GPU cache. In some embodiments, the ATS response may be merged with one or more GMMU attributes generated during the aforementioned GMMU walk.

[0023] Advantages of the techniques disclosed herein include, but are not limited to, achieving bidirectional CPU-GPU coherence with an unambiguous view of the SPA, since the GPU can cache the SPA in its TLB. Furthermore, the ATS scheme described herein can provide system-wide atomic support for GPU atomics. Furthermore, by caching the SPA in the GPU TLB, bandwidth issues associated with IOMMU translations in the CPU subsystem can be avoided. Furthermore, the GPU can use the SPA returned from the ATS response and GPU attributes obtained from the GMMU address translation to obtain the benefits associated with PASID ATS and use the obtained GMMU attributes to provide GPU-specific optimizations.

[0024] Figure 1 is a block diagram illustrating a computer system 100 configured to implement one or more aspects of the present disclosure. In some embodiments, the system 100 includes interconnected hardware devices, while in other embodiments, the system 100 may be an exemplary system on a chip (SoC). In at least one embodiment, the devices of the computer system 100 may be interconnected via a proprietary interconnect, a standard interconnect (e.g., PCI or PCIe), or a combination thereof. In at least one embodiment, the devices of the computer system 100 may be interconnected with a chip-to-chip (C2C) interconnect or a chip-to-package interconnect. The computer system 100 includes a central processing unit (CPU) subsystem 102 and a system memory 116 interconnected using an interface. The interface also connects the CPU subsystem 102 and the system memory 116 to a fabric to which a plurality of I / O devices / peripherals are also connected.

[0025] The CPU subsystem 102 may include a CPU 104, including one or more processing cores 106A to 106N (generally referred to herein as "processing cores 106"), a last level cache (LLC) 110, and a translation agent 112 including one or more TLBs 114. The CPU 104 may include one or more processing cores 106A to 106N (generally referred to herein as "processing cores 106"). Each processing core 106 may be a separate processing unit within the CPU 104 that independently executes instructions. Each processing core 106 may be a complete processing unit with its own arithmetic logic unit (ALU), control unit, registers, and other components required to execute program instructions. In some embodiments, the translation agent 112 may be a memory management unit (MMU) that translates between virtual addresses used by the CPU 104 and the SPA. The translation agent 112 may access a page table to determine the translation for a given virtual address. The TLB 114 may store recently accessed virtual-to-physical internal mappings. Each processing core may have a dedicated cache (e.g., L1, L2 108A to 108N, etc.) and may further access a system cache, such as the LLC 110. In some embodiments, CPU 104 may run programs such as programs that are started when the system boots, an operating system, or device drivers that manage the allocation of structural resources and peripheral device configurations in system 100 .

[0026] System 100 includes a CPU subsystem 102 coupled to one or more I / O devices that perform various tasks for system 100. In this example, the I / O devices connected to the fabric may include one or more parallel processing units (PPUs), such as PPU 120. In some embodiments, one or more of the I / O devices may include an accelerator or other coprocessor implemented in a field programmable gate array (FPGA) (not shown) and connected to the bus.

[0027] The PPU 120 may include one or more multiprocessors 122, a graphics memory management unit (GMMU) 124, one or more translation lookaside buffers (TLBs) 126, a cache 130, and a video memory (VMEM) 128. For example, each multiprocessor 122 may be a streaming multiprocessor (SM), a compute unit (CU), a multi-integrated core (MIC), etc. Each multiprocessor 122 may be responsible for executing parallel processing tasks that involve performing the same operation on multiple pieces of data concurrently. Each multiprocessor 122 may execute a certain number of threads simultaneously, so that the PPU 120 as a whole can execute a large number of threads concurrently across all multiprocessors 122.

[0028] It should be understood that the system 100 shown herein is illustrative and that variations and modifications are possible. The connection topology, the number of CPUs 104 within the CPU subsystem 102, the number of PPUs 120, the number of processing cores 106 within the CPU 104, and the number of multiprocessors 122 within the PPU 120 can be modified as desired. Furthermore, the specific components shown herein are not exhaustive; for example, any number of add-in cards, peripherals, switches, network adapters, etc. may be supported but are not shown herein.

[0029] In at least one embodiment, PPU 120 is a graphics processing unit having a rendering pipeline that can be configured to perform various tasks associated with generating pixel data from graphics data provided by CPU subsystem 102 and / or system memory 116. In operation, CPU 104 is the central processing unit of computer system 100, controlling and coordinating the operations of other system components. Specifically, CPU 104 can issue instructions that control the operation of PPU 120.

[0030] In at least one embodiment, PPU 120 may incorporate circuitry optimized for graphics and video processing, including, for example, video output circuitry, and constitute a graphics processing unit (GPU). In another embodiment, PPU 120 incorporates circuitry optimized for general-purpose processing while preserving the underlying compute architecture. PPU 120 may further include video memory (VMEM) 128, which PPU 120 may use to store necessary data, such as textures, frame buffers, shaders, and other graphics elements. In at least one embodiment, VMEM 128 may include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as video random access memory (VRAM) or synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) SGRAM. In at least one embodiment, VMEM 130 may include one or more memory stacks, such as multiple DRAM dies stacked vertically to form high-bandwidth memory (HBM). It should be understood that the specific implementation of VMEM 128 may vary and may be selected from many available designs.

[0031] VMEM 128 can be addressed according to SPA. In some embodiments, PPU 120 includes a graphics memory management unit (GMMU) 124 for translating between virtual addresses used by applications and GPA. One or more engines (e.g., compute, graphics, copy engine, video engine, etc.) associated with PPU 120 or a virtual machine (VM) can be virtually addressed and make requests using virtual addresses. GMMU 124 can translate virtual addresses received from applications into GPA. In at least one embodiment, when the aperture is VMEM 128 and virtualization is disabled, GMMU 124 can generate a GPA equivalent to SPA because (1) the possible GPA range falls under an address block (e.g., 512GB) and the GPU is assigned a SPA block that is equal to or larger than the GPA range, and (2) the GPA is padded on top with address bits above the GPU's SPA block for consistency purposes. In at least one embodiment, when the aperture is VMEM 128 and virtualization is enabled, GMMU 124 generates a GPA as an intermediate physical address, which can be further translated by a virtual memory management unit (VMMU) to obtain a SPA.

[0032] The GMMU 124 may include one or more translation lookaside buffers (TLBs) 126. The TLBs 126 may store cache lines containing multiple page table entries (PTEs), some of which may be valid and some of which may be invalid. The PTEs may maintain translations between virtual addresses and GPAs, or between virtual addresses obtained from the CPU subsystem 102 and system physical addresses (SPAs). The GPU engine may issue translation requests using the virtual address space. The translation requests may be received by one or more local TLBs and sent down through the GMMU TLB hierarchy of the TLBs 126 (e.g., L2TLBs or hub TLBs). If a translation request is lost on the TLB 126, a page walk may be performed on the GMMU 124 to retrieve a GMMU PTE containing the address to be translated. Each valid GMMU PTE may include a separate GPA address and a set of GMMU attributes associated with the page. These GPAs may be further translated by the CPU subsystem 102 using the ATS techniques described herein.

[0033] The GMMU 124 may send an address translation request (ATR) to the translation agent 112 to translate the GPA into a system physical address (SPA) associated with the CPU subsystem 102. In some embodiments, the translation agent constitutes an input / output memory management unit (IOMMU) or a system memory management unit (SMMU). In some embodiments, the GMMU 124 may provide translation context information to the translation agent 112 via a bus / device / function (BDF) field transmitted in the ATR. In some embodiments, the BDF field is an N-bit identifier in the ATR that includes identifiers associated with the bus, device, and function of the ATR. It may be noted that from the perspective of the translation agent 112, the GPA received with the ATR may be considered a virtual address. Therefore, the GPA may also be referred to as an input / output virtual address (IOVA).

[0034] The translation agent 112 can translate device-generated addresses (e.g., GPA / IOVA) into SPAs. SPAs are fully resolved physical addresses used to reference system address mappings. The translation agent 112 can be configured and initialized by the operating system (OS), or in the case of virtualization, by the hypervisor. In some embodiments, the translation agent 112 can use one or more TLBs 114 to cache recently used translations. If the translation is not found in one or more TLBs 114 associated with the translation agent 112, the translation agent 112 can initiate a page walk to retrieve the translation from system memory 116. The translation agent 112 can accordingly use the ATS protocol so that the cache in the PPU 120 can be tagged with SPAs, thereby achieving full coherency between the CPU subsystem 102 and the PPU 120. The translation agent 112 can provide and address the translation response, including the translated SPA, to the PPU 120. GMMU 124 may cache the received SPA in TLB 126, tag the cache with SPA for full coherency, and merge the SPA with the GMMU attributes so that the GMMU 124 PTE contains the SPA translation and a set of GMMU 124 attributes.

[0035] In some embodiments, the translation agent 112 may include features to support virtualization, allowing multiple virtual machines to run on the physical system 100 while maintaining isolation between their memory spaces and the I / O devices they access. In such embodiments, the translation agent 112 may perform multiple stages of address translation. For example, in a first stage, the translation agent 112 may perform a translation from a GPA / IOVA programmed by a guest or process to an intermediate physical address (IPA). In a second stage, the translation agent may perform a translation from an IPA programmed by a hypervisor to an SPA.

[0036] Figure 2 2 is an example diagram 200 illustrating address translation operations for implementing memory consistency according to at least one embodiment of the present disclosure. Schematic diagram 200 may include similar elements shown in computing system 100, as shown in FIG. Figure 1 It should be noted that Figure 1 The elements may be used herein to help describe Figure 2 For purposes of illustration and not limitation, the following is shown Figure 2 The operations described are performed serially. Although shown in a particular order or sequence, the order of operations may be modified unless otherwise indicated. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated operations may be performed in a different order, and some operations may be performed in parallel. In addition, one or more operations may be omitted in at least one embodiment. Therefore, not all illustrated operations are required in every embodiment, and other processing flows are possible. In at least one embodiment, the same, different, fewer, or more operations may be performed. In at least one embodiment, Figure 2 One or more of the operations described in the preceding paragraphs may be performed by the PPU 120 or Figure 1 In at least one embodiment, Figure 2 One or more operations described in may be performed by a general-purpose CPU or GPU based on instructions generated by software that manages and allocates hardware resources, such as an operating system (OS) or a hypervisor.

[0037] Schematic diagram 200 includes a GPU 201 and a CPU subsystem 230 communicating via an interface such as a C2C interface, a PCIe interface, etc. In at least one embodiment, GPU 201 may correspond to Figure 1 In at least one embodiment, the CPU subsystem 230 may correspond to the PPU 120. Figure 1 The diagram 200 shows a system including a GPU 201 having dedicated local memory 208. In at least one embodiment, aspects and implementations of the present disclosure can be applied to a unified memory system (e.g., a SoC) such that the GPU 201 does not include dedicated local memory 208.

[0038] Schematic diagram 200 includes one or more GPU clients 202 of a GPU 201. The one or more GPU clients 202 may request access to memory resources using virtual addresses (VAs) 203. In at least one embodiment, the one or more GPU clients 202 may include applications or processes associated with the GPU 201, such as Figure 1The PPU 120 is used for computing and rendering tasks. In some embodiments, the GPU 201 client may also include a virtual machine (VM) or a specific application running in conjunction with the VM that utilizes GPU resources. Virtualization can allow multiple VMs to share the same physical hardware, such as the GPU 201, and each VM can act as an independent GPU client, utilizing resources provided by the host system.

[0039] The TLB hierarchy 204 may receive a memory request from the GPU client 202 including a VA 203. The VA 203 may be a guest virtual address (GVA) associated with a VM or a GPU virtual address associated with the GPU 201. Thus, the VA 203 may also be referred to herein as the GVA 203. In at least one embodiment, the TLB hierarchy 204 may include one or more TLBs (e.g., an L1 TLB, an L2 TLB, etc.) associated with a memory management unit (GMMU) of the GPU 201. For example, the TLB hierarchy 204 may include Figure 1 The TLB 126 of the GPU may be accessed through the TLB hierarchy 204. The TLB hierarchy 204 may store cache lines containing multiple page table entries (PTEs). The PTEs may maintain translations between VA, GPA, and SPA. Each valid GMMU PTE may also include an independent GPA address and a set of GMMU attributes associated with the page. The request may be sent down through the TLB hierarchy 204. If the request hits the TLB hierarchy 204, the process flow may continue to the GPU NoC 206. If the request does not hit the TLB hierarchy 204, the process flow may continue to the GMMU fill unit 220.

[0040] If the request hits the TLB hierarchy 204, the physical address retrieved from the associated PTE can be sent to the GPU network-on-chip (NoC) 206. In at least one embodiment, the TLB hierarchy 204 can also provide translation context information to the GPU NoC 206 using a bus / device / function (BDF) identifier. The BDF can identify the source of the request. For example, the BDF can be mapped (e.g., using a mapping table, using one or more integer constants, etc.) to a GPU function identifier (GFID) associated with the GPU and / or a guest identifier associated with the requesting VM. In at least one embodiment, the PTE of the TLB hierarchy 204 can include a BDF identifier, and the BDF identifier includes a GFID. The GFID can be used to identify the function number of the address translation request (ATR) and memory request as they flow through the diagram 200. In at least one embodiment, the GFID can be an encoded PCIe function number that identifies the physical function (physical device) or virtual function (virtual machine) associated with the ATR / memory request. GFID, GVA, valid bits, address type identifiers, and other identifiers used in conjunction with ATR / memory requests may be included within the PTE, as described below with reference to Figure 3 shown.

[0041] The GPU NoC 206 may be an on-chip network architecture within the GPU. Specifically, the GPU NoC 206 is a communication infrastructure that connects various components within the chip (e.g., functional blocks, processing units, memory, etc.). For example, the GPU NoC 206 may facilitate communication between different memories, such as local memory 208, remote memory 210, and system memory 212. The GPU NoC 206 may retrieve data from the local memory 208, remote memory 210, or system memory 212 based on the type of physical address received from the TLB hierarchy 204. In at least one embodiment, the GPU NoC 206 may decode the physical address to determine whether it is associated with the local memory 208, the remote memory 210 (or any other memory associated with a remote I / O device), or the system memory 212.

[0042] For example, GPU NoC 206 may receive a physical address from TLB hierarchy 204 and decode the physical address to determine that it is a SPA associated with a local I / O device that is part of the system, such as GPU 201. In response to determining that the physical address is a SPA associated with local memory, GPU NoC 206 may retrieve data associated with the SPA from local memory 208. In some embodiments, local memory 208 may be video memory that is part of the system, such as VMEM 128.

[0043] In another example, GPU NoC 206 may receive a physical address from TLB hierarchy 204, decode the physical address, and determine that the physical address is associated with a remote device, such as a remote GPU. For example, the physical address may be a GPA associated with remote memory 210 or a fabric linear address (FLA) of remote memory 210 associated with a remote I / O device. A remote I / O device, such as a remote GPU, may be a peer GPU physically separate from a system including GPU 201 and CPU subsystem 230. GPU NoC 206 may retrieve data associated with the GPA / FLA directly from remote memory 210. In at least one embodiment, GPU NoC 206 may perform memory accesses to remote memory 210 over a standardized and / or C2C interconnect, such as a high-speed GPU-GPU NVLINK-C2C interconnect. In this manner, data may be retrieved directly from remote memory 210 without involving a host CPU, such as CPU 104. In this illustrative example, GPU NoC 206 may send a request to a remote GPU using FLA addressing. The remote GPU can perform GMMU translation and optional ATS translation to determine the SPA and route the request to the correct memory destination. Since the remote GPU has the SPA address of the request, it can be cached coherently for that remote GPU if necessary.

[0044] In another example, GPU NoC 206 may receive a physical address from TLB hierarchy 204 and decode the physical address to determine that it is associated with system memory 212. In at least one embodiment, system memory 212 may correspond to Figure 1 In response to determining that the received physical address is a SPA associated with system memory 212, GPU NoC 206 may use the SPA to retrieve data directly from system memory 212. In response to determining that the physical address is a GPA, GPU NoC 206 may send the physical address to translation agent 214 to translate the received physical address inline into a SPA.

[0045] If the request from GPU client 202 misses on TLB hierarchy 204, virtual address 203 may be sent to GMMU fill unit 220. In response to determining that PASID ATS is enabled (e.g., via software), GMMU fill unit 220 may send an ATR including virtual address 203 and a PASID associated with virtual address 203 to translation agent 214. In some embodiments, a given virtual address space may have (1) a GMMU mapping followed by a non-PASID ATS mapping and / or (2) a PASID ATS mapping. Unless otherwise configured by software, two types of translations may be attempted when a request from GPU client 202 misses on TLB hierarchy 204.

[0046] A page walk can be performed on the GMMU of the GPU 201 to obtain a GMMU PTE that maps a virtual address range including the VA 203 to be translated. Each valid GMMU PTE can include a separate GPA address and a set of GMMU attributes associated with the page. The translation of the VA 203 to the GPA can be retrieved from the GMMU PTE. In at least one embodiment, the GMMU walk can be performed on a page table indicated by a page directory base (PDB) identifier. A unique PDB identifier can be assigned (e.g., by a hypervisor, operating system (OS), etc.) to a virtual machine (VM) using the GPU and / or GPU engine that issues the address translation request. The PDB can serve as a context identifier. When an I / O device or VM generates a memory request, the PDB identifier and the VA 203 can be included in the request. The GMMU can use the PDB identifier to determine the page table within the GMMU to translate the VA 203 to the GPA.

[0047] The GPA and associated GMMU attributes may be sent to attribute merger 222. Additionally, if non-PASIDATS is enabled (e.g., via software), the translated GPA (IOVA from the perspective of translation agent 214) and associated BDF identifier may be sent to translation agent 214.

[0048] In at least one embodiment, translation agent 214 can directly translate VA 203 into SPA based on PASID ATS technology. In at least one embodiment, translation agent 214 can use one or more address translation stages to translate the GPA generated as a result of GMMU traversal 221 into SPA. For example, translation agent 214 can receive a BDF and GPA from the GMMU and indicate that non-PASID ATS is enabled. Translation agent 214 can use one or more fields of the BDF to select a translation context. For example, translation agent 214 can use one or more fields of the BDF to determine a page table within translation agent 214 to translate the GPA into SPA.

[0049] In response to converting the GPA to the SPA, the translation agent 214 may provide the SPA in the ATS response (such as a standard protocol packet) to the attribute merger 222. The attribute merger 222 may merge the GMMU attributes determined by the GMMU traversal 221 with the SPA received from the translation agent 214 into a merged response. The final translated SPA may be stored in a TLB entry in the TLB hierarchy 204. The final translated SPA may be the result of one or more translation stages of a serial GMMU translation and a translation agent 214 translation. Due to the serialized nature of the translation, the entries in the TLB hierarchy 204 are merged with the GMMU attributes and the translation agent 214 PTE. The GMMU attributes may include an aperture type indicating the type and / or location of the memory being accessed, an attribute indicating whether the request is cached in the GPU 201, and the like. The GMMU attributes may also include attributes controlling data compression and data format. In addition, the GMMU attributes may include page permissions, such as permissions indicating whether atomic requests are allowed.

[0050] The GPU 201 may cache the VA 203 SPA translation in the TLB hierarchy 204 and use the SPA in the GPU caches (e.g., L1 cache, L2 cache, L3 cache, etc.) to achieve bidirectional coherency between the GPU 201 caches and the CPU subsystem 230 caches. In addition, the SPA may be cached and associated with received GMMU attributes, which may be used to achieve memory coherency, system atomicity, memory compression, and other GPU-specific optimizations.

[0051] When a page in memory is unmapped, the corresponding PTE in the TLB hierarchy 204 that describes the mapping can be updated to mark the PTE as invalid. After the PTE is marked invalid, any TLBs in the TLB hierarchy 204 that may have a cached copy of the translation can also be invalidated. When all TLB cache lines corresponding to the invalidated virtual address are deleted and there are no running instructions using the address obtained from the invalidated PTE, the invalidation is considered complete. In at least one embodiment, invalidation can be performed in different scopes. For example, the system can invalidate any TLB entry containing a specific virtual address (such as VA 203); the system can invalidate all TLB entries within a virtual address range; the system may invalidate an entire context or function; and the system can invalidate all TLBs. When invalidating PTEs based on one or more VAs, the TLB entries can be invalidated by comparing the TLB tags because the PTEs are tagged by the VA.

[0052] In at least one embodiment, invalidation can be supported at each translation level (e.g., GMMU translation and one or more translation levels performed by translation agent 214). When nested page tables are used with multiple translation stages, the TLB in each translation stage does not contain intermediate translations. In this case, invalidation of a lower level of the page table hierarchy requires invalidation of all upper nested levels to ensure that there are no invalid translations that span lower translation stages. Invalidation can be targeted at lower levels because the virtual address range of that level is known. However, the virtual address used to mark the TLB in the higher level is unknown to the lower level (hypervisor or kernel driver) performing the invalidation, so the higher level invalidation can be a global invalidation of the entire context or function using GFID, PASID and / or PDB.

[0053] Figure 3 An example of a page table entry (PTE) 300 for implementing the address translation operations described herein is shown in accordance with at least one embodiment of the present disclosure. For example, the PTE 300 may include Figure 2 300. The PTE 300 may be a TLB hierarchy 204 that stores a tag for a PTE in the TLB hierarchy 204. Specifically, the PTE 300 includes a field 302 for a valid bit (V). If the field 302 indicates that the PTE 300 is invalid, address translation may be stopped and a fault may be caused upon a client request. The PTE 300 may include a field 304 for a GPU function identifier (GFID) that identifies the GPU and / or guest (e.g., VM) associated with the client request. The PTE 300 includes a field 306 for an untranslated (UT) indicator that indicates whether the associated physical address is a translated SPA.

[0054] PTE 300 includes a field 308 for a page directory base identifier (PDBID). The PDBID is a unique identifier assigned by the GMMU hardware. If PASID ATS applies (e.g., PASID ATS is enabled), the GMMU hardware uses the PDB and PASID to assign the PDBID. For non-PASID ATS, the GMMU hardware may use the PDB alone to assign the PDB. PTE 300 includes a field 310 for a GVA. The GVA may be referred to as a GPU virtual address or a guest virtual address associated with a VM. For example, field 310 may store Figure 2 VA 203.

[0055] In some embodiments, PTE 300 may include one or more fields corresponding to non-tag attributes. In the example shown, PTE 300 includes field 312 for a corresponding GPA or SPA. For example, field 312 may store a GPA corresponding to a GVA stored in field 310 or a SPA corresponding to a GWA stored in field 310. In some embodiments, PTE 300 may include one or more additional fields corresponding to non-tag attributes, such as compression enabled, volatility, aperture, etc.

[0056] Figure 4 A flow chart of a method for implementing address translation services for memory consistency according to at least one embodiment of the present disclosure is shown. Although method 400 is described in the context of a processing unit, method 400 can also be performed by a program, custom circuitry, or a combination of custom circuitry and a program. For example, method 400 can be performed by an I / O device, a GPU (graphics processing unit), a CPU (central processing unit), or any processor capable of virtual addressing. In addition, it will be understood by those skilled in the art that any system that performs method 400 is within the scope and spirit of the embodiments of the present invention.

[0057] Now, more illustrative information will be provided regarding various optional architectures and features that can implement the above framework, depending on the user's desires. It should be noted that the following information is for illustrative purposes only and should not be construed as limiting in any way. Any of the following features may be selectively combined with or without excluding the other features described.

[0058] Although shown in a particular order or sequence, the order of operations may be modified unless otherwise indicated. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated operations may be performed in a different order, and some operations may be performed in parallel. Furthermore, one or more operations may be omitted in various embodiments. Therefore, not all operations are required in every embodiment.

[0059] At operation 402 of method 400, processing circuitry may use a first translation agent associated with a first I / O device of the system to translate a first virtual address (VA) (e.g., Figure 2 In at least one embodiment, the first I / O device may be a GPU, and the first conversion agent may be a GMMU. For example, the first I / O device may be Figure 2 The first conversion agent may be a GPU 201, and the first physical address may be a GPA associated with the GPU 201. Figure 1 GMMU 124. In at least one embodiment, to translate the first VA into the first physical address using the first translation agent, the processing circuitry may use a page directory base (PDB) identifier to identify a page table of the first translation agent, wherein the page table is for a VM service of the system as indicated by the PDB identifier; and translate the virtual address into the first physical address using the identified page table.

[0060] At operation 404 of method 400, the processing circuit may send a first address translation request (ATR) to a second translation agent associated with a CPU of the system, such as regarding Figure 1 The CPU 104 of the system 100 is shown. For example, the second conversion agent may correspond to Figure 1 Conversion agent 112 or Figure 2 The first address translation request may include the first physical address. In at least one embodiment, the second translation agent may include an IOMMU. In at least one embodiment, in response to determining that the first physical address is associated with the system memory of the CPU, the first ATR is sent to the second translation agent.

[0061] At operation 406 of method 400, the processing circuitry may receive a first address translation response from the second translation agent. The first address translation response includes a second physical address. The second physical address is associated with an address space of the system. For example, the second physical address may be a SPA. In at least one embodiment, the first I / O device includes a cache, and the processing circuitry may use the second physical address as a tag within the cache. In at least one embodiment, the cache of the first I / O device is coherent with one or more caches of the CPU. In an illustrative example, the processing circuitry may cache the translated SPA in one or more TLBs 126 of the PPU 120 or within the cache hierarchy 204 of the GPU 201.

[0062] In at least one embodiment, the processing logic may merge the first address translation response with one or more attributes obtained by the first translation agent, wherein the first address translation response further includes one or more attributes obtained from the second translation agent. In at least one embodiment, the system includes a second I / O device coupled to the first I / O device via a chip-to-chip (C2C) interconnect. In at least one embodiment, the processing logic may use the first translation agent to translate the second virtual address into a fabric linear address (FLA). The processing logic may determine that the FLA is associated with the second I / O device, wherein the second I / O device is a remote I / O device. The processing logic may send a second address translation request to the second I / O device, wherein the second address translation request includes the FLA. The processing logic may receive a second address translation response from the second I / O device, the second address translation response including data associated with the FLA.

[0063] Parallel processing architecture

[0064] Figure 5 A parallel processing unit (PPU) 500 is shown according to one embodiment. In one embodiment, the PPU 500 is a multi-threaded processor implemented on one or more integrated circuit devices. The PPU 500 is a latency-hiding architecture designed for processing many threads in parallel. A thread (e.g., an execution thread) is an instance of an instruction set configured to be executed by the PPU 500. In one embodiment, the PPU 500 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device (such as a liquid crystal display (LCD) device). In other embodiments, the PPU 500 can be used to perform general-purpose computations. Although an exemplary parallel processor is provided herein for illustrative purposes, it should be specifically noted that the processor is described for illustrative purposes only and any processor may be used in addition to and / or in place of the processor.

[0065] One or more PPUs 500 can be configured to accelerate thousands of high-performance computing (HPC), data center, and machine learning applications. PPUs 500 can be configured to accelerate numerous deep learning systems and applications, including autonomous vehicle platforms, deep learning, high-precision speech, image, and text recognition systems, intelligent video analysis, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations, among others.

[0066] like Figure 5As shown, the PPU 500 includes an input / output (I / O) unit 505, a front-end unit 515, a scheduler unit 520, a work distribution unit 525, a hub 530, a crossbar switch (Xbar) 570, one or more processing clusters 550 (e.g., general processing clusters GPCs), and one or more partition units 580. The PPU 500 can be connected to a host processor or other PPUs 500 via one or more high-speed NVLink 510 interconnects. The PPU 500 can be connected to a host processor or other peripheral devices via an interconnect 502. The PPU 500 can also be connected to a local memory including a plurality of memory devices 504. In one embodiment, the local memory can include a plurality of dynamic random access memory (DRAM) devices. The DRAM devices can be configured as a high-bandwidth memory (HBM) subsystem in which multiple DRAM dies are stacked within each device.

[0067] The NVLink 510 interconnect enables the system to scale and include one or more PPUs 500 in conjunction with one or more CPUs, supporting cache coherency between the PPU 500 and the CPU, and CPU mastering. Data and / or commands can be sent by the NVLink 510 through the hub 530 to or from other units of the PPU 500, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). Figure 7B NVLink 510 is described in more detail.

[0068] I / O unit 505 is configured to send and receive communications (e.g., commands, data, etc.) from a host processor (not shown) via interconnect 502. I / O unit 505 can communicate directly with the host processor via interconnect 502, or through one or more intermediate devices (such as a memory bridge). In one embodiment, I / O unit 505 can communicate with one or more other processors (e.g., one or more PPUs 500) via interconnect 502. In one embodiment, I / O unit 505 implements a Peripheral Component Interconnect Express (PCIe) interface for communicating over a PCIe bus, and interconnect 502 is a PCIe bus. In alternative embodiments, I / O unit 505 can implement other types of known interfaces for communicating with external devices.

[0069] The I / O unit 505 decodes data packets received via the interconnect 502. In one embodiment, the data packets represent commands configured to cause the PPU 500 to perform various operations. The I / O unit 505 sends the decoded commands to various other units of the PPU 500 as specified by the commands. For example, some commands may be sent to the front end unit 515. Other commands may be sent to the hub 530 or other units of the PPU 500, such as one or more copy engines, video encoders, video decoders, a power management unit, etc. (not explicitly shown). In other words, the I / O unit 505 is configured to route communications between and among the various logical units of the PPU 500.

[0070] In one embodiment, a program executed by a host processor encodes a command stream in a buffer that provides a workload to the PPU 500 for processing. The workload may include many instructions and data to be processed by those instructions. A buffer is an area of ​​memory that is accessible (e.g., read / write) by both the host processor and the PPU 500. For example, the I / O unit 505 may be configured to access a buffer in system memory connected to the interconnect 502 via a memory request transmitted over the interconnect 502. In one embodiment, the host processor writes a command stream into the buffer and then sends a pointer to the start of the command stream to the PPU 500. The front end unit 515 receives pointers to one or more command streams. The front end unit 515 manages the one or more streams, reads commands from the streams, and forwards the commands to the various units of the PPU 500.

[0071] The front-end unit 515 is coupled to a scheduler unit 520, which configures various processing clusters 550 to process tasks defined by one or more flows. The scheduler unit 520 is configured to track status information associated with the various tasks managed by the scheduler unit 520. The status may indicate which processing cluster 550 the task is assigned to, whether the task is active or inactive, the priority associated with the task, and the like. The scheduler unit 520 manages the execution of multiple tasks on one or more processing clusters 550.

[0072] Scheduler unit 520 is coupled to work distribution unit 525, which is configured to dispatch tasks for execution on processing clusters 550. Work distribution unit 525 can track a number of scheduled tasks received from scheduler unit 520. In one embodiment, work distribution unit 525 manages a pending task pool and an active task pool for each processing cluster 550. The pending task pool can include a number of time slots (e.g., 32 time slots) containing tasks assigned to be processed by a particular processing cluster 550. The active task pool can include a number of time slots (e.g., 4 time slots) for tasks being actively processed by processing cluster 550. When a processing cluster 550 completes execution of a task, the task is evicted from the active task pool of processing cluster 550, and one of the other tasks from the pending task pool is selected and scheduled for execution on processing cluster 550. If an active task on processing cluster 550 has become idle, such as while waiting for data dependencies to be resolved, then the active task can be evicted from processing cluster 550 and returned to the pending task pool, while another task in the pending task pool is selected and scheduled for execution on processing cluster 550.

[0073] Work distribution unit 525 communicates with one or more processing clusters 550 via XBar (crossbar) 570. XBar 570 is an interconnect network that couples many units of PPU 500 to other units of PPU 500. For example, XBar 570 can be configured to couple work distribution unit 525 to a specific processing cluster 550. Although not explicitly shown, one or more other units of PPU 500 can also be connected to XBar 570 via hub 530.

[0074] Tasks are managed by the scheduler unit 520 and dispatched to the processing clusters 550 by the work distribution unit 525. The processing clusters 550 are configured to process tasks and generate results. The results can be consumed by other tasks within the processing cluster 550, routed to a different processing cluster 550 via the XBar 570, or stored in the memory 504. The results can be written to the memory 504 via the partition unit 580, which implements a memory interface for reading data from and writing data to the memory 504. The results can be sent to another PPU 504 or CPU via the NVLink 510. In one embodiment, the PPU 500 includes a number U of partition units 580, which is equal to the number of independent and different memory devices 504 coupled to the PPU 500. Figure 6B The partition unit 580 is described in more detail.

[0075] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API) that enables one or more applications to be executed on the host processor to schedule operations for execution on the PPU 500. In one embodiment, multiple computing applications are executed simultaneously by the PPU 500, and the PPU 500 provides isolation, quality of service (QoS), and independent address spaces for the multiple computing applications. The application can generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks to be executed by the PPU 500. The driver kernel outputs the tasks to one or more streams being processed by the PPU 500. Each task can include one or more related groups of threads, referred to herein as warps. In one embodiment, a warp includes 32 related threads that can execute in parallel. Collaborating threads can refer to multiple threads that include instructions to perform tasks and that can exchange data through shared memory. In combination Figure 7A Describes threads and cooperative threads in more detail.

[0076] Figure 6A According to one embodiment, Figure 5 The processing cluster 550 of the PPU 500 is Figure 6A As shown, each processing cluster 550 includes multiple hardware units for processing tasks. In one embodiment, each processing cluster 550 includes a pipeline manager 610, a pre-raster operation unit (PROP) 615, a raster engine 625, a work distribution crossbar (WDX) 680, a memory management unit (MMU) 690, and one or more data processing clusters (DPCs) 620. It should be understood that Figure 6A The processing cluster 550 may include instead Figure 6A Other hardware units or other than the units shown in Figure 6A Other hardware units besides those shown in .

[0077] In one embodiment, the operation of processing cluster 550 is controlled by pipeline manager 610. Pipeline manager 610 manages the configuration of one or more DPCs 620 for processing tasks assigned to processing cluster 550. In one embodiment, pipeline manager 610 may configure at least one of one or more DPCs 620 to implement at least a portion of a graphics rendering pipeline. For example, DPC 620 may be configured to execute vertex shading programs on programmable multiprocessor 640. Multiprocessor 640 may typically include a streaming multiprocessor, a compute unit, a number of integrated cores, and the like. Pipeline manager 610 may also be configured to route packets received from work distribution unit 625 to appropriate logic units within processing cluster 550. For example, some packets may be routed to fixed-function hardware units within PROP 615 and / or raster engine 625, while other packets may be routed to DPC 620 for processing by primitive engine 635 or multiprocessor 640. In one embodiment, pipeline manager 610 may configure at least one of one or more DPCs 620 to implement a neural network model and / or a compute pipeline.

[0078] PROP unit 615 is configured to route data generated by raster engine 625 and DPC 620 to the raster operations (ROP) unit, in conjunction with Figure 6B The PROP unit 615 may also be configured to perform optimizations for color blending, organize pixel data, perform address translation, and the like.

[0079] The raster engine 625 includes several fixed-function hardware units configured to perform various raster operations. In one embodiment, the raster engine 625 comprises a setup engine, a coarse raster engine, a culling engine, a clipping engine, a fine raster engine, and a tile aggregation engine. The setup engine receives the transformed vertices and generates plane equations associated with the geometric primitives defined by the vertices. The plane equations are sent to the coarse raster engine to generate coverage information for the primitives (e.g., the x, y coverage mask of the tile). The output of the coarse raster engine is sent to the culling engine, where fragments associated with primitives that fail the z-test are culled, and to the clipping engine, where fragments outside the viewing frustum are clipped. Those fragments remaining after clipping and culling can be passed to the fine raster engine to generate pixel fragment attributes based on the plane equations generated by the setup engine. The raster engine 625 output includes, for example, fragments to be processed by the fragment shader implemented in the DPC 620.

[0080] Each DPC 620 included in the processing cluster 550 includes an M-pipe controller (MPC) 630, a primitive engine 635, and one or more multiprocessors 640. The MPC 630 controls the operation of the DPC 620 and routes packets received from the pipeline manager 610 to appropriate units within the DPC 620. For example, packets associated with vertices may be routed to the primitive engine 635, which is configured to retrieve vertex attributes associated with the vertices from the memory 504. Conversely, packets associated with shading programs may be sent to the multiprocessor 640.

[0081] In some embodiments, multiprocessor 640 includes a programmable multiprocessor, such as a programmable streaming multiprocessor, configured to process tasks represented by multiple threads. Each multiprocessor 640 is multithreaded and configured to simultaneously execute multiple threads (e.g., 32 threads) from a particular thread group. In one embodiment, multiprocessor 640 implements a SIMD (single instruction, multiple data) architecture, in which each thread in a thread group (e.g., a warp) is configured to process a different data set based on the same instruction set. All threads in a thread group execute the same instructions. In another embodiment, multiprocessor 640 implements a SIMT (single instruction, multiple thread) architecture, in which each thread in a thread group is configured to process a different data set based on the same instruction set, but in which individual threads in a thread group are allowed to diverge during execution. In one embodiment, a program counter, call stack, and execution state are maintained for each warp, enabling concurrency between warps and serial execution within warps when threads within the warp diverge. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, thereby achieving equal concurrency between all threads within and between warps. When execution state is maintained for each individual thread, threads executing the same instruction can be converged and executed in parallel for maximum efficiency. Figure 7A Multiprocessor 640 is described in more detail.

[0082] The MMU 690 provides an interface between the processing cluster 550 and the partition unit 580. The MMU 690 can provide virtual address to physical address translation, memory protection, and arbitration of memory requests. In one embodiment, the MMU 690 provides one or more translation lookaside buffers (TLBs) for performing translations from virtual addresses to physical addresses in the memory 504.

[0083] Figure 6B According to one embodiment, Figure 5 The memory partition unit 580 of the PPU 500. Figure 6BAs shown, the memory partition unit 580 includes a level 2 (L2) cache 660 and a memory interface 670. The memory interface 670 is coupled to the memory 504. The memory interface 670 can implement a 32-, 64-, 128-, or 1024-bit data bus for high-speed data transfer. In one embodiment, the PPU 500 incorporates U memory interfaces 670, one for each pair of partition units 580, where each pair of partition units 580 is connected to a corresponding memory device 504. For example, the PPU 500 can be connected to up to Y memory devices 504, such as a high-bandwidth memory stack or graphics double data rate version 5 synchronous dynamic random access memory or other types of persistent memory.

[0084] In one embodiment, memory interface 670 implements an HBM2 memory interface, and Y is equal to half of U. In one embodiment, the HBM2 memory stack is located on the same physical package as PPU 500, providing significant power and area savings compared to conventional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory dies and Y is equal to 4, where the HBM2 stack includes two 128-bit channels per die, for a total of 8 channels and a data bus width of 1024 bits.

[0085] In one embodiment, memory 504 supports single-error correction, double-error detection (SECDED) error correction code (ECC) to protect data. ECC provides increased reliability for computing applications sensitive to data corruption. Reliability is particularly important in large cluster computing environments where PPU 500 processes very large data sets and / or long-running applications.

[0086] In one embodiment, the PPU 500 implements a multi-level memory hierarchy. In one embodiment, the memory partitioning unit 580 supports unified memory to provide a single, unified virtual address space for the CPU and PPU 500 memory, enabling data sharing between virtual memory systems. In one embodiment, the frequency of PPU 500 accesses to memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 500 where the pages are accessed more frequently. In one embodiment, NVLink 510 supports address translation services that allow the PPU 500 to directly access the CPU's page tables and provide full access to the CPU's memory by the PPU 500.

[0087] In one embodiment, the copy engine transfers data between multiple PPUs 500 or between a PPU 500 and a CPU. The copy engine can generate a page fault for an address that is not mapped to a page table. The memory partition unit 580 can then service the page fault, map the address into a page table, and then the copy engine can perform the transfer. In conventional systems, multiple copy engines operate on fixed memory (e.g., non-pageable) between multiple processors, which significantly reduces the available memory. Due to hardware page faults, addresses can be passed to the copy engine without worrying about whether the memory page is resident, and the copy process is transparent.

[0088] Data from memory 504 or other system memory can be retrieved by memory partition unit 580 and stored in L2 cache 660, which is located on-chip and shared between the various processing clusters 550. As shown, each memory partition unit 580 includes a portion of L2 cache 660 associated with the corresponding memory device 504. Lower-level caches can then be implemented in multiple units within a processing cluster 550. For example, each multiprocessor 640 can implement a level 1 (L1) cache. The L1 cache is a dedicated memory dedicated to a particular multiprocessor 640. Data from the L2 cache 660 can be retrieved and stored in each L1 cache for processing in the functional units of the multiprocessor 640. The L2 cache 660 is coupled to the memory interface 670 and the XBar 570.

[0089] Figure 7A According to one embodiment, Figure 6A The multiprocessor 640. Figure 7A As shown, the multiprocessor 640 includes an instruction cache 705, one or more scheduler units 710, a register file 720, one or more processing cores 750, one or more special function units (SFUs) 752, one or more load / store units (LSUs) 754, an interconnection network 780, and a shared memory / L1 cache 770.

[0090] As described above, the work distribution unit 525 schedules tasks for execution on the processing cluster 550 of the PPU 500. Tasks are assigned to specific DPCs 620 within the processing cluster 550, and if the task is associated with a shader program, the task may be assigned to a multiprocessor 640. The scheduler unit 710 receives tasks from the work distribution unit 525 and manages the scheduling of instructions assigned to one or more thread blocks assigned to the multiprocessor 640. The scheduler unit 710 schedules the thread blocks for execution as warps of parallel threads, where each thread block is assigned at least one warp. In one embodiment, each warp executes 32 threads. The scheduler unit 710 can manage multiple different thread blocks, assign warps to different thread blocks, and then dispatch instructions from multiple different cooperative groups to various functional units (e.g., core 750, SFU 752, and LSU 754) during each clock cycle.

[0091] Cooperative Groups is a programming model for organizing groups of communicating threads that allows developers to express the granularity at which threads are communicating, enabling the expression of richer and more efficient decompositions of parallelism. The cooperative launch API supports synchronization between thread blocks to execute parallel algorithms. The conventional programming model provides a single simple structure for synchronizing cooperating threads: a barrier across all threads of a thread block (i.e., the syncthreads() function). However, programmers often want to define thread groups at a granularity smaller than the thread block granularity and synchronize within the defined group, enabling higher performance, design flexibility, and software reuse in the form of a collective group-wide function interface.

[0092] Cooperative Groups enable programmers to explicitly define thread groups at sub-block (e.g., as small as a single thread) and multi-block granularity and perform collective operations, such as synchronization, on threads in a cooperative group. The programming model supports clean composition across software boundaries so that libraries and utility functions can safely synchronize in their local environment without making assumptions about convergence. Cooperative Group primitives enable new patterns of cooperative parallelism, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire grid of thread blocks.

[0093] The dispatch unit 715 is configured to deliver instructions to one or more functional units. In this embodiment, the scheduler unit 710 includes two dispatch units 715, which enable scheduling two different instructions from the same warp during each clock cycle. In alternative embodiments, each scheduler unit 710 may include a single dispatch unit 715 or additional dispatch units 715.

[0094] Each multiprocessor 640 includes a register file 720 that provides a set of registers for the functional units of the multiprocessor 640. In one embodiment, the register file 720 is divided between each functional unit so that each functional unit is allocated a dedicated portion of the register file 720. In another embodiment, the register file 720 is divided between the different warps executed by the multiprocessor 640. The register file 720 provides temporary storage for operands connected to the data paths of the functional units.

[0095] Each multiprocessor 640 includes L processing cores 750. In one embodiment, the multiprocessor 640 includes a large number (e.g., 128, etc.) of different processing cores 750. Each core 750 may include a fully pipelined, single-precision, double-precision, and / or mixed-precision processing unit, including a floating-point arithmetic logic unit (FLU) and an integer arithmetic logic unit (ALU). In one embodiment, the FLU implements the IEEE 754-2008 standard for floating-point operations. In one embodiment, the core 750 includes 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0096] Tensor cores are configured to perform matrix operations, and in one embodiment, one or more tensor cores are included in core 750. Specifically, the tensor cores are configured to perform deep learning matrix operations, such as convolution operations for neural network training and inference. In one embodiment, each tensor core operates on a 4×4 matrix and performs a matrix multiplication and accumulation operation D=A×B+C, where A, B, C, and D are 4×4 matrices.

[0097] In one embodiment, the matrix multiplication inputs A and B are 16-bit floating point matrices, while the accumulation matrices C and D can be 16-bit floating point or 32-bit floating point matrices. The tensor cores operate on 16-bit floating point input data as well as 32-bit floating point accumulations. The 16-bit floating point multiplication requires 64 operations to produce a full-precision product, which is then accumulated using 32-bit floating point additions with other intermediate products of the 4×4×4 matrix multiplication. In practice, tensor cores are used to perform larger two-dimensional or higher-dimensional matrix operations built from these smaller elements. APIs (such as the CUDA 9 C++ API) expose specialized matrix load, matrix multiplication and accumulation, and matrix store operations to efficiently use tensor cores from CUDA-C++ programs. At the CUDA level, the warp-level interface assumes that the 16×16 size matrix spans all 32 threads of the warp.

[0098] Each multiprocessor 640 also includes M SFUs 752 that perform special functions (e.g., attribute evaluation, reciprocal square root, etc.). In one embodiment, the SFUs 752 may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, the SFUs 752 may include a texture unit configured to perform texture map filtering operations. In one embodiment, the texture unit is configured to load a texture map (e.g., a 2D array of texels) from memory 504 and sample the texture map to generate sampled texture values ​​for use in shader programs executed by the multiprocessor 640. In one embodiment, the texture map is stored in shared memory / L1 cache 670. The texture unit performs texture operations, such as filtering operations using mip maps (e.g., texture maps of different levels of detail). In one embodiment, each SM 540 includes two texture units.

[0099] Each multiprocessor 640 also includes N LSUs 754 that perform load and store operations between the shared memory / L1 cache 770 and the register file 720. Each multiprocessor 640 includes an interconnect network 780 that connects each functional unit to the register file 720 and connects the LSUs 754 to the register file 720 and the shared memory / L1 cache 770. In one embodiment, the interconnect network 780 is a crossbar switch that can be configured to connect any functional unit to any register in the register file 720 and to connect the LSUs 754 to memory locations in the register file and the shared memory / L1 cache 770.

[0100] The shared memory / L1 cache 770 is an on-chip memory array that allows data storage and communication between the multiprocessor 640 and the primitive engine 635, as well as between threads in the multiprocessor 640. In one embodiment, the shared memory / L1 cache 770 includes 128KB of storage capacity and is in the path from the multiprocessor 640 to the partition unit 580. The shared memory / L1 cache 770 can be used to cache reads and writes. One or more of the shared memory / L1 cache 770, the L2 cache 660, and the memory 504 is a backing store.

[0101] Combining data cache and shared memory functionality into a single memory block provides the best overall performance for both types of memory access. This capacity can be used by programs as a cache that is not using shared memory. For example, if the shared memory is configured to use half of its capacity, texture and load / store operations can use the remaining capacity. Integration within shared memory / L1 cache 770 enables shared memory / L1 cache 770 to function as a high-throughput pipeline for streaming data, while providing high-bandwidth and low-latency access to frequently reused data.

[0102] When configured for general parallel computing, a simpler configuration can be used compared to graphics processing. Specifically, Figure 5 The fixed-function graphics processing unit is shown bypassed, creating a simpler programming model. In a general-purpose parallel computing configuration, the work distribution unit 525 assigns and distributes thread blocks directly to the DPC 620. The threads in the block execute the same program, using unique thread IDs in computations to ensure each thread produces unique results, using the multiprocessor 640 to execute the program and perform computations, using the shared memory / L1 cache 770 to communicate between threads, and using the LSU 754 to read and write to global memory through the shared memory / L1 cache 770 and the memory partition unit 580. When configured for general-purpose parallel computing, the multiprocessor 640 can also write commands that the scheduler unit 520 can use to start new work on the DPC 620.

[0103] The PPU 500 may be included in a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, etc. In one embodiment, the PPU 500 is included on a single semiconductor substrate. In another embodiment, the PPU 500 is included on a system-on-chip (SoC) along with one or more other devices (such as an additional PPU 500, a memory 504, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), etc.).

[0104] In one embodiment, PPU 500 may be included on a graphics card that includes one or more memory devices 504. The graphics card may be configured to interface with a PCIe slot on a motherboard of a desktop computer. In another embodiment, PPU 500 may be an integrated graphics processing unit (iGPU) or parallel processor included in a chipset of the motherboard.

[0105] Exemplary Computing System

[0106] Systems with multiple GPUs and CPUs are being used across various industries as developers expose and exploit greater parallelism in applications such as artificial intelligence computing. High-performance GPU-accelerated systems with tens to thousands of computing nodes are deployed in data centers, research institutions, and supercomputers to solve larger problems. As the number of processing devices within high-performance systems increases, communication and data transmission mechanisms need to scale to support this increased bandwidth.

[0107] Figure 7B According to one embodiment, the Figure 5Conceptual diagram of a processing system 700 implemented by a PPU 500. The exemplary system 765 may be configured to implement Figure 4 The processing system 700 includes a CPU 730, a switch 712, and each of the plurality of PPUs 500 and a corresponding memory 504. The NVLink 510 provides a high-speed communication link between each PPU 500. Figure 7B 502 connections, but the number of connections connected to each PPU 500 and CPU 730 may vary. Switch 712 interfaces between interconnect 502 and CPU 730. PPU 500, memory 504, and NVLink 510 may be located on a single semiconductor platform to form parallel processing module 725. In one embodiment, switch 712 supports two or more protocols that interface between various different connections and / or links.

[0108] In another embodiment (not shown), NVLink 510 provides one or more high-speed communication links between each PPU 500 and CPU 730, and switch 712 interfaces between interconnect 502 and each PPU 500. The PPUs 500, memory 504, and interconnect 502 may be located on a single semiconductor platform to form a parallel processing module 725. In yet another embodiment (not shown), interconnect 502 provides one or more communication links between each PPU 500 and CPU 730, and switch 712 interfaces between each PPU 500 using NVLink 510 to provide one or more high-speed communication links between the PPUs 500. In another embodiment (not shown), NVLink 510 provides one or more high-speed communication links between the PPUs 500 and CPU 730 through switch 712. In yet another embodiment (not shown), interconnect 502 provides one or more communication links directly between each PPU 500. One or more NVLink 510 high-speed communication links may be implemented as a physical NVLink interconnect or as an on-chip or on-die interconnect using the same protocol as NVLink 510.

[0109] In the context of this specification, a single semiconductor platform may refer to a unique, single semiconductor-based integrated circuit fabricated on a die or chip. It should be noted that the term single semiconductor platform may also refer to a multi-chip module with increased connectivity that emulates on-chip operation and is substantially improved by utilizing conventional bus implementations. Of course, various circuits or devices may also be placed separately or in various combinations of semiconductor platforms, depending on the needs of the user. Alternatively, the parallel processing module 725 may be implemented as a circuit board substrate, and each of the PPU 500 and / or memory 504 may be a packaged device. In one embodiment, the CPU 730, switch 712, and parallel processing module 725 are located on a single semiconductor platform.

[0110] In one embodiment, the signaling rate of each NVLink 510 is 20 to 25 Gbit / s, and each PPU 500 includes six NVLink 510 interfaces (e.g., Figure 7B As shown, each PPU 500 includes five NVLink 510 interfaces. Each NVLink 510 provides a data transfer rate of 25 Gbit / s in each direction, with six links providing 500 Gbit / s. When the CPU 730 also includes one or more NVLink 510 interfaces, the NVLink 510 can be used specifically for Figure 7B PPU to PPU communication shown, or some combination of PPU to PPU and PPU to CPU.

[0111] In one embodiment, NVLink 510 allows direct load / store / atomic access from the CPU 730 to the memory 504 of each PPU 500. In one embodiment, NVLink 510 supports coherency operations, allowing data read from memory 504 to be stored in the cache hierarchy of the CPU 730, reducing cache access latency for the CPU 730. In one embodiment, NVLink 510 includes support for Address Translation Services (ATS), allowing the PPU 500 to directly access page tables within the CPU 730. One or more NVLinks 510 can also be configured to operate in a low-power mode.

[0112] Figure 7C An exemplary system 765 is shown in which various architectures and / or functionalities of various previous embodiments may be implemented. The exemplary system 765 may be configured to implement Figure 4 The method 400 shown in FIG.

[0113] As shown, a system 765 is provided that includes at least one central processing unit 730 connected to a communication bus 775. The communication bus 775 can be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or one or more point-to-point communication protocols. The system 765 also includes a main memory 740. Control logic (software) and data are stored in the main memory 740, which can take the form of random access memory (RAM).

[0114] System 765 also includes an input device 760, a parallel processing system 725, and a display device 745, such as a conventional CRT (cathode ray tube), an LCD (liquid crystal display), an LED (light emitting diode), a plasma display, etc. User input can be received from input device 760 (e.g., a keyboard, a mouse, a touchpad, a microphone, etc.). Each of the aforementioned modules and / or devices can even be located on a single semiconductor platform to form system 765. Alternatively, the modules can be located separately or in various combinations of semiconductor platforms, depending on the user's needs.

[0115] Furthermore, system 765 may be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, etc.) through network interface 735 for communication purposes.

[0116] System 765 may also include auxiliary storage (not shown). Auxiliary storage 610 includes, for example, a hard drive and / or a removable storage drive, representative of a floppy disk drive, a tape drive, an optical disk drive, a digital versatile disk (DVD) drive, a recording device, or a universal serial bus (USB) flash memory. The removable storage drive reads from and / or writes to a removable storage unit in a well-known manner.

[0117] Computer programs or computer control logic algorithms may be stored in the main memory 740 and / or secondary storage. These computer programs, when executed, enable the system 765 to perform various functions. The memory 740, storage, and / or any other storage are possible examples of computer-readable media.

[0118] The architecture and / or functionality of the various preceding figures can be implemented in the context of a general-purpose computer system, a circuit board system, a game console system dedicated to entertainment purposes, a dedicated system, and / or any other desired system. For example, system 765 can take the form of a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, a mobile telephone device, a television, a workstation, a game console, an embedded system, and / or any other type of logic.

[0119] Although various embodiments have been described above, it should be understood that they are presented by way of example only, not limitation. Therefore, the breadth and scope of a preferred embodiment should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

[0120] Graphics processing pipeline

[0121] In one embodiment, the PPU 500 includes a graphics processing unit (GPU). The PPU 500 is configured to receive commands specifying a shader for processing graphics data. Graphics data can be defined as a set of primitives, such as points, lines, triangles, quadrilaterals, triangle strips, etc. Typically, a primitive includes data specifying a plurality of vertices of the primitive (e.g., in a model space coordinate system) and attributes associated with each vertex of the primitive. The PPU 500 can be configured to process the primitives to generate a frame buffer (e.g., pixel data for each of the pixels of a display).

[0122] An application writes model data for a scene (e.g., a collection of vertices and attributes) to memory (such as system memory or memory 504). The model data defines each of the objects that may be visible on the display. The application then makes an API call to the driver kernel, requesting the model data to be rendered and displayed. The driver kernel reads the model data and writes commands to one or more streams to perform operations to process the model data. These commands may reference different shader programs to be implemented on the multiprocessors 640 of the PPU 500, including one or more of vertex shading, hull shading, domain shading, geometry shading, and pixel shading. For example, one or more of the multiprocessors 640 may be configured to execute a vertex shading program that processes multiple vertices defined by the model data. In one embodiment, different multiprocessors 640 may be configured to execute different shading programs simultaneously. For example, a first subset of the multiprocessors 640 may be configured to execute a vertex shading program, while a second subset of the multiprocessors 640 may be configured to execute a pixel shading program. The first subset of the multiprocessors 640 processes the vertex data to generate processed vertex data and writes the processed vertex data to the L2 cache 460 and / or memory 504. After the processed vertex data is rasterized (e.g., converted from three-dimensional data to two-dimensional data in screen space) to generate fragment data, a second subset of the multiprocessors 640 performs pixel shading to generate processed fragment data, which is then blended with other processed fragment data and written to a frame buffer in the memory 504. The vertex shading program and the pixel shading program can be executed simultaneously, processing different data from the same scene in a pipelined manner, until all model data for the scene has been rendered to the frame buffer. The contents of the frame buffer are then transmitted to the display controller for display on the display device.

[0123] Figure 8 According to one embodiment, Figure 5 500 . The graphics processing pipeline 800 is an abstract flow chart of the processing steps implemented to generate a 2D computer-generated image from 3D geometric data. As is well known, pipeline architectures can perform long-latency operations more efficiently by dividing the operations into multiple stages, where the output of each stage is coupled to the input of the next consecutive stage. Thus, the graphics processing pipeline 800 receives input data 801 that is passed from one stage of the graphics processing pipeline 800 to the next stage to generate output data 802. In one embodiment, the graphics processing pipeline 800 may represent a graphics processing pipeline composed of API-defined graphics processing pipeline. Alternatively, graphics processing pipeline 800 can be implemented within the functional and architectural context of the previous figures and / or one or more of any subsequent figures.

[0124] like Figure 8As shown, the graphics processing pipeline 800 includes a pipeline architecture comprising multiple stages. These stages include, but are not limited to, a data assembly stage 810, a vertex shading stage 820, a primitive assembly stage 830, a geometry shading stage 840, a viewport scale, cull, and clip (VSCC) stage 850, a rasterization stage 860, a fragment shading stage 870, and a raster operation stage 880. In one embodiment, input data 801 includes commands that configure a processing unit to implement the stages of the graphics processing pipeline 800 and configure geometric primitives (e.g., points, lines, triangles, quads, triangle strips, or fans, etc.) to be processed by these stages. Output data 802 may include pixel data (i.e., color data), which is copied to a frame buffer or other type of surface data structure in memory.

[0125] The data assembly stage 810 receives input data 801, which specifies vertex data for high-level surfaces, primitives, etc. The data assembly stage 810 collects the vertex data in temporary storage or queues, such as by receiving a command from the host processor that includes a pointer to a buffer in memory and reading the vertex data from the buffer. The vertex data is then passed to the vertex shading stage 820 for processing.

[0126] The vertex shading stage 820 processes vertex data by executing a set of operations (e.g., a vertex shader or program) on each vertex at a time. A vertex may be specified, for example, as a 4-coordinate vector (e.g., , ) associated with one or more vertex attributes (e.g., color, texture coordinates, surface normal, etc.).<x,y,z,w> ). The vertex shading stage 820 can manipulate various vertex attributes, such as position, color, texture coordinates, etc. In other words, the vertex shading stage 820 performs operations on the vertex coordinates or other vertex attributes associated with the vertex. These operations typically include lighting operations (e.g., modifying the color attribute of the vertex) and transformation operations (e.g., modifying the coordinate space of the vertex). For example, a vertex can be specified using coordinates in object coordinate space, which are transformed by multiplying the coordinates by a matrix that converts the coordinates from the object coordinate space to world space or normalized-device-coordinate (NCD) space. The vertex shading stage 820 generates transformed vertex data that is passed to the primitive assembly stage 830.

[0127] The primitive assembly stage 830 collects the vertices output by the vertex shading stage 820 and groups the vertices into geometric primitives for processing by the geometry shading stage 840. For example, the primitive assembly stage 830 can be configured to group every three consecutive vertices into geometric primitives (e.g., triangles) for transmission to the geometry shading stage 840. In some embodiments, particular vertices can be reused for consecutive geometric primitives (e.g., two consecutive triangles in a triangle strip can share two vertices). The primitive assembly stage 830 transmits the geometric primitives (e.g., a collection of associated vertices) to the geometry shading stage 840.

[0128] The geometry shading stage 840 processes geometric primitives by executing a set of operations (e.g., geometry shaders or programs) on the geometric primitives. A tessellation operation can generate one or more geometric primitives from each geometric primitive. In other words, the geometry shading stage 840 can subdivide each geometric primitive into a finer mesh of two or more geometric primitives for processing by the rest of the graphics processing pipeline 800. The geometry shading stage 840 passes the geometric primitives to the viewport SCC stage 850.

[0129] In one embodiment, the graphics processing pipeline 800 may operate within a streaming multiprocessor and vertex shading stage 820, primitive assembly stage 830, geometry shading stage 840, fragment shading stage 870, and / or hardware / software associated therewith, and may perform processing operations sequentially. Once the sequential processing operations are completed, in one embodiment, the viewport SCC stage 850 may utilize the data. In one embodiment, primitive data processed by one or more stages in the graphics processing pipeline 800 may be written to a cache (e.g., an L1 cache, a vertex cache, etc.). In this case, in one embodiment, the viewport SCC stage 850 may access the data in the cache. In one embodiment, the viewport SCC stage 850 and the rasterization stage 860 are implemented as fixed function circuits.

[0130] The viewport SCC stage 850 performs viewport scaling, culling, and clipping of geometric primitives. Each surface being rendered is associated with an abstract camera position. The camera position represents the position of the viewer viewing the scene and defines a viewing cone that surrounds the objects of the scene. The viewing cone can include a viewing plane, a back plane, and four clipping planes. Any geometric primitives that are completely outside the viewing cone can be culled (e.g., discarded) because they will not contribute to the final rendered scene. Any geometric primitives that are partially inside the viewing cone and partially outside the viewing cone can be clipped (e.g., converted to new geometric primitives that are enclosed within the viewing cone). In addition, each geometric primitive can be scaled based on the depth of the viewing cone. All potentially visible geometric primitives are then transferred to the rasterization stage 860.

[0131] The rasterization stage 860 converts 3D geometric primitives into 2D fragments (e.g., capable of being used for display, etc.). The rasterization stage 860 can be configured to use the vertices of the geometric primitives to set a set of plane equations from which various attributes can be interpolated. The rasterization stage 860 can also calculate a coverage mask for multiple pixels, which indicates whether one or more sample positions of a pixel intercept the geometric primitive. In one embodiment, a z test can also be performed to determine whether the geometric primitive is occluded by other geometric primitives that have been rasterized. The rasterization stage 860 generates fragment data (e.g., interpolated vertex attributes associated with a specific sample position for each covered pixel), which is transmitted to the fragment shading stage 870.

[0132] The fragment shading stage 870 processes the fragment data by executing a set of operations (e.g., a fragment shader or program) on each of the fragments. The fragment shading stage 870 can generate pixel data (e.g., color values) for the fragment, such as by performing lighting operations or sampling a texture map using the fragment's interpolated texture coordinates. The fragment shading stage 870 generates pixel data, which is sent to the raster operations stage 880.

[0133] The raster operations stage 880 may perform various operations on the pixel data, such as performing alpha tests, stencil tests, and blending the pixel data with other pixel data corresponding to other fragments associated with the pixel. When the raster operations stage 880 has completed processing the pixel data (e.g., output data 802), the pixel data may be written to a render target, such as a frame buffer, a color buffer, etc.

[0134] It should be appreciated that one or more additional stages may be included in the graphics processing pipeline 800 in addition to or in place of one or more of the above-described stages. Various implementations of the abstract graphics processing pipeline may implement different stages. Furthermore, in some embodiments, one or more of the above-described stages may be excluded from the graphics processing pipeline (such as the geometry shading stage 840). Other types of graphics processing pipelines are contemplated within the scope of the present disclosure. Furthermore, any stage of the graphics processing pipeline 800 may be implemented by one or more dedicated hardware units within a graphics processor (such as the PPU 500). Other stages of the graphics processing pipeline 800 may be implemented by programmable hardware units (such as the multiprocessor 640 of the PPU 500).

[0135] The graphics processing pipeline 800 can be implemented via an application program executed by a host processor (such as a CPU). In one embodiment, a device driver can implement an application programming interface (API) that defines various functions that can be utilized by the application program to generate graphics data for display. A device driver is a software program that includes multiple instructions that control the operation of the PPU 500. The API provides an abstraction for programmers, allowing them to utilize specialized graphics hardware (such as the PPU 500) to generate graphics data without requiring them to utilize the specific instruction set of the PPU 500. An application program can include API calls that are routed to the device driver of the PPU 500. The device driver interprets the API calls and performs various operations in response to the API calls. In some cases, the device driver can perform operations by executing instructions on the CPU. In other cases, the device driver can perform operations at least in part by initiating operations on the PPU 500 using an input / output interface between the CPU and the PPU 500. In one embodiment, the device driver is configured to implement the graphics processing pipeline 800 using the hardware of the PPU 500.

[0136] Various programs can be executed within the PPU 500 to implement the various stages of the graphics processing pipeline 800. For example, a device driver can launch a kernel on the PPU 500 to execute the vertex shading stage 820 on one multiprocessor 640 (or multiple multiprocessors 640). The device driver (or the initial kernel executed by the PPU 500) can also launch other kernels on the PPU 500 to execute other stages of the graphics processing pipeline 800, such as the geometry shading stage 840 and the fragment shading stage 870. In addition, some of the stages of the graphics processing pipeline 800 can be implemented on fixed unit hardware (such as a rasterizer or data assembler implemented within the PPU 500). It should be appreciated that the results from one kernel may be processed by one or more intermediate fixed-function hardware units before being processed by subsequent kernels on the multiprocessor 640.

[0137] Machine Learning

[0138] Deep neural networks (DNNs) developed on processors such as the PPU 500 are already being used in a variety of use cases: from self-driving cars to faster drug development, from automatic image captioning in online image databases to intelligent real-time language translation in video chat applications. Deep learning is a technology that models the neural learning process of the human brain, constantly learning, getting smarter, and delivering more accurate results faster over time. A child, initially taught by an adult to correctly identify and classify various shapes, eventually becomes able to recognize shapes without any tutoring. Similarly, deep learning or neural learning systems need to be trained in object recognition and classification to become smarter and more efficient at recognizing basic objects, occluded objects, and assigning context to objects.

[0139] At the simplest level, neurons in the human brain examine the various inputs they receive, assign a level of importance to each of these inputs, and pass outputs to other neurons for processing. An artificial neuron, or perceptron, is the most basic model of a neural network. In one example, a perceptron can receive one or more inputs representing various features of the object it is being trained to recognize and classify, and each of these features is assigned a certain weight based on its importance in defining the object's shape.

[0140] Deep neural network (DNN) models consist of multiple layers of connected nodes (e.g., perceptrons, Boltzmann machines, radial basis functions, convolutional layers, etc.), which can be trained with large amounts of input data to solve complex problems quickly and accurately. In one example, the first layer of a DNN model breaks down an input image of a car into its components and looks for basic patterns (such as lines and angles). The second layer assembles the lines to find higher-level patterns, such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the final layers generate labels for the input image, identifying the model of a specific car brand.

[0141] Once trained, a DNN can be deployed and used to recognize and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include recognizing handwritten numbers on check deposits at an ATM, identifying images of friends in photos, providing movie recommendations to over 50 million users, identifying and classifying different types of cars, pedestrians, and road hazards in self-driving cars, or translating human speech in real time.

[0142] During training, data flows through the DNN in a forward propagation phase until a prediction is produced, which indicates the label corresponding to the input. If the neural network does not correctly label the input, the error between the correct label and the predicted label is analyzed, and the weights are adjusted for each feature during a backward propagation phase until the DNN correctly labels that input and other inputs in the training dataset. Training complex neural networks requires a large amount of parallel computing performance, including floating-point multiplications and additions supported by the PPU 500. Inference is a less computationally intensive and latency-sensitive process than training, in which a trained neural network is applied to new inputs it has not seen before to classify images, translate speech, and generally reason about new information.

[0143] Neural networks rely heavily on matrix math operations, and complex, multi-layer networks require significant floating-point performance and bandwidth for efficiency and speed. With thousands of processing cores optimized for matrix math operations and delivering tens to hundreds of TFLOPS of performance, the PPU 300 is a computing platform capable of delivering the performance required for deep neural network-based artificial intelligence and machine learning applications.

Claims

1. A system comprising: Central Processing Unit (CPU); as well as a first input / output (I / O) device coupled to the CPU, wherein the first I / O device is configured to: translating a first virtual address into a first physical address using a first translation agent associated with the first I / O device; sending a first address translation request to a second translation agent associated with the CPU, wherein the first address translation request includes the first physical address; and A first address translation response is received from the second translation agent, the first address translation response including a second physical address, wherein the second physical address is associated with an address space of the system.

2. The system of claim 1 , wherein the first I / O device is further configured to: The first address translation response is merged with one or more attributes obtained by the first translation agent, wherein the first address translation response further includes one or more attributes obtained from the second translation agent. 3 . The system of claim 1 , wherein the second translation agent comprises an input / output memory management unit (IOMMU) or a system memory management unit (SMMU). 4 . The system of claim 1 , wherein the first I / O device comprises a graphics processing unit (GPU), and the first translation agent comprises a graphics memory management unit (GMMU).

5. The system of claim 1, wherein the first I / O device comprises a cache, and wherein the second physical address is used as a tag within the cache.

6. The system of claim 5, wherein the cache of the first I / O device is coherent with one or more caches of the CPU.

7. The system of claim 1, wherein the first address translation request is sent to the second translation agent in response to determining that the first physical address is associated with a system memory of the CPU.

8. The system of claim 1 , wherein to translate the first virtual address to the first physical address using the first translation agent, the first I / O device is configured to: identifying a page table of the first translation agent using a page directory base PDB identifier, wherein the page table serves a virtual machine VM of the system indicated by the PDB identifier; and The first virtual address is translated into the first physical address using the identified page table.

9. The system of claim 1 , wherein the system further comprises a second I / O device coupled to the first I / O device via a chip-to-chip (C2C) interconnect, and wherein the first I / O device is further configured to: using the first translation agent to translate the second virtual address into a structure linear address FLA; determining that the FLA is associated with the second I / O device, wherein the second I / O device is a remote I / O device; sending a second address translation request to the second I / O device, wherein the second address translation request includes the FLA; and A second address translation response is received from the second I / O device, the second address translation response including data associated with the FLA.

10. A method comprising: translating the first virtual address into a first physical address using a first translation agent associated with a first input / output (I / O) device of the system; sending a first address translation request to a second translation agent associated with a central processing unit (CPU) of the system, wherein the first address translation request includes the first physical address; and A first address translation response is received from the second translation agent, the first address translation response including a second physical address, wherein the second physical address is associated with an address space of the system.

11. The method of claim 10, further comprising: The first address translation response is merged with one or more attributes obtained by the first translation agent, wherein the first address translation response further includes one or more attributes obtained from the second translation agent. 12 . The method of claim 10 , wherein the second translation agent comprises an input / output memory management unit (IOMMU) or a system memory management unit (SMMU). 13 . The method of claim 10 , wherein the first I / O device comprises a graphics processing unit (GPU), and the first translation agent comprises a graphics memory management unit (GMMU).

14. The method of claim 10, wherein the first I / O device comprises a cache, and wherein the second physical address is used as a tag within the cache.

15. The method of claim 14, wherein the cache of the first I / O device is coherent with one or more caches of the CPU.

16. The method of claim 10, wherein in response to determining that the first physical address is associated with a system memory of the CPU, sending the first address translation request to the second translation agent.

17. The method of claim 10, wherein translating the first virtual address to the first physical address using the first translation agent comprises: identifying a page table of the first translation agent using a page directory base PDB identifier, wherein the page table serves a virtual machine VM of the system indicated by the PDB identifier; as well as The first virtual address is translated into the first physical address using the identified page table.

18. The method of claim 10, further comprising: using the first translation agent to translate the second virtual address into a structure linear address FLA; determining that the FLA is associated with a second I / O device, wherein the second I / O device is a remote I / O device; Sending a second address translation request to the second I / O device, wherein the second address translation request includes the FLA; as well as A second address translation response is received from the second I / O device, the second address translation response including data associated with the FLA.

19. One or more processors comprising processing circuitry configured to: translating the first virtual address into a first physical address using a first translation agent associated with a first input / output (I / O) device of the system; sending a first address translation request to a second translation agent associated with a central processing unit (CPU) of the system, wherein the first address translation request includes the first physical address; and A first address translation response is received from the second translation agent, the first address translation response including a second physical address, wherein the second physical address is associated with an address space of the system.

20. The one or more processors of claim 19, wherein the processing circuitry is further configured to: The first address translation response is merged with one or more attributes obtained by the first translation agent, wherein the first address translation response further includes one or more attributes obtained from the second translation agent.

Images