Method and apparatus for supporting distributed graphics and compute engines and synchronization in multi-die parallel processor architecture

Abstract

The invention relates to a method and apparatus for supporting distributed graphics and compute engines and synchronization in a multi-grain parallel processor architecture. The present disclosure describes supporting distributed graphics and computing engines in multi-grain processor (e.g., multi-grain graphics processing unit (GPU)) architectures and synchronization in such architectures. Each multi-grain processor includes a hardware-implemented remapping capability and / or a hardware-implemented memory barrier capability.

Description

Technical Field

[0001] The present technology generally relates to multi-dielet processing systems, such as multi-die graphics processing units (GPUs). More specifically, the technology herein relates to distributed graphics and compute engines and synchronization in such processing systems. Background Art

[0002] The demand for processors with extensive parallel processing capabilities, such as graphics processing units (GPUs), continues to grow. The processing demands on such processors are also growing rapidly in terms of complexity, workload size, and throughput.

[0003] The demand for GPUs to excel at workloads of rapidly increasing complexity, magnitude, and throughput has driven the need to pack an ever-increasing number of components onto a GPU semiconductor die—typically a very small square of flat semiconductor material, such as silicon, cut from a wafer, on which circuitry is fabricated. The more components packed onto a die, the more functionality can be provided by the integrated circuit chip package that includes the die. Consequently, chip designers have been working to pack an ever-increasing number and variety of components onto each physical die.

[0004] There are physical limits to how many components can be packed onto a single die. For example, packing more transistors generates more heat, which can damage the chip unless cooling is properly managed. More components (usually smaller components) on a single die can also make interconnects difficult to implement and can also cause signal problems on the interconnects, among other things. Furthermore, despite Moore's Law, some components may have a minimum physical size beyond which they cannot be easily miniaturized further.

[0005] Therefore, while workload demands continue to grow, the push to pack more components onto a single die may encounter insurmountable limitations on the number of components, component types, or physical size of the processor. In addition to packing a greater number and type of components onto a single die, other approaches to expanding the processing power and functionality of processors, such as GPUs, may be explored to meet growing workload demands. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A multi-die GPU according to some embodiments of the present disclosure is shown.

[0007] Figure 2 Another multi-die GPU according to some embodiments of the present disclosure is shown.

[0008] Figure 3An example of a request message that may be remapped relative to local engine IDs and global engine IDs of hardware engines located on multiple dies is shown according to some embodiments of the present invention.

[0009] Figure 4A A diagram illustrating example interactions between a CPU, a first die of a multi-die GPU, and a second die of a multi-die GPU when a request from the CPU is received at the multi-die GPU according to some embodiments of the present invention.

[0010] Figure 4B A diagram illustrating example interactions between a CPU, a first die of a multi-die GPU, and a second die of a multi-die GPU when a request from a hardware engine is received at a coordination component in the multi-die GPU according to some embodiments of the present disclosure.

[0011] Figure 5A A diagram illustrating example interactions between a CPU, a first die of a multi-die GPU, and a second die of a multi-die GPU when a memory request from a hardware engine of the multi-die GPU is received at a coordination component of the multi-die GPU according to some embodiments of the present disclosure.

[0012] Figure 5B A diagram illustrates example interactions between a CPU, a first die of a multi-die GPU, and a second die of the multi-die GPU when software on the CPU sends one or more memory requests to the multi-die GPU and then sets a register to trigger a memory barrier (“membar”) operation, according to some embodiments of the present disclosure.

[0013] Figure 6 A dual finite state machine (FSM) hardware arrangement for input / output (I / O) refresh operations in a membar process is shown, according to some embodiments of the present disclosure.

[0014] Figure 7 An example GPU on a die of a multi-die GPU is shown, wherein its frame buffer hub (FBHUB) and high speed hub (HSHUB) are identified according to some of their interconnections, according to some embodiments of the present disclosure.

[0015] Figure 8 An example parallel processing unit of a GPU is shown in accordance with some embodiments.

[0016] Figure 9A According to some embodiments, Figure 8 The parallel processing units within the general processing cluster (GPC)

[0017] Figure 9B Shown Figure 8 An example memory partitioning unit of a parallel processing unit.

[0018] Figure 10A FIG. 1 shows a circuit with an MMA state machine according to some embodiments. Figure 9A An example of a streaming multiprocessor (SM).

[0019] Figure 10B Conceptually illustrates a method for performing a multi-processor (M / M) process in accordance with some embodiments. Figure 10A The four sub-partitions implemented in the SM shown.

[0020] Figure 11A According to one embodiment, using Figure 8 An example conceptual diagram of a processing system implemented with a parallel processing unit (PPU) of FIG.

[0021] Figure 11B Block diagrams illustrating example systems in which the various architecture and / or functionality of the various previous embodiments may be implemented. DETAILED DESCRIPTION

[0022] As CMOS process scaling slows, silicon dies are reaching photolithographic mask limits. On the other hand, packaging technology has advanced to the point where multiple dies ("dies" / "chiplets") can be integrated to achieve higher packaging densities. Given these factors, it has been proposed to interconnect multiple physical dies together to form larger, more complex processing systems, such as GPUs. Such larger processing systems are referred to herein as "multi-die processing systems." Designing processing systems that go beyond the boundaries of a single manufactured die (as in the case of multi-die processing systems) provides a new path to scalability and removes some of the previously existing physical limitations.

[0023] For ease of description, the present disclosure refers to a multi-die GPU having two independently manufactured dies, where each die contains one or more streaming multiprocessors (SMs), specialized and general-purpose hardware engines, and associated routing components for executing applications on behalf of processes launched by the CPU or GPU. In some embodiments, the GPUs on the dies can be identical. In this case, the multi-die GPU can have duplicate components between the two dies. Depending on the application, such duplicate components can be used or made redundant. In some embodiments, the GPUs on the dies can be different, and some GPUs can have different sets of hardware units. Example embodiments are necessary to accommodate different scenarios in which multiple GPUs with different capabilities are combined together.

[0024] In the case of a standalone (i.e., single-die) GPU, software (e.g., GPU driver software running on a CPU) typically treats a single GPU as a single GPU with which it interacts to schedule tasks on the GPU. In at least some embodiments, the front end of the GPU memory system may be a frame buffer hub (FBHUB). Therefore, one of the issues with multi-die GPUs is that, when each die in a multi-die GPU has its own FBHUB, software and various hardware entities (e.g., the compute and graphics engines on each die) that access the multi-die GPU's memory system are provided with a way to continue treating it as a single GPU, in the same or similar manner as with a single-die GPU. The FBHUB is a hardware component of the GPU and can be considered the gateway to the GPU memory system within the GPU. To schedule and coordinate work across the compute cores, etc., many GPU components can interoperate (e.g., many direct memory access (DMA) targets, etc.). These hardware units access the memory system via the FBHUB. The FBHUB also serves as the synchronization point for any entity within the GPU that needs to access the memory system. More specifically, the FBHUB coordinates synchronization between other hardware components accessing memory on the GPU and other GPU endpoints, such as the memory partition unit (MPU) and high-speed hub (HSHUB) that interface with the connected CPU or GPU.

[0025] However, with the formation of such larger GPUs, it is often necessary or effective to protect software so that it does not have to understand the physical layout of such larger GPUs. For example, such protection may be necessary to ensure that multi-die GPUs can interoperate in many usage scenarios without requiring extensive modifications to customize the software stack for each scenario. Such protection software does not have to understand the detailed organization of the multi-die GPU, and different multi-die GPUs can be retrofitted to existing software (e.g., GPU driver software) in at least some cases. Such protection can also "future-proof" multi-die designs by ensuring that software can run with such multi-die GPUs regardless of the specific design (e.g., the number of die, the type and number of hardware engines on each die, etc.). From a system-level perspective, the methods in the disclosed embodiments are seamless and avoid the burden of the application programmer having to deal with how to stitch multiple die together to provide the computational power of a single large GPU.

[0026] Example embodiments provide for isolating GPU software from many details of the hardware organization of a multi-die GPU. An important feature of a multi-die GPU architecture is presenting a monolithic view of the GPU (e.g., a unified view) to software deployed on it and promoting reusability and portability between multiple variants of multi-die GPUs. Ideally, software should be able to understand the hardware as if it were a single GPU, regardless of how many dies are present and the individual composition of those dies. For example, if the same DMA engine is present in multiple dies, and all of those DMA engines should run as separate, independent engines, software must be able to uniquely identify each of those engines and its software-exposed engine identifier. Events such as faults, interrupts, page table binds, etc. all depend on this uniqueness. Maintaining a unified view of the GPU to the software stack running on top may be key to scaling GPU hardware, using die technology, beyond reticle limits, and beyond the end of Moore's Law.

[0027] Some embodiments can be considered to address the problem of how to uniquely combine all components (e.g., hardware units) in a GPU when the same GPU is replicated multiple times to form a single multi-die GPU or any number of asymmetric dies that will be viewed by software as a single hardware component (a single GPU). This is different from the case where the CPU, GPU, and other hardware components are combined on separate dies.

[0028] The features described in this disclosure are designed to facilitate scaling of many aspects of GPU design across multiple dies, including but not limited to syspipe, MiG, micro-GPU or other partial GPU features, and multi-die distributed GPC and memory endpoints.

[0029] This monolithic view provided by the example embodiments also extends to the hardware units in the individual dies. When reference is made to "hardware units" in this document, this includes, but is not limited to, hardware units used in graphics and compute applications, video encoding or decoding, units in the GPC and SYS clusters, and any unit with DMA capabilities. Hardware units expect "system-level awareness" of various operations in the memory subsystem. For example, when a DMA engine needs to ensure that memory writes are visible system-wide, that scope should include all memory locations local to the die containing this DMA engine as well as any other memory locations on the connected die in the GPU. Without a hardware implementation for this processing, software would require explicit knowledge of die-specific hardware units and memory locations and the ability to manually synchronize parallel operations, resulting in software complexity and performance degradation. In this disclosure, the terms "hardware units" and "hardware engines" are used interchangeably.

[0030] Embodiments of the present disclosure focus on hardware mechanisms to present a multi-die architecture (where multiple dies have at least one GPU) as a monolithic view (e.g., as a single cohesive GPU) to software. This approach differs from software-based solutions, which must provide software with information about the die structure, such as what hardware engines are on each die and how to address each hardware engine. This approach also requires fewer hardware units to understand the die structure than other potential solutions, and it also keeps more hardware units in each GPU unaware of the details of the multi-die GPU's structure, thereby promoting reuse and scalability.

[0031] Embodiments of the present disclosure provide multiple hardware features on each of the multiple dies in a multi-die GPU to create a monolithic view of the multi-die GPU for both software and hardware units. These hardware features, implemented as one or more hardware circuits, allow for hardware-driven synchronization across multiple dies and limit any need to expose these complex architectures to software applications. The multiple hardware features of this solution include address remapping features and synchronization features.

[0032] In some embodiments, the hardware engine remapper is configured to provide a unique or distinct engine ID per hardware unit that is intended to be operable in more than one die in order to uniquely or otherwise distinguish page table bindings, address translations, security checks, fault handling, etc. In some embodiments, each engine request using a non-BAR (non-base address register) engine ID includes a flag that instructs the GPU (or more specifically, the GPU's FBHUB) to convert the local die-relative engine ID ("local engine ID") included in the request by the source hardware engine to an equivalent globally unique engine ID ("global engine ID").

[0033] Each individual GPU, or more specifically, the frame buffer hub (FBHUB) of that GPU, may calculate an offset based on a dielet-identifying fuse signal and may replace the relative engine ID provided by the source hardware engine with an equivalent global engine ID determined based on the calculated offset. When responding to or sending an acknowledgment of a request to the source hardware engine, the FBHUB may reverse-map the global engine ID to a local engine ID equivalent.

[0034] In an embodiment of the present disclosure, a globally unique identifier is determined for each FBHUB instance (e.g., based on a die identification fuse signal) to facilitate request and message routing between hardware units in different dies of a multi-die GPU. Each FBHUB instance is represented by a unique node ID so that the FBHUB can know whether certain traffic is intended for use by itself (e.g., by one or more hardware units on its die) or forwarded to a remote FBHUB instance located on a different die.

[0035] FBHUB provides channels for certain units on its die to communicate directly with other units on different dies. Messages from a hardware engine on the FBHUB's die and directed to another hardware engine in a multi-die GPU are either sent to a destination within the same die or forwarded to a remote FBHUB where they can be processed and sent to the target hardware engine in the remote die. This channel essentially removes the physical die from the equation and allows communication between hardware units as if they were all on a single die. When communication is forwarded to a remote FBHUB instance, the target instance is also able and responsible for sending an ack (acknowledgement indication) back to the forwarding FBHUB instance. This communication channel and acknowledgment mechanism facilitates memory barrier operations (membars) across all die.

[0036] Some embodiments of the present disclosure include a hardware mechanism for deadlock-free, independent, software-triggered membars of a configurable scope. For example, software is enabled to write to an FBHUB register that is configured as a membar trigger. When the register is written, the FBHUB performs a membar of the scope determined by the register write without interrupting pending or ongoing hardware-initiated membars. The register written to trigger the membar can be a privileged register (e.g., a pri register). The FBHUB can acknowledge the software's membar request by writing to a readable status register and clearing the trigger.

[0037] Some embodiments may also include a high-priority direct interface that connects FBHUBs across dies to facilitate system-level membar synchronization. The high-priority direct interface bypasses high-traffic crossbar interfaces (e.g., the crossbar that interconnects the individual GPUs in a multi-die GPU) to reduce latency in FBHUB-to-FBHUB communication and ultimately the membar operation itself.

[0038] Regarding the hardware mechanism of membar, FBHUB can also implement remote refresh request confirmation handshake through a direct interface to push remote bare die traffic to a system-level consistency point. Independently operable finite state machines (FSMs) are implemented to separate the input / output (I / O) refresh phase of membar processing. The FBHUB design can include multiple consecutive but independently executable FSMs to segment the membar processing into scope-specific components, thereby improving merging and minimizing the time cost of membar.

[0039] With this segmented FSM design, the membar-based synchronous I / O refresh phase can be initiated by the direct memory access (DMA) engine on the local die, by a membar initiated by the local die GPC, or by an FBHUB located on a remote die performing its own I / O refresh.

[0040] Each FBHUB instance can simultaneously issue remote I / O refresh requests to remote FBHUBs and receive and process remote I / O refresh requests from remote FBHUBs. This simultaneous feature allows multi-die expansion without introducing interference between dies and avoiding mutual deadlock.

[0041] Figure 1 A multi-die GPU 100 according to some embodiments of the present disclosure is shown. The multi-die GPU 100 includes two dies—die-1 102a and die-2 102b. Each die includes a GPU, where die-1 102a has GPU-1 104a and die-2 102b has GPU-2 104b. The dies can be interconnected via one or more high-speed links 116. In some embodiments, the high-speed link is a chip-to-chip high-bandwidth interface (CTCHBI), an NVLink interface, or other high-bandwidth interface. Each multi-die GPU can have two or more dies, each having one or more GPUs. However, embodiments are not limited to the number of dies included in a multi-die GPU or the number of GPUs on each die. A multi-die GPU can also have any number of dies without GPUs.

[0042] Each GPU includes multiple hardware engines. For example, each die has its own one or more streaming multiprocessors (SMs) and other hardware engines that perform processing to support the processing executed on the SMs. In the illustrated embodiment, GPU-1 includes hardware engine-1 106a and hardware engine-2 108a, while GPU-2 includes hardware engine-1 106b and hardware engine-2 108b. Hardware engines can include compute units, graphics units, encoding / decoding units, encryption / decryption units, etc. Embodiments are not limited to any particular number of hardware engines or any type of hardware engines on each GPU.

[0043] Each GPU also includes a frame buffer hub (FBHUB). In the illustrated embodiment, GPU-1 104a includes FBHUB-1 110a, while GPU-2 104b includes FBHUB-2 110b. The FBHUBs on each die are interconnected with a high-speed connection 112. In some embodiments, circuitry 114, which includes at least a portion of each FBHUB in multi-die GPU 100, provides a monolithic view of multi-die GPU 100, enabling external entities (such as, for example, driver software 122 executing on CPU 120 communicating with multi-die GPU 100 via interface 118 (e.g., PCI interface, NVLink interface, etc.) to view multi-die GPU 100 as a monolithic GPU. In other words, the monolithic view implemented by circuitry 114 allows software (such as, for example, driver software 122) to be agnostic with respect to the specific architecture of multi-die GPU 100.

[0044] Figure 2 Another multi-die GPU 200 is shown, in accordance with some embodiments. Figure 2 Also schematically shown are hardware components 204 that may be included on each die and configured to provide the functionality described in this disclosure.

[0045] Multi-die GPU 200 includes four dies 202a, 202b, 202c, and 202d. In some embodiments, a crossbar switch 206 interconnects dies 202a, 202b, 202c, and 202d. In at least some embodiments, each die may have another crossbar switch 208 connecting the die to memory 205.

[0046] FBHUB 204 of die 202 a may include a hardware engine remapper 210, a register-triggered membar processor 212, a general-purpose membar processor 214, an FBHUB-to-FBHUB transmitter / receiver 216, and an external device controller 217. Similarly, each of the dies 202 b-202 d includes an FBHUB, each of which may include a respective hardware engine remapper, a register-triggered membar processor, a general-purpose membar handler, an FBHUB-to-FBHUB transmitter / receiver, and an external device controller configured to operate in the same or similar manner as the corresponding components described for die 202 a.

[0047] Engine remapper 210 on die 202 a can be configured to map the local engine IDs of the hardware engines on die 202 a to their corresponding globally unique engine IDs. In some embodiments, this mapping is performed based on an offset determined by an identifier assigned to die 202 a, where the assigned identifier is unique within multi-die GPU 200.

[0048] The engine remapper 210 can also be configured to perform reverse mapping of a globally unique engine ID (global engine ID) to a corresponding local hardware engine ID (local engine ID). That is, the remapper 210 on die 202 a is configured to perform reverse mapping of the global engine ID of the hardware engine on die 202 a to its corresponding local engine ID.

[0049] For example, remapper 210 can map memory requests sent by any memory requester 218 located in the same die or a memory requester external to the die.Memory requester 218 can include a hardware engine on die 202a.

[0050] Fuse 220 for each of die 202a-202d is set to specify a unique identifier assigned to the die within multi-die GPU 200. As described above, mapping and remapping between a hardware unit's local engine ID and its corresponding global engine ID is performed by remapper 210 using the die's unique identifier.

[0051] For example, each of the four dies in multi-die GPU 200 can be assigned a unique number between 0 and 3 (e.g., if the multi-die GPU has n dies, each individual die can be assigned a unique number between 0 and n-1). Given a configured offset size, in a multi-die GPU where each die is identified by a different number between 0 and n-1, the corresponding global engine ID for the local hardware engine ID p on the i-th die can be determined as p + (offset size) xi.

[0052] Register-triggered membar handling 212 and general membar handling 214 on die 202a enable FBHUB 204 to handle membar operations on die 202a. Membar requests received by FBHUB 204 can be initiated by a local hardware engine on die 202a, by dies 202b-202d, or by software (e.g., driver software on a CPU).

[0053] For membar requests initiated by the local hardware engine of die 202a, general membar processing device 214 generates membar requests to other dies 202b-202d as needed and coordinates the receipt of acknowledgments from the local request and any requests to other dies before responding to the requesting local hardware engine.

[0054] For membar requests received from any other die 202b-202d, the general membar processing device 214 performs the requested operation on the locally accessible memory 205 and returns an acknowledgment. In some embodiments, the general membar processing device 214 can also perform the requested operation on units handling external communication interfaces (e.g., HSHUBs) through the external device controller 217. The membar synchronization mechanism in this embodiment handles the matrix processing unit (MPU), but is not limited to this. The synchronization mechanism can be configured to enable any set of synchronization points "range."

[0055] For software-initiated membar operations, a mechanism is provided for software to notify FBHUB 204 to initiate a corresponding membar. Software can instruct FBHUB 204 to trigger a membar by sending a write request 222 to a register on die 202a (e.g., a privileged register (pri register)). The pri register can also specify the scope of the membar (e.g., local scope or system scope). Register-triggered membar processing device 212 can be configured to trigger a membar in response to the state of the pri register set by software.

[0056] The triggered membar is then processed in conjunction with the general membar processing means 214, and an ack 224 of the triggered membar with respect to the requested pri register is provided to the requesting software. In some embodiments, the ack to the requesting software can be provided by writing 224 to a register. This register can be monitored by the software requesting the triggered membar.

[0057] The FBHUB-to-FBHUB transmitter / receiver 216 on the FBHUB 204 can be configured to provide a channel for certain hardware units on the die 202a to communicate directly with other hardware units on different dies. Messages are either sent to a destination within the same die or forwarded to a remote FBHUB where they can be processed and sent to the target hardware engine on the remote die.

[0058] Each FBHUB instance can also be configured to forward acks to the corresponding FBHUB instance in the remote die if the acks are not intended for the FBHUB that received them. This mechanism may be needed to facilitate membar between all die in a multi-die GPU 200.

[0059] Figure 3 An example of a message showing that remapper 210 may perform remapping between globally unique engine IDs of hardware engines in multi-die GPU 200 and corresponding local engine identifiers is shown.

[0060] Message 302 may be a request sent by an external processor (e.g., a CPU) to multi-die GPU 100 or 200. It may include a request field 306 (which includes the request (e.g., an instruction)), a source identifier field 308 identifying the sender, and a destination identifier 310. Destination identifier 310 is a globally unique engine identifier that is intended to uniquely identify a hardware engine in multi-die GPU 200. The sender (e.g., driver software on the CPU) may know that there is a hardware engine in multi-die GPU 200 that can handle a particular request, but the sender may not know which specific die the hardware engine is located on. When there is more than one hardware engine of a particular type in multi-die GPU 200 (e.g., more than one DMA engine), the sender may know the number of such hardware engines in multi-die GPU 200, but not their arrangement on different dies. For example, if multi-die GPU 200 has five hardware DMA engines, the unified view provided by multi-die GPU 200 may require software to know that there are five DMA engines in the multi-die GPU, but does not require software to know how the DMA engines are distributed across the four dies (e.g., die 202a has the first two DMA engines, while dies 202b-202d each have one DMA engine). A globally unique engine ID enables a sender to uniquely identify a specific hardware engine in a multi-die GPU without having to know where the hardware engine is located in the multi-die GPU (e.g., on which die among several dies).

[0061] The message 302 may also include a remap flag 312 that instructs the FBHUB to remap the target address.

[0062] Message 304 may represent message 302 after being remapped by FBHUB. Message 304 includes the same request 306 and the same source identifier 308 as message 302, but the globally unique engine identifier 310 is replaced by the corresponding local engine identifier.

[0063] By providing this remapping capability in FBHUB, multi-GPU hardware enables software (e.g., driver software executing on the CPU) to interpret the multi-die GPU hardware as if it were a single GPU, regardless of how many dies are present and their individual composition. For example, as previously mentioned, if the same DMA engine is present in multiple dies, and all of these DMA engines are expected to operate as separate, independent engines, software must be able to uniquely identify each of these engines and their software-exposed engine identifiers. Events such as faults, interrupts, page table bindings, etc. can depend on this uniqueness.

[0064] As described above, each FBHUB can determine the engine identifier of the hardware unit on its die by applying an offset. The offset can be determined based on the unique identifier of the die. For example, for a hardware engine p present on each die 202a-202d, its globally unique engine identifier on each die can be calculated by applying a different offset to each die. As described above, the offset can be determined by each FBHUB based on the die identifier of its die. The die identifier is unique within a multi-die GPU. Each FBHUB can determine the die identifier based on the fuse settings of the die.

[0065] Figure 4A An example interaction diagram 400 is shown between a CPU, a first die of the multi-die GPU, and a second die of the multi-die GPU when a request from the CPU is received at the multi-die GPU.

[0066] An example remapping interaction may begin at operation 402. At operation 402, a request is received by each of two die in a multi-die GPU (e.g., multi-die GPU 100 or 200). The request may be received via a crossbar interface to which each die and / or its FBHUB is connected. The request may be in the form of Figure 3 The format 302 is shown in FIG, where the global engine ID is the address of the target hardware engine. The source of the request may be driver software executing on the CPU (eg, driver 122 executing on CPU 120).

[0067] Upon receiving the request, the FBHUB on each die checks whether the target hardware engine is local to its die. As described above, each FBHUB can determine its die ID based on fuse settings. According to some embodiments, each FBHUB determines the corresponding globally unique engine ID for each of its hardware engines by adding an offset based on the die ID to the local engine ID. In the example shown, the FBHUB on Die-2 determines that the request is not intended for that die and ignores it, while the FBHUB on Die-1 recognizes that the request is intended for its die and continues to process the received request further.

[0068] Therefore, at operation 404, the FBHUB on Die-1 remaps the global engine ID in the request to the corresponding local engine ID. Figure 3 An example request 302 is provided with a global engine ID, and a remapped request 304 that replaces the global engine ID with a corresponding local engine ID.

[0069] At operation 406, the remapped request is transmitted to the target hardware engine identified by the local engine ID, and an ack or fault is received back by the target hardware engine at the FBHUB at operation 408. Embodiments are not limited to any particular set of requests or any particular set or type of hardware engines.

[0070] In operation 410, the FBHUB maps the received ack or fault (which identifies its source by the local engine ID) to replace or otherwise include the corresponding global engine ID. In operation 412, the FBHUB of Die-1 transmits the mapped ack or fault notification to the CPU. Identifying the source of any ack or fault notification by the corresponding global engine ID enables the CPU or other sender of the corresponding request to take appropriate processing steps. For example, if a fault notification is received in response to a request from the CPU, the CPU can reset the hardware engine that issued the fault notification.

[0071] Figure 4B An example interaction diagram 420 is shown between a CPU, a first die of a multi-die GPU, and a second die of a multi-die GPU when a request from a hardware engine is received at a FBHUB in the multi-die GPU.

[0072] The example mapping interaction begins at operation 422. At operation 422, a hardware engine with local engine ID x on die-1 issues a request for services required by another hardware engine on a multi-die GPU. The request is received by the FBHUB on die-1, which maps the source address from local engine ID x to its corresponding global engine ID at operation 424. The mapping has been described above.

[0073] At operation 426, the mapped request (i.e., a request whose source address is mapped from a local engine ID to a corresponding global engine ID) is transmitted by the FBHUB on die-1 on the crossbar switch to the other dies in the multi-die GPU. According to an example, if the request is for accessing memory, each FBHUB can determine whether it is for a local DMA hardware engine based on the memory address of the request.

[0074] In operation 428, the FBHUB on Die-2 detects that the request is for one of its local hardware engines and transmits the request to the appropriate local hardware engine. If the request includes the target hardware engine ID as the global engine ID, remapping can be performed by the FBHUB on Die-2 before transmitting 428.

[0075] In operation 430, the target hardware engine returns an ack or fault notification in response to request 428. In operation 432, the FBHUB on Die-2 performs a mapping to map the local engine ID of the hardware engine that issued the ack or fault to its global equivalent. The ack is then transmitted to the FBHUB on Die-1 at operation 434. The ack or fault notification received by the FBHUB on Die-1 can have both its target engine and source engine identified by their corresponding global engine IDs.

[0076] At operation 436, the FBHUB on die-1 remaps at least the destination address of the ack or fault notification from the global engine ID to the corresponding local engine ID, and then transmits it to the request-initiating hardware engine at operation 438. It should be noted that 430 may not be limited to acks or fault indications, and may include, for example, other types of responses triggered by the requested operation itself (e.g., data from a memory read).

[0077] The FBHUB is responsible for ensuring that the membar.gpu operation pushes all previous memory accesses from all FBHUB clients (e.g., DMA, front-end, etc.) to GPU-wide visibility. The FBHUB is also responsible for ensuring that the membar.sys operation pushes all previously GPU-wide visible requests (from all GPU clients, including SMs) to system-wide visibility. The membar.gpu and membar.sys operations are performed to provide a consistent view of the memory model that multiple processes / threads can rely on when sharing memory. In example embodiments of a multi-die GPU (e.g., multi-die GPU 100 or 200), the FBHUB of the GPU receiving the request ensures that related operations executed by any other GPU in the multi-die GPU are pushed to a consistency point. Because in some embodiments, multi-die GPUs have unified memory where GPUs can directly access each other's memory, multiple consistency points can exist depending on where the request is directed. The FBHUB of the GPU that receives or triggers the initial membar operation waits for acks from all destinations of the membar. Once all acks are received, the FBHUB confirms that the memory barrier has completed. After the FBHUB has acknowledged the memory barrier, the source of the memory barrier request (eg, a hardware or software source) or any other entity can safely assume that the request prior to the memory barrier has taken effect or will take effect.

[0078] This process requires coordination between all hardware engines and all memories distributed across multiple dies in a multi-die GPU. The membar command is routed to all reachable dies. For example, the membar command is sent to all dies reachable through a crossbar, and all acks in response to the membar command are collected.

[0079] After all acks have been received for the membar commands to ensure their GPU-wide visibility, the next sequence can then be issued to ensure their system-wide visibility (if the original membar command had a system-wide requirement). The next sequence is used to flush I / O. I / O flushing is to ensure that the I / O, in some embodiments, the interconnect between the die / GPU (e.g., external communication interface such as NVLINK PCI, etc.), has been flushed for the previous request. This is to ensure that the request before the membar command has been acknowledged by the target / peer GPU.

[0080] Since I / O is split across multiple dies (e.g., 2 dies in a multi-die GPU 100), this process requires coordination. The coordination process is as follows:

[0081] The FBHUB that receives the membar request issues an I / O refresh request to its own I / O (i.e., the I / O on the same die) and also issues an I / O refresh request to the second die via the FBHUB-to-FBHUB point-to-point connection. The FBHUB on the second die issues an I / O refresh request to the I / O entity on that die, collects the corresponding ACKs, and confirms the request from the first FBHUB.

[0082] There may be situations where more than one FBHUB in a multi-die GPU is executing a memory barrier at a particular time. In this case, since two or more memory barriers are being executed concurrently, steps need to be taken to ensure that they occur in a deadlock-free manner (e.g., without circular dependencies).

[0083] In some embodiments, each FBHUB runs two separate state machines—the first state machine controls local memory barrier I / O refreshes, while the second state machine controls remote memory barrier I / O refresh operations to the remote FBHUB. This provides an operating mode where each die can handle local membar I / O refresh operations and membar I / O refresh requests from another die in parallel and independently.

[0084] – Assume that the IO refresh FSM can be called by multiple clients to execute their own IO refresh commands. When the FSM receives an IO refresh request from a local client, it must also send requests to other die. Processing of the local IO refresh can continue without waiting for responses from other die. On each other die, the FBHUB in that die will issue an IO refresh request to the local client, collect the ACKs from the local clients, and send an ACK to the requesting die, indicating the completion of the I / O refresh.

[0085] While waiting for the ack from the remote FBHUB, the FSM can continuously process IO flush requests from other entities (such as SM). The source of memory can come from many places, for example, hardware engines, syspipe, or even through software.

[0086] For multi-die GPUs, a PRI-based view is provided so that software can issue membar commands through the PRI interface on any one or more of the FBHUBs. This is a way for software to quickly inject membars without using dedicated hardware units. Through PRI-based registers, software can trigger membars on any die. In the case of software-initiated membars through PRI registers, software is responsible for triggering membars on any group of die or all dies.

[0087] Figure 5A A diagram illustrating example interactions between a CPU, a first die of a multi-die GPU, and a second die of the multi-die GPU when a memory request from a hardware engine of the multi-die GPU is received at the FBHUB of the multi-die GPU.

[0088] At operation 502, a source hardware engine on die-1 sends one or more memory requests. For example, one or more threads executing on an SM on die-1 may issue memory request 502. Following memory request 502, the source hardware engine ID may issue a memory barrier (membar) request 504. The FBHUB on die-1 receives the memory request and membar request and, at operation 506, maps the source address from the local engine ID to the corresponding global ID.

[0089] The mapped memory request and mapped membar request are communicated to the other die at operations 508 and 510, respectively. In this case, the mapped request and mapped membar are received at the FBHUB on die-2.

[0090] In operation 512, the FBHUB on Die-2 determines the appropriate hardware engine and transmits the memory request to the hardware engine unit. In operation 514, the hardware unit returns an ack or fault notification corresponding to the memory request to the FBHUB on Die-2.

[0091] At operation 516, the FBHUB on die-2 monitors whether it has received acks for all memory requests destined for its local hardware engines. When it is determined that all acks have been received, at operation 518, the FBHUB on die-1 sends an ack and clears the membar on die-2.

[0092] At operation 520, the FBHUB on die-1 monitors whether it has received acks for all memory requests destined for its local hardware engine. When it is determined that all acks have been received, at operation 522, the acks are remapped to convert the global engine ID to the corresponding local engine ID of the source hardware engine. At operation 524, the remapped ack is sent to the source hardware engine ID.

[0093] Figure 5B A diagram illustrates example interactions between a CPU, a first die of a multi-die GPU, and a second die of the multi-die GPU when software on the CPU sends one or more memory requests to the multi-die GPU and subsequently sets a pri register to trigger a membar operation.

[0094] Example interaction 530 can begin at operation 532, whereby software executing on a CPU can trigger a membar on a multi-die GPU. At operation 532, the software issues a memory request, which is received at the FBHUB of die-1, which determines that the request is destined for a local hardware engine. At operation 538, the FBHUB can transmit the request to the local hardware engine after mapping the target engine ID from the global ID specified in the received request to the local engine ID.

[0095] As described above, determining whether to process a memory request locally can be based on the memory address or memory address range in the received request. Thus, if the FBHUB on Die-1 determines that a portion of the request involves another die, it sends the request to the FBHUB on Die-2 at operation 540. At operation 544, the FBHUB on Die-2 transmits the request to its local hardware engine after performing any remapping of the target engine ID as needed.

[0096] In response to operation 538, the target hardware engine on Die-1 returns an ack at operation 550. Similarly, in response to operation 544, the target hardware engine on Die-2 returns an ack to the FBHUB on Die-2 at operation 546. At operation 552, the FBHUB on Die-2, after performing any mapping of the source hardware address as needed, sends an ack to acknowledge the request received at operation 540.

[0097] For the transferred memory request, software may cause a membar to be triggered by setting a register accessible to the FBHUB on die- 1 at operation 534. For example, a dedicated privilege register (pri register) may be set.

[0098] At operation 536, after detecting the write to the register, the FBHUB on Die-1 triggers membar and accordingly monitors for an ack corresponding to the memory request at operation 554. Since the FBHUB on Die-1 has already transmitted the memory request to the second die, the FBHUB on Die-1 also transmits the membar request to the FBHUB on Die-2 at operation 542. In response to the membar request, the FBHUB on Die-2 starts monitoring for an ack corresponding to the memory request at operation 548.

[0099] When monitoring at 548 determines that the required ack has been received, the FBHUB on die-2 sends an ack to the FBHUB on die-1 at operation 552. When monitoring at 554 determines that the locally issued memory request is acknowledged at 550 and the memory request to die-2 is acknowledged at 552, the FBHUB on die-1 determines that the membar has completed and sets the ack register at operation 556. Software can periodically read the ack register to determine whether the membar has completed. In some embodiments, external interface units of the MPU or HSHUB type are distributed across multiple dies. While in a single-die system, synchronization across MPUs means seeing that all required traffic travels through one point in the hardware, in a multi-die system according to embodiments of the present disclosure, software can continue to operate as if there is a single synchronization point in the hardware, even though it is now a single point in each of the multiple dies (i.e., multiple points in a multi-die system).

[0100] Figure 6 A dual finite state machine (FSM) hardware arrangement 600 for input / output (I / O) refresh operations in a membar process is shown. The refresh operation can be part of the membar operation.

[0101] A dual FSM hardware arrangement 600 is provided for each FBHUB to ensure that I / O is flushed after a membar operation. The hardware arrangement 600 includes a first FSM 602 and a second FSM 604. The first FSM 602 tracks membar requests from local hardware engines (e.g., from GPCs and video memory). The second FSM 604 tracks membar requests from remote FBHUBs.

[0102] The local FSM 602 can track each membar issued by a local hardware engine (e.g., GPC or video memory). For each membar request, the FSM 602 issues a refresh request 608 to the associated local I / O. In some embodiments, the local refresh request is sent to an external interface controller arbiter (e.g., HSHUB) 606. The external interface controller arbiter 606 transmits the refresh request 610 and collects the corresponding refresh ack. The external interface controller arbiter 606 returns the refresh ack to the requesting FSM 602.

[0103] Additionally, if the local FSM 602 determines that memory traffic has been routed from a remote die to an external interface, the local FSM 602 issues a remote refresh request 612 to one or more remote dies, and the corresponding acks 612 are monitored.

[0104] The second FS M 604 services refresh requests from the remote die. For example, for each refresh request 614 received from the remote die, the FSM 604 issues one or more refresh requests 616 to the local hardware engine and collects the corresponding ACKs. The external interface controller arbiter 606 transmits the refresh requests 618 to the corresponding hardware engine and collects the corresponding ACKs, which are then returned to the FSM 604.

[0105] The dual-state machine design enables each FBHUB instance to simultaneously issue remote I / O refresh requests to the remote FBHUB and receive and process remote I / O refresh requests from the remote FBHUB. This simultaneous feature allows multi-die expansion without introducing interference between dies and avoiding mutual deadlock.

[0106] In an example scenario, software wants to issue DMA commands to move some memory traffic spread across two dies in a multi-die GPU. According to an example embodiment, the FBHUB allows software to simply issue some memory traffic and issue a membar command with system memory scope. The FBHUB in the hardware must understand how to communicate between all dies so that it covers all engines on any die.

[0107] Without the FBHUB in the example embodiment, the software must know that the relevant memory involves accessing two dies, issue the corresponding commands to both dies, and handle synchronization on its own (for example, both transfers are at a consistency point, and a separate membar command is issued for each die). The software knows that there are two DMA engines, but does not know where they are (that is, which die). Each DMA engine also does not know its own engine ID in a multi-die GPU configuration, only its local engine ID. The FBHUB does the mapping. The fuse settings determine the ID of the FBHUB.

[0108] It is important to understand that the mapping can be for commands received from a driver running on the CPU or from a thread running on a streaming multiprocessor (SM) of the GPU. The addition of hardware components on the FBHUB hides the complexity of the software design.

[0109] During operation, it can be one software context spread across several die in a multi-die GPU.

[0110] The dies may be of the same design or may be of different generations, sizes, etc. (eg, stacked / asymmetric, concentrated die, etc.) One or more dies may have one or more GPUs, enabling GPUs of arbitrary complexity.

[0111] Figure 7 An example GPU 700 is shown on a die of a multi-die GPU according to some embodiments of the present disclosure, with its frame buffer hub (FBHUB) 702 and high-speed hub (HSHUB) 704 identified along with some of their interconnections. GPU 700 includes two processors 705-1 and 705-2, each of which includes multiple processing units 710 connected to cache memory 715. Respective crossbar switches 712-1 and 712-2 interconnect the processing units 710 and memory 715 in processors 705-1 and 705-2. High-speed hub (HSHUB) 718 connects all processing units 710 and cache memory 715. The HSBUB can also be a controller for communicating with external devices / interfaces. Frame buffer hub (FBHUB) 720 provides access to system and / or host memory 722 for processing units 710, which are connected to FBHUB 720 via HSHUB 718 or directly through crossbar network 712-2.

[0112] Example GPU Architecture

[0113] An example illustrative architecture will now be described in which remapping and synchronization are performed efficiently. The following information is for illustrative purposes only and should not be construed as limiting in any way. Any of the following features may optionally be combined with the other features described, or not exclude the other features described.

[0114] Figure 8 A parallel processing unit (PPU) 800 is shown according to one embodiment. In one embodiment, the PPU 800 is a multi-threaded processor implemented on one or more integrated circuit devices. The PPU 800 is a latency-hidden architecture designed for processing many threads in parallel. A thread (e.g., an execution thread) is an instantiation of an instruction set configured to be executed by the PPU 800. In one embodiment, the PPU 800 is a graphics processing unit (GPU) that is configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device (such as a liquid crystal display (LCD) device). In other embodiments, the PPU 800 can be used to perform general-purpose computations. In some other embodiments, the PPU 800 is configured to implement large neural networks in deep learning applications or other high-performance computing applications.

[0115] One or more PPUs 800 can be configured to accelerate thousands of high-performance computing (HPC), data center, and machine learning applications. PPUs 800 can be configured to accelerate numerous deep learning systems and applications, including autonomous vehicle platforms, deep learning, high-precision speech, image, and text recognition systems, intelligent video analysis, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations, among others.

[0116] like Figure 8As shown, the PPU 800 includes an input / output (I / O) unit 805, a front-end unit 815, a scheduler unit 820, a work distribution unit 825, a hub 830, a crossbar switch (Xbar) 870, one or more general processing clusters (GPCs) 850, and one or more memory partition units (MPUs) 880. The PPU 800 can be connected to a host processor or other PPUs 800 via one or more high-speed NVLink 810 interconnects. The PPU 800 can be connected to a host processor or other peripheral devices via an interconnect 802. The PPU 800 can also be connected to a memory including a plurality of memory devices 804. In one embodiment, the memory 804 can include a plurality of dynamic random access memory (DRAM) devices. The DRAM devices can be configured as a high-bandwidth memory (HBM) subsystem, in which multiple DRAM dies are stacked within each device.

[0117] The NVLink 810 interconnect enables the system to scale and include one or more PPUs 800 in conjunction with one or more CPUs, supporting cache coherency between the PPU 800 and the CPU, and CPU mastering. Data and / or commands can be sent by the NVLink 810 through the hub 830 to or from other units of the PPU 800, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). Figure 11A and Figure 11B NVLink 810 is described in more detail.

[0118] The I / O unit 805 is configured to send and receive communications (e.g., commands, data, etc.) from a host processor (not shown) via the interconnect 802. The I / O unit 805 can communicate directly with the host processor via the interconnect 802, or through one or more intermediate devices (such as a memory bridge). In one embodiment, the I / O unit 805 can communicate with one or more other processors (e.g., one or more PPUs 800) via the interconnect 802. In one embodiment, the I / O unit 805 implements a Peripheral Component Interconnect Express (PCIe) interface for communicating over a PCIe bus, and the interconnect 802 is a PCIe bus. In alternative embodiments, the I / O unit 805 can implement other types of known interfaces for communicating with external devices.

[0119] The I / O unit 805 decodes data packets received via the interconnect 802. In one embodiment, the data packets represent commands configured to cause the PPU 800 to perform various operations. The I / O unit 805 sends the decoded commands to various other units of the PPU 800 as specified by the commands. For example, some commands may be sent to the front end unit 815. Other commands may be sent to the hub 830 or other units of the PPU 800, such as one or more copy engines, video encoders, video decoders, a power management unit, etc. (not explicitly shown). In other words, the I / O unit 805 is configured to route communications between and among the various logical units of the PPU 800.

[0120] In one embodiment, a program executed by a host processor encodes a command stream in a buffer that provides a workload to the PPU 800 for processing. The workload may include many instructions and data to be processed by those instructions. A buffer is an area in memory that is accessible (i.e., read / write) by both the host processor and the PPU 800. For example, the I / O unit 805 can be configured to access a buffer in system memory connected to the interconnect 802 via a memory request transmitted through the interconnect 802. In one embodiment, the host processor writes a command stream into the buffer and then sends a pointer to the start of the command stream to the PPU 800. The front end unit 815 receives pointers to one or more command streams. The front end unit 815 manages the one or more streams, reads commands from the streams, and forwards the commands to the various units of the PPU 800.

[0121] The front-end unit 815 is coupled to the scheduler unit 820, which configures the various GPCs 850 to process tasks defined by one or more streams. The scheduler unit 820 is configured to track state information related to the various tasks managed by the scheduler unit 820. The state may indicate which GPC 850 the task is assigned to, whether the task is active or inactive, the priority associated with the task, etc. The scheduler unit 820 manages the execution of multiple tasks on one or more GPCs 850.

[0122] Scheduler unit 820 is coupled to work distribution unit 825, which is configured to dispatch tasks for execution on GPCs 850. Work distribution unit 825 can track a number of scheduled tasks received from scheduler unit 820. In one embodiment, work distribution unit 825 manages a pending task pool and an active task pool for each GPC 850. The pending task pool can include a number of time slots (e.g., 32 time slots) containing tasks assigned to be processed by a particular GPC 850. The active task pool can include a number of time slots (e.g., 4 time slots) for tasks being actively processed by GPC 850. When a GPC 850 completes execution of a task, the task is evicted from the active task pool of GPC 850, and one of the other tasks from the pending task pool is selected and scheduled for execution on GPC 850. If an active task on GPC 850 has become idle, such as while waiting for a data dependency to be resolved, the active task may be evicted from GPC 850 and returned to the pending task pool, while another task in the pending task pool is selected and scheduled for execution on GPC 850 .

[0123] Work distribution unit 825 communicates with one or more GPCs 850 via XBar (crossbar) 870. XBar 870 is an interconnect network that couples many units of PPU 800 to other units of PPU 800. For example, XBar 870 can be configured to couple work distribution unit 825 to a specific GPC 850. Although not explicitly shown, one or more other units of PPU 800 can also be connected to XBar 870 via hub 830.

[0124] Tasks are managed by the scheduler unit 820 and dispatched to the GPCs 850 by the work distribution unit 825. The GPCs 850 are configured to process tasks and generate results. The results can be consumed by other tasks within the GPC 850, routed to a different GPC 850 via the XBar 870, or stored in the memory 804. The results can be written to the memory 804 via the partition unit 880, which implements a memory interface for reading data from and writing data to the memory 804. The results can be sent to another PPU 804 or CPU via the NVLink 810. In one embodiment, the PPU 800 includes a number U of partition units 880, which is equal to the number of independent and different memory devices 804 coupled to the PPU 800. Figure 9B The partition unit 880 is described in more detail.

[0125] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API) that enables one or more applications to be executed on the host processor to schedule operations for execution on the PPU 800. In one embodiment, multiple computing applications are executed simultaneously by the PPU 800, and the PPU 800 provides isolation, quality of service (QoS), and independent address spaces for the multiple computing applications. The application can generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks for execution by the PPU 800. The driver kernel outputs the tasks to one or more streams being processed by the PPU 800. Each task can include one or more related thread groups, referred to herein as warps. In one embodiment, a warp includes 32 related threads that can execute in parallel. Collaborating threads can refer to multiple threads that include instructions to perform tasks and can exchange data through shared memory (SMEM). According to some embodiments, threads, cooperative threads, and hierarchical groupings of threads (e.g., cooperative thread arrays (CTAs) and cooperative group arrays (CGAs)) are described in more detail in U.S. application Ser. No. 17 / 691,621, the contents of which are hereby incorporated by reference. According to some embodiments, SMEM is described in U.S. application Ser. No. 17 / 691,690, which is hereby incorporated by reference.

[0126] Figure 9A According to one embodiment, Figure 8 PPU 800 GPC 850. Figure 9A As shown, each GPC 850 includes multiple hardware units for processing tasks. In one embodiment, each GPC 850 includes a pipeline manager 910, a pre-raster operation unit (PROP) 915, a raster engine 925, a work distribution crossbar (WDX) 980, a memory management unit (MMU) 990, and one or more data processing clusters (DPCs) 920. It should be understood that Figure 9A The GPC 850 may include instead Figure 9A Other hardware units or other than the units shown in Figure 9A Other hardware units besides those shown in .

[0127] In one embodiment, the operation of GPC 850 is controlled by pipeline manager 910. Pipeline manager 910 manages the configuration of one or more DPCs 920 for processing tasks assigned to GPC 850. In one embodiment, pipeline manager 910 can configure at least one of one or more DPCs 920 to implement at least a portion of a graphics rendering pipeline, a neural network, and / or a compute pipeline. For example, with respect to the graphics rendering pipeline, DPC 920 can be configured to execute vertex shading programs on programmable streaming multiprocessors (SMs) 940. Pipeline manager 910 can also be configured to route packets received from work distribution units 925 to appropriate logic units within GPC 850. For example, some packets can be routed to fixed-function hardware units within PROP 915 and / or raster engine 925, while other packets can be routed to DPC 920 for processing by primitive engines 935 or SMs 940.

[0128] PROP unit 915 is configured to route data generated by raster engine 925 and DPC 920 to the raster operations (ROP) unit, in conjunction with Figure 9B The PROP unit 915 may also be configured to perform color blending optimizations, organize pixel data, perform address translations, and the like.

[0129] The GPC 850 includes a DPC 920, a controller (MPC) 930, a primitive engine 935, and one or more SMs 940. The MPC 930 controls the operation of the DPC 920 and routes data packets received from the pipeline manager 910 to the appropriate units in the DPC 920. For example, data packets associated with vertices may be routed to the primitive engine 935, which is configured to fetch vertex attributes associated with the vertices from the memory 804. Conversely, data packets associated with shading programs may be sent to the SM 940.

[0130] SM 940 includes a programmable streaming processor configured to process tasks represented by multiple threads. Each SM 940 is multi-threaded and configured to simultaneously execute multiple threads (e.g., 32 threads) from a specific thread group. In one embodiment, SM 940 implements a SIMT (single instruction, multiple thread) architecture, in which each thread in a thread group (e.g., a warp) is configured to process a different data set based on the same instruction set. All threads in a thread group execute the same instructions. In another embodiment, SM 940 implements a SIMT (single instruction, multiple thread) architecture, in which each thread in a thread group is configured to process a different data set based on the same instruction set, but in which individual threads in a thread group are allowed to diverge during execution. In one embodiment, a program counter, call stack, and execution state are maintained for each warp, enabling concurrency between warps and serial execution within warps when threads within the warp diverge. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, thereby achieving equal concurrency between all threads within and between warps. When execution state is maintained for each individual thread, threads executing the same instruction can be converged and executed in parallel for maximum efficiency. Figure 10A SM 940 is described in more detail.

[0131] The MMU 990 provides an interface between the GPC 850 and the partition unit 880. The MMU 990 can provide virtual to physical address translation, memory protection, and arbitration of memory requests. In one embodiment, the MMU 990 provides one or more translation lookaside buffers (TLBs) for performing translations from virtual addresses to physical addresses in memory 804.

[0132] Figure 9B According to one embodiment, Figure 8 The memory partition unit 880 of the PPU 800. Figure 9B As shown, the memory partition unit 880 includes a raster operations (ROP) unit 950, a level 2 (L2) cache 960, and a memory interface 970. The memory interface 970 is coupled to the memory 804. The memory interface 970 can implement a 32-, 64-, 128-, or 1024-bit data bus for high-speed data transfer. In one embodiment, the PPU 800 incorporates U memory interfaces 970, one for each pair of partition units 880, where each pair of partition units 880 is connected to a corresponding memory device 804. For example, the PPU 800 can be connected to up to Y memory devices 804, such as a high-bandwidth memory stack or graphics double data rate version 5 synchronous dynamic random access memory or other types of persistent memory.

[0133] In one embodiment, memory interface 970 implements an HBM2 memory interface and Y is equal to half a U. In one embodiment, the HBM2 memory stack is located on the same physical package as the PPU 800, providing significant power and area savings compared to traditional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory dies and Y is equal to 4, where each die in the HBM2 stack includes two 128-bit channels, for a total of 8 channels and a data bus width of 824 bits.

[0134] In one embodiment, memory 804 supports single-error correction with double-error detection ("SECDED") error correction code ("ECC") to protect data. ECC provides increased reliability for computer applications that are sensitive to data corruption. Reliability is particularly important in large-scale cluster computing environments where PPU 800 processes very large data sets and / or long-running applications.

[0135] In one embodiment, the PPU 800 implements a multi-level memory hierarchy. In one embodiment, the memory partitioning unit 880 supports unified memory to provide a single unified virtual address space for the CPU and PPU 300 memory, thereby enabling data sharing between virtual memory systems. In one embodiment, the frequency of PPU 800 accesses to memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 800 where the pages are accessed more frequently. In one embodiment, NVLink 810 supports address translation services, allowing the PPU 800 to directly access the CPU's page tables and providing full access to the CPU's memory through the PPU 800.

[0136] In one embodiment, the copy engine transfers data between multiple PPUs 800 or between a PPU 800 and a CPU. The copy engine can generate a page fault for an address that is not mapped to a page table. The memory partition unit 880 can then service the page fault, mapping the address to a page table, after which the copy engine can perform the transfer. In conventional systems, memory is fixed (e.g., non-pageable) for multiple copy engine operations between multiple processors, significantly reducing the available memory. With hardware page faults, addresses can be passed to the copy engine regardless of whether the memory page is resident, and the copy process is transparent.

[0137] Data from memory 804 or other system memory can be retrieved by memory partition unit 880 and stored in L2 cache 960, which is located on-chip and shared between various GPCs 850. As shown, each memory partition unit 880 includes a portion of L2 cache 960 associated with the corresponding memory device 804. Then, low-level caches can be implemented in various units within GPC 850. For example, each SM 940 can implement a level 1 (L1) cache. The L1 cache is private memory dedicated to a specific SM 940. Data from L2 cache 960 can be retrieved and stored in each L1 cache for processing in the functional units of SM 940. L2 cache 960 is coupled to memory interface 970 and XBar 870.

[0138] The ROP unit 950 performs graphics raster operations associated with pixel color, such as color compression, pixel blending, etc. The ROP unit 950 also implements depth testing in conjunction with the raster engine 925, receiving the depth of the sample position associated with the pixel fragment from the culling engine of the raster engine 925. The depth is tested against the corresponding depth in the depth buffer of the sample position associated with the fragment. If the fragment passes the depth test for the sample position, the ROP unit 950 updates the depth buffer and sends the result of the depth test to the raster engine 925. It should be understood that the number of partition units 880 can be different than the number of GPCs 850, and therefore, each ROP unit 950 can be coupled to each GPC 850. The ROP unit 950 tracks the packets received from the different GPCs 850 and determines to which GPC 850 the results generated by the ROP unit 950 are routed via the Xbar 870. Although in Figure 9B In the embodiment, ROP unit 950 is included in memory partition unit 880, but in other embodiments, ROP unit 950 may be located outside of memory partition unit 880. For example, ROP unit 950 may be located in GPC 850 or another unit.

[0139] Figure 10A According to one embodiment, Figure 9A Streaming multiprocessor 940. Figure 10B Conceptually illustrates an example of a Figure 10A The four sub-partitions 1091-1094 are implemented in an SM such as the SM shown in FIG. Figure 10AAs shown, SM 640 includes an instruction cache 1005, one or more scheduler units 1010 (K), a register file 1020, one or more processing cores 1050, one or more special function units (SFUs) 1052, one or more load / store units (LSUs) 1054, an interconnection network 1080, and a SMEM / L1 cache 1070.

[0140] As described above, the work distribution unit 825 schedules tasks for execution on the GPCs 850 of the PPU 800. Tasks are assigned to specific DPCs 920 within the GPCs 850 and, if the task is associated with a shader program, may be assigned to the SMs 940. The scheduler unit 1010(K) receives tasks from the work distribution unit 825 and manages the scheduling of instructions for one or more thread blocks assigned to the SMs 940. The scheduler unit 1010(K) schedules the thread blocks for execution as warps of parallel threads, where each thread block consists of at least one warp. In one embodiment, each warp includes 32 threads. The scheduler unit 1010(K) can manage multiple different thread blocks, assigning different thread blocks to different warps, and then dispatching instructions from multiple different cooperative groups to various functional units (e.g., cores 1050, SFUs 1052, and LSUs 1054) during each clock cycle.

[0141] The Cooperative Group Array (CGA) provides a programming model for organizing groups of communicating threads that allows developers to express the granularity at which threads are communicating, enabling the expression of richer and more efficient decompositions of parallelism. The cooperative launch API supports synchronization between thread blocks to execute parallel algorithms. The conventional programming model provides a single simple construct for synchronizing cooperating threads: a barrier across all threads of a thread block (e.g., the syncthreads() function). However, programmers often want to define thread groups at a granularity smaller than the thread block granularity and synchronize within the defined group, enabling higher performance, design flexibility, and software reuse in the form of a collective group-wide function interface.

[0142] Cooperative group arrays enable programmers to explicitly define thread groups at sub-block (e.g., as small as a single thread) and multi-block granularity and perform collective operations on threads, such as synchronization in cooperative groups. The programming model supports clean composition across software boundaries so that libraries and utility functions can safely synchronize in their local environment without making assumptions about convergence. Cooperative group array primitives enable new patterns of cooperative parallelism, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across an entire grid of thread blocks. Hierarchical grouping of threads (e.g., cooperative thread arrays (CTAs) and cooperative group arrays (CGAs)) according to some embodiments is described in more detail in U.S. application Ser. No. 17 / 691,621, the entire contents of which are incorporated herein by reference.

[0143] The dispatch unit 1015 is configured to deliver instructions to one or more functional units. In this embodiment, the scheduler unit 1010(K) includes two dispatch units 1015, which enable scheduling of two different instructions from the same warp during each clock cycle. In alternative embodiments, each scheduler unit 1010(K) may include a single dispatch unit 1015 or additional dispatch units 1015.

[0144] Each SM 940 includes a register file 1020 that provides a set of registers for the functional units of the SM 940. In one embodiment, register file 1020 is partitioned between each functional unit so that each functional unit is allocated a dedicated portion of register file 1020. In another embodiment, register file 1020 is partitioned between different warps organized by the SM 940. Register file 1020 provides temporary storage for operands connected to the data paths of the functional units.

[0145] Each SM 940 includes multiple processing cores 1050. In one embodiment, the SM 940 includes a large number (e.g., 128, etc.) of different processing cores 1050. Each core 1050 may include a fully pipelined, single-precision, double-precision, and / or mixed-precision processing unit, including a floating-point arithmetic logic unit (FPU) and an integer arithmetic logic unit (ILU). In one embodiment, the FPU implements the IEEE 754-2008 standard for floating-point operations.

[0146] Tensor cores are configured to perform matrix operations, and in one embodiment, one or more tensor cores are included in core 1050. Specifically, tensor cores are configured to perform deep learning matrix operations, such as convolution operations for neural network training and inference.

[0147] In some embodiments, the transposition hardware is included in the processing core 1050 or another functional unit (e.g., SFU 1052 or LSU 1054) and is configured to generate diagonally stored matrix data and / or generate the original matrix and / or the transposed matrix from the diagonally stored matrix data. The transposition hardware can be provided within the SMEM 1070 to load the register file 1020 of the SM 940 into the path.

[0148] Each SM 940 also includes multiple SFUs 1052 that perform special functions (e.g., attribute evaluation, reciprocal square root, etc.). In one embodiment, the SFUs 1052 may include a tree traversal unit (e.g., TTU 943) configured to traverse a hierarchical tree data structure. In one embodiment, the SFUs 1052 may include a texture unit (e.g., texture unit 942) configured to perform texture map filtering operations. In one embodiment, the texture unit is configured to load a texture map (e.g., a 2D array of texels) from memory 804 and sample the texture map to generate sampled texture values ​​for use in shader programs executed by the SM 940. In one embodiment, the texture map is stored in the SMEM / L1 cache 970. The texture unit performs texture operations, such as filtering operations using mip maps (i.e., texture maps at different levels of detail). In one embodiment, each SM 940 includes two texture units.

[0149] Each SM 940 also includes multiple LSUs 1054 that perform load and store operations between the SMEM / L1 cache 1070 and the register file 1020. Each SM 940 includes an interconnect network 1080 that connects each functional unit to the register file 1020 and connects the LSUs 1054 to the register file 1020 and the SMEM / L1 cache 1070. In one embodiment, the interconnect network 1080 is a crossbar switch that can be configured to connect any functional unit to any register in the register file 1020 and to connect the LSUs 1054 to memory locations in the register file and the SMEM / L1 cache 1070.

[0150] The SMEM / L1 cache 1070 is an on-chip memory array that allows data storage and communication between the SM 940 and the primitive engine 935, as well as between threads in the SM 940. In one embodiment, the SMEM / L1 cache 1070 includes 128KB of storage capacity and is in the path from the SM 940 to the partition unit 880. The SMEM / L1 cache 1070 can be used to cache reads and writes. One or more of the SMEM / L1 cache 1070, the L2 cache 960, and the memory 804 is a backing store.

[0151] Combining data cache and SMEM functionality into a single memory block provides the best overall performance for both types of memory access. This capacity can be used by programs as a cache that does not utilize SMEM. For example, if SMEM is configured to use half its capacity, texture and load / store operations can use the remaining capacity. Integration within SMEM / L1 cache 1070 enables SMEM / L1 cache 1070 to function as a high-throughput pipeline for streaming data, while providing both high-bandwidth and low-latency access to frequently reused data.

[0152] In the context of the present disclosure, an SM or "streaming multiprocessor" refers to a processor of an architecture as described in US Patent No. 7,447,873 to Nordquist, including improvements and advancements thereto, and as implemented, for example, in multiple generations of NVIDIA GPUs. For example, an SM may include multiple processing engines or cores configured to concurrently execute multiple threads arranged in multiple single instruction, multiple data (SIMD) groups (e.g., warps), wherein each thread in the same group of the SIMD groups executes the same data processing program comprising a sequence of instructions on different input objects, and different threads in the same group of the SIMD groups execute using different processing engines or cores. The SM may also typically provide (a) a local register file having multiple channels, wherein each processing engine or core is configured to access a different subset of the channels; and instruction issue logic configured to select one of the SIMD groups and issue one of the instructions of the same data processing program to each of the multiple processing engines in parallel, wherein each processing engine executes the same instruction in parallel with the other processing engines using a subset of the accessible local register file channels. The SM typically further includes core interface logic that is configured to initiate execution of one or more SIMD groups. As shown in the figure, the architecture of such an SM is to provide fast local SMEM, thereby enabling data sharing / reuse and synchronization between all threads of a CTA executing on the SM.

[0153] When configured for general parallel computing, a simpler configuration can be used compared to graphics processing. Specifically, Figure 9AThe fixed-function graphics processing unit is shown bypassed, creating a simpler programming model. In a general-purpose parallel computing configuration, the work distribution unit 825 assigns and dispatches thread blocks directly to the DPC 920. The threads in the block execute the same program, using unique thread IDs in computations to ensure each thread produces unique results, using the SM 940 to execute the program and perform computations, using the SMEM / L1 cache 1070 to communicate between threads, and using the LSU 1054 to read and write global memory through the SMEM / L1 cache 1070 and the memory partition unit 880. When configured for general-purpose parallel computing, the SM 940 can also write commands that the scheduler unit 820 can use to start new work on the DPC 920.

[0154] The PPU 800 may be included in a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, etc. In one embodiment, the PPU 800 is included on a single semiconductor substrate. In another embodiment, the PPU 800 is included on a system-on-chip (SoC) along with one or more other devices (such as an additional PPU 800, a memory 804, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), etc.).

[0155] In an embodiment, the PPU 800 may be included on a graphics card that includes one or more memory devices 804. The graphics card may be configured to interface with a PCIe slot on a desktop computer motherboard. In another embodiment, the PPU 800 may be an integrated graphics processing unit (iGPU) or a parallel processor included in a motherboard chipset. In some embodiments, two or more PPUs 800 are integrated into a single package to form a multi-chip PPU (e.g., a multi-chip GPU described herein).

[0156] Exemplary Computing System

[0157] Systems with multiple GPUs and CPUs are being used across various industries as developers expose and exploit greater parallelism in applications such as artificial intelligence computing. High-performance GPU-accelerated systems with tens to thousands of computing nodes are deployed in data centers, research institutions, and supercomputers to solve larger problems. As the number of processing devices within high-performance systems increases, communication and data transmission mechanisms need to scale to support this increased bandwidth.

[0158] Figure 11B The use according to one embodiment includes Figure 8Conceptual diagram of a processing system 1100 implemented with two or more multi-die PPUs 1101 of PPU 800. Processing system 1100 includes a CPU 1130, a switch 1110, and each of the multiple multi-die PPUs 1101 and a corresponding memory 1104. NVLink 1110 provides a high-speed communication link between each multi-die PPU 1101. Figure 11A 1102, but the number of connections connected to each multi-die PPU 1101 and CPU 1130 may vary. Switch 1110 interfaces between interconnect 1102 and CPU 1130. Multi-die PPU 1101, memory 1104, and NVLink 1110 may be located on a single semiconductor platform to form parallel processing module 1125. In one embodiment, switch 1110 supports two or more protocols that interface between various different connections and / or links.

[0159] In another embodiment (not shown), NVLink 1110 provides one or more high-speed communication links between each multi-die PPU 1101 and CPU 1130, and switch 1110 interfaces between interconnect 1102 and multi-die PPUs 1101. Multi-die PPUs 1101, memory 1104, and interconnect 1102 may be located on a single semiconductor platform to form parallel processing module 1125. In yet another embodiment (not shown), interconnect 1102 provides one or more communication links between each multi-die PPU 1101 and CPU 1130, and switch 1155 interfaces between the various multi-die PPUs 1101 using NVLink 1110 to provide one or more high-speed communication links between the multi-die PPUs 1101. In another embodiment (not shown), NVLink 1110 provides one or more high-speed communication links between PPUs 1101 and CPU 1130 through switch 1110. In yet another embodiment (not shown), interconnect 1102 provides one or more communication links directly between each multi-die PPU 1101. The one or more NVLink 1110 high-speed communication links may be implemented as a physical NVLink interconnect or an on-chip or on-die interconnect using the same protocol as NVLink 1110.

[0160] In the context of this specification, a single semiconductor platform may refer to a unique, single semiconductor-based integrated circuit fabricated on a die or chip. It should be noted that the term single semiconductor platform may also refer to a multi-chip module with increased connectivity that emulates on-chip operation and is substantially improved by utilizing conventional bus implementations. Of course, the various circuits or devices may also be placed separately or in various combinations of semiconductor platforms, depending on the needs of the user. Optionally, the parallel processing module 1125 may be implemented as a circuit board substrate, and each of the multi-die PPU 1101 and / or memory 1104 may be a packaged device. In one embodiment, the CPU 1130, switch 1155, and parallel processing module 1125 are located on a single semiconductor platform.

[0161] In one embodiment, NVLink 1110 allows direct load / store / atomic access from the CPU 1130 to the memory 1104 of each multi-die PPU 1101. In one embodiment, NVLink 1110 supports coherency operations, allowing data read from memory 1104 to be stored in the cache hierarchy of the CPU 1130, reducing cache access latency for the CPU 1130. In one embodiment, NVLink 1110 includes support for Address Translation Services (ATS), allowing the multi-die PPU 1101 to directly access page tables within the CPU 1130. One or more NVLinks 1110 can also be configured to operate in a low-power mode.

[0162] Figure 11B An exemplary system 1165 is shown in which various architectures and / or functionality of various previous embodiments can be implemented. The exemplary system 1165 can be configured to implement the methods disclosed herein.

[0163] As shown, a system 1165 is provided that includes at least one central processing unit 1130 connected to a communication bus 1175. The communication bus 1175 can be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or one or more point-to-point communication protocols. The system 1165 also includes a main memory 1140. Control logic (software) and data are stored in the main memory 1140, which can take the form of random access memory (RAM).

[0164] System 1165 also includes an input device 1160, a parallel processing system 1125, and a display device 1145, such as a conventional CRT (cathode ray tube), an LCD (liquid crystal display), an LED (light emitting diode), a plasma display, etc. User input can be received from input device 1160 (e.g., a keyboard, a mouse, a touchpad, a microphone, etc.). Each of the aforementioned modules and / or devices can even be located on a single semiconductor platform to form system 1165. Alternatively, the modules can be located separately or in various combinations of semiconductor platforms, depending on user needs.

[0165] Furthermore, the system 1165 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, etc.) through the network interface 1135 for communication purposes.

[0166] System 1165 may also include auxiliary storage (not shown). Auxiliary storage 610 includes, for example, a hard drive and / or a removable storage drive, representative of a floppy disk drive, a tape drive, an optical drive, a digital versatile disk (DVD) drive, a recording device, or a universal serial bus (USB) flash memory. The removable storage drive reads from and / or writes to a removable storage unit in a well-known manner.

[0167] Computer programs or computer control logic algorithms may be stored in the main memory 1140 and / or the secondary storage. These computer programs, when executed, enable the system 1165 to perform various functions. The memory 1140, storage, and / or any other storage are possible examples of computer-readable media.

[0168] The architecture and / or functionality of the various preceding figures can be implemented in the context of a general-purpose computer system, a circuit board system, a game console system dedicated for entertainment purposes, a dedicated system, and / or any other desired system. For example, system 1165 can take the form of a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, a mobile telephone device, a television, a workstation, a game console, an embedded system, and / or any other type of logic.

[0169] An application program may be implemented by an application program executed by a host processor, such as a CPU. In one embodiment, a device driver may implement an application programming interface (API) that defines various functions that an application program can use to generate graphics data for display. A device driver is a software program that includes multiple instructions that control the operation of a multi-die PPU 1101. The API provides an abstraction for programmers, allowing them to utilize specialized graphics hardware, such as the multi-die PPU 1101, to generate graphics data without requiring them to utilize the specific instruction set of the multi-die PPU 1101. The application program may include API calls that are routed to the device driver of the multi-die PPU 1101. The device driver interprets the API calls and performs various operations in response to the API calls. In some cases, the device driver may perform operations by executing instructions on the CPU. In other instances, the device driver may perform operations at least in part by initiating operations on the multi-die PPU 1101 using an input / output interface between the CPU and the multi-die PPU 1101. In one embodiment, the device driver is configured to implement a graphics processing pipeline using the hardware of the multi-die PPU 1101.

[0170] Various programs may be executed within the multi-die PPU 1101 to implement various stages of processing for an application. For example, a device driver may launch a kernel on the multi-die PPU 1101 to perform one stage of processing on one SM 940 (or multiple SMs 940). The device driver (or the initial kernel executed by the multi-die PPU 1101) may also launch other kernels on the multi-die PPU 1101 to perform other stages of processing. If the application processing includes a graphics processing pipeline, some stages of the graphics processing pipeline may be implemented on fixed unit hardware, such as a rasterizer or data assembler implemented within the multi-die PPU 1101. It should be understood that the results from one kernel may be processed by one or more intermediate fixed-function hardware units before being processed by a subsequent kernel on the SM 940.

[0171] The technology disclosed herein can be incorporated into any processor that can be used to process neural networks, such as a central processing unit (CPU), a graphics processing unit (GPU), an intelligence processing unit (IPU), a neural processing unit (NPU), a tensor processing unit (TPU), a neural network processor (NNP), a data processing unit (DPU), a vision processing unit (VPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and the like. Such a processor can be integrated into a personal computer (e.g., a laptop), a data center, an Internet of Things (IoT) device, a handheld device (e.g., a smartphone), a vehicle, a robot, or any other device that performs inference, training, or any other processing of a neural network. Such a processor can be used in a virtualized system so that an operating system executing in a virtual machine on the system can utilize the processor.

[0172] As an example, a processor incorporating the techniques disclosed herein can be used to process one or more neural networks in a machine to identify, classify, manipulate, process, operate, modify, or navigate physical objects in the real world. For example, such a processor can be used in an autonomous vehicle (e.g., a car, motorcycle, helicopter, drone, airplane, ship, submarine, delivery robot, etc.) to enable the vehicle to move in the real world. Additionally, such a processor can be used in a robot in a factory to select parts and assemble the parts into assemblies.

[0173] As an example, a processor incorporating the techniques disclosed herein may be employed to process one or more neural networks to identify one or more features in an image, or to alter, generate, or compress an image. For example, such a processor may be used to enhance images rendered using rasterization, ray tracing (e.g., using NVIDIA RTX), and / or other rendering techniques. In another example, such a processor may be employed to reduce the amount of image data transmitted from a rendering device to a display device over a network (e.g., the Internet, a mobile telecommunications network, a WIFI network, and any other wired or wireless network system). Such transmission may be used to stream image data from a server or a data center in the cloud to a user device (e.g., a personal computer, a video game console, a smartphone, other mobile device, etc.) to enhance services that stream images, such as NVIDIA GeForce Now (GFN), Google Stadia, and the like.

[0174] As an example, a processor incorporating the techniques disclosed herein can be used to process one or more neural networks for any other type of application that can utilize neural networks. For example, these applications may involve translating from one spoken language to another, recognizing and negating sounds in audio, detecting anomalies or defects in the production of goods and services, monitoring living and / or non-living things, medical diagnosis, decision making, etc.

[0175] As an example, a processor incorporating the techniques disclosed herein may be used to implement a neural network, such as a large language model (LLM), to generate content (e.g., images, video, text, essays, audio, etc.), respond to user queries, solve problems in mathematics and other fields, and the like.

[0176] All patents, patent applications, and publications cited herein are incorporated by reference for all purposes as if expressly stated.

[0177] While the invention has been described in connection with what are presently considered to be the most practical and preferred embodiments, it should be understood that the invention is not limited to the disclosed embodiments, but on the contrary, the invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A multi-die processing system comprising: a plurality of processors comprising at least a first processor on a first die and at least a second processor on a second die different from the first die, wherein the plurality of processors each include a plurality of hardware engines; and Hardware engine remapping circuitry is configured to map between respective local die-based engine identifiers and corresponding respective engine identifiers of the plurality of hardware engines.

2. The multi-die processing system according to claim 1, wherein: Each of the respective engine identifiers indicates a corresponding hardware engine, regardless of which die the corresponding hardware engine is located on.

3. The multi-die processing system according to claim 1, wherein: The hardware engine remapping circuit is configured to perform the mapping on a page table binding request, an address translation request, a security check request, or a fault handling request, or corresponding responses to these requests.

4. The multi-die processing system according to claim 3, wherein: The source of the request is software executing on a processor external to the die.

5. The multi-die processing system according to claim 3, wherein: The source of the request or the corresponding response is a task executing on one of the plurality of processors.

6. The multi-die processing system according to claim 3, wherein: The request includes a flag indicating conversion of a local engine identifier to a unique engine identifier, wherein the local engine identifier is a die-associated engine ID, and the unique engine identifier is a globally unique engine ID that is identical to the die-associated engine ID.

7. The multi-die processing system of claim 6, wherein: The hardware engine remapping circuit is further configured to calculate an offset for each of the dies based on a die identification fuse signal and to use the calculated offset in the mapping.

8. The multi-die processing system of claim 1, wherein: The hardware engine remapping circuit is configured to determine a globally unique die identifier for each die based on the die identification fuse signal.

9. The multi-die processing system of claim 1, wherein: The hardware engine remapping circuitry includes a plurality of die-level remapping circuits, and each die-level remapping circuit is configured to operate independently of other ones of the die-level remapping circuits.

10. The multi-die processing system of claim 9, wherein: Each die-level remapping circuit is configured to apply an offset to a hardware engine identifier in a received message, wherein the offset is determined based on the die identification fuse signal.

11. The multi-die processing system of claim 10, wherein: Each die-level remapping circuit is further configured to determine, based on the application, whether to forward the received message to another one of the die-level remapping circuits located on a different die.

12. The multi-die processing system of claim 11, wherein: Each die-level remapping circuit is configured to forward received messages to other ones of the die-level remapping circuits located on different dies via dedicated communication links between the frame buffer hubs of each of the dies.

13. The multi-die processing system of claim 10, wherein: Each die-level remapping circuit is further configured to determine whether to forward the received message to at least one local hardware engine based on the application.

14. The multi-die processing system of claim 1 , further comprising: a respective memory barrier processing circuit disposed on each die of the plurality of dies, wherein each memory barrier processing circuit is configured to determine a status of a memory barrier based on acknowledgements received from one or more local hardware engines located on the same die as the memory barrier processing circuit and one or more acknowledgements received from other dies of the plurality of dies.

15. The multi-die processing system of claim 14, wherein: A set of the acknowledgements to monitor is determined based on a scope of the memory barrier.

16. The multi-die processing system of claim 15, wherein: The corresponding memory barrier processing circuit includes at least a first state machine configured for a locally initiated input / output refresh request and a second state machine configured for a remotely initiated input / output refresh request.

17. The multi-die processing system of claim 1, wherein: The multiple processors on respective different dies are located in a single package, wherein the package is configured to be connected to a central processing unit (CPU) in a computer.

18. The multi-die processing system of claim 1, wherein: The hardware engine remapping circuitry includes a plurality of die-level remapping circuits, and each die of the plurality of dies includes a respective one of the die-level remapping circuits.

19. The multi-die processing system of claim 1, wherein: Each of the plurality of processors is a graphics processing unit (GPU).

20. A method performed by hardware engine remapping circuitry in a multi-die processing system, wherein the multi-die processing system includes a plurality of processors located on respective different dies, each processor including a plurality of hardware engines, the method comprising: determining an offset, wherein the offset is different for each of the die; and In response to the received message, a mapping is performed between the respective local engine identifiers of the hardware engines and the corresponding unique engine identifiers based on the determined offset for the particular die.

Images