Hardware management of direct memory access commands

Abstract

A method for hardware management of DMA transfer commands includes accessing, by a first DMA engine [314], a DMA transfer command and determining a first portion of a data transfer requested by the DMA transfer command. Transfer of the first portion of the data transfer by the first DMA engine is initiated based at least in part on the DMA transfer command. Similarly, a second portion of the data transfer by a second DMA engine is initiated based at least in part on the DMA transfer command. After transferring the first and second portions of the data transfer, an indication is generated indicating completion of the data transfer requested by the DMA transfer command.

Description

[Background technology]

[0001] A system direct memory access (DMA) engine is a module that coordinates direct memory access transfers of data between devices (e.g., input / output interfaces and display controllers) and memory in a computer system, or between different locations in memory. A DMA engine is often located on a processor, such as a central processing unit (CPU) or a graphics processor (GPU), and receives commands from an application running on the processor. Based on the command, the DMA engine reads data from a DMA source (e.g., a first memory buffer defined in memory) and writes data to a DMA destination (e.g., a second buffer defined in memory).

[0002] The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the accompanying drawings, in which: The use of the same reference numbers in different drawings indicates similar or identical items. [Brief description of the drawings]

[0003] [Figure 1] FIG. 1 is a block diagram of a computing system implementing a multi-die processor, according to some embodiments. [Diagram 2] FIG. 2 is a block diagram of a portion of an example computing system for implementing hardware management of DMA commands, in accordance with some embodiments. [Diagram 3] FIG. 1 is a block diagram illustrating a portion of an exemplary multi-processor computing system for implementing hardware management of DMA commands, in accordance with some embodiments. [Figure 4] FIG. 1 is a block diagram illustrating an example of a system that implements hardware-managed splitting of transfer commands based on cache status, according to some embodiments. [Diagram 5]FIG. 2 is a block diagram illustrating another example of a system that implements hardware-managed splitting of transfer commands, according to some embodiments. [Figure 6] 4 is a flow diagram illustrating a method for performing hardware-managed splitting of a DMA transfer command according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] A conventional processor includes one or more direct memory access engines for reading and writing blocks of data stored in a system memory. The direct memory access engine relieves the processor core from the burden of managing the transfer. In response to a data transfer request from the processor core, the direct memory access engine provides the necessary control information to the corresponding source and destination so that the data transfer operation can be performed without delaying the computation code, thus allowing communication and computation to overlap in time. By asynchronously handling the formation and communication of control information by the direct memory access engine, the processor core is freed to perform other tasks while waiting for the satisfaction of the data transfer request.

[0005] Distributed architectures are becoming increasingly popular as an alternative to monolithic processing architectures in which physically or logically separated processing units work cooperatively through high performance interconnects. One example of a distributed architecture is the chiplet architecture, which allows some parts of a processing unit to be manufactured at a node having larger dimensions while capturing the benefits of manufacturing at a smaller node, where other parts do not benefit from the reduced scale of the smaller node. The number of direct memory access engines is likely to be increased in chiplet-based systems (e.g., compared to a comparable monolithic non-chiplet-based design).

[0006] To improve system performance by improving utilization of direct memory access engines, FIGS. 1-6 illustrate systems and methods that utilize hardware-managed throttling for processing of direct memory transfer commands. In various embodiments, a method for hardware management of DMA transfer commands includes accessing, by a first DMA engine, a DMA transfer command and determining a first portion of a data transfer requested by the DMA transfer command. Transfer of the first portion of the data transfer by the first DMA engine is initiated based at least in part on the DMA transfer command. Similarly, transfer of a second portion of the data transfer by a second DMA engine (i.e., a DMA engine different from the first DMA engine) is initiated based at least in part on the DMA transfer command. After transferring the first and second portions of the data transfer, an indication is generated that signals completion of the data transfer requested by the DMA transfer command. In this manner, the work specified by the transfer command is divided among the DMA engines, thereby increasing aggregate bandwidth utilization without the need to enlarge the individual DMA engines or add features to increase overall DMA throughput or bandwidth utilization of the data fabric.

[0007] FIG. 1 illustrates a block diagram of one embodiment of a computing system 100 implementing a multi-die processor, according to some embodiments. In various embodiments, computing system 100 includes at least one or more processors 102A-102N, fabric 104, input / output (I / O) interface 106, memory controller 108, display controller 110, and other devices 112. In various embodiments, computing system 100 includes a host processor 114, such as a central processing unit (CPU), to support execution of instructions for graphics and other types of workloads. In various embodiments, computing system 100 includes any of a computer, laptop, mobile device, server, or various other types of computing systems or devices. It should be noted that the number of components of computing system 100 varies in some embodiments. It should also be noted that in some embodiments, computing system 100 includes other components not shown in FIG. 1. Additionally, in other embodiments, computing system 100 is constructed in other manners than that shown in FIG. 1.

[0008] Fabric 104 represents any communication interconnect conforming to any of a variety of types of protocols utilized to communicate between components of computing system 100. Fabric 104 provides data paths, switches, routers, and other logic that connect processing units 102, I / O interfaces 106, memory controller 108, display controller 110, and other devices 112 to one another. Fabric 104 handles request, response, and data traffic, as well as probe traffic, to facilitate coherency. Fabric 104 also handles interrupt request routing and configuration access paths to the various components of computing system 100. In addition, fabric 104 handles configuration request, response, and configuration data traffic. In some embodiments, fabric 104 is bus-based, including shared bus configurations, crossbar configurations, and hierarchical buses with bridges. In other embodiments, fabric 104 is packet-based and hierarchical with bridges, crossbar, point-to-point, or other interconnects. From the perspective of fabric 104, the other components of computing system 100 are referred to as "clients." Fabric 104 is configured to process requests generated by various clients and pass the requests on to other clients.

[0009] Memory controller 108 represents any number and type of memory controller coupled to any number and type of memory devices. For example, the types of memory devices coupled to memory controller 108 include dynamic random access memory (DRAM), static random access memory (SRAM), NAND flash memory, NOR flash memory, ferroelectric random access memory (FeRAM), etc. Memory controller 108 is accessible by processor 102, I / O interface 106, display controller 110, and other devices 112 via fabric 104. I / O interface 106 represents any number and type of I / O interfaces (e.g., peripheral component interconnect (PCI) bus, PCI-Extended (PCI-X), PCI Express (PCI Express) bus, gigabit Ethernet (GBE) bus, universal serial bus (USB)). Various types of peripheral devices are coupled to I / O interface 106. Such peripheral devices include (but are not limited to) displays, keyboards, mice, printers, scanners, joysticks, other types of game controllers, media recording devices, external storage devices, network interface cards, etc. Other devices 112 represent any number and type of devices (e.g., multimedia devices, video codecs).

[0010] In various embodiments, each of the processors 102 is a parallel processor (e.g., a vector processor, a graphics processing unit (GPU), a general-purpose GPU (GPGPU), a non-scalar processor, a highly parallel processor, an artificial intelligence (AI) processor, an inference engine, a machine learning processor, other multi-threaded processing unit, etc.). Each parallel processor 102 is constructed as a multi-chip module (e.g., a semiconductor die package) that includes two or more base integrated circuit dies (described in more detail below with respect to FIG. 2) communicatively coupled together with a bridge chip such that the parallel processors are usable (e.g., addressable) like a single semiconductor integrated circuit. As used in this disclosure, the terms "die" and "chip" are used interchangeably. Those skilled in the art will recognize that conventional (e.g., non-multi-chip) semiconductor integrated circuits are manufactured as wafers or as dies (e.g., single-chip ICs) that are formed in a wafer and later separated from the wafer (e.g., when the wafer is diced), and often multiple ICs are manufactured simultaneously in a wafer. The ICs and possibly discrete circuits and possibly other components (such as non-semiconductor packaging substrates including printed circuit boards, interposers and possibly others) are assembled into a multi-die parallel processor.

[0011] As described in more detail below with respect to FIGS. 2-6, in various embodiments, each of the individual processors 102 includes one or more base IC dies that use processing stacked die chiplets according to some embodiments. The base die is formed as a single semiconductor chip package that includes N communicatively coupled graphics processing stacked die chiplets. In various embodiments, the base IC die includes two or more DMA engines that coordinate DMA transfers of data between the device and memory (or between different locations in the memory). Although various embodiments are described below in the specific context of a CPU and a GPU for ease of illustration and explanation, it should be appreciated that the concepts described herein are similarly applicable to other processors, including accelerated processing units (APUs), discrete GPUs (dGPUs), artificial intelligence (AI) accelerators, other parallel processors, and the like.

[0012] Referring to FIG. 2, a block diagram of portions of an exemplary computing system 200 is shown. In some examples, the computing system 200 is implemented using some or all of the device 100 as shown and described with respect to FIG. 1. The computing system 200 includes at least a first semiconductor die 202. In various embodiments, the semiconductor die 202 includes one or more processors 204A-204N, an input / output (I / O) interface 206, an intra-die interconnect 208, a memory controller 210, and a network interface 212. In other examples, the computing system 200 includes additional components, different components, and / or is arranged in a different manner. In some embodiments, the semiconductor die 202 is a multi-chip module configured as a semiconductor die package that includes two or more integrated circuit (IC) dies such that the processors can be used as a single semiconductor integrated circuit. As used in this disclosure, the terms "die" and "chip" may be used interchangeably.

[0013] In some embodiments, each of the processors 204A-204N includes one or more processing devices. In one embodiment, at least one of the processors 204A-204N includes one or more general-purpose processing devices, such as a CPU. In some embodiments, such processing devices are implemented using the processor 102 as shown and described with respect to FIG. 1. In another embodiment, at least one of the processors 204A-204N includes one or more parallel processors. Examples of parallel processors include GPUs, digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.

[0014] I / O interface 206 includes one or more I / O interfaces (e.g., a Peripheral Component Interconnect (PCI) bus, a PCI-Extension (PCI-X), a PCI Express (PCIe) bus, a Gigabit Ethernet (GBE) bus, a Universal Serial Bus (USB), etc.). In some embodiments, I / O interface 206 is implemented using input drivers 112 and / or output drivers 114 as shown and described with respect to FIG. 1. Various types of peripheral devices can be coupled to I / O interface 206. Such peripheral devices include, but are not limited to, displays, keyboards, mice, printers, scanners, joysticks, other types of game controllers, media recording devices, external storage devices, network interface cards, etc. In some embodiments, such peripheral devices are implemented using input devices 108 and / or output devices 118 as shown and described with respect to FIG. 1.

[0015] In various embodiments, each processor includes a cache subsystem having one or more levels of cache. In some embodiments, each of processors 204A-204N includes a cache (e.g., a level 3 (L3) cache) that is shared among multiple processor cores of a core complex. Memory controller 210 includes at least one memory controller accessible by processors 204A-204N, such as accessible via intra-die interconnect 208. In various embodiments, memory controller 210 includes one or more of any suitable type of memory controller. Each of the memory controllers is coupled to (or in communication with) and controls access to any number and type of memory devices (not shown). In some embodiments, such memory devices include dynamic random access memory (DRAM), static random access memory (SRAM), NAND flash memory, NOR flash memory, ferroelectric memory (FeRAM), or any other suitable memory device. Intra-die interconnect 208 includes any computer communication medium suitable for communication between the devices shown in FIG. 2, such as a bus, a data fabric, or the like.

[0016] FIG. 3 is a block diagram illustrating a portion of an exemplary multi-processor computing system 300. System 300, or portions thereof, may be implemented using some or all of semiconductor die 202 (as shown and described with respect to FIG. 2) and / or device 100 (as shown and described with respect to FIGS. 1 and 2). In various embodiments, system 300 includes a processor multi-chip module 302 that uses processing stacked die chiplets according to some embodiments. Processor multi-chip module 302 is formed as a single semiconductor chip package that includes N=3 communicatively coupled graphics processing stacked die chiplets 304. As shown, processor multi-chip module 302 includes a first graphics processing stacked die chiplet 304A, a second graphics processing stacked die chiplet 304B, and a third graphics processing stacked die chiplet 304C.

[0017] The graphics processing stacked die chiplet 304 is described below in the specific context of GPU terminology for ease of illustration and explanation, however, it should be appreciated that in various embodiments, the described architecture may be applicable to any of the various types of parallel processors (as more broadly described above with reference to FIGS. 2 and 3) without departing from the scope of this disclosure. Additionally, in various embodiments, the term "chiplet" as used herein refers to any device that includes, but is not limited to, the following characteristics: 1) the chiplet includes active silicon dies that include at least a portion of the computational logic used to solve a complete problem (i.e., the computational workload is distributed across multiple of these active silicon dies), 2) the chiplets are packaged together as a monolithic unit on the same substrate, and 3) the programming model maintains the notion of combining these separate computational dies (i.e., the graphics processing stacked die chiplets) as a single monolithic unit (i.e., each chiplet is not exposed as a separate device to the application that uses the chiplet to process the computational workload).

[0018] In various embodiments, the processor multi-chip module 302 includes an inter-chip data fabric 306 that operates as a high bandwidth inter-die interconnect between the chiplets (e.g., between any combination of the first graphics processing stacked die chiplet 304A, the second graphics processing stacked die chiplet 304B, and the third graphics processing stacked die chiplet 304C). In some embodiments, the processor multi-chip module 302 includes one or more processor cores 308 (e.g., CPUs and / or GPUs, or processor core dies) formed on each of the chiplets 304A-304C. Additionally, in various embodiments, each of the chiplets 304A-304C includes one or more levels of cache memory 310 and one or more memory PHYs (not shown) for communicating with external system memory modules 312, such as dynamic random access memory (DRAM) modules.

[0019] Each of the chiplets 304A-304C also includes one or more DMA engines 314. In various embodiments, the one or more DMA engines 314 coordinate DMA transfers of data between devices and memory (or between different locations in memory) in the system 300. In various embodiments, the one or more DMA engines 314 coordinate the movement of data between multiple devices / accelerators while computations are performed on other data, for example in the processor core 308. In various embodiments, the one or more DMA engines 314 are part of a DMA controller (not shown) in some embodiments, although the terms DMA engine and DMA controller are used interchangeably herein. The DMA engines 314 operate upon command to transfer data, for example, to and from one or more memory modules 312, without the involvement of the processor core 308. Similarly, the DMA engines 314 perform intra-chip data transfers in some embodiments. As will be appreciated, the DMA engines 314 relieve the processor cores of the burden of managing data transfers and, in various embodiments, are used as a global data transfer agent to handle various data transfer requirements from software, such as memory-to-memory data copies.

[0020] The one or more DMA engines 314 provide for the fetching and decoding of command packets from application / agent queues and respective DMA buffers to perform the desired data transfer operations specified by DMA commands, also known as descriptors. DMA commands include memory flow commands (e.g., read / get or write / put commands to transfer data into or out of memory) that control the transfer or movement of memory locations containing data or instructions. The DMA command descriptor, in various embodiments, indicates, for each data transfer operation, the source address from which the data is read, the transfer size, and the destination address to which the data is written. The descriptors are generally organized in memory as a linked list or chain, with each descriptor including a field indicating the address in memory of the next descriptor to be executed. In various embodiments, the descriptors are an array of commands with a valid bit, the commands are of a known size, and the one or more DMA engines 314 stop when they reach an invalidate command. The last descriptor in the list has a null pointer in the "next descriptor" field, indicating to the DMA engine that there are no more commands to be executed and that the DMA should go idle upon reaching the end of the chain.

[0021] In response to a data transfer request from a processor core, the DMA engine 314 provides the necessary control information to the corresponding source and destination so that the data transfer request can be fulfilled. Because the DMA engine 314 handles the formation and communication of control information, the processor core can perform other tasks while waiting for the data transfer request to be fulfilled. In various embodiments, the DMA engines 314 each include one or more dedicated auxiliary processors that transfer data between memory and / or peripheral input / output (I / O) devices and locations in memory without going through the processor core or CPU.

[0022] In some embodiments, DMA requests are processed by placing DMA commands generated by one or more of the processors 308 in a memory-mapped IO (MMIO) location, such as a DMA buffer 316 (also referred to interchangeably as a DMA queue for holding DMA transfer commands). In various embodiments, a DMA buffer is a hardware structure to which read or write instructions are transferred so that the DMA engine 314 can read the DMA command (e.g., without having to go to DRAM memory). To perform a data transfer operation, in various embodiments, the DMA engine 314 receives instructions (e.g., DMA transfer commands / data transfer requests) generated by the processors 308 by accessing a sequence of commands in the DMA buffer 316. The DMA engine 314 then retrieves the DMA command (also known as a descriptor) from the DMA buffer 316 for processing. In some embodiments, the DMA command specifies, for example, a starting address for direct virtual memory access (DVMA) and I / O bus access, as well as the length of the transfer up to a predetermined maximum.

[0023] DMA buffer 316 is illustrated in FIG. 3 as being implemented in chiplet 304 for ease of explanation, but one of ordinary skill in the art will recognize that DMA buffer 316 can be implemented in various components of the systems and devices described herein without departing from the scope of the present disclosure. For example, in some embodiments, DMA buffer 316 is configured in a main memory, such as memory module 312. That location of the command queue in memory is where DMA engine 314 goes to read transfer commands. In various embodiments, DMA buffer 316 is configured as one or more ring buffers (e.g., addressed by modulo addressing).

[0024] DMA engine 314 accesses (or receives) DMA transfer commands from DMA buffer 316 via a bus (not shown). Based on the received instructions, in some embodiments, DMA engine 314 reads and buffers data from any memory (e.g., memory module 312) via data fabric 306 and writes buffered data to any memory via data fabric 306. In some embodiments, the DMA source and the DMA destination are physically located on different devices (e.g., different chiplets). Similarly, in a multiprocessor system, the DMA source and the DMA destination are possibly located on different devices associated with different processors. In such cases, DMA engine 314 resolves the virtual address to obtain a physical address and issues remote read and / or write commands to affect the DMA transfer. For example, in various embodiments, based on the received instructions, DMA engine 314 sends a message to a data fabric device with instructions to affect the DMA transfer.

[0025] During DMA, one or more processor cores 308 may queue DMA commands in DMA buffer 316 and signal their presence to DMA engine 314. For example, in some embodiments, an application program executing on system 300 prepares an appropriate chain of descriptors in memory accessible to the DMA engine (e.g., DMA buffer 316) to initiate a chain of DMA data transfers. Processor core 308 then sends a message (or other notification) to DMA engine 314 indicating the memory address of the first descriptor in the chain, which is a request for the DMA engine to begin executing the descriptor. Typically, the application sends a message to the DMA engine's "doorbell," a control register with a specific bus address designated for this purpose. Sending such a message to begin a DMA execution is known as "ringing the doorbell" of DMA engine 314. DMA engine 314 responds by reading and executing the first descriptor. It then updates the status field of the descriptor to indicate to the application that the descriptor has been executed. The DMA engine 314 walks the "next" field through the linked list, marking each descriptor as executed until it reaches the null pointer in the last descriptor. After executing the last descriptor, the DMA engine 314 goes into an idle state, ready to receive a new list for execution.

[0026] In various embodiments as shown in FIG. 3, the system 300 includes two or more accelerators connected to each other by an inter-chip data fabric 306. Further, as shown in FIG. 3, the components of the graphics processing stacked die chiplets 304 (e.g., one or more processor cores 308, DMA engines 314, DMA buffers 316, etc.) communicate with each other via an interconnect 318 (e.g., via other components). In this manner, the interconnect 318 forms part of a data fabric that facilitates communication between the components of the multiprocessor computing system 300. Further, the inter-chip data fabric 306 extends the data fabric over the various communicatively coupled graphics processing stacked die chiplets 304 and I / O interfaces (not shown) that also form part of the data fabric. In various embodiments, the interconnect 318 includes any computer communication medium suitable for communication between the devices shown in FIG. 3, such as a bus, a data fabric, etc. In some embodiments, the interconnect 318 is connected to and / or communicates with other components not shown in FIG. 3 for ease of illustration. For example, in some embodiments, the interconnect 318 includes a connection to one or more input / output (I / O) interfaces 206 as shown and described with respect to FIG.

[0027] As will be appreciated, inter-chip data fabric 306 and / or interconnect 318 often have such high bandwidth that a single DMA engine cannot saturate the available data fabric bandwidth. In various embodiments, as described in more detail below, system 300 utilizes an increased number of DMA engines 314 (e.g., one per chiplet 304, as shown in the embodiment of FIG. 3) to perform hardware-managed (i.e., mediated by device hardware without input by a system software application, including, for example, one or both of an operating system or hypervisor) division of transfer commands to multiple DMA engines 314 for processing of memory transfer commands via DMA. In this manner, the work specified by a transfer command is essentially divided across multiple chiplets 304 and their respective DMA engines 314, thereby increasing aggregate bandwidth usage without the need for individual DMA engines 314 to be larger or have more capabilities to increase overall DMA throughput and data fabric bandwidth usage.

[0028] In operation, in response to a notification (e.g., a doorbell ring), the DMA engine 314 reads and executes a DMA transfer command (with associated parameters) from the DMA buffer 316 to perform a data transfer operation and a packet transfer. In various embodiments, the operational parameters (e.g., DMA command parameters) are typically the base address, stride, element size, and number of elements to communicate for both the sender and receiver. Specifically, the DMA engines 314 are configured such that multiple DMA engines 314 across multiple dies (e.g., MCMs 302) or chiplets 304 read the same location containing a packet with the DMA transfer parameters. The DMA engines 314 then synchronize and coordinate with each other via hardware mechanisms to work together in the DMA transfer, as described in more detail below. In various embodiments, the DMA engines 314 perform hardware-managed partitioning of the command transfer such that a single DMA engine performs only a portion of the DMA transfer. For example, splitting a DMA transfer between two DMA engines 314 has the potential to double the bandwidth usage per unit time or DMA transfer throughput, since each DMA engine is performing half of the transfer simultaneously with the other DMA engine.

[0029] Referring to FIG. 4, a block diagram of an example of a system for implementing hardware-managed partitioning of transfer commands based on cache status is shown, according to some embodiments. The device 400, or portions thereof, can be implemented using some or all of the semiconductor die 202 (as shown and described with respect to FIG. 2) and / or the device 100 (as shown and described with respect to FIGS. 1 and 2). In various embodiments, the device 400 includes a base die 402 that uses processing stacked die chiplets according to some embodiments. The base die 402 is formed as a single semiconductor chip package that includes N=2 communicatively coupled graphics processing stacked die chiplets 404. As shown, the processor base die 402 includes a first graphics processing stacked die chiplet 404A and a second graphics processing stacked die chiplet 404B.

[0030] In various embodiments, the base die 402 includes an inter-chip data fabric 406 that operates as a high bandwidth inter-die interconnect between chiplets (e.g., between the first graphics processing stacked die chiplet 404A and the second graphics processing stacked die chiplet 404B). In some embodiments, the base die 402 includes one or more processor cores 408 (e.g., CPUs and / or GPUs, or processor core dies) formed on each of the chiplets 404A-404B. Additionally, in various embodiments, each of the chiplets 404A-404B includes one or more levels of cache memory 410 and one or more memory PHYs (not shown) for communicating with an external system memory module 412, such as a dynamic random access memory (DRAM) module. Taken as a whole, the main memory (e.g., memory module 412) communicatively coupled to the multiple chiplets (e.g., chiplets 404A and 404B) and their local caches form a shared memory for the device 400. As will be appreciated, each chiplet 404 only has a direct physical connection to a portion of the overall shared memory system.

[0031] In various embodiments, the base die 402 includes two or more DMA engines 414 (e.g., a first DMA engine 414A and a second DMA engine 414B) disposed on the base die 402. In various embodiments, the DMA engine 414 coordinates DMA transfers of data between devices and memory (or between different locations in memory) in the system 400. In various embodiments, the DMA engine 414 coordinates the movement of data between multiple devices / accelerators while computations are performed on other data, for example, in the processor core 408. In various embodiments, the one or more DMA engines 414 are part of a DMA controller (not shown) in some embodiments, although the terms DMA engine and DMA controller are used interchangeably herein. The DMA engine 414 operates in response to commands to transfer data to and from, for example, one or more memory modules 412, without the involvement of the processor core 408. Similarly, the DMA engine 414 performs intra-chip data transfers in some embodiments.

[0032] The graphics processing stacked die chiplet 304 is described below in the particular context of GPU terminology for ease of illustration and explanation, but it should be appreciated that in various embodiments, the described architecture may be applicable to any of a variety of types of parallel processors (as more broadly described above with reference to FIGS. 2 and 3) without departing from the scope of the present disclosure. Additionally, while the DMA engine 414 is shown in FIG. 4 as being on top of an individual compute chiplet, one skilled in the art will appreciate that the hardware-managed division of transfer commands may be performed by a DMA engine in a variety of system locations without departing from the scope of the present disclosure. For example, in some embodiments, the DMA engine is located on top of the base die (e.g., as described with respect to FIG. 5). Similarly, while the systems and devices are described herein in the context of a chiplet-based system, one skilled in the art will appreciate that the hardware-managed division of transfer commands is not limited to a particular architecture that may be implemented in any system configuration that includes multiple DMA engines, including a monolithic die.

[0033] In some embodiments, DMA requests are processed by placing DMA commands (also interchangeably referred to as packets) generated by one or more of the processors 408 into a memory-mapped IO (MMIO) location, such as a DMA buffer 416 (also interchangeably referred to as a DMA queue for holding DMA transfer commands). In various embodiments, the DMA buffer 416 is a hardware structure into which read or write instructions are transferred so that the DMA engine 414 can read the DMA command (e.g., without having to go to DRAM memory). To perform a data transfer operation, in various embodiments, the DMA engine 414 receives instructions (e.g., DMA transfer commands / data transfer requests) generated by the processor 408 by accessing a sequence of commands in the DMA buffer 416. The DMA engine 414 then retrieves the DMA command (also known as a descriptor) from the DMA buffer 416 for processing. In some embodiments, the DMA command specifies, for example, a starting address for direct virtual memory access (DVMA) and I / O bus access, as well as the length of the transfer up to a predetermined maximum.

[0034] DMA buffer 416 is shown in FIG. 4 as being implemented in base die 402 for ease of explanation, but one of ordinary skill in the art will recognize that DMA buffer 416 can be implemented in various components of the systems and devices described herein without departing from the scope of the present disclosure. For example, in some embodiments, DMA buffer 416 is configured in a main memory, such as memory module 412. That location of the command queue in memory is where DMA engine 414 goes to read transfer commands. In various embodiments, DMA buffer 416 is configured as one or more ring buffers (e.g., addressed by modulo addressing).

[0035] As illustrated in FIG. 4, DMA engines 414A and 414B receive DMA notifications 418 indicating that one or more DMA workloads or DMA jobs have been created and filled in memory (such as DMA buffer 416 or system memory module 412). In various embodiments as described above, DMA notification 418 includes a doorbell ring or other notifier indicating that a DMA descriptor is ready in memory. In response to receiving DMA notification 418, DMA engines 414A and 414B each independently fetch a DMA job description 420 from DMA buffer 416. In some embodiments, the communication path through which DMA engine 414 retrieves DMA transfer commands includes a PCIE interface (e.g., I / O interface 106 as described above with respect to FIG. 1) as a path for connecting the processor to chiplet 404. In other embodiments, the communication path through which DMA engine 414 retrieves DMA transfer commands includes an internal data fabric (e.g., interconnect 318 as described above with respect to FIG. 3) as a path for connecting the on-die processor to DMA engine 414.

[0036] In addition to independently fetching the same DMA job description from the same location (i.e., DMA buffer 416), each of DMA engines 414A and 414B independently determines the portion of the data transfer requested by the DMA transfer command to perform. In one embodiment, each of DMA engines 414A and 414B issues speculative DMA transfers by broadcasting cache probe requests 422 (e.g., read probes and / or write probes) to the cache memory 410 and main memory (e.g., memory module 412) of their respective chiplet 404. In response to the cache probe requests, the cache memory 410 and main memory of each chiplet 404 return one or more return responses to the requesting DMA engine. For example, in the case of a DMA read probe, the cache memory 410 can return a cache hit or a cache miss to indicate whether the requested data was found in the cache memory 410.

[0037] 4, the retrieved DMA job description 420 is a single transfer command to read data from physical addresses X and Y. To determine whether a portion of the data associated with addresses X and Y is currently resident in the cache memory 410 (e.g., an L3 or some last level cache) of the first graphics processing stacked die chiplet 404A, the DMA engine 414A broadcasts a cache probe request 422 to the cache memory 410 (including a cache controller [not shown]) before the memory module 412 and receives one or more return responses indicating a cache hit for address X and a cache miss for address Y. To determine whether a portion of the data associated with addresses X and Y is currently resident in the cache memory 410 of the second graphics processing stacked die chiplet 404B, the DMA engine 414B similarly broadcasts a cache probe request 422 to the cache memory 410 and receives one or more return responses indicating a cache miss for address X and a cache miss for address Y.

[0038] In various embodiments, a probe comprises a message passed from a coherency point (e.g., at DMA engine 414) to one or more caches in a computer system to request a response indicating whether the cache has a copy of the data block, and in some embodiments, to indicate the cache state that the cache should place the data block in. In some embodiments, when DMA engine 414 receives a memory request that targets its corresponding memory controller (e.g., a memory request for data stored at an address or address region in a memory controlled by the memory controller), DMA engine 414 performs a lookup (e.g., a tag-based lookup) against its corresponding cache directory to determine whether the request targets a memory address or region cached in at least one cache line of any cache subsystem.

[0039] Furthermore, a particular physical address only exists in one place on the memory module 412. For example, in a chiplet-based system, each die is assigned or otherwise physically connected to a memory channel with a particular range of addresses (or a particular address pattern belonging to the local die and the remote die). From the virtual to physical address translation, each DMA engine 414 knows whether the address of a DMA transfer command is within the range of memory connected to its physical die. If the DMA engine 414 decodes the DMA transfer command and determines that the data is not in local memory, the DMA engine 414 skips those parts of the DMA job. Furthermore, if all DMA engines 414 operate in this same manner, the entire memory channel is covered.

[0040] 4, data at physical address X and physical address Y is stored in a memory module 412 directly connected to the second graphics processing stacked die chiplet 404B. However, because the cache memory 410 of the first graphics processing stacked die chiplet 404A already contains the requested data for physical address X, the DMA engine 414A performs the portion of the data transfer associated with address X (as opposed to retrieving the data from a different chiplet's cache). Thus, having the DMA engine 414A perform the DMA transfer of the data associated with address X is more energy optimal and has lower latency than accessing the data-owning memory module 412 (or cache) of a different chiplet across the inter-chip data fabric 406.

[0041] Similarly, due to the cache probe request 422 resulting in a cache miss in both the cache memories 410 of the first and second chiplets 404, the DMA engine 414B of the data owning cache memory 410 performs a DMA transfer of the data associated with physical address Y. In other words, the DMA engine 414 closer to the data is the engine that performs the DMA transfer. The DMA engine 414 then signals that the DMA transfer is complete, such as by sending an interrupt signal to the processor core 408. In this manner, the hardware-managed coordination of the division of the entire DMA transfer packet and the performance of the DMA transfer (each DMA engine performs its transfer in parallel with the other DMA engines) by the closer DMA engines 414 avoids the need for the DMA transfer operation to read data from memory across all different channels (e.g., reading data from cache memories or DRAM associated with different chiplets requires traffic that goes through the interconnect and takes an energy performance hit).

[0042] Once all DMA engines 414 have completed their portions of the DMA job, a signal is returned to the requesting processor indicating that the DMA job is complete. For example, in some embodiments, this is accomplished by writing a value indicating completion to the DMA buffer, writing a value to some other location in memory, interrupting the requesting processor, etc. Since the requesting processor is only notified when the entire DMA job is complete, the DMA engines 414 synchronize in some manner such that DMA job completion is indicated only when all DMA engines 414 have completed their portions of the DMA job. In some embodiments, this is accomplished by a variety of techniques, such as atomically incrementing a counter in the DMA buffer to indicate how many DMA engines have completed their portions, through synchronization structures in memory, or through direct messaging between the DMA engines via fabric 406.

[0043] In other embodiments, described in more detail below, the DMA engine determines the division of transfer commands into packets based on the amount of data transfer and the interleaving of workload among multiple DMA engines (instead of looking at the physical address to be accessed as described in FIG. 4).

[0044] Referring to FIG. 5, a block diagram illustrating another example of a system implementing hardware-managed partitioning of transfer commands is shown, according to some embodiments. The device 500, or portions thereof, can be implemented using some or all of the semiconductor die 202 (as shown and described with respect to FIG. 2) and / or the device 100 (as shown and described with respect to FIGS. 1 and 2). In various embodiments, the device 500 includes a base die 502 that uses processing stacked die chiplets according to some embodiments. The base die 502 is formed as a single semiconductor chip package that includes N=2 communicatively coupled graphics processing stacked die chiplets 504. As shown, the processor base die 502 includes a first graphics processing stacked die chiplet 504A and a second graphics processing stacked die chiplet 504B.

[0045] In various embodiments, the base die 502 includes an inter-chip data fabric 506 that operates as a high bandwidth inter-die interconnect between chiplets (e.g., between the first graphics processing stacked die chiplet 504A and the second graphics processing stacked die chiplet 504B). In some embodiments, the base die 502 includes one or more processor cores 508 (e.g., CPUs and / or GPUs, or processor core dies) formed on each of the chiplets 504A-504B. Additionally, in various embodiments, each of the chiplets 504A-504B includes one or more levels of cache memory 510 and one or more memory PHYs (not shown) for communicating with an external system memory module 512, such as a dynamic random access memory (DRAM) module. Taken as a whole, the main memory (e.g., memory module 512) communicatively coupled to the multiple chiplets (e.g., chiplets 504A and 504B) and their local caches form a shared memory for the device 500. As will be appreciated, each chiplet 504 only has a direct physical connection to a portion of the overall shared memory system.

[0046] In various embodiments, base die 502 includes two or more DMA engines 514 (e.g., a first DMA engine 514A and a second DMA engine 514B) disposed on base die 502. In various embodiments, DMA engine 514 coordinates DMA transfers of data between devices and memory (or between different locations in memory) in system 500. In various embodiments, DMA engine 514 coordinates the movement of data between multiple devices / accelerators while computations are performed on other data, for example in processor core 508.

[0047] In various embodiments, one or more DMA engines 514 are part of a DMA controller (not shown) in some embodiments, although the terms DMA engine and DMA controller are used interchangeably herein. DMA engine 514 operates in response to commands to transfer data, for example, to or from one or more memory modules 512 without the involvement of processor core 508. Similarly, DMA engine 514 performs intra-chip data transfers in some embodiments. In some embodiments, DMA engine 514 includes a front-end packet processor (not shown) and a back-end command engine and transfer engine (not shown). The transfer engine is the portion of DMA engine 514 that performs the actual DMA transfers / data movement, and the command engine drives the transfer engine to tell it what operations to perform. In various embodiments, the packet processor includes one or more processing engines, such as an Advanced RISC Machine (ARM), a Microprocessor with Non-Interlocking Pipeline Stages (MIPS), a Serial Data Processor (SDP) and / or other RISC cores capable of performing buffer management, table lookups, queue management, fabric processing, and host processing functions known in the art. The packet processor, in various embodiments, is a microcontroller that waits for DMA packets to arrive. The packet processor receives and reads the DMA packets, then turns on the backend of the DMA controller to perform the data transfer.

[0048] In various embodiments, the back-end command engine is a microcontroller (or other firmware, state machine, etc.) that other system components communicate with to initiate DMA transfers. In some embodiments, the command engine includes a single command engine that drives multiple transfer engines. In other embodiments, the command engine includes a single command engine per transfer engine. Additionally, the DMA engine 514 includes some distributed algorithms that allow the command engines to agree on what operations to perform. For example, if a single DMA engine 514 includes two command engines, each command engine may be indicated by hardware that there is a packet waiting in memory. After reading the packet, the command engines synchronize with each other via a synchronization mechanism such as a command engine bus or a special location in memory that both command engines know to perform software synchronization such as a barrier or lock. In various embodiments, both command engines read the packet. Then, both command engines read the command and communicate with each other to split the data transfer. As will be appreciated, the command engines synchronize with each other at various times, such as to inform the requesting processor (e.g., the CPU or processor core 508) that a DMA transfer is complete (after both command engines have finished executing their respective portions of the DMA transfer).

[0049] Graphics processing stacked die chiplets 504 are described below in the particular context of GPU terminology for ease of illustration and explanation, but it should be appreciated that in various embodiments, the described architecture may be applicable to any of a variety of types of parallel processors (as more broadly described above with reference to FIGS. 2 and 3) without departing from the scope of the present disclosure. Additionally, while DMA engine 514 is shown in FIG. 5 as being located on top of base die 502, one skilled in the art will appreciate that the hardware-managed division of transfer commands may be performed by a DMA engine in a variety of system locations without departing from the scope of the present disclosure. For example, in some embodiments, the DMA engine is located on top of an individual compute chiplet (as described in more detail above with respect to FIG. 4). Similarly, while systems and devices are described herein in the context of a chiplet-based system, one skilled in the art will appreciate that the hardware-managed division of transfer commands is not limited to a particular architecture that may be implemented in any system configuration including multiple DMA engines, including a monolithic die.

[0050] In some embodiments, DMA requests are processed by placing DMA commands (also interchangeably referred to as packets) generated by one or more of the processors 508 into a memory-mapped IO (MMIO) location, such as a DMA buffer 516 (also interchangeably referred to as a DMA queue for holding DMA transfer commands). In various embodiments, a DMA buffer is a hardware structure into which read or write instructions are transferred so that the DMA engine 514 can read the DMA command (e.g., without having to go to DRAM memory). To perform a data transfer operation, in various embodiments, the DMA engine 514 receives an instruction (e.g., a DMA transfer command / data transfer request generated by the processor 508). For example, in some embodiments, the DMA engine 514 retrieves the DMA command (also known as a descriptor) from the DMA buffer 516 for processing.

[0051] DMA buffer 516 is shown in FIG. 5 as being implemented in base die 502 for ease of explanation, but one of ordinary skill in the art will recognize that DMA buffer 516 can be implemented in various components of the systems and devices described herein without departing from the scope of the present disclosure. For example, in some embodiments, DMA buffer 516 is configured in a main memory, such as memory module 512. That location of the command queue in memory is where DMA engine 514 goes to read transfer commands.

[0052] 5, in some embodiments, DMA engines 514A and 514B receive DMA job notifications 518 (a first DMA job notification 518A to DMA engine 514A and a second DMA job notification 518B to DMA engine 514B) from primary DMA engine 522. Primary DMA engine 522 acts as a central agent that mediates between the submitter of the DMA job (e.g., a server, a CPU, etc.) and the secondary / remote DMA engines (e.g., first DMA engine 514A and second DMA engine 514B) that perform the actual data transfer.

[0053] In some embodiments, such as those described above with respect to FIG. 4, primary DMA engine 522 sends first DMA job notification 518A and second DMA job notification 518B, which are the same signal, indicating that one or more DMA workloads or DMA jobs have been formed and filled into memory (such as DMA buffer 516 or system memory module 512). In response to receiving DMA notification 518, DMA engines 514A and 514B each independently fetch a DMA job description 520 from DMA buffer 516 and divide the DMA job workload in a distributed decision scheme in which two or more separate entities (e.g., two DMA engines 514A and 514B) each read the same packet and independently decide how to perform the DMA transfer. Alternatively, in other embodiments, primary DMA engine 522 sends the DMA job descriptor directly to DMA engines 514A and 514B, skipping the step of instructing the retrieval of the DMA job description from buffer 516.

[0054] In addition to independently fetching the same DMA job description from the same location (i.e., DMA buffer 516), each of DMA engines 514A and 514B independently determines the portion of the data transfer requested by the DMA transfer command to perform. For example, as described in more detail in connection with FIG. 4, DMA engine 514 performs hardware-managed arbitration based on splitting the entire DMA transfer packet and performing the DMA transfer (each DMA engine performs its transfer in parallel with the other DMA engines) with closer DMA engines 514 (e.g., closer via same-die cache or local DRAM without traversing the inter-chiplet interconnect).

[0055] In some embodiments, DMA engine 514 performs hardware-managed throttling via splitting transfer command packets based on data transfer amount and deciding to interleave workload among multiple DMA engines. For example, as illustrated in FIG. 5, DMA job description 520 is a single transfer command directing the transfer of 1000 megabytes of data from physical address X to physical address Y. In one embodiment, DMA engine 514 divides the total DMA transfer size equally between them. For example, device 500 includes two DMA engines, where first DMA engine 514A has global ID=0 and second DMA engine 514B has global ID=1. Thus, device 500 hardware includes information identifying the total number and relative order of DMA engines in the hardware configuration. In this example, after reading the same DMA job description 520, first DMA engine 514A (e.g., global ID=0) performs the first half of the DMA transfer by transferring the first 500 megabytes of data. Similarly, the second DMA engine 415B (eg, global ID=1) performs the second half of the DMA transfer in parallel by transferring the second 500 megabytes of data.

[0056] Those skilled in the art will recognize that this division of transfer command workload is not limited to the specific embodiment described herein, and any hardware-managed adjustment of DMA transfers and interleaving of DMA workloads among multiple DMA engines can be implemented without departing from the scope of the present disclosure. As non-limiting examples, in some embodiments, hardware-managed DMA workload division embodiments include odd / even byte number division (e.g., on a single byte basis), memory channel addressing, memory page size, memory page location, alternating data block basis that is not fully interleaved, or any other type of interleaving. As will be appreciated, in various embodiments, the size of the particular interleave depends on the particular interconnect hardware, memory transfer size, and caches in the system. The optimization of the interleave type and any data division type is variable depending on the type of system micro-architecture for a particular implementation. For example, in one embodiment, the interleave division size is based on the page translation size such that the transfer size matches the page translation size to reduce the amount of page translation that needs to be performed. In another embodiment, each DMA engine 514 is assigned a particular amount of data (e.g., 1000 megabytes) to be transferred in place of an address range. For example, in such an embodiment, each DMA engine 514 transfers 1000 megabytes of data in parallel with the other DMA engines, whereby DMA job description 520 is a single transfer command directing the transfer of 1000 megabytes of data from physical address X to physical address Y, and simply turns on the first DMA engine 514A while the second DMA engine 514B remains idle.

[0057] In another embodiment, the first DMA job notification 518A to DMA engine 514A and the second DMA job notification 518B to DMA engine 514B do not indicate that the DMA engines should fetch DMA job description 520 from DMA buffer 516. Rather, primary DMA engine 522 splits a single DMA job (e.g., DMA job description 520 directing the transfer of 1000 megabytes of data from physical address X to physical address Y) into multiple smaller jobs and sends different workloads separately to each separate DMA engine 514. Thus, each secondary / remote DMA engine sees a portion of the original DMA job description 520.

[0058] In this example, primary DMA engine 522 splits DMA job description 520 into two smaller workloads by sending a first DMA job notification 518A instructing first DMA engine 514A (as a secondary / remote DMA engine) to perform the first half of the DMA transfer by transferring the first 500 megabytes of data. Similarly, primary DMA engine 522 sends a second DMA job notification 518B instructing second DMA engine 514B (as a secondary / remote DMA engine) to perform the second half of the DMA transfer by transferring the second 500 megabytes of data. After each individual secondary / remote DMA engine finishes its respective portion of the DMA job, secondary DMA engine 514 notifies primary DMA engine 522 of their work completion. Primary DMA engine 522 then communicates that the entire job is complete.

[0059] In this example, an individual DMA engine 514 executes the entirety of the job sent to it (e.g., DMA engine 514 acts as a pure dummy by executing the workload assigned to it without any discretion, as opposed to the embodiments described above in which each DMA engine individually determines what portion of the workload is locally applicable to it.) In various embodiments, primary DMA engine 522 determines the number of DMA engines available to the primary DMA engine in order to allocate available work and / or available bandwidth or other processing resources to each remote DMA engine.

[0060] 6 is a block diagram of a method 600 for performing hardware-managed partitioning of DMA transfer commands, according to some embodiments. For ease of illustration and explanation, method 600 is described below in the exemplary context with reference to the systems and devices of FIGS. 1-5. However, method 600 is not limited to these exemplary contexts, and instead may be employed in different embodiments for any of a variety of possible system configurations using the guidelines provided herein.

[0061] Method 600 begins at block 602 by a first DMA engine accessing a DMA transfer command to determine a first portion of a data transfer requested by the DMA transfer command. For example, as shown in FIG. 4 with respect to DMA engine 414, DMA engines 414A and 414B receive DMA notification 418 indicating that one or more DMA workloads or DMA jobs have been created and filled in memory (such as DMA buffer 416 or system memory module 412). In various embodiments as described above, DMA notification 418 includes a doorbell ring or other notifier indicating that DMA descriptors are ready in memory. In response to receiving DMA notification 418, DMA engines 414A and 414B each independently fetch a DMA job description 420 from DMA buffer 416.

[0062] In some embodiments, each of the DMA engines 414A and 414B independently determines the portion of the data transfer requested by the DMA transfer command to perform. In one embodiment, each of the DMA engines 414A and 414B issues a speculative DMA transfer by broadcasting a cache probe request 422 (e.g., a read probe and / or a write probe) to the cache memory 410 and main memory (e.g., memory module 412) of their respective chiplet 404. In response to the cache probe request, the cache memory 410 and main memory of each chiplet 404 return one or more return responses to the requesting DMA engine. For example, in the case of a DMA read probe, the cache memory 410 can return a cache hit or a cache miss to indicate whether the requested data was found in the cache memory 410.

[0063] 5 for DMA engine 514, DMA engines 514A and 514B receive DMA job notifications 518 (a first DMA job notification 518A to DMA engine 514A and a second DMA job notification 518B to DMA engine 514B) from primary DMA engine 522. Primary DMA engine 522 splits a single DMA job (e.g., DMA job description 520 directing the transfer of 1000 megabytes of data from physical address X to physical address Y) into multiple smaller jobs and sends different workloads separately to each separate DMA engine 514. Thus, each secondary / remote DMA engine sees a portion of the original DMA job description 520.

[0064] Method 600 continues at block 604 with initiating a transfer of a first portion of the data transfer by the first DMA engine based at least in part on the DMA transfer command. For example, as shown in FIG. 4 with respect to the first DMA engine 414A, to determine whether a portion of the data associated with addresses X and Y is currently resident in a cache memory 410 (e.g., an L3 or some last level cache) of the first graphics processing stacked die chiplet 404A, the DMA engine 414A broadcasts a cache probe request 422 to the cache memory 410 (including a cache controller [not shown]) before the memory module 412 and receives one or more return responses indicating a cache hit for address X and a cache miss for address Y. In the exemplary embodiment of FIG. 4, the data for physical address X and physical address Y is stored in a memory module 412 directly connected to the second graphics processing stacked die chiplet 404B. Because the cache memory 410 of the first graphics processing stacked die chiplet 404A already contains the requested data for physical address X, the DMA engine 414A performs the portion of the data transfer associated with address X (as opposed to retrieving the data from the cache of a different chiplet).

[0065] In some embodiments, as illustrated in FIG. 5 for DMA engine 514, DMA engine 514 performs hardware-managed throttling via splitting packets of transfer commands based on data transfer amount and deciding to interleave workload among multiple DMA engines. For example, as illustrated in FIG. 5, DMA job description 520 is a single transfer command directing the transfer of 1000 megabytes of data from physical address X to physical address Y. In one embodiment, DMA engine 514 divides the total DMA transfer size equally between them. For example, device 500 includes two DMA engines, where first DMA engine 514A has global ID=0 and second DMA engine 514B has global ID=1. Thus, the hardware of device 500 includes information identifying the total number and relative order of DMA engines in the hardware configuration. In this example, after reading the same DMA job description 520, first DMA engine 514A (e.g., global ID=0) performs the first half of the DMA transfer by transferring the first 500 megabytes of data. Similarly, the second DMA engine 415B (eg, global ID=1) performs the second half of the DMA transfer in parallel by transferring the second 500 megabytes of data.

[0066] Method 600 continues at block 606 with initiating a transfer of a second portion of the data transfer by the second DMA engine based at least in part on the DMA transfer command. For example, as shown in FIG. 4 with respect to the first DMA engine 414A, to determine whether a portion of the data associated with addresses X and Y is currently resident in the cache memory 410 of the second graphics processing stacked die chiplet 404B, the DMA engine 414B similarly broadcasts a cache probe request 422 to the cache memory 410 and receives one or more return responses indicating a cache miss for address X and a cache miss for address Y. Because the cache probe request 422 results in a cache miss in the cache memories 410 of both the first and second chiplets 404, the DMA engine 414B of the data owning cache memory 410 performs a DMA transfer of the data associated with physical address Y. In other words, the DMA engine 414 that is closer to the data is the engine that performs the DMA transfer.

[0067] At block 608, after transferring the first and second portions of the data transfer, an indication is generated that the data transfer requested by the DMA transfer command is complete. For example, as illustrated in FIG. 4, DMA engine 414 signals that the DMA transfer is complete, such as by sending an interrupt signal to processor core 408 after some form of synchronization between DMA engines 414. Similarly, with reference to FIG. 5, secondary DMA engine 514 notifies primary DMA engine 522 of the completion of each secondary / remote DMA engine's work after they finish their respective portions of the DMA job. Primary DMA engine 522 then communicates that the entire job is complete.

[0068] Thus, as described herein, the hardware-managed coordination of splitting of an entire DMA transfer packet by closer DMA engines and the execution of the DMA transfer (each DMA engine performs its transfer in parallel with other DMA engines) avoids the need for the DMA transfer operation to read data from memory across all different channels (e.g., reading data from cache memory or DRAM associated with different chiplets requires traffic that traverses the interconnect and takes an energy performance hit). For example, one skilled in the art will recognize that round-trip DMA traffic competes with other traffic and consumes power. Thus, the hardware-managed splitting of DMA commands results in reduced power and increased bandwidth for other traffic uses.

[0069] As disclosed herein, in some embodiments, a method includes initiating a transfer of a first portion of a data transfer by a first DMA engine based at least in part on a DMA transfer command, and initiating a transfer of a second portion of the data transfer by a second DMA engine based at least in part on the DMA transfer command. In one aspect, the method includes receiving, by the first DMA engine, a DMA notification indicating that the DMA transfer command is stored in a DMA buffer in a system memory, and fetching, by the first DMA engine, the DMA transfer command from the DMA buffer. In another aspect, initiating a transfer of the first portion of the data transfer by the first DMA engine includes sending, by the first DMA engine, a cache probe request to a cache memory, and transferring the first portion of the data transfer based on receiving a return response indicating a cache hit in the cache memory. In yet another aspect, initiating a transfer of the second portion of the data transfer by the second DMA engine includes sending, by the second DMA engine, a cache probe request to the cache memory, and transferring the second portion of the data transfer from an owner main memory based on receiving a return response indicating a cache miss in the cache memory.

[0070] In one aspect, determining the first portion of the data transfer further includes interleaving a total DMA transfer size between the first and second DMA engines. In another aspect, the method includes receiving a DMA transfer command at a primary DMA engine and dividing the DMA transfer command into multiple smaller workloads. In yet another aspect, the method includes receiving any of the multiple smaller workloads from the primary DMA engine.

[0071] In some embodiments, a processor device includes a base integrated circuit (IC) die including a plurality of processing stacked die chiplets 3D stacked on the base IC die, the base IC die including an inter-chip data fabric communicatively coupling the processing stacked die chiplets to one another, and a plurality of DMA engines 3D stacked on the base IC die, each of the plurality of DMA engines configured to perform a portion of a data transfer requested by a DMA transfer command. In one aspect, each of the plurality of DMA engines includes a single command engine that drives the plurality of transfer engines. In another aspect, each of the plurality of DMA engines is configured to receive a DMA notification indicating that the DMA transfer command is stored in a DMA buffer in a system memory. In yet another aspect, a first DMA engine of the plurality of DMA engines is configured to transmit a cache probe request to a cache memory communicatively coupled to the first processing stacked die chiplet and transfer a first portion of the data transfer based on receiving a return response indicating a cache hit in the cache memory. In yet another aspect, a second DMA engine of the plurality of DMA engines is configured to transmit a cache probe request to a cache memory communicatively coupled to the second processing stacked die chiplet and transfer a second portion of the data transfer from an owner main memory based on receiving a return response indicating a cache miss in the cache memory.

[0072] In one aspect, each of the multiple DMA engines is configured to independently determine portions of the data transfer by interleaving a total DMA transfer size among the multiple DMA engines. In another aspect, the processor device includes a primary DMA engine configured to receive a DMA transfer command and divide the DMA transfer command into multiple smaller workloads. In yet another aspect, the primary DMA engine is further configured to send different ones of the multiple smaller workloads to each of the multiple DMA engines.

[0073] In some embodiments, a system includes a host processor communicatively coupled to a parallel processor multi-chip module, the parallel processor multi-chip module including a base integrated circuit (IC) die including a plurality of processing stacked die chiplets 3D stacked on the base IC die including an inter-chip data fabric communicatively coupling the processing stacked die chiplets to one another, and a plurality of DMA engines 3D stacked on the base IC die, each of the plurality of DMA engines configured to perform a portion of a data transfer requested by a DMA transfer command. In one aspect, the system includes a primary DMA engine configured to receive a DMA transfer command and divide the DMA transfer command into a plurality of smaller workloads, the primary DMA engine further configured to send different ones of the plurality of smaller workloads to each of the plurality of DMA engines. In another aspect, each of the plurality of DMA engines is configured to independently determine a portion of the data transfer by interleaving a total DMA transfer size among the plurality of DMA engines. In yet another aspect, a first DMA engine of the plurality of DMA engines is configured to transfer a first portion of the data transfer based on sending a cache probe request to a cache memory communicatively coupled to the first processing stacked die chiplet and receiving a return response indicative of a cache hit in the cache memory. In yet another aspect, a second DMA engine of the plurality of DMA engines is configured to transfer a second portion of the data transfer from an owner main memory based on sending a cache probe request to a cache memory communicatively coupled to the second processing stacked die chiplet and receiving a return response indicative of a cache miss in the cache memory.

[0074] A computer-readable storage medium includes any non-transitory storage medium or combination of non-transitory storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tape, magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or micro-electromechanical systems (MEMS) based storage media. The computer-readable storage medium (e.g., system RAM or ROM) may be internal to the computing system, the computer-readable storage medium (e.g., a magnetic hard drive) may be permanently attached to the computing system, the computer-readable storage medium (e.g., an optical disk or Universal Serial Bus (USB)-based flash memory) may be removably attached to the computing system, or the computer-readable storage medium (e.g., network-accessible storage (NAS)) may be coupled to the computer system via a wired or wireless network.

[0075] In some embodiments, certain aspects of the techniques described above are implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied in a non-transitory computer-readable storage medium. The software may include instructions and specific data that, when executed by the one or more processors, operate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer-readable storage medium may include, for example, a magnetic or optical disk storage device, a solid-state storage device such as a flash memory, a cache, a random access memory (RAM), or other non-volatile memory device(s), etc. The executable instructions stored in the non-transitory computer-readable storage medium may be implemented as source code, assembly language code, object code, or other form of instructions that can be interpreted or otherwise executed by one or more processors.

[0076] In addition to the above, it should be noted that not all activities or elements described in the summary description are required, some of the specific activities or devices may not be required, one or more additional activities may be performed, and one or more additional elements may be included. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, those skilled in the art will appreciate that various changes and modifications can be made without departing from the scope of the invention as set forth in the claims. Thus, the specification and drawings should be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the invention.

[0077] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, the benefits, advantages, solutions to problems, and features by which any benefit, advantage, or solution may occur or be manifested are not to be construed as critical, essential, or essential features of any or all claims. Moreover, the specific embodiments described above are illustrative only, as the disclosed invention may be modified and practiced in different but similar manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as set forth in the appended claims. It is therefore apparent that the specific embodiments described above may be altered or modified, and all such variations are considered to be within the scope of the disclosed invention. Accordingly, the protection sought herein is set forth in the appended claims.

Claims

1. 1. A method comprising: initiating a transfer of a first portion of the data transfer by the first DMA engine based at least in part on the DMA transfer command; and initiating a transfer of a second portion of the data transfer by a second DMA engine based at least in part on the DMA transfer command. method.

2. the first DMA engine receiving a DMA notification indicating that the DMA transfer command is stored in a DMA buffer in system memory; the first DMA engine fetching the DMA transfer command from the DMA buffer; 10. The method of claim 1.

3. Initiating the transfer of the first portion of the data transfer by the first DMA engine includes: the first DMA engine sending a cache probe request to a cache memory; transferring the first portion of the data transfer based on receiving a return response indicating a cache hit in the cache memory. The method of claim 2.

4. Initiating the transfer of the second portion of the data transfer by the second DMA engine includes: the second DMA engine sending a cache probe request to a cache memory; transferring the second portion of the data transfer from the owner main memory based on receiving a return response indicating a cache miss in the cache memory. The method of claim 2.

5. determining the first portion of the data transfer includes interleaving a total DMA transfer size between the first DMA engine and the second DMA engine; 10. The method of claim 1.

6. receiving the DMA transfer command at a primary DMA engine and dividing the DMA transfer command into a plurality of smaller workloads; 10. The method of claim 1.

7. receiving any of the plurality of smaller workloads from the primary DMA engine; The method of claim 6.

8. 1. A processor device, comprising: a base integrated circuit (IC) die including a plurality of processing stacked die chiplets 3D stacked on the base IC die, the base IC die including an inter-chip data fabric communicatively coupling the plurality of processing stacked die chiplets to one another; a plurality of DMA engines 3D stacked on the base IC die, each DMA engine configured to perform a portion of a data transfer requested by a DMA transfer command; Processor device.

9. each of the plurality of DMA engines includes a single command engine that drives a plurality of transfer engines; The processor device of claim 8.

10. each of the plurality of DMA engines is configured to receive a DMA notification indicating that the DMA transfer command has been stored in a DMA buffer in a system memory; The processor device of claim 8.

11. a first DMA engine of the plurality of DMA engines configured to transmit a cache probe request to a cache memory communicatively coupled to a first processing stacked die chiplet and transfer a first portion of the data transfer based on receiving a return response indicating a cache hit in the cache memory; The processor device of claim 8.

12. a second DMA engine of the plurality of DMA engines configured to transmit the cache probe request to a cache memory communicatively coupled to a second processing stacked die chiplet, and transfer a second portion of the data transfer from an owner main memory based on receiving a return response indicating a cache miss in the cache memory; The processor device of claim 11.

13. each of the plurality of DMA engines configured to independently determine portions of the data transfer by interleaving a total DMA transfer size among the plurality of DMA engines; The processor device of claim 8.

14. a primary DMA engine configured to receive the DMA transfer command and divide the DMA transfer command into a plurality of smaller workloads; The processor device of claim 8.

15. the primary DMA engine is configured to send a different one of the plurality of smaller workloads to each of the plurality of DMA engines; The processor device of claim 14.

16. A system comprising: a host processor communicatively coupled to the parallel processor multi-chip module; The parallel processor multi-chip module comprises: a base integrated circuit (IC) die including a plurality of processing stacked die chiplets 3D stacked on the base IC die, the base IC die including an inter-chip data fabric communicatively coupling the plurality of processing stacked die chiplets to one another; a plurality of DMA engines 3D stacked on the base IC die, each DMA engine configured to perform a portion of a data transfer requested by a DMA transfer command; system.

17. The method of claim 1, further comprising: a primary DMA engine configured to receive the DMA transfer command and divide the DMA transfer command into a plurality of smaller workloads, the primary DMA engine configured to send a different one of the plurality of smaller workloads to each of the plurality of DMA engines.

17. The system of claim 16.

18. Each of the plurality of DMA engines is configured to independently determine portions of the data transfer by interleaving a total DMA transfer size among the plurality of DMA engines.

17. The system of claim 16.

19. The method of claim 18, wherein a first DMA engine of the plurality of DMA engines is configured to transmit a cache probe request to a cache memory communicatively coupled to a first processing stacked die chiplet, and transfer a first portion of the data transfer based on receiving a return response indicating a cache hit in the cache memory.

17. The system of claim 16.

20. A second DMA engine of the plurality of DMA engines is configured to transmit the cache probe request to a cache memory communicatively coupled to a second processing stacked die chiplet, and transfer a second portion of the data transfer from an owner main memory based on receiving a return response indicating a cache miss in the cache memory.

20. The system of claim 19.

Images