Hardware management of direct memory access commands

Abstract

This invention provides processor devices and systems that improve the use of direct memory access (DMA) engines. [Solution] In system 300, a method for hardware management of DMA transfer commands includes, by a first DMA engine 314, accessing the DMA transfer command and determining a first portion of the data transfer requested by the DMA transfer command. The transfer of the first portion of the data transfer by the first DMA engine is initiated at least partially based on the DMA transfer command. Similarly, the second portion of the data transfer by the second DMA engine is initiated at least partially based on the DMA transfer command. After the first and second portions of the data transfer have been transferred, an indicator is generated to signal the completion of the data transfer requested by the DMA transfer command.

Description

Background Art

[0001] A system direct memory access (DMA) engine is a module that coordinates direct memory access transfers of data between devices (e.g., input / output interfaces and display controllers) and memory within a computer system, or between different locations within memory. The DMA engine is often located on a processor such as a central processing unit (CPU) or a graphics processing unit (GPU), and receives commands from an application operating on the processor. Based on the commands, the DMA engine reads data from a DMA source (e.g., a first memory buffer defined within memory) and writes the data to a DMA destination (e.g., a second buffer defined within memory).

[0002] The present disclosure may be better understood by reference to the accompanying drawings, in which numerous features and advantages will become apparent to those skilled in the art. The use of the same reference numerals in different drawings indicates similar or identical items.

Brief Description of the Drawings

[0003] [Figure 1] FIG. 1 is a block diagram of a computing system implementing a multi-die processor according to some embodiments. [Figure 2] FIG. 2 is a block diagram of a portion of an exemplary computing system for implementing hardware management of DMA commands according to some embodiments. [Figure 3] FIG. 3 is a block diagram showing a portion of an exemplary multi-processor computing system for implementing hardware management of DMA commands according to some embodiments. [Figure 4] FIG. 4 is a block diagram showing an example of a system for implementing hardware management partitioning of transfer commands based on cache status according to some embodiments. [Figure 5]This block diagram shows another example of a system that implements hardware-managed partitioning of transfer commands, according to several embodiments. [Figure 6] This flowchart illustrates how to perform hardware-managed partitioning of DMA transfer commands according to several embodiments. [Modes for carrying out the invention]

[0004] Conventional processors include one or more direct memory access engines (DMCs) for reading and writing blocks of data stored in system memory. The DMCs relieve the processor cores of the burden of managing transfers. In response to data transfer requests from the processor cores, the DMCs provide the necessary control information to the corresponding source and destination, enabling the data transfer operation to be performed without delaying the computation code, thus allowing communication and computation to overlap in time. By processing the formation of control information and communication asynchronously, the DMCs are freed to perform other tasks while awaiting the satisfaction of data transfer requests.

[0005] Distributed architectures are becoming increasingly common as an alternative to monolithic processing architectures, where physically or logically separated processing units work collaboratively through high-performance interconnects. An example of a distributed architecture is the chiplet architecture, which gains the advantage of manufacturing some parts of the processing unit on smaller nodes, while allowing other parts to be manufactured on larger nodes if they do not benefit from the reduced scale of the smaller nodes. The number of direct memory access engines is likely to increase in chiplet-based systems (for example, compared to an equivalent monolithic non-chiplet-based design).

[0006] To improve system performance by enhancing the use of direct memory access engines, Figures 1 to 6 illustrate systems and methods that utilize hardware-managed adjustments for processing direct memory transfer commands. In various embodiments, the method for hardware management of DMA transfer commands includes a first DMA engine accessing the DMA transfer command and determining a first portion of the data transfer requested by the DMA transfer command. The transfer of the first portion of the data transfer by the first DMA engine is initiated at least partially based on the DMA transfer command. Similarly, a second portion of the data transfer by a second DMA engine (i.e., a DMA engine different from the first DMA engine) is initiated at least partially based on the DMA transfer command. After transferring the first and second portions of the data transfer, an indicator is generated to signal the completion of the data transfer requested by the DMA transfer command. In this way, the work specified by the transfer command is divided among the DMA engines, thus increasing total bandwidth usage without the need to enlarge individual DMA engines or add features to increase overall DMA throughput or data fabric bandwidth usage.

[0007] Figure 1 shows a block diagram of one embodiment of a computing system 100 implementing a multi-die processor, according to several embodiments. In various embodiments, the computing system 100 includes at least one or more processors 102A-102N, a fabric 104, an input / output (I / O) interface 106, a memory controller 108, a display controller 110, and other devices 112. In various embodiments, the computing system 100 includes a host processor 114, such as a central processing unit (CPU), to support the execution of instructions for graphics and other types of workloads. In various embodiments, the computing system 100 includes any of a computer, laptop, mobile device, server, or various other types of computing systems or devices. Note that the number of components of the computing system 100 varies in some embodiments. Also note that in some embodiments, the computing system 100 includes other components not shown in Figure 1. In addition, in other embodiments, the computing system 100 is constructed in a manner other than that shown in Figure 1.

[0008] Fabric 104 represents any communication interconnection conforming to any of the various types of protocols used to communicate between components of computing system 100. Fabric 104 provides data paths, switches, routers, and other logic connecting processing units 102, I / O interfaces 106, memory controllers 108, display controllers 110, and other devices 112 to each other. Fabric 104 handles request, response, and data traffic, as well as probe traffic, to facilitate coherence. Fabric 104 also handles interrupt request routing and configuration access paths to various components of computing system 100. In addition, Fabric 104 handles configuration request, response, and configuration data traffic. In some embodiments, Fabric 104 is bus-based, including hierarchical buses with shared bus configurations, crossbar configurations, and bridges. In other embodiments, Fabric 104 is packet-based and hierarchical, having bridges, crossbars, point-to-point, or other interconnections. From the perspective of Fabric 104, the other components of computing system 100 are referred to as “clients”. Fabric 104 is configured to process requests generated by various clients and pass those requests on to other clients.

[0009] The memory controller 108 represents any number and type of memory controller coupled to any number and type of memory devices. For example, the types of memory devices coupled to the memory controller 108 include Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), NAND flash memory, NOR flash memory, Ferroelectric Random Access Memory (FeRAM), etc. The memory controller 108 is accessible via the fabric 104 by the processor 102, I / O interface 106, display controller 110, and other devices 112. The I / O interface 106 represents any number and type of I / O interface (e.g., peripheral component interconnect (PCI) bus, PCI-Extended (PCI-X), PCI Express (PCIE) bus, Gigabit Ethernet (GBE) bus, Universal Serial Bus (USB)). Various types of peripheral devices are coupled to the I / O interface 106. Such peripheral devices include (but are not limited to) displays, keyboards, mice, printers, scanners, joysticks, other types of game controllers, media recording devices, external storage devices, network interface cards, etc. Other devices 112 represent any number and type of devices (e.g., multimedia devices, video codecs).

[0010] In various embodiments, each of the processors 102 is a parallel processor (e.g., a vector processor, a graphics processing unit (GPU), a general-purpose GPU (GPGPU), a non-scalar processor, a highly parallel processor, an artificial intelligence (AI) processor, an inference engine, a machine learning processor, or other multithreaded processing unit). Each parallel processor 102 is constructed as a multi-chip module (e.g., a semiconductor die package) including two or more base integrated circuit dies (described in more detail below with respect to Figure 2) communicatively coupled with a bridge chip so that the parallel processors can be used (e.g., addressable) like a single semiconductor integrated circuit. As used in this disclosure, the terms “die” and “chip” are used interchangeably. Those skilled in the art will recognize that conventional (e.g., non-multi-chip) semiconductor integrated circuits are manufactured as dies (e.g., single-chip ICs) formed on or within a wafer and later separated from the wafer (e.g., when the wafer is diced), and in many cases multiple ICs are manufactured simultaneously within a wafer. ICs, and possibly discrete circuits, as well as possibly other components (such as printed circuit boards, interposers, and possibly other non-semiconductor packaging substrates), are assembled within a multi-die parallel processor.

[0011] As will be described in more detail with respect to Figures 2 to 6 below, in various embodiments, each of the individual processors 102 includes one or more base IC dies using processing stacked die chiplets according to several embodiments. The base die is formed as a single semiconductor chip package including N communicably coupled graphics processing stacked die chiplets. In various embodiments, the base IC die includes two or more DMA engines that coordinate DMA transfers of data between the device and memory (or between different locations within memory). Various embodiments are described below in the specific context of CPUs and GPUs for the sake of illustration and explanation, but it should be recognized that the concepts described herein are equally applicable to other processors, including accelerated processing units (APUs), discrete GPUs (dGPUs), artificial intelligence (AI) accelerators, and other parallel processors.

[0012] Referring to Figure 2, a block diagram of a portion of an exemplary computing system 200 is shown. In some examples, the computing system 200 is implemented using some or all of the device 100 as illustrated and described with respect to Figure 1. The computing system 200 includes at least a first semiconductor die 202. In various embodiments, the semiconductor die 202 includes one or more processors 204A-204N, an input / output (I / O) interface 206, an in-die interconnect 208, a memory controller 210, and a network interface 212. In other examples, the computing system 200 includes further components, different components, and / or are arranged in different ways. In some embodiments, the semiconductor die 202 is a multi-chip module configured as a semiconductor die package including two or more integrated circuit (IC) dies, so that the processors can be used as a single semiconductor integrated circuit. As used in this disclosure, the terms “die” and “chip” may be used interchangeably.

[0013] In some embodiments, each of the processors 204A to 204N includes one or more processing devices. In one embodiment, at least one of the processors 204A to 204N includes one or more general-purpose processing devices such as CPUs. In some embodiments, such processing devices are implemented using processor 102 as illustrated and described with respect to Figure 1. In another embodiment, at least one of the processors 204A to 204N includes one or more parallel processors. Examples of parallel processors include GPUs, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and the like.

[0014] The I / O interface 206 includes one or more I / O interfaces (e.g., Peripheral Component Interconnection (PCI) bus, PCI Expansion (PCI-X), PCI EIE (PCI Express) bus, Gigabit Ethernet® (GBE) bus, Universal Serial Bus (USB), etc.). In some embodiments, the I / O interface 206 is implemented using an input driver 112 and / or an output driver 114, as illustrated and described with respect to Figure 1. Various types of peripheral devices can be coupled to the I / O interface 206. Such peripheral devices include (but are not limited to) displays, keyboards, mice, printers, scanners, joysticks, other types of game controllers, media recording devices, external storage devices, network interface cards, etc. In some embodiments, such peripheral devices are implemented using an input device 108 and / or an output device 118, as illustrated and described with respect to Figure 1.

[0015] In various embodiments, each processor includes a cache subsystem having one or more levels of cache. In some embodiments, each of processors 204A to 204N includes a cache (e.g., a Level 3 (L3) cache) shared among multiple processor cores of the core complex. The memory controller 210 includes at least one memory controller accessible by processors 204A to 204N, such as accessible via the die interconnect 208. In various embodiments, the memory controller 210 includes one or more memory controllers of any suitable type. Each memory controller is coupled to (or communicates with) any number and types of memory devices (not shown) and controls access to them. In some embodiments, such memory devices include dynamic random access memory (DRAM), static random access memory (SRAM), NAND flash memory, NOR flash memory, ferroelectric memory (FeRAM), or any other suitable memory device. The die interconnect 208 includes any computer communication medium suitable for communication between devices shown in Figure 2, such as a bus or data fabric.

[0016] Figure 3 is a block diagram showing a portion of an exemplary multiprocessor computing system 300. The system 300 or a portion thereof can be implemented using part or all of the semiconductor die 202 (as illustrated and described with respect to Figure 2) and / or device 100 (as illustrated and described with respect to Figures 1 and 2). In various embodiments, the system 300 includes a processor multichip module 302 using processing stacked die chiplets according to some embodiments. The processor multichip module 302 is formed as a single semiconductor chip package including N=3 communicably coupled graphics processing stacked die chiplets 304. As shown, the processor multichip module 302 includes a first graphics processing stacked die chiplet 304A, a second graphics processing stacked die chiplet 304B, and a third graphics processing stacked die chiplet 304C.

[0017] The graphics processing stacked die chiplet 304 is described below in the specific context of GPU terminology for the sake of illustrative and explanatory purposes, but it should be noted that in various embodiments, the described architecture is applicable to any of the various types of parallel processors (as more broadly described above with reference to Figures 2 and 3) without departing from the scope of this disclosure. In addition, in various embodiments, the term “chiplet” as used herein refers to any device having the following characteristics, but is not limited to: 1) the chiplet includes an active silicon die containing at least a portion of the computational logic used to solve the entire problem (i.e., the computational workload is distributed across multiple of these active silicon dies); 2) the chiplets are packaged together as a monolithic unit on the same substrate; and 3) the programming model maintains the concept that these individual computational dies (i.e., graphics processing stacked die chiplets) are combined as a single monolithic unit (i.e., each chiplet is not exposed as a separate device to applications that use the chiplet to process computational workloads).

[0018] In various embodiments, the processor multichip module 302 includes an interchip data fabric 306 that operates as a high-bandwidth inter-die interconnect between chiplets (e.g., between any combination of the first graphics processing stacked die chiplet 304A, the second graphics processing stacked die chiplet 304B, and the third graphics processing stacked die chiplet 304C). In some embodiments, the processor multichip module 302 includes one or more processor cores 308 (e.g., a CPU and / or GPU, or a processor core die) formed on each of the chiplets 304A to 304C. Furthermore, in various embodiments, each of the chiplets 304A to 304C includes one or more levels of cache memory 310 and one or more memory PHYs (not shown) for communicating with an external system memory module 312, such as a dynamic random access memory (DRAM) module.

[0019] Furthermore, each of the chiplets 304A to 304C includes one or more DMA engines 314. In various embodiments, one or more DMA engines 314 coordinate DMA transfers of data between devices and memory (or between different locations within memory) within the system 300. In various embodiments, one or more DMA engines 314 coordinate the movement of data between multiple devices / accelerators, while calculations are performed on other data in, for example, the processor core 308. In various embodiments, one or more DMA engines 314 are part of a DMA controller (not shown) in some embodiments, but the terms DMA engine and DMA controller are used interchangeably herein. The DMA engines 314 operate to transfer data to, for example, one or more memory modules 312, without the involvement of the processor core 308, in response to commands. Similarly, the DMA engines 314 perform in-chip data transfers in some embodiments. As understood, the DMA engines 314 relieve the processor core of the burden of managing data transfers and are used in various embodiments as global data transfer agents to handle various data transfer requirements from software, such as inter-memory data copies.

[0020] One or more DMA engines 314 provide fetching and decoding of command packets from the application / agent queue and their respective DMA buffers to perform desired data transfer operations specified by DMA commands, also known as descriptors. DMA commands include memory flow commands (e.g., read / get or write / put commands for transferring data into or out of memory) that control the transfer or forwarding of memory locations containing data or instructions. In various embodiments, a DMA command descriptor indicates, for each data transfer operation, the source address from which to read data, the transfer size, and the destination address to which to write data. Descriptors are generally organized in memory as a linked list or chain, and each descriptor includes a field indicating the memory address of the next descriptor to be executed. In various embodiments, a descriptor is an array of commands with valid bits, the commands are of a known size, and one or more DMA engines 314 stop when they reach an invalid command. The last descriptor in the list has a null pointer in the "next descriptor" field, indicating to the DMA engine that there are no more commands to execute and the DMA should become idle once the end of the chain is reached.

[0021] In response to a data transfer request from the processor core, the DMA engine 314 provides the necessary control information to the corresponding source and destination so that the data transfer request is fulfilled. Because the DMA engine 314 handles the formation and communication of the control information, the processor core can perform other tasks while waiting for the data transfer request to be fulfilled. In various embodiments, each of the DMA engines 314 includes one or more dedicated auxiliary processors that transfer data between memory and / or peripheral input / output (I / O) devices and locations in memory without going through the processor core or CPU.

[0022] In some embodiments, a DMA request is processed by placing DMA commands generated by one or more processors 308 into a memory-mapped I / O (MMIO) location such as a DMA buffer 316 (also interchangeably called a DMA queue for holding DMA transfer commands). In various embodiments, the DMA buffer is a hardware structure to which read or write instructions are transferred so that a DMA engine 314 can read DMA commands (without having to go to DRAM memory, for example). To perform a data transfer operation, in various embodiments, the DMA engine 314 receives instructions generated by processors 308 (e.g., DMA transfer commands / data transfer requests) by accessing a sequence of commands in the DMA buffer 316. The DMA engine 314 then retrieves the DMA commands (also known as descriptors) from the DMA buffer 316 for processing. In some embodiments, the DMA commands specify, for example, the start addresses for direct virtual memory access (DVMA) and I / O bus access, as well as the transfer length up to a predetermined maximum value.

[0023] Although the DMA buffer 316 is shown in Figure 3 as being implemented on a chiplet 304 for ease of explanation, those skilled in the art will recognize that the DMA buffer 316 can be implemented in various components of the systems and devices described herein without departing from the scope of this disclosure. For example, in some embodiments, the DMA buffer 316 is configured in main memory, such as a memory module 312. Its location in the command queue in memory is where the DMA engine 314 goes to read the transfer commands. In various embodiments, the DMA buffer 316 is configured as one or more ring buffers (addressed, for example, by modulo addressing).

[0024] The DMA engine 314 accesses (or receives) DMA transfer commands from the DMA buffer 316 via a bus (not shown). Based on the received command, in some embodiments, the DMA engine 314 reads data from any memory (e.g., memory module 312) via the data fabric 306, buffers it, and writes the buffered data to any memory via the data fabric 306. In some embodiments, the DMA source and the DMA destination are physically located on different devices (e.g., different die). Similarly, in a multi-processor system, the DMA source and the DMA destination may, in some cases, be located on different devices associated with different processors. In such cases, the DMA engine 314 resolves virtual addresses to obtain physical addresses and issues remote read and / or write commands to affect the DMA transfer. For example, in various embodiments, based on the received command, the DMA engine 314 transmits a message having commands that affect the DMA transfer to the data fabric device.

[0025] During DMA, one or more processor cores 308 can queue DMA commands in the DMA buffer 316 and signal their presence to the DMA engine 314. For example, in some embodiments, an application program running on system 300 prepares a suitable chain of descriptors in memory accessible to the DMA engine (e.g., DMA buffer 316) to initiate a chain of DMA data transfers. The processor core 308 then sends a message (or other notification) to the DMA engine 314 indicating the memory address of the first descriptor in the chain, which is a request to the DMA engine to begin executing the descriptor. Typically, the application sends the message to the DMA engine's "doorbell," i.e., a control register with a specific bus address designated for this purpose. Sending such a message to initiate DMA execution is known as "ringing the doorbell" of the DMA engine 314. The DMA engine 314 responds by reading and executing the first descriptor and updating the descriptor's status field to indicate to the application that the descriptor has been executed. The DMA engine 314 traverses the "next" field throughout the linked list, marking each descriptor as executed until it reaches the null pointer in the last descriptor. After executing the last descriptor, the DMA engine 314 becomes idle and ready to receive a new list for execution.

[0026] In various embodiments as shown in FIG. 3, system 300 includes two or more accelerators connected to each other by an inter-chip data fabric 306. Further, as shown in FIG. 3, components of the graphics processing stacked dielet 304 (e.g., one or more processor cores 308, DMA engine 314, DMA buffer 316, etc.) communicate with each other via an interconnect 318 (e.g., via other components). In this way, interconnect 318 forms part of a data fabric that facilitates communication between components of the multiprocessor computing system 300. Further, the inter-chip data fabric 306 extends the data fabric over various communicatively coupled graphics processing stacked dielets 304 and I / O interfaces (not shown) that also form part of the data fabric. In various embodiments, interconnect 318 includes any computer communication medium suitable for communication between the devices shown in FIG. 3, such as a bus, a data fabric, etc. In some embodiments, interconnect 318 is connected to and / or communicates with other components not shown in FIG. 3 for ease of explanation. For example, in some embodiments, interconnect 318 includes connections to one or more input / output (I / O) interfaces 206 as illustrated and described with respect to FIG. 2.

[0027] As can be understood, the inter-chip data fabric 306 and / or interconnect 318 often have high bandwidth such that a single DMA engine cannot saturate the available data fabric bandwidth. In various embodiments, as will be described in more detail later, the system 300 utilizes an increased number of DMA engines 314 (e.g., one for each chiplet 304, as shown in the embodiment of Figure 3) to perform hardware-managed (i.e., mediated by device hardware without input from system software applications, including, for example, the operating system or hypervisor) partitioning of transfer commands to multiple DMA engines 314 for processing memory transfer commands via DMA. In this way, the work specified by the transfer command is essentially partitioned across multiple chiplets 304 and their respective DMA engines 314, thereby increasing the total bandwidth usage without the individual DMA engines 314 needing to be larger or have more capabilities to increase the overall DMA throughput and data fabric bandwidth usage.

[0028] During operation, in response to notifications (e.g., doorbell ringing), the DMA engine 314 reads and executes a DMA transfer command (along with associated parameters) from the DMA buffer 316 to perform data transfer operations and packet transfers. In various embodiments, the operational parameters (e.g., DMA command parameters) are typically the base address, stride, element size, and number of elements to communicate, for both the sender and receiver. Specifically, the DMA engine 314 is configured such that multiple DMA engines 314 across multiple dies (e.g., MCM 302) or chiplets 304 read the same location containing the packet with the DMA transfer parameters. Subsequently, as described in more detail below, the DMA engines 314 operate in cooperation in DMA transfers, synchronizing and coordinating with each other via hardware mechanisms. In various embodiments, the DMA engines 314 perform hardware-managed partitioning of command transfers so that a single DMA engine performs only a portion of the DMA transfer. For example, splitting a DMA transfer between two DMA engines 314 has the potential to double the bandwidth usage per unit time or DMA transfer throughput, as each DMA engine performs half of the transfer simultaneously with the other DMA engine.

[0029] Referring to Figure 4, a block diagram is shown illustrating an example of a system that implements hardware-managed partitioning of transfer commands based on cache status, according to several embodiments. Device 400 or a portion thereof can be implemented using a semiconductor die 202 (as illustrated and described with respect to Figure 2) and / or part or all of device 100 (as illustrated and described with respect to Figures 1 and 2). In various embodiments, device 400 includes a base die 402 using a processing stacked die chiplet according to several embodiments. The base die 402 is formed as a single semiconductor chip package including N=2 communicably coupled graphics processing stacked die chiplets 404. As illustrated, the processor base die 402 includes a first graphics processing stacked die chiplet 404A and a second graphics processing stacked die chiplet 404B.

[0030] In various embodiments, the base die 402 includes an inter-chip data fabric 406 that acts as a high-bandwidth inter-die interconnect between chiplets (e.g., between a first graphics processing stacked die chiplet 404A and a second graphics processing stacked die chiplet 404B). In some embodiments, the base die 402 includes one or more processor cores 408 (e.g., a CPU and / or GPU, or a processor core die) formed on each of the chiplets 404A to 404B. Furthermore, in various embodiments, each of the chiplets 404A to 404B includes one or more levels of cache memory 410 and one or more memory PHYs (not shown) for communicating with an external system memory module 412, such as a dynamic random access memory (DRAM) module. Considered as a whole, the main memory (e.g., memory module 412) and their local caches, which are communicably coupled to the multiple chiplets (e.g., chiplets 404A and 404B), form a shared memory for the device 400. To be understood, each chiplet 404 has only a direct physical connection to a portion of the entire shared memory system.

[0031] In various embodiments, the base die 402 includes two or more DMA engines 414 (e.g., a first DMA engine 414A and a second DMA engine 414B) located on the base die 402. In various embodiments, the DMA engines 414 coordinate DMA transfers of data between devices and memory (or between different locations within memory) in the system 400. In various embodiments, the DMA engines 414 coordinate the movement of data between multiple devices / accelerators, while calculations are performed on other data in, for example, a processor core 408. In various embodiments, one or more DMA engines 414 are part of a DMA controller (not shown) in some embodiments, but the terms DMA engine and DMA controller are used interchangeably herein. The DMA engines 414 operate to transfer data to, for example, one or more memory modules 412, in response to a command, without the involvement of the processor core 408. Similarly, the DMA engines 414 perform intra-chip data transfers in some embodiments.

[0032] The graphics processing stacked die chiplet 304 is described below in the specific context of GPU terminology for the sake of illustrative and explanatory purposes, but it should be noted that in various embodiments, the described architecture is applicable to any of the various types of parallel processors (as more broadly described above with reference to Figures 2 and 3) without departing from the scope of this disclosure. Furthermore, although the DMA engine 414 is shown in Figure 4 as being on top of the individual compute chiplets, those skilled in the art will recognize that hardware-managed partitioning of transfer commands can be performed by the DMA engine in various system locations without departing from the scope of this disclosure. For example, in some embodiments, the DMA engine is located on the base die (as described, for example, with reference to Figure 5). Similarly, although the systems and devices are described here in the context of chiplet-based systems, those skilled in the art will recognize that hardware-managed partitioning of transfer commands is not limited to any particular architecture and can be performed in any system configuration including multiple DMA engines, including a monolithic die.

[0033] In some embodiments, DMA requests are processed by placing DMA commands (also known as interchangeable packets) generated by one or more processors 408 into memory-mapped I / O (MMIO) locations such as a DMA buffer 416 (also known as a DMA queue for interchangeable DMA transfer commands). In various embodiments, the DMA buffer 416 is a hardware structure to which read or write instructions are transferred, so that the DMA engine 414 can read DMA commands (without having to go to DRAM memory, for example). To perform a data transfer operation, in various embodiments, the DMA engine 414 receives instructions (e.g., DMA transfer commands / data transfer requests) generated by processors 408 by accessing a sequence of commands in the DMA buffer 416. The DMA engine 414 then retrieves the DMA commands (also known as descriptors) from the DMA buffer 416 for processing. In some embodiments, the DMA command specifies, for example, the start address for direct virtual memory access (DVMA) and I / O bus access, as well as the transfer length up to a predetermined maximum value.

[0034] Although the DMA buffer 416 is shown in Figure 4 as being implemented on the base die 402 for the sake of clarity, those skilled in the art will recognize that the DMA buffer 416 can be implemented in various components of the systems and devices described herein without departing from the scope of this disclosure. For example, in some embodiments, the DMA buffer 416 is configured in main memory, such as a memory module 412. Its location in the command queue in memory is where the DMA engine 414 goes to read the transfer commands. In various embodiments, the DMA buffer 416 is configured as one or more ring buffers (for example, addressed by modulo addressing).

[0035] As illustrated in Figure 4, DMA engines 414A and 414B receive a DMA notification 418 indicating that one or more DMA workloads or DMA jobs (such as a DMA buffer 416 or a system memory module 412) have been formed and filled into memory. In various embodiments as described above, the DMA notification 418 includes a doorbell ring or other notifier indicating that a DMA descriptor is prepared in memory. In response to receiving the DMA notification 418, DMA engines 414A and 414B independently fetch a DMA job description 420 from the DMA buffer 416. In some embodiments, the communication path for the DMA engine 414 to retrieve DMA transfer commands includes a PCIe interface (e.g., I / O interface 106 as described above with respect to Figure 1) as a path for connecting the processor and the chiplet 404. In other embodiments, the communication path for the DMA engine 414 to retrieve DMA transfer commands includes an internal data fabric (e.g., interconnect 318 as described above with respect to Figure 3) as a path for connecting the on-die processor and the DMA engine 414.

[0036] In addition to independently fetching the same DMA job description from the same location (i.e., DMA buffer 416), each of the DMA engines 414A and 414B independently determines the portion of the data transfer required by the DMA transfer command to be executed. In one embodiment, each of the DMA engines 414A and 414B issues a speculative DMA transfer by broadcasting a cache probe request 422 (e.g., a read probe and / or write probe) to the cache memory 410 and main memory (e.g., memory module 412) of their respective chiplets 404. In response to the cache probe request, the cache memory 410 and main memory of each chiplet 404 return one or more return responses to the requesting DMA engine. For example, in the case of a DMA read probe, the cache memory 410 may return a cache hit or cache miss to indicate whether the requested data was found in the cache memory 410.

[0037] As shown in Figure 4, the retrieved DMA job description 420 is a single transfer command to read data from physical addresses X and Y. To determine whether some of the data associated with addresses X and Y is currently present in the cache memory 410 of the first graphics processing stacked die chiplet 404A (e.g., L3 or some last-level cache), the DMA engine 414A broadcasts a cache probe request 422 to the cache memory 410 (including a cache controller [not shown]) in front of the memory module 412 and receives one or more return responses indicating a cache hit for address X and a cache miss for address Y. To determine whether some of the data associated with addresses X and Y is currently present in the cache memory 410 of the second graphics processing stacked die chiplet 404B, the DMA engine 414B similarly broadcasts a cache probe request 422 to the cache memory 410 and receives one or more return responses indicating a cache miss for address X and a cache miss for address Y.

[0038] In various embodiments, the probe includes messages passed from a coherency point (e.g., in the DMA engine 414) to one or more caches in the computer system to request a response indicating whether the cache has a copy of a data block, and, in some embodiments, to indicate the cache state in which the cache should place the data block. In some embodiments, when the DMA engine 414 receives a memory request targeting its corresponding memory controller (e.g., a memory request for data stored in an address or address region in memory controlled by the memory controller), the DMA engine 414 performs a lookup (e.g., a tag-based lookup) against its corresponding cache directory to determine whether the request targets a memory address or region cached in at least one cache line of any cache subsystem.

[0039] Furthermore, a specific physical address exists in only one location on the memory module 412. For example, in a chiplet-based system, each die is assigned to a memory channel with a specific range of addresses (or a specific address pattern belonging to the local die and the remote die), or is otherwise physically connected. From the translation from virtual addresses to physical addresses, each DMA engine 414 knows whether the address of the DMA transfer command is within the range of memory connected to its physical die. If the DMA engine 414 decodes the DMA transfer command and determines that the data is not in local memory, the DMA engine 414 skips those parts of the DMA job. Furthermore, if all DMA engines 414 operate in this same manner, the entire memory channel is covered.

[0040] In the exemplary embodiment shown in Figure 4, the data for physical addresses X and Y is stored in a memory module 412 directly connected to the second graphics processing stacked die chiplet 404B. However, since the cache memory 410 of the first graphics processing stacked die chiplet 404A already contains the requested data for physical address X, the DMA engine 414A performs the portion of the data transfer associated with address X (as opposed to retrieving data from the cache of a different chiplet). Therefore, having the DMA engine 414A perform the DMA transfer of the data associated with address X is more energy-optimal and has lower latency than accessing the data-owning memory module 412 (or cache) of a different chiplet across the inter-chip data fabric 406.

[0041] Similarly, due to a cache probe request 422 resulting in a cache miss in both the first and second chiplets 404's cache memory 410, the DMA engine 414B of the data-owning cache memory 410 performs a DMA transfer of the data associated with physical address Y. In other words, the DMA engine 414 closer to the data is the engine that performs the DMA transfer. The DMA engine 414 then signals that the DMA transfer is complete, for example, by sending an interrupt signal to the processor core 408. In this way, the hardware-managed coordination of the division of the entire DMA transfer packet by the closer DMA engine 414 and the execution of the DMA transfer (each DMA engine performs its transfer in parallel with the other DMA engines) avoids the need for the DMA transfer operation to read data from memory across all different channels (for example, reading data from cache memory or DRAM associated with different chiplets requires traffic that takes an energy performance hit across the interconnect).

[0042] Once all DMA engines 414 have completed their respective parts of the DMA job, a signal indicating that the DMA job is complete is returned to the requesting processor. For example, in some embodiments, this is achieved by writing a value indicating completion to the DMA buffer, writing a value to some other location in memory, or by an interrupt to the requesting processor. Since the requesting processor is only notified when the entire DMA job is complete, the DMA engines 414 synchronize in some way so that DMA job completion is indicated only when all DMA engines 414 have completed their respective parts of the DMA job. In some embodiments, this is achieved by various techniques such as atomically incrementing a counter in the DMA buffer to indicate how many DMA engines have completed their parts, via a synchronization structure in memory, or via direct messaging between DMA engines through the fabric 406.

[0043] In other embodiments, as described in more detail below, the DMA engine determines packet splitting of a forwarding command based on the amount of data being transferred and the interleaving of workloads between multiple DMA engines (instead of looking at the physical address to be accessed, as shown in Figure 4).

[0044] Referring to Figure 5, a block diagram is shown illustrating another example of a system that implements hardware management partitioning of transfer commands according to several embodiments. Device 500 or part thereof can be implemented using part or all of the semiconductor die 202 (as illustrated and described with respect to Figure 2) and / or device 100 (as illustrated and described with respect to Figures 1 and 2). In various embodiments, device 500 includes a base die 502 using a processing stacked die chiplet according to several embodiments. The base die 502 is formed as a single semiconductor chip package including N=2 communicably coupled graphics processing stacked die chiplets 504. As illustrated, the processor base die 502 includes a first graphics processing stacked die chiplet 504A and a second graphics processing stacked die chiplet 504B.

[0045] In various embodiments, the base die 502 includes an inter-chip data fabric 506 that acts as a high-bandwidth inter-die interconnect between chiplets (e.g., between a first graphics processing stacked die chiplet 504A and a second graphics processing stacked die chiplet 504B). In some embodiments, the base die 502 includes one or more processor cores 508 (e.g., a CPU and / or GPU, or a processor core die) formed on each of the chiplets 504A to 504B. Furthermore, in various embodiments, each of the chiplets 504A to 504B includes one or more levels of cache memory 510 and one or more memory PHYs (not shown) for communicating with an external system memory module 512, such as a dynamic random access memory (DRAM) module. Considered as a whole, the main memory (e.g., memory module 512) and their local caches, which are communicably coupled to the multiple chiplets (e.g., chiplets 504A and 504B), form a shared memory for the device 500. To be understood, each chiplet 504 has only a direct physical connection to a portion of the entire shared memory system.

[0046] In various embodiments, the base die 502 includes two or more DMA engines 514 (e.g., a first DMA engine 514A and a second DMA engine 514B) located on the base die 502. In various embodiments, the DMA engines 514 coordinate DMA transfers of data between devices and memory (or between different locations within memory) in the system 500. In various embodiments, the DMA engines 514 coordinate the movement of data between multiple devices / accelerators, while calculations are performed on other data in, for example, a processor core 508.

[0047] In various embodiments, one or more DMA engines 514 are part of a DMA controller (not shown), but the terms DMA engine and DMA controller are used interchangeably herein. The DMA engine 514 operates to transfer data to, for example, one or more memory modules 512, without the involvement of the processor core 508, in response to a command. Similarly, the DMA engine 514 performs in-chip data transfers in some embodiments. In some embodiments, the DMA engine 514 includes a front-end packet processor (not shown) and a back-end command engine and transfer engine (not shown). The transfer engine is the part of the DMA engine 514 that performs the actual DMA transfer / data movement, and the command engine drives the transfer engine to tell it which operation to perform. In various embodiments, the packet processor includes one or more processing engines such as an Advanced RISC Machine (ARM), a pipeline stage uninterlocked microprocessor (MIPS), a serial data processor (SDP), and / or other RISC cores capable of performing buffer management, table lookup, queue management, fabric processing, and host processing functions known in the art. In various embodiments, the packet processor is a microcontroller that waits for DMA packets to arrive. After receiving and reading the DMA packets, the packet processor turns on the backend of the DMA controller to perform data transfer.

[0048] In various embodiments, the backend command engine is a microcontroller (or other firmware, state machine, etc.) with which other system components communicate to initiate DMA transfers. In some embodiments, the command engine includes a single command engine that drives multiple transfer engines. In other embodiments, the command engine includes a single command engine for each transfer engine. Furthermore, the DMA engine 514 includes several distributed algorithms that allow the command engines to agree on which operation to perform. For example, if a single DMA engine 514 includes two command engines, each command engine may be instructed by hardware that there is a packet waiting in memory. After reading the packet, the command engines synchronize with each other via a command engine bus or a synchronization mechanism such as a special location in memory that both command engines are aware of performing software synchronization such as a barrier or lock. In various embodiments, both command engines read the packet. Then, both command engines read the command and communicate with each other to divide the data transfer. As you can understand, the command engines synchronize with each other in various ways, such as to inform the requesting processor (e.g., the CPU or processor core 508) that the DMA transfer is complete (after both command engines have finished executing their respective parts of the DMA transfer).

[0049] The graphics processing stacked die chiplet 504 is described below in the specific context of GPU terminology for the sake of illustrative and explanatory purposes, but it should be noted that in various embodiments, the described architecture is applicable to any of the various types of parallel processors (as more broadly described above with reference to Figures 2 and 3) without departing from the scope of this disclosure. Furthermore, although the DMA engine 514 is shown located on top of the base die 502 in Figure 5, those skilled in the art will recognize that hardware-managed partitioning of transfer commands can be performed by the DMA engine in various system locations without departing from the scope of this disclosure. For example, in some embodiments, the DMA engine is located on top of individual compute chiplets (as more broadly described above with reference to Figure 4). Similarly, although the systems and devices are described here in the context of chiplet-based systems, those skilled in the art will recognize that hardware-managed partitioning of transfer commands is not limited to any particular architecture and can be performed in any system configuration including multiple DMA engines, including a monolithic die.

[0050] In some embodiments, a DMA request is processed by placing DMA commands (also known as interchangeable packets) generated by one or more processors 508 into a memory-mapped I / O (MMIO) location such as a DMA buffer 516 (also known as a DMA queue for interchangeable DMA transfer commands). In various embodiments, the DMA buffer is a hardware structure to which read or write instructions are transferred so that a DMA engine 514 can read DMA commands (without having to go to DRAM memory, for example). To perform a data transfer operation, in various embodiments, the DMA engine 514 receives an instruction (e.g., a DMA transfer command / data transfer request generated by processor 508). For example, in some embodiments, the DMA engine 514 retrieves a DMA command (also known as a descriptor) from the DMA buffer 516 for processing.

[0051] Although the DMA buffer 516 is shown in Figure 5 as being implemented on the base die 502 for the sake of clarity, those skilled in the art will recognize that the DMA buffer 516 can be implemented in various components of the systems and devices described herein without departing from the scope of this disclosure. For example, in some embodiments, the DMA buffer 516 is configured in main memory, such as a memory module 512. Its location in the command queue within the memory is where the DMA engine 514 goes to read the transfer commands.

[0052] As illustrated in Figure 5, in some embodiments, DMA engines 514A and 514B receive DMA job notifications 518 (a first DMA job notification 518A to DMA engine 514A and a second DMA job notification 518B to DMA engine 514B) from the primary DMA engine 522. The primary DMA engine 522 acts as a central agent mediating between the DMA job submitter (e.g., a server, CPU, etc.) and the secondary / remote DMA engines (e.g., the first DMA engine 514A and the second DMA engine 514B) that perform the actual data transfer.

[0053] In some embodiments as described above with respect to Figure 4, the primary DMA engine 522 transmits a first DMA job notification 518A and a second DMA job notification 518B, which are the same signal and indicate that one or more DMA workloads or DMA jobs have been formed and filled into memory (such as the DMA buffer 516 or system memory module 512). Upon receiving the DMA notification 518, the DMA engines 514A and 514B independently fetch the DMA job description 520 from the DMA buffer 516, and divide the DMA job workload in a distributed decision scheme in which two or more separate entities (e.g., two DMA engines 514A and 514B) each read the same packet and independently decide how to perform the DMA transfer. Alternatively, in other embodiments, the primary DMA engine 522 directly transmits the DMA job descriptor to the DMA engines 514A and 514B, skipping the step of instructing them to retrieve the DMA job description from the buffer 516.

[0054] In addition to independently fetching the same DMA job description from the same location (i.e., DMA buffer 516), each of the DMA engines 514A and 514B independently determines the portion of the data transfer required by the DMA transfer command to be executed. For example, as described in more detail in relation to Figure 4, the DMA engine 514 performs hardware-managed coordination based on the division of the entire DMA transfer packet and the execution of the DMA transfer by the closer DMA engine 514 (for example, closer via the same die cache or local DRAM without traversing the inter-chiplet interconnect) (each DMA engine performs its transfer in parallel with the other DMA engines).

[0055] In some embodiments, the DMA engine 514 performs hardware-managed coordination through decisions to divide the packets of a transfer command based on the amount of data being transferred and to interleave the workload among multiple DMA engines. For example, as illustrated in Figure 5, a DMA job description 520 is a single transfer command instructing the transfer of 1000 megabytes of data from physical address X to physical address Y. In one embodiment, the DMA engines 514 divide the total DMA transfer size equally among them. For example, device 500 includes two DMA engines, a first DMA engine 514A having global ID=0 and a second DMA engine 514B having global ID=1. Thus, the hardware of device 500 includes information that identifies the total number and relative order of DMA engines in the hardware configuration. In this example, after reading the same DMA job description 520, the first DMA engine 514A (e.g., global ID=0) performs the first half of the DMA transfer by transferring the first 500 megabytes of data. Similarly, a second DMA engine 415B (e.g., global ID=1) performs the second half of the DMA transfer in parallel by transferring a second 500 megabytes of data.

[0056] Those skilled in the art will recognize that this partitioning of the transfer command workload is not limited to the specific embodiments described herein, and that any hardware-managed coordination of DMA transfers and interleaving of DMA workloads between multiple DMA engines can be performed without departing from the scope of this disclosure. As a non-limiting example, in some embodiments, embodiments of hardware-managed DMA workload partitioning include odd / even byte count partitioning (e.g., on a single-byte basis), memory channel addressing, memory page size, memory page location, alternating data block criteria that are not fully interleaved, or any other type of interleaving. As will be understood, in various embodiments, the size of a particular interleave depends on the specific interconnect hardware, memory transfer size, and cache in the system. Optimization of interleave type and any data partitioning type is variable depending on the type of system microarchitecture for a particular implementation. For example, in one embodiment, the interleave partition size is based on the page conversion size so that the transfer size matches the page conversion size in order to reduce the amount of page conversions that need to be performed. In another embodiment, each DMA engine 514 is allocated a specific amount of data (e.g., 1000 megabytes) to be transferred instead of an address range. For example, in such an embodiment, each DMA engine 514 transfers 1,000 megabytes of data in parallel with other DMA engines, thereby the DMA job description 520 is a single transfer command that instructs the transfer of 1,000 megabytes of data from physical address X to physical address Y, and simply turns on the first DMA engine 514A while the second DMA engine 514B remains idle.

[0057] In another embodiment, the first DMA job notification 518A to DMA engine 514A and the second DMA job notification 518B to DMA engine 514B do not indicate that the DMA engines should fetch the DMA job description 520 from the DMA buffer 516. Rather, the primary DMA engine 522 divides a single DMA job (e.g., a DMA job description 520 instructing the transfer of 1000 megabytes of data from physical address X to physical address Y) into several smaller jobs, sending different workloads individually to each separate DMA engine 514. Thus, each secondary / remote DMA engine sees a portion of the original DMA job description 520.

[0058] In this example, the primary DMA engine 522 divides the DMA job description 520 into two smaller workloads by sending a first DMA job notification 518A instructing the first DMA engine 514A (as a secondary / remote DMA engine) to perform the first half of the DMA transfer by transferring a first 500 megabytes of data. Similarly, the primary DMA engine 522 sends a second DMA job notification 518B instructing the second DMA engine 514B (as a secondary / remote DMA engine) to perform the second half of the DMA transfer by transferring a second 500 megabytes of data. After each individual secondary / remote DMA engine has completed its respective part of the DMA job, the secondary DMA engine 514 notifies the primary DMA engine 522 of their completion. The primary DMA engine 522 then communicates that the entire job is complete.

[0059] In this example, each individual DMA engine 514 executes the entire job that is sent to it (for example, DMA engine 514 acts as a pure dummy by executing the workload assigned to it without any discretion, in contrast to the embodiments described above in which each DMA engine individually determines which part of the workload is locally applicable to it). In various embodiments, the primary DMA engine 522 determines how many DMA engines are available in order to allocate available work and / or available bandwidth or other processing resources to each remote DMA engine.

[0060] Figure 6 is a block diagram of Method 600 for performing hardware-managed partitioning of DMA transfer commands, according to several embodiments. For ease of illustration and explanation, Method 600 is described below in the illustrative context with reference to the systems and devices in Figures 1 to 5. However, Method 600 is not limited to these illustrative contexts, and instead, in different embodiments, it may be employed for any of the various possible system configurations using the guidelines provided herein.

[0061] Method 600 begins in block 602 by having a first DMA engine access a DMA transfer command and determine a first portion of the data transfer requested by the DMA transfer command. For example, as shown in Figure 4 with respect to DMA engine 414, DMA engines 414A and 414B receive a DMA notification 418 indicating that one or more DMA workloads or DMA jobs (such as DMA buffer 416 or system memory module 412) have been formed and filled into memory. In the various embodiments described above, the DMA notification 418 includes a doorbell ring or other notifier indicating that a DMA descriptor is prepared in memory. In response to receiving the DMA notification 418, DMA engines 414A and 414B independently fetch a DMA job description 420 from DMA buffer 416, respectively.

[0062] In some embodiments, each of the DMA engines 414A and 414B independently determines a portion of the data transfer required by the DMA transfer command to be executed. In one embodiment, each of the DMA engines 414A and 414B issues a speculative DMA transfer by broadcasting a cache probe request 422 (e.g., a read probe and / or write probe) to the cache memory 410 and main memory (e.g., a memory module 412) of their respective chiplets 404. In response to the cache probe request, the cache memory 410 and main memory of each chiplet 404 return one or more return responses to the requesting DMA engine. For example, in the case of a DMA read probe, the cache memory 410 may return a cache hit or cache miss to indicate whether the requested data was found in the cache memory 410.

[0063] In other embodiments with respect to DMA engine 514, as illustrated in Figure 5, DMA engines 514A and 514B receive DMA job notifications 518 (a first DMA job notification 518A to DMA engine 514A and a second DMA job notification 518B to DMA engine 514B) from the primary DMA engine 522. The primary DMA engine 522 divides a single DMA job (for example, a DMA job description 520 instructing the transfer of 1000 megabytes of data from physical address X to physical address Y) into several smaller jobs, sending different workloads individually to each separate DMA engine 514. Thus, each secondary / remote DMA engine sees a portion of the original DMA job description 520.

[0064] Method 600, following block 604, initiates the transfer of a first portion of the data transfer by the first DMA engine, at least in part, based on the DMA transfer command. For example, as shown in Figure 4 with respect to the first DMA engine 414A, in order to determine whether a portion of the data associated with addresses X and Y is currently present in the cache memory 410 (e.g., L3 or some last-level cache) of the first graphics processing stacked die chiplet 404A, the DMA engine 414A broadcasts a cache probe request 422 to the cache memory 410 (including a cache controller [not shown]) in front of the memory module 412 and receives one or more return responses indicating a cache hit for address X and a cache miss for address Y. In the exemplary embodiment of Figure 4, the data for physical addresses X and Y is stored in a memory module 412 directly connected to the second graphics processing stacked die chiplet 404B. Since the cache memory 410 of the first graphics processing stacked die chiplet 404A already contains the requested data for physical address X, the DMA engine 414A performs the portion of the data transfer associated with address X (as opposed to retrieving data from the cache of a different chiplet).

[0065] In some embodiments with respect to the DMA engine 514, as illustrated in Figure 5, the DMA engine 514 performs hardware-managed coordination through decisions to divide the packets of a transfer command based on the amount of data being transferred and to interleave the workload among multiple DMA engines. For example, as illustrated in Figure 5, a DMA job description 520 is a single transfer command instructing the transfer of 1000 megabytes of data from physical address X to physical address Y. In one embodiment, the DMA engines 514 divide the total DMA transfer size equally among them. For example, device 500 includes two DMA engines, a first DMA engine 514A having global ID=0 and a second DMA engine 514B having global ID=1. Thus, the hardware of device 500 includes information that identifies the total number and relative order of DMA engines in the hardware configuration. In this example, after reading the same DMA job description 520, the first DMA engine 514A (e.g., global ID=0) performs the first half of the DMA transfer by transferring the first 500 megabytes of data. Similarly, a second DMA engine 415B (e.g., global ID=1) performs the second half of the DMA transfer in parallel by transferring a second 500 megabytes of data.

[0066] Method 600, following block 606, initiates the transfer of the second portion of the data transfer by the second DMA engine, at least partially based on the DMA transfer command. For example, as shown in Figure 4 with respect to the first DMA engine 414A, in order to determine whether a portion of the data associated with addresses X and Y is currently present in the cache memory 410 of the second graphics processing stacked die chiplet 404B, the DMA engine 414B also broadcasts a cache probe request 422 to the cache memory 410 and receives one or more return responses indicating a cache miss for address X and a cache miss for address Y. As the cache probe request 422 results in a cache miss in the cache memories 410 of both the first and second chiplets 404, the DMA engine 414B of the data-owning cache memory 410 performs a DMA transfer of the data associated with physical address Y. In other words, the DMA engine 414 that is closer to the data is the engine that performs the DMA transfer.

[0067] In block 608, after the first and second parts of the data transfer have been transferred, an indicator (indicator) is generated to signal the completion of the data transfer requested by the DMA transfer command. For example, as illustrated in Figure 4, the DMA engine 414 signals the completion of the DMA transfer by sending an interrupt signal to the processor core 408 after some form of synchronization between the DMA engines 414. Similarly, with respect to Figure 5, the secondary DMA engine 514 notifies the primary DMA engine 522 of the completion of each part of the DMA job after the individual secondary / remote DMA engines have completed their respective parts. The primary DMA engine 522 then communicates that the entire job is complete.

[0068] Therefore, as described herein, hardware-managed coordination of the division of entire DMA transfer packets by closer DMA engines and the execution of DMA transfers (each DMA engine performing its transfer in parallel with other DMA engines) avoids the need for DMA transfer operations to read data from memory across all different channels (for example, reading data from cache memory or DRAM associated with different chiplets requires traffic that traverses the interconnect and incurs energy performance hits). For example, those skilled in the art will recognize that round-trip DMA traffic competes with other traffic and consumes power. Thus, hardware-managed division of DMA commands results in reduced power consumption and increased bandwidth for other traffic use.

[0069] As disclosed herein, in some embodiments, the method includes initiating the transfer of a first portion of a data transfer by a first DMA engine, at least in part on a DMA transfer command, and initiating the transfer of a second portion of a data transfer by a second DMA engine, at least in part on a DMA transfer command. In one embodiment, the method includes the first DMA engine receiving a DMA notification indicating that a DMA transfer command is stored in a DMA buffer in system memory, and the first DMA engine fetching the DMA transfer command from the DMA buffer. In another embodiment, initiating the transfer of a first portion of a data transfer by the first DMA engine includes the first DMA engine transferring the first portion of the data transfer based on sending a cache probe request to cache memory and receiving a return response indicating a cache hit in cache memory. In yet another embodiment, initiating the transfer of a second portion of a data transfer by the second DMA engine includes the second DMA engine transferring the second portion of the data transfer from owner main memory based on sending a cache probe request to cache memory and receiving a return response indicating a cache miss in cache memory.

[0070] In one embodiment, determining a first portion of a data transfer further includes interleaving the total DMA transfer size between a first DMA engine and a second DMA engine. In another embodiment, the method includes, in the primary DMA engine, receiving a DMA transfer command and dividing the DMA transfer command into several smaller workloads. In yet another embodiment, the method includes receiving one of the several smaller workloads from the primary DMA engine.

[0071] In some embodiments, the processor device includes a base integrated circuit (IC) die comprising a base IC die including a plurality of processing stacked die chiplets 3D stacked on the base IC die and an inter-chip data fabric that connects the processing stacked die chiplets to each other in a communicative manner; and a plurality of DMA engines 3D stacked on the base IC die, each of which is configured to perform a portion of a data transfer requested by a DMA transfer command. In one embodiment, each of the plurality of DMA engines includes a single command engine that drives the plurality of transfer engines. In another embodiment, each of the plurality of DMA engines is configured to receive a DMA notification indicating that a DMA transfer command is stored in a DMA buffer in system memory. In yet another embodiment, a first DMA engine of the plurality of DMA engines is configured to transfer a first portion of a data transfer based on sending a cache probe request to a cache memory communicatively coupled to a first processing stacked die chiplet and receiving a return response indicating a cache hit in the cache memory. In yet another embodiment, a second DMA engine among a plurality of DMA engines is configured to send a cache probe request to a cache memory communicatively coupled to a second processing stacked die chiplet and to transfer a second portion of the data transfer from owner main memory based on receiving a return response indicating a cache miss in the cache memory.

[0072] In one embodiment, each of the multiple DMA engines is configured to independently determine portions of data transfer by interleaving the total DMA transfer size among the multiple DMA engines. In another embodiment, the processor device includes a primary DMA engine configured to receive DMA transfer commands and divide the DMA transfer commands into multiple smaller workloads. In yet another embodiment, the primary DMA engine is further configured to send different workloads from the multiple smaller workloads to each of the multiple DMA engines.

[0073] In some embodiments, the system comprises a host processor commutably coupled to a parallel processor multichip module, the parallel processor multichip module comprising: a base integrated circuit (IC) die comprising a plurality of processing stacked die chiplets 3D stacked on the base IC die and comprising an interchip data fabric that commutably couples the processing stacked die chiplets to one another; and a plurality of DMA engines 3D stacked on the base IC die, each of the plurality of DMA engines configured to perform a portion of a data transfer requested by a DMA transfer command. In one embodiment, the system comprises a primary DMA engine configured to receive a DMA transfer command and divide the DMA transfer command into a plurality of smaller workloads, and further configured to transmit different workloads from the plurality of smaller workloads to each of the plurality of DMA engines. In another embodiment, each of the plurality of DMA engines is configured to independently determine portions of data transfer by interleaving the total DMA transfer size among the plurality of DMA engines. In yet another embodiment, a first DMA engine among a plurality of DMA engines is configured to transfer a first portion of data transfer based on sending a cache probe request to a cache memory communicably coupled to a first processing stacked die chiplet and receiving a return response indicating a cache hit in the cache memory. In yet another embodiment, a second DMA engine among a plurality of DMA engines is configured to transfer a second portion of data transfer from owner main memory based on sending a cache probe request to a cache memory communicably coupled to a second processing stacked die chiplet and receiving a return response indicating a cache miss in the cache memory.

[0074] Computer-readable storage media include any non-temporary storage media or combination of non-temporary storage media that are accessible by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray® discs), magnetic media (e.g., floppy disks, magnetic tapes, magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical system (MEMS) based storage media. Computer-readable storage media (e.g., system RAM or ROM) may be built into the computing system, computer-readable storage media (e.g., magnetic hard drives) may be permanently mounted to the computing system, computer-readable storage media (e.g., optical disks or Universal Serial Bus (USB) based flash memory) may be detachably mounted to the computing system, and computer-readable storage media (e.g., network-accessible storage (NAS)) may be connected to the computer system via a wired or wireless network.

[0075] In some embodiments, certain aspects of the technology described above are implemented by one or more processors of a processing system that executes the software. The software includes one or more sets of executable instructions, which are stored in a non-temporary computer-readable storage medium or otherwise clearly embodied. The software may also include instructions and specific data, which, when executed by one or more processors, operate the one or more processors to execute one or more aspects of the technology described above. Non-temporary computer-readable storage mediums may include, for example, magnetic or optical disk storage devices, solid-state storage devices such as flash memory, caches, random-access memory (RAM), or other non-volatile memory devices (one or more). Executable instructions stored in a non-temporary computer-readable storage medium can be implemented as source code, assembly language code, object code, or other instruction forms that can be interpreted or otherwise executed by one or more processors.

[0076] In addition to the foregoing, it should be noted that not all activities or elements described in the summary are required, and certain activities or parts of devices may not be required, and one or more additional activities may be performed, and one or more additional elements may be included. Furthermore, the order in which the activities are listed does not necessarily indicate the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, those skilled in the art will understand that various modifications and variations can be made without departing from the scope of the invention as described in the claims. Therefore, the specification and drawings should be considered illustrative rather than restrictive, and all of these variations are intended to fall within the scope of the invention.

[0077] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, benefits, advantages, solutions to problems, and features that may give rise to or manifest any benefits, advantages, or solutions are not to be construed as essential, necessary, or indispensable features to any or all of the claims. Furthermore, the disclosed invention can be modified and implemented in different but similar ways, in ways that are obvious to those skilled in the art who are interested in the teachings of this specification; therefore, the specific embodiments described above are merely illustrative. There are no limitations to the details of the configuration or design shown herein beyond those described in the appended claims. Accordingly, the specific embodiments described above may be modified or altered, and it is clear that all such modifications are within the scope of the disclosed invention. Accordingly, the protection sought herein is described in the appended claims.

Claims

1. A processor device, A base integrated circuit (IC) die comprising a plurality of processing stacked die chiplets 3D stacked on the base IC die, and an inter-chip data fabric that connects the plurality of processing stacked die chiplets in a communicative manner, The system comprises a plurality of DMA engines stacked in 3D on the base IC die, each configured to perform a portion of the data transfer requested by a DMA transfer command. Processor device.

2. Each of the plurality of DMA engines includes a single command engine that drives the plurality of transfer engines, A processor device according to claim 1.

3. Each of the plurality of DMA engines is configured to receive a DMA notification indicating that the DMA transfer command is stored in a DMA buffer in system memory. A processor device according to claim 1.

4. The first DMA engine among the plurality of DMA engines is configured to transmit a cache probe request to a cache memory communicably coupled to a first processing stacked die chiplet and to transmit the first portion of the data transfer based on receiving a return response indicating a cache hit in the cache memory. A processor device according to claim 1.

5. The second DMA engine among the plurality of DMA engines is configured to transmit the cache probe request to a cache memory communicably coupled to the second processing stacked die chiplet and to transfer the second portion of the data transfer from the owner main memory based on receiving a return response indicating a cache miss in the cache memory. The processor device according to claim 4.

6. Each of the plurality of DMA engines is configured to independently determine the portion of the data transfer by interleaving the total DMA transfer size among the plurality of DMA engines. A processor device according to claim 1.

7. A primary DMA engine is provided which receives the DMA transfer command and is configured to divide the DMA transfer command into a plurality of smaller workloads. A processor device according to claim 1.

8. The primary DMA engine is configured to transmit different workloads from among the multiple smaller workloads to each of the multiple DMA engines. The processor device according to claim 7.

9. It is a system, A host processor is connected to a parallel processor multichip module in a communicative manner. The aforementioned parallel processor multichip module is A base integrated circuit (IC) die comprising a plurality of processing stacked die chiplets 3D stacked on the base IC die, and an inter-chip data fabric that connects the plurality of processing stacked die chiplets in a communicative manner, The system comprises a plurality of DMA engines stacked in 3D on the base IC die, each configured to perform a portion of the data transfer requested by a DMA transfer command. system.

10. A primary DMA engine configured to receive the DMA transfer command and divide the DMA transfer command into a plurality of smaller workloads, comprising a primary DMA engine configured to transmit different workloads from the plurality of smaller workloads to each of the plurality of DMA engines. The system according to claim 9.

11. Each of the plurality of DMA engines is configured to independently determine the portion of the data transfer by interleaving the total DMA transfer size among the plurality of DMA engines. The system according to claim 9.

12. The first DMA engine among the plurality of DMA engines is configured to transmit a cache probe request to a cache memory communicably coupled to a first processing stacked die chiplet and to transmit the first portion of the data transfer based on receiving a return response indicating a cache hit in the cache memory. The system according to claim 9.

13. The second DMA engine among the plurality of DMA engines is configured to transmit the cache probe request to a cache memory communicably coupled to the second processing stacked die chiplet and to transfer the second portion of the data transfer from the owner main memory based on receiving a return response indicating a cache miss in the cache memory. The system according to claim 12.

Images