Systems, methods, and apparatuses of accelerators with virtualization and tiered memory

Abstract

An apparatus can include an interconnect interface; a memory system including one or more first type memory devices to receive first data, one or more second type memory devices to receive second data; and an accelerator configured to perform an operation using the first data and the second data. The memory system can further include a cache configured to cache the second data of the one or more second type memory devices. An apparatus can include an interconnect interface; a memory system coupled to the interconnect interface to receive data; an accelerator coupled to the memory system; and virtualization logic configured to partition one or more resources of the accelerator into one or more virtual accelerators, wherein a first virtual accelerator of the one or more virtual accelerators can be configured to perform a first operation on a first portion of the data.

Description

Technical Field

[0001] This disclosure relates generally to accelerator devices, and more specifically to systems, methods, and apparatuses for accelerators having virtualized and tiered memory. Background Technology

[0002] Data processing systems can use one or more accelerator devices to increase system throughput. Some data processing workloads, such as machine learning workloads, may involve using models that can utilize large amounts of memory. The model can be partitioned and spread across multiple accelerator devices. A portion of the model can be stored at each accelerator device that can perform operations on the corresponding portion of the model.

[0003] The information disclosed in the background section is only used to enhance the understanding of the background of the present invention, and therefore may contain information that does not constitute prior art. Summary of the Invention

[0004] An apparatus may include: an interconnect interface; a memory system including one or more first-type memory devices coupled to the interconnect interface to receive first data, and one or more second-type memory devices coupled to the interconnect interface to receive second data; and an accelerator coupled to the one or more first-type memory devices and the one or more second-type memory devices, and configured to perform operations using the first data and the second data. The memory system may also include a cache configured to cache the second data from the one or more second-type memory devices. The one or more first-type memory devices may include one or more stacked memory devices, the one or more second-type memory devices may include one or more non-volatile memory devices, and the cache may include one or more volatile memory devices. The apparatus may also include a coherency engine configured to maintain consistency of at least one of the one or more first-type memory devices. The apparatus may also include a memory switch configured to configure one or more connections between the one or more first-type memory devices and the accelerator. The memory switch may be configured to configure one or more connections between the one or more second-type memory devices and the accelerator. The apparatus may also include a memory manager configured to exchange the first data between the one or more first-type memory devices and the accelerator. The device may also include a preprocessor coupled to a memory system and configured to preprocess the first data. The device may be a first device, which may further include a reduce engine configured to exchange the first data with a second device via an interconnect interface.

[0005] An apparatus may include: an interconnect interface; a memory system coupled to the interconnect interface to receive data; an accelerator coupled to the memory system; and virtualization logic configured to partition one or more resources of the accelerator into one or more virtual accelerators, wherein a first virtual accelerator among the one or more virtual accelerators may be configured to perform a first operation on a first portion of the data. A second virtual accelerator among the one or more virtual accelerators may be configured to perform a second operation on a second portion of the data. The virtualization logic may include one or more registers to configure the one or more virtual accelerators based on configuration information received through the interconnect interface. The virtualization logic may include flow control logic configured to control the data flow between the one or more virtual accelerators and a host through the interconnect interface. The first virtual accelerator among the one or more virtual accelerators may be configured to receive one or more operation parameters through the interconnect interface and perform the first operation based on the one or more operation parameters. The one or more operation parameters may include one or more of memory usage, dependency, or timing information. The apparatus may also include a memory manager configured to schedule one or more portions of the memory system for one or more portions of the first operation. The memory manager may be configured to schedule one or more portions of the memory system based on memory access patterns. The memory manager can be configured to determine a first portion of the memory system based on bandwidth sensitivity and to direct data from the first portion of the memory system to a second portion. The device may also include a scheduler configured to multiplex one or more command streams to one or more virtual accelerators.

[0006] A method may include partitioning a model into a first part and a second part, storing the first part of the model in the memory of a device, storing the second part of the model in the memory of the device, having a first virtual accelerator at the device perform a first operation using the first part of the model, and having a second virtual accelerator at the device perform a second operation using the second part of the model. The model may include one or more of a graph, a machine learning model, or a neural network. Partitioning the model may include partitioning the model at least partially by a host. Partitioning the model may include partitioning the model based on one or more parameters of the first part of the model and one or more parameters of the first virtual accelerator. The one or more parameters of the first part of the model may include one or more of memory usage or computation usage. The one or more parameters of the first virtual accelerator may include one or more of memory resources, computation resources, or interconnect resources. Partitioning the model may include associating the first part of the model with a first process address space and associating the second part of the model with a second process address space. The method may also include generating a first data set for the first virtual accelerator based on the first part of the model. The memory may include a first type of memory device and a second type of memory device, and the method may further include scheduling at least a portion of the first data set for the first type of memory device and the second type of memory device. The first type of memory device may include the working memory of the first virtual accelerator. The method may further include a model-based second part generating a second data set for the second virtual accelerator. The memory may include a first memory device of a first type associated with the first virtual accelerator, a first memory device of a second type associated with the first virtual accelerator, and a second memory device of a first type associated with the second virtual accelerator. The method may further include scheduling at least a portion of the first data set for the first type of first memory device and the second type of first memory device, and scheduling at least a portion of the second data set for the first type of first memory device and the second type of first memory device. Attached Figure Description

[0007] The accompanying drawings are not necessarily drawn to scale, and for illustrative purposes, elements with similar structures or functions are generally indicated by similar reference numerals or portions thereof throughout the drawings. The drawings are intended only to facilitate the description of the various embodiments described herein. The drawings do not depict every aspect of the teachings disclosed herein and do not limit the scope of the claims. To prevent obscurity, not all components, connections, etc., may be shown, and not all components may have reference numerals. However, the pattern of component configuration can be readily discerned from the drawings. The drawings, together with the specification, illustrate exemplary embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0008] Figure 1An embodiment of a framework for analyzing parallel processing operations according to an example embodiment of the present disclosure is shown.

[0009] Figure 2 An embodiment of a device having a layered memory is shown according to an example embodiment of the present disclosure.

[0010] Figure 3 An embodiment of a device with accelerator virtualization according to an example embodiment of the present disclosure is shown.

[0011] Figure 4A An example embodiment of a host according to an example embodiment of this disclosure is shown.

[0012] Figure 4B An example embodiment of a device having a hierarchical memory and a virtual accelerator is shown according to an example embodiment of the present disclosure.

[0013] Figure 5 An embodiment of a system having dis-aggregation according to an example embodiment of the present disclosure is shown.

[0014] Figure 6 The diagram illustrates an example embodiment of the present disclosure and an embodiment of a method for global partitioning of the diagram.

[0015] Figure 7 A partial scheduling operation according to an example embodiment of this disclosure is illustrated.

[0016] Figure 8 An embodiment of a device virtualization method according to an example embodiment of the present disclosure is shown.

[0017] Figure 9 An embodiment of a workflow for a device virtualization method according to an example embodiment of the present disclosure is shown.

[0018] Figure 10A and Figure 10B (Together forming Figure 10) illustrates an embodiment of a memory scheduling method according to an example embodiment of the present disclosure.

[0019] Figure 11 An embodiment of the allreduce operation method according to an example embodiment of the present disclosure is shown.

[0020] Figure 12 An embodiment of a method for processing data using a model according to an example embodiment of the present disclosure is shown. Detailed Implementation

[0021] Some models for applications such as graph processing and machine learning (ML) may be too large to fit into the memory of accelerator devices. Therefore, models can be partitioned and distributed across multiple accelerator devices. However, this can increase data transfer between the host and accelerator devices, as accelerator devices may swap different parts of the model into their memory as they process the input using different parts of the model.

[0022] Some accelerator devices according to exemplary embodiments of this disclosure may include a multi-tier memory system. For example, the first tier may include a high-bandwidth memory that can be used as the accelerator's working memory. The second tier may include a high-capacity memory that can store some or all of the model data. Depending on the implementation details, this can reduce the time, cost, energy, etc., required to move portions of a large model (and / or the model's input data) into the accelerator device.

[0023] In some embodiments, the second layer may include a cache to improve access speed to more frequently used portions of data stored in high-capacity memory (e.g., more frequently used portions of a model). Depending on the implementation details, this can reduce latency in the second layer of memory, which in turn can improve the performance of accelerator devices in applications such as latency-sensitive ML inference.

[0024] Some accelerator devices according to exemplary embodiments of this disclosure may implement one or more virtual accelerators. For example, one or more resources of a physical accelerator (e.g., processor cores, working memory, cache, etc.) may be divided into multiple virtual accelerators, each of which may appear as a separate accelerator for a host or other device. According to the implementation details, this can enable the accelerator device to implement model, data, and / or workload parallelism.

[0025] Some embodiments may include a memory switch and / or a memory manager. For example, a memory switch may be configured to physical connections to and / or between various types of memory that can be used for a first memory tier, a second memory tier, a cache, etc., while a memory manager may manage data movement between memory devices and / or between an accelerator device and a host and / or other devices. In some embodiments, the memory manager may control data movement based on one or more learned memory access patterns.

[0026] In some embodiments, various quantities of control can be provided to users and / or applications, for example, through one or more registers and / or application programming interfaces (APIs) that can determine the allocation of resources between virtual accelerators, the allocation of memory resources in a multi-layer memory system, etc.

[0027] In some embodiments, according to exemplary embodiments of this disclosure, workflows and / or models, such as graphs, machine learning models, neural networks, etc., can be partitioned among multiple accelerator devices and / or virtual accelerators. For example, a host can partition a model among virtual accelerators based on the memory requirements and / or computation counts of a portion of the model, and the memory resources and / or cores of the virtual accelerators. In some embodiments, based on the partitioning, the host can generate a clustered graph with groups of data to be executed by the virtual accelerators and scheduled by a memory manager.

[0028] The principles disclosed herein have independent applicability and can be embodied independently, and not every embodiment can utilize every principle. However, the principles can also be embodied in various combinations, some of which can synergistically amplify the benefits of the individual principles. For example, some accelerator devices can implement a multi-level memory system without a virtual accelerator, while other accelerators can implement a virtual accelerator without hierarchical memory. However, some embodiments can implement both a multi-level memory system and a virtual accelerator in the same device, which, depending on the details of the implementation, can amplify the individual benefits of both features.

[0029] Figure 1 An embodiment of a framework for analyzing parallel processing operations according to an example embodiment of the present disclosure is shown. Figure 1 The framework shown can be used, for example, for analyzing graph processing and / or deep learning (DL) applications (e.g., deep neural networks (DNNs)), where the computation and / or portions of the DL model can be distributed across multiple machines (accelerator devices (e.g., multiple neural processing units (NPUs))). Figure 1 In the embodiment shown, model 103 can be segmented across multiple machines Machine 1 (machine 1) to Machine 4 (machine 4), as described below.

[0030] Using one or more accelerator devices with large models can be challenging, for example, because only a small portion of the model and / or the model's input data can fit into the accelerator device's working memory. Therefore, performing computations using an accelerator device can involve the following operations: (1) a portion of the trained model can be moved to the accelerator's working memory; (2) one or more inputs (e.g., one or more vectors) used for computation can be moved to the accelerator device's working memory; (3) the accelerator device can use these one or more inputs and the portion of the trained model to perform computations; and (4) the accelerator device can store one or more computation results in its working memory and / or transfer one or more results to a host or other device. Furthermore, operations (1) through (4) can be repeated to perform computations on different portions of the model.

[0031] refer to Figure 1 According to exemplary embodiments of this disclosure, various types of parallelism can be used to reduce processing time. For example, data parallelism can involve splitting workload data across multiple machines (e.g., accelerator devices). In some embodiments, data parallelism can reduce processing time (e.g., training and / or inference time). However, depending on the implementation details, data parallelism may be computationally constrained (e.g., limited by the amount of computing resources), and therefore, processing time may increase if computational constraints are met.

[0032] Model parallelism can involve partitioning a model across multiple machines (e.g., accelerator devices), each capable of processing a specific portion of the model's data. In some embodiments, model parallelism can reduce processing time; however, depending on the implementation details, model parallelism may be memory-constrained (e.g., limited by the amount of memory resources). For example, large deep learning models may cause systems with model parallelism to reach memory limitations.

[0033] Workload partitioning can involve dividing a workload (e.g., data and models) across multiple machines (e.g., accelerator devices). In some embodiments, workload partitioning can reduce processing time. For example, with a larger batch size, processor core utilization and / or accuracy can be improved, and / or the number of training iterations can be reduced. However, depending on the implementation details, workload partitioning may be memory-constrained and / or computationally-constrained. For example, large deep learning models may cause systems with model parallelism to reach memory limits, in which case computational resources (e.g., processor cores) may be underutilized.

[0034] In any of the parallel scenarios described above, the memory factor affecting system performance can be based on factors such as the width and / or depth of the machine learning model, the batch size of the data, and / or the quality of the input. Moving data can be costly in terms of time and energy. For workloads with high computational demands, adding additional machines may be efficient, but for memory-constrained workloads, the machines may be underutilized (and therefore costly). Furthermore, in some embodiments, implementing parallelism in ultra-large-scale environments can emphasize inference workloads and / or low-latency workloads. For example, some training workloads can be performed offline, where relatively high latency may be acceptable. However, online (e.g., real-time) ML workloads may be more sensitive to latency, regardless of batch size. For example, in the case of small-batch-size inference workloads (such as mobile image recognition), users may expect fast results. Similarly, in the case of large-batch-size inference workloads (such as language processing, image search, and / or e-commerce and / or social media recommendations), users may also expect fast results.

[0035] Figure 2 An embodiment of a device having a layered memory is shown according to an example embodiment of the present disclosure. Figure 2 The device 200 shown may include an interconnect interface 202, a memory system 204, and an accelerator 206. The memory system 204 may include a first memory layer 208 and a second memory layer 210. The first memory layer 208 may be implemented using at least a first type of memory, and the second memory layer 210 may be implemented using at least a second type of memory. The memory system 204 and the accelerator 206 can communicate with the interconnect interface 202 via an interconnect 212. The accelerator 206 can access the first type of memory 208 via a first memory interface 214. The accelerator 206 can access the second type of memory 208 via a second memory interface 216.

[0036] In some embodiments, the first type of memory in the first memory layer 208 may be implemented using one or more types of memory that can provide relatively high bandwidth. Therefore, the first memory layer 208 may be referred to as bandwidth-enhanced memory. For example, the first type of memory may be implemented using high-bandwidth memory (HBM) that may include one or more memory device stacks, one or more interposers, and one or more point-to-point interconnects. As another example, the first type of memory may be implemented using dynamic random access memory (DRAM) such as double data rate (DDRX) DRAM of any generation (where X may indicate a generation (e.g., DDR2, DDR3, DDR4, etc.), low-power double data rate (LPDDRX) DRAM, etc. Other examples may include synchronous DRAM (SDRAM), static random access memory (SRAM), etc. In some embodiments, the first memory layer 208 may include a combination of different memory types. In some embodiments, the first memory layer 208 may be optimized for bandwidth within the memory system 204.

[0037] In some embodiments, the second type of memory in the second memory layer 210 may be implemented using one or more types of memory that can provide relatively high capacity. Therefore, the second memory layer 210 may be referred to as capacity-enhanced memory. For example, the second type of memory may be implemented using non-volatile memory that may include flash memory (such as NAND flash memory, low-latency NAND flash memory), persistent memory (PMEM) (such as cross-grid non-volatile memory, memory with volume resistance variation, phase-change memory (PCM), etc.) or any combination thereof. In some embodiments, the second memory layer may be optimized for capacity within the memory system 204.

[0038] In some embodiments, memory system 204 may include cache 218 configured to cache data in first memory layer 208 and / or second memory layer 210. Cache 218 may be implemented, for example, using relatively low-latency memory such as DRAM, SRAM, etc. In some embodiments, cache 218 may be part of second memory layer 210. Therefore, second memory layer 210 may be referred to as latency-enhancing memory. In some embodiments, the second memory layer may be optimized for latency within memory system 204.

[0039] Figure 2The device 200 shown can be used to implement a wide variety of applications. For example, in some embodiments, device 200 can be used for ML training and / or inference (e.g., DL and / or DNN), speech recognition, language processing, image recognition, graph processing, recommendation generation, etc. In some embodiments, the first memory layer 208 can be configured as a high-bandwidth memory that can be used as the working memory of the accelerator, and the second memory layer 210 can be configured as a low-latency, high-capacity memory (e.g., using cache 218) that can store some or all of a model, such as a graph or ML model. Depending on the implementation details, this can reduce the time, cost, energy, etc., of moving parts of the model (and / or the model's input and / or output data) into the accelerator device, improve the access speed of one or more portions of the first and / or second memory layers, reduce the latency of one or more accelerator operations, etc.

[0040] Interconnect interface 202 and / or interconnect 212 may be implemented using one or more of any type of interface and / or protocol, including Peripheral Component Interconnect Express (PCIe), Nonvolatile Memory Express (NVMe), NVMe-over-fabric (NVMe-oF), Advanced eXtensible Interface (AXI), Ultra Path Interconnect (UPI), Ethernet, Transmission Control Protocol / Internet Protocol (TCP / IP), Remote Direct Memory Access (RDMA), RDMA over Converged Ethernet (ROCE), Fibre Channel, InfiniBand, Serial ATA (SATA), Small Computer Systems Interface (SCSI), Serial Attached SCSI (SAS), iWARP, etc., or any combination thereof. In some embodiments, the interconnect interface 202 may be implemented using one or more memory semantics and / or memory coherence interfaces and / or protocols (such as Compute Express Link (CXL), and / or CXL.mem, CXL.io, and / or CXL.cache, Gen-Z, Coherent Accelerator Processor Interface (CAPI), CacheCoherent Interconnect for Accelerators (CCIX), etc., or any combination thereof).

[0041] Memory interfaces 214 and 216 can be implemented using one or more of any type of interface, including DDRX, LPDDRX, Open Memory Interface (OMI), NVLink, High Bandwidth Memory (HBM), HBM2, HBM3, and / or any interconnect interface and / or protocol mentioned above, including CX1. Memory interfaces 214 and 216 can be implemented using coherent and / or non-coherent interfaces. For example, a non-coherent memory interface can be used for memory interface 214 between the first memory layer 208 and accelerator 206, while a coherent interface can be used for memory interface 216 between the second memory layer 210 and accelerator 206.

[0042] One or more of interconnect interfaces 202, 212, and memory interfaces 214 and 216 can be implemented as discrete components or integrated into the interconnect structure (e.g., using one or more switches for configuration). Figure 2 (The connections between the components are shown in the diagram).

[0043] Accelerator 206 can be implemented using any type of device, which may include one or more processing resources suitable for the accelerator, such as a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), a combinational logic-based accelerator, sequential logic, one or more timers, counters, registers, state machines, complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), central processing units (CPUs) (such as complex instruction set computer (CISC) processors (such as x86 processors) and / or reduced instruction set computer (RISC) processors (such as ARM processors)), or any combination thereof.

[0044] Device 200 can be implemented in any physical and / or electrical configuration and / or form factor (such as a standalone device, a plug-in card such as a PCIe adapter or expansion card, a plug-in device such as a connector and / or slot that can be inserted into a server chassis (e.g., a connector on the backplane and / or middleboard of a server or other device), etc.). In some embodiments, device 200 can be implemented in any connector configuration of interconnect interface 202 (such as SATA connector, SCSI connector, SAS connector, M.2 connector, U.2 connector, U.3 connector, etc.) in form factors of storage devices (such as 3.5-inch, 2.5-inch, 1.8-inch, M.2, Enterprise and Data Center SSD Form Factor (EDSFF), NF1, etc.). Any device disclosed herein can be implemented and / or used in conjunction with server chassis, server racks, data rooms, data centers, edge data centers, mobile edge data centers, and / or any combination thereof, in whole or in part. In some embodiments, device 200 may be implemented as a CXL Type-1 device, a CXL Type-2 device, a CXL Type-3 device, etc.

[0045] Figure 3 An embodiment of a device with accelerator virtualization according to an example embodiment of the present disclosure is shown. Figure 3 The device 300 shown may include an interconnect interface 302, a memory system 304, an accelerator 306, and an interconnect 312, which may be similar to those referenced above. Figure 2 Although memory system 304 may or may not implement a hierarchical memory structure, and may instead use any type of memory or combination thereof in any configuration, accelerator 306 may interface to memory system 304 via one or more memory interfaces 315 of any type.

[0046] Device 300 may also include virtualization logic 320, which can partition one or more resources of accelerator 306 into one or more virtual accelerators 306-1, ..., 306-N. The partitionable accelerator resources may include processing resources (e.g., processor cores), registers, memory resources, interconnect resources, etc. For example, in some embodiments, virtualization logic 320 may allocate one or more physical processor cores of accelerator 306 to one or more of virtual accelerators 306-1, ..., 306-N. Virtualization logic 320 may also allocate a portion of memory from memory system 304 to one or more of virtual accelerators 306-1, ..., 306-N. In embodiments that implement memory system 304 using layers, the virtualization logic may allocate a portion of memory from each layer to one or more of virtual accelerators 306-1, ..., 306-N. In embodiments where memory layers may include caches, a portion of the cache may be allocated to one or more of virtual accelerators 306-1, ..., 306-N.

[0047] In some embodiments, virtualization logic 320 may also allocate discrete interconnect resources, such as ports, to one or more of virtual accelerators 306-1, ..., 306-N.

[0048] According to the implementation details, one or more of the virtual accelerators 306-1, ..., 306-N can appear as decoupling accelerators for a host or other device. For example, each of the virtual accelerators 306-1, ..., 306-N can be implemented as a decoupling endpoint with a decoupling device identifier (ID). In some embodiments, the virtual accelerators can be implemented as decoupling endpoints by implementing the device 300 as a multi-headed device (where each of the virtual accelerators 306-1, ..., 306N can have a decoupling port).

[0049] Virtual accelerators 306-1, ..., 306N can be implemented using various degrees of isolation between virtual accelerators. For example, in some embodiments, all resources of each virtual accelerator (e.g., processor cores, memory, ports, etc.) can be completely isolated, so that the virtual accelerators do not compete for resources, and / or each virtual accelerator does not pollute the memory (e.g., cache) of any other virtual accelerator. As another example, in some embodiments, one or more resources (e.g., interconnect ports, a portion of cache, etc.) can be partially or completely shared between virtual accelerators.

[0050] In some embodiments, virtualization logic 320 may be programmable, for example, using one or more registers 322 that can be accessed by a user or application via interconnect interface 302 (e.g., using an API). For example, virtualization logic 320 may be programmed to configure a first virtual accelerator 306-1 to perform a first operation on a first portion of data received via interconnect interface 302 and stored in memory system 304. Virtualization logic 320 may also be programmed to configure a second virtual accelerator 306-2 to perform a second operation on a second portion of data received via interconnect interface 302 and stored in memory system 304. In some embodiments, the first and second portions of data may be stored in separate portions of memory system 304 that may be allocated to separate virtual accelerators 306-1 and 306-2.

[0051] Devices with accelerator virtualization according to exemplary embodiments of the present disclosure can be used in a wide variety of applications. For example, in some embodiments, a model (e.g., a graph, ML model, etc.) can be partitioned into multiple parts, each of which can be assigned to a virtual accelerator to implement the model in parallel. Depending on the implementation details, this can enable relatively large models to be implemented efficiently across multiple virtual accelerators. Furthermore, the use of virtual accelerators according to exemplary embodiments of the present disclosure can reduce or eliminate memory-constrained limitations, particularly if the virtual accelerator is implemented in conjunction with a hierarchical memory system according to exemplary embodiments of the present disclosure.

[0052] For the purpose of illustrating the principles of this disclosure, some exemplary embodiments are described below in the context of systems, methods, and / or devices, wherein the systems, methods, and / or devices may use specific implementation details, such as CXL interconnects and / or protocols, CXL type 2 devices, cached DRAM, capacity-enhancing flash memory, accelerator-based NPUs, DL and / or graph models, and / or other implementation details. However, the principles are not limited to these exemplary implementation details and can be applied to embodiments that may use any other interconnects, protocols, memory types, device types, accelerators, models, etc.

[0053] Figure 4A An example embodiment of a host according to an example embodiment of this disclosure is shown. Figure 4B An example embodiment of a device having a hierarchical memory and a virtual accelerator is shown according to an example embodiment of the present disclosure. Figure 4A and Figure 4B Examples of systems according to exemplary embodiments of this disclosure are illustrated and are collectively referred to as FIG4. For example, the embodiments shown in FIG4 can be used, for example, to implement the above-referenced... Figure 2 and Figure 3 Any of the embodiments described (including one or more of its features).

[0054] Referring to FIG4, the system may include a device 400 and a host 424 that can communicate via interconnect 430. In some embodiments, the system may also include a remote storage device 426 that can communicate with the host 424 via network 431 and network interface 432, as shown in FIG4. However, in other embodiments, the remote storage device 426 may be connected to the system in any other way (e.g., via one or more interconnects such as interconnect 430 using switches).

[0055] Device 400 may include a tiered memory system 404 having bandwidth-enhancing memory 408 and latency-enhancing memory 410, wherein the latency-enhancing memory 410 may include capacity-enhancing memory 417 and cache 418. Bandwidth-enhancing memory 408 may be implemented at least partially using one or more bandwidth-enhancing point-to-point memory devices, such as stacked LPDDRX devices and / or HBM devices that may include memory dies with an interposer layer. Latency-enhancing memory 410 may be implemented at least partially using capacity-enhancing memory 417, such as low-latency NAND memory and DRAM cache 418.

[0056] Device 400 may also include one or more accelerators 406, which in this example can be implemented as an NPU. The NPU 406 may be partitioned into one or more virtual NPUs 406-1, ..., 406-N by virtualization logic 420, wherein the virtualization logic 420 may be programmed, for example, using one or more registers 422. The virtualization logic 420 may be, for example, similar to the above reference. Figure 3 The virtualization logic described is operated in the manner of 320.

[0057] Referring to Figure 4, virtual NPUs 406-1, ..., 406-N can interface to bandwidth-enhancing memory 408 via a non-coherent memory interface 414, such as DDRX and / or NVLINK. Virtual NPUs 406-1, ..., 406-N can interface to latency-enhancing memory 410 via a coherent interface 416, such as CXL. Although memory interfaces 414 and 416 are shown as separate interfaces, in some embodiments they can be integrated into, for example, a memory interface structure. In some embodiments, some or all of the bandwidth-enhancing memories 408 can be used as working memory for one or more of NPUs 406 and / or virtual NPUs 406-1, ..., 406-N.

[0058] Device 400 may also include an interconnect interface 402, which in this example can be implemented as a CXL interface, and uses the CXL.io protocol interface 434, the CXL.cache protocol interface 436, and / or the CXL.mem protocol interface 438. Device Coherence (DCOH) engine 440 (e.g., a consistency broker) can resolve the consistency of one or more caches at the device and / or manage one or more CXL bias states.

[0059] In some embodiments, device 400 may also include a memory switch 448. In some embodiments, memory switch 448 may be configured to physical connections to and / or between various types of memory that can be used for first-level memory, second-level memory, cache, etc. Although memory switch 448 is shown as a single component, in some embodiments it may be implemented in a distributed manner, wherein one or more portions of memory switch 448 are located within and / or between any components of hierarchical memory system 404, any components of NPU 406, etc., for example, as shown in FIG. 4 as the overlap between memory switch 448, hierarchical memory system 404, and NPU 406. Therefore, in some embodiments, memory switch 448 and / or any components of hierarchical memory system 404 and / or NPU 406 may form a memory structure, wherein memory switch 448 may serve as a memory structure manager. In some embodiments, the memory manager may implement one or more memory configurations using topologies such as torus, mesh, point-to-point, etc.

[0060] In some embodiments, memory switch 448 may be (e.g., by an application via an API) programmable to configure memory switch 448 and / or any component of tiered memory system 404 and / or NPU 406. For example, a programmer may know that a particular workload to run on one of NPU 406 or virtual NPUs 406-1, ..., 406-N may require or benefit from a specific amount of cache 418. Therefore, a programmer (e.g., by an application) can program the memory switch (e.g., via an API) to configure a specific amount of cache to one of NPU 406 or virtual NPUs 406-1, ..., 406-N.

[0061] In some embodiments, device 400 may also include memory manager 450. Memory manager 450 can manage data movement within tiered memory system 404, between any component of tiered memory system 404 and NPU 406, and between any component of tiered memory system 404 and host and / or other devices. Therefore, memory switch 448 can configure the physical arrangement of memory and other resources, while memory manager 450 can manage data movement within the configured memory arrangement. In some embodiments, memory switch 448 and / or memory manager 450 can configure memory and / or control data movement based on one or more observed or learned memory access patterns, for example, determined by memory access pattern analyzer 452. In some embodiments, memory manager 450 can implement one or more caching algorithms and / or caching strategies.

[0062] In some embodiments, device 400 may also include one or more preprocessors 442, one or more input and / or output (I / O or IO) engines 444, and / or one or more compression and / or decompression logic 446. The preprocessors can perform any function that can prepare data for processing by NPU 406 and / or one or more virtual NPUs 406-1, ..., 406-N. For example, preprocessing can be used for data cleaning (e.g., noise removal, filling in missing or null values, etc.), data transformation (e.g., standardization, normalization, feature selection, etc.), data organization, data reduction (e.g., aggregation, numerosity reduction, dimensionality reduction, etc.), etc.

[0063] The IO engine 444 can implement one or more IO-related functions, such as data deduplication, offloading all or part of the preprocessing of the dataset (e.g., to the preprocessor 442), etc. Compression and / or decompression logic 446 can help improve the effective bandwidth of the interconnect interface 402 by compressing data before it is transmitted out through the interconnect interface 402 and / or decompressing data after it is received through the interconnect interface 402.

[0064] In some embodiments, device 400 may also include an allreduce engine 454, which can perform one or more operations associated with the allreduce operation. For example, the allreduce engine 454 can help schedule data transfers (e.g., tensors, updates from the NPU, etc.) between the NPU, the device, etc.

[0065] Some embodiments may include one or more internal interconnects between various components, some examples of which are identified as 456a and 456b in FIG. 4 (which may be collectively referred to as 456). Any of the internal interconnects 456 can be implemented using any type of bus, point-to-point connection, etc. In some embodiments, one or more internal interconnects 456 may be implemented using PCIe, NVLink, AXI, etc. In some embodiments, one or more internal interconnects 456 may be implemented using packet processing (e.g., to increase the bandwidth of data movement within and / or between any of the components shown in FIG. 4). Packet processing may be implemented, for example, by performing packet processing on any of the internal interconnects 456 using one or more network-on-chip (NOC) devices.

[0066] Host 424 may include CPU 458 having an interface (e.g., CXL) controller 460, a memory controller 462, and an internal cache 464. The memory controller 462 may control the cache 464 and / or one or more host (e.g., system) memories 466. In some embodiments, the cache may be implemented using SRAM, and the host memory 466 may be implemented using DRAM.

[0067] Host 424 may include interconnect interface 428, which in this example may be implemented as a CXL interface, and uses one or more of the CXL.io, CXL.cache and / or CXL.mem protocols.

[0068] The host 424 may also include a local storage device 468, which may be implemented, for example, using any type of storage device(s) based on any type of memory and / or storage medium (including solid-state media, magnetic media, optical media, etc.).

[0069] Host 424 may also include a network interface 432 that provides access to remote storage device 426 and / or any other system, host, device, etc. Network interface 432 may be implemented as, for example, a network interface card (NIC), which can use any suitable network interface and / or protocol, including Ethernet, TCP / IP, RDMA, ROCE, and / or any other interface and / or protocol including those mentioned above. Remote storage device 426 may be implemented, for example, using any type(s) of storage devices based on any type of memory and / or storage medium, including solid-state media, magnetic media, optical media, etc., and configured in a server chassis, server rack, data center, data center, edge data center, mobile edge data center, and / or any combination thereof.

[0070] CPU 458, local storage device 468 and / or network interface 432 may communicate, for example, via system bus 470.

[0071] Host 424 may also include accelerator (e.g., NPU) virtualization logic 472, which may cooperate with virtualization logic 420 at device 400 to partition, manage, and / or use one or more virtual NPUs 406-1, ..., 406-N. Either host-side virtualization logic 472 and / or device-side virtualization logic 420 may be implemented in hardware, software, or a combination thereof. For example, in some embodiments, device-side virtualization logic 420 may be implemented primarily in hardware to partition resources associated with NPU 406 into one or more virtual NPUs 406-1, ..., 406-N, while host-side virtualization logic 472 may be implemented primarily in software, for example, at least partially in a device driver of device 400 and / or at least partially in one or more applications 474. In some embodiments, the device driver in the host-side virtualization logic 472 may receive one or more commands from the application 474 via an API to program one or more registers 422 in the device-side virtualization logic 420, thereby creating, managing and / or using one or more of the virtual NPUs 406-1, ..., 406-N.

[0072] In some embodiments, device-side virtualization logic 420 and / or host-side virtualization logic 472 (collectively referred to as virtualization logic) can provide flow control for one or more of the virtual NPUs 406-1, ..., 406-N. In some embodiments, flow control can determine that each virtual NPU can be saturated (e.g., limited to a specific amount of bandwidth) between the virtual NPU and the host 424. In some embodiments, flow control can be provided on a per-partition basis (e.g., per virtual NPU). This can be referred to as spatial multiplexing, which can be implemented in a parallel form across multiple virtual NPUs.

[0073] In some embodiments, the virtualization logic may include a hardware and / or software scheduler 423 to identify command flows that can be multiplexed among virtual NPUs. This may be referred to as time multiplexing and / or scheduling.

[0074] In some embodiments, temporal multiplexing can be combined with spatial multiplexing. For example, ten different command streams can be temporally multiplexed across ten different virtual NPUs to present the appearance of 100 different processors. Depending on the implementation details, temporal and / or spatial multiplexing can be distributed among virtual NPUs based on, for example, the amount of resources each virtual NPU can have.

[0075] In some embodiments, network interface 432 and / or remote storage device 426 may initiate DMA transactions to tiered memory system 404. For such transactions, the data path may be via network interface 432, CXL controller 460, host-side interconnect interface 428, interconnect 430 (using CXL.io protocol), device-side interconnect interface 402, and memory switch 448.

[0076] When handling DMA transactions, memory switch 448 can use one or more queues that can be separate from one or more queues used for CXL transactions (e.g., for vector transfers using flits). Using separate queues can avoid conflicts that may arise when receiving requests from the host via the CXL.mem and / or CXL.cache interfaces when performing large block transfers (e.g., DMA) through CXL.io.

[0077] In some embodiments, the list of physical address ranges may be maintained, for example, by a remote server to facilitate the initiation of peer-to-peer RDMA requests via network interface 432 using one or more DMA engines. Address ranges may be maintained for one or more devices, such as device 400 and / or any virtual NPU or other accelerators within each device.

[0078] To perform a DMA transfer to the NPU 406, the initiating device can use a destination (e.g., target) buffer in device 400. Therefore, in some embodiments, memory manager 450 can allocate one or more target buffer locations for the DMA transfer.

[0079] In some embodiments, the CXL interface 402 may implement CXL.io, CXL.mem, and / or CXL.cache capabilities using host-managed device memory (HDM), which may be allocated from one or more portions of the tiered memory system 404. For example, data transferred from host 424 to device 400 using CXL may be placed in an HDM region of the memory. In some embodiments, host 424 may be unaware of any other memory on device 400 besides the HDM memory. One or more memory regions exposed to host 424 may be indicated, for example, by one or more start and / or end addresses in a base address register (BAR). For example, in device 400 with a total of 1TB of memory in the second memory tier 410, only 10GB of the 1TB may be exposed to host 424 as HDM.

[0080] Any or all components and / or capabilities of device 400 can be configured, managed, operated, implemented, etc., with the help of firmware, which may run, for example, on a RISC processor core on device 400. For example, in some embodiments, the firmware at device 400 may determine how much and / or what type of memory (e.g., as HDM) of the hierarchical memory system 404 is exposed to host 424 and / or what device capabilities are exposed to host 424. These determinations may be based on, for example, the total resources available at device 400, resources that can be allocated to other uses, etc.

[0081] In embodiments where device 400 is implemented as a CXL type 2 device, I / O transactions can follow any generation of the PCIe protocol using an Address Translation Service (ATS). For example, in order for device 400 to request access to host memory 466 (e.g., using CXL.cache), device 400 can use the ATS to make the request because device 400 may not know the actual physical address in host memory 466 to which the request is made. For example, the ATS can be implemented using an input-output memory management unit (IOMMU) and / or a device-side translation lookaside buffer (DTLB) that can cache one or more address translations in the CXL controller 460. In some embodiments, this can enable device 400 to retrieve data from host memory 466 (e.g., using CXL.cache) and / or remote storage device 426 as needed.

[0082] Host 424 may also include a memory access pattern analyzer 452, which can monitor and / or analyze memory access throughout the system, including memory access within and between one or more of the tiered memory system 404, NPU 406, and / or virtual NPUs 406-1, ..., 406-N, to determine one or more memory access patterns that can be used, for example, to configure one or more components of the tiered memory system 404, NPU 406, etc., for more efficient operation. In some embodiments, the memory access pattern analyzer 452 may use one or more ML techniques to identify one or more memory access patterns.

[0083] Host 424 can run any type of application 474 related to the use of one or more NPUs and / or other types of accelerators that can be implemented by device 400. For example, host 424 can run one or more applications that can use device 400 to implement graph processing, ML frameworks (including one or more DL frameworks such as TensorFlow, PyTorch, etc.).

[0084] As mentioned above, using one or more accelerator devices with large models can be challenging, for example, because only a small portion of the model and / or the model's input data can fit into the accelerator device's working memory. Depending on the implementation details, adding sufficient additional working memory to store the entire model can be prohibitively expensive, for example, in terms of cost, space, and power consumption. Therefore, using one or more accelerator devices with large models can involve costly data transfers between the host and / or from the host to swap different portions of the model into the accelerator device's working memory.

[0085] However, in some embodiments, the system shown in FIG4 can be used to implement large DL models (e.g., large DNN models), for example, by using a capacity-enhancing memory 417, such as flash memory (e.g., low-latency NAND memory), in the second memory layer 410. Depending on the implementation details, this capacity-enhancing memory 417 can be used to store the entire DL model at an acceptable cost in terms of economy, space, and power consumption. Therefore, the system can reduce or eliminate data transfers associated with using accelerator devices with DL models.

[0086] Furthermore, using a cache 418 (e.g., DRAM) in the latency-enhancing memory 410 can compensate for potential latency issues of flash memory (e.g., data can be quickly moved from the second memory layer 410 to the first memory layer 408).

[0087] Furthermore, according to exemplary embodiments of this disclosure, the system enables the implementation of DL models using a variety of techniques. For example, CXL can be used in conjunction with PCIe, thus the system and / or device 400 can select between PCIe and CXL transfers depending on the specific circumstances. For example, large trained models can be transferred to a second memory layer (latency-enhanced and / or capacity-enhanced memory) 410 in 4KB granularity, for example, before the trained model used for inference. This can be referred to as placing the model behind the NPU 406. In some embodiments, PCIe transfers can be implemented using the CXL.io protocol, which, depending on the implementation details, can operate substantially like PCIe.

[0088] The model's runtime input data can be received at host 424, for example, via network interface 432. The input data can be transferred from host 424 to the working memory of NPU 406 (e.g., the first memory layer 408 of bandwidth-enhanced memory), so that NPU 406 can use the model to process the input data. Although PCIe may not provide consistency, it can still be used to transfer input data (e.g., DL vectors) into working memory, for example, by pushing the input data into working memory and then notifying NPU 406 that the data is available. (For simplicity, in some embodiments, NPU 406 and / or any of the virtual NPUs 406-1, ..., 406-N may be collectively referred to as NPU 406.)

[0089] As another example, CXL can be used to transfer input data from host 424 to working memory 408. In some embodiments, a potential advantage of using CXL to transfer input data (e.g., DL vectors) is that the consistency of CXL allows data to be transferred passively, for example, on demand by NPU 406 (and / or any of the virtual NPUs 406-1, ..., 406-N). For example, because NPU 406, working memory 408, and / or host 424 can be in the same consistency domain, it can make cache line requests when NPU 406 can work with cache lines available. Furthermore, CXL can provide finer-grained data transfer (e.g., 64B microchips). Depending on the implementation details, this can reduce or eliminate unnecessary data transfers and / or achieve more efficient data transfer.

[0090] In some embodiments, cache memory 418 may be implemented with hardware control, software control, or a combination thereof. For example, in some embodiments, some or all of the control over cache 418 may be determined by the hardware microarchitecture with little or no software control. Hardware control over cache 418 may be beneficial, for example, in cases where data has good spatial and / or temporal locality, in which case offloading the control work to hardware may be more efficient. Hardware control over cache 418 may also be beneficial for fine-grained data transfers, as software may find it difficult to migrate small portions of data due to the software overhead associated with controlling cache 418. In the case of a hardware-based cache, cache 418 may be transparent to NPU 406; for example, NPU 406 may only see capacity-enhanced memory 417 (e.g., NAND flash memory) but have the benefit of reduced latency provided by faster memory (e.g., DRAM) provided by cache 418.

[0091] As another example, in some embodiments, cache 418 can be primarily controlled by software. For instance, software control of cache 418 can allow a programmer (e.g., through an application) to control which input data vectors are prefetched into the cache, which portion of the model stored in NAND flash memory is prefetched into the cache, and so on. Software control of cache 418 can be beneficial, for example, in situations where data lacks spatial and / or temporal locality (e.g., utilizing streaming data access), where the programmer has better control over which data to cache. Furthermore, in cases of large-granularity data transfer, the software overhead may be smaller; therefore, providing programmers and / or applications with greater control over placing and / or moving data can be advantageous.

[0092] In some embodiments, memory switch 448 and / or memory manager 450 may implement hardware intelligence to identify hot data (e.g., frequently used or potentially used blocks, pages, lines, etc.) to be moved and / or retained in cache 418 and / or cold data to be removed from cache 418. In some embodiments, memory switch 448 and / or memory manager 450 may implement control logic for hardware and / or software control of cache 418. In some embodiments, memory switch 448 and / or memory manager 450 may implement flow detection and / or prefetching to predict which data will be prefetched into the cache. In some embodiments, cache 418 may operate as a buffer for portions of a model that can be prefetched and / or input data (e.g., hot vectors and / or hot indices) to reduce or prevent access to capacity-enhanced memory 417 (e.g., flash memory).

[0093] In some embodiments, an NPU 406 or other type of accelerator can be implemented using a single instruction, multiple SIMD engines, and / or multiple streaming units. One or more NPU 406s can be used to implement a dataflow processing engine distributed across one or more clusters, where each cluster can include a dataflow engine or SIMD engine, one or more registers, floating-point (FP) units, and / or a single SRAM cache. In some embodiments, a translation lookaside buffer (TLB) can be used to accelerate addressing. In some embodiments, one or more multi-tiered caches can be implemented across a computer cluster.

[0094] In some embodiments, host 424 and NPU 406 may share one or more cache lines in a coherent memory space, which may be maintained in a coherent state by CXL controller 460 and / or DCOH engine 440. DCOH engine 440 may implement cache line snooping to track the presence and / or state of one or more cache lines on the device side, enabling device 400 to respond when host 424 requests a cache line from device 400. In some embodiments, for example, DCOH engine 440 may respond to host snooping on a device-to-host (D2H) response channel because the device may be holding dirty data. To implement the CXL.cache interface, DCOH engine 440 may implement response and / or request logic to generate responses to requests for D2H transfers from host 424, and / or requests for host-to-device (H2D) transfers generated by device 400. In some embodiments, the DCOH engine 440 enables the device 400 to request one or more cache lines from other locations in the system besides the host. For example, the device 400 may request one or more cache lines from different devices, accelerators, memory extensions, memory buffers, etc., and the CXL controller 460 at the host 424 facilitates the transfer of the requested cache lines to the device 400.

[0095] In embodiments where device 400 is implemented as a CXL type 2 device, DCOH engine 440 can track bias states such as device bias and / or host bias. In a device bias state, an NPU or other type of accelerator can access device-local memory without the performance penalty that might be associated with a host bias state; however, the host can still access device-local memory using consistent semantics.

[0096] Some embodiments of device 400 may provide one or more enhancements to CXL or other memory coherence and / or memory semantic interfaces and / or protocols. For example, coherence regions of device memory (e.g., any memory in tiered memory system 404) may be written to by multiple sources, such as host 424, device 400 and / or other hosts, devices, etc. Implementations of device 400 according to exemplary embodiments of this disclosure may handle writes to coherence regions from these multiple sources by (1) restricting host-visible coherence regions (e.g., HDM) and / or (2) using DCOH to order (and / or enforce their order) writes. Depending on the implementation details, this may improve performance, for example, where the order of multiple writes from multiple sources may be undefined. Thus, DCOH can define the order of writes (e.g., strictly, loosely, etc.). Similar techniques can be applied to reads.

[0097] In some embodiments, host 424 may use a master-to-subordinate protocol to perform read and / or write accesses to consistent regions of the device memory. Upon a read request, the device may respond with data and / or no-data response (NDR) fields.

[0098] The memory manager 450 can manage how memory rows are distributed between the first memory layer 408 and the second memory layer 410. The memory manager 450 can also implement cache management between the cache (e.g., DRAM) 418 and the capacity-enhancing memory 417 (e.g., flash memory) in the second memory layer 410. In some embodiments, the memory manager 450 can implement hardware-based caching and / or software-based caching, for example, by utilizing one or more parameters to expose regions of the cache 418 and the capacity-enhancing memory 417 to one or more software layers, such as an application 474 that may implement a framework.

[0099] In some embodiments, memory manager 450 may implement a tag cache manager and / or a controller for cache 418 (e.g., a DDR controller if DDR DRAM is used for cache 418).

[0100] In some embodiments, large-granularity transfers, such as a system-wide copy of the entire model from a network buffer or external storage device to capacity-enhanced memory 417, can be performed using a DMA stream over PCIe via CXL.io interface 434. Smaller-granularity transfers can be performed using a CXL stream over CXL.mem interface 438. In some embodiments, in addition to initially filling capacity-enhanced memory 417 with the model, memory manager 450 can enable application 474 on host 424 to access capacity-enhanced memory 417 directly or indirectly through a window of cache 418 to facilitate swapping in and / or swapping out one or more portions of the model between capacity-enhanced memory 417 and cache 418. In some embodiments, memory manager 450 can separate one or more regions of the first memory layer 408 and the second memory layer 410, which can be exposed to host 424 in a configurable manner, for example, through one or more registers in memory manager 450 that are accessible via an API. Therefore, in some embodiments, large DL models can be swapped in and / or swapped out of capacity-enhancing memory 417 (e.g., NAND flash) for initial filling prior to runtime. However, during runtime, such as when inference is performed using the model, there may be one or more portions of the model that can be accessed more frequently, which can be facilitated by caching those portions of the model using cache 418 between layers 408 and 410.

[0101] In some embodiments, a device memory manager 476 may be included to manage one or more aspects of memory devices, storage devices, etc., within management layers 408 and 410. While memory manager 450 may implement caching algorithms, data movement between management layers 408 and 410, etc., device memory manager 476 may perform lower-level functions such as wear leveling, address translation, hash tables, tag management, additional logic, memory devices, data movement within storage devices, etc. In some embodiments, device memory manager 476 may implement a signaling scheme that enables application 474 (e.g., a DL framework) to indicate one or more base addresses, offset addresses, etc., for active portions of the model. This allows device memory manager 476 to move data transparently to the host and / or NPU using hardware. In some embodiments, device memory manager 476 may implement one or more hardware techniques to determine bandwidth-sensitive portions of memory without the involvement of host 424, NPU 406, etc. In some embodiments, device manager 476 may expose one or more controls via an API, for example, to enable programmers who may know the wear level patterns of a particular application to implement wear leveling schemes for that application.

[0102] In some embodiments, one or more devices having accelerator virtualization and / or tiered memory according to exemplary embodiments of this disclosure can be used to implement memory and / or accelerator deaggregation, for example, by using single-level and / or multi-level switching.

[0103] Figure 5 An embodiment of a system with depolymerization according to an example embodiment of the present disclosure is shown. Figure 5 The system shown may include a host 524 connected to a first switch 578a and a second switch 578b via interconnects 556a and 556b, respectively. The first switch 578a may be connected to a first device 500a and a first memory 580a via interconnects 556c and 556d, respectively. The second switch 578b may be connected to a second device 500b and a second memory 580b via interconnects 556e and 556f, respectively. In some embodiments, the first switch 578a and the second switch 578b, along with interconnects 556a-556f (which may collectively form an interconnect structure), may be implemented using the CXL protocol with PCIe switches and / or interconnects.

[0104] According to exemplary embodiments of this disclosure, the first device 500a and the second device 500b may be implemented, for example, using one or more devices having accelerator virtualization and / or hierarchical memory.

[0105] Figure 5 The system shown allows resources such as the first switch 578a and the second switch 578b, as well as the first memory 580a and the second memory 580b, to be available online and / or offline based on, for example, current processing and / or memory requirements. One or more of the resources can be time-multiplexed, e.g., cross-domain. Furthermore, one or more of the downstream ports (e.g., host root ports 582a and 582b) and / or switches (e.g., switches 578a and 578b) can be virtualized, for example, for... Figure 5 The diagram shows multiple virtual NPUs 406-1, ..., 406-N. In some embodiments, one or more of devices 500a and 500b may be implemented as multi-head devices having multiple ports for, for example, multiple virtual accelerators.

[0106] Figure 5The system illustrated may also include a structure manager, which can perform one or more functions, such as device discovery, virtual switch creation and / or management, binding virtual ports to physical ports, etc. In some embodiments, the structure manager may reside at host 524 and be implemented, for example, using one or more device drivers. In some embodiments, the structure manager may be implemented in a sideband configuration, for example, using a system management bus (SMBus). Although Figure 5 The embodiment shown is illustrated as having two switches, two devices, and two memories, but any number of components can be used with any number of switches, interconnections, etc., using any level of switching.

[0107] Devices with accelerator virtualization and / or tiered memory according to exemplary embodiments of this disclosure can be used to implement various types of parallelism. For example, data parallelism can involve partitioning workload data across multiple accelerator devices, each of which may have the same model. Depending on the implementation details, data parallelism may be computationally constrained, particularly for large models. However, devices with tiered memory systems according to exemplary embodiments of this disclosure can, for example, accommodate large models in capacity-enhanced memory in a second memory tier. Furthermore, using a cache in the second memory tier can compensate for the potential latency issues of the capacity-enhanced memory.

[0108] In model parallelism, parts of the model can be partitioned across multiple accelerators, and the same data can be processed by each accelerator. For example, if the model is split in half between NPU0 and NPU1, all or part of the data can be processed first by NPU0, and the same data can be processed by NPU1. Model parallelism can be used for full reduction algorithms and / or all-to-all (A2A) communication, where one or more (sometimes all) NPUs communicate with one or more (sometimes all) other NPUs after each epoch. A device with accelerator virtualization according to an example embodiment of this disclosure can accommodate model parallelism, for example, by enabling each of a plurality of virtual accelerators to process a portion of the model.

[0109] In some embodiments, choosing between data parallelism and model parallelism may involve one or more tradeoffs. For example, data parallelism may be difficult to implement with large models because the entire model may be copied and stored at each accelerator. Furthermore, in the case of data parallelism, it may be necessary to synchronize data for training the DL model. For example, weights can be synchronized during training because each accelerator can work on the same model with different training data, so the system can synchronize the data and determine one or more averages of the weights for each epoch. In the case of model parallelism, less memory and / or storage resources can be used to store only a portion of the model at each accelerator, and training synchronization problems can be avoided. However, depending on the implementation details, communication between accelerators may increase.

[0110] According to implementation details, the device 400 shown in FIG4 can reduce or eliminate tradeoffs, for example, when implementing model parallelism using full reduction and / or all-to-all communication primitives. For example, the use of a consistency interface such as CXL can enable multiple NPUs (e.g., multiple NPUs 406 at different instances of device 400 and / or multiple virtual NPUs 406-1, ..., 406-N within a single device 400) to exchange data at a finer granularity (e.g., using CXL shards). Thus, the full reduction engine 454 can be configured as a station that can collect updates from various NPUs using CXL primitives and / or the data lines required to start various NPUs. In some embodiments, the full reduction engine 454 can also be configured to implement message passing interface (MPI) transmit and / or receive primitives via interconnect 430 and / or network 431. According to implementation details, this can enable the system to be scaled out to multiple accelerators via network 431 in model parallelism. Furthermore, CXL can achieve consistency across devices (e.g., as one or more NPUs 406 and / or one or more virtual NPUs 406-1, ..., 406-N), servers, multiple accelerators within a rack, across networks, etc. Therefore, in some embodiments, the full reduction engine 454 can implement a combination of MPI and CXL, which can clear CXL requests to collect updates for use by device 400 and / or generate updates that other devices can use to scale to multiple NPUs. Depending on the implementation details, this can enable the full reduction engine 454 to efficiently exchange and / or schedule tensors, for example, using CXL semantics.

[0111] Figure 6 The diagram illustrates an example embodiment of the present disclosure and an embodiment of a method for global partitioning of the diagram. Figure 6 Figure 684 shown may include twelve vertices 683, represented as v1 to v3 connected by edges (x,y).12 Here, x and y can indicate vertices connected by an edge. In the case of a directed edge, x and y can indicate the origin vertex and the destination vertex, respectively. In the case of an undirected edge, x and y are unordered. Therefore, edge (1, 4) indicates the edge between vertex v1 and vertex v4. (To avoid obscuring the attached diagram, not all edges may be labeled.)

[0112] Figure 684 can be divided into three parts at the dashed lines 686a and 686b: 684a (which may include vertices v1 to v6), 684b (which may include vertices v7 to v8), and 684b (which may include vertices v7 to v8). 10 ) and 684c (which may include vertex v) 11 to v 12 The global partitioning can be performed using this method. Figure 684 can be partitioned, for example, by applications running on the host machine and implementing ML frameworks such as TensorFlow, PyTorch, etc.

[0113] The application can partition vertices into three accelerator devices 688, denoted as dev1 to dev3. Specifically, the application can partition the vertices of the first part 684a of the graph as dev1, the vertices of the second part 684b of the graph as dev3, and the vertices of the third part 684c of the graph as dev2. The application can partition vertices based on one or more parameters of the graph elements (e.g., vertices and / or edges) (such as memory to be used, computation time, etc.) and one or more parameters of the accelerator devices 688 (such as computational resources, memory, interconnects, etc.).

[0114] although Figure 6 The graph shown may include twelve vertices and 17 edges and may be divided into three parts, but in other embodiments, any number of vertices, edges and / or partitions may be used in any configuration.

[0115] Figure 7 A partial scheduling operation according to an example embodiment of this disclosure is illustrated. Figure 7 The embodiments shown can be used, for example, to schedule graph processing operations of portions 684a, 684b, and 684c of the graph using accelerator devices 788, respectively denoted as dev1, dev3, and dev2, as... Figure 6 As shown, these parts have been divided into the accelerator device 788.

[0116] refer to Figure 7 An edge between two vertices within one device (e.g., edge (1, 4)) can be scheduled for processing by the corresponding device (e.g., dev1). An edge between vertices spanning two devices (e.g., edge (4, 11)) can be scheduled for processing by one or both of the corresponding devices. For example, an edge containing the destination vertex v... 11The device dev2 can process the edge (4, 11).

[0117] Figure 8 An embodiment of a device virtualization method according to an example embodiment of the present disclosure is shown. Although Figure 8 The embodiments shown can be used in any type of application; however, depending on the implementation details, some embodiments may be particularly useful for virtualizing devices implementing multiple large DNNs. For the purpose of illustrating the principles of this disclosure, Figure 8 The method shown in the figure uses Figure 6 and Figure 7 The diagram and its partitioning are shown in the figure. However, this principle can be implemented using any other diagram, model, etc., and its partitioning.

[0118] refer to Figure 8 For illustrative purposes, Figure 884 can be shown in a similar manner. Figure 6 The implementation is shown in Figure 684, but any other diagram, model, etc., can be used. In this embodiment, operations 881, 883, and 885 can be performed by a host, and operation 887 can be performed by a device; however, in other embodiments, the operations may be distributed differently. The host and device can be implemented, for example, using any host and / or device disclosed herein, including those referenced. Figure 2 , Figure 3 Figure 4 and / or Figure 5 Those described. Figure 884 can be implemented, for example, by application 874 (such as an ML framework running on the host).

[0119] refer to Figure 8 During operation 881 (part 1), in the configuration process prior to runtime, the device may present itself to the host as multiple virtual NPUs, each virtual NPU having one or more resources, including, for example, one or more processor cores, memory (e.g., tiered or non-tiered memory), interconnects, etc. In some embodiments, one or more interconnects to the host may be implemented as inter-device and / or intra-device.

[0120] In operation of the 883 (part 2.1), the host can annotate graph inputs and / or outputs with one or more parameters, such as the amount of memory to use, computation time, etc.

[0121] In operation 885 (part 2.2), the host can partition Figure 884 into devices dev1, dev2, and dev3 based on one or more parameters of the virtual NPU (such as computing resources (e.g., processor cores), memory, interconnects, etc.). In some embodiments, the host can attempt to match one or more parameters of the graph portion (such as memory to be used, compute time, etc.) with corresponding parameters of the virtual accelerator devices (such as computing resources (e.g., processor cores), memory, interconnects, etc.).

[0122] Some example values ​​for the parameters in the graph section are as follows:

[0123] dev1: 30GB, Delta = 0;

[0124] dev2: 20GB, Delta = max(0, 10 seconds);

[0125] dev3: 28GB, Delta = 0.

[0126] In Operation 887 (Part 3), a device (e.g., a device implementing a virtual NPU) can extract one or more operational parameters (such as memory usage, task dependencies, timing information for one or more graph partitions, etc.) from a graph partition provided by the host for runtime use. In some embodiments, the device can establish one or more process address spaces for each virtual device and partition. Each process address space can have a process address space identifier (PASID). Figure 8 In the example shown, the device can establish process address spaces PASID0, PASID1, and PASID2 for virtual accelerators dev1, dev2, and dev3 respectively in physical accelerator 806. In some embodiments, PASIDs can be implemented in the context of CXL, for example, by utilizing dedicated CPU time-sharing, shared virtual machines (VMs), and / or shared single root input and / or output virtualization (SR-IOV).

[0127] Figure 9 An embodiment of the workflow of a device virtualization method according to an example embodiment of the present disclosure is shown. Figure 9 The workflow shown can be, for example, with... Figure 8 Use it together with the method shown in Figure 10.

[0128] refer to Figure 9 In operation 902, for example, one or more user applications running on a host may create one or more graph representations. In operation 904, the one or more applications may use an ML framework to implement one or more graph representations. In operation 906, the one or more applications may partition the graph into one or more parts and map the one or more parts to one or more NPUs (e.g., virtual NPUs), for example, for memory performance (e.g., optimization) based on operational parameters such as task dependencies, peak memory capacity, minimum synchronization overhead, etc.

[0129] Figure 10A and Figure 10B (Together forming Figure 10) illustrates an embodiment of a memory scheduling method according to an exemplary embodiment of the present disclosure. While the embodiments shown in Figure 10 can be used for any type of application, some embodiments may be particularly useful for implementing memory scheduling of multiple large DNNs, depending on the implementation details. For the purpose of illustrating the principles of the present disclosure, the method shown in Figure 10 uses... Figure 6 and Figure 7 The diagram and graph partitions are shown in the figure. However, these principles can be implemented using any other graph, model, etc., and their partitions.

[0130] refer to Figure 10A For illustrative purposes, Figure 1084 can be shown in a similar manner. Figure 6 The implementation is as shown in Figure 684, but any other graph, model, etc., can be used. In this embodiment, operations 1081, 1083, 1085, and 1089 can be performed by the host. Figure 1084 can be implemented, for example, by an application 1074 such as an ML framework running on the host, and the memory graph generator 1075 can also be implemented on the host. However, in other embodiments, the operations can be distributed differently, for example, between the host and the device. The host and the device can be implemented, for example, using any host and / or device disclosed herein, including those referenced. Figure 2 , Figure 3 Figure 4 and / or Figure 5 Those described.

[0131] Referring to Figure 10, during operation 1081 (part 1), during the configuration process prior to runtime, a device (e.g., a CXL type 2 device) can present itself to the host as multiple virtual NPUs, each virtual NPU having one or more resources, including, for example, one or more processor cores, memory (e.g., tiered or non-tiered memory), interconnects, etc.

[0132] In Operation 1083 (Part 2.1), the host can annotate graph inputs and / or outputs with one or more parameters, such as the amount of memory to use, computation time, etc.

[0133] In operation 1085 (part 2.2), the host may partition Figure 884 into devices dev1, dev2, and dev3 based on one or more parameters of the virtual NPU (such as computing resources (e.g., processor cores), memory, interconnects, etc.). In some embodiments, the host may attempt to match one or more parameters of the graph portion (such as memory to be used, compute time, etc.) with corresponding parameters of the virtual accelerator devices (such as computing resources (e.g., processor cores), memory, interconnects, etc.).

[0134] Some example values ​​for the parameters in the graph section are as follows:

[0135] dev1: 30GB, Delta = 0;

[0136] dev2: 20GB, Delta = max(0, 10 seconds);

[0137] dev3: 28GB, Delta = 0.

[0138] In operation 1089 (part 2.3), the host (e.g., using a memory graph generator 1075) can generate a cluster graph 1090 with memory capacity and / or duration information. Cluster graph 1090 may include data groups (denoted as group 1, group 2, and group 3) distributed among virtual NPUs (denoted as VNPU1, VNPU2, and VNPU3). Figure 10A In the embodiments shown, VNPU1 (group 1) and VNPU2 (group 3) can be implemented on a first physical device (device 1), and VNPU3 (group 2) can be implemented on a second physical device (device 2). Figure 10A The memory usage and timing values ​​shown in cluster diagram 1090 can be displayed as follows: Figure 10B Implemented on the two physical devices shown.

[0139] refer to Figure 10B The NPU memory 1008-1 of device 1, which can be used as working memory for VNPU1 and VNPU2, can be implemented, for example, using the bandwidth-enhancing memory 408 shown in FIG. 4. The hierarchical memory 1010-1 of device 1 can be implemented, for example, using the latency-enhancing memory 410 shown in FIG. 4. Similarly, the NPU memory 1008-2 of device 2, which can be used as working memory for VNPU3, can be implemented, for example, using the bandwidth-enhancing memory 408 shown in FIG. 4, and the hierarchical memory 1010-2 of device 2 can be implemented, for example, using the latency-enhancing memory 410 shown in FIG. 4.

[0140] Refer again Figure 10BGroup 1 data can initially be loaded into NPU memory 1008-1 of device 1. As VNPU1 processes Group 1 data, it can gradually swap the data to hierarchical memory 1010-1, for example, based on priority, as shown in (1). Furthermore, as VNPU1 processes Group 1 data, it can transfer data (e.g., a 15GB block) to NPU memory 1008-2 of device 2 at time T=8, as shown in (2), where it can be processed by VNPU3 in Group 2. Similarly, after processing a 13GB block of data, VNPU3 can transfer the data to NPU memory 1008-1 of device 1, as shown in (4), where it can be processed by VNPU2 as part of Group 3. At intermediate times, VNPU2 at device 1 can begin processing Group 3 data, for example, once it has a minimum amount of data and / or memory to process, as shown in (3).

[0141] Figure 11 An embodiment of a full reduction operation method according to an example embodiment of the present disclosure is shown. Figure 11 The embodiments shown can be implemented, for example, using any host and / or device disclosed herein, including references to Figure 2 , Figure 3 Figure 4 and / or Figure 5 Those described.

[0142] refer to Figure 11 This method can be implemented using a host 1124, an interconnect controller (e.g., a CXL controller) 1160, and one or more devices 1106 having one or more accelerators. For illustrative purposes, Figure 11 The embodiments shown utilize eight accelerators 1106-1 to 1106-8; however, these principles can be applied to any number of devices. Accelerators 1106-1 to 1106-8 can be implemented using physical accelerators, virtual accelerators, or any combination thereof. In some embodiments, Figure 11 The embodiments shown can be partially or wholly utilized Figure 4B The full reduction engine 454 shown in the figure is used for implementation.

[0143] refer to Figure 11 This method can be performed as shown by the solid and dashed arrows, where different shades indicate different datasets, and the ⊕ operator indicates a full reduction operation.

[0144] Figure 12An embodiment of a method for processing data using a model according to an example embodiment of the present disclosure is shown. The method may begin at operation 1202. At operation 1204, the method may divide the model into a first part and a second part. For example, the method may divide the model based on one or more parameters of the model and / or one or more parameters of a first virtual accelerator and a second virtual accelerator. At operation 1206, the method may store the first part of the model in the memory of the device. At operation 1208, the method may store the second part of the model in the memory of the device. In some embodiments, the memory may be implemented as a hierarchical memory. At operation 1210, the model may be used by a first virtual accelerator at the device to perform a first operation using the first part of the model. At operation 1212, the method may be used by a second virtual accelerator at the device to perform a second operation using the second part of the model. The method may end at operation 1214.

[0145] Figure 12 The embodiments illustrated herein, as well as all other embodiments described herein, are example operations and / or components. In some embodiments, some operations and / or components may be omitted, and / or other operations and / or components may be included. Furthermore, in some embodiments, the temporal and / or spatial order of operations and / or components may vary. Although some components and / or operations may be shown as separate components, in some embodiments, some components and / or operations shown separately may be integrated into a single component and / or operation, and / or some components and / or operations shown as single components and / or operations may be implemented using multiple components and / or operations.

[0146] Any functionality described herein, including any host functionality, device functionality, etc., including references, is subject to these provisions. Figures 1 to 12 The described functions, such as accelerators, I / O engines, full reduction engines, compression and / or decompression logic, memory switches, memory managers, preprocessors, virtualization logic, memory access pattern analyzers, DCOH, etc., can be implemented in hardware, software, or any combination thereof, including combinational logic, sequential logic, one or more timers, counters, registers, state machines, CPLDs, FPGAs, ASICs, CPUs, GPUs, NPUs, TPUs, etc., including CISC processors such as x86 processors and / or RISC processors such as ARM processors. In some embodiments, one or more components can be implemented as a system-on-a-chip (SoC).

[0147] Some embodiments disclosed above have been described in the context of various implementation details, but the principles of this disclosure are not limited to these or any other specific details. For example, some functions have been described as being implemented by a particular component, but in other embodiments, functions may be distributed among different systems and components in different locations and having various user interfaces. Specific embodiments have been described as having specific processes, operations, etc., but these terms also cover embodiments in which specific processes, operations, etc. may be implemented by multiple processes, operations, etc., or multiple processes, operations, etc. may be integrated into a single process, step, etc. References to components or elements may refer to only a portion of a component or element. For example, a reference to a block may refer to the entire block or one or more sub-blocks. The use of terms such as “first” and “second” in this disclosure and claims may be merely for the purpose of distinguishing what they modify and may not indicate any spatial or temporal order unless it is obvious from the context. In some embodiments, a reference to something may refer to at least a portion of something; for example, “based on” may mean “at least partially based on”, etc. A reference to a first element may not imply the presence of a second element. The principles disclosed herein have independent utility and may be embodied independently, and not every principle can be utilized in every embodiment. However, this principle can also be embodied in various combinations, some of which can synergistically amplify the benefits of each principle.

[0148] Based on the inventive principles disclosed in this patent, the various details and embodiments described above can be combined to produce additional embodiments. Since the inventive principles disclosed in this patent can be modified in arrangement and detail without departing from the inventive concept, such changes and modifications are considered to fall within the scope of the appended claims.

Claims

1. An apparatus for processing data, comprising: Interconnection interface; Memory system, including: One or more first-type memory devices are coupled to the interconnect interface to receive first data and are configured to serve as working memory for an accelerator of the device for processing the data. One or more second-type memory devices are coupled to the interconnect interface to receive second data, and wherein the first type relates to high bandwidth and the second type relates to high capacity; The accelerator is coupled to the one or more first-type memory devices and the one or more second-type memory devices, and is configured to perform operations using the first data and the second data; and A memory switch is configured to configure one or more connections between the one or more first-type memory devices and the one or more second-type memory devices and the accelerator, wherein the memory switch is programmable to configure a specific amount of cache to the accelerator.

2. The device according to claim 1, wherein, The memory system also includes a cache configured to cache second data from the one or more second-type memory devices.

3. The device of claim 1, further comprising a consistency engine configured to maintain consistency of at least one of the one or more first type memory devices.

4. The device according to claim 1, wherein, The memory switch can also be programmed to configure the one or more first-type memory devices and the one or more second-type memory devices.

5. The device of claim 1, further comprising a memory manager configured to exchange the first data between the one or more first-type memory devices and the accelerator.

6. A method for processing data, the method comprising: The model is divided into a first part and a second part; The first part of the model is stored in the device's memory; The second part of the model is stored in the memory of the device; The first operation is performed by the first virtual accelerator at the device using the first part of the model; as well as The second virtual accelerator at the device performs the second operation using the second part of the model; The method further includes generating a first data set for the first virtual accelerator based on the first part of the model. The memory includes a first type of memory device and a second type of memory device, and the method further includes scheduling at least a portion of the first data group for the first type of memory device and the second type of memory device. The first type of memory device includes the working memory of the first virtual accelerator, and the first type relates to high bandwidth, while the second type relates to high capacity. The memory switch is configured to configure one or more connections between the first type of memory device and the second type of memory device and at least one of the first virtual accelerator and the second virtual accelerator, wherein the memory switch is programmable to configure a specific amount of cache to at least one of the first virtual accelerator and the second virtual accelerator.

7. The method according to claim 6, wherein, Partitioning the model includes partitioning the model at least in part by the host.

8. The method according to claim 6, wherein, The model is divided based on the following: One or more parameters of the first part of the model; and One or more parameters of the first virtual accelerator.

9. The method according to claim 6, wherein, The model is divided into the following categories: Associating the first portion of the model with the address space of the first process; and The second part of the model is associated with the address space of the second process.

10. The method according to claim 6, wherein, The memory switch can also be programmed to configure the first type of memory device and the second type of memory device.

11. The method of claim 6, further comprising generating a second data set for the second virtual accelerator based on the second portion of the model, wherein, The memory includes: A first type of first memory device associated with the first virtual accelerator; A first memory device of a second type associated with the first virtual accelerator; and A second memory device of a first type associated with the second virtual accelerator.

12. The method of claim 11, further comprising: Schedule at least a portion of the first data group for the first memory device of the first type and the first memory device of the second type; as well as Schedule at least a portion of the second data group for the first memory device of the first type and the second memory device of the first type.

Images