Heterogeneous ML Accelerator Cluster with Flexible System Resource Balancing

Abstract

Aspects of the present disclosure are directed to heterogeneous machine learning accelerator systems with compute nodes and memory nodes connected by high-speed chip-to-chip interconnects. While existing remote / non-aggregated memories may require memory expansion via a remote processing unit, aspects of the present disclosure add memory nodes to the machine learning accelerator cluster via chip-to-chip interconnects without requiring assistance from the remote processing unit to achieve higher performance, simpler software stacks and / or lower costs. Memory nodes may support pre-fetching and intelligent compression, allowing the use of lower-cost memory without performance degradation.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Patent Application No. 18 / 152,428, filed January 10, 2023, the disclosure of which is incorporated herein by reference. Summary of the Invention [Problem to be solved by the invention]

[0002] background Training a state-of-the-art machine learning model can require hundreds to tens of thousands of modern machine learning model accelerators, which can be connected with high-speed chip-to-chip interconnects to form a system that performs the required total computation and storage. For example, thousands of tensor processing unit (TPU) chips can be connected by inter-chip interconnects (ICIs) to form a TPU pod. Because each TPU accelerator has fixed compute and memory resources, the overall system balance between aggregate compute and memory resources is also fixed. This leads to a suboptimal situation because different machine learning models may require different resource balances with respect to compute and memory. In a system with a fixed resource ratio between compute and memory, some models may be stuck with memory capacity, while other models may be stuck with compute resources. [Effects of the Invention]

[0003] Quick Overview Aspects of the present disclosure are directed to heterogeneous machine learning accelerator systems with compute nodes and memory nodes connected by high-speed chip-to-chip interconnects. While existing remote / disaggregated memories may require memory expansion via a remote processing unit, aspects of the present disclosure add memory nodes to a machine learning accelerator cluster via the chip-to-chip interconnect without requiring assistance from the remote processing unit, achieving higher performance, a simpler software stack, and / or lower cost. Memory nodes may support prefetching and intelligent compression, allowing the use of lower-cost memory without performance degradation. [Means for solving the problem]

[0004] One aspect of the disclosure provides a machine learning accelerator system that includes one or more accelerator clusters including a plurality of heterogeneously distributed memory nodes and a plurality of compute nodes connected by a high-speed chip-to-chip interconnect, and one or more processors configured to prefetch machine learning models or data from slower memory nodes of the plurality of heterogeneously distributed memory nodes via the high-speed interconnect to on-chip memory in the compute nodes.

[0005] In one example, the high-speed interconnect includes an inter-chip interconnect (ICI) having multiple off-chip links and multiple on-chip links. In another example, the multiple memory nodes are of various types including high-bandwidth memory (HBM) and at least one of double data rate synchronous dynamic random access memory (DDR) or low-power DDR (LPDDR). In yet another example, the multiple compute nodes are of various types including at least two of tensor processing units (TPUs), graphics processing units (GPUs), or central processing units (CPUs).

[0006] In yet another example, the one or more processors are further configured to determine that a memory node of the plurality of memory nodes constitutes a slow memory node based on a latency threshold. In yet another example, the one or more processors are further configured to prefetch machine learning models or data from a remote memory node of the plurality of memory nodes to an on-chip memory at the compute node via a high-speed interconnect. In yet another example, the one or more processors are further configured to determine that a memory node of the plurality of memory nodes constitutes a remote memory node based on a location of the memory node within one or more accelerator clusters.

[0007] In yet another example, the one or more processors are further configured to swap the machine learning model or data between memory nodes of the plurality of memory nodes based on a hotness of the machine learning model or data. In yet another example, the one or more processors are further configured to determine a hotness of the machine learning model or data based on at least one of recency of use or frequency of use.

[0008] In yet another example, the one or more processors are further configured to keep the machine learning model or portion of the data uncompressed in a fast memory node of the plurality of memory nodes. In yet another example, the one or more processors are further configured to determine that additional machine learning models or data are needed to perform the calculation and to decompress the additional machine learning models or data from a slower memory node of the plurality of memory nodes. In yet another example, keeping the machine learning model or portion of the data uncompressed is based on at least one of recency or frequency of use.

[0009] Another aspect of the present disclosure provides a method for a machine learning accelerator system. The machine learning accelerator system includes one or more accelerator clusters including a plurality of unevenly distributed memory nodes and a plurality of compute nodes connected by a high-speed chip-to-chip interconnect. The method includes one or more processors determining, based on a latency threshold, that a memory node of the plurality of memory nodes constitutes a slow memory node, and the one or more processors prefetching, via the high-speed interconnect, a machine learning model or data from the slow memory node to an on-chip memory in the compute node.

[0010] In one example, the method further includes one or more processors determining, based on a location of the memory node in the one or more accelerator clusters, that a memory node of the plurality of memory nodes constitutes a remote memory node; and one or more processors prefetching, via a high-speed interconnect, the machine learning model or data from the remote memory node to on-chip memory at the compute node.

[0011] In another example, the method further includes one or more processors determining a hotness of the machine learning model or data based on at least one of recent use or frequency of use, and one or more processors swapping the machine learning model or data between memory nodes of the plurality of memory nodes based on the hotness of the machine learning model or data.

[0012] In yet another example, the method further includes one or more processors keeping a portion of the machine learning model or data uncompressed in a fast memory node of the plurality of memory nodes based on at least one of recent use or frequency of use, the one or more processors determining that additional machine learning models or data are needed to perform the calculation, and the one or more processors decompressing the additional machine learning model or data from a slower memory node of the plurality of memory nodes.

[0013]

[0009] Yet another aspect of the present disclosure provides a non-transitory computer-readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for a machine learning accelerator system. The machine learning accelerator system includes one or more accelerator clusters including a plurality of unevenly distributed memory nodes and a plurality of compute nodes connected by a high-speed chip-to-chip interconnect. The operations include determining, based on a latency threshold, that a memory node of the plurality of memory nodes constitutes a slow memory node, and prefetching, via the high-speed interconnect, a machine learning model or data from the slow memory node to an on-chip memory in the compute node.

[0014] In one example, the operations further include determining that a memory node of the plurality of memory nodes constitutes a remote memory node based on a location of the memory node within one or more accelerator clusters, and prefetching the machine learning model or data from the remote memory node to on-chip memory at the compute node via a high-speed interconnect.

[0015] In another example, the operations further include determining a hotness of the machine learning model or the data based on at least one of recent use or frequency of use, and swapping the machine learning model or the data between memory nodes of the plurality of memory nodes based on the hotness of the machine learning model or the data.

[0016] In yet another example, the operations further include keeping a portion of the machine learning model or data uncompressed in a fast memory node of the plurality of memory nodes based on at least one of recent use or frequency of use, determining that additional machine learning model or data is needed to perform the calculation, and decompressing the additional machine learning model or data from a slower memory node of the plurality of memory nodes. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a block diagram of an exemplary heterogeneous machine learning accelerator system according to aspects of the present disclosure. [Figure 2] FIG. 1 is a block diagram of an exemplary environment for implementing a heterogeneous accelerator system according to aspects of the present disclosure. [Figure 3] FIG. 1 is a block diagram of an exemplary accelerator cluster according to aspects of the present disclosure. [Figure 4] FIG. 1 is a block diagram of an exemplary accelerator cluster with additional compute nodes and additional memory nodes according to aspects of the present disclosure. [Figure 5] FIG. 10 is a flow diagram of an exemplary process for active prefetching according to aspects of the present disclosure. [Figure 6] FIG. 10 is a flow diagram of an exemplary process for memory swapping according to aspects of the present disclosure. [Figure 7] FIG. 1 is a flow diagram of an exemplary process for intelligent memory compression according to aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0018] Detailed Description Generally disclosed herein is an implementation in a heterogeneous machine learning accelerator system involving compute nodes and memory nodes connected by a high-speed chip-to-chip interconnect. The memory nodes support intelligent compression, allowing the use of low-cost memory without performance degradation. The high-speed interconnected memory nodes are distinct from remote / non-aggregated memory. In this regard, the high-speed interconnected memory nodes may be accessed directly by a processing unit host, while the remote / non-aggregated memory is accessed by the processing unit host through a remote processing unit system stack. Accessing the remote / non-aggregated memory through a remote processing unit system stack may incur delays, which may degrade the performance of the remote / non-aggregated memory compared to the interconnected memory nodes. Furthermore, supporting such access to the remote / non-aggregated memory may leave expensive processing unit resources behind in the accelerator system.

[0019] The accelerator system can include one or more accelerator clusters, each having multiple memory nodes and multiple compute nodes connected by the same high-speed inter-chip interconnect, such as an inter-chip interconnect (ICI). The high-speed interconnect can include multiple off-chip links and multiple on-chip links connected with an on-chip router. The off-chip and on-chip links can have high data throughput, allowing large amounts of data to be transmitted quickly both on-chip and off-chip. The high-speed interconnect can include an optical interconnect or high-speed Ethernet.

[0020] The memory nodes can include the same or different memory types in the accelerator cluster, including high-bandwidth memory (HBM), double-data-rate synchronous dynamic random-access memory (DDR), low-power DDR (LPDDR), dynamic random-access memory (DRAM), flash memory, or non-volatile RAM. The compute nodes can also include the same or different compute types, including tensor processing units (TPUs), graphics processing units (GPUs), and central processing units (CPUs).

[0021] Compute nodes can include different hardware with different computational capabilities. For example, some compute nodes can have different supporting hardware, such as some compute nodes being designed or optimized for one function, such as video / image processing, while other compute nodes in the same cluster can be designed or optimized for a different function, such as natural language processing. Additionally, compute nodes can be software-managed or hardware-managed, and memory nodes can have different power efficiencies and memory persistence depending on the type of memory included. For example, compute node power efficiency can vary between LPDDR, DDR, and HBM, and memory persistence can vary between DRAM, flash memory, or non-volatile RAM.

[0022] The compute-to-memory resource balance in an accelerator cluster can be changed by installing or pre-installing and activating additional memory and / or compute nodes. The additional memory and / or compute nodes can be uniformly or unevenly distributed throughout the accelerator cluster. Uniformly distributed memory nodes can correspond to additional memory nodes that are the same, such as having the same capacity and / or latency, while unevenly distributed memory nodes can correspond to additional memory nodes that vary, such as having different capacity and / or latency.

[0023] Uniformly or non-uniformly distributed memory nodes can be collocated in adjacent memory racks and connected to accelerator clusters via high-speed interconnects. Memory racks can be physically or logically partitioned to accommodate distant / disjoint memories that can serve as memory pools for various accelerator clusters.

[0024] Multiple memory nodes can implement a memory hierarchy based on unevenly distributed memory nodes with different memory latencies. To account for the memory hierarchy, software in the accelerator cluster can utilize active prefetching and / or memory swapping. Compute nodes can include high-performance on-chip memory such as CMEM.

[0025] Software in the accelerator cluster can prefetch machine learning models and / or data from remote and / or slow memory nodes to on-chip memory in the compute nodes. The software can determine which memory nodes can constitute slow memory nodes based on a configurable latency threshold, which can be based on configurable performance and / or efficiency requirements. For example, a memory node can be considered slow if its access latency is longer than the high-bandwidth memory access latency. The software can also determine which memory nodes can constitute remote memory nodes based on whether the memory node is part of the accelerator cluster. The machine learning models can include tensors and / or layers, and the data can include training data and / or validation data. For example, DDR and / or LPDDR memory nodes can have their machine learning models and / or data prefetched, while HBM memory nodes do not need their machine learning models and / or data prefetched. Prefetching ensures that working sets of machine learning models and data are always accessible from on-chip memory when computations are performed.

[0026] Software in the accelerator cluster may also swap machine learning models and / or data between memory nodes based on the hotness of the machine learning models and / or data. Hotness may refer to the recent or frequent use of the machine learning models and / or data. For example, machine learning models and / or data that are likely to be used recently or more frequently may be placed on closer or faster memory nodes, such as HBM, while machine learning models and / or data that are likely to be used less recently or rarely may be placed on more distant / further away memory nodes, such as DDR, or slower memory nodes. The recent and frequent use thresholds may be configurable depending on performance and / or efficiency requirements.

[0027] The use of low-cost, high-density memory, such as DDR and LPDDR, allows for scaling of the total memory capacity of a machine learning accelerator cluster and dynamic balancing of compute and memory resources. Additionally, adding intelligent memory compression to remote memory nodes can further scale memory capacity, allowing software to prefetch machine learning models and / or data from remote and / or slower memory nodes. Memory nodes can include hardware compression engines that can perform intelligent memory compression and prefetching instead of compressing / decompressing on fetch.

[0028] The software can keep portions of the machine learning models and / or data uncompressed in fast memory nodes, such as HBM, and decompress additional models and / or data as needed. The software can implement cache replacement policies, such as a least recently used (LRU) or most frequently used (MFU) algorithm, to keep portions of the machine learning models and / or data uncompressed in fast memory nodes. The software can also implement more advanced cache replacement policies, such as an algorithm that combines LRU and MFU, to determine which portions of the machine learning models and / or data should be kept uncompressed. For example, the software can prefetch or decompress based on input data in a machine learning model, and fetch or decompress additional models / data as the input data is processed.

[0029] 1 shows a block diagram of an exemplary heterogeneous machine learning accelerator system 100. The heterogeneous accelerator system 100 can be configured to receive input data, such as through a user interface. For example, the heterogeneous accelerator system 100 can receive input data as part of a call to an API that exposes the heterogeneous accelerator system 100. The heterogeneous accelerator system 100 can be implemented on one or more computing devices. Input to the heterogeneous accelerator system 100 can be provided, for example, via a storage medium, including remote storage, connected to one or more computing devices via a network, or as input via a user interface on a client computing device coupled to the heterogeneous accelerator system 100.

[0030] The heterogeneous accelerator system 100 may be configured to receive training data 102 for training a machine learning model and target computing resource data 104 that specifies a target computing resource. The training data 102 may correspond to a machine learning task, such as a neural network task performed by a neural network. The training data 102 may be split into a training set, a validation set, and / or a test set. An exemplary training / test split may be an 80 / 20 split. The machine learning model may be configured to receive any type of input data and generate output data 106 for performing the machine learning task. By way of example, the output data 106 may be any type of score, classification, or regression output based on the input data. Correspondingly, the machine learning task may be a scoring, classification, and / or regression task for predicting some output given some input. These machine learning tasks may correspond to various applications that process image, video, text, audio, or other types of data.

[0031] The training data 102 can be in any format suitable for training a machine learning model according to one of a variety of different learning techniques. Learning techniques for training a machine learning model can include supervised learning, unsupervised learning, and semi-supervised learning techniques. For example, the training data 102 can include multiple training examples that can be received as input by the machine learning model. The training examples can be labeled with a desired output for the machine learning model when processing the labeled training examples. The labels and model output can be evaluated through a loss function to determine an error, which can be backpropagated through the machine learning model to update weights in the machine learning model. For example, if the machine learning task is a classification task, the training examples can be images labeled with one or more classes that classify the objects depicted in the images. As another example, a supervised learning technique can be applied to calculate the error between outputs using ground truth labels of training samples processed by the machine learning model. Various loss or error functions appropriate to the type of task for which the machine learning model is being trained can be used, such as cross-entropy loss for classification tasks or mean squared error for regression tasks. The gradient of the error with respect to the various weights of the candidate model on the candidate hardware can be calculated, for example, using a backpropagation algorithm, and the model weights can be updated. The machine learning model can be trained until a stopping criterion is met, such as a number of training iterations, a maximum duration, convergence, or when a minimum accuracy threshold is met.

[0032] The target computing resource data 104 may specify characteristics of a computing resource on which the neural network may be at least partially deployed. The computing resources may be housed in one or more data centers or other physical locations that host any of various types of hardware devices. Examples of types of hardware include central processing units (CPUs), graphics processing units (GPUs), edge or mobile computing devices, field programmable gate arrays (FPGAs), and various types of application-specific Accumulation Examples include circuits (ASICs).

[0033] Some devices can be configured for hardware acceleration, which can include devices configured to efficiently perform specific types of operations. These hardware accelerators, which can include, for example, GPUs and tensor processing units (TPUs), can implement specialized functionality for hardware acceleration. An example of hardware acceleration functionality is performing operations commonly associated with running machine learning models, such as matrix multiplication. These specialized functionality can include, for example, matrix multiply-accumulate units available in various types of GPUs and matrix multiplication units available in TPUs.

[0034] The target computing resource data 104 may include data for one or more target sets of computing resources. The target set of computing resources may refer to a group of computing devices on which it is desired to deploy a machine learning model. The information specifying the target set of computing resources may refer to the type and / or quantity of hardware accelerators or other computing devices in the target set. The target sets may include devices of the same or different types. For example, the target set of computing resources may specify the hardware characteristics and quantity of a particular type of hardware accelerator, such as its processing power, throughput, memory capacity, etc. As described herein, the heterogeneous accelerator system 100 may generate one or more machine learning model architectures for each device specified in the target set of computing resources. In some examples, the target set of computing resources may specify computing resources in devices with lower overall computing power than devices in a data center, such as mobile phones or wearable devices (e.g., headphones, earphones, or smartwatches) on which the machine learning model will be deployed. Furthermore, the target computing resource data 104 may specify different target sets of computing resources, for example, reflecting different potential configurations of computing resources housed in a data center.

[0035] From the training data 102 and the target computing resource data 104, the heterogeneous accelerator system 100 can be configured to output one or more results of the generated machine learning task as output data 106. The output data 106 can be transmitted for display on a user display, as an example. In some implementations, the heterogeneous accelerator system 100 can be configured to provide the output data 106 as a set of computer-readable instructions, such as one or more computer programs. The computer programs can be written in any type of programming language and according to any programming paradigm, such as declarative, procedural, assembly, object-oriented, data-oriented, functional, or imperative. The computer programs can perform one or more different functions and can be written to run, for example, on a physical device, on a virtual machine, or within a computing environment spanning multiple devices. The computer programs can also implement the functions described herein, for example, to be executed by a system, engine, module, or model.

[0036] The heterogeneous accelerator system 100 can be configured to forward the output data 106 to one or more other devices configured to convert the output data 106 into an executable program written in a computer programming language, and optionally as part of a framework for generating a machine learning model. The heterogeneous accelerator system 100 can also be configured to send the output data 106 to a storage device for storage and subsequent retrieval.

[0037] The heterogeneous accelerator system 100 may include an accelerator cluster engine 108. The accelerator cluster engine 108 may be implemented as one or more computer programs, specially configured electronic circuits, or any combination thereof. The accelerator cluster engine 108 may be configured to generate one or more accelerator clusters, each having multiple memory nodes and multiple compute nodes connected by high-speed chip-to-chip interconnects. The memory nodes may include the same or different types of memory, and the compute nodes may include the same or different types of processing units. The accelerator cluster engine 108 may generate the accelerator cluster based on the target computing resource data 104. Generating the accelerator cluster may include determining whether to activate additional memory nodes and / or compute nodes and whether to distribute the additional memory nodes and / or compute nodes uniformly or non-uniformly throughout the accelerator cluster based on the target computing resource data 104.

[0038] The heterogeneous accelerator system 100 may further include a prefetch / memory swap engine 110. The prefetch / memory swap engine 110 may be implemented as one or more computer programs, specially configured electronic circuits, or any combination thereof. The prefetch / memory swap engine 110 may be configured to prefetch input data, such as machine learning models and / or training data 102, from remote and / or slower memory nodes to on-chip memory in the computational node so that a working set of the machine learning models and / or input data is ready to access when computation is performed. The prefetch / memory swap engine 110 may determine which memory nodes constitute slow memory nodes based on a configurable latency threshold that may be defined by performance and / or efficiency requirements of the target computing resource data 104. The prefetch / memory swap engine 110 may determine which memory nodes constitute remote memory nodes based on their location within or outside the accelerator cluster.

[0039] The prefetch / memory swap engine 110 may also be configured to swap the machine learning models and / or input data between memory nodes based on the hotness of the machine learning models and / or input data. The prefetch / memory swap engine 110 may determine hotness based on configurable thresholds for recent and / or frequent use, which may be dictated by performance and / or efficiency requirements of the target computing resource data 104.

[0040] The heterogeneous accelerator system 100 may further include an intelligent compression engine 112. The intelligent compression engine 112 may be implemented as one or more computer programs, specially configured electronic circuits, or any combination thereof. The intelligent compression engine 112 may be configured to keep a portion of the machine learning models and / or input data uncompressed in a high-speed memory node and to decompress additional machine learning models and / or input data as needed. The intelligent compression engine 112 may implement a cache replacement policy, such as a least recently used (LRU) algorithm, a most frequently used (MFU) algorithm, or a combination thereof, to determine which portions of the machine learning models and / or input data should be kept uncompressed.

[0041] 2 shows a block diagram of an exemplary environment 200 for implementing a heterogeneous accelerator system. The system 200 can be implemented on one or more devices having one or more processors in one or more locations, such as a server computing device 202. The client computing device 204 and the server computing device 202 can be communicatively coupled to one or more storage devices 206 via a network 208. The storage device 206 can be a combination of volatile and non-volatile memory and can be in the same or a different physical location as the computing devices 202, 204. For example, the storage device 206 can include any type of non-transitory computer-readable medium capable of storing information, such as a hard drive, a solid-state drive, a tape drive, optical storage, a memory card, ROM, RAM, DVD, CD-ROM, writable memory, and read-only memory.

[0042] The server computing device 202 may include one or more processors 210 and memory 212. The memory 212 may store information accessible by the processor 210, including instructions 214 that may be executed by the processor 210. The memory 212 may also include data 216 that may be retrieved, manipulated, or stored by the processor 210. The memory 212 may be a type of non-transitory computer-readable medium that may store information accessible by the processor 210, such as volatile and non-volatile memory. The processor 210 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and / or application specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0043] The instructions 214 may include one or more instructions that, when executed by the processor 210, cause one or more processors to perform the actions specified by the instructions 214. The instructions 214 may be stored in object code format for direct processing by the processor 210, or may be stored in other formats, including an interpretable script or a collection of independent source code modules that are interpreted on demand or pre-compiled. The instructions 214 may include instructions for implementing a heterogeneous accelerator system 218, which may correspond to the heterogeneous accelerator system 100 of FIG. 1. The heterogeneous accelerator system 218 may be executed using the processor 210 and / or using other processors located remotely from the server computing device 202.

[0044] The data 216 may be retrieved, stored, or modified by the processor 210 in accordance with the instructions 214. The data 216 may be stored in a computer register, a relational or non-relational database, as a table with multiple different fields and records, or as a JSON, YAML, proto, or XML document. The data 216 may also be formatted in a computer-readable format, such as, but not limited to, binary values, ASCII, or Unicode. Furthermore, the data 216 may include sufficient information to identify related information, such as numbers, descriptive text, unique codes, pointers, references to data stored in other memory, including other network locations, or information used by a function to calculate the related data.

[0045] The client computing device 204 may be configured similarly to the server computing device 202, with one or more processors 220, memory 222, instructions 224, and data 226. The client computing device 204 may also include a user input 228 and a user output 230. The user input 228 may include any suitable mechanism or technology for receiving input from a user, such as a keyboard, a mouse, a mechanical actuator, a soft actuator, a touchscreen, a microphone, and a sensor.

[0046] The server computing device 202 may be configured to transmit data to the client computing device 204, which may be configured to display at least a portion of the received data on a display implemented as part of the user output 230. The user output 230 may also be used to display an interface between the client computing device 204 and the server computing device 202. The user output 230 may alternatively or additionally include one or more speakers, transducers or other audio outputs, a haptic interface or other tactile feedback that provides non-visual and non-auditory information to a platform user of the client computing device 204.

[0047] While Figure 2 depicts the processors 210, 220 and memories 212, 222 as being within the computing devices 202, 204, the components described herein may include multiple processors and memories that may operate in different physical locations rather than within the same computing device. For example, some of the instructions 214, 224 and data 216, 226 may be stored on a removable SD card, while other instructions may be stored in a read-only computer chip. Some or all of the instructions and data may be stored in a location physically separate from but still accessible by the processors 210, 220. Similarly, the processors 210, 220 may include a group of processors capable of performing simultaneous and / or sequential operations. The computing devices 202, 204 each include one or more internal clocks that provide timing information that can be used to time operations and programs executed by the computing devices 202, 204.

[0048] The server computing device 202 can be connected via a network 208 to a data center 232 that houses any number of hardware accelerators 232A-N. The data center 232 can be one of multiple data centers or other facilities in which various types of computing devices, such as hardware accelerators, are located. The computing resources housed in the data center 232 can be designated for deploying a neural network architecture on the hardware architecture, as described herein.

[0049] The server computing device 202 can be configured to receive requests to process data from the client computing devices 204 on computing resources in the data center 232. For example, the environment 200 can be part of a computing platform configured to provide various services to users through various user interfaces and / or application programming interfaces (APIs) that expose platform services. One or more services can be a machine learning framework or set of tools for generating a neural network or other machine learning model according to a specified task and training data. The client computing device 204 can send and receive data specifying target computing resources to be assigned to execute the machine learning model trained to perform a particular machine learning task. The heterogeneous accelerator system 218 can receive data specifying the target computing resources and / or training data and, in response, generate output data for deployment on the target computing resources.

[0050] As another example of a potential service provided by a platform implementing environment 200, server computing device 202 may maintain different machine learning models according to different potential target computing resources available at data center 232. For example, server computing device 202 may maintain different families for deploying neural networks on different types of TPUs and / or GPUs housed at or available for processing at data center 232.

[0051] The devices 202, 204 and the data center 232 can communicate directly and indirectly over the network 208. For example, using network sockets, the client computing device 204 can connect to services running on the data center 232 through the Internet Protocol. The devices 202, 204 can set up listening sockets that can accept initiating connections to send and receive information. The network 208 itself can include a variety of configurations and protocols, including the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local networks, and private networks that use communication protocols proprietary to one or more companies. The network 208 can support a variety of short- and long-range connections. Short- and long-range connections can occur over different bandwidths, such as 2.402 GHz to 2.480 GHz, commonly associated with the Bluetooth® standard, and 2.4 GHz and 5 GHz, commonly associated with the Wi-Fi® communication protocol, or associated with various communication standards, such as the LTE® standard for wireless broadband communications. The network 208 may additionally or alternatively support wired connections between the devices 202, 204 and the data center 232, including various types of Ethernet connections.

[0052] 2 depicts a single server computing device 202, client computing device 204, and data center 232, it is understood that aspects of the disclosure can be implemented according to a variety of different configurations and quantities of computing devices, including in serial or parallel processing paradigms, or via a distributed network of multiple devices. In some implementations, aspects of the disclosure can be performed on a single device connected to a hardware accelerator configured to process machine learning models and any combination thereof.

[0053] 3 shows a block diagram of an exemplary accelerator cluster 300. The accelerator cluster 300 includes multiple memory nodes and multiple compute nodes. Although two memory nodes and two compute nodes are shown in FIG. 3, the accelerator cluster 300 may include any number of memory nodes and compute nodes. As shown, the multiple memory nodes include a first memory node 302 and a second memory node 304, and the multiple compute nodes include a first compute node 306 and a second compute node 308.

[0054] The memory nodes may include the same or different memory types, including high bandwidth memory (HBM), double data rate synchronous dynamic random access memory (DDR), low power DDR (LPDDR), dynamic random access memory (DRAM), flash memory, or non-volatile RAM, within accelerator cluster 300. For example, memory node 302 may support HBM, while memory node 304 may support LPDDR.

[0055] The compute nodes can include the same or different compute types, such as tensor processing units (TPUs), graphics processing units (GPUs), and central processing units (CPUs). The compute nodes can also include different hardware with different computing capabilities or different supporting hardware. For example, compute node 306 can be designed or optimized for one function, such as video / image processing, while compute node 308 can be designed or optimized for another function, such as natural language processing. The compute nodes can be software-managed or hardware-managed, and the power efficiency and memory persistence can vary depending on the type of memory included in the memory nodes. For example, the power efficiency of a compute node can vary between LPDDR, DDR, and HBM, and the memory persistence can vary between DRAM, flash memory, or non-volatile RAM.

[0056] The memory nodes and compute nodes are connected by a high-speed inter-chip interconnect 310, such as an inter-chip interconnect (ICI). The high-speed interconnect 310 may include multiple off-chip links 312 and multiple on-chip links 314 connected via an interconnect router 316. The off-chip links 312 may connect to an interconnect router on a separate chip (not shown) to connect the compute nodes and / or memory nodes to other compute nodes and / or memory nodes on another chip. The on-chip links 314 may connect the compute nodes and / or memory nodes on the same chip via the interconnect router 316. Although four off-chip links 312 and two on-chip links 314 are shown in FIG. 3, the accelerator cluster 300 may include any number of off-chip links 312 and on-chip links 314. The off-chip links 312 and on-chip links 314 may have high data throughput, allowing large amounts of data to be sent quickly both on-chip and off-chip. For example, the off-chip link 312 may be capable of 500 Gbit / s or more or less, and the on-chip link may be capable of 1000 Gbit / s or more or less. The high-speed interconnect 310 may include an optical interconnect or high-speed Ethernet.

[0057] High-speed interconnected memory nodes and compute nodes differ from remote / non-aggregated memory. In this regard, high-speed interconnected memory nodes can be directly accessed by compute nodes, while remote / non-aggregated memory must be accessed by the processing unit host through the remote processing unit system stack. Accessing remote / non-aggregated memory through the remote processing unit system stack can introduce latency, which can degrade the performance of remote / non-aggregated memory compared to interconnected memory nodes. Furthermore, expensive processing unit resources can be left behind from the accelerator cluster when facilitating access to such remote / non-aggregated memory.

[0058] Based on the target computing resource data, additional memory nodes and / or compute nodes can be installed or pre-installed and activated to change the computing and memory resource balance on the accelerator cluster. FIG. 4 shows a block diagram of an exemplary accelerator cluster 400 with an additional compute node and an additional memory node. Similar to the accelerator cluster shown in FIG. 3, the accelerator cluster 400 includes multiple memory nodes and multiple compute nodes. The multiple memory nodes include a first memory node 402 and a second memory node 404, and the multiple compute nodes include a first compute node 406 and a second compute node 408. For the accelerator cluster 400, based on the target computing resource, the multiple compute nodes include an additional memory node 410 and an additional compute node 412. The memory nodes and compute nodes are connected by a high-speed inter-chip interconnect 414, such as an inter-chip interconnect (ICI). Although FIG. 4 shows one additional memory node 410 and one additional compute node 412, the accelerator cluster 400 can include any number of additional memory nodes or compute nodes.

[0059] The additional memory nodes and / or compute nodes may be uniformly or unevenly distributed throughout the accelerator cluster 400. Uniformly distributed memory nodes may correspond to additional memory nodes that are the same, e.g., have the same capacity and / or latency, while unevenly distributed memory nodes may correspond to additional memory nodes that are different, e.g., have different capacity and / or latency. For example, unevenly distributed memory nodes may include one memory node corresponding to HBM and another memory node corresponding to LPDDR. Similarly, uniformly distributed compute nodes may correspond to additional compute nodes that are the same, e.g., have the same processing power, while unevenly distributed compute nodes may correspond to additional compute nodes that are different, e.g., have different processing power. For example, unevenly distributed compute nodes may include one compute node corresponding to a TPU and another compute node corresponding to a CPU.

[0060] Multiple memory nodes can introduce a memory hierarchy based on unevenly distributed memory nodes with different memory latencies. To take this into account, software in the accelerator cluster can utilize active prefetching and memory swapping.

[0061] 5 shows a flow diagram of an example process 500 for actively prefetching machine learning models and / or input data from a remote and / or slow memory node to on-chip memory at a compute node. The example process 500 can be executed on a system of one or more processors in one or more locations, such as the heterogeneous accelerator system 100 of FIG.

[0062] As shown in block 510, the accelerator cluster engine 108 may maintain the unevenly distributed memory nodes and / or the unevenly distributed compute nodes as one or more accelerator clusters.

[0063] As shown in block 520, the prefetch / memory swap engine 110 may determine which memory nodes constitute slow memory nodes based on a configurable latency threshold. The latency threshold may depend on configurable performance and / or efficiency requirements, such as from the target computing resource data 104. For example, if the access latency of a memory node is longer than a predetermined high-bandwidth memory access latency, the prefetch / memory swap engine 110 may determine that the memory node is slow. The prefetch / memory swap engine 110 may also determine which memory nodes constitute slow memory nodes based on the type of memory node. For example, DDR and / or LPDDR memory nodes may constitute slow memory nodes, while HBM may constitute fast memory nodes.

[0064] As shown in block 530, the prefetch / memory swap engine 110 may determine which memory nodes constitute remote memory nodes based on the location of the memory nodes within or outside one or more accelerator clusters. For example, a memory node within an accelerator cluster that includes one or more compute nodes utilized in a computation does not constitute a remote memory node, but a memory node outside that accelerator cluster does constitute a remote memory node.

[0065] As shown in block 540, the prefetch / memory swap engine 110 can prefetch the machine learning model and / or input data from slower and / or more distant memory nodes, eliminating the need to prefetch the machine learning model and / or input data from faster and / or closer memory nodes. Prefetching allows a working set of the machine learning model and / or input data to be readily accessible from on-chip memory in the compute node when computation is performed.

[0066] 6 shows a flow diagram of an example process 600 for swapping machine learning models and / or input data between memory nodes based on the hotness of the machine learning models and / or input data. The example process 600 can be executed on a system of one or more processors in one or more locations, such as the heterogeneous accelerator system 100 of FIG.

[0067] As shown in block 610, the accelerator cluster engine 108 may maintain the unevenly distributed memory nodes and / or the unevenly distributed compute nodes as one or more accelerator clusters.

[0068] As shown in block 620, the prefetch / memory swap engine 110 may determine which memory nodes contain recently used machine learning models and / or input data based on a recently used threshold. The recently used threshold may depend on configurable performance and / or efficiency requirements, such as from the target computing resource data 104.

[0069] As shown in block 630, the prefetch / memory swap engine 110 may determine which memory nodes contain frequently used machine learning models and / or input data based on a frequency threshold, which may depend on configurable performance and / or efficiency requirements, such as from the target computing resource data 104.

[0070] As shown in block 640, the prefetch / memory swap engine 110 can swap recently and / or frequently used machine learning models and / or input data from slower and / or more distant memory nodes to faster and / or closer memory nodes. The prefetch / memory swap engine 110 can also swap less recently and / or rarely used machine learning models and / or input data from faster and / or closer memory nodes to slower and / or more distant memory nodes. For example, very recently and / or frequently used machine learning models and / or data can be placed in the HBM, while less recently and / or infrequently used machine learning models and / or data can be placed in the DDR.

[0071] The total memory capacity of a machine learning accelerator cluster can be expanded by using low-cost, high-density memory and prefetching or swapping from that memory as needed. Memory capacity can be further expanded by adding intelligent memory compression to distant memory nodes. Software can perform intelligent memory compression and prefetching instead of compressing / decompressing on fetch.

[0072] 7 shows a flow diagram of an example process 700 for intelligent memory compression. The example process 700 can be executed on a system of one or more processors in one or more locations, such as the heterogeneous accelerator system 100 of FIG.

[0073] As shown in block 710, the accelerator cluster engine 108 may maintain the unevenly distributed memory nodes and / or the unevenly distributed compute nodes as one or more accelerator clusters.

[0074] As shown in block 720, the intelligent compression engine 112 can keep portions of the machine learning model and / or input data uncompressed in faster and / or closer memory nodes. For example, the intelligent compression engine 112 can keep portions of the machine learning model and / or input data uncompressed in the HBM. The intelligent compression engine 112 can implement a cache replacement policy, such as a least recently used (LRU) algorithm or a most frequently used (MFU) algorithm, to keep portions of the machine learning model and / or input data uncompressed in faster and / or closer memory nodes, such as the HBM. The intelligent compression engine 112 can also implement more advanced cache replacement policies, such as an algorithm that is a combination of LRU and MFU, to determine which portions of the machine learning model and / or input data should be kept uncompressed.

[0075] As shown in block 730, the intelligent compression engine 112 can decompress additional machine learning models and / or input data if needed for further computation. For example, for a Mixture of Experts (MoE) model, if K experts from the MoE model are stored in the HBM with compression, n*K experts can be stored in the HBM, where n is the compression ratio. Then, during training or inference, the intelligent compression engine 112 can keep a subset of all experts from the MoE model uncompressed and resident in the HBM for normal training or inference. If model training or inference wants to access an expert but finds that the expert is not present in the HBM, the model training or inference misses the expert, and the intelligent compression engine 112 can fetch it from remote memory and / or decompress a new expert. A memory access penalty based on waiting for memory contents to become available can be included when decompressing experts, as well as fetching them from remote memory. Software can implement an LRU or MFU algorithm to keep a subset of experts uncompressed in the HBM. If the MoE model has individual experts processing different input data, the software can also run more advanced algorithms based on the input data, such as a combination with LRU and / or MFU.

[0076] As described herein, aspects of the present disclosure provide a heterogeneous machine learning accelerator system. Examples of machine learning models or neural network tasks are as follows:

[0077] As an example, input to a machine learning model may be in the form of an image or video. The machine learning model may be configured to extract, identify, or generate features as part of processing the given input, e.g., as part of a computer vision task. A machine learning model trained to perform this type of neural network task may be trained to generate an output classification from a set of different potential classifications. Additionally or alternatively, the machine learning model may be trained to output a score corresponding to an estimated probability that an identified object in an image or video belongs to a particular class.

[0078] As another example, the input to a machine learning model may be formatted metadata obtained from a data file corresponding to a particular format, e.g., an HTML file, a word processing document, or other type of data, such as metadata for an image file. A machine learning task in this context may be to classify, score, or predict some characteristic about the received input. For example, a machine learning model may be trained to predict the probability that the received input contains text related to a particular subject. A machine learning model may also be trained to generate text predictions as part of performing a particular task, e.g., as part of a tool that auto-completes text in a document while the document is being composed. A machine learning model may also be trained to predict the translation of text in an input document into a target language, e.g., while composing a message.

[0079] Other types of input documents can be data related to the characteristics of a network of interconnected devices. These input documents can include activity logs and records regarding the access privileges of various computing devices to access various sources of potentially sensitive data. Machine learning models can be trained to process these and other types of documents to predict current and future security breaches of the network. For example, a machine learning model can be trained to predict intrusions into a network by malicious attackers.

[0080] As another example, input to a machine learning model can be audio input, including streamed audio, pre-recorded audio, audio as part of a video or other source or media. Machine learning tasks in an audio context can include speech recognition, which involves separating audio from other identified audio sources and / or enhancing the audibility of identified audio features. Machine learning models can be trained to predict accurate translations of input audio into a target language in real time, for example, as part of a translation tool.

[0081] In addition to data inputs, including the various types of data described herein, machine learning models can also be trained to process features corresponding to a given input. A feature is a value, such as a numeric value or a category, that relates to some characteristic of the input. For example, in the context of an image, an image's features may relate to the RGB values ​​of each pixel in the image. A machine learning task in an image / video context may be to classify the content of an image or video, for example, for the presence of various people, places, objects, etc. Machine learning models can be trained to extract and select features relevant to processing to generate an output for a particular input, as well as to generate new features based on learned relationships between various characteristics of the input data.

[0082] Aspects of this disclosure may be implemented as digital circuitry, as one or more computer programs in a computer-readable storage medium, or as a combination of one or more of the foregoing. The computer-readable storage medium may be non-transitory, for example, executable by a cloud computing platform, and as one or more instructions stored on a tangible storage device.

[0083] In this specification, the phrase "configured to" is used in various contexts in reference to a computer system, hardware, or part of a computer program. When a system is said to be configured to perform one or more operations, this means that the system has appropriate software, firmware, and / or hardware installed and, during operation, causes the system to perform one or more operations. When a piece of hardware is said to be configured to perform one or more operations, this means that the hardware includes one or more circuits that, during operation, receive inputs and generate outputs corresponding to the one or more operations according to the inputs. When a computer program is said to be configured to perform one or more operations, this means that the computer program includes one or more program instructions that, when executed by one or more computers, cause the one or more computers to perform one or more operations.

[0084] Unless otherwise specified, the foregoing alternatives are not mutually exclusive and can be implemented in various combinations to achieve unique advantages. Because these and other variations and combinations of the above-described features can be utilized without departing from the subject matter defined by the claims, the foregoing description of embodiments should be construed as illustrative, and not limiting, of the subject matter defined by the claims. Furthermore, the provision of examples described herein, as well as clauses such as "such as," "including," and the like, should not be construed as limiting the subject matter of the claims to any particular example. Rather, these examples are intended to illustrate only one of many possible embodiments. Furthermore, the same reference numbers in different drawings may identify the same or similar elements.

Claims

1. 1. A machine learning accelerator system, comprising: One or more accelerator clusters including a plurality of unevenly distributed memory nodes and a plurality of compute nodes, the unevenly distributed memory nodes and the plurality of compute nodes being connected by a high-speed chip-to-chip interconnect, the machine learning accelerator system further comprising: one or more processors configured to prefetch machine learning models or data from slower memory nodes of the unevenly distributed plurality of memory nodes via the high-speed chip-to-chip interconnect into on-chip memory in the compute node; an access latency of the slow memory node is longer than an access latency of the plurality of memory nodes other than the slow memory node; the one or more processors are further configured to keep a portion of the machine learning model or data uncompressed in a fast memory node of the plurality of memory nodes; The system, wherein the access latency of the fast memory node is less than the access latency of the slow memory node.

2. 10. The system of claim 1, wherein the high-speed inter-chip interconnect comprises an inter-chip interconnect (ICI) having multiple off-chip links and multiple on-chip links.

3. The one or more processors: determining that additional machine learning models or data are required to perform the calculations; 3. The system of claim 1 or 2, further configured to unpack the additional machine learning model or data from the slower memory node of the plurality of memory nodes.

4. The system of claim 1 or 2, wherein keeping the machine learning model or portions of the data uncompressed is based on at least one of recency of use or frequency of use.

5. 1. A method for a machine learning accelerator system, the machine learning accelerator system including one or more accelerator clusters including a plurality of unevenly distributed memory nodes and a plurality of compute nodes, the unevenly distributed memory nodes and the plurality of compute nodes being connected by a high-speed chip-to-chip interconnect, the method comprising: one or more processors determining, based on a latency threshold, which memory nodes of the plurality of memory nodes constitute slow memory nodes; the one or more processors prefetching machine learning models or data from the slower memory nodes to on-chip memory in the compute nodes via the high-speed chip-to-chip interconnect; the one or more processors keeping a portion of the machine learning model or data uncompressed in a high-speed memory node of the plurality of memory nodes based on at least one of recent use or frequency of use; the one or more processors determining that additional machine learning models or data are needed to perform the calculations; and the one or more processors decompressing the additional machine learning model or data from the slower memory nodes of the plurality of memory nodes; The method, wherein the access latency of the fast memory node is less than the access latency of the slow memory node.

6. A computer program product which, when executed by one or more processors, causes the one or more processors to perform the method of claim 5.

Images