Heterogeneous ML Accelerator Cluster with Flexible System Resource Balancing

Abstract

Aspects of the present disclosure are directed to heterogeneous machine learning accelerator systems with compute nodes and memory nodes connected by high-speed chip-to-chip interconnects. While existing remote / non-aggregated memories may require memory expansion via a remote processing unit, aspects of the present disclosure add memory nodes to the machine learning accelerator cluster via chip-to-chip interconnects without requiring assistance from the remote processing unit to achieve higher performance, simpler software stacks and / or lower costs. Memory nodes may support pre-fetching and intelligent compression, allowing the use of lower-cost memory without performance degradation.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Patent Application No. 18 / 152,428, filed January 10, 2023, the disclosure of which is incorporated herein by reference. Summary of the Invention [Problem to be solved by the invention]

[0002] background Training a state-of-the-art machine learning model may require hundreds to tens of thousands of modern machine learning model accelerators, which may be connected with high-speed chip-to-chip interconnects to form a system that performs the total computation and storage required. For example, thousands of tensor processing unit (TPU) chips may be connected by inter-chip interconnects (ICIs) to form a TPU pod. Each TPU accelerator has fixed computing and memory resources, so the overall system balance of aggregated computing and memory resources is also fixed. This leads to a suboptimal situation, since different machine learning models may require different resource balances in terms of computing and memory. For a system with a fixed resource ratio between computing and memory, some models may be stuck with memory capacity, while other models may be stuck with computing resources. Effect of the Invention

[0003] Quick Overview Aspects of the present disclosure are directed to heterogeneous machine learning accelerator systems with compute nodes and memory nodes connected by high-speed chip-to-chip interconnects. While existing remote / non-aggregated memories may require memory expansion via a remote processing unit, aspects of the present disclosure add memory nodes to the machine learning accelerator cluster via chip-to-chip interconnects without requiring assistance from the remote processing unit to achieve higher performance, simpler software stacks and / or lower costs. Memory nodes may support pre-fetching and intelligent compression, allowing the use of lower-cost memory without performance degradation. [Means for solving the problem]

[0004] One aspect of the disclosure provides a machine learning accelerator system that includes one or more accelerator clusters including a plurality of non-uniformly distributed memory nodes and a plurality of computational nodes connected by a high-speed chip-to-chip interconnect, and one or more processors configured to pre-fetch machine learning models or data from slower memory nodes of the plurality of non-uniformly distributed memory nodes via the high-speed interconnect to an on-chip memory in the computational node.

[0005] In one example, the high-speed interconnect includes an inter-chip interconnect (ICI) having multiple off-chip links and multiple on-chip links. In another example, the multiple memory nodes are of various types including high-bandwidth memory (HBM) and at least one of double data rate synchronous dynamic random access memory (DDR) or low power DDR (LPDDR). In yet another example, the multiple computational nodes are of various types including at least two of a tensor processing unit (TPU), a graphics processing unit (GPU), or a central processing unit (CPU).

[0006] In yet another example, the one or more processors are further configured to determine that the memory node of the plurality of memory nodes constitutes a slow memory node based on a latency threshold. In yet another example, the one or more processors are further configured to prefetch the machine learning model or data from a remote memory node of the plurality of memory nodes to an on-chip memory at the compute node via a high-speed interconnect. In yet another example, the one or more processors are further configured to determine that the memory node of the plurality of memory nodes constitutes a remote memory node based on a location of the memory node within one or more accelerator clusters.

[0007] In yet another example, the one or more processors are further configured to swap the machine learning model or data between memory nodes of the plurality of memory nodes based on a hotness of the machine learning model or data. In yet another example, the one or more processors are further configured to determine a hotness of the machine learning model or data based on at least one of recency or frequency of use.

[0008] In yet another example, the one or more processors are further configured to keep the machine learning model or portion of the data uncompressed in a fast memory node of the plurality of memory nodes. In yet another example, the one or more processors are further configured to determine that additional machine learning models or data are needed to perform the calculations and to decompress the additional machine learning models or data from a slow memory node of the plurality of memory nodes. In yet another example, keeping the machine learning model or portion of the data uncompressed is based on at least one of recent use or frequency of use.

[0009] Another aspect of the present disclosure provides a method for a machine learning accelerator system. The machine learning accelerator system includes one or more accelerator clusters including a plurality of unevenly distributed memory nodes and a plurality of computation nodes connected by a high-speed chip-to-chip interconnect. The method includes one or more processors determining that a memory node of the plurality of memory nodes constitutes a slow memory node based on a latency threshold, and the one or more processors prefetching machine learning models or data from the slow memory node to an on-chip memory in the computation node via the high-speed interconnect.

[0010] In one example, the method further includes the one or more processors determining that a memory node of the plurality of memory nodes constitutes a remote memory node based on a location of the memory node in the one or more accelerator clusters, and the one or more processors prefetching the machine learning model or data from the remote memory node to an on-chip memory in the compute node via a high-speed interconnect.

[0011] In another example, the method further includes the one or more processors determining a hotness of the machine learning model or data based on at least one of recent use or frequency of use, and the one or more processors swapping the machine learning model or data between memory nodes of the plurality of memory nodes based on the hotness of the machine learning model or data.

[0012] In yet another example, the method further includes the one or more processors keeping a portion of the machine learning model or data in an uncompressed state in a fast memory node of the plurality of memory nodes based on at least one of recent use or frequency of use, the one or more processors determining that additional machine learning models or data are needed to perform a calculation, and the one or more processors decompressing the additional machine learning model or data from a slower memory node of the plurality of memory nodes.

[0013] Yet another aspect of the present disclosure provides a non-transitory computer-readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for a machine learning accelerator system. The machine learning accelerator system includes one or more accelerator clusters including a plurality of unevenly distributed memory nodes and a plurality of computational nodes connected by a high-speed chip-to-chip interconnect. The operations include determining, based on a latency threshold, that a memory node of the plurality of memory nodes constitutes a slow memory node, and prefetching, via the high-speed interconnect, a machine learning model or data from the slow memory node to an on-chip memory in the computational node.

[0014] In one example, the operations further include determining that a memory node of the plurality of memory nodes constitutes a remote memory node based on a location of the memory node in the one or more accelerator clusters, and prefetching the machine learning model or data from the remote memory node to an on-chip memory at the compute node via a high-speed interconnect.

[0015] In other examples, the operations further include determining a hotness of the machine learning model or data based on at least one of recent use or frequency of use, and swapping the machine learning model or data between memory nodes of the plurality of memory nodes based on the hotness of the machine learning model or data.

[0016] In yet another example, the operations further include keeping a portion of the machine learning model or data uncompressed in a faster memory node of the plurality of memory nodes based on at least one of recent use or frequency of use, determining that additional machine learning models or data are needed to perform a calculation, and decompressing the additional machine learning model or data from a slower memory node of the plurality of memory nodes. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a block diagram of an example heterogeneous machine learning accelerator system according to aspects of the present disclosure. [Diagram 2] FIG. 1 is a block diagram of an example environment for implementing a heterogeneous accelerator system according to aspects of the present disclosure. [Diagram 3] FIG. 2 is a block diagram of an example accelerator cluster according to an aspect of the present disclosure. [Figure 4] FIG. 2 is a block diagram of an example accelerator cluster with additional compute nodes and additional memory nodes according to aspects of the disclosure. [Diagram 5] FIG. 1 is a flow diagram of an example process for active prefetching in accordance with an aspect of the present disclosure. [Figure 6] FIG. 1 is a flow diagram of an example process for memory swapping in accordance with aspects of the present disclosure. [Figure 7] FIG. 2 is a flow diagram of an exemplary process for intelligent memory compression according to aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Detailed Description Generally disclosed herein is an implementation in a heterogeneous machine learning accelerator system with compute nodes and memory nodes connected by a high-speed chip-to-chip interconnect. The memory nodes support intelligent compression, allowing the use of low-cost memory without performance degradation. The high-speed interconnected memory nodes are distinct from remote / non-aggregated memory. In this regard, the high-speed interconnected memory nodes may be accessed directly by the processing unit host, whereas the remote / non-aggregated memory is accessed by the processing unit host via a remote processing unit system stack. Accessing the remote / non-aggregated memory via the remote processing unit system stack may cause delays, which may reduce the performance of the remote / non-aggregated memory compared to the interconnected memory nodes. Furthermore, expensive processing unit resources may be left behind from the accelerator system in facilitating access to such remote / non-aggregated memory.

[0019] The accelerator system may include one or more accelerator clusters, each having multiple memory nodes and multiple computational nodes connected by the same high-speed inter-chip interconnect, such as an inter-chip interconnect (ICI). The high-speed interconnect may include multiple off-chip links and multiple on-chip links connected with an on-chip router. The off-chip and on-chip links may have high data throughput, allowing large amounts of data to be transmitted quickly both on-chip and off-chip. The high-speed interconnect may include an optical interconnect or a high-speed Ethernet.

[0020] The memory nodes may include the same or different memory types in the accelerator cluster, including high bandwidth memory (HBM), double data rate synchronous dynamic random access memory (DDR), low power DDR (LPDDR), dynamic random access memory (DRAM), flash memory, or non-volatile RAM. The compute nodes may include the same or different compute types, including tensor processing units (TPUs), graphics processing units (GPUs), and central processing units (CPUs).

[0021] Compute nodes can include different hardware with different computational capabilities. For example, some compute nodes can have different supporting hardware, such as some compute nodes are designed or optimized for one function, such as video / image processing, while other compute nodes in the same cluster can be designed or optimized for another function, such as natural language processing. Compute nodes can also be software or hardware managed, and have different power efficiency and memory persistence depending on the type of memory contained in the memory nodes. For example, the power efficiency of a compute node can be different for LPDDR, DDR, and HBM, and the memory persistence can be different for DRAM, flash memory, or non-volatile RAM.

[0022] The compute-to-memory resource balance in an accelerator cluster can be changed by installing or pre-installing and activating additional memory and / or compute nodes. The additional memory and / or compute nodes can be distributed uniformly or non-uniformly across the accelerator cluster. Uniformly distributed memory nodes can correspond to additional memory nodes that are the same, such as having the same capacity and / or latency, while non-uniformly distributed memory nodes can correspond to additional memory nodes that vary, such as having different capacity and / or latency.

[0023] Uniformly or non-uniformly distributed memory nodes can be collocated in adjacent memory racks and connected to accelerator clusters via high-speed interconnects. Memory racks can be physically or logically partitioned to accommodate remote / non-aggregated memories that can serve as memory pools for various accelerator clusters.

[0024] Multiple memory nodes can introduce a memory hierarchy based on unevenly distributed memory nodes with different memory latencies. To account for the memory hierarchy, software in the accelerator cluster can utilize active prefetching and / or memory swapping. Compute nodes can include high performance on-chip memory such as CMEM.

[0025] Software in the accelerator cluster can prefetch machine learning models and / or data from remote and / or slow memory nodes to on-chip memory in the compute nodes. The software can determine which memory nodes can constitute slow memory nodes based on a configurable latency threshold, which can be based on configurable performance and / or efficiency requirements. For example, a memory node can be considered slow if its access latency is longer than the high bandwidth memory access latency. The software can also determine which memory nodes can constitute remote memory nodes based on whether the memory node is part of the accelerator cluster. The machine learning models can include tensors and / or layers, and the data can include training data and / or validation data. For example, DDR and / or LPDDR memory nodes can have their machine learning models and / or data prefetched, while HBM memory nodes do not need their machine learning models and / or data prefetched. Prefetching ensures that a working set of machine learning models and data is always ready to be accessed from on-chip memory when computation is performed.

[0026] Software in the accelerator cluster may also swap machine learning models and / or data between memory nodes based on hotness of the machine learning models and / or data. Hotness may refer to recent or frequent use of the machine learning models and / or data. For example, machine learning models and / or data that are likely to be used recently or more frequently may be placed on closer or faster memory nodes such as HBM, while machine learning models and / or data that are likely to be used less recently or rarely may be placed on more / further away memory nodes such as DDR, or slower memory nodes. The recent and frequent use thresholds may be configurable depending on performance and / or efficiency requirements.

[0027] The use of low-cost, high-density memories such as DDR and LPDDR can scale the total memory capacity of a machine learning accelerator cluster and dynamically balance compute and memory resources. Additionally, adding intelligent memory compression to the distant memory nodes can further scale memory capacity, as software can pre-fetch machine learning models and / or data from distant and / or slower memory nodes. Memory nodes can include hardware compression engines that can perform intelligent memory compression and pre-fetching instead of compressing / decompressing on fetch.

[0028] The software can keep portions of the machine learning models and / or data uncompressed in a fast memory node, e.g., HBM, and decompress additional models and / or data as needed. The software can implement cache replacement policies, such as least recently used (LRU) or most frequently used (MFU) algorithms, to keep portions of the machine learning models and / or data uncompressed in a fast memory node. The software can also implement more advanced cache replacement policies, such as algorithms that combine LRU and MFU, to determine which portions of the machine learning models and / or data should be kept uncompressed. For example, the software can prefetch or decompress based on input data in the machine learning models, and fetch or decompress additional models / data as the input data is processed.

[0029] FIG. 1 illustrates a block diagram of an exemplary heterogeneous machine learning accelerator system 100. The heterogeneous accelerator system 100 can be configured to receive input data, such as through a user interface. For example, the heterogeneous accelerator system 100 can receive input data as part of a call to an API that exposes the heterogeneous accelerator system 100. The heterogeneous accelerator system 100 can be implemented on one or more computing devices. Input to the heterogeneous accelerator system 100 can be provided, for example, through a storage medium, including remote storage, connected to one or more computing devices via a network, or as input through a user interface on a client computing device coupled to the heterogeneous accelerator system 100.

[0030] The heterogeneous accelerator system 100 can be configured to receive training data 102 for training a machine learning model and target computing resource data 104 specifying a target computing resource. The training data 102 can correspond to a machine learning task, such as a neural network task performed by a neural network. The training data 102 can be split into a training set, a validation set, and / or a test set. An exemplary training / test split can be an 80 / 20 split. The machine learning model can be configured to receive any type of input data and generate output data 106 for performing the machine learning task. By way of example, the output data 106 can be any kind of score, classification, or regression output based on the input data. Correspondingly, the machine learning task can be a scoring, classification, and / or regression task for predicting some output given some input. These machine learning tasks can correspond to various applications that process images, videos, text, audio, or other types of data.

[0031] The training data 102 may be in any form suitable for training a machine learning model according to one of a variety of different learning techniques. Learning techniques for training a machine learning model may include supervised, unsupervised, and semi-supervised learning techniques. For example, the training data 102 may include a number of training examples that may be received as input by the machine learning model. The training examples may be labeled with a desired output for the machine learning model when processing the labeled training examples. The labels and the model's output may be evaluated through a loss function to determine an error, which may be backpropagated through the machine learning model to update weights in the machine learning model. For example, if the machine learning task is a classification task, the training examples may be images labeled with one or more classes that classify the object depicted in the image. As another example, a supervised learning technique may be applied to calculate the error between the outputs using ground truth labels of the training samples processed by the machine learning model. A variety of loss or error functions may be utilized that are appropriate for the type of task for which the machine learning model is being trained, such as cross-entropy loss for classification tasks or mean squared error for regression tasks. A gradient of the error with respect to various weights of the candidate model on the candidate hardware can be calculated, for example using a backpropagation algorithm, and the model weights can be updated. The machine learning model can be trained until a stopping criterion is met, such as a number of training iterations, a maximum duration, convergence, or when a minimum accuracy threshold is met.

[0032] The target computing resource data 104 may specify characteristics of a computing resource on which the neural network may be at least partially deployed. The computing resources may be housed in one or more data centers or other physical locations that host any of a variety of types of hardware devices. Examples of hardware types include central processing units (CPUs), graphics processing units (GPUs), edge or mobile computing devices, field programmable gate arrays (FPGAs), and various types of application specific circuits (ASICs).

[0033] Some devices can be configured for hardware acceleration, which can include devices configured to efficiently perform certain types of operations. These hardware accelerators, which can include, for example, GPUs and tensor processing units (TPUs), can implement special functionality for hardware acceleration. Example hardware acceleration functionality can include configurations that perform operations commonly associated with running machine learning models, such as matrix multiplication. These special functionality can include, for example, matrix multiply-accumulate units available in various types of GPUs and matrix multiplication units available in TPUs.

[0034] The target computing resource data 104 may include data for one or more target sets of computing resources. The target set of computing resources may refer to a group of computing devices on which it is desired to deploy the machine learning model. The information specifying the target set of computing resources may refer to the type and / or quantity of hardware accelerators or other computing devices in the target set. The target sets may include devices of the same or different types. For example, the target set of computing resources may specify the hardware characteristics and quantity of a particular type of hardware accelerator, such as its processing power, throughput, memory capacity, etc. As described herein, the heterogeneous accelerator system 100 may generate one or more machine learning model architectures for each device specified in the target set of computing resources. In some examples, the target set of computing resources may specify computing resources in devices that have less overall computational power than devices in a data center, such as mobile phones or wearable devices, e.g., headphones, earbuds, smart watches, on which the machine learning model is deployed. Additionally, the target computing resource data 104 may specify different target sets of computing resources, e.g., reflecting different potential configurations of computing resources housed in a data center.

[0035] From the training data 102 and the target computing resource data 104, the heterogeneous accelerator system 100 can be configured to output one or more results of the machine learning task generated as output data 106. The output data 106 can be transmitted for display on a user display, as an example. In some implementations, the heterogeneous accelerator system 100 can be configured to provide the output data 106 as a set of computer-readable instructions, such as one or more computer programs. The computer programs can be written in any type of programming language and according to any programming paradigm, such as declarative, procedural, assembly, object-oriented, data-oriented, functional, or imperative. The computer programs can be written to perform one or more different functions and run, for example, on a physical device, on a virtual machine, or within a computing environment across multiple devices. The computer programs can also implement the functions described herein, for example, as executed by a system, engine, module, or model.

[0036] The heterogeneous accelerator system 100 can be configured to forward the output data 106 to one or more other devices configured to convert the output data 106 into an executable program written in a computer programming language, and optionally as part of a framework for generating a machine learning model. The heterogeneous accelerator system 100 can also be configured to transmit the output data 106 to a storage device for storage and subsequent retrieval.

[0037] The heterogeneous accelerator system 100 may include an accelerator cluster engine 108. The accelerator cluster engine 108 may be implemented as one or more computer programs, specially configured electronic circuits, or any combination thereof. The accelerator cluster engine 108 may be configured to generate one or more accelerator clusters, each having multiple memory nodes and multiple computational nodes connected by high-speed chip-to-chip interconnects. The memory nodes may include the same or different types of memory, and the computational nodes may include the same or different types of processing units. The accelerator cluster engine 108 may generate the accelerator cluster based on the target computing resource data 104. Generating the accelerator cluster may include determining whether to activate additional memory nodes and / or computational nodes, and whether to distribute the additional memory nodes and / or computational nodes uniformly or non-uniformly throughout the accelerator cluster based on the target computing resource data 104.

[0038] The heterogeneous accelerator system 100 may further include a prefetch / memory swap engine 110. The prefetch / memory swap engine 110 may be implemented as one or more computer programs, specially configured electronic circuits, or any combination thereof. The prefetch / memory swap engine 110 may be configured to prefetch input data, such as machine learning models and / or training data 102, from a remote and / or slow memory node to an on-chip memory in a computation node such that a working set of the machine learning models and / or input data is ready to be accessed when computation is performed. The prefetch / memory swap engine 110 may determine which memory nodes constitute slow memory nodes based on a configurable latency threshold that may be defined by the performance and / or efficiency requirements of the target computing resource data 104. The prefetch / memory swap engine 110 may determine which memory nodes constitute remote memory nodes based on their location within or outside the accelerator cluster.

[0039] The prefetch / memory swap engine 110 may also be configured to swap the machine learning models and / or input data between memory nodes based on the hotness of the machine learning models and / or input data. The prefetch / memory swap engine 110 may determine hotness based on configurable thresholds for recent and / or frequent use, which may be dictated by performance and / or efficiency requirements of the target computing resource data 104.

[0040] The heterogeneous accelerator system 100 may further include an intelligent compression engine 112. The intelligent compression engine 112 may be implemented as one or more computer programs, specially configured electronic circuits, or any combination thereof. The intelligent compression engine 112 may be configured to keep a portion of the machine learning models and / or input data in an uncompressed state in a high-speed memory node, and to decompress additional machine learning models and / or input data as needed. The intelligent compression engine 112 may execute a cache replacement policy, such as a least recently used (LRU) algorithm, a most frequently used (MFU) algorithm, or a combination thereof, to determine which portions of the machine learning models and / or input data should be kept in an uncompressed state.

[0041] 2 illustrates a block diagram of an exemplary environment 200 for implementing a heterogeneous accelerator system. The system 200 can be implemented on one or more devices having one or more processors at one or more locations, such as a server computing device 202. The client computing device 204 and the server computing device 202 can be communicatively coupled to one or more storage devices 206 via a network 208. The storage device 206 can be a combination of volatile and non-volatile memory and can be in the same or a different physical location as the computing devices 202, 204. For example, the storage device 206 can include any type of non-transitory computer-readable medium capable of storing information, such as a hard drive, a solid-state drive, a tape drive, optical storage, a memory card, ROM, RAM, DVD, CD-ROM, writable memory, and read-only memory.

[0042] The server computing device 202 may include one or more processors 210 and memory 212. The memory 212 may store information accessible by the processor 210, including instructions 214 that may be executed by the processor 210. The memory 212 may also include data 216 that may be retrieved, manipulated, or stored by the processor 210. The memory 212 may be a type of non-transitory computer-readable medium that may store information accessible by the processor 210, such as volatile and non-volatile memory. The processor 210 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and / or application specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0043] The instructions 214 may include one or more instructions that, when executed by the processor 210, cause one or more processors to perform actions prescribed by the instructions 214. The instructions 214 may be stored in object code format for direct processing by the processor 210, or may be stored in other formats, including an interpretable script or a collection of independent source code modules that are interpreted on demand or pre-compiled. The instructions 214 may include instructions for implementing a heterogeneous accelerator system 218 that may correspond to the heterogeneous accelerator system 100 of FIG. 1. The heterogeneous accelerator system 218 may be executed using the processor 210 and / or using other processors located remotely from the server computing device 202.

[0044] The data 216 may be retrieved, stored, or modified by the processor 210 according to the instructions 214. The data 216 may be stored in a computer register, a relational or non-relational database, as a table with multiple different fields and records, or as a JSON, YAML, proto, or XML document. The data 216 may also be formatted in a computer readable format, such as, but not limited to, binary values, ASCII, or Unicode. Additionally, the data 216 may include sufficient information to identify related information, such as numbers, descriptive text, unique codes, pointers, references to data stored in other memory, including other network locations, or information used by a function to calculate the related data.

[0045] The client computing device 204 may be configured similarly to the server computing device 202, with one or more processors 220, memory 222, instructions 224, and data 226. The client computing device 204 may also include user input 228 and user output 230. The user input 228 may include any suitable mechanism or technology for receiving input from a user, such as a keyboard, a mouse, a mechanical actuator, a soft actuator, a touch screen, a microphone, and a sensor.

[0046] The server computing device 202 may be configured to transmit data to the client computing device 204, which may be configured to display at least a portion of the received data on a display implemented as part of the user output 230. The user output 230 may also be used to display an interface between the client computing device 204 and the server computing device 202. The user output 230 may alternatively or additionally include one or more speakers, transducers or other audio outputs, a haptic interface or other haptic feedback that provides non-visual and non-auditory information to a platform user of the client computing device 204.

[0047] 2 illustrates the processors 210, 220 and memories 212, 222 as being within the computing devices 202, 204, the components described herein may include multiple processors and memories that may operate in different physical locations rather than within the same computing device. For example, some of the instructions 214, 224 and data 216, 226 may be stored on a removable SD card, while other instructions may be stored in a read-only computer chip. Some or all of the instructions and data may be stored in a location that is physically separate from the processors 210, 220, but still accessible by the processors 210, 220. Similarly, the processors 210, 220 may include a group of processors capable of performing simultaneous and / or sequential operations. The computing devices 202, 204 may each include one or more internal clocks that provide timing information that may be used for time measurements in operations and programs executed by the computing devices 202, 204.

[0048] The server computing device 202 can be connected via a network 208 to a data center 232 that houses any number of hardware accelerators 232A-N. The data center 232 can be one of multiple data centers or other facilities in which various types of computing devices, such as hardware accelerators, are located. The computing resources housed in the data center 232 can be designated for deploying a neural network architecture on a hardware architecture, as described herein.

[0049] The server computing device 202 can be configured to receive requests to process data from the client computing devices 204 on computing resources in the data center 232. For example, the environment 200 can be part of a computing platform configured to provide various services to users through various user interfaces and / or application programming interfaces (APIs) that expose platform services. One or more of the services can be a machine learning framework or set of tools for generating a neural network or other machine learning model according to a specified task and training data. The client computing devices 204 can send and receive data specifying a target computing resource to be assigned to execute the machine learning model trained to perform a particular machine learning task. The heterogeneous accelerator system 218 can receive data specifying the target computing resource and / or the training data and, in response, generate output data for deployment on the target computing resource.

[0050] As another example of potential services provided by a platform implementing environment 200, server computing device 202 may maintain different machine learning models according to different potential target computing resources available at data center 232. For example, server computing device 202 may maintain different families for deploying neural networks on different types of TPUs and / or GPUs housed at data center 232 or available for processing.

[0051] The devices 202, 204 and the data center 232 can communicate directly and indirectly through the network 208. For example, using network sockets, the client computing device 204 can connect to services running on the data center 232 through the Internet Protocol. The devices 202, 204 can set up listening sockets that can accept initiating connections to send and receive information. The network 208 itself can include a variety of configurations and protocols, including the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local networks, and private networks that use communication protocols proprietary to one or more companies. The network 208 can support a variety of short-range and long-range connections. The short-range and long-range connections can be made over different bandwidths, such as 2.402 GHz to 2.480 GHz, commonly associated with the Bluetooth® standard, 2.4 GHz and 5 GHz, commonly associated with the Wi-Fi® communication protocol, or associated with various communication standards, such as the LTE® standard for wireless broadband communication. The network 208 may additionally or alternatively support wired connections between the devices 202, 204 and the data center 232, including various types of Ethernet connections.

[0052] 2 depicts a single server computing device 202, client computing devices 204, and data center 232, it is understood that aspects of the disclosure can be implemented according to a variety of different configurations and quantities of computing devices, including in serial or parallel processing paradigms, or via a distributed network of multiple devices. In some implementations, aspects of the disclosure may be performed on a single device connected to a hardware accelerator configured to process machine learning models and any combination thereof.

[0053] 3 illustrates a block diagram of an exemplary accelerator cluster 300. The accelerator cluster 300 includes multiple memory nodes and multiple computational nodes. Although two memory nodes and two computational nodes are shown in FIG. 3, the accelerator cluster 300 may include any number of memory nodes and computational nodes. As illustrated, the multiple memory nodes include a first memory node 302 and a second memory node 304, and the multiple computational nodes include a first computational node 306 and a second computational node 308.

[0054] The memory nodes may include the same or different memory types, including high bandwidth memory (HBM), double data rate synchronous dynamic random access memory (DDR), low power DDR (LPDDR), dynamic random access memory (DRAM), flash memory, or non-volatile RAM, within accelerator cluster 300. For example, memory node 302 may support HBM, while memory node 304 may support LPDDR.

[0055] The computational nodes may include the same or different computational types, such as tensor processing units (TPUs), graphics processing units (GPUs), and central processing units (CPUs). The computational nodes may also include different hardware having different computational capabilities or different supporting hardware. For example, the computational node 306 may be designed or optimized for one function, such as video / image processing, while the computational node 308 may be designed or optimized for another function, such as natural language processing. The computational nodes may be software or hardware managed, and have different power efficiencies and memory persistence depending on the type of memory included in the memory node. For example, the power efficiency of a computational node may differ for LPDDR, DDR, and HBM, and the memory persistence may differ for DRAM, flash memory, or non-volatile RAM.

[0056] The memory nodes and the computational nodes are connected by a high-speed inter-chip interconnect 310, such as an inter-chip interconnect (ICI). The high-speed interconnect 310 may include multiple off-chip links 312 and multiple on-chip links 314 connected via an interconnect router 316. The off-chip links 312 may connect to an interconnect router on a separate chip (not shown) to connect the computational nodes and / or memory nodes to other computational nodes and / or memory nodes on another chip. The on-chip links 314 may connect the computational nodes and / or memory nodes on the same chip via the interconnect router 316. Although four off-chip links 312 and two on-chip links 314 are shown in FIG. 3, the accelerator cluster 300 may include any number of off-chip links 312 and on-chip links 314. The off-chip links 312 and on-chip links 314 may have high data throughput that allows large amounts of data to be sent quickly both on-chip and off-chip. For example, the off-chip link 312 may be capable of 500 Gbits / sec or greater or less, and the on-chip link may be capable of 1000 Gbits / sec or greater or less. The high speed interconnect 310 may include an optical interconnect or high speed Ethernet.

[0057] High-speed interconnected memory nodes and compute nodes are distinct from remote / non-aggregated memory. In this regard, high-speed interconnected memory nodes can be directly accessed by compute nodes, while remote / non-aggregated memory must be accessed by the processing unit host through the remote processing unit system stack. Accessing remote / non-aggregated memory through the remote processing unit system stack can introduce delays, which can degrade the performance of remote / non-aggregated memory compared to interconnected memory nodes. Furthermore, expensive processing unit resources can be left behind from the accelerator cluster in facilitating access to such remote / non-aggregated memory.

[0058] Based on the target computing resource data, additional memory nodes and / or computing nodes can be installed or pre-installed and activated to change the computing and memory resource balance on the accelerator cluster. FIG. 4 illustrates a block diagram of an example accelerator cluster 400 with additional computing nodes and additional memory nodes. Similar to the accelerator cluster illustrated in FIG. 3, the accelerator cluster 400 includes multiple memory nodes and multiple computing nodes. The multiple memory nodes include a first memory node 402 and a second memory node 404, and the multiple computing nodes include a first computing node 406 and a second computing node 408. For the accelerator cluster 400, based on the target computing resource, the multiple computing nodes include an additional memory node 10 and an additional computing node 412. The memory nodes and computing nodes are connected by a high-speed inter-chip interconnect 414, such as an inter-chip interconnect (ICI). Although one additional memory node 410 and one additional computing node 412 are illustrated in FIG. 4, the accelerator cluster 400 can include any number of additional memory nodes or computing nodes.

[0059] The additional memory nodes and / or computational nodes may be uniformly or non-uniformly distributed throughout the accelerator cluster 400. The uniformly distributed memory nodes may correspond to additional memory nodes that are the same, such as having the same capacity and / or latency, while the non-uniformly distributed memory nodes may correspond to additional memory nodes that are different, such as having different capacity and / or latency. For example, the non-uniformly distributed memory nodes may include one memory node corresponding to HBM and another memory node corresponding to LPDDR. Similarly, the uniformly distributed computational nodes may correspond to additional computational nodes that are the same, such as having the same processing power, while the non-uniformly distributed computational nodes may correspond to additional computational nodes that are different, such as having different processing power. For example, the non-uniformly distributed computational nodes may include one computational node corresponding to a TPU and another computational node corresponding to a CPU.

[0060] Multiple memory nodes can introduce a memory hierarchy based on unevenly distributed memory nodes with different memory latencies. To account for the memory hierarchy, software in the accelerator cluster can utilize active prefetching and memory swapping.

[0061] 5 illustrates a flow diagram of an example process 500 for actively prefetching machine learning models and / or input data from a remote and / or slow memory node to on-chip memory at a compute node. The example process 500 may be executed on a system of one or more processors in one or more locations, such as the heterogeneous accelerator system 100 of FIG.

[0062] As indicated in block 510, the accelerator cluster engine 108 may maintain the unevenly distributed memory nodes and / or the unevenly distributed compute nodes as one or more accelerator clusters.

[0063] As shown in block 520, the prefetch / memory swap engine 110 can determine which memory nodes constitute slow memory nodes based on a configurable latency threshold. The latency threshold can depend on configurable performance and / or efficiency requirements, such as from the target computing resource data 104. For example, if the access latency of a memory node is longer than a predetermined high-bandwidth memory access latency, the prefetch / memory swap engine 110 can determine that the memory node is slow. The prefetch / memory swap engine 110 can also determine which memory nodes constitute slow memory nodes based on the type of memory node. For example, DDR and / or LPDDR memory nodes can constitute slow memory nodes, while HBM can constitute fast memory nodes.

[0064] As shown in block 530, the prefetch / memory swap engine 110 may determine which memory nodes constitute remote memory nodes based on the location of the memory nodes within one or more accelerator clusters or outside one or more accelerator clusters. For example, a memory node within an accelerator cluster that includes one or more compute nodes utilized in a computation does not constitute a remote memory node, but a memory node outside the accelerator cluster constitutes a remote memory node.

[0065] As shown in block 540, the prefetch / memory swap engine 110 can prefetch the machine learning models and / or input data from slower and / or distant memory nodes, without the need to prefetch the machine learning models and / or input data from faster and / or closer memory nodes. Prefetching allows a working set of the machine learning models and / or input data to be readily accessible from on-chip memory in the compute node when computation is performed.

[0066] 6 illustrates a flow diagram of an example process 600 for swapping machine learning models and / or input data between memory nodes based on the hotness of the machine learning models and / or input data. The example process 600 may be executed on a system of one or more processors in one or more locations, such as the heterogeneous accelerator system 100 of FIG.

[0067] As indicated in block 610, the accelerator cluster engine 108 may maintain the unevenly distributed memory nodes and / or the unevenly distributed compute nodes as one or more accelerator clusters.

[0068] As shown in block 620, the prefetch / memory swap engine 110 may determine which memory nodes contain recently used machine learning models and / or input data based on a recently used threshold. The recently used threshold may depend on configurable performance and / or efficiency requirements, such as from the target computing resource data 104.

[0069] As shown in block 630, the prefetch / memory swap engine 110 may determine which memory nodes contain frequently used machine learning models and / or input data based on a frequency threshold. The frequency threshold may depend on configurable performance and / or efficiency requirements, such as from the target computing resource data 104.

[0070] As shown in block 640, the prefetch / memory swap engine 110 can swap recently and / or frequently used machine learning models and / or input data from slower and / or more distant memory nodes to faster and / or closer memory nodes. The prefetch / memory swap engine 110 can also swap recently and / or rarely used machine learning models and / or input data from faster and / or closer memory nodes to slower and / or more distant memory nodes. For example, very recently and / or frequently used machine learning models and / or data can be placed in the HBM, while recently and / or infrequently used machine learning models and / or data can be placed in the DDR.

[0071] The total memory capacity of a machine learning accelerator cluster can be expanded by using low-cost, high-density memory and prefetching or swapping from that memory as needed. Memory capacity can be further expanded by adding intelligent memory compression to distant memory nodes. Software can perform intelligent memory compression and prefetching instead of compressing / decompressing on fetch.

[0072] 7 illustrates a flow diagram of an example process 700 for intelligent memory compression. The example process 700 may be executed on one system of one or more processors in one or more locations, such as the heterogeneous accelerator system 100 of FIG.

[0073] As indicated in block 710, the accelerator cluster engine 108 may maintain the unevenly distributed memory nodes and / or the unevenly distributed compute nodes as one or more accelerator clusters.

[0074] As shown in block 720, the intelligent compression engine 112 can keep portions of the machine learning model and / or input data uncompressed in faster and / or closer memory nodes. For example, the intelligent compression engine 112 can keep portions of the machine learning model and / or input data uncompressed in the HBM. The intelligent compression engine 112 can execute a cache replacement policy, such as a most recently used (LRU) algorithm or a most frequently used (MFU) algorithm, to keep portions of the machine learning model and / or input data uncompressed in faster and / or closer memory nodes, such as the HBM. The intelligent compression engine 112 can also execute a more advanced cache replacement policy, such as an algorithm that is a combination of LRU and MFU, to determine which portions of the machine learning model and / or input data should be kept uncompressed.

[0075] As shown in block 730, the intelligent compression engine 112 can decompress additional machine learning models and / or input data if needed for further computation. For example, for a Mixture of Experts (MoE) model, if K experts of the MoE model are stored in the HBM with compression, then n*K experts can be stored in the HBM, where n is the compression ratio. Then, during training or inference, the intelligent compression engine 112 can keep a portion of all experts of the MoE model uncompressed and resident in the HBM for normal training or inference. If the model training or inference wants to access an expert but finds that the expert does not exist in the HBM, the model training or inference misses the expert and the intelligent compression engine 112 can fetch it from the remote memory and / or decompress a new expert. Memory access penalties based on waiting for memory contents to be available can be included when decompressing experts as well as fetching experts from the remote memory. The software can run an LRU or MFU algorithm to keep a subset of experts in the HBM uncompressed. If the MoE model has individual experts processing various input data, the software can also run more advanced algorithms based on the input data, such as combinations with LRU and / or MFU.

[0076] As described herein, aspects of the present disclosure provide a heterogeneous machine learning accelerator system. Examples of machine learning models or neural network tasks are as follows:

[0077] As an example, the input to the machine learning model may be in the form of an image, video. The machine learning model may be configured to extract, identify, generate features as part of processing a given input, e.g., as part of a computer vision task. A machine learning model trained to perform this type of neural network task may be trained to generate an output classification from a set of different potential classifications. Additionally or alternatively, the machine learning model may be trained to output a score corresponding to an estimated probability that an identified object in an image or video belongs to a particular class.

[0078] As another example, the input to the machine learning model may be formatted metadata obtained from a data file corresponding to a particular format, e.g., metadata of an HTML file, a word processing document, or an image file, or other type of data. A machine learning task in this context may be to classify, score, or predict some characteristic about the received input. For example, a machine learning model may be trained to predict the probability that a received input contains text related to a particular subject matter. A machine learning model may also be trained to generate text predictions as part of performing a particular task, e.g., as part of a tool that auto-completes text in a document while the document is being composed. A machine learning model may also be trained to predict the translation of text in an input document into a target language, e.g., while composing a message.

[0079] Other types of input documents can be data related to the characteristics of a network of interconnected devices. These input documents can include activity logs as well as records regarding the access privileges of various computing devices to access various sources of potentially sensitive data. Machine learning models can be trained to process these and other kinds of documents to predict current and future security breaches to the network. For example, a machine learning model can be trained to predict intrusions into a network by malicious attackers.

[0080] As another example, the input to the machine learning model can be audio input, including streamed audio, pre-recorded audio, audio as part of a video or other source or media. Machine learning tasks in an audio context can include speech recognition, including separating audio from other identified audio sources and / or enhancing audibility of identified speech features. Machine learning models can be trained to predict accurate translations of input speech into a target language, for example in real time as part of a translation tool.

[0081] In addition to data inputs, including the various types of data described herein, machine learning models can also be trained to process features corresponding to a given input. A feature is a value, such as a numerical value or a category, that is related to some characteristic of the input. For example, in the context of an image, an image feature may be related to the RGB values ​​of each pixel in the image. A machine learning task in an image / video context may be, for example, to classify the content of an image or video for the presence of various people, places, objects, etc. Machine learning models can be trained to extract and select features that are relevant for processing to generate an output for a particular input, as well as to generate new features based on learned relationships between various characteristics of the input data.

[0082] Aspects of the disclosure may be implemented as one or more computer programs in a digital circuit, in a computer-readable storage medium, or as a combination of one or more of the foregoing. The computer-readable storage medium may be non-transitory, for example executable by a cloud computing platform, as one or more instructions stored on a tangible storage device.

[0083] In this specification, the phrase "configured to" is used in various contexts in relation to a computer system, hardware, or part of a computer program. When a system is said to be configured to perform one or more operations, this means that the system has appropriate software, firmware, and / or hardware installed thereon, and that, during operation, causes the system to perform one or more operations. When a piece of hardware is said to be configured to perform one or more operations, this means that the hardware includes one or more circuits that, during operation, receive inputs and generate outputs corresponding to the one or more operations in accordance with the inputs. When a computer program is said to be configured to perform one or more operations, this means that the computer program includes one or more program instructions that, when executed by one or more computers, cause the one or more computers to perform one or more operations.

[0084] Unless otherwise stated, the above alternatives are not mutually exclusive and can be implemented in various combinations to achieve unique advantages. These and other variations and combinations of the above-mentioned features can be utilized without departing from the subject matter defined by the claims, and therefore the above description of the embodiments should be construed as illustrative, and not limiting, of the subject matter defined by the claims. Furthermore, the provision of examples described herein, as well as clauses expressed as "such as," "including," and the like, should not be construed as limiting the subject matter of the claims to any particular example. Rather, these examples are intended to illustrate only one of many possible embodiments. Furthermore, the same reference numbers in different drawings may identify the same or similar elements.

Claims

1. 1. A machine learning accelerator system, comprising: One or more accelerator clusters comprising a plurality of non-uniformly distributed memory nodes and a plurality of computational nodes, the plurality of non-uniformly distributed memory nodes and the plurality of computational nodes being connected by a high-speed chip-to-chip interconnect, the machine learning accelerator system further comprising: one or more processors configured to prefetch machine learning models or data from slower memory nodes of the plurality of unevenly distributed memory nodes via the high-speed interconnect to an on-chip memory in the compute node; A machine learning accelerator system comprising:

2. 10. The system of claim 1, wherein the high speed interconnect comprises an inter-chip interconnect (ICI) having multiple off-chip links and multiple on-chip links.

3. 2. The system of claim 1, wherein the plurality of memory nodes are of different types, the different types comprising: high bandwidth memory (HBM) and at least one of double data rate synchronous dynamic random access memory (DDR) or low power DDR (LPDDR).

4. 2. The system of claim 1, wherein the plurality of compute nodes are of different types, the different types comprising at least two of a tensor processing unit (TPU), a graphics processing unit (GPU), or a central processing unit (CPU).

5. 10. The system of claim 1, wherein the one or more processors are further configured to determine which memory nodes of the plurality of memory nodes constitute the slow memory nodes based on a latency threshold.

6. 2. The system of claim 1, wherein the one or more processors are further configured to prefetch the machine learning models or data from a remote memory node of the plurality of memory nodes to the on-chip memory in the compute node via the high-speed interconnect.

7. 7. The system of claim 6, wherein the one or more processors are further configured to determine which memory nodes of the plurality of memory nodes constitute the remote memory node based on a location of the memory node within the one or more accelerator clusters.

8. 2. The system of claim 1, wherein the one or more processors are further configured to swap the machine learning model or data between memory nodes of the plurality of memory nodes based on a hotness of the machine learning model or data.

9. 10. The system of claim 8, wherein the one or more processors are further configured to determine hotness of the machine learning model or data based on at least one of recent use or frequency of use.

10. 2. The system of claim 1, wherein the one or more processors are further configured to keep a portion of the machine learning model or data in an uncompressed state in a fast memory node of the plurality of memory nodes.

11. The one or more processors: determining that additional machine learning models or data are required to perform the calculations; 11. The system of claim 10, further configured to unpack the additional machine learning models or data from the slower memory nodes of the plurality of memory nodes.

12. 11. The system of claim 10, wherein keeping the machine learning models or portions of the data uncompressed is based on at least one of recency or frequency of use.

13. 1. A method for a machine learning accelerator system, the machine learning accelerator system including one or more accelerator clusters including a plurality of non-uniformly distributed memory nodes and a plurality of computational nodes, the plurality of non-uniformly distributed memory nodes and the plurality of computational nodes being connected by a high speed chip-to-chip interconnect, the method comprising: determining, by one or more processors, that a memory node of the plurality of memory nodes constitutes a slow memory node based on a latency threshold; the one or more processors prefetching machine learning models or data from the slower memory nodes to on-chip memory in the compute nodes via the high-speed interconnect; A method comprising:

14. determining, by the one or more processors, that a memory node of the plurality of memory nodes constitutes a remote memory node based on a location of the memory node within the one or more accelerator clusters; the one or more processors prefetching the machine learning model or data from the remote memory node to the on-chip memory in the compute node via the high-speed interconnect; The method of claim 13 further comprising:

15. determining a hotness of the machine learning model or data based on at least one of recency or frequency of use, the one or more processors swapping the machine learning model or data between memory nodes of the plurality of memory nodes based on the hotness of the machine learning model or data; The method of claim 13 further comprising:

16. the one or more processors keeping a portion of the machine learning model or data in an uncompressed state in a high speed memory node of the plurality of memory nodes based on at least one of recent use or frequency of use; determining, by the one or more processors, that additional machine learning models or data are required to perform the calculations; and the one or more processors decompressing the additional machine learning model or data from the slower memory nodes of the plurality of memory nodes; The method of claim 13 further comprising:

17. 1. A non-transitory computer-readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for a machine learning accelerator system, the machine learning accelerator system including one or more accelerator clusters including a plurality of non-uniformly distributed memory nodes and a plurality of computational nodes, the plurality of non-uniformly distributed memory nodes and the plurality of computational nodes being connected by a high-speed chip-to-chip interconnect, the operations comprising: determining, based on a latency threshold, that a memory node of the plurality of memory nodes constitutes a slow memory node; prefetching machine learning models or data from the slower memory nodes to on-chip memory in the compute nodes via the high-speed interconnect; A non-transitory computer readable medium comprising:

18. The operation includes: determining that a memory node of the plurality of memory nodes constitutes a remote memory node based on a location of the memory node within the one or more accelerator clusters; prefetching the machine learning model or data from the remote memory node to the on-chip memory in the compute node via the high-speed interconnect; 20. The non-transitory computer-readable medium of claim 17, further comprising:

19. The operation includes: determining a hotness of the machine learning model or data based on at least one of recent use or frequency of use; swapping the machine learning model or data between memory nodes of the plurality of memory nodes based on the hotness of the machine learning model or data; 20. The non-transitory computer-readable medium of claim 17, further comprising:

20. The operation includes: keeping a portion of the machine learning model or data in an uncompressed state in a fast memory node of the plurality of memory nodes based on at least one of recent use or frequency of use; determining that additional machine learning models or data are required to perform the calculation; decompressing the additional machine learning model or data from the slower memory node of the plurality of memory nodes; 20. The non-transitory computer-readable medium of claim 17, further comprising:

Images