Dynamic allocation of artificial intelligence workloads among heterogeneous processing units

Abstract

Components of an artificial intelligence (AI) task may be dynamically allocated among heterogenous processing units. Computational tasks within the AI task may be identified and classified as either a task in neural network interface or a task outside neural network inference. When a task is classified as a task outside neural network interface, the task may be allocated to a CPU. Data for performing the task may be transferred to the CPU. When a task is classified as a task in neural network interface (e.g., a task of executing a neural network layer), a processing unit may be selected from eligible processing units based on estimated costs of performing the task by the eligible processing units. The cost of switching from another processing unit performing a precedent computation task to the processing unit may also be estimated. The task may be allocated to the processing unit based on these costs.

Description

DYNAMIC ALLOCATION OF ARTIFICIAL INTELLIGENCE WORKLOADS AMONG HETEROGENEOUS PROCESSING UNITSCross-Reference to Related Application

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 662,182, filed June 20, 2024, and entitled "EFFICIENT LARGE LANGUAGE MODEL PARTITION ON ARTIFICIAL INTELLIGENCE ACCELERATOR," which is incorporated by reference in its entirety.Technical Field

[0002] This disclosure relates generally to artificial intelligence (Al), and more specifically, dynamic allocation of Al workloads among heterogeneous processing units.Background

[0003] Neural networks (also referred to as "deep neural networks" or "DNNs") are used extensively for a variety of Al applications ranging from computer vision to speech recognition and natural language processing due to their ability to achieve high accuracy. However, the high accuracy comes at the expense of significant computation cost. DNNs have extremely high computing demands as there can be a large number of operations as well as a large amount of data to read and write. Therefore, techniques to improve efficiency of DNNs are needed.Brief Description of the Drawings

[0004] Embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0005] FIG. 1 illustrates an example transformer model, in accordance with various embodiments.

[0006] FIG. 2 illustrates an example embedding layer, in accordance with various embodiments.

[0007] FIG. 3 illustrates positional encoding, in accordance with various embodiments.

[0008] FIGS. 4A and 4B illustrate an example multi-head attention (MHA) layer, in accordance with various embodiments.

[0009] FIG. 5 is a block diagram of a heterogeneous computing system, in accordance with various embodiments.

[0010] FIG. 6 illustrates an example process of dynamic allocation of Al workloads among heterogeneous processing units, in accordance with various embodiments.

[0011] FIG. 7 illustrates partitioning of an example DNN, in accordance with various embodiments.

[0012] FIG. 8 illustrates partitioning of another example DNN, in accordance with various embodiments.

[0013] FIG. 9 illustrates partitioning of yet another example DNN, in accordance with various embodiments.

[0014] FIG. 10 illustrates coarse-grained partition of an Al task across heterogenous processing units with graph mode execution, in accordance with various embodiments.

[0015] FIG. 11 illustrates coarse-grained partition of an Al task across heterogenous processing units with eager mode execution, in accordance with various embodiments.

[0016] FIG. 12 illustrates model execution with both graph mode and eager mode, in accordance with various embodiments.

[0017] FIG. 13 is a flowchart of a method for dynamic allocation of Al tasks workloads, in accordance with various embodiments.

[0018] FIG. 14 is a flowchart of another method for dynamic allocation of Al tasks workloads, in accordance with various embodiments.

[0019] FIG. 15 is a block diagram of a DNN accelerator 1500, in accordance with various embodiments.

[0020] FIG. 16 is a block diagram of an example computing device, in accordance with various embodiments.Detailed DescriptionOverview

[0021] The last decade has witnessed a rapid rise in Al based data processing, particularly based on DNNs. DNNs are widely used in various domains (e.g., computer vision, speech recognition, autonomous driving, language processing, image processing, video processing,etc.) mainly due to their ability to achieve beyond human-level accuracy. A DNN typically includes a sequence of layers. A DNN layer may include one or more deep learning operations (also referred to as "neural network operations"), such as convolution, matrix multiplication, layer normalization, batch normalization, SoftMax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on.

[0022] For example, large language models (LLMs), including transformer-based neural networks like GPT (Generative Pretrained Transformer) and BERT (Bidirectional Encoder Representations from Transformers) has ushered in a new era of capability in natural language processing. LLMs can generate human-like text and perform various natural language processing tasks. LLMs are a class of foundation models that are generally built using transformer layers trained on huge corpus of data. These models can be used on various down-stream tasks even at a zero-shot or few-shot manner with or without fine- tuning. Autoregressive LLMs have found various use cases for generative Al tasks including complex reasoning, code-generation, question answering, text and context summarization. LLMs can also act as the backbone for vision language foundation models (VLMs). Many such multimodal models (e.g., large language and vision assistant (LLaVA) models) can consume input from both vision and language modalities to generate autoregressive outputs.

[0023] Despite the remarkable capabilities of LLMs, deploying these models efficiently and executing LLM inference on end user devices with resource-constrained Al accelerators is not trivial due to their high computational intensity and expensive memory requirements. LLM workloads may be partitioned to maximize performance and resource utilization while minimizing latency and energy consumption. The partition strategy may involve addressing issues such as load balancing, data movement, and synchronization overheads to achieve efficient utilization of computational resources during inference and training tasks.

[0024] Some currently available approaches for partitioning LLMs leverage the hardware capabilities of custom-designed chips and tailor hardware and software components to achieve efficient execution of LLMs while minimizing power consumption and maximizing battery life. Such approaches may involve fine-tuning model architectures, optimizing memory access patterns, and leveraging specialized hardware instructions for accelerating key operations involved in LLM inference and training tasks. Other techniques can also be used. For instance, kernel fusion, layer pruning, and precision calibration can be used toreduce computational complexity and memory footprint. Parallel or diverse hardware can also be used to accelerate LLM inference and training tasks, achieving high performance and scalability. Further, LLMs may be deployed using open standards and libraries such as HIP (Heterogeneous Compute Interface for Portability) and MIOpen (Machine Intelligence Open) for efficient execution of LLMs and balancing performance, power consumption, and costeffectiveness. Cloud service provides may offer specialized instances and services for deploying LLMs at scale and provide pre-configured environments, optimized drivers, and libraries for running LLM workloads efficiently on cloud-based hardware, enabling developers to leverage the scalability and flexibility of cloud computing for their Al projects.

[0025] DNN optimization and deployment toolkits may be used. For instance, OpenVINO has two execution modes for hetero compute execution: AUTO and MULTI. In the AUTO execution mode, OpenVINO can automatically select the suitable processing unit based on factors such as model size, available hardware resources, and performance requirements. This mode can simplify deployment for developers by abstracting away the complexity of hardware selection and optimization. However, even though convenient, the AUTO mode may not always deliver the highest performance or energy efficiency since it relies on heuristics rather than exhaustive optimization. On the other hand, the MULTI execution mode in OpenVINO allocates computing devices dynamically for inference requests. It can enhance throughput by utilizing multiple devices simultaneously and improves performance stability by distributing the workload across devices. It can maintain application logic intact, eliminating the need for manual model compilation or request balancing. However, its effectiveness relies on consistently supplying sufficient inference requests to keep all devices occupied.

[0026] However, these currently available techniques fail to provide a seamless and optimized solution for leveraging heterogeneous compute for DNNs, such as LLMs. This deficiency usually stems from the absence of an effective cost model to determine the most suitable hardware for running specific tasks. Without this guidance, it can be challenging to determine which tasks should run on which Al accelerator to maximize efficiency and performance. Another drawback is the absence of effective cost model / mapping tool for selecting where to execute tasks. Without a comprehensive understanding of the cost implications of running tasks on different hardware accelerators, it can be difficult to make informed decisions regarding task allocation and resource utilization. An additionaldrawback is the latency introduced by existing frameworks when switching between different accelerators during inference. This latency can hinder the real-time responsiveness required for many applications and makes it challenging to adaptively allocate tasks to the most appropriate accelerators based on dynamic workload changes. These drawbacks can hinder the ability to fully exploit the capabilities of heterogeneous compute resources for LLMs, resulting in suboptimal performance and efficiency

[0027] Embodiments of the present disclosure may improve on at least some of the challenges and issues described above by providing dynamic allocation of Al workloads among heterogeneous processing units. Heterogeneous processing units may be processing units of different types, such as central processing unit (CPU), graphics processing unit (GPU), neural processing unit (NPU, e.g., Al accelerators), other types of processing units, or some combination thereof. An Al task may be partitioned into smaller tasks that can be performed by different ones of the heterogeneous processing units to optimize efficiency. Examples of the Al task may include tasks of deploying DNNs for language processing, speech recognition, image or video analysis, autonomous driving, project management, and so on.

[0028] In various embodiments of the present disclosure, a task manager may identify computational tasks within the Al task and determine whether each computational task is part of neural network inference. The neural network inference may be a process of applying a pretrained neural network on input data to obtain an output of the neural network, which may be a prediction made by the neural network. The Al task may include tasks in the neural network inference, such as tasks of executing neural network layers, and tasks outside the neural network inference, such as tasks of compiling the neural network, generating the input data, processing the neural network output, and so on.

[0029] When a computational task is determined to be outside the neural network inference, the task manager may allocate the computational task to a CPU. The task manager may command the CPU to perform the computational task. The task manager may also facilitate transfer of data to be used by the CPU for performing the computational task, e.g., transfer from a memory of another processing unit to a memory of the CPU. When a computational task is determined to be part of the neural network inference (e.g., the computational task is a task to execute a neural network layer), the task manager may selecta processing unit from a group of eligible processing units and command the processing unit to perform the computation task.

[0030] To select a processing unit for a computational task that is part of the neural network inference, the task manager may identify processing units eligible for executing the neural network layer. For instance, the task manager may identify all the available processing units, which may include various types of processing unit. The task manager may prune the eligible list by removing one or more processing units. For instance, the task manager may remove a processing unit based on historical data indicating a cost of a historical execution of a similar computational task by the processing unit. For instance, the historical data may indicate that the cost is higher than the cost of another historical execution of a similar computational task by each of the other processing units. For each processing unit in the eligible list, the task manager may estimate various types of costs of the processing unit performing the computational task. The costs may include a latency cost indicating an estimation of the latency caused by performing the computational task by the processing unit, an energy cost indicating an estimation of energy consumed by the processing unit for performing the computational task, a performance cost indicating an estimation of a performance of the processing unit for performing the computational task, and so on. The task manager may select one cost type, e.g., based on a user selection. The task manager may receive a user input indicating a preference of a user for a cost type and select the cost type based on the user input. The task manager may further compare the costs of the selected type that are estimated for the processing units in the pruned eligible list and select a processing unit based on the comparison. The task manager may also estimate a switching cost of switching from another processing unit executing a precedent computational task to the processing unit for performing the computational task, such as cost of transferring data from the other processing unit to the processing unit. The task manager may determine whether to switch to the processing unit based on the switching cost. After the task manager determines to switch, the task manager may command the processing unit to perform the computational task.

[0031] The Al workload allocation approach in the present disclosure can enhance the efficiency and scalability of deploying LLMs on heterogenous processing units by strategically offloading specific network computations and layers to heterogenous processing unit. These computations, known as kernels, can range from fundamentalmathematical operations like matrix multiplication to intricate functional blocks like decoder layers in DNNs. The dynamic allocation can lead to a significant boost in overall system efficiency, making it possible to execute demanding Al workloads locally on client devices, such as users' personal computers. The approach in the present disclosure can also enable a toolchain to be highly integrated with DNN libraries, such as PyTorch and Huggingface Transformers libraries. The approach in the present disclosure can lead to improved performance, resource utilization, and energy efficiency. By optimizing workload partitioning and minimizing communication overheads, the approach can achieve higher throughput, reduced latency, and lower operational costs when running LLM inference and training tasks on heterogenous platforms.

[0032] For purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without the specific details or / and that the present disclosure may be practiced with only some of the described aspects. In other instances, well known features are omitted or simplified in order not to obscure the illustrative implementations.

[0033] Further, references are made to the accompanying drawings that form a part hereof, and in which is shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.

[0034] Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the claimed subject matter.However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order from the described embodiment. Various additional operations may be performed or described operations may be omitted in additional embodiments.

[0035] For the purposes of the present disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (Band C), or (A, B, and C). The term "between," when used with reference to measurement ranges, is inclusive of the ends of the measurement ranges.

[0036] The description uses the phrases "in an embodiment" or "in embodiments," which may each refer to one or more of the same or different embodiments. The terms "comprising," "including," "having," and the like, as used with respect to embodiments of the present disclosure, are synonymous. The disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to explain various features of the drawings, but these terms are simply for ease of discussion, and do not imply a desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of the ordinal adjectives "first," "second," and "third," etc., to describe a common object, merely indicates that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking or in any other manner.

[0037] In the following detailed description, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0038] The terms "substantially," "close," "approximately," "near," and "about," generally refer to being within + / - 20% of a target value as described herein or as known in the art. Similarly, terms indicating orientation of various elements, e.g., "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between the elements, generally refer to being within + / - 5-20% of a target value as described herein or as known in the art.

[0039] In addition, the terms "comprise," "comprising," "include," "including," "have," "having" or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, device, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, device, or DNN accelerators. Also, the term "or" refers to an inclusive "or" and not to an exclusive "or."

[0040] The systems, methods and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for all desirable attributes disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the description below and the accompanying drawings.Example Transformer Models

[0041] FIG. 1 illustrates an example transformer model 100, in accordance with various embodiments. The transformer model 100 may transform input sequences into output sequences. In some embodiments, the transformer model 100 is a neural network that can learn context and meaning by tracking relationships in sequential data, such as sequential words in a sentence, sequential audio signals, sequential images, and so on. In an example, the transformer model 100 may be an LLM. The transformer model 100 includes an encoder block 110, a decoder block 120, and a head block 130. In other embodiment, different or additional components may be included in the transformer model 100. Further, functionality attributed to a component of the transformer model 100 may be accomplished by a different component included in the transformer model 100 or a different model or module.

[0042] The encoder block 110 receives input sequences and generates matrix representations of the input sequences. In the embodiments of FIG. 1, the encoder block 110 receives inputs 101 and generates encoder outputs 102. In some embodiments, the inputs 101 may include one or more input tokens, such as words, phrases, sentences, images, audio signals, other types of input tokens, or some combination thereof. In an example, the inputs 101 may include a prompt received from a user of the transformer model 100. The prompt may include a question or request made by the user. A word in the prompt may be an input token. The encoder outputs 102 may include one or more vectors that are contextualized representations of the input 101. Each vector in the encoder outputs 102 may represent a token in the input 101 with contextual understanding.

[0043] The encoder block 110 includes an embedding layer 113, a positional encoding layer 115, and a plurality of layers 140 (individually referred to as "layer 140"). In other embodiments, the encoder block 110 may have different, fewer, or more components. Also, the arrangement of the components in the encoder block 110 may be different from the arrangement shown in FIG. 1. For the purpose of illustration, the encoder block 110 has N layers in FIG. 1, where N is an integer. Each layer 140 may include one or more neural network operations. The layers 140 may transform a sequence of embeddings into a representation that encapsulates the learned information from the input 101. Different layers 140 may have different internal parameters, e.g., different weights, bias, or other types of internal parameters. In some embodiments, the layers 140 have identical components. The components in a layer 140 may be layers and may also be referred to assub-layers of the layer 140. As shown in FIG. 1, a layer 140 includes four sub-layers: an MHA layer 141, an add & norm layer 142, a feed forward layer 143, and another add & norm layer 144.

[0044] The decoder block 120 iteratively generates outputs 103 using encoded representations generated by the encoder block 110. The decoder block 120 includes an embedding layer 123, a positional encoding layer 125, and a plurality of layers 150 (individually referred to as "layer 150"). For the purpose of illustration, the decoder block 120 has N layers in FIG. 1, where N is an integer. In the embodiments of FIG. 2, the number of layers 150 in the decoder block 120 is the same as the number of layers 140 in the encoder block 110. In other embodiments, the number of layers 150 in the decoder block 120 may be different from the number of layers 140 in the encoder block 110. Each layer 150 may include one or more neural network operations. Different layers 150 may have different internal parameters. In some embodiments, the layers 150 may have identical components. The components in a layer 150 may be layers and may also be referred to as sub-layers of the layer 150. As shown in FIG. 1, a layer 150 includes six sub-layers: an MHA layer 151, an add & norm layer 152, an encoder-decoder attention layer 153, another add & norm layer 154, a feed forward layer 155, and another add & norm layer 156.

[0045] In some embodiments, a sequence of inference phases is performed in the decoder block 120 using encoder outputs, e.g., the encoder outputs 102. A matrix may be predicted through each inference phase. The outputs 103 may include a plurality of matrices. Each matrix may be further processed in the head block 130 to predict a token. The plurality of matrices may be used to predict a sequence of tokens. For the first inference phase, the decoder block 120 may receive one or more start tokens as input tokens and compute a first matrix from the input tokens and the output of the encoder block 110. The first matrix may be used by the head block 130 to predict a first token. The predicted token may be used as a new input token, in addition to the start token(s), in the second inference phase. Similarly, a second token may be predicted through the second inference phase and may be used in the third inference phase. This iteration may continue till all the inference phases are complete.

[0046] The head block 130 receives the output of the decoder block 120 and processes it in a linear layer 133 and a SoftMax layer 135. A linear operation may be performed on the output of the decoder block 120 in the linear layer 133. The linear operation may include a multiplication of the output of the decoder block 120 with a weight matrix. The output ofthe linear layer 133 may be a vector. In some embodiments, the head block 130 may function as a classifier. The number of data elements in the vector computed in the linear layer 133 may depend on the number of classes involved. In an example where there are M classes, where M is an integer, the vector computed in the linear layer 133 may have M data elements representing the prediction for the M classes, respectively.

[0047] The output of the linear layer 133 may be input into the SoftMax layer 135. A SoftMax function may be applied on the output of the linear layer 133 to compute probability scores. A probability score may have a value in the range from 0 to 1. In some embodiments, a probability value is computed for each data element in the vector computed in the linear layer 133. The highest one of the probability scores may be the key. The corresponding index of the key may point to the token that the transformer model 100 predicts as the next in the sequence. The final output of the transformer model 100 may be the sequence of predicted tokens. In some embodiments, the head block 130 may be a language modeling head.

[0048] An embedding layer (e.g., the embedding layer 113 or the embedding layer 123) converts an input of the embedding layer (e.g., the inputs 101 or the outputs 103) into one or more embeddings. An embedding may be a vector, which is also referred to as an embedding vector or a vector embedding. The vector embedding may include a sequence of data elements. In some embodiments, the embedding layer 113 may generate a plurality of embeddings, each of which may be converted from a different input token in the inputs 101. The embeddings may capture the semantic meaning of the tokens in the input 101. The embeddings may be numerical representations that capture the relationships or meanings of words, phrases, or other data types. In an example where the input 101 is a prompt including a sequence of words, the embedding layer 113 may generate an embedding from each word in the input 101. The embedding layer 123 in the decoder block 120 may generate a plurality of embeddings from tokens received by the decoder block 120 in a similar manner as the embedding layer 113. Certain aspects of embedding layers are described below in conjunction with FIG. 2.

[0049] A positional encoding layer (e.g., the positional encoding layer 115 or the positional encoding layer 125) performs positional encoding on embeddings generated in the corresponding embedding layer. In some embodiments, the positional encoding layer may apply one or more positional encoding vectors (e.g., a positional encoding vector 104 orpositional encoding vector 105) on vector embeddings from the corresponding embedding layer to generate new vector embeddings that represents the embeddings with positional context. The positional encoding vector may encode information about the position of the embedding in a sequence of embeddings. In some embodiments, the positional encoding layer performs an addition operation on a positional encoding vector and a vector embedding. The addition operation may be elementwise addition. The positional encoding layer may output an embedding matrix that includes the vector embeddings computed in the positional encoding layer. Certain aspects of positional encoding layers are described below in conjunction with FIG. 3.

[0050] An MHA layer (e.g., the MHA layer 141, the MHA layer 151, or the MHA layer 153) may implement a multi-head attention mechanism, which may be a multi-head selfattention mechanism or a multi-head cross-attention mechanism. In some embodiments, the MHA layer 141 or the MHA layer 151 may implement a self-attention mechanism. For self-attention, the queries, keys, and values may come from the same place. For instance, for the MHA layer 141, the queries, keys, and values may all come from the positional encoding layer 115. For the MHA layer 151, the queries, keys, and values may all come from the positional encoding layer 125. The self-attention mechanism may enable the transformer model 100 to relate each token with other tokens. The MHA layer may compute attention scores from embeddings generated in the corresponding positional encoding layer. In some embodiments, the MHA layer may receive one or more queries, one or more keys, and one or more values. In some embodiments, the MHA layer has a number of heads that receive different linearly projected versions of the queries, keys, and values and produce outputs in parallel that are then used to generate the final result.

[0051] In some embodiments, the queries, keys, and values input into the MHA layer 141 may be computed from vector embeddings generated by the positional encoding layer 115. The queries, keys, and values input into the MHA layer 151 may be computed from vector embeddings generated by the positional encoding layer 125. A query, key, or value may be a vector the represents a token in a sequence. In some embodiments, a query matrix Q G may bgconnpU ed by multiply an embedding matrix X G ffilNxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WqG ]Rdxft, where d is the dimension of a vector embedding, N is the number of vector embeddings in the embedding matrix, and h is the number of attention heads. Each row in the query matrixmay be a query. A key matrix K E BWx / lmay be computed by multiple an embedding matrix X E BWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WkE Bdxh. Each row in the key matrix may be a key. A value matrix V E BWx / lmay be computed by multiple an embedding matrix X E BWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WvE Bdx / l. Each row in the value matrix may be a value.

[0052] In some embodiments, the MHA layer 151 may implement masked multi-head selfattention. The MHA layer 151 may prevent positions from attending to subsequent positions. For instance, each token in the sequence may not be influenced by future tokens. This masking can ensure that the predictions of a particular position can depend on known outputs at positions before it and not depend on unknown outputs at positions after it.

[0053] In some embodiments, the MHA layer 153 may implement a cross-attention mechanism, such as encoder-decoder cross-attention. The MHA layer 153 may use outputs from the previous layer (i.e., the add & norm layer 152) as queries and use outputs from the encoder block 110 as keys and values. The cross-attention can align the encoder's input with the decoder's, empowering the decoder block 120 to identify and emphasize the most relevant parts of the encoder's input. Certain aspects of MHA layers are described below in conjunction with FIGS. 4A and 4B.

[0054] An add & norm layer in the transformer model 100, such as the add & norm layer 142, 144, 152, 154, and 156, has an addition operation followed by a layer normalization operation. The addition operation may be an addition of the output of the preceding layer and the input of the preceding layer. The preceding layer is a layer that is arranged right before the add & norm layer. For example, the preceding layer of the add & norm layer 142 is the MHA layer 141. As another example, the preceding layer of the add & norm layer 154 is the encoder-decoder attention layer 153.

[0055] Then the layer normalization operation is applied on the result of the addition operation, which may be denoted as LayerNorm(x + sublay er (x)), where LayerNorm denotes layer normalization, x is the input of the preceding layer, and sublayer^x) denotes the output of the preceding layer. In some embodiments, the layer normalization operation may include a sequence of computations. In an example, the layer normalization operation may include a mean computation, which may be denoted as .xy= ~xz=i^xyz > whereAxyzdenotes a data element in the input tensor, x may be the positional index of the data element in one of the spatial dimensions, y may be the positional index of the data element in the other one of the spatial dimensions, z may be the positional index of the data element in the channel dimension, and ixydenotes the output of the mean computation, which may be a 2D matrix. The mean computation may be channel-wise reduction operation. The layer normalization operation may convert .xyto a 3D tensor yxyz, e.g., by replicating every data element over z output points.

[0056] The layer normalization operation may also include an elementwise subtraction, which may be denoted as Dxyz= Axyz— .xyz. The layer normalization operation may further include a variance computation denoted asD2xyzand a division computation denoted as Mxy=1Mxymay be a 2D tensor. The layernormalization operation may also convert Mxyto a 3D tensor Mxyz, e.g., by replicating every data element over z output points. Further, the layer normalization operation may have an element multiplication denoted as ' y Xlayer normalization operation may further compute A"xyzX yz. LNxyzmay be the output of the layernormalization operation.

[0057] A feed forward layer (e.g., the feed forward layer 143 and the feed forward layer 155) may be a position-wise fully-connected feed forward network. In an example, the feed forward layer may include two linear layers with an activation function in between. An example of the activation function is Rectified Linear Unit (ReLU).

[0058] FIG. 2 illustrates an embedding operation in an embedding layer 200, in accordance with various embodiments. The embedding layer 200 may be an example of the embedding layer 113 or the embedding layer 123 in FIG. 1. As shown in FIG. 2, the embedding layer 200 receives an input sequence 201, which includes three words 202, 203, and 204. Each word may be a token. The embedding layer 200 generates a vector embedding 205 from the word 202. The embedding layer 200 also generates a vector embedding 206 from the word 203. The embedding layer 200 further generates a vector embedding 207 from the word 204. In the embodiments of FIG. 2, the vector embeddings 205, 206, and 207 have the samedimension, i.e., they each have five data elements. In other embodiments, the vector embedding 205, 206, or 207 may have a different dimension. Also, the input to the embedding layer 200 may be data of a type other than words, such as audio signals, images, and so on.

[0059] In some embodiments where the embedding layer 200 is in an encoder (e.g., the encoder block 110), the input sequence 201 may be an input received by the encoder, such as a prompt made by a user. The input sequence 201 may remain the same during inference of the encoder. In some embodiments where the embedding layer 200 is in a decoder (e.g., the decoder block 120), the input sequence 201 may change and the dimension of the input sequence 201 may be dynamic during inference of the decoder. In an example, the decoder inference may include a sequence of phases. Each inference phase may be conducted for predicting a token. For the first inference phase, the input sequence 201 may include one or more start tokens. For each subsequent inference phase (e.g., the second inference phase, the third inference phase, etc.), the input sequence 201 may include tokens predicted in the previous inference phases. The dimension of the input sequence may be increased by one after each inference phase.

[0060] FIG. 3 illustrates a positional encoding operation in a positional encoding layer, in accordance with various embodiments. The positional encoding layer may be an example of the positional encoding layer 115 or the positional encoding layer 125 in FIG. 1. The positional encoding operation includes an addition of a vector embedding 310 and a positional encoding vector 320. The vector embedding 310 may be generated by an embedding layer. The positional encoding vector 320 may encode information of the position of the token represented by the vector embedding 310 in a sequence of tokens. The positional encoding operation computes a vector embedding 330, which represents the token with positional context. In some embodiments, the positional encoding operation may be an elementwise addition operation. A data element in the vector embedding 330 may equal the sum of a data element in the vector embedding 310 and a data element in the positional encoding vector 320. In the embodiments of FIG. 3, the vector embedding 310, positional encoding vector 320, and vector embedding 330 have the same dimension, i.e., they each have five data elements. In other embodiments, the vector embedding 310, positional encoding vector 320, or vector embedding 330 may have a different dimension.

[0061] FIGS. 4A and 4B illustrate an example MHA layer 400, in accordance with various embodiments. The MHA layer 400 may be an example of the MHA layer 141 or the MHA layer 151 in FIG. 1. As shown in FIG. 4A, the MHA layer 400 includes linear layers 410, 420, and 430, a MatMul layer 440, a scale layer 450, a SoftMax layer 460, another MatMul layer 470, a concatenation layer 480, and another linear layer 490. In other embodiments, the MHA layer 400 may include fewer, more, or different layers.

[0062] The MHA layer 400 receive three input matrices: a query matrix 401, a key matrix 402, and a value matrix 403, which are inputs of the linear layers 410, 420, and 430, respectively. The linear layers 410, 420, and 430 are in a linear block 415 of the MHA layer 400. In some embodiments, the MHA layer 400 includes a plurality of linear blocks that includes the linear block 415. For the purpose of illustration, the MHA layer 400 includes h linear blocks in FIG. 4A, where h is an integer. Each of the linear blocks may have the same layers as the linear block 415. Each linear block may compute three parameter matrices from the query matrix 401, key matrix 402, and value matrix 403, respectively. For instance, the linear layer 410 may perform a multiplication of the query matrix 401 with a weight matrix to compute a parameter matrix 404 shown in FIG. 4B. The linear layer 420 may perform a multiplication of the key matrix 402 with a weight matrix to compute a parameter matrix 405 shown in FIG. 4B. The linear layer 430 may perform a multiplication of the value matrix 403 with a weight matrix to compute a parameter matrix 406 shown in FIG. 4B.

[0063] The parameter matrix 404 may be denoted as QW , where Q is the query matrix 401 and W G ]dmodei*dq js t e wejght matrix. The parameter matrix 405 may be denoted as KWIK, where K is the key matrix 401 and W G ]dmodeixdkjs the wejghtmatrix. The parameter matrix 406 may be denoted as VW-7, where V is the value matrix 401 andG ]^dmodei^dk jsthe weight matrix, i may indicate the index of the head. dqis the dimension of a query vector. dkis the dimension of a key vector. dvis the dimension of a value vector. In some embodiments, dq= dk= dv= dmodei / h.

[0064] The MatMul layer 440, scale layer 450, mask layer 455, SoftMax layer 460, and MatMul layer 470 are in an attention block 425 of the MHA layer. The attention block 425 may implement a scaled dot-product attention mechanism. In some embodiments, the MHA layer 400 includes a plurality of attention blocks that includes the attention block 425. For the purpose of illustration, the MHA layer 400 includes h attention blocks in FIG. 4A. Each ofthe attention blocks may have the same layers as the attention block 425. The linear block 415 and attention block 425 may constitute a head of the MHA layer 400. As the MHA layer 400 has h linear blocks and h attention blocks, the MHA layer 400 has h heads. A head may be denoted as headt= Attention QWjQ,KWiK,VW^').

[0065] A matrix multiplication operation may be performed on the parameter matrices 404 and 405 in the MatMul layer 440, which computes a score matrix 406. In some embodiments, the score matrix 406 may establish the degree of emphasis each token should place on other tokens. The 406 may include a plurality of scores. Each token may be assigned a score in relation to other tokens within the same time step. A higher score may indicate a higherfocus or emphasis. The score matrix 406 may be scaled in the scale layer 450. In some embodiments, the score matrix 406 is scaled down in the scale layer 450 by dividing the scores in the score matrix by the square root of the dimension of the query vector and the key vector, which may be denoted as fd^. The output of the scale layer 450 may be a scaled matrix 408, which includes adjusted scores. The mask layer 455 may be optional in some embodiments. The mask layer 455 may add an attention mask (which may be an input to the attention block 425) to the output of the scale layer 450 to mask out some elements in the output of the scale layer 450. The positions of the masked out elements may be defined by the attention mask. A SoftMax function may be applied on the scaled matrix 408 in the SoftMax layer 460 to compute an attention weight matrix 409. The attention weight matrix 409 includes attention weights. The attention weights may be probability values ranging from 0 to 1. The SoftMax function may emphasize high scores while diminishing low scores, which can enhance the model's ability to determine which tokens should get more attention.

[0066] In the MatMul layer 470, a matrix multiplication operation is performed on the attention weight matrix 409 computed in the SoftMax layer 460 and the parameter matrix 407 computed from value matrix 403 in the linear layer 430. The result of the matrix multiplication operation is a single-head output matrix 411, which is an output of the attention block 425.

[0067] As the MHA layer 400 has h attention blocks, there will be h single-head output matrices. The single-head output matrices are concatenated in the concatenation layer 480 to form a concatenated matrix. A linear operation (also referred to as "linearY1transformation") is performed on the concatenated matrix using a weight matrix in the linear layer 490. In some embodiments, the multi-head attention may be denoted as MultiHead Q, K, V) = Concat (head}, head2, ... , head^W0, where Concat denotes concatenation, and W° E ]hd xdmo<iei jsthe weight matrix in the linear layer 490.Example Heterogenous Computing System

[0068] FIG. 5 is a block diagram of a heterogeneous computing system 500, in accordance with various embodiments. The heterogeneous computing system 500 includes a task manager 510, a CPU 520A, a GPU 520B, and a NPU 520C. In other embodiments, alternative configurations, different or additional components may be included in the heterogeneous computing system 500. For instance, the heterogeneous computing system 500 may include multiple CPUs, GPUs, or NPUs. Also, the heterogeneous computing system 500 may include other types of processing units. Further, functionality attributed to a component of the heterogeneous computing system 500 may be accomplished by a different component included in the heterogeneous computing system 500 or a different system. For instance, functionality attributed to the task manager 510 may be accomplished by the CPU 520A. The CPU 520A, GPU 520B, and NPU 520C may be collectively referred to as heterogenous processing units 520, individually referred to as "heterogenous processing unit 520." The heterogenous processing units 520 may be implemented in separate chips. In an example, each heterogenous processing unit 520 may be implemented as a separate chip. Even though a single CPU, GPU, or NPU is used in some embodiments or examples described below, multiple CPUs, GPUs, or NPUs may be used in these embodiments or examples.

[0069] The task manager 510 manages performance of Al tasks by the heterogenous processing units 520 in the heterogeneous computing system 500. Al tasks may be tasks for various Al applications, such as language processing, speech recognition, image or video analysis, autonomous driving, project management, and so on. An Al task involves deployment of one or more DNNs. The deployment of a DNN is also referred to as DNN inference, which is a process of applying a pretrained DNN on an input to obtain a prediction made by the DNN. The Al task may include tasks in the neural network inference, such as tasks of executing neural network layers, and tasks outside the neural network inference, such as tasks of compiling the neural network, generating the input data, processing the neural network output, and so on. Taking an autonomous driving task for example, sensor data from sensors on the vehicle may be pre-processed (e.g., to removenoise, integrate sensor data from different sensors, and so on) to generate an input to a DNN. The DNN may process the input and may output a prediction, e.g., a prediction that the vehicle needs to decelerate to avoid an accident. The output of the DNN may go through a post-processing operation to generate a control signal that can trigger deacceleration of the vehicle. The task manager 510 may dynamically allocate computational tasks within the Al task to the heterogenous processing units 520 to optimize efficiency.

[0070] As shown in FIG. 5, the task manager 510 includes an interface module 530, a coarsegrained partition module 540, a fine-grained mapping module 550, and a command module 560. The task manager 510 may include fewer, more, or different components. Further, functionality attributed to a component of the task manager 510 may be accomplished by a different component included in the task manager 510 or a different module.

[0071] The interface module 530 facilitates communications of the task manager 510 with other modules or systems. In some embodiments, the interface module 530 establishes communications between the task manager 510 with an external database or system to receive data that can be used to perform Al tasks. For example, the interface module 530 may receive a user input from a client device associated with the user. The user input may be processed to generate input data for DNN inference or may be used to select a processing unit that performs DNN inference. As another example, the interface module 530 may receive images from a camera for image or video processing. As yet another example, the interface module 530 may receive sensor data from a robot for robot navigation. In some embodiments, the interface module 530 may transmit DNN input data to the heterogenous processing units 520 for DNN inference. The interface module 530 may also facilitate transmission of data used for controlling operations of the heterogenous processing units 520 to the heterogenous processing units 520. The data may be commands generated by the command module 560 or configuration descriptors generated in the compilation stage.

[0072] The coarse-grained partition module 540 partitions Al tasks into computational tasks. The coarse-grained partition module 540 may perform partitioning of an Al task across the heterogenous processing units. In some embodiments, the coarse-grained partition module 540 may identify each step in the Al task as a computational task. For example, the coarse-grained partition module 540 may identify the step of compiling the DNN as a computational task. Also, the coarse-grained partition module 540 may identifythe step of generating a DNN input as a computational task and the step of processing the DNN output to generate the Al task result as another computational task. Further, the coarse-grained partition module 540 may identify the step of executing each DNN layer as an individual computational task. Some computational tasks may be part of the DNN inference, e.g., tasks of executing DNN layers. Other computational tasks may not be part of the DNN inference. For example, a preprocessing operation may be needed to prepare the input to the DNN. As another example, a post-processing operation may be needed to generate a task result from the output of the DNN.

[0073] The coarse-grained partition module 540 may also determine whether each computational task within the Al task can be accelerated by specialized processing units. In some embodiments, the coarse-grained partition module 540 classifies computational tasks into two categories: computational tasks that cannot be accelerated by specialized processing units and computational tasks that can be accelerated by specialized processing units. For each computational task, the coarse-grained partition module 540 may determine whether it is part of DNN inference. A computational task that is part of DNN inference is a computational task in DNN inference; otherwise, it is a computational task outside DNN inference. The coarse-grained partition module 540 may determine that all computational tasks outside DNN inference cannot be accelerated by specialized processing units and map these computational tasks to the CPU 520A. The coarse-grained partition module 540 may map these computational tasks to the CPU 520A. In some embodiments, the coarse-grained partition module 540 may use historical data to determine whether to map a computational task to the CPU 520A. The historical data may represent prior knowledge. The historical data may be a user input. For instance, the heterogeneous computing system 500 may receive the historical data from a client device associated with a user. In an embodiment where the historical data indicates that CPU is historically superior for performing the same or similar task or that the same or similar task did not historically benefit from specialized hardware, the coarse-grained partition module 540 may map the computational task to the CPU 520A.

[0074] Taking object detection application for example, image preprocessing, nonmaximum suppression, and result visualization may be three components crucial for the overall functionality of the application but are not part of DNN inference and do not benefit from the specialized hardware acceleration that the GPU 520B or the NPU 520C can provide for neural network computations. The coarse-grained partition module 540 may map thesecomponents to the CPU 520A. As another example, certain control operations, such as conditional branching or dynamic control flow, can be efficiently executed on CPU due to its superior single-thread performance and control flow capabilities. In addition, LLM generation typically involves post-processing steps after LLM inference such as tokenization, detokenization, beam-search, etc. that are not parallelizable and can be efficiently handled by the CPU 520A.

[0075] For each computational task in DNN inference, the coarse-grained partition module 540 identifies processing units that are eligible for performing computational tasks in neural network inference. The eligible processing units may include the CPU 520A, the GPU 520B, the NPU 520C, or some combination thereof. In some embodiments, for each DNN layer, the coarse-grained partition module 540 determines whether one or more processing units are eligible for executing the DNN layer based on one or more characteristics of the DNN layer or Al task. The coarse-grained partition module 540 may also determine whether one or more processing units are eligible for executing the DNN layer further based on historical data. The historical data may indicate that certain layers are better suited for execution on the CPU 520A or may benefit from specialized hardware accelerator that the GPU 520B or NPU 520C can provide for DNN computations. Taking LLMs for example, although LLMs are designed to leverage the parallel processing capabilities of the GPU 520B or NPU 520C, there are certain scenarios or layers within these models where execution on the CPU 520A might be more suitable based on prior knowledge or specific characteristics. For example, LLMs often start with an embedding layer that maps tokens (words or sub-words) to highdimensional vectors. When the embedding layer is relatively large or the operation involves looking up a small number of embeddings from the large layer, it might be more efficient to perform this on the CPU 520A to avoid the overhead of transferring data to the GPU 520B or NPU 520C during inference. By using historical data, the coarse-grained partition module 540 can reduce the search space.

[0076] In some embodiments, the coarse-grained partition module 540 may identify all the available processing units, which may include various types of processing unit, and obtain an eligibility list of processing units for each DNN layer. The coarse-grained partition module 540 may prune the eligible list by removing one or more processing units. For instance, the coarse-grained partition module 540 may remove a processing unit based on historical data. The historical data may represent prior knowledge about efficiency of a type of processingunit executing the same or similar DNN layer. In some embodiments, the historical data may indicate a cost of a historical execution of a similar computational task by the processing unit. For instance, the historical data may indicate that the cost is higher than the cost of another historical execution of a similar computational task by each of the other processing units.

[0077] In some embodiments (e.g., embodiments in which prior CPU affinity indication does not exist), the coarse-grained partition module 540 may check the compatibility of the DNN layer with each available processing unit. As the CPU 520A is a general-purpose processor, the coarse-grained partition module 540 may include the CPU 520A in the eligibility list for each DNN layer by default. In some embodiments, it may be unnecessary that every layer will be supported on the GPU 520B and NPU 520C. Based on the prior knowledge (such as hardware or software specifications) about accelerator supported layers, the coarse-grained partition module 540 may add every processing unit that supports the layer to the eligibility list and exclude other processing units. For instance, the coarse-grained partition module 540 may refine the search space for subsequent processing by excluding the CPU 520A from consideration when the GPU 520B or NPU 520C is identified as a more suitable option based on prior knowledge. In some embodiments, DNN layers forming the core computational blocks in DNNs, such as convolutional layers, MHA layers, fully-connected layers, etc., are better suited for the GPU 520B or NPU 520C as these layers can be accelerated resulting in better overall performance of the DNN. The pruning step may be crucial for streamlining the decision-making process in the subsequent fine-grained mapping stage, as it reduces the search complexity.

[0078] After the coarse-grained partition module 540 finds the eligibility list the entire DNN, the coarse-grained partition module 540 may provide the eligibility list to the fine-grained mapping module 550 for fine-grained mapping. In embodiments where there are multiple DNNs chained together in the Al task, the fine-grained mapping module 550 may be called for each DNN separately.

[0079] The fine-grained mapping module 550 may select an optimal processing unit for each computational task in neural network inference, e.g., based on comprehensive cost models and user preference. The mapping generated by the fine-grained mapping module 550 (e.g., mapping of a DNN layer to the optimal processing unit for the DNN layer) may be utilized to guide the DNN execution across the heterogeneous processing units duringinference. In some embodiments, a computational task may be the task of executing a single DNN layer. The fine-grained mapping module 550 may leverage pre-developed cost models Ccpu, GPU> CNPU t° identify the most efficient processing unit for executing each layer from the eligibility list of the layer.

[0080] In some embodiments, for each processing unit in the eligible list, the fine-grained mapping module 550 may estimate various types of costs of the processing unit performing the computational task. The costs may include a latency cost indicating an estimation of the latency caused by performing the computational task by the processing unit, an energy cost indicating an estimation of energy consumed by the processing unit for performing the computational task, a performance cost indicating an estimation of a performance of the processing unit for performing the computational task, and so on.

[0081] In some embodiments, the fine-grained mapping module 550 may input data indicating one or more layer configurations into the cost models. The cost models may output estimates for latency LIP, energy consumption EIP, and performance per watt P[p, which are pivotal metrics for decision-making. Examples of the layer configurations include input activation shape (which may include batch size), filter or kernel shape, activation datatype, weight datatype, type of layer, computations inside layer, other layer configurations, or some combinations thereof. For example, cost model for the CPU 520A may use event-based power model to provide performance and energy consumption of any workload on the CPU 520A. Cost model for the GPU 520B provides performance and energy usage for the GPU 520B computing workloads. Cost model for the NPU 520C may provide performance and energy consumption for any layer executing on the NPU 520C. For other accelerators, simulation tools may also be used for obtaining similar information. Along with cost models for specific types of processing units, the fine-grained mapping module 550 may use a pretrained DNN to predict the execution time, power consumption, and performance / watt of each layer on each accelerator.

[0082] In some embodiments, the fine-grained mapping module 550 may select one cost type, e.g., based on a user selection. The fine-grained mapping module 550 may receive a user input indicating a preference of a user for a cost type and select the cost type based on the user input. The fine-grained mapping module 550 may further compare the costs of the selected type that are estimated for the processing units in the pruned eligible list and select a processing unit based on the comparison. For instance, the fine-grained mapping module550 may select the processing unit that has the lowest cost or best performance. The finegrained mapping module 550 may map the DNN layer to the selected processing unit.

[0083] In some embodiments, the user's preference may prioritize latency, throughput, or energy efficiency to device the optimal mapping. For running Al models, different users might have varying preferences depending on their specific needs, constraints, and objectives. Some Al applications may have high latency sensitivity. In an example, for realtime applications such as voice assistants or live translations, users may prioritize low latency to ensure a seamless and responsive user experience. In such cases, the preference would be to minimize the time it takes to compute each inference, possibly at the expense of higher energy consumption. For other Al applications, users may prefer energy efficiency. For instance, in scenarios where power consumption is a concern, such as battery-powered edge computing devices like mobile phones, drones, laptops, etc., users may prefer energyefficient execution. This preference may aim to minimize the energy required to perform computations, which might allow for slower response times if necessary. These two constraints may not be mutually exclusive, as low latency and low energy might be obtained for the same processing unit. Users of some Al applications may prefer throughput maximization. For instance, for batch processing tasks, such as video processing, users might prioritize high throughput. The goal may be to process the largest amount of data in the shortest amount of time, regardless of the power consumption of individual inferences. Balanced performance may be preferred by users in some scenarios. For instance, some users may seek a balance between latency, energy, and throughput, aiming for a solution that provides reasonable performance across all metrics without significant trade-offs. The user may also choose to not provide any preference when the balanced case is selected by default. In this case, performance / watt may be considered as the metric as it considers both latency and energy.

[0084] In some embodiments, the fine-grained mapping module 550 may also estimate a switching cost of switching from another processing unit executing a precedent DNN layer ("previously used processing unit") to the processing unit selected from the eligible list for the current layer ("new processing unit"). In some embodiments, the fine-grained mapping module 550 estimates the total cost for the particular layer across the system. The total cost may account for the costs associated with switching between processing units, including latency and energy overheads, in addition to processing-unit-level cost. For instance, theswitching cost may include the cost of transferring data from the other processing unit to the processing unit. The fine-grained mapping module 550 may determine whether to switch to the processing unit based on the switching cost. When the previously used processing unit is the same as the new processing unit identified for the current layer, no switching costs are incurred.

[0085] When the previously used processing unit is different from (e.g., of a different type from) the new processing unit, the fine-grained mapping module 550 may estimate the total cost of executing the layer, including the switching overhead. This cost encumbers data transfer costs from previously used processing unit to the new processing unit. When two layers are executed on the same processing unit and the size of output of last layer is small enough to be stored in local memory (e.g., local static random-access memory (SRAM)) of the processing unit, then the extra cost incurred in case of non-repeating processing unit may include the cost to write the data to dynamic random-access memories (DRAM) by the previously used processing unit and cost to read from DRAM by the new processing unit. In some embodiments, it may be imperative that all the processing units have a shared memory space in DRAM for DNN inference, as disjoint space can result in another extra copy in DRAM, increasing the switching cost. In addition, switching between processing units may require synchronization mechanisms to ensure data consistency and correct execution order. This can introduce delays and additional computational overhead, and these are also included in the switching cost. In some embodiments, the fine-grained mapping module 550 does not estimate or consider switching cost is considered for the first DNN layer.

[0086] The fine-grained mapping module 550 may decide the optimal processing unit mapping for the layer is based on a comparison of the total costs with and without switching, ensuring that the chosen mapping minimizes latency or energy consumption as per the user's preference. After the fine-grained mapping module 550 determines to switch, the fine-grained mapping module 550 may map the DNN layer to the new processing unit. Switching costs may be important considerations in systems that allocate tasks across heterogeneous processing units. Minimizing these costs can be crucial for achieving efficient and effective computation. Therefore, this comparison ensures that the fine-grained mapping module 550 selects the optimal mapping after careful consideration of above constraints.

[0087] This 2-stage (i.e., coarse-grained partition and fine-grained mapping) approach to DNN execution across heterogeneous processing units can significantly enhance the efficiency and adaptability of computational resource utilization. By considering both the application-level device preference, prior knowledge about layer compatibility, and computational characteristics of DNN layers, this approach can ensure that each layer of the DNN is executed on the processing unit that offers the best balance of speed and energy efficiency, leading to improved overall performance of the application. After all the layers in a DNN are evaluated and corresponding mapping has been obtained, the coarse-grained partition module 540 or fine-grained mapping module 550 may find out mapping for next DNN.

[0088] The command module 560 command processing units mapped to computational tasks by the coarse-grained partition module 540 or fine-grained mapping module 550 to perform the computational tasks. In an example in which the coarse-grained partition module 540 maps a computational task to the CPU 520A, the command module 560 commands the CPU 520A to perform the computational task. The command module 560 may facilitate transfer of data needed to perform the computational task to the CPU 520A. For instance, the command module 560 may command the CPU 520A to read the data from a memory or command another processing unit to transfer the data to the CPU 520A.

[0089] In another example, when after the fine-grained mapping module 550 maps a DNN layer to the GPU 520B or NPU 520C, the command module 560 commands the GPU 520B or NPU 520C to execute the DNN layer. The command module 560 may facilitate transfer of data needed to execute the DNN layer, such as input data of the DNN layer, to the GPU 520B or NPU 520C. For instance, the command module 560 may command the GPU 520B or NPU 520C to read the data from a memory or command another processing unit (e.g., a previously used processing unit) to transfer the data to the GPU 520B or NPU 520C. The data may include configuration descriptors generated from compiling the DNN. Components of the GPU 520B or NPU 520C may be configured by the configuration descriptors for the DNN layer execution.

[0090] FIG. 6 illustrates an example process of dynamic allocation of Al workloads among heterogeneous processing units, in accordance with various embodiments. The process may be performed by the task manager 510 in FIG. 5. Although the process is described with reference to the flowchart illustrated in FIG. 6, many other methods for dynamic allocationof Al workloads may alternatively be used. For example, the order of execution of the steps in FIG. 6 may be changed. As another example, some of the steps may be changed, eliminated, or combined.

[0091] In Step 610, an Al task is partitioned into computational tasks. In Step 820, for each computational task, it is determined whetherthe computational task is part of neural network interface. In embodiments in which the computational task is not part of neural network interface, the process proceeds to Step 630, in which the computational task is allocated to a CPU.

[0092] In embodiments in which the computational task is part of neural network interface, the process proceeds to Step 640, eligible processing units are found for the computational task. The eligible processing units may include one or more CPUs, GPUs, NPUs, other types of processing units, or some combination thereof. It may be determined that a processing unit is eligible for performing the computational task when the processing unit has the processing elements that can perform computations in the computational task.

[0093] In Step 650, prior knowledge is obtained. The prior knowledge may indicate that one or more eligible processing units may be better at performing the computational task than one or more other eligible processing units. For instance, the cost for performing the computational task may be lower. The list of eligible processing units ("eligible list") is pruned based on the prior knowledge in Step 660. For instance, the one or more processing units that are better would be kept in the eligible list, while the other processing unit(s) would be removed from the eligible list.

[0094] In Step 670, the best processing unit in the pruned eligible list is selected. The best processing unit may be selected based on estimated costs of the processing units in the pruned eligible list performing the computational task. In some embodiments, multiple types of costs (e.g., latency cost, energy cost, performance / watt cost, throughput cost, etc.) may be estimated for each processing unit, e.g., by inputting data indicating characteristics of the computational task or the processing unit into multiple cost models. Each cost model may output the estimated cost of a particular type. One of the cost types may be selected based on a user preference. The costs of the selected type for all the processing units may be compared. The processing unit with the lowest cost may be considered as the best processing unit.

[0095] In Step 680, it is determined whether another processing unit (e.g., a processing unit that is different from the best processing unit found in Step 670) is used for the previous computational task. In embodiments in which the best processing unit is used for the previous computational task, the process proceeds to Step 695, in which the computational task is allocated to the best processing unit found in Step 670.

[0096] In embodiments in which the other processing unit is used for the previous computational task, the process proceeds to Step 685, in which it is determined whether to switch processing unit. For instance, a total cost may be determined by aggregating the cost for the best processing unit performing the computational task plus a cost of switching from the other processing unit to the best processing unit. The total cost may be compared with the cost of the other processing unit performing the computation task. When the total cost is the same or greater, it is determined not to switch to the best processing unit. The process proceeds to Step 690, in which the computational task is allocated to the other processing unit. When the total cost is lower, it is determined to switch to the best processing unit. The process proceeds to Step 695, in which the computational task is allocated to the best processing unit found in Step 670.

[0097] After the computational task is allocated. The process may be repeated for subsequent computational tasks in the Al task. In some embodiments, the process may be performed online, e.g., during the performance of the Al tasks. For instance, the process may be performed after the previous computational task within the Al task is started or complete but before the current computational task is started. After all the computational tasks within the Al task are allocated, the process may be repeated for computational tasks within another Al task.

[0098] FIG. 7 illustrates partitioning of an example DNN 700, in accordance with various embodiments. For the purpose of illustration and simplification, FIG. 7 shows a DNN 700 that includes blocks 710A-710D, an IF operation 720, and an unsupported operation 730. Each of the blocks 710A-710D may be at least part of a layer in the DNN 700. The IF operation 720 or unsupported operation 730 may each be at least part of a layer. The DNN 700 receives an input 701, which is used for the DNN inference. The DNN inference includes execution of the blocks 710A-710D, IF operation 720, and unsupported operation 730. In some embodiments, the blocks 710A-710D, IF operation 720 and, unsupported operation730 are referred to as kernels in the DNN 700. In some embodiments, the DNN inference may be partitioned into six computational tasks of executing the seven kernels.

[0099] The computational tasks may be dynamically allocated across heterogenous processing units, which can facilitate seamless integration with the DNN optimization and deployment toolkit (e.g., OpenVINO) to leverage pre-optimized kernels and workflows. It can reduce the time and effort required to implement Al applications with the DNN. In some embodiments, the DNN is prepared for execution, which may involve analyzing the DNN to identify computational tasks that are well-suited for offloading to GPU or NPU. A just-in- time (JIT) compilation approach may be used to tailor these identified computational tasks specifically for the GPU or NPU architecture. The compiled code, along with potential caching mechanisms for faster loading, may be transferred to the GPU or NPU for execution.

[0100] To achieve seamless integration with the existing runtime platform, the computational tasks to be performed by GPU or NPU (e.g., the computational tasks of executing the blocks 710A-710D) may be swapped with their original CPU counterpart. In the embodiments of FIG. 7, not all operations in the DNN are compatible with the GPU or NPU. A key strength of the dynamic allocation approach lies in its ability to identify and offload supported computational tasks (e.g., the computational tasks of executing the blocks 710A-710D) to the GPU or NPU, while excluding the other computational tasks not natively supported (e.g., the computational tasks of executing the IF operation 720 and unsupported operation 730). This enables a smooth heterogeneous inference where the CPU inference can handle control flow and unsupported operations while computeintensive layers are offloaded to GPU or NPU. This selective offloading strategy leverages strengths of both CPU and specialized processing unit, resulting in a balanced and effective resource utilization for LLM workloads running on Al personal computers.

[0101] Various neural network operations can be allocated to specialized processing unit that can accelerate the neural network operations, such as GPU or NPU. For example, a layer with matrix multiplication (MatMul), such as floating-point MatMul, may implement y = x * wT, in which x denotes the input to the layer, w denotes weight of the layer, and y denotes the output of the layer. Such a layer may be the backbone of LLM enabling as it is one of the biggest contributors on overall network latency. Weight w may be treated as an input to the layer so the same kernel can be reused multiple time during the same DNN inference. As another example, a layer with quantized MatMul may implement y = S *( x * wTin which S denotes the quantization scale, x denotes the input to the layer, w denotes weight of the layer, and y denotes the output of the layer. The layer may be the (symmetrically) quantized version of the previous layer. It can be important to support quantization as the model performances heavily depends on the model size and NPU has specific hardware acceleration to support low precision and mixed precision inference. As yet another example, LLaMA FeedFoward block, which may be part of a DNN layer, can be offloaded to GPU or NPU while another part of the layer may be allocated to CPU. As the model inference requires numerous instances of this block, each requiring distinct weights may be provided as inputs during runtime saving on compilation time and system memory. This is not a comprehensive list. Other types of neural network operations or DNN layers may be offloaded to GPU or NPU. The size of an individual kernel can vary from single layer to blocks to full decoder layers up to the entire model.

[0102] FIG. 8 illustrates partitioning of another example DNN 800, in accordance with various embodiments. For the purpose of illustration and simplification, the DNN 800 may be a MatMul-based Llama model. The DNN 800 includes various neural network operations, such as batch normalization, linear transformation, flash attention, accumulation, and SwigGLU activation function. Each neural network operation is represented by a box in FIG.8. The DNN 800 receives an input 801, which is used for the DNN inference. The DNN inference includes execution of neural network operations. In some embodiments, the neural network operations are referred to as kernels in the DNN 800.

[0103] In some embodiments, the DNN inference may be partitioned into nine computational tasks of executing the nine neural network operations, respectively. The computational tasks may be dynamically allocated across heterogenous processing units. In an example, the two batch normalization operations and the two accumulation operations may be allocated to a CPU; the three linear transformation operations and the SwigGLU activation function may be allocated to a NPU; and the flash attention operation may be allocated to a GPU.

[0104] FIG. 9 illustrates partitioning of yet another example DNN 900, in accordance with various embodiments. The DNN 900 may be a block-based Llama model. For the purpose of illustration and simplification, the DNN 900 includes a block 910 for batch normalization, a block 920 for multi-head attention, a block 930 for accumulation, a block 940 for batch normalization, a block 950 for feed forward, and another block 960 for accumulation. TheDNN 900 receives an input 901, which is used for the DNN inference. The DNN inference includes execution of the six blocks. In some embodiments, the blocks are referred to as kernels in the DNN 900.

[0105] In some embodiments, the DNN inference may be partitioned into six computational tasks of executing the six blocks, respectively. The computational tasks may be dynamically allocated across heterogenous processing units. In an example, the block 910, block 930, block 940, and block 960 may be allocated to a CPU; while the block 920 and the block 950 may be allocated to an NPU. There may be other forms of Llama models, such as decoderbased Llama models and graph-based Llama models. A decoder-based Llama model may include a plurality of decoders, which may be allocated to an NPU. The inference of a graphbased Llama model may also be carried out by an NPU.

[0106] FIG. 10 illustrates coarse-grained partition of an Al task across heterogenous processing units with graph mode execution, in accordance with various embodiments. For the purpose of illustration and simplicity, the heterogenous processing units includes a CPU 1010 and eligible processing units 1020. The CPU 1010 performs computational tasks including loading a DNN 1001 and compiles the DNN 1001. In some embodiments, the CPU 1010 may execute instructions stored in a memory associated with the CPU 1010 to compile the DNN 1001. The CPU 1010, or a module implemented by the CPU 1010, may be a DNN compiler.

[0107] In some embodiments, the DNN compiler compiles information of the DNN 1001 to executable instructions that can be executed, e.g., by the eligible processing units 1020, to carry out neural network operations in DNNs. The DNN compiler may generate an executable DNN that can be at least partially executed by the eligible processing units 1020. In some embodiments, the DNN compiler may generate a graph representing the DNN 1001. The graph may include nodes and edges. A node may represent a specific neural network operation in the DNN 1001. An edge may connect two nodes and represent a connection between the two corresponding neural network operations. In an example, an edge may encode a tensor that flows from one of the neural network operations to the other neural network operation. The tensor may be an output tensor of the first neural network operation and an input tensor of the second neural network operation. The edge may encode one or more attributes of the tensor, such as size, shape, storage format, and so on. The DNN compiler may use the graph to generate instructions (e.g., compilationdescriptors). The instructions would be executed by components of the eligible processing units 1020 to execute the DNN.

[0108] The CPU 1010 also processes input data 1002 to generate layer input 1003. This is referred to as pre-process. The eligible processing units 1020, or one of the eligible processing units 1020, is allocated to execute the DNN layer using the layer input 1003, which generates layer output 1004. The CPU 1010 may process layer output 1004 to generate output data 1005, which may be the result of the Al task. This is referred to as post-process. In some embodiments, the pre-process or post-process may be skipped. The input data 1002 may be input directly into the DNN 1001, or the output of the DNN 1001 may be directly used as the result of the Al task.

[0109] The eligible processing units 1020, or one of the eligible processing units 1020, may execute multiple or all layers in the DNN 1001. In some embodiments, at least part of a DNN layer is a parameter-free kernel with the weights being treated as inputs to the kernel. This streamlined approach can simplify the process of offloading kernels to the NPU, ensuring a smooth and efficient execution pipeline. As many DNNs (e.g., LLMs) are made of the same kernels in a repeated fashion, this approach can be very effective as it can leverage caching mechanisms in DNN optimization and deployment toolkits and avoid re-compiling the same sub-networks.

[0110] In the embodiments of FIG. 10, the DNN execution may be graph mode execution. Many DNN optimization and deployment toolkits (e.g., OpenVINO) work in graph mode, where it constructs a computational graph beforehand, e.g., by compiling the DNN. The graph may include nodes and edges. A node may represent a neural network operation, while each edge may indicate data flow between neural network operations. The graph may outline the sequence of operations in the DNN, enabling optimizations like pre-computing intermediate values and efficient memory management. While it may be less intuitive and hard to implement in the compiler, graph mode can translate generally to faster execution. This performance gain can stem from the ability of the compiler to plan and optimize the entire computation rather than processing each step independently. However, it can be much harder for a compiler to effectively optimize in a short amount of time massive networks like local LLMs. Also, static graph does not allow for efficient control flow or dynamic shapes inference, which needs to be assumed overestimated hindering the graph mode performance advantage.

[0111] FIG. 11 illustrates coarse-grained partition of an Al task across heterogenous processing units with eager mode execution, in accordance with various embodiments. For the purpose of illustration and simplicity, the heterogenous processing units includes a CPU 1110 and eligible processing units 1120. The CPU 1110 performs computational tasks including loading a DNN 1101 and performing JIT compilation of the DNN 1101. The CPU 1110 also receives input data 1102 and provides input for each block, which provides layer input 1103 for layer execution by at least one of the eligible processing units 1120. The layer weights 1104 are also used as input data for layer execution so the kernel can be parameter free. The layer output 1105 computed form the layer execution is provided to the CPU 1110, which may be used for the next layer execution. After the inference is done, the CPU 1110 may provide output data 1106, which may be the result of the Al task. The eligible processing units 1120, or one of the eligible processing units 1120, is allocated to execute the DNN layer using the layer input 1103, which generates layer output 1105. The CPU 1110 may process the layer output 1105 to generate output data 1106, which may be the result of the Al task.

[0112] Kernel-based inference may be an implementation of DNN eager mode execution. This inference mode may directly run a model layer-by-layer, resembling standard sequential programming. This can make it intuitive and easier to debug, allowing for dynamic control flow using conditional statements and loops. However, this flexibility may come at a cost. Each operation may run independently, limiting compiler optimizations. In some embodiments, the CPU 1110 can manage the overall program flow. The CPU 1110 may retain control over the dynamic control flow, dictating the sequence of execution for the kernels. This can ensure that the computations are performed in the correct order and the Al model functions as intended. It also enables dynamic control flow as the individual kernels are JIT compiled statically and executed on demand. Graph mode execution can reign supreme for performance-critical tasks, while eager mode offers a more user-friendly development environment.

[0113] FIG. 12 illustrates model execution with both graph mode and eager mode, in accordance with various embodiments. Forthe purpose of illustration and simplicity, FIG. 12 shows a DNN 1200 that includes a backbone 1210, an encoder 1220, a decoder 1230, and heads 1240 (individually referred to as "head 1240"). The backbone 1210 includes a CNN 1213 that generates a feature map 1201. The CNN 1213 may extract the feature map 1201from an image. The feature map 1201 may be a two-dimensional orthree-dimensional representation of the image. The feature map 1201 is input into the encoder 1220. The encoder 1220 also receives positional encoding tensors 1202, individually referred to as "positional encoding tensors 1202." The DNN 1200, e.g., an accumulator 1215 in the DNN 1200, may flatten and supplement the feature map 1201 with the positional encoding tensors 1202 before passing it into the encoder 1220. The encoder outputs 1203 are input into the decoder 1230. The decoder 1230 may take as input a small, fixed number of learned positional embeddings 1204, such as object queries, and additionally attends to the encoder outputs 1203. The decoder outputs 1205 are provided to the heads 1240 to generate predictions 1206, e.g., predicted tokens.

[0114] In some embodiments, both execution mode and graph mode may be used, e.g., by the task manager 510 for dynamic allocation of Al task workloads. The task manager 510 may select what part of the DNN 1200 runs where and in which mode. This can ensure maximum flexibility and performance at the same time. In some embodiments, the backbone 1210 may be compiled in the graph mode, which can fully exploit the static shape inference and its speed while keeping the encoder 1220 and decoder 1230 in eager mode while exploiting their dynamic nature. In some embodiments, the encoder 1220 may also be compiled in graph mode, e.g., when the entire compiled graph is static, to increase the overall performance of the Al task. This can lead to maximum performance while running different parts of the network with their optimal execution mode. The task manager 510 can automatically identify the dynamic part of the DNN 1200 and autonomously decide on the partitioning, compilation mode, and optimal device selection.Example Methods of Dynamic Allocation of Al Task Workloads

[0115] FIG. 13 is a flowchart of a method for dynamic allocation of Al tasks workloads, in accordance with various embodiments. The method 1300 may be performed by the task manager 510 in FIG. 5. Although the method 1300 is described with reference to the flowchart illustrated in FIG. 13, many other methods for dynamic allocation of Al tasks workloads may alternatively be used. For example, the order of execution of the steps in FIG. 13 may be changed. As another example, some of the steps may be changed, eliminated, or combined.

[0116] The task manager 510 determines 1310 whether a computational task within an Al task is part of neural network inference. The Al task is to be performed based on the neuralnetwork inference. In some embodiments, the computational task is a task of compiling a neural network to be deployed for the neural network inference, and the task is not part of the neural network inference. In some embodiments, the computational task is a task of generating an input to a neural network to be deployed for the neural network inference or processing an output of the neural network, and the task is not part of the neural network inference. In some embodiments, the computational task is a task of executing a layer in a neural network, and the task is part of the neural network inference.

[0117] The task manager 510 commands 1320 a CPU to perform the computational task, in response to determining that the computational task is not part of the neural network inference.

[0118] In response to determining that the computational task is part of the neural network inference, the task manager 510 identifies 1330 a plurality of processing units eligible for performing the computational task. In some embodiments, the plurality of processing units comprises the CPU.

[0119] In some embodiments, the task manager 510 identifies a group of processing units eligible for performing the computational task. The group of processing units includes the plurality of processing units and an additional processing unit. The task manager 510 receives historical data indicating a cost of a historical execution of a similar computational task by the additional processing unit. In some embodiments, the historical data indicates that the cost of the historical execution of the similar computational task by the additional processing unit is higher than a cost of a historical execution of a similar computational task by each of the plurality of processing units. The task manager 510 removes the additional processing unit from the group of processing units based on the historical data. In some embodiments, the task manager 510 receives data indicating one or more characteristics of the computational task. The additional processing unit is removed further based on the data.

[0120] The task manager 510 estimates 1340 costs of the plurality of processing units performing the computational task. A cost indicates a measurement of latency, consumed energy, or performance of a processing unit performing the computational task. In some embodiments, for each processing unit, the task manager 510 estimates costs of different cost types using a plurality of cost models. The task manager 510 selects a cost from thecosts based on a user input. The user input indicates a preference of a user for one of the different cost types.

[0121] The task manager 510 selects 1350 a processing unit from the plurality of processing units based on the estimated costs. In some embodiments, the task manager 510 estimates a switching cost indicating a cost of switching from a different processing unit to the processing unit for executing the layer. A precedent layer in the neural network is executed by the different processing unit. The task manager 510 selects the processing unit further based on the switching cost.

[0122] The task manager 510 commands 1360 the selected processing unit to perform the computational task. In some embodiments, the task manager 510 commands another processing unit to provide one or more configuration descriptors to the selected processing unit. One or more components of the selected processing unit are to be configured by the one or more configuration descriptors for performing the computational task.

[0123] FIG. 14 is a flowchart of another method for dynamic allocation of Al tasks workloads, in accordance with various embodiments. The method 1400 may be performed by the task manager 510 in FIG. 5. Although the method 1400 is described with reference to the flowchart illustrated in FIG. 14, many other methods for dynamic allocation of Al tasks workloads may alternatively be used. For example, the order of execution of the steps in FIG. 14 may be changed. As another example, some of the steps may be changed, eliminated, or combined.

[0124] The task manager 510 estimates 1410 a cost of a first processing unit executing a first layer in a neural network. In some embodiments, the task manager 510 selects, based on the estimated cost of the first processing unit, the first processing unit from a group of processing units eligible for executing the first layer. In some embodiments, the estimated cost of the first processing unit indicates an estimated latency cost of the first processing unit executing the first layer, estimated energy cost of the first processing unit executing the first layer, or estimated performance cost of the first processing unit executing the first layer.

[0125] In some embodiments, the task manager 510 estimates costs of different cost types for the first processing unit. The task manager 510 receives a user input indicating a selection of a cost type. For instance, the task manager 510 receives the user input from a client device associated with a user. The task manager 510 selects the estimated cost of thefirst processing unit from the estimated costs of the different cost types based on the user input.

[0126] In some embodiments, the task manager 510 determines whether a second layer in the neural network is executed by the first processing unit or a second processing unit. In some embodiments, the second layer is executed before the first processing unit is selected by the task manager 510 for executing the first layer. In some embodiments, the task manager 510 commands the first processing unit to execute the first layer, in response to determining that the second layer is executed by the first processing unit.

[0127] The task manager 510 estimates 1420 a cost of a second processing unit executing the first layer. In some embodiments, the task manager 510 estimates costs of different cost types for the second processing unit. The task manager 510 receives a user input indicating a selection of a cost type. For instance, the task manager 510 receives the user input from a client device associated with a user. The task manager 510 selects the estimated cost of the second processing unit from the estimated costs of the different cost types based on the user input.

[0128] The task manager 510 estimates 1430 a switching cost indicating a cost of switching from the second processing unit to the first processing unit for executing the first layer, in response to determining that the second layer is executed by the second processing unit. In some embodiments, the task manager 510 receives a user input indicating a preference of a user for minimizing a type of overhead to be caused by switching from the second processing unit to the first processing unit. The task manager 510 selects a switching cost model developed for estimating the type of overhead. The task manager 510 estimates the switching cost using the switching cost model. In some embodiments, the switching cost includes a cost of transferring input data of the first layer from a memory of the second processing unit to a memory of the first processing unit.

[0129] The task manager 510 determines 1440 whether the estimated cost of the second processing unit is greater than an aggregation of the estimated cost of the first processing unit and the switching cost. In some embodiments, the task manager 510 commands the second processing unit to execute the first layer, in response to determining that the estimated cost of the second processing unit is not greater than an aggregation.

[0130] The task manager 510 causes 1450 the first processing unit to execute the first layer, in response to determining that the estimated cost of the second processing unit is greaterthan an aggregation. In some embodiments, the task manager 510 commands a third processing unit to provide one or more configuration descriptors to the first processing unit. One or more components of the first processing unit are to be configured by the one or more configuration descriptors for executing the first layer.Example DNN Accelerator

[0131] FIG. 15 is a block diagram of a DNN accelerator 1500, in accordance with various embodiments. The DNN accelerator 1500 can execute DNNs. For instance, the DNN accelerator 1500 can execute a DNN by running deep learning operations in the DNN. The process of carrying out a deep learning operation is also referred to as a process of executing the deep learning operation or a process of performing the deep learning operation. The execution of the DNN may be for training the DNN or for using the DNN to perform Al tasks. The DNN accelerator 1500 may be an example of the NPU 520C in FIG. 5. In some embodiments, the DNN accelerator 1500 may perform computational tasks in DNN inference based on commands from the task manager 510. The DNN accelerator 1500 may be an example of the eligible processing units 1020 in FIG. 10 or an example of the NPU 520C in FIG. 5.

[0132] As shown in FIG. 15, the DNN accelerator 1500 includes a memory 1510, a DMA (direct memory access) engine 1520, and data processing units 1530 (individually referred to as "data processing unit 1530"). In other embodiments, alternative configurations, different or additional components may be included in the DNN accelerator 1500. For example, the DNN accelerator 1500 may include more than one memory 1510 or DMA engine 1520. As another example, the DNN accelerator 1500 may include a single data processing unit 1530. Further, functionality attributed to a component of the DNN accelerator 1500 may be accomplished by a different component included in the DNN accelerator 1500 or by a different system. A component of the DNN accelerator 1500 may be implemented in hardware, software, firmware, or some combination thereof.

[0133] The memory 1510 stores data associated with deep learning operations performed by the DNN accelerator 1500. The memory 1510 may be a system memory. In some embodiments, the memory 1510 includes a DRAM. When the DNN accelerator 1500 executes operations in transformer models, at least part of the memory 1510 may be used to implement KV caches, such as self-attention KV caches and cross-attention KV caches, in the transformer models. The KV caches may be updated during inference of thetransformation model. Layout of data in the KV caches may be determined to optimize the efficiency of the DNN accelerator 1501.

[0134] In some embodiments, the memory 1510 may store data to be used by the data processing units 1530 for DNN execution. The memory 1510 may store weights, such as weights of convolutional layers, which are determined by training DNNs. The memory 1510 may further store inputs to DNN layers or outputs of DNN layers, such as data generated by the data processing units 1530 from performing deep learning operations in DNNs. Example deep learning operations include convolutions (also referred to as "convolutional operations"), layer normalization operations, SoftMax operations, matrix multiplication operations, pooling operations, elementwise operations, activation functions, other types of deep learning operations, or some combination thereof. The memory 1510 may be a main memory of the DNN accelerator 1500. In some embodiments, the memory 1510 includes one or more DRAMs.

[0135] The DMA engine 1520 facilitates data transfer between the memory 1510 and local memories of the data processing units 1530. For example, the DMA engine 1520 can read data from the memory 1510 and write data into a local memory of a data processing unit 1530. As another example, the DMA engine 1520 can read data from a local memory of a data processing unit 1530and write data into the memory 1510. The DMA engine 1520 provides a DMA feature that allows the data processing unit 1530 to initiate data transfer between the memory 1510 and the local memories of the data processing units 1530 and to perform other operations while the data transfer is being conducted. In some embodiments, the DMA engine 1520 may read tensors from the memory 1510, modify the tensors in a way that is optimized for the data processing unit 1530 before it writes the tensors into the local memories of the data processing units 1530.

[0136] The data processing units 1530 perform deep learning operations in DNNs. For instance, a data processing unit 1530 may execute a DNN layer by running one or more deep learning operations in the DNN layer. A data processing unit 1530 may execute a layer, or a portion of a layer, at a time. In some embodiments, the operations of the DNN layers may be run by multiple data processing units 1530 in parallel. For instance, multiple data processing units 1530 may each perform a portion of a workload for a deep learning operation. Data may be shared between the data processing units 1530. A data processing unit 1530 may also be referred to as a compute block or compute tile.

[0137] The data processing units 1530 may be capable of running various types of deep learning operations, such as convolution, layer normalization, SoftMax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on. Deep learning operations performed by the data processing units 1530 include tensor operations, i.e., operations whose inputs are tensors or operations whose outputs are tensors. In an example, the data processing unit 1530 receives an input tensor and one or more convolutional kernels and performs a convolution with the input tensor and convolutional kernels. The result of the convolution may be an output tensor, which can be further computed, e.g., by the data processing unit 1530 or another data processing unit 1530.

[0138] In the embodiments of FIG. 15, each data processing unit 1530 includes a local memory 1540, a sparsity mode module 1550, a load module 1560, a processing engine 1570, a post-processing engine 1580, and a drain module 1590. Some or all the components of the data processing unit 1530 can be implemented on the same chip. In other embodiments, alternative configurations, different or additional components may be included in the data processing unit 1530. Further, functionality attributed to a component of the data processing unit 1530 may be accomplished by a different component included in the data processing unit 1530, a different data processing unit 1530, another component of the DNN accelerator 1500, or a different system. A component of the data processing unit 1530 may be implemented in hardware, software, firmware, or some combination thereof.

[0139] The local memory 1540 is local to the corresponding data processing unit 1530. In the embodiments of FIG. 15, the local memory 1540 is inside the data processing unit 1530. In other embodiments, the local memory 1540 may be outside the data processing unit 1530. Data in the local memory 1540 may be transferred to or from the memory 1510, e.g., through the DMA engine 1520. For instance, KV caches may be copied from the memory 1510 to the local memory 1540. In some embodiments, data in the local memory 1540 may be transferred to or from the local memory of another data processing unit 1530. The local memory 1540 may store data received, used, or generated by the sparsity mode module 1550, the load module 1560, the processing engine 1570, the post-processing engine 1580, or the drain module 1590. Examples of the data may include input activations, weights, output activations, sparsity bitmaps, and so on.

[0140] In some embodiments, the local memory 1540 may store tensors to be processed by the processing engine 1570 or the post-processing engine 1580. The tensors may be inputtensors of deep learning operations. The local memory 1540 may also store tensors generated by the processing engine 1570 or the post-processing engine 1580. The tensors may be output tensors of deep learning operations. The layout of data points of a tensor in the local memory 1540 may depend on the format in which the tensor is stored.

[0141] In some embodiments, the local memory 1540 may store dense tensors (e.g., dense activation tensors, dense weight tensors, etc.), sparse tensors (e.g., sparse activation tensors, sparse weight tensors, etc.), and so on. A dense tensor may be a tensor from which zero-valued elements (if any) are not removed. A dense tensor may be converted to a sparse tensor by removing one or more zero-valued elements in the dense tensor. A sparse tensor may also be referred to as a compressed tensor or packed tensor. The process of converting a dense tensor to a sparse tensor may be referred to as sparsity encoding. Sparsity encoding may also generate a sparsity tensor. Each element in the sparsity tensor may correspond to a different element in the dense tensor and indicate whether the element in the dense tensor is zero or not. The sparsity tensor may indicate positions of elements of the sparse tensor in the dense tensor. The sparsity tensor may be a sparsity bitmap, each element of which is a bit. A sparse tensor may be converted to a dense tensor through a densifying process, in which one or more zeros may be added to the sparse tensor based on the sparsity tensor.

[0142] In some embodiments, the local memory 1540 includes one or more SRAMs. The local memory 1540 may be byte-addressable, and each memory address identifies a single byte (eight bits) of storage. In some embodiments, the local memory 1540 may include memory banks. The number of data banks in the local memory 1540 may be 16, 64, 128, 1556, 512, 1024, 2048, or other numbers. A memory bank may include a plurality of storage units. In an example, a data bank may include 8, 16, 64, or a different number of storage units. A memory bank or a storage unit in a memory bank may have a memory address. In an example, a storage unit may store a single byte, and data larger than a single byte may be stored in storage units with consecutive memory addresses, i.e., adjacent storage units. For instance, a storage unit can store an integer number in the INT8 format, versus two storage units may be needed to store a number in the FP16 or BF16 format, which has 16 bits. In some embodiments, 16 bits can be transferred from the local memory 1540 in a single read cycle. In other embodiments, 16 bits can be transferred from the local memory 1540 in multiple read cycles, such as two cycles.

[0143] The sparsity mode module 1550 determines sparsity modes in which the data processing unit 1530 operates to execute DNN layers. For instance, the sparsity mode module 1550 may determine whether to accelerate a layer based on weight sparsity, activation sparsity, or both. The sparsity mode module 1550 select the sparsity mode for a layer from a group of sparsity modes that includes, for example, combined sparsity mode in which the layer is accelerated based on both weight sparsity and activation sparsity, activation sparsity mode in which the layer is accelerated based on activation sparsity but not based on weight sparsity, weight sparsity mode in which the layer is accelerated based on weight sparsity but not based on activation sparsity, and a dense mode in which the layer is not accelerated based on sparsity. In some embodiments (e.g., embodiments where a layer is executed by multiple data processing units 1530), the sparsity mode module 1550 may determine the sparsity mode for all the data processing units 1530 that executes the layer. In some embodiments, the sparsity mode module 1550 may receive one or more configuration descriptors from the compiler. A configuration descriptor may correspond to a layer and indicate whether to accelerate the layer based on weight sparsity. The sparsity mode module 1550 may determine the sparsity mode of the layer based on the configuration descriptor.

[0144] The load module 1560 loads data from the local memory 1540 to the processing engine 1570 or to the post-processing engine 1580. The load module 1560 may read tensors from the local memory 1540. The tensors may include sparse activation tensors, sparse weight tensors, activation sparsity tensors, weight sparsity tensors, and so on. In some embodiments, the load module 1560 may load data based on the sparsity mode determined by the sparsity mode module 1550. The load module 1560 may select different data to transmit to the processing engine 1570 in different sparsity modes. For instance, the load module 1560 may transmit an activation sparsity tensor and a weight sparsity tensor of a layer to the processing engine 1570 in the combined sparsity mode, while transmit the activation sparsity tensor but not the weight sparsity tensor to the processing engine 1570 in the activation sparsity mode and transmit the weight sparsity tensor but not the activation sparsity tensor to the processing engine 1570 in the weight sparsity mode. In the dense mode, the load module 1560 does not transmit either the activation sparsity tensor or the weight sparsity tensor to the processing engine 1570.

[0145] In some embodiments, the load module 1560 may process (e.g., densify) data stored in the local memory 1540 before providing the data to the processing engine 1570. In an example, the load module 1560, while operating in the weight sparsity mode, may densify sparse activation tensors to generate dense activation tensors based on corresponding activation sparsity tensors. For instance, the load module 1560 may add one or more zeros into a sparse activation tensor based on an activation sparsity tensor associated with the sparse activation tensor to generate the dense activation tensor. The dense activation tensor includes one or more elements than the sparse activation tensor. The additional element(s) are zero-valued. The load module 1560 may identify one or more elements in the activation sparsity tensor that correspond to the zero-valued element(s), determine the position of each of the zero-valued element(s) in the dense activation tensor, and insert the zero-valued element(s) into the sparse activation tensor based on the determined positions. After the densification, the load module 1560 may transmit the dense activation tensors to the processing engine 1570. The load module 1560 may also transmit corresponding sparse weight tensors and weight sparsity tensors to the processing engine 1570. Activation sparsity tensor of the dense activation tensors may not be loaded to the processing engine 1570.

[0146] In another example, the load module 1560, while operating in the activation sparsity mode, may densify sparse weight tensors to generate dense weight tensors based on corresponding weight sparsity tensors by inserting zeros into sparse weight tensors. The densification of sparse weight tensors may be similar to the densification of sparse activation tensors described above. After the densification, the load module 1560 may transmit the dense weight tensors to the processing engine 1570. The load module 1560 may also transmit corresponding sparse activation tensors and activation sparsity tensors to the processing engine 1570. Weight sparsity tensor of the dense weight tensors may not be loaded to the processing engine 1570. In yet another example, the load module 1560, while operating in the dense mode, may densify both sparse weight tensors and sparse activation tensors. The load module 1560 may generate the input tensor and weight tensor of the layer and transmit the tensors to the processing engine 1570 for executing the layer without sparsity acceleration.

[0147] The processing engine 1570 performs operations in DNNs. The processing engine 1570 may accelerate neural network operations based on sparsity in data. In someembodiments, the processing engine 1570 may operate in a dense mode in which sparsity acceleration is not performed. The processing engine 1570 may include one or more processing cells. In some embodiments, the processing cells may be arranged in one or more rows and one or more columns in the processing engine 1570. Each processing cell may include processing elements (PEs) that may be arranged in an array that includes rows and columns. All the PEs in the processing engine 1570 may constitute a bigger array that includes more rows and columns.

[0148] An example PE may be or may include one or more multiply-accumulate (MAC) units that can perform MAC operations. In some embodiments (e.g., embodiments where the data processing unit 1530 executes a convolutional layer), a computation in an MAC unit may be an MAC operation on an activation operand and a weight operand. The activation operand may be an activation tensor that may include one or more activations in the input tensor of the convolution. Different activations may be in different input channels. The weight operand may be a weight tensor that may include one or more weights in the filter of the convolution. The values of the weights are determined through training the DNN. The weights in the weight operand may be in different input channels.

[0149] In some embodiments, an MAC unit includes one or more multipliers for performing multiplications. An MAC unit may also include one or more accumulators ("adders") for performing accumulations. A column of MAC units is referred to as an MAC column. An MAC column may be associated with one or more MAC lanes. A MAC lane is a path for loading data e.g., by the load module 1560, into an MAC column. A MAC lane may be also referred to as a data transmission lane or data loading lane. An MAC column may have multiple MAC lanes. The loading bandwidth of the MAC column is an aggregation of the loading bandwidths of all the MAC lanes associated with the MAC column. With a certain number of MAC lanes, data can be fed into the same number of independent MAC units simultaneously. In some embodiments where an MAC column has four MAC lanes for feeding activations or weights into the MAC column and each MAC lane may have a bandwidth of 16 bytes, the four MAC lanes can have a total loading bandwidth of 64 bytes.

[0150] In some embodiments, the processing engine 1570 may be capable of depthwise convolution, standard convolution, or both. In a depthwise convolution, an MAC unit may perform an MAC operation that includes a sequence of multiplications for an input operand and a weight operand. Each multiplication in the sequence (also referred to as a cycle) is amultiplication of a different activation in the input operand with a different weight in the weight operand. The activation and weight in the same cycle may correspond to the same channel. The sequence of multiplication produces a product operand that includes a sequence of products. The MAC operation may also include accumulations in which multiple product operands are accumulated to produce an output operand of the MAC unit. The processing engine 1570 may output multiple output operands at a time, each of which is generated by a different MAC unit. In a standard convolution, MAC operations may include accumulations across the channels. For instance, as opposed to generating an output operand, a MAC unit may accumulate products across different channels to generate a single output point.

[0151] In some embodiments, the processing engine 1570 may perform MAC operations in quantized deep learning operations, such as MAC operations in a quantized convolution. In some embodiments, an MAC unit in the processing engine 1570 may receive quantized activation and quantized weights and compute a quantized MAC result. The quantized MAC result may be a quantized value in an integer format and may be the output of the MAC unit. In some embodiments, the MAC unit may also include a quantization multiplier that can multiply a quantization scale with the quantized MAC result, and the output of the MAC unit may be a real value in a floating-point format. The MAC unit may include no quantization subtractors as zero-point offsetting is not needed for the MAC operations in quantized deep learning operations.

[0152] In some embodiments, the processing engine 1570 may include sparsity acceleration logic for facilitating sparsity acceleration. For instance, each processing cell in the processing engine 1570 may include one or more sparsity modules. In an example, each MAC column or each MAC row may have a corresponding sparsity module that accelerates MAC operations in the MAC column or MAC row. In some embodiments, a sparsity module accelerates computations in the processing engine 1570 based on sparsity in activations, sparsity in weights, or both. The sparsity module may include a storage unit that stores a sparsity tensor, which may be loaded to the storage unit by the load module 1560. The sparsity tensor may be an activation sparsity tensor, a weight sparsity tensor, or a combined sparsity tensor.

[0153] An activation sparsity tensor may be the sparsity tensor of an activation tensor and has the same number of elements as the activation tensor. An element in the activationsparsity tensor may indicate whether the corresponding element in the activation tensor is zero or not. For instance, a zero-valued in the activation sparsity tensor may indicate that the corresponding element in the activation tensor is zero. A one-valued in the activation sparsity tensor may indicate that the corresponding element in the activation tensor is nonzero. A weight sparsity tensor may be the sparsity tensor of a weight tensor and has the same number of elements as the weight tensor. An element in the weight sparsity tensor may indicate whether the corresponding element in the weight tensor is zero or not. For instance, a zero-valued in the weight sparsity tensor may indicate that the corresponding element in the weight tensor is zero. A one-valued in the weight sparsity tensor may indicate that the corresponding element in the weight tensor is nonzero. The sparsity module may generate a combined sparsity tensor using an activation sparsity tensor and a weight sparsity tensor. For instance, the sparsity module may multiply an element of the activation sparsity tensor with a corresponding element of the weight sparsity tensor to compute an element of the combined sparsity tensor. The positions of the three elements in their corresponding sparsity tensors may match. In some embodiments, each element in a sparsity tensor may be a bit, and the sparsity tensor may be referred to as a sparsity bitmap.

[0154] The sparsity module may use the sparsity tensor to identify activations and weights to be used in MAC operations by the MAC units. In an embodiment where the processing engine 1570 operates in the combined sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of a combined sparsity tensor. In an embodiment where the processing engine 1570 operates in the activation sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of an activation sparsity tensor. In an embodiment where the processing engine 1570 operates in the weight sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of a weight sparsity tensor. The sparsity module may be bypassed in the dense mode as no sparsity acceleration would be conducted.

[0155] The post-processing engine 1580 processes outputs of the processing engine 1570. The post-processing engine 1580 may include one or more post-processing elements. In some embodiments, the post-processing elements in the post-processing engine 1580 may be arranged in an array that has rows and columns. In some embodiments, the postprocessing engine 1580 computes activation functions. The post-processing engine 1580may receive outputs of the processing engine 1570 as inputs to the activation functions. In addition or alternative to activation functions, the post-processing engine 1580 may perform other types of post-processing on outputs of the processing engine 1570. For instance, the post-processing engine 1580 may apply a bias on an output of the processing engine 1570. In some embodiments, the post-processing engine 1580 may be bypassed for certain neural network operations.

[0156] The drain module 1590 drains data from the processing engine 1570 or from the post-processing engine 1580. The drain module may write the data to the local memory 1540. The drained data may be tensors, such as output tensors of neural network operations. In some embodiments, the drain module 1590 may drain data on a cell level. For each processing cell, the drain module 1590 may drain outputs of PEs in the processing cell based on a row index or column index of each PE. For instance, the drain module 1590 may use a sequence of cycles to drain data from a processing cell. The drain module 1590 may drain the output of some of the PEs in each cycle. The sequence of the cycles may be configured based on a configuration descriptor indicating the operation mode of the load module 1560.

[0157] In some embodiments, the drain module 1590 includes sparsity encoding logic that can convert outputs of the processing engine 1570 from a dense format to a sparse format. For instance, the drain module 1590 may be implemented with one or more sparsity encoders. A sparsity encoder converts dense data to compressed data based on sparsity in the dense data. For instance, the sparsity encoder may remove zeros in an activation tensor computed by the processing engine 1570 to convert the activation tensor to a compressed activation tensor. The sparsity encoder may also generate sparsity tensors, including activation sparsity tensors.

[0158] In some embodiments, the data drained from the processing engine 1570 may be at least part of an output tensor (e.g., the output tensor 1630 in FIG. 16) of a deep learning operation. The sparsity encoder may generate a compressed version of the output tensor. The sparsity encoder may identify every zero-valued activation in the output tensor and remove these activations from the output tensor to generate a compressed activation tensor (aka "sparse activation tensor"). The sparsity encoder may also generate one or more sparsity tensors for the output tensor. A sparsity tensor may correspond to a portion of the output tensor (e.g., the vector 1635 in FIG. 16). The sparsity tensor may include sparsityelements (e.g., bits), each of which corresponds to a different activation in the vector and indicates whether the corresponding activation is zeroed or not.

[0159] The drain module 1590 may write the compressed activation tensor and the one or more sparsity tensors into the local memory 1540. The sparse activation tensor and the one or more sparsity tensors may be further loaded to the memory 1510, e.g., through the DMA engine 1520. Additionally or alternatively, the sparse activation tensor and the one or more sparsity tensors may be loaded by the load module 1560 to the processing engine 1570 for further computation, e.g., for performing a deep learning operation in the next layer.Example Computing Device

[0160] FIG. 16 is a block diagram of an example computing device 1600, in accordance with various embodiments. In some embodiments, the computing device 1600 can be used as at least part of the DNN system 1700. A number of components are illustrated in FIG. 16 as included in the computing device 1600, but any one or more of these components may be omitted or duplicated, as suitable for the application. In some embodiments, some or all of the components included in the computing device 1600 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated onto a single system on a chip (SoC) die. Additionally, in various embodiments, the computing device 1600 may not include one or more of the components illustrated in FIG. 16, but the computing device 1600 may include interface circuitry for coupling to the one or more components. For example, the computing device 1600 may not include a display device 1606, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which a display device 1606 may be coupled. In another set of examples, the computing device 1600 may not include an audio input device 1618 or an audio output device 1608 but may include audio input or output device interface circuitry (e.g., connectors and supporting circuitry) to which an audio input device 1618 or audio output device 1608 may be coupled.

[0161] The computing device 1600 may include a processing device 1602 (e.g., one or more processing devices). The processing device 1602 processes electronic data from registers and / or memory to transform that electronic data into other electronic data that may be stored in registers and / or memory. The computing device 1600 may include a memory 1604, which may itself include one or more memory devices such as volatile memory (e.g., DRAM), nonvolatile memory (e.g., read-only memory (ROM)), high bandwidth memory(HBM), flash memory, solid state memory, and / or a hard drive. In some embodiments, the memory 1604 may include memory that shares a die with the processing device 1602. In some embodiments, the memory 1604 includes one or more non-transitory computer- readable media storing instructions executable to perform operations for dynamically allocating Al task workloads (e.g., the method 1300 described in conjunction with FIG. 13 or the method 1400 described in conjunction with FIG. 14) or some operations performed by one or more components of the heterogeneous computing system 500 (e.g., the task manager 510). The instructions stored in the one or more non-transitory computer-readable media may be executed by the processing device 1602.

[0162] In some embodiments, the computing device 1600 may include a communication chip 1612 (e.g., one or more communication chips). For example, the communication chip 1612 may be configured for managing wireless communications for the transfer of data to and from the computing device 1600. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data through the use of modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not.

[0163] The communication chip 1612 may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.10 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment), Long-Term Evolution (LTE) project along with any amendments, updates, and / or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to as "3GPP2"), etc.). IEEE 802.16 compatible Broadband Wireless Access (BWA) networks are generally referred to as WiMAX networks, an acronym that stands for worldwide interoperability for microwave access, which is a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standards. The communication chip 1612 may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E- HSPA), or LTE network. The communication chip 1612 may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). Thecommunication chip 1612 may operate in accordance with Code-division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. The communication chip 1612 may operate in accordance with other wireless protocols in other embodiments. The computing device 1600 may include an antenna 1622 to facilitate wireless communications and / or to receive other wireless communications (such as AM or FM radio transmissions).

[0164] In some embodiments, the communication chip 1612 may manage wired communications, such as electrical, optical, or any other suitable communication protocols (e.g., the Ethernet). As noted above, the communication chip 1612 may include multiple communication chips. For instance, a first communication chip 1612 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second communication chip 1612 may be dedicated to longer-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, a first communication chip 1612 may be dedicated to wireless communications, and a second communication chip 1612 may be dedicated to wired communications.

[0165] The computing device 1600 may include battery / power circuitry 1614. The battery / power circuitry 1614 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 1600 to an energy source separate from the computing device 1600 (e.g., AC line power).

[0166] The computing device 1600 may include a display device 1606 (or corresponding interface circuitry, as discussed above). The display device 1606 may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display, for example.

[0167] The computing device 1600 may include an audio output device 1608 (or corresponding interface circuitry, as discussed above). The audio output device 1608 may include any device that generates an audible indicator, such as speakers, headsets, or earbuds, for example.

[0168] The computing device 1600 may include an audio input device 1618 (or corresponding interface circuitry, as discussed above). The audio input device 1618 may include any device that generates a signal representative of a sound, such as microphones, microphone arrays, or digital instruments (e.g., instruments having a musical instrument digital interface (MIDI) output).

[0169] The computing device 1600 may include a GPS device 1616 (or corresponding interface circuitry, as discussed above). The GPS device 1616 may be in communication with a satellite-based system and may receive a location of the computing device 1600, as known in the art.

[0170] The computing device 1600 may include another output device 1610 (or corresponding interface circuitry, as discussed above). Examples of the other output device 1610 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0171] The computing device 1600 may include another input device 1620 (or corresponding interface circuitry, as discussed above). Examples of the other input device 1620 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0172] The computing device 1600 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smart phone, a mobile internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultramobile personal computer, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 1600 may be any other electronic device that processes data.

[0173] The following paragraphs provide various examples of the embodiments disclosed herein.

[0174] Example 1 provides a method, including determining whether a computational task within an Al task is part of neural network inference, the Al task to be performed based onthe neural network inference; in response to determining that the computational task is not part of the neural network inference, causing a CPU to perform the computational task; and in response to determining that the computational task is part of the neural network inference: identifying a plurality of processing units eligible for performing the computational task, estimating costs of the plurality of processing units performin the computational task, in which a cost indicates a measurement of latency, consumed energy, or performance of a processing unit performing the computational task, selecting a processing unit from the plurality of processing units based on the estimated costs, and causing the selected processing unit to perform the computational task.

[0175] Example 2 provides the method of example 1, in which identifying the plurality of processing units includes identifying a group of processing units eligible for performing the computational task, the group of processing units includes the plurality of processing units and an additional processing unit; receiving historical data indicating a cost of a historical execution of a similar computational task by the additional processing unit; and removing the additional processing unit from the group of processing units based on the historical data.

[0176] Example 3 provides the method of example 2, in which the historical data indicates that the cost of the historical execution of the similar computational task by the additional processing unit is higher than a cost of a historical execution of a similar computational task by each of the plurality of processing units.

[0177] Example 4 provides the method of example 2 or 3, in which identifying the plurality of processing units further includes receiving data indicating one or more characteristics of the computational task, in which the additional processing unit is removed further based on the data.

[0178] Example 5 provides the method of any one of examples 1-4, in which estimating the costs of the plurality of processing units performing the computational task includes for each processing unit: estimating costs of different cost types using a plurality of cost models, and selecting a cost from the costs based on a user input, the user input indicating a preference of a user for one of the different cost types.

[0179] Example 6 provides the method of any one of examples 1-5, in which the plurality of processing units includes the CPU.

[0180] Example 7 provides the method of any one of examples 1-6, in which the computational task is a task of compiling a neural network to be deployed forthe neural network inference, and the task is not part of the neural network inference.

[0181] Example 8 provides the method of any one of examples 1-6, in which the computational task is a task of generating an input to a neural network to be deployed for the neural network inference or processing an output of the neural network, and the task is not part of the neural network inference.

[0182] Example 9 provides the method of any one of examples 1-6, in which the computational task is a task of executing a layer in a neural network, and the task is part of the neural network inference.

[0183] Example 10 provides the method of example 9, in which selecting the processing unit includes estimating a switching cost indicating a cost of switching from a different processing unit to the processing unit for executing the layer, a precedent layer in the neural network executed by the different processing unit; and selecting the processing unit further based on the switching cost.

[0184] Example 11 provides one or more non-transitory computer-readable media storing instructions executable to perform operations, the operations including determining whether a computational task within an Al task is part of neural network inference, the Al task to be performed based on the neural network inference; in response to determining that the computational task is not part of the neural network inference, causing a CPU to perform the computational task; and in response to determining that the computational task is part of the neural network inference: identifying a plurality of processing units eligible for performing the computational task, estimating costs of the plurality of processing units performing the computational task, in which a cost indicates a measurement of latency, consumed energy, or performance of a processing unit performing the computational task, selecting a processing unit from the plurality of processing units based on the estimated costs, and causing the selected processing unit to perform the computational task.

[0185] Example 12 provides the one or more non-transitory computer-readable media of example 11, in which identifying the plurality of processing units includes identifying a group of processing units eligible for performing the computational task, the group of processing units includes the plurality of processing units and an additional processing unit; receivinghistorical data indicating a cost of a historical execution of a similar computational task by the additional processing unit; and removing the additional processing unit from the group of processing units based on the historical data.

[0186] Example 13 provides the one or more non-transitory computer-readable media of example 12, in which the historical data indicates that the cost of the historical execution of the similar computational task by the additional processing unit is higher than a cost of a historical execution of a similar computational task by each of the plurality of processing units.

[0187] Example 14 provides the one or more non-transitory computer-readable media of example 12 or 13, in which identifying the plurality of processing units further includes receiving data indicating one or more characteristics of the computational task, in which the additional processing unit is removed further based on the data.

[0188] Example 15 provides the one or more non-transitory computer-readable media of any one of examples 11-14, in which estimating the costs of the plurality of processing units performing the computational task includes for each processing unit: estimating costs of different cost types using a plurality of cost models, and selecting a cost from the costs based on a user input, the user input indicating a preference of a user for one of the different cost types.

[0189] Example 16 provides the one or more non-transitory computer-readable media of any one of examples 11-15, in which the plurality of processing units includes the CPU.

[0190] Example 17 provides the one or more non-transitory computer-readable media of any one of examples 11-16, in which the computational task is a task of compiling a neural network to be deployed for the neural network inference, and the task is not part of the neural network inference.

[0191] Example 18 provides the one or more non-transitory computer-readable media of any one of examples 11-16, in which the computational task is a task of generating an input to a neural network to be deployed for the neural network inference or processing an output of the neural network, and the task is not part of the neural network inference.

[0192] Example 19 provides the one or more non-transitory computer-readable media of any one of examples 11-16, in which the computational task is a task of executing a layer in a neural network, and the task is part of the neural network inference.

[0193] Example 20 provides the one or more non-transitory computer-readable media of example 19, in which selecting the processing unit includes estimating a switching cost indicating a cost of switching from a different processing unit to the processing unit for executing the layer, a precedent layer in the neural network executed by the different processing unit; and selecting the processing unit further based on the switching cost.

[0194] Example 21 provides an apparatus, including a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations including determining whether a computational task within an Al task is part of neural network inference, the Al task to be performed based on the neural network inference, in response to determining that the computational task is not part of the neural network inference, causing a CPU to perform the computational task, and in response to determining that the computational task is part of the neural network inference: identifying a plurality of processing units eligible for performing the computational task, estimating costs of the plurality of processing units performing the computational task, in which a cost indicates a measurement of latency, consumed energy, or performance of a processing unit performing the computational task, selecting a processing unit from the plurality of processing units based on the estimated costs, and causing the selected processing unit to perform the computational task.

[0195] Example 22 provides the apparatus of example 21, in which identifying the plurality of processing units includes identifying a group of processing units eligible for performing the computational task, the group of processing units includes the plurality of processing units and an additional processing unit; receiving historical data indicating a cost of a historical execution of a similar computational task by the additional processing unit; and removing the additional processing unit from the group of processing units based on the historical data.

[0196] Example 23 provides the apparatus of example 21 or 22, in which estimating the costs of the plurality of processing units performing the computational task includes for each processing unit: estimating costs of different cost types using a plurality of cost models, and selecting a cost from the costs based on a user input, the user input indicating a preference of a user for one of the different cost types.

[0197] Example 24 provides the apparatus of any one of examples 21-23, in which the computational task is a task of compiling a neural network to be deployed forthe neural network inference, generating an input to the neural network, or processing an output of the neural network, and the task is not part of the neural network inference.

[0198] Example 25 provides the apparatus of any one of examples 21-23, in which the computational task is a task of executing a layer in a neural network, the task is part of the neural network inference, and selecting the processing unit includes estimating a switching cost indicating a cost of switching from a different processing unit to the processing unit for executing the layer, a precedent layer in the neural network executed by the different processing unit; and selecting the processing unit further based on the switching cost.

[0199] The following paragraphs provide additional examples of the embodiments disclosed herein.

[0200] Example 1 provides a method, including estimating a cost of a first processing unit executing a first layer in a neural network; estimating a cost of a second processing unit executing the first layer; estimating a switching cost indicating a cost of switching from the second processing unit to the first processing unit for executing the first layer; determining whether the estimated cost of the second processing unit is greater than an aggregation of the estimated cost of the first processing unit and the switching cost; and in response to determining that the estimated cost of the second processing unit is greater than an aggregation, causing the first processing unit to execute the first layer.

[0201] Example 2 provides the method of example 1, further including selecting, based on the estimated cost of the first processing unit, the first processing unit from a group of processing units eligible for executing the first layer.

[0202] Example 3 provides the method of example 2, in which the second layer is executed before the first processing unit is selected for executing the first layer.

[0203] Example 4 provides the method of example 2 or 3, further including estimating costs of different cost types for the first processing unit; receiving a user input indicating a selection of a cost type; and selecting the estimated cost of the first processing unit from the estimated costs of the different cost types based on the user input.

[0204] Example 5 provides the method of any one of examples 1-4, in which estimating the switching cost includes receiving a user input indicating a preference of a user for minimizing a type of overhead to be caused by switching from the second processing unit tothe first processing unit; selecting a switching cost model developed for estimating the type of overhead; and estimating the switching cost using the switching cost model.

[0205] Example 6 provides the method of any one of examples 1-5, further including in response to determining that the estimated cost of the second processing unit is not greater than an aggregation, causing the second processing unit to execute the first layer.

[0206] Example 7 provides the method of any one of examples 1-6, further including in response to determining that the second layer is executed by the first processing unit, causing the first processing unit to execute the first layer.

[0207] Example 8 provides the method of any one of examples 1-7, in which the estimated cost of the first processing unit indicates an estimated latency cost of the first processing unit executing the first layer, estimated energy cost of the first processing unit executing the first layer, or estimated performance cost of the first processing unit executing the first layer.

[0208] Example 9 provides the method of any one of examples 1-8, in which the switching cost includes a cost of transferring input data of the first layer from a memory of the second processing unit to a memory of the first processing unit.

[0209] Example 10 provides the method of any one of examples 1-9, further including causing a third processing unit to provide one or more configuration descriptors to the first processing unit, in which one or more components of the first processing unit are to be configured by the one or more configuration descriptors for executing the first layer.

[0210] Example 11 provides one or more non-transitory computer-readable media storing instructions executable to perform operations, the operations including estimating a cost of a first processing unit executing a first layer in a neural network; estimating a cost of a second processing unit executing the first layer; estimating a switching cost indicating a cost of switching from the second processing unit to the first processing unit for executing the first layer; determining whether the estimated cost of the second processing unit is greater than an aggregation of the estimated cost of the first processing unit and the switching cost; and in response to determining that the estimated cost of the second processing unit is greater than an aggregation, causing the first processing unit to execute the first layer.

[0211] Example 12 provides the one or more non-transitory computer-readable media of example 11, in which the operations further include selecting, based on the estimated costof the first processing unit, the first processing unit from a group of processing units eligible for executing the first layer.

[0212] Example 13 provides the one or more non-transitory computer-readable media of example 12, in which the second layer is executed before the first processing unit is selected for executing the first layer.

[0213] Example 14 provides the one or more non-transitory computer-readable media of example 12 or 13, in which the operations further include estimating costs of different cost types for the first processing unit; receiving a user input indicating a selection of a cost type; and selecting the estimated cost of the first processing unit from the estimated costs of the different cost types based on the user input.

[0214] Example 15 provides the one or more non-transitory computer-readable media of any one of examples 11-14, in which estimating the switching cost includes receiving a user input indicating a preference of a user for minimizing a type of overhead to be caused by switching from the second processing unit to the first processing unit; selecting a switching cost model developed for estimating the type of overhead; and estimating the switching cost using the switching cost model.

[0215] Example 16 provides the one or more non-transitory computer-readable media of any one of examples 11-15, in which the operations further include in response to determining that the estimated cost of the second processing unit is not greater than an aggregation, causing the second processing unit to execute the first layer.

[0216] Example 17 provides the one or more non-transitory computer-readable media of any one of examples 11-16, in which the operations further include in response to determining that the second layer is executed by the first processing unit, causing the first processing unit to execute the first layer.

[0217] Example 18 provides the one or more non-transitory computer-readable media of any one of examples 11-17, in which the estimated cost of the first processing unit indicates an estimated latency cost of the first processing unit executing the first layer, estimated energy cost of the first processing unit executing the first layer, or estimated performance cost of the first processing unit executing the first layer.

[0218] Example 19 provides the one or more non-transitory computer-readable media of any one of examples 11-18, in which the switching cost includes a cost of transferring inputdata of the first layer from a memory of the second processing unit to a memory of the first processing unit.

[0219] Example 20 provides the one or more non-transitory computer-readable media of any one of examples 11-19, in which the operations further include causing a third processing unit to provide one or more configuration descriptors to the first processing unit, in which one or more components of the first processing unit are to be configured by the one or more configuration descriptors for executing the first layer.

[0220] Example 21 provides a system, including a first processing unit; a second processing unit; and a task manager configured to perform operations including estimating a cost of a first processing unit executing a first layer in a neural network, estimating a cost of a second processing unit executing the first layer, estimating a switching cost indicating a cost of switching from the second processing unit to the first processing unit for executing the first layer, determining whether the estimated cost of the second processing unit is greater than an aggregation of the estimated cost of the first processing unit and the switching cost, and in response to determining that the estimated cost of the second processing unit is greater than an aggregation, causing the first processing unit to execute the first layer.

[0221] Example 22 provides the system of example 21, in which the operations further include selecting, based on the estimated cost of the first processing unit, the first processing unit from a group of processing units eligible for executing the first layer, in which the second layer is executed before the first processing unit is selected for executing the first layer.

[0222] Example 23 provides the system of example 22, in which the operations further include estimating costs of different cost types for the first processing unit; receiving a user input indicating a selection of a cost type; and selecting the estimated cost of the first processing unit from the estimated costs of the different cost types based on the user input.

[0223] Example 24 provides the system of any one of examples 21-23, in which estimating the switching cost includes receiving a user input indicating a preference of a user for minimizing a type of overhead to be caused by switching from the second processing unit to the first processing unit; selecting a switching cost model developed for estimating the type of overhead; and estimating the switching cost using the switching cost model.

[0224] Example 25 provides the system of any one of examples 21-24, in which the operations further include causing a third processing unit to provide one or moreconfiguration descriptors to the first processing unit, in which one or more components of the first processing unit are to be configured by the one or more configuration descriptors for executing the first layer.

[0225] The above description of illustrated implementations of the disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. While specific implementations of, and examples for, the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art will recognize. These modifications may be made to the disclosure in light of the above detailed description.

Claims

Claims1. A method, comprising: estimating a cost of a first processing unit executing a first layer in a neural network; estimating a cost of a second processing unit executing the first layer; estimating a switching cost indicating a cost of switching from the second processing unit to the first processing unit for executing the first layer; determining whether the estimated cost of the second processing unit is greater than an aggregation of the estimated cost of the first processing unit and the switching cost; and in response to determining that the estimated cost of the second processing unit is greater than an aggregation, causing the first processing unit to execute the first layer.

2. The method of claim 1, further comprising: selecting, based on the estimated cost of the first processing unit, the first processing unit from a group of processing units eligible for executing the first layer.

3. The method of claim 2, wherein the second layer is executed before the first processing unit is selected for executing the first layer.

4. The method of claim 2 or 3, further comprising: estimating costs of different cost types for the first processing unit; receiving a user input indicating a selection of a cost type; and selecting the estimated cost of the first processing unit from the estimated costs of the different cost types based on the user input.

5. The method of any one of claims 1-4, wherein estimating the switching cost comprises: receiving a user input indicating a preference of a user for minimizing a type of overhead to be caused by switching from the second processing unit to the first processing unit; selecting a switching cost model developed for estimating the type of overhead; and estimating the switching cost using the switching cost model.

6. The method of any one of claims 1-5, further comprising: in response to determining that the estimated cost of the second processing unit is not greater than an aggregation, causing the second processing unit to execute the first layer.

7. The method of any one of claims 1-6, further comprising: in response to determining that the second layer is executed by the first processing unit, causing the first processing unit to execute the first layer.

8. The method of any one of claims 1-7, wherein the estimated cost of the first processing unit indicates an estimated latency cost of the first processing unit executing the first layer, estimated energy cost of the first processing unit executing the first layer, or estimated performance cost of the first processing unit executing the first layer.

9. The method of any one of claims 1-8, wherein the switching cost comprises a cost of transferring input data of the first layer from a memory of the second processing unit to a memory of the first processing unit.

10. The method of any one of claims 1-9, further comprising: causing a third processing unit to provide one or more configuration descriptors to the first processing unit, wherein one or more components of the first processing unit are to be configured by the one or more configuration descriptors for executing the first layer.

11. One or more non-transitory computer-readable media storing instructions executable to perform operations, the operations comprising: estimating a cost of a first processing unit executing a first layer in a neural network; estimating a cost of a second processing unit executing the first layer; estimating a switching cost indicating a cost of switching from the second processing unit to the first processing unit for executing the first layer;determining whether the estimated cost of the second processing unit is greater than an aggregation of the estimated cost of the first processing unit and the switching cost; and in response to determining that the estimated cost of the second processing unit is greater than an aggregation, causing the first processing unit to execute the first layer.

12. The one or more non-transitory computer-readable media of claim 11, wherein the operations further comprise: selecting, based on the estimated cost of the first processing unit, the first processing unit from a group of processing units eligible for executing the first layer.

13. The one or more non-transitory computer-readable media of claim 12, wherein the second layer is executed before the first processing unit is selected for executing the first layer.

14. The one or more non-transitory computer-readable media of claim 12 or 13, wherein the operations further comprise: estimating costs of different cost types for the first processing unit; receiving a user input indicating a selection of a cost type; and selecting the estimated cost of the first processing unit from the estimated costs of the different cost types based on the user input.

15. The one or more non-transitory computer-readable media of any one of claims 11-14, wherein estimating the switching cost comprises: receiving a user input indicating a preference of a user for minimizing a type of overhead to be caused by switching from the second processing unit to the first processing unit; selecting a switching cost model developed for estimating the type of overhead; and estimating the switching cost using the switching cost model.

16. The one or more non-transitory computer-readable media of any one of claims 11-15, wherein the operations further comprise:in response to determining that the estimated cost of the second processing unit is not greater than an aggregation, causing the second processing unit to execute the first layer.

17. The one or more non-transitory computer-readable media of any one of claims 11-16, wherein the operations further comprise: in response to determining that the second layer is executed by the first processing unit, causing the first processing unit to execute the first layer.

18. The one or more non-transitory computer-readable media of any one of claims 11-17, wherein the estimated cost of the first processing unit indicates an estimated latency cost of the first processing unit executing the first layer, estimated energy cost of the first processing unit executing the first layer, or estimated performance cost of the first processing unit executing the first layer.

19. The one or more non-transitory computer-readable media of any one of claims 11-18, wherein the switching cost comprises a cost of transferring input data of the first layer from a memory of the second processing unit to a memory of the first processing unit.

20. The one or more non-transitory computer-readable media of any one of claims 11-19, wherein the operations further comprise: causing a third processing unit to provide one or more configuration descriptors to the first processing unit, wherein one or more components of the first processing unit are to be configured by the one or more configuration descriptors for executing the first layer.

21. A system, comprising: a first processing unit; a second processing unit; and a task manager configured to perform operations comprising: estimating a cost of the first processing unit executing a first layer in a neural network,estimating a cost of the second processing unit executing the first layer, estimating a switching cost indicating a cost of switching from the second processing unit to the first processing unit for executing the first layer, determining whether the estimated cost of the second processing unit is greater than an aggregation of the estimated cost of the first processing unit and the switching cost, and in response to determining that the estimated cost of the second processing unit is greater than an aggregation, causing the first processing unit to execute the first layer.

22. The system of claim 21, wherein the operations further comprise: selecting, based on the estimated cost of the first processing unit, the first processing unit from a group of processing units eligible for executing the first layer, wherein the second layer is executed before the first processing unit is selected for executing the first layer.

23. The system of claim 22, wherein the operations further comprise: estimating costs of different cost types for the first processing unit; receiving a user input indicating a selection of a cost type; and selecting the estimated cost of the first processing unit from the estimated costs of the different cost types based on the user input.

24. The system of any one of claims 21-23, wherein estimating the switching cost comprises: receiving a user input indicating a preference of a user for minimizing a type of overhead to be caused by switching from the second processing unit to the first processing unit; selecting a switching cost model developed for estimating the type of overhead; and estimating the switching cost using the switching cost model.

25. The system of any one of claims 21-24, wherein the operations further comprise: causing a third processing unit to provide one or more configuration descriptors to the first processing unit,wherein one or more components of the first processing unit are to be configured by the one or more configuration descriptors for executing the first layer.

Images