Efficient SOFTMAX calculations without loss of accuracy

Abstract

The invention relates to an efficient SOFTMAX calculation without loss of accuracy. The name of the invention is as follows: SOFTMAX calculation is efficient without loss of accuracy. In particular, for deep learning neural networks such as transformer-based neural networks and large language models (LLMs), a modified 2-pass version of the SoftMax operation may be implemented without loss of accuracy to address reduced computational costs. The first pass is modified to include two scalar operations at the end. At the end of the first pass, a first scalar operation is performed to calculate a logarithm of the denominator, and a second scalar operation is performed to calculate an operand value based on a sum of the logarithm of the denominator and a maximum value. The second pass is modified to perform addition and exponentiation. In a second pass, an exponent is obtained by subtracting the operand value from the elements of the input tensor, and the exponent of the base is ascertained. And division is avoided in the second time.

Description

Background Technology

[0001] Deep neural networks (DNNs) are widely used in a wide range of artificial intelligence applications, from computer vision to speech recognition and natural language processing, due to their ability to achieve high accuracy. However, this high accuracy comes at the cost of significant computational cost. DNNs have extremely high computational requirements because they can involve a large number of operations and a large amount of data to be read and written. Attached Figure Description

[0002] The embodiments will be readily understood from the accompanying drawings and the following detailed description. For ease of description, the same reference numerals denote the same structural elements. In the accompanying drawings, embodiments are illustrated by way of example rather than limitation.

[0003] Figure 1 This describes a block in a transformer-based neural network according to some disclosed embodiments.

[0004] Figure 2 The operation in the attention layer according to some disclosed embodiments is described.

[0005] Figure 3 A DNN system based on some disclosed embodiments is described.

[0006] Figure 4 The DNN module according to some of the disclosed embodiments is described.

[0007] Figure 5 The processing engine according to some of the disclosed embodiments is described.

[0008] Figure 6 A simplified 3-pass version of an exemplary operation for performing SoftMax on an input tensor, according to some of the disclosed embodiments, is described.

[0009] Figure 7 A two-pass version of an exemplary operation for performing SoftMax on an input tensor, according to some disclosed embodiments, is described.

[0010] Figure 8 This describes a modified version of an exemplary operation for performing SoftMax on an input tensor, based on some disclosed embodiments.

[0011] Figure 9 This describes a modified, two-pass version of an operation, based on some disclosed embodiments, for performing SoftMax on an input tensor and matrix multiplication of the output tensor and matrix, with approximate exemplary characteristics.

[0012] Figure 10This is a flowchart illustrating a method for compiling a SoftMax operation to be performed by a processor, according to some disclosed embodiments.

[0013] Figure 11 This is a flowchart illustrating a method for performing SoftMax operations to be executed by a processor, according to some disclosed embodiments.

[0014] Figure 12 This is a block diagram of an exemplary computing device based on some disclosed embodiments. Detailed Implementation Overview

[0015] The past decade has witnessed a dramatic rise in data processing based on artificial intelligence (AI), particularly deep neural networks (DNNs). DNNs are widely used in computer vision, speech recognition, image and video processing primarily due to their ability to achieve accuracy exceeding human levels. A DNN typically consists of a series of layers. A DNN layer can include one or more deep learning operations (also known as "neural network operations"), such as convolution, matrix multiplication, layer normalization, batch normalization, SoftMax, pooling, element-wise operations, linear operations, non-linear operations, and so on. While DNNs are efficient in analysis and prediction, they come at the cost of enormous computational power. DNNs can consume significant power and runtime during training and inference.

[0016] Transformer-based neural networks, or transformer-based models, are a type of DNN that can be used to power large language models (LLMs) and computer vision models (referred to as ViTs in the literature). Transformer-based neural networks are used in services and applications such as natural language processing, speech processing, conversational AI assistants, image captioning, object detection, video understanding, recommender systems, bioinformatics, time series prediction, reinforcement learning, and generative models to produce text, images, or music. Unlike other types of DNNs (such as convolutional neural networks (CNNs), where the SoftMax operation is only used in the output layer), transformer-based neural networks include a SoftMax operation in every layer. SoftMax operations can be a bottleneck when executing transformer-based neural networks on a processor, especially as sequence lengths become longer and models become larger. Within the computational pipeline of transformer-based neural networks, SoftMax operations can consume a significant amount of runtime (second only to matrix multiplication or MatMul). In some pipelines, SoftMax operations can consume 20%–40% of the total runtime on the processor. The ability to improve the efficiency of SoftMax operations performed on a processor can significantly improve the overall runtime of transformer-based neural networks.

[0017] Input or output data for deep learning operations can be arranged in a data structure called a tensor. In the case of the SoftMax operation, the SoftMax operation operates on the input tensor and outputs an output tensor. In the case of the matrix multiplication (MatMul) operation, the MatMul operation operates on a first input tensor and a second input tensor and outputs an output tensor representing the product of the first and second input tensors multiplied together. A tensor is a data structure with multiple elements across one or more dimensions. Examples of tensors include vectors (which are one-dimensional (1D) tensors), matrices (which are two-dimensional (2D) tensors), three-dimensional (3D) tensors, four-dimensional (4D) tensors, and even higher-dimensional tensors. The dimensions of a tensor can correspond to axes, such as axes in a coordinate system. Dimension can be measured by the number of data points along an axis. The dimensions of a tensor define its shape. A DNN layer can receive one or more input tensors and compute an output tensor based on one or more input tensors. Tensors in a DNN can be stored in X-major formats (e.g., XYZ or XZY), Y-major formats (e.g., YXZ or YZX), or Z-major formats (e.g., ZXY or ZYX). The format of a tensor defines the order in which data points are stored, written, or read from the tensor. The first character represents the dimension in which data points are contiguous in memory. The second character represents the dimension in which data points can be accessed after contiguous data points have been accessed in memory. The third character represents the dimension in which data points are accessed after all data points in the dimension represented by the second character have been used. For example, in the ZXY format, the access order starts in the Z dimension, then moves to the X dimension, and finally to the Y dimension. Data points in a tensor are contiguous in the Z dimension in memory, meaning that data points with the same (x, y) coordinates are contiguous in memory. Using tensor permutation, tensors can be read from memory in different formats.

[0018] The SoftMax operation can use x with N elements i The input tensor (along the specified dimension) and in element x i Apply the SoftMax operation to generate a product with element y i The output tensor. The SoftMax operation can convert element x into a tensor. i The probability y of converting to a probability distribution i The values ​​or probabilities are non-negative and sum to 1. Values ​​can range from 0 to 1. The mathematical representation of the SoftMax operation is as follows:

[0019] Where max(x) i ) represents element x i The maximum value. Subtract max(x)i This ensures numerical stability.

[0020] In some implementations, a simple 3-pass version of the SoftMax operation can be performed on the processor. Figure 6 The text explains and describes a simple 3-pass version. In the first pass, element x is determined. i The maximum value is calculated. In the second pass, the denominator value is calculated. In the third pass, the numerator is divided by the denominator value.

[0021] In some implementations, a two-pass version of the SoftMax operation can be executed on the processor. Figure 7 The text describes and explains a 2-pass version. In the first pass, the maximum value and the denominator value are determined. In the second pass, the numerator is divided by the denominator. While the 2-pass version reduces the number of passes for the data, the second pass involves N divisions, which may take multiple cycles (e.g., 2 cycles) to complete. In other words, the second pass may have the high computational cost associated with N divisions.

[0022] Embodiments of this disclosure can address at least some of the challenges and problems described above by modifying a two-pass version of the SoffMax operation. Figure 8-9 The text describes and explains a modified two-pass version and a two-pass version with approximate modifications. The first pass is modified to include two scalar operations at the end. At the end of the first pass, the first scalar operation is performed to calculate the logarithm of the denominator, and the second scalar operation is performed to calculate the operand value based on the sum of the logarithm of the denominator and the maximum value. The second pass is modified to perform addition and exponentiation. In the second pass, the operand value is subtracted from the elements of the input tensor to obtain the exponent, and the exponent of the base is calculated. Division is omitted in the second pass.

[0023] It is worth noting that inverse multiplication or division is eliminated in the second pass. The calculations in the modified 2-pass version are mathematically equivalent to Equation 1, meaning that there is no loss of model accuracy due to the modifications made and no specific fine-tuning is needed to compensate for accuracy losses otherwise incurred due to approximation-based techniques. The modified 2-pass version can be extended to utilize approximations. Furthermore, the modified 2-pass version and the modified 2-pass version with approximations are compatible with various data formats. Transformer-based neural networks or transformer-based models

[0024] Transformer-based neural networks can include one or more blocks. A block can be an encoder. A block can be a decoder. A neural network can contain many encoder blocks and decoder blocks.

[0025] Figure 1Block 100 in a transformer-based neural network according to some disclosed embodiments is illustrated. For illustration, block 100 is a decoder block in a transformer-based neural network. Transformer block 100 includes transformer decoder 110 (referred to individually as "transformer decoder 110"). As an example, each transformer decoder 110 may include multiple layers, such as a self-attention layer 120, a layer normalization layer 130, a cross-attention layer 140, another layer normalization layer 150, a feedforward layer 160, and another layer normalization layer 170. In other embodiments, transformer decoder 110 may include different, fewer, or more layers. Transformer block 100 may include different numbers of transformer decoders 110.

[0026] Self-attention layer 120 or cross-attention layer 140 may have an attention function. In some embodiments, the attention function may map one or more queries and one or more key-value pairs to an output, where the query, each key, each value, or the output may be a tensor, such as a vector. The output may be computed as a weighted sum of values. The weight assigned to each value may be computed via a compatibility function of the query with the corresponding key. Layer normalization layer 130 may concatenate all positions with a constant number of sequentially executed operations. Self-attention (sometimes referred to as internal attention) may be an attention mechanism relating to different positions in a single sequence to facilitate the computation of a representation of the sequence. Self-attention layer 120 may allow each position in the corresponding transformer decoder 110 to attend to all positions in the transformer decoder 110, up to and including that position.

[0027] like Figure 1 As shown, the self-attention layer 120 receives vector 102C. Vector 102C can be generated by flattening the current feature map 101C. In some embodiments, the current feature map 101C can be a 2D or 3D tensor, while vector 102C can be a 1D tensor. Data points in the current feature map 101C can be rearranged so that all data points lie in one dimension to generate vector 102C. In the example, the current feature map 101C can have three dimensions: X, Y, and Z, where the Z dimension can be a channel dimension. Data points in the current feature map 101C can be rearranged so that all data points lie in one of the X, Y, and Z dimensions to form vector 102C. Vector 102C is input into the self-attention layer 120 and can be used as a query, key, and value for the self-attention function. In some embodiments, the self-attention function in the self-attention layer 120 can be represented as: Where Q, K, and V represent query, key, and value, respectively; f(Q, K, V) represents the self-attention function; and σ represents the SoftMax function or operation.

[0028] The output of the self-attention layer 120 is further accumulated with vector 102C via concatenator 125. The concatenated result is fed into layer normalization layer 130. Layer normalization layer 130 applies layer normalization to the output of concatenator 125. Layer normalization layer 130 can normalize its input (i.e., the output of concatenator 125) across feature dimensions (rather than batch dimensions). Feature dimensions can be the dimensions of vector 102C or vector 102P.

[0029] In some embodiments, the layer normalization operation may include a series of calculations. Layer normalization layer 130 may include a mean calculation, which may be represented as... Where A xyz Let x represent the data element in the input tensor, y represent the position index of the data element in one spatial dimension, z represent the position index of the data element in another spatial dimension, and μ represent the position index of the data element in the channel dimension. xy The output of the mean calculation can be a 2D matrix. The mean calculation can be a channel-wise reduction operation. Layer normalization layer 130 can, for example, reduce μ by copying each data element at z output points. xy Convert to 3D tensor μ xyz .

[0030] Layer normalization layer 130 may also include layers that can be represented as D xyz =A xyz -μ xyz Element-wise subtraction. Layer normalization layer 130 can also perform subtraction that can be represented as... The variance calculation. Layer normalization layer 130 may further include elements represented as... Division calculation. M xy It can be a 2D tensor. Layer normalization layer 130 can also, for example, by copying each data element at z output points to normalize M. xy Convert to 3D tensor M xyz Furthermore, the layer normalization layer 130 can have a representation as Element-wise multiplication. Layer normalization layer 130 can be further calculated. and LN xyz =A″ xyz ×γ z LN xyz It can be the output of a layer normalization operation.

[0031] The output of the layer normalization layer 130, which encodes the flattened current feature, can be input into the cross-attention layer 140. The cross-attention layer 140 also receives vector 102P as another input. Vector 102P is generated by flattening a previous feature map, such as a previous feature map 101P. The previous feature map 101P can be generated by an encoder block of a transformer-based neural network. In some embodiments, the previous feature map 101P can be a 2D or 3D tensor, while vector 102P can be a 1D tensor. Data points in the previous feature map 101P can be rearranged such that all data points lie in one dimension to generate vector 102P. In the example, each previous feature map 101P can have three dimensions: X, Y, and Z, where the Z dimension can be a channel dimension. Data points in the previous feature map 101P can be rearranged so that all data points lie in one of the X, Y, and Z dimensions to form vector 102P. Data points in the same previous feature map can be arranged together, for example, in a manner that does not interrupt the flow of data points from different previous feature maps. In some embodiments, vector 102P lies on the same dimension as vector 102C. For illustrative purposes, each may... Figure 1 A series of boxes in the vector map represent vectors 102C and 102P. The number of boxes in vector 102C or vector 102P does not represent the number of data points in vector 102C or vector 102P. In some embodiments, the number of data points in vector 102C may be equal to the number of data points in the current feature map 101C, and the number of data points in vector 102P may be equal to the total number of data points in all previous feature maps 101P.

[0032] The cross-attention layer 140 has a cross-attention function applied to vectors 102P and 102C. Vector 102P can be used as a query, and vector 102C can be used as a key and value. The cross-attention function can be expressed as: Where σ is the SoftMax function or operation; Represents vector 102C; Represents vector 102P; and These are the projection matrices used for querying, keys, and values, respectively.

[0033] The output of the cross-attention layer 140 can be summed with the output of the layer normalization layer 130 via another connector 145. The output of connector 145 is fed into the layer normalization layer 150 to perform another layer normalization operation within it. The output of the layer normalization layer 150 is fed into the feedforward layer 160. In some embodiments, the feedforward layer 160 may include linear and nonlinear transformations. The linear transformation may be the same as or similar to the operation in the fully connected layer. The nonlinear transformation may be an activation function. In some embodiments, the feedforward layer 160 may multiply the input with keys to obtain a weight for each key. The feedforward layer 160 may use the weights to compute a weighted sum of values ​​to produce an output.

[0034] The output of feedforward layer 160 can be summed with the output of layer normalization layer 150 via connector 165. The output of connector 165 can be fed into layer normalization layer 170, where another layer normalization operation is performed. The output of layer normalization layer 170 can be fed into the next transformer decoder 110 for further processing. The output of the final transformer decoder 110 can constitute the output of transformer block 100, which can be an intermediate feature map representing the collection of the current feature map 101C and the previous feature map 101P.

[0035] Figure 2 Operations in an attention layer according to some disclosed embodiments are described. For transformer-based neural networks, attention layers (e.g., self-attention layers, cross-attention layers, multi-head attention layers, masked multi-head attention layers) can be included in the encoder block. Attention layers (e.g. Figure 1 The self-attention layer 120 and the cross-attention layer 140 can be included in the decoder block.

[0036] To perform at least some of the operations associated with the attention mechanism, the attention layer will include operations such as MatMul, scaling operations, and the SoftMax function or operation. To illustrate, in Figure 2 The text describes the operation of calculating values ​​based on Equation 2.

[0037] The attention layer can include the MatMul operation 202. The MatMul operation 202 can convert matrix Q and matrix K. T Multiply to calculate QK T .

[0038] The attention layer can include scaling operation 204. Scaling operation 204 can scale QK. T Scaling of the value of the output tensor of (MatMul operation 202) To calculate

[0039] The attention layer can include the SoftMax operation 206. The SoftMax operation 206 can be... Apply the SoffMax function to the output tensor of scaling operation 204 to calculate The SoftMax operation 206 can convert the output of the scaling operation 204 into a probability matrix or an attention score.

[0040] The attention layer can include the MatMul operation 208. The MatMul operation 208 can convert the matrix The output tensor of SoftMax operation 206 is multiplied by matrix V to calculate the output. MatMul operation 208 combines a probability matrix or attention score with a value tensor V.

[0041] The output of MatMul operation 208 can be passed to the next layer, such as the layer normalization layer in a transformer block.

[0042] In fact, attention layers can be included in Figure 2 The following are many examples of parallel operations. For a self-attention layer, the SoftMax operation 206 can be applied to compute b×N times, where b is the batch size and N is the sequence length. N can also be the size of the input tensor of the SoftMax operation 206. N can also be the size of the output tensor of the SoftMax operation 206. For a multi-head attention layer, the SoftMax operation 206 can be applied to compute b×h×N times, where b is the batch size, h is the number of heads, and N is the sequence length.

[0043] Figure 2 The operations of the attention layer described herein implement a self-attention mechanism. The attention layer has operations capable of performing at least some of the calculations used in the computation of values ​​based on Equation 2. It can also implement... Figure 2 The operations described herein are used to perform cross-attention or other attention mechanisms. The attention layer has operations that can perform at least some of the calculations used in the calculation of values ​​based on Equation 3.

[0044] In some cases, such as Figure 2The scenario described above can be implemented in the attention layer of a transformer-based neural network. The SoftMax operation can be performed after the scaling operation in the attention layer. The SoftMax operation is used in the attention mechanism or attention layer of a transformer-based neural network because it helps determine the importance weight, or attention score, assigned to each element in the input sequence when computing the weighted sum of the value vectors. In the attention layer of a transformer-based neural network, the input to the attention layer includes the query, key, and value derived from the input sequence. The attention score is computed as a scaled dot product between the query and key vectors, representing the relevance between the query and each key. However, these attention scores are not necessarily normalized and can have values ​​outside the range of 0 to 1. Applying the SoftMax operation to these attention scores transforms them into a probability distribution, where each value represents the relative importance or weight assigned to the corresponding value vector. The SoftMax operation ensures that the sum of the attention weights is up to 1, which is a desirable property for probability distributions. The normalization functionality of the SoftMax operation applied in this way prevents the attention mechanism from assigning excessive importance to a single element or position in the input sequence, which could lead to unstable or biased representations. By applying the SoftMax operation, the attention weights are transformed into a well-performing probability distribution, allowing the attention mechanism to distribute importance weights more evenly across relevant elements of the input sequence. Furthermore, the SoftMax function is differentiable, which allows transformer-based neural networks to be trainable using gradient-based optimization techniques such as backpropagation.

[0045] In some cases, the SoftMax operation can be implemented in the output layer of a neural network. The SoftMax operation can be provided in the output layer after a linear layer. It can also be provided in the output layer after one or more fully connected layers. In the context of neural networks used for classification tasks, the SoftMax operation can be provided in the output layer. Each element (or vector of input tensors) of the input tensor represents an activation value for a specific class, and the SoftMax function transforms these activation values ​​into probabilities. The class with the highest probability can then be selected as the predicted class. The SoftMax operation is particularly useful when dealing with multi-class classification problems because it provides a way to represent the probability distribution for all classes. The SoftMax operation is also differentiable, which allows gradient-based optimization techniques to be used to train the neural network. Implementing DNN on a processor

[0046] The significant improvements in DNN model size and accuracy, combined with the rapid increase in the computing power of execution platforms, have led to the adoption of DNN applications, even in resource-constrained mobile and edge devices with limited power availability. DNN models can be executed by DNN accelerators, for example, for training or inference. A DNN accelerator can be or includes one or more data processing units. Data processing units can also be referred to as computational blocks or tiles. Data processing units can include one or more processing engines capable of performing neural network operations. Processing engines can include one or more processing cells used to perform arithmetic operations associated with neural network operations.

[0047] Figure 3 A DNN system 300 according to some disclosed embodiments is described. It can be implemented on one or more computing devices (such as...) Figure 12 The entire DNN system 300 or a part of the DNN system 300 is implemented in the computing device 1200. The DNN system 300 can generate and execute DNNs, such as transformer-based neural networks, convolutional neural networks, etc. Figure 3 As shown, the DNN system 300 includes a DNN module 301 and a DNN accelerator 302. In other embodiments, alternative configurations, different, or additional components may be included in the DNN system 300. For example, the DNN system 300 may include multiple DNN modules or multiple DNN accelerators. Furthermore, the functionality of components attributed to the DNN system 300 may be implemented by different components included in the DNN system 300 or different systems. In some embodiments, the DNN module 301 and the DNN accelerator 302 may include different types of processing units or may be implemented by different types of processing units. In the example, the DNN module 301 may be implemented by one or more central processing units (CPUs). The DNN accelerator 302 may also be referred to as a neural processing unit, an AI accelerator, or an AI processor. The DNN module 301 and the DNN accelerator 302 may be implemented in the same chip or may be implemented as separate chips.

[0048] DNN module 301 facilitates the generation and deployment of DNNs. In some embodiments, DNN module 301 can generate and train DNNs. For example, DNN module 301 can define the hierarchical architecture of the DNN. DNN module 301 can also determine the intrinsic parameters of the DNN through the DNN training process. DNN module 301 can also determine one or more hyperparameters that define how the DNN is trained. Example hyperparameters are sparsity rates that define the sparsity level of one or more deep learning tensors of the DNN.

[0049] DNN module 301 may compress the DNN, for example, during or after training. In some embodiments, DNN module 301 may prune weights in one or more layers of the DNN by reducing non-zero value weights to zero. DNN module 301 may prune weights based on a target weight sparsity rate. The weight sparsity rate may be the ratio of the number of zero-value weights to the total number of weights. In the example where DNN module 301 prunes weights during DNN training, DNN module 301 may prune the layer weights after one or more epochs to achieve the target sparsity rate. DNN module 301 may prevent pruned weights from changing values ​​during the remainder of the training process. Alternatively, DNN module 301 may allow pruned weights to change values ​​such that pruned zero-value weights may have non-zero values ​​after further training. DNN module 301 may prune the layer weights again after one or more additional epochs.

[0050] DNN module 301 can deploy trained, compressed, or validated DNNs for use in deep learning applications. In some embodiments, DNN module 301 can distribute trained, compressed, or validated DNNs to means or systems that can use the DNNs to perform tasks for which the DNNs have been trained (e.g., image classification, motion planning, etc.). In other embodiments, DNN module 301 can use DNN accelerator 302 to facilitate DNN deployment. For example, DNN module 301 can receive data from means or systems coupled to DNN system 300 and input the received data (or data generated by DNN module 301, for example, based on the received data) into the DNN. DNN module 301 can compile instructions executable by DNN accelerator 302 to perform DNN operations according to the DNN's model definition. DNN module 301 can generate instructions (e.g., configuration files) to control the operation of DNN accelerator 302 during DNN execution. DNN module 301 can receive the output of the DNN from DNN accelerator 302. DNN module 301 can transmit the output of the DNN (or the result of processing the output of the DNN by DNN module 301) to a device or system. In some embodiments, DNN module 301 can control the execution process of a trained, compressed, or validated DNN. DNN module 301 can act as a compiler for the DNN executed by DNN accelerator 302. DNN module 301 can compile the DNN and generate a configuration file based on which the DNN can be executed. Figure 4 The document describes and illustrates certain aspects of the DNN module 301.

[0051] DNN accelerator 302 executes the DNN provided by DNN module 301. For example, DNN accelerator 302 can execute the DNN by running deep learning operations within the DNN. The process of performing deep learning operations is also referred to as the process of executing deep learning operations or performing deep learning operations. The execution of the DNN can be used to train the DNN or to use the DNN to perform AI tasks.

[0052] like Figure 3As shown, the DNN accelerator 302 includes a memory 310, a DMA (Direct Memory Access) engine 320, and a data processing unit 330 (referred to separately as "data processing unit 330"). In other embodiments, alternative configurations, different, or additional components may be included in the DNN accelerator 302. For example, the DNN accelerator 302 may include more than one memory 310 or DMA engine 320. As another example, the DNN accelerator 302 may include a single data processing unit 330. Furthermore, the functionality of the components attributed to the DNN accelerator 302 may be implemented by different components included in the DNN accelerator 302 or by different systems. The components of the DNN accelerator 302 may be implemented in hardware, software, firmware, or some combination thereof.

[0053] Memory 310 stores data associated with deep learning operations performed by DNN accelerator 302. In some embodiments, memory 310 may store data to be used by data processing unit 330 for DNN execution. Memory 310 may store weights, such as the weights of convolutional layers determined by training the DNN. Memory 310 may further store inputs to or outputs of DNN layers, such as data generated by data processing unit 330 based on deep learning operations performed in the DNN. Example deep learning operations include convolution (also referred to as "convolution operation"), layer normalization operation, SoftMax operation, matrix multiplication operation, pooling operation, element-wise operation, activation function, other types of deep learning operations, or some combination thereof. Memory 310 may store instructions executable by DNN accelerator 302, such as instructions executable by data processing unit 330. Memory 310 may be the main memory of DNN accelerator 302. In some embodiments, memory 310 includes one or more dynamic random access memories (DRAM).

[0054] DMA engine 320 facilitates data transfer between local memory 340 and memory 310 of data processing unit 330. For example, DMA engine 320 can read data from memory 310 and write data to local memory 340 of data processing unit 330. As another example, DMA engine 320 can read data from local memory 340 of data processing unit 330 and write data to memory 310. DMA engine 320 provides DMA features that allow data processing unit 330 to initiate data transfer between local memory 340 and memory 310 and to perform other operations while data transfer is being initiated. In some embodiments, DMA engine 320 can read tensors from memory 310 and modify the tensors in a manner optimized for data processing unit 330 before writing the tensors to local memory 340 of data processing unit 330.

[0055] Data processing unit 330 performs deep learning operations within a DNN. For example, data processing unit 330 can execute a DNN layer by running one or more deep learning operations within the DNN layer. Data processing unit 330 can execute a layer or a portion of a layer at a time. In some embodiments, multiple data processing units 330 can run the operations of a DNN layer in parallel. For example, multiple data processing units 330 can each execute a portion of the deep learning workload. Data can be shared among data processing units 330. Data processing unit 330 can also be referred to as a computation block or computation tile.

[0056] Data processing unit 330 can perform various types of deep learning operations, such as convolution, layer normalization, SoftMax operations, pooling, element-wise operations, linear operations, non-linear operations, and so on. Deep learning operations performed by data processing unit 330 include tensor operations, i.e., operations whose input is a tensor or whose output is a tensor. In the example, data processing unit 330 receives an input tensor and one or more convolution kernels and performs convolution using the input tensor and the convolution kernels. The result of the convolution can be an output tensor, which can be further computed, for example, by data processing unit 330 or another data processing unit 330.

[0057] exist Figure 3 In some embodiments, each data processing unit 330 includes a local memory 340, a loading module 360, a processing engine 370, a post-processing engine 380, and an exhaust module 390. Some or all of the components in the data processing unit 330 can be implemented on the same chip. In other embodiments, alternative configurations, different or additional components can be included in the data processing unit 330. Furthermore, the functionality of the components attributed to the data processing unit 330 can be implemented by different components included in the data processing unit 330, different data processing units 330, another component of the DNN accelerator 302, or different systems. The components of the data processing unit 330 can be implemented in hardware, software, firmware, or some combination thereof.

[0058] Local memory 340 is local to the corresponding data processing unit 330. Figure 3In one embodiment, the local memory 340 is internal to the data processing unit 330. In other embodiments, the local memory 340 may be external to the data processing unit 330. Data in the local memory 340 may be transferred to or from the memory 310, for example, via the DMA engine 320. In some embodiments, data in the local memory 340 may be transferred to or from the local memory 310 of another data processing unit 330. The local memory 340 may store data received, used, or generated by the loading module 360, processing engine 370, post-processing engine 380, or discharge module 390. Examples of such data may include input activations, weights, output activations, instructions, configuration files, etc.

[0059] In some embodiments, local memory 340 may store tensors to be processed by processing engine 370 or post-processing engine 380. The tensors may be input tensors for deep learning operations. Local memory 340 may also store tensors generated by processing engine 370 or post-processing engine 380. The tensors may be output tensors for deep learning operations. The layout of data points of tensors in local memory 340 may depend on the format in which the tensors are stored. In some embodiments, local memory 340 may store tensors in various formats, including Z-major format, X-major format, and Y-major format. For tensors with Z-major format, local memory 340 may store data points with the same (x, y) coordinates contiguously. For example, data points with the same (x, y) coordinates may be stored at a series of memory addresses in local memory 340. For tensors with ZXY or ZYX format, local memory 340 may store data points with the same (x, y) coordinates contiguously. For example, data points with the same (x, y) coordinates may be stored at a series of memory addresses in local memory 340. For tensors with an X-major format, local memory 340 can continuously store data points with the same (y, z) coordinates. For tensors with a Y-major format, local memory 340 can continuously store data points with the same (x, z) coordinates.

[0060] In some embodiments, local memory 340 may store dense tensors (e.g., dense activation tensors, dense weight tensors, etc.), sparse tensors (e.g., sparse activation tensors, sparse weight tensors, etc.), and so on. A dense tensor may be a tensor from which no zero-value elements (if any) have been removed. A dense tensor can be converted into a sparse tensor by removing one or more zero-value elements from it. A sparse tensor may also be referred to as a compressed tensor or a compacted tensor. The process of converting a dense tensor into a sparse tensor may be referred to as sparsity encoding. Sparsity encoding can also generate sparse tensors. Each element in a sparse tensor may correspond to a distinct element in a dense tensor and indicate whether an element in the dense tensor is zero. A sparse tensor may indicate the position of an element of a sparse tensor within a dense tensor. A sparse tensor may be a sparse bitmap, where each element is a bit. A sparse tensor can be converted into a dense tensor by a compaction process, where one or more zeros may be added to the sparse tensor based on the sparse tensor.

[0061] In some embodiments, local memory 340 includes one or more static random access memories (SRAMs). Local memory 340 may be byte-addressable, and each memory address identifies a single byte (eight bits) stored. In some embodiments, local memory 340 may include a memory bank. The number of data banks in local memory 340 may be 16, 64, 128, 356, 512, 1024, 2048, or other numbers. A memory bank may include multiple memory cells. In an example, a data bank may include 8, 16, 64, or different numbers of memory cells. A memory bank or a memory cell within a memory bank may have a memory address. In an example, a memory cell may store a single byte, and data larger than a single byte may be stored in memory cells with contiguous memory addresses (i.e., adjacent memory cells). For example, a memory cell may store an integer in INT8 format, whereas two memory cells might be needed to store a 16-bit number in FP16 or BF16 format. In some embodiments, 16 bits can be transferred from local memory 340 in a single read cycle. In other embodiments, 16 bits can be transferred from local memory 340 in multiple read cycles (such as two cycles).

[0062] The loading module 360 ​​loads data from local memory 340 to processing engine 370 or to post-processing engine 380. The loading module 360 ​​can read tensors from local memory 340. Tensors may include sparse activation tensors, sparse weight tensors, activation sparsity tensors, weight sparsity tensors, etc. In some embodiments, the loading module 360 ​​may load data based on a sparsity pattern. The loading module 360 ​​can select different data to send to processing engine 370 with different sparsity patterns.

[0063] Loading module 360 ​​loads instructions (such as configuration files) from local memory 340 into processing engine 370. The instructions can be used to configure or control the processing units of processing engine 370 to perform one or more deep neural network operations.

[0064] Processing engine 370 performs neural network operations on the DNN. In Figure 5 The exemplary processing engine 370 is described and illustrated in the document.

[0065] Post-processing engine 380 processes the output of processing engine 370. Post-processing engine 380 may include one or more post-processing elements. In some embodiments, the post-processing elements in post-processing engine 380 may be arranged in a row and column layout. In some embodiments, post-processing engine 380 computes an activation function. Post-processing engine 380 may receive the output of processing engine 370 as input to the activation function. In addition to or as an alternative to the activation function, post-processing engine 380 may also perform other types of post-processing on the output of processing engine 370. For example, post-processing engine 380 may apply a bias to the output of processing engine 370. In some embodiments, post-processing engine 380 may be bypassed for certain neural network operations.

[0066] The discharge module 390 discharges data from the processing engine 370 and / or from the post-processing engine 380. The discharge module 390 may write the data to local memory 340. The discharged data may be tensors, such as the output tensors of neural network operations. In some embodiments, the discharge module 390 may discharge data at the unit level of the processing engine 370. For each processing unit, the discharge module 390 may discharge the output of a processing element within the processing unit based on the row or column index of each processing element. For example, the discharge module 390 may use a series of cycles to discharge data from the processing unit. The discharge module 390 may discharge the output of some processing elements within the processing element in each cycle. The order of cycles may be configured based on configuration parameters indicating the operating mode of the loading module 360. The discharged data (e.g., tensors) may be further loaded into memory 310, for example, via DMA engine 320. Alternatively or additionally, the discharged data may be loaded by the loading module 360 ​​into the processing engine 370 for further computation, such as for performing deep learning operations in the next layer.

[0067] Figure 4A DNN module 301 according to some disclosed embodiments is described. The DNN module 301 includes an interface module 410, a training module 420, a validation module 440, and a data storage 460. In other embodiments, alternative configurations, different, or additional components may be included in the DNN module 301. Furthermore, the functionality of the components attributed to the DNN module 301 may be implemented by different components included in the DNN module 301, or by different modules or systems.

[0068] Interface module 410 facilitates communication between DNN module 301 and other modules or systems. For example, interface module 410 establishes communication between DNN module 301 and external data storage to receive data that can be used to train the DNN or input into the DNN to perform tasks. As another example, interface module 410 supports DNN module 301 in distributing the DNN to other systems, such as computing devices configured to apply the DNN to perform tasks.

[0069] Training module 420 trains the DNN using a training dataset. Training module 420 forms the training dataset. In an example where training module 420 trains the DNN to recognize objects in an image, the training dataset includes training images and training labels. The training labels describe the ground-truth classification of objects in the training images. In some embodiments, each label in the training dataset corresponds to an object in a training image. In an example where training module 420 trains a transformer-based neural network to predict the next token, the training dataset may include a large library of sequences of tokens. In some embodiments, a portion of the training dataset may be used to initially train the DNN, and the remainder of the training dataset may be reserved as a validation subset used by validation module 440 to validate the performance of the trained DNN. The portion of the training dataset excluding the tuning subset and validation subset may be used to train the DNN.

[0070] Training module 420 also determines the hyperparameters used to train the DNN. Hyperparameters are variables that specify the DNN training process. Hyperparameters are different from parameters inside the DNN (such as filter weights). In some embodiments, hyperparameters include variables that determine the architecture of the DNN, such as the number of hidden layers. Hyperparameters also include variables that determine how the DNN is trained, such as batch size, number of epochs, etc. Batch size defines the number of training samples to be processed (workthrough) before updating the parameters of the DNN. Batch size is the same as or less than the number of samples in the training dataset. The training dataset can be divided into one or more batches. The number of epochs defines how many times the entire training dataset is passed forward and backward through the entire network. The number of epochs defines how many times the deep learning algorithm processes the entire training dataset once. An epoch means that each training sample in the training dataset has the opportunity to update the parameters inside the DNN. Episodes can include one or more batches. The number of epochs can be 1, 5, 10, 50, 100, 500, 1000, or even larger.

[0071] Training module 420 may define the architecture of the DNN, for example, based on some hyperparameters. In some cases, training module 420 may receive a model definition that defines or specifies the architecture of the DNN. The architecture of the DNN may include multiple layers. Examples of layers may include convolutional layers, pooling layers, fully connected layers, normalization layers, SoftMax or logistic regression (LOGIT) layers, etc. After training module 420 defines the architecture of the DNN, training module 420 feeds a training dataset into the DNN. The training dataset includes multiple training samples. Training module 420 modifies the parameters internal to the DNN (“internal parameters of the DNN”) to minimize the error between the labels of the training objects generated by the DNN and the true labels of the objects. Internal parameters include weights used in the layers of the DNN. In some embodiments, training module 420 uses a cost function to minimize the error.

[0072] Training module 420 can train the DNN for a predetermined number of periods. The number of periods is a hyperparameter that defines how many times the deep learning algorithm will process the entire training dataset. One period means that each sample in the training dataset has the opportunity to update the internal parameters of the DNN. After training module 420 has completed the predetermined number of periods, training module 420 can stop updating the parameters in the DNN. The DNN with updated parameters is called the trained DNN.

[0073] The validation module 440 verifies the accuracy of the trained DNN. In some embodiments, the validation module 440 feeds samples from the validation dataset into the trained DNN and uses the DNN's output to determine model accuracy. In some embodiments, the validation dataset may consist of some or all of the samples from the training dataset. Alternatively or additionally, the validation dataset may include supplementary samples in addition to those in the training set. In some embodiments, the validation module 440 may determine an accuracy score that measures the precision, recall, or a combination of precision and recall of the DNN. The validation module 440 may use the following metrics to determine the accuracy score: precision = TP / (TP+FP) and recall = TP / (TP+FN), where precision can be how many (TP) the DNN correctly predicted out of the total number of predictions (TP or true positives + FP or false positives), and recall can be how many (TP) the DNN correctly predicted out of the total number of objects that actually have the problematic nature (TP+FN or false negatives). The F-score (F-score = 2*PR / (P+R)) unifies precision and recall into a single measurement.

[0074] The verification module 440 can compare the accuracy score with a threshold score. In an example where the verification module 440 determines that the accuracy score of the DNN is less than the threshold score, the verification module 440 instructs the training module 420 to retrain the DNN. In one embodiment, the training module 420 can iteratively retrain the DNN until a stopping condition is met, such as an accuracy measurement indicating that the DNN may be accurate enough or multiple training epochs have occurred.

[0075] Compiler 450 compiles information about the DNN (such as model definitions) into executable instructions (e.g., configuration files) that can be executed, for example, by DNN accelerator 302, to perform neural network operations in the DNN. The model definition may include one or more neural network operations to be performed by the DNN. In some embodiments, compiler 450 may generate a graph representing the DNN. The graph may include nodes and edges. Nodes may represent specific neural network operations in the DNN. Edges may connect two nodes and represent a connection between two corresponding neural network operations. In an example, an edge may encode a tensor flowing from one neural network operation to another. The tensor may be the output tensor of the first neural network operation and the input tensor of the second neural network operation. Edges may encode one or more properties of the tensor, such as size, shape, storage format, etc. Compiler 450 may use the graph to generate instructions (e.g., configuration files, configuration parameters, etc.). Instructions may include low-level machine code that can be executed by components of DNN accelerator 302 (e.g., processing engine 370). Low-level machine code can be in binary format, which can be loaded onto the DNN accelerator 302 and executed by components of the DNN accelerator 302 (e.g., processing engine 370). Instructions will be executed by components of the DNN accelerator 302 (e.g., processing engine 370) to execute the DNN.

[0076] Data storage 460 stores data received, generated, used, or otherwise associated with DNN module 301. For example, data storage 460 stores datasets used by training module 420 and validation module 440. Data storage 460 may also store data generated by training module 420 and validation module 440, such as hyperparameters used to train the DNN, intrinsic parameters of the trained DNN (e.g., weights), data used for sparsity acceleration (e.g., sparse bitmaps), etc. Data storage 460 may store configuration parameters, configuration files, instructions generated by compiler 450, etc. Data storage 460 may include one or more memories. Figure 4 In one embodiment, data storage 460 is a component of DNN module 301. In other embodiments, data storage 460 may be external to DNN module 301 and communicate with DNN module 301 via a network.

[0077] Figure 5 A processing engine 370 according to some disclosed embodiments is described. The processing engine 370 may be included as part of a data processing unit, such as... Figure 3The data processing unit 330. The processing engine 370 may include one or more processing units 502. In some embodiments, the processing units 502 may be arranged in one or more rows and / or one or more columns in the processing engine 370. In some embodiments, the processing units 502 may be arranged as one or more sets or arrays of processing units 502 performing different operations.

[0078] Each processing unit may include one or more processing elements. In some cases, a processing unit includes a single processing element. In some cases, a processing unit includes multiple processing elements. Processing elements may be arranged in an array. Processing elements may be arranged in rows and / or columns. In some cases, a processing unit may include one or more processing elements performing the same operation. In some cases, a processing unit may include one or more processing elements performing different operations. In some cases, at least some of the processing elements in a processing unit may be arranged to perform operations in parallel. In some cases, at least some of the processing elements in a processing unit may be arranged to perform operations serially.

[0079] Processing elements can perform arithmetic operations associated with neural network or DNN operations. In some cases, one or more processing elements can be arranged in an array including rows and columns. Examples of processing elements may include multiplication units, division units, scaling units, addition units, accumulator units, subtractor units, logarithmic units, exponentiation units, multiply-accumulate (MAC) units, bit shift units, square root units, etc. Processing elements in a processing unit can be arranged to perform arithmetic operations on an input vector to generate an output vector (in parallel), sometimes referred to as vector processing. Processing elements in a processing unit can also perform scalar operations.

[0080] Processing engine 370 may include a controller 504 that can configure circuit modules of one or more processing units 502 to perform arithmetic operations. In some cases, controller 504 may configure one or more processing units 502 (or individual processing elements within processing units 502) to perform operations in a specific order or manner. In some cases, controller 504 may configure one or more processing units 502 (or individual processing elements within processing units 502) based on instructions loaded into instruction buffer 506. Controller 504 may include a program counter for determining the instructions loaded into instruction buffer 506 to be executed by one or more processing units 502 (or individual processing elements within processing units 502).

[0081] (for example, through) Figure 3The instructions loaded into the instruction buffer 506 by the loading module 360 ​​can signal which processing units 502 (or individual processing elements in processing units 502) want to execute or perform one or more operations.

[0082] It can be done Figure 3 The loading module 360 ​​loads data into the data buffer 508. The data can then be used by the processing unit 502. The data generated by the processing unit 502 can be discharged from the data buffer 508 by the discharge module 390. Figure 3 Local memory 340.

[0083] Data buffer 508 may include one or more of the following: one or more input data buffers and one or more output data buffers. Data buffer 508 may include one or more weight / parameter buffers. Data buffer 508 may store operands for one or more processing elements of processing unit 502. Data buffer 508 may store outputs generated by one or more processing elements of processing unit 502.

[0084] Instructions loaded into instruction buffer 506 can signal which data stored in data buffer 508 should be processed by processing unit 502 (or the individual processing elements within processing unit 502). In some cases, processing unit 502 (or the individual processing elements within processing unit 502) can read data from data buffer 508 at the default location of processing unit 502 or an individual processing element within processing unit 502.

[0085] After the processing unit 502 generates output data, the instruction loaded into the instruction buffer 506 can signal where to store the output data in the data buffer 508. In some cases, the processing unit 502 (or the individual processing elements in the processing unit 502) can write data to the data buffer 508 at the default location of the processing unit 502 or the individual processing elements in the processing unit 502.

[0086] Figure 3 The loading module 360 ​​can load data into certain locations in the data buffer 508. Figure 3 The discharge module 390 can discharge data to be stored from the data buffer 508. Figure 3 Data in local memory 340 and / or memory 310.

[0087] refer to Figure 3-5The DNN System 300 illustrates one implementation of a processor designed to accelerate the execution of DNNs. The processor's architecture design can vary depending on the application requirements. The architecture design can differ based on factors such as the number of data processing units, the number of processing engines, the number of processing units, support for vector processing, support for sparse patterns, the type or set of elements processed, storage requirements, and buffer size.

[0088] The performance of the DNN execution on the DNN system 300 can be limited by the availability of certain types of processing units 502 or certain types of processing elements. Performance can be limited by the amount of parallel processing (such as vector processing) the architecture can support. Performance can be limited by the amount of storage available in the data buffer 508. Performance can be limited by the amount of storage available in the local memory 340. Performance can be limited by the speed of memory access at different cache or memory levels (e.g., the number of cycles) (e.g., the speed of loading module 360, the speed of evading module 390, etc.). Performance can be limited by the number of cycles that processing unit 502 or one or more processing elements in processing unit 502 will take to execute operations. In some cases, performance can be limited by how processing unit 502 or one or more processing elements in processing unit 502 are used to compile and execute neural network operations (e.g., MatMul operations, SoftMax operations, etc.). SoftMax's Simple 3-Pass Version

[0089] Figure 6 This describes a simplified 3-pass version of an exemplary operation of performing SoftMax on an input tensor according to some disclosed embodiments. The SoftMax operation is mathematically defined by Equation 1. The input tensor (or input vector) for the SoftMax operation may include N elements x. i Where i = 1, ..., N. The input tensor can have a sequence length of N. The output tensor (or output vector) of the SoftMax operation can include N elements y. i , where i = 1, ..., N. The output tensor can have a sequence length of N. The mathematical representation of the SoftMax operation is reproduced here:

[0090] In 602, the maximum value m is initialized to negative infinity. The denominator d is initialized to 0.

[0091] In 604, the process iterates over the input tensor x. i The first pass through the elements. The first pass through the input tensor x. i Iterate through the elements to determine the input tensor x. iThe maximum value m of the elements. For each element x of the input tensor. j (For j in the range (1, N), if element x j Greater than the maximum value m(if(x) j If the maximum value m is greater than or equal to the value of x, then the maximum value m is set to the value of the element x. j (m=x j If element x j Not greater than the maximum value m(if(x) j If ≤m), then the maximum value m remains unchanged. The maximum value m determined in the first pass corresponds to max(x) in equation 1. i ).

[0092] In 606, the iteration over the input tensor x is performed. i The elements are passed through a second time. The second pass iterates through the input tensor x. i The elements of the input tensor are iterated to determine the denominator value d. For each element x of the input tensor... j (For j in the range (1, N), the denominator d is updated to the denominator and the natural base e (element x). j Subtract the sum of the powers of the maximum value (m). The denominator value d determined in the second pass corresponds to the value in equation 1.

[0093] In 608, the iteration over the input tensor x is performed. i The third pass through the elements. The third pass passes through the input tensor x. i Iterate through the elements to determine the output tensor y. i The value of x for each element x of the input tensor. j (For j in the range (1, N), the elements x of the input tensor j The element x is updated to the natural base e. j Subtract the maximum value m to the power of (i.e.) Divide by the denominator value d determined in the second pass (i.e. The input tensor x has the updated value determined in the third pass. i It can be used as the output tensor y i Output. Output tensor y i The value corresponds to the value in equation 1.

[0094] In fact, in 606 for The calculated values ​​are cached and reused in 608 to avoid having to perform the same exponentiation again. This limits the memory of SoftMax's simple 3-pass version, or in other words, it limits the memory available for storage in 606. The fast storage of computed values ​​limits performance. SoftMax's second version

[0095] Figure 7 This describes a two-pass version of an exemplary operation of performing SoftMax on an input tensor according to some disclosed embodiments. The SoftMax operation is mathematically defined by Equation 1. The input tensor (or input vector) for the SoftMax operation may include N elements x. i The input tensor can have a sequence length of N. The output tensor (or output vector) of the SoftMax operation can include N elements yi, where i = 1, ..., N. The output tensor can have a sequence length of N. The mathematical representation of the SoftMax operation is reproduced here:

[0096] In 702, the maximum value m is initialized to negative infinity. The denominator d is initialized to 0.

[0097] In 704, the iteration over the input tensor x is performed. i The first pass through the elements. The first pass through the input tensor x. i Iterate through the elements to determine the input tensor x. i The maximum value m and the denominator d of the elements in the input tensor. For each element x of the input tensor j (For j in the range (1, N), if element x j Greater than the maximum value m(if(x) j If the denominator d is greater than or equal to the natural base e, then the denominator d is set to the maximum value m minus the element x. j The product of powers And the maximum value m is set to element x j (m=x j If element x j If x is not greater than the maximum value m (if x ≤ m), then the maximum value m remains unchanged. This applies regardless of the element x. j Whether the value is greater than the maximum value m for each element x of the input tensor j (For j in the range (1, N), the denominator d is updated to the denominator d and the natural base e (element x). j Subtract the sum of the powers of the maximum value (m). The maximum value m determined in the first pass corresponds to max(x) in Equation 1. i The denominator value d determined in the first pass corresponds to the value in equation 1. The first pass in 704 combines the first pass in 602 and the second pass in 604 to pass through the input tensor x. i The element is repeated only once.

[0098] In 706, iterate over the input tensor x. i The elements are passed through a second time. The second pass iterates through the input tensor x. i Iterate through the elements to determine the output tensor y. i The value of x for each element x of the input tensor. j (For j in the range (1, N), the elements x of the input tensor j The element x is updated to the natural base e. j Subtract the maximum value m to the power of (i.e.) Divide by the denominator value d determined in the second pass (i.e. This update corresponds to what we see in 706. The input tensor x has the updated value determined in the second pass. i It can be used as the output tensor y i Output. Output tensor y i The value corresponds to the value in equation 1. SoftMax's two revised versions

[0099] Understand the simple 3-pass version (in) Figure 6 (See in) and 2-pass version (in) Figure 7 (As seen in the text) involves recognizing that different sets of operations can be used to compute equation 1 differently. The mathematical representation of the SoftMax operation is reproduced here:

[0100] Instead of calculating SoftMax directly, it is possible to first calculate the logarithm of SoftMax and then raise the logarithm to obtain SoftMax. The logarithm of SoftMax can be precisely rewritten using the following formula: in c = max(x) i )+ln(d) (Equation 6)

[0101] The logarithm of SoffMax raised to the natural base e yields SoftMax as shown below:

[0102] For example, as previously stated Figure 6 The simple 3-times version explained in the text and in Figure 7 The calculation is performed in the second version as described in the text. As required, generating SoffMax using Equation 4-8 does not necessitate performing N divisions. This is achieved by introducing two scalar operations, such as ln(d) and c = max(x), through the use of Equation 4-7. i )+ln(d).

[0103] Figure 8 This describes a modified, two-pass version of an exemplary operation for performing SoftMax on an input tensor, based on some disclosed embodiments. The input tensor (or input vector) for the SoftMax operation may include N elements x. i Where i = 1, ..., N. The input tensor can have a sequence length of N. The output tensor (or output vector) of the SoftMax operation can include N elements y. i , where i = 1, ..., N. The output tensor can have a sequence length of N. The modified 2-pass version of the operation follows the mathematical formula of Equation 4-7.

[0104] Brief Reference Figure 3-4 When the DNN module 301 (e.g., compiler 450) receives or determines a neural network model definition specifying the SoftMax operation being applied to the input tensor, the DNN module 301 can convert the SoftMax operation into a form that can be processed by one or more processing engines (e.g., Figure 3 The processing engine 370) executes instructions to perform SoftMax operations. The DNN module 301 can generate instructions that can be executed by one or more processing engines. It does not generate instructions that are executed in... Figure 6-7 The operations described herein are to execute the simple 3-pass and 2-pass versions of SoftMax instructions, but the DNN module 301 can generate instructions to execute such operations. Figure 8 The instructions for the operation are shown in the modified second-version description. These instructions will cause one or more processing engines of the DNN accelerator 302 (e.g., Figure 3 The processing engine 370) executes as follows: Figure 8 The operation is shown in the modified version 2. The instruction can cause one or more processing engines of the DNN accelerator 302 to perform the SoffMax operation on the input tensor.

[0105] The processing engine can accept the input tensor x from the SoftMax operation of a neural network. i For example, the processing engine can receive... Figure 2 The SoftMax operation takes a 206-bit input tensor x. i .

[0106] In 802, the maximum value m is initialized to negative infinity. The denominator d is initialized to 0.

[0107] In 804, the processing engine performs the iteration over the input tensor x. i The first pass through the elements. The processing engine can perform an pass through the input tensor x. i The first pass through the elements, for example, passing through the input tensor x. i Iterate over the elements based on the input tensor x i One or more elements are used to maintain the maximum value m and the denominator value d.

[0108] For each element x of the input tensor j (For j in the range (1, N), if element x j Greater than the maximum value m(if(x) j If the denominator d is greater than or equal to the natural base e, then the denominator d is set to the maximum value m minus the element x. j The product of powers And the maximum value m is set to element x j (m=x j If element x j Not greater than the maximum value m(if(x) j If x ≤ m), then the maximum value m remains unchanged. Regardless of the element x... j Whether the value is greater than the maximum value m for each element x of the input tensor j (For j in the range (1, N), the denominator d is updated to the denominator d and the natural base (element x). j Subtract the sum of the powers of the maximum value (m). By iterating through the elements x of the input tensor in the processor engine i During iteration, this update of the denominator d is performed, and according to Equation 5, iterates through the elements x of the input tensor. i The denominator d at the end of the iteration will be equal to

[0109] In some embodiments, maintaining the maximum running value m and the denominator d in 804 includes determining the first element x of the input tensor. j Is it greater than the running maximum value m? In 804, maintaining the running maximum value m and the denominator d can further include, in response to determining the first element x of the input tensor. j A value greater than the maximum running value m is used to determine the product of the denominator d and the exponent of the base. Determining the product of the denominator d and the exponent of the base may include determining what is seen in 804. The base can be a natural number e. The exponent can be the maximum value m minus the first element x of the input tensor. j The exponent can be correlated with mx as seen in 804. j In 804, maintaining the maximum running value m and the denominator d can further include the first element x in response to determining the input tensor.j Greater than the maximum running value m and based on the first element x of the input tensor j To update the maximum running value m. Updating the maximum running value corresponds to m = x as seen in 804. j .

[0110] In some embodiments, maintaining the maximum running value m and the denominator d in 804 may further include pervading the input tensor x in the execution of 804. i After iterating over the elements, based on the product (e.g.) ) and the exponent of the base (e.g. The sum of the values ​​of x and y is used to update the denominator d. The exponent can be the first element x of the input tensor. j Subtract the maximum running value m (i.e., x) j -m). The sum can correspond to 804. This update of the denominator d can correspond to According to Equation 5, the elements x of the input tensor are iterated over. i The denominator d at the end of the iteration will be equal to

[0111] In 804, when iterating over the input tensor x i Iterate through the elements to obtain the denominator value d (e.g. ) and running at the maximum value m (e.g., m = max(x) j After that, the processing engine updates the denominator d based on the logarithm of the denominator d (e.g., ln(d)). This update corresponds to d←ln(d) in 804. The logarithm of the denominator d can be the natural logarithm of the denominator value or the logarithm with the natural number e as the base. The update of the denominator d based on the logarithm of the denominator d calculates the value ln(d) used in Equation 6.

[0112] In 804, after updating the denominator d based on the logarithm of the denominator d, the processing engine determines the operand c based on the sum of the denominator d (which has the value ln(d)) and the running maximum value m. In 804, the sum can correspond to d + m. In 804, this determination corresponds to c = d + m. This determination corresponds to calculating the value c according to Equation 6.

[0113] Return to reference Figure 7 The 704 error, executed twice in SoftMax, causes the processing engine to perform P operations to complete the operation in the 704 error. (Return to reference) Figure 8The 804, which executes a modified 2-pass version of SoftMax, can cause the processing engine to perform P operations plus scalar logarithmic operations (e.g., d←ln(d) in 804) and scalar 2-input additions (e.g., c=d+m in 804) to complete the operations in 804.

[0114] In 806, the processing engine performs the iteration over the input tensor x. i The elements are passed through a second time. The second pass iterates through the input tensor x. i Iterate through the elements to determine the output tensor y. i The value of x for each element x of the input tensor. j (For j in the range (1, N), the elements x of the input tensor j The element x is updated to the natural base e. j Subtract the operand from the power of c). The power can be represented by x as seen in 806. j -c. This update corresponds to what we see in 806. The input tensor x has the updated value determined in the second pass. i It can be output as an output tensor y i Output tensor y i The value corresponds to the value in equation 7.

[0115] In some embodiments, the input tensor x i The second pass of the elements involves the processor engine calculating the base by raising it to the power of the exponent (e.g., To update at least the first element x of the input tensor j The exponent can be the first element x of the input tensor. j Subtract the operand c. The exponent can be represented by x as seen in 806. j -c. This update corresponds to what we see in 806. The operations in 806 ultimately calculate the output tensor according to Equation 7. The SoftMax value.

[0116] Return to reference Figure 7 The 706, executing a 2-pass version of SoftMax, can cause the processing engine to perform N 2-input subtractions (to determine x). j -m), N exponential operations (to determine) ) and N divisions (to determine This completes the operation in 706. (Return to reference) Figure 8 The 806, executing a 2-pass version of SoffMax's modifications, allows the processing engine to perform N 2-input subtractions (to determine x). j-c), N exponential operations (to determine) This completes the operations in 806. 806 does not have division.

[0117] Although the 804 added two scalar operations, saving N divisions in the 806 means a dramatic reduction in the computational cost of performing the SoftMax operation. In the attention layer of a transformer-based neural network, the SoftMax operation is performed / repeated N times for a sequence length N. This means that the modified 2-pass version of SoftMax saves quadratic (N) divisions. 2 The division (e.g., FP32 inverse multiplication) is performed, and only scalar operations on linear quantities (N) are added. Because the modified 2-pass version of SoftMax implements the operations corresponding to Equation 4-7, and Equation 4-7 is mathematically equivalent to Equation 1, therefore... Figure 8 The modified SoftMax version described herein will not result in a loss of accuracy. Computational savings increase as the sequence length N becomes longer.

[0118] After the processing engine executes the instructions, it can output an output tensor y containing the result of the SoftMax operation. i In some cases, the processing engine can output the input tensor x. i (with the updated values ​​of its elements) as the output tensor y i Output tensor y i Including an input tensor x that is transformed into one or more probabilities of a probability distribution. i One or more elements. Output tensor y. i One or more elements of the form have values ​​ranging from 0 to 1, such that the output tensor y has one or more values ​​ranging from 0 to 1. i The sum of one or more values ​​of one or more elements is equal to 1. A two-pass version with approximate matrix multiplication and SoftMax modifications.

[0119] Further understanding involves recognizing that integer arithmetic can be used to approximate equations 4-7 in order to enable faster binary and integer operations in the processing engine. Instead of performing exponentiation using a base of e, approximation or quantization techniques can be used to modify... Figure 8 The operations described in the text. Specifically, exponentiation involving a base of e can be replaced by exponentiation involving a base of 2. Exponentiation involving a base of e would require floating-point operations. Bit shifting and integer operations can be used to efficiently implement exponentiation involving a base of 2 without requiring floating-point operations. Bit shifting can also be used to perform multiplication with numbers represented as powers of 2 faster.

[0120] Figure 9This describes a modified two-pass version of an operation, based on some disclosed embodiments, for performing SoftMax on an input tensor and matrix multiplication of the output tensor and matrix V, with approximate exemplary characteristics. The input tensor (or input vector) for the SoftMax operation may include N elements x. i Where i = 1, ..., N. The input tensor can have a sequence length of N. The output tensor (or output vector) of the SoftMax operation can include N elements y. i , where i = 1, ..., N. The output tensor can have a sequence length of N. The output tensor (or output vector) of the SoftMax operation can be multiplied by the matrix V.

[0121] Brief Reference Figure 3-4 When the DNN module 301 (e.g., compiler 450) receives or determines a neural network model definition specifying the SoftMax operation applied to the input tensor and matrix multiplication operations on the output tensor and matrix, the DNN module 301 can convert the SoftMax operation and matrix multiplication operations into operations that can be processed by one or more processing engines (e.g., Figure 3 The processing engine 370 executes instructions to perform SoffMax operations and matrix multiplication. The DNN module 301 can generate instructions that can be executed by one or more processing engines. It does not generate instructions that can be executed by one or more processing engines. Figure 6-7 The operations described herein are to execute the simple 3-pass and 2-pass versions of SoftMax instructions, but the DNN module 301 can generate instructions to execute them in, for example, Figure 9 The instructions described in the second-pass version, with similar modifications, illustrate the computational instructions. These instructions will cause one or more processing engines of the DNN accelerator 302 (e.g., Figure 3 The processing engine 370) executes in such a way Figure 9 The operation described is similar to that seen in the 2-pass version with modifications. The instructions can cause one or more processing engines of the DNN accelerator 302 to perform a SoftMax operation on the input tensor and a matrix multiplication operation on the output tensor and matrix V.

[0122] The processing engine can accept the input tensor x from the SoftMax operation of a neural network. i For example, the processing engine can receive... Figure 2 The SoftMax operation takes a 206-bit input tensor x. i .

[0123] In 902, the maximum value m is initialized to negative infinity. The denominator d is initialized to 0.

[0124] In 904, the processing engine performs the iteration over the input tensor x. iThe first pass through the elements. The processing engine can perform an pass through the input tensor x. i The first pass through the elements, for example, passing through the input tensor x. i Iterate over the elements based on the input tensor x i One or more elements are used to maintain the maximum value m and the denominator value d.

[0125] For each element x of the input tensor j (For j in the range (1, N), if element x j Greater than the maximum value m(if(x) j If the sum of the denominator values ​​is greater than or equal to m, then the denominator value d is set to the base of 2 (the maximum value m minus the element x). j The product of powers (i.e.) And the maximum value m is set to element x. j (m=x j If element x j Not greater than the maximum value m(if(x) j If x ≤ m), then the maximum value m remains unchanged. Regardless of the element x... j Whether the value is greater than the maximum value m for each element x of the input tensor j (For j in the range (1, N), the denominator d is updated to the sum of the denominator d and the base 2 (element x). j Subtract the sum of the powers of the maximum value m (i.e.) By iterating through the elements x of the input tensor in the processor engine... i This update of the denominator d is performed during iteration, iterating over the elements x of the input tensor. i The denominator d at the end of the iteration will be equal to

[0126] In some embodiments, maintaining the maximum running value m and the denominator d in 904 includes determining the first element x of the input tensor. j Is it greater than the running maximum value m? In 904, maintaining the running maximum value m and the denominator d can further include, in response to determining the first element x of the input tensor. j A value greater than the maximum value m is used to determine the product of the denominator d and the exponent of the base. Determining the product of the denominator d and the exponent of the base may include determining what is seen in 904. The base can be a power of 2 (such as 2). The exponent can be the maximum value m minus the first element x of the input tensor. j The exponent can be correlated with mx as seen in 904. j In 904, maintaining the maximum value m and the denominator d can further include the first element x in response to determining the input tensor. j Greater than the maximum running value m and based on the first element x of the input tensorj To update the maximum running value m. Updating the maximum running value corresponds to m = x seen in 904. j .

[0127] In some embodiments, maintaining the maximum value m and the denominator d in 904 may further include pervading the input tensor x in the execution of 904. i After iterating over the elements, based on the product (e.g.) ) and the exponent of the base (e.g. The sum of the values ​​of x and y is used to update the denominator d. The exponent can be the first element x of the input tensor. j Subtract the maximum running value m (i.e., x) j -m). The sum can correspond to 904. This update of the denominator d can correspond to The element x that pervades the input tensor i The denominator d at the end of the iteration will be equal to

[0128] In 904, the input tensor x is... i Iterate through the elements to obtain the denominator value d (e.g. ) and running at the maximum value m (e.g., m = max(x) j After that, the processing engine updates the denominator d based on the logarithm of the denominator d (e.g., ln(d)). This update corresponds to d←ln(d) in 904. The logarithm of the denominator d can be the natural logarithm of the denominator value or the logarithm with the natural number e as the base. The update of the denominator d based on the logarithm of the denominator d calculates the value ln(d) used in Equation 6.

[0129] In 904, after updating the denominator d based on the logarithm of the denominator d, the processing engine determines the operand c based on the sum of the denominator d (which has the value ln(d)) and the running maximum value m. In 904, the sum can correspond to d + m. In 904, this determination corresponds to c = d + m. This determination corresponds to calculating the value c according to Equation 6.

[0130] Return to reference Figure 7 The 704 error, executed twice in SoftMax, causes the processing engine to perform P operations to complete the operation in the 704 error. (Return to reference) Figure 9In 904, a modified 2-pass version of SoftMax is executed. 904 can cause the processing engine to perform P operations plus scalar logarithmic operations (e.g., d ← ln(d)) and scalar 2-input additions (e.g., c = d + m) to complete the operation in 904. When comparing a modified 2-pass version of SoftMax with a modified 2-pass version of SoftMax, 904 is used to determine... The N exponent operations can be compared to those used in the 806 to determine The N-fold exponentiation is faster.

[0131] In 906, the processing engine performs the iteration over the input tensor x. i The elements are passed through a second time. The second pass iterates through the input tensor x. i Iterate through the elements to determine the output tensor y. i The value of x for each element x of the input tensor. j (For j in the range (1, N), the elements x of the input tensor j The element x is updated to a base of 2. j Subtract the operand from the power of c). The power can correspond to x as seen in 906. j -c. This update corresponds to what we see in 906. The input tensor x has the updated value determined in the second pass. i It can be output as an output tensor y i Output tensor y i The value corresponds to

[0132] In some embodiments, the input tensor x i The second pass of the elements involves the processor engine calculating the exponent of the base 2 (e.g., ...). To update at least the first element x of the input tensor j The exponent can be the first element x of the input tensor. j Subtract the operand c. The exponent can be represented by x as seen in 906. j -c. This update corresponds to what we see in 906. The operations in 906 ultimately calculate the output tensor. The approximation of the SoffMax value.

[0133] Return to reference Figure 7 The 706, executing a 2-pass version of SoftMax, can cause the processing engine to perform N 2-input subtractions (to determine x). j -m), N exponential operations (to determine) ) and N divisions (to determine This completes the operation in 706. (Return to reference) Figure 9 The 906 version, a modified SoftMax implementation, allows the processing engine to perform N 2-input subtractions (to determine x). j -c), N exponential operations (to determine) This completes the operation in 906. 906 does not have division. When comparing a two-pass version with approximate SoftMax modifications to a two-pass version of SoftMax modifications, 906 uses [method / method] to determine [the result / function]. The N exponent operations can be determined more efficiently than those used in the 806. The N-fold exponentiation is faster.

[0134] Although the 904 version adds two scalar operations, saving N divisions in the 906 version means a dramatic reduction in the computational cost of performing the SoftMax operation. In the attention layer of a transformer-based neural network, the SoftMax operation is performed / repeated N times for a sequence length N. This means that the modified 2-pass version of SoftMax saves quadratic (N) times. 2 The division (e.g., FP32 inverse multiplication) is performed in 904 and 906, with only scalar operations added for the linear quantity (N). The computational savings increase as the sequence length N becomes longer. When comparing a 2-pass version with a modified SoftMax approximation to a 2-pass version with a modified SoffMax, the exponentiation performed in 904 and 906 can be faster than that performed in 802 and 806. Minimal accuracy loss can occur due to the approximation or quantization introduced in 904 and 906. However, fine-tuning can be used to reduce accuracy loss if desired.

[0135] After the processing engine executes the instructions, it can output an output tensor y containing the result of the SoftMax operation. i In some cases, the processing engine can output the input tensor x. i (with the updated values ​​of its elements) as the output tensor y Z Output tensor y i Including an input tensor x that is transformed into one or more probabilities of a probability distribution. i One or more elements. Output tensor y. i One or more elements of the form have values ​​ranging from 0 to 1, such that the output tensor y has one or more values ​​ranging from 0 to 1. i The sum of one or more values ​​of one or more elements is equal to 1.

[0136] In 908, the processing engine can perform output tensor operations. Matrix multiplication with matrix V. In the attention layer, matrix V can correspond to a value matrix. Matrix multiplication performed in 908 can correspond to... Figure 2 The MatMul operation 208 receives the output from the SoftMax operation 206 and determines the output O. The value of the output tensor can be represented as a power of 2, where the power corresponds to the exponent (x). j -c). The exponent or power is the first element minus the operand c (i.e., x). j -c). Performing matrix multiplication in 908 involves performing many dot products, where each dot product includes the element y. i Multiplication of corresponding elements in matrix V and accumulation of products. Not performing operations on element y. i Instead of multiplying the elements of matrix V to determine the dot product, 908 could alternatively include shifting the elements of matrix V by multiple positions. The shifts could correspond to V << ROUND[x] as seen in 908. j -c]. The number of positions can correspond to the rounded integer value of an exponent or power, such as x. j -c. Performing bitwise shifts instead of floating-point multiplication of numbers can be significantly faster, although approximations may result in some loss of accuracy. Exemplary method for compiling SoftMax operations to be performed by the processing engine

[0137] Figure 10 This is a flowchart illustrating a method for compiling a SoftMax operation to be executed by a processor, according to some disclosed embodiments. A computing device (such as...) can be used. Figure 12 The computing device 1200 in the middle) executes method 1000. It can be used in... Figure 3-5 Method 1000 is executed by one or more parts as described in the document. Method 1000 can be executed by, for example, as described in... Figure 3-5 The exemplary method performed by the DNN module 301 and / or DNN accelerator 302 described herein.

[0138] In 1002, the DNN module can receive a neural network model definition that specifies the SoftMax operation being applied to the input tensor.

[0139] In 1004, the DNN module can generate one or more first instructions that can be executed by the processing engine to maintain the maximum and denominator values ​​based on one or more elements of the input tensor.

[0140] In 1006, the DNN module can generate a second instruction that can be executed by the processing engine to update the denominator value based on the logarithm of the denominator value.

[0141] In 1008, the DNN module can generate third instructions that can be executed by the processing engine to determine the operand value based on the first sum of the denominator value and the running maximum value.

[0142] In 1010, the DNN module can generate one or more fourth instructions that can be executed by the processing engine to update at least the first element of the input tensor by raising the first exponent of the base, where the first exponent is the first element of the input tensor minus the operation value.

[0143] One or more first instructions, second instructions, third instructions, and one or more fourth instructions cause the processing engine to perform a SoftMax operation on the input tensor.

[0144] In some embodiments, the SoftMax operation on the input tensor transforms one or more elements of the input tensor into one or more values ​​in the range from 0 to 1, such that the second sum of one or more values ​​of the input tensor equals 1.

[0145] In some embodiments, one or more first instructions may maintain the running maximum value and the denominator value by determining whether a first element of the input tensor is greater than the running maximum value. One or more first instructions may further maintain the running maximum value and the denominator value in response to determining that the first element of the input tensor is greater than the running maximum value by: determining the product of the denominator value and a second exponent of the base, where the second exponent is the running maximum value minus the first element of the input tensor; and updating the running maximum value based on the first element of the input tensor. One or more first instructions may further maintain the running maximum value and the denominator value by updating the denominator value based on a third sum of the product and a third exponent of the base, where the third exponent is the first element of the input tensor minus the running maximum value.

[0146] In some embodiments, the logarithm of the denominator is the natural logarithm of the denominator.

[0147] In some embodiments, the base is a natural number e. In some alternative embodiments, the base is a power of 2 (such as 2).

[0148] In some embodiments, the neural network model definition further specifies matrix multiplication operations on the output tensor and matrix of the SoftMax operation. Method 1000 may further include generating one or more fifth instructions executable by the processing engine to perform displacement of a plurality of positions, the number of positions being a rounded integer value of a first exponent. Exemplary method for performing SoftMax operations by a processing engine

[0149] Figure 11 This is a flowchart illustrating a method for performing a SoftMax operation to be executed by a processor, according to some disclosed embodiments. A computing device (such as...) can be used. Figure 12 The computing device 1200 in the middle) is used to execute method 1100. It can be used in... Figure 3-5Method 1100 is executed by one or more parts as described in the document. Method 1100 can be performed by, for example, as described in... Figure 3-5 The exemplary method performed by the DNN module 301 and / or DNN accelerator 302 described herein.

[0150] In 1102, the processing engine (e.g., the processing engine of DNN accelerator 302) can receive the input tensors of the SoftMax operation of the neural network.

[0151] In 1104, the processing engine can maintain the maximum and denominator values ​​based on one or more elements of the input tensor.

[0152] In 1106, the processing engine can update the denominator value based on the logarithm of the denominator value.

[0153] In 1108, the processing engine can determine the operation value based on the first sum of the denominator value and the maximum running value.

[0154] In 1110, the processing engine can update at least the first element of the input tensor by raising the first exponent of the base, where the first exponent is the first element of the input tensor minus the operation value.

[0155] In 1112, the processing engine can output an output tensor. The output tensor can include one or more elements of the input tensor, which has been transformed into one or more probabilities of a probability distribution.

[0156] In some embodiments, one or more elements of the output tensor have values ​​ranging from 0 to 1, such that a second sum of one or more elements of the output tensor equals 1.

[0157] In some embodiments, maintaining the running maximum value and the denominator value may include determining whether a first element of the input tensor is greater than the running maximum value. In some embodiments, maintaining the running maximum value and the denominator value may further include, in response to determining that the first element of the input tensor is greater than the running maximum value: determining the product of the denominator value and a second exponent raised to the power of the base, where the second exponent is the running maximum value minus the first element of the input tensor; and updating the running maximum value based on the first element of the input tensor. In some embodiments, maintaining the running maximum value and the denominator value may further include updating the denominator value based on a third sum of the product and a third exponent raised to the power of the base, where the third exponent is the first element of the input tensor minus the running maximum value.

[0158] In some embodiments, the logarithm of the denominator is the natural logarithm of the denominator.

[0159] In some embodiments, the base is a natural number e. In some alternative embodiments, the base is a power of 2 (such as 2).

[0160] In some embodiments, the neural network further includes matrix multiplication operations on the output tensor and the matrix. Method 1100 may further include a processing engine performing displacements at multiple positions, the number of positions being a rounded integer value of a first exponent. Exemplary calculation of θ setting

[0161] Figure 12 This is a block diagram of an apparatus or system, such as an exemplary computing device 1200, according to some disclosed embodiments. One or more computing devices 1200 may be used to implement the functionality described herein using the diagrams. Multiple components illustrated in the diagrams may be included in the computing device 1200, but any one or more of these components may be omitted or copied as appropriate for the application. In some embodiments, some or all of the components included in the computing device 1200 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated on a single system-on-a-chip (SoC) die. Additionally, in various embodiments, the computing device 1200 may not include… Figure 12 The computing device 1200 may include one or more of the components described herein, and may include interface circuit modules for coupling to one or more components. For example, the computing device 1200 may not include the display device 1206, but may include a display device interface circuit module (e.g., a connector and driver circuit module) to which the display device 1206 may be coupled. In another set of examples, the computing device 1200 may not include the audio input device 1218 or the audio output device 1208, but may include an audio input or output device interface circuit module (e.g., a connector and support circuit module) to which the audio input device 1218 or the audio output device 1208 may be coupled.

[0162] Computing device 1200 may include processing device 1202 (e.g., one or more processing devices, one or more of the same type of processing devices, or one or more of different types of processing devices). Processing device 1202 may include electronic circuit modules that process electronic data from data storage elements (e.g., registers, memories, resistors, capacitors, qubit units) to convert that electronic data into other electronic data that can be stored in registers and / or memories. Examples of processing device 1202 may include CPUs, graphics processing units (GPUs), quantum processors, machine learning processors, artificial intelligence processors, neural network processors, artificial intelligence accelerators, application-specific integrated circuits (ASICs), analog signal processors, analog computers, microprocessors, digital signal processors, field-programmable gate arrays (FPGAs), tensor processing units (TPUs), data processing units (DPUs), etc.

[0163] The computing device 1200 may include a memory 1204, which itself may include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high-bandwidth memory (HBM), flash memory, solid-state memory, and / or hard disk drives. The memory 1204 includes one or more non-transitory computer-readable storage media. In some embodiments, the memory 1204 may include memory sharing a die with the processing device 1202.

[0164] In some embodiments, memory 1204 includes storage for operations that can be executed to utilize the graph and the operations described herein (such as in...). Figure 8-11 One or more non-transitory computer-readable media containing instructions for the methods and operations described herein. In some embodiments, memory 1204 includes storage for executable instructions to perform... Figure 10 Method 1000 and Figure 11 Method 1100 operates instructions on one or more non-transitory computer-readable media. An exemplary portion is depicted that can be encoded as instructions and stored in memory 1204. Memory 1204 can store instructions encoding one or more exemplary portions (such as DNN module 301 or one or more portions of DNN module 301). The instructions stored in the one or more non-transitory computer-readable media can be executed by processing device 1202.

[0165] In some embodiments, as illustrated in the figures and described herein, memory 1204 may store data such as data structures, binary data, bits, metadata, files, binary large objects (blobs), etc. Exemplary data that may be stored in memory 1204 is depicted.

[0166] In some embodiments, memory 1204 may store one or more DNNs (and / or portions thereof). Memory 1204 may store training data used to train (trained) DNNs. Memory 1204 may store instructions for performing operations associated with training the DNNs. Memory 1204 may store input data, output data, intermediate outputs, and intermediate inputs of one or more DNNs. Memory 1204 may store one or more parameters used by one or more DNNs. Memory 1204 may store information encoding how the nodes of one or more DNNs are interconnected. Memory 1204 may store instructions used to perform one or more operations of one or more DNNs. Memory 1204 may store model definitions specifying one or more operations of the DNNs. Memory 1204 may store instructions (such as configuration files) generated by a compiler based on the model definitions.

[0167] In some embodiments, computing device 1200 may include communication device 1212 (e.g., one or more communication devices). For example, communication device 1212 may be configured to manage wired and / or wireless communications for transmitting data to and from computing device 1200. The term “wireless” and its derivatives may be used to describe circuits, apparatus, systems, methods, techniques, communication channels, etc., that can transmit data over a non-solid medium by using modulated electromagnetic radiation. The term does not imply that the associated apparatus does not contain any wiring, although in some embodiments they may not contain any wiring. Communication device 1212 may implement any of a variety of wireless standards or protocols, including but not limited to Institute of Electrical and Electronics Engineers (IEEE) standards, including Wi-Fi (IEEE 802.10 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendments), Long Term Evolution (LTE) projects along with any amendments, updates, and / or revisions (e.g., Advanced LTE project, Ultra Mobile Broadband (UMB) project (also known as “3GPP2”), etc.). IEEE 802.16 compliant Broadband Wireless Access (BWA) networks are commonly referred to as WiMAX (an acronym for Global Microwave Access Interoperability) networks, and are certification marks for products that have passed conformance and interoperability testing of the IEEE 802.16 standard. Communication device 1212 can operate according to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE networks. Communication device 1212 can operate according to Enhanced Data GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). Communication device 1212 can operate according to Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolved Data Optimized (EV-DO) and its derivatives, as well as any other wireless protocol designated as 3G, 4G, 5G, and above. In other embodiments, the communication device 1212 may operate according to other wireless protocols. The computing device 1200 may include an antenna 1222 to facilitate wireless communication and / or receiving other wireless communications (such as radio frequency transmissions). The computing device 1200 may include receiver circuitry and / or transmitter circuitry. In some embodiments, the communication device 1212 may manage wired communication, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication device 1212 may include multiple communication chips.For example, the first communication device 1212 may be dedicated to shorter-range wireless communication such as Wi-Fi or Bluetooth, and the second communication device 1212 may be dedicated to longer-range wireless communication such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first communication device 1212 may be dedicated to wireless communication, and the second communication device 1212 may be dedicated to wired communication.

[0168] The computing device 1200 may include a power supply / power circuit module 1214. The power supply / power circuit module 1214 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuit modules for coupling components of the computing device 1200 to an energy source (e.g., DC power, AC power, etc.) that is separate from the computing device 1200.

[0169] The computing device 1200 may include a display device 1206 (or a corresponding interface circuit module, as discussed above). For example, the display device 1206 may include any visual indicator, such as a head-up display, computer monitor, projector, touch screen display, liquid crystal display (LCD), light-emitting diode display, or flat panel display.

[0170] The computing device 1200 may include an audio output device 1208 (or a corresponding interface circuit module, as discussed above). For example, the audio output device 1208 may include any device that generates an auditory indicator, such as a speaker, headphones, or earphones.

[0171] The computing device 1200 may include an audio input device 1218 (or a corresponding interface circuit module, as discussed above). The audio input device 1218 may include any device that generates a signal representing sound, such as a microphone, microphone array, or digital musical instrument (e.g., a musical instrument with a Musical Instrument Digital Interface (MIDI) output).

[0172] The computing device 1200 may include a GPS device 1216 (or a corresponding interface circuit module, as discussed above). As is known in the art, the GPS device 1216 can communicate with a satellite-based system and can receive the location of the computing device 1200.

[0173] The computing device 1200 may include a sensor 1230 (or one or more sensors). As discussed above, the computing device 1200 may include a corresponding interface circuit module. The sensor 1230 can sense physical phenomena and convert them into electrical signals that can be processed, for example, by the processing device 1202. Examples of sensors 1230 may include: capacitive sensors, inductive sensors, resistive sensors, electromagnetic field sensors, light sensors, cameras, imagers, microphones, pressure sensors, temperature sensors, vibration sensors, accelerometers, gyroscopes, strain sensors, moisture sensors, humidity sensors, distance sensors, ranging sensors, time-of-flight sensors, pH sensors, particle sensors, air quality sensors, chemical sensors, gas sensors, biosensors, ultrasonic sensors, scanners, etc.

[0174] The computing device 1200 may include another output device 1210 (or a corresponding interface circuit module, as discussed above). Examples of another output device 1210 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, a haptic output device, a gas output device, a vibration output device, a lighting output device, a home automation controller, or an additional storage device.

[0175] The computing device 1200 may include another input device 1220 (or a corresponding interface circuit module, as discussed above). Examples of the other input device 1220 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a barcode reader, a quick-response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0176] The computing device 1200 can have any desired form factor, such as a handheld or mobile computer system (e.g., mobile phone, smartphone, mobile internet device, music player, tablet computer, laptop computer, netbook computer, personal digital assistant (PDA), personal computer, remote control, wearable device, helmet, glasses, footwear, electronic clothing, etc.), desktop computer system, server or other networked computing component, printer, scanner, monitor, set-top box, entertainment control unit, vehicle control unit, digital camera, digital video recorder, Internet of Things device, or wearable computer system. In some embodiments, the computing device 1200 can be any other electronic device that processes data. Selected Examples

[0177] Example 1 includes a method comprising: receiving a neural network model definition specifying a SoftMax operation to be applied to an input tensor; generating one or more first instructions executable by a processing engine based on one or more elements of the input tensor to maintain a running maximum and a denominator; generating a second instruction executable by the processing engine to update the denominator based on the logarithm of the denominator; generating a third instruction executable by the processing engine to determine an operand value based on a first sum of the denominator and the running maximum; and generating one or more fourth instructions executable by the processing engine to update at least a first element of the input tensor by raising the base to a first exponent, the first exponent being the first element of the input tensor minus the operand value; wherein the one or more first, second, third, and one or more fourth instructions cause the processing engine to perform a SoftMax operation on the input tensor.

[0178] In Example 2, the method described in Example 1 may optionally include a SoftMax operation on the input tensor, which transforms one or more elements of the input tensor into one or more values ​​in the range from 0 to 1 such that a second sum of one or more values ​​of the input tensor equals 1.

[0179] In Example 3, the method described in Example 1 or 2 may optionally include one or more first instructions to maintain the running maximum value and denominator value by determining whether a first element of the input tensor is greater than the running maximum value.

[0180] In Example 4, the method described in Example 3 may optionally include one or more first instructions to maintain the running maximum value and the denominator value by: in response to determining that a first element of the input tensor is greater than the running maximum value; determining the product of the denominator value and a second exponent raised to the power of the base, the second exponent being the running maximum value minus the first element of the input tensor; and updating the running maximum value based on the first element of the input tensor.

[0181] In Example 5, the method described in Example 4 may optionally include one or more first instructions to maintain the running maximum value and denominator value by updating the denominator value based on a third sum of the product and the third exponent of the base, where the third exponent is the first element of the input tensor minus the running maximum value.

[0182] In Example 6, the method described in any of the examples in Examples 1-5 may optionally include the logarithm of the denominator value being the natural logarithm of the denominator value.

[0183] In Example 7, the method described in any of the examples in Examples 1-6 may optionally include a base that is a natural number e.

[0184] In Example 8, the method described in any of the examples in Examples 1-6 may optionally include powers of 2.

[0185] In Example 9, the method described in Example 8 may optionally include: the neural network model definition further specifying matrix multiplication operations on the output tensor and matrix of the SoftMax operation; and the method further includes: generating one or more fifth instructions executable by the processing engine to perform displacement of a plurality of positions, the number of positions being a rounded integer value of a first exponent.

[0186] In Example 10, the method described in any of the examples 1-9 may optionally include: a SoftMax operation in the attention layer of the neural network model definition.

[0187] In Example 11, the method described in any of the examples 1-10 may optionally include: the SoftMax operation following the scaling operation in the neural network model definition.

[0188] Example 12 includes a method comprising: receiving an input tensor of a SoftMax operation of a neural network by a processing engine; maintaining a running maximum and a denominator value by the processing engine based on one or more elements of the input tensor; updating the denominator value by the processing engine based on the logarithm of the denominator value; determining an operation value by the processing engine based on a first sum of the denominator value and the running maximum; updating at least a first element of the input tensor by the processing engine by raising the base to a first exponent, the first exponent being the first element of the input tensor minus the operation value; and outputting an output tensor by the processing engine, the output tensor comprising one or more elements of the input tensor that have been transformed into one or more probabilities of a probability distribution.

[0189] In Example 13, the method described in Example 12 may optionally include having one or more values ​​of one or more elements of the output tensor in the range from 0 to 1, such that a second sum of one or more values ​​of one or more elements of the output tensor equals 1.

[0190] In Example 14, the method described in Example 12 or 13 may optionally include maintaining the running maximum value and the denominator value by determining whether the first element of the input tensor is greater than the running maximum value.

[0191] In Example 15, the method described in Example 14 may optionally include maintaining the running maximum value and the denominator value, further including: in response to determining that a first element of the input tensor is greater than the running maximum value: determining the product of the denominator value and a second exponent of the base, the second exponent being the running maximum value minus the first element of the input tensor; and updating the running maximum value based on the first element of the input tensor.

[0192] In Example 16, the method described in Example 15 may optionally include maintaining the running maximum value and the denominator value, further including updating the denominator value based on the third sum of the product and the third exponent of the base, where the third exponent is the first element of the input tensor minus the running maximum value.

[0193] In Example 17, the method is described as in any of the examples 12-16, where the logarithm of the denominator is the natural logarithm of the denominator.

[0194] In Example 18, the method is described as in any of the examples 12-17, where the base is the natural number e.

[0195] In Example 19, the method is described as in any of the examples 12-17, where the base is a power of 2.

[0196] In Example 20, the method is as described in Example 19, wherein: the neural network further includes matrix multiplication of the output tensor and the matrix; and the method further includes: performing displacement of a plurality of positions, the number of positions being a rounded integer value of a first exponent.

[0197] In Example 21, the method is described as in any of the examples 12-20, wherein the SoftMax operation is performed in the attention layer of the neural network.

[0198] In Example 22, the method is described as in any of the examples 12-21, wherein the SoftMax operation follows the scaling operation in the neural network.

[0199] Example 23: One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause one or more processors to: receive a neural network model definition specifying a SoftMax operation being applied to an input tensor; generate one or more first instructions executable by a processing engine based on one or more elements of the input tensor to maintain a running maximum value and a denominator value; generate a second instruction executable by the processing engine to update the denominator value based on the logarithm of the denominator value; generate a third instruction executable by the processing engine to determine an operand value based on a first sum of the denominator value and the running maximum value; and generate one or more fourth instructions executable by the processing engine to update at least a first element of the input tensor by raising the base to a first exponent, the first exponent being the first element of the input tensor minus the operand value; wherein the one or more first, second, third, and one or more fourth instructions cause the processing engine to perform a SoftMax operation on the input tensor.

[0200] In Example 24, one or more non-transitory computer-readable media as described in Example 23 may optionally include a SoffMax operation on the input tensor that transforms one or more elements of the input tensor into one or more values ​​in the range from 0 to 1 such that a second sum of one or more values ​​of the input tensor equals 1.

[0201] In Example 25, one or more non-transitory computer-readable media as described in Examples 23 or 24 may optionally include one or more first instructions for maintaining the running maximum value and denominator value by determining whether a first element of the input tensor is greater than the running maximum value.

[0202] In Example 26, one or more non-transitory computer-readable media as described in Example 25 may optionally include one or more first instructions to maintain the running maximum value and the denominator value by: in response to determining that a first element of the input tensor is greater than the running maximum value; determining the product of the denominator value and a second exponent of the base, the second exponent being the running maximum value minus the first element of the input tensor; and updating the running maximum value based on the first element of the input tensor.

[0203] In Example 27, one or more non-transitory computer-readable media as described in Example 26 may optionally include one or more first instructions to maintain the running maximum value and denominator value by updating the denominator value based on a third sum of the product and the third exponent of the base, the third exponent being the first element of the input tensor minus the running maximum value.

[0204] In Example 28, one or more non-transitory computer-readable media as described in any of the examples 23-27 may optionally include a logarithm of the denominator value that is the natural logarithm of the denominator value.

[0205] In Example 29, one or more non-transitory computer-readable media as described in any of the examples 23-28 may optionally include a base that is a natural number e.

[0206] In Example 30, one or more non-transitory computer-readable media as described in any of the examples 23-28 may optionally include powers of 2.

[0207] In Example 31, one or more non-transitory computer-readable media as described in Example 30 may optionally include: a neural network model definition further specifying matrix multiplication operations on the output tensor and matrix of the SoffMax operation; and instructions further causing one or more processors to: generate one or more fifth instructions executable by a processing engine to perform displacement of a plurality of positions, the number of positions being a rounded integer value of a first exponent.

[0208] In Example 32, one or more non-transitory computer-readable media as described in any of the examples 23-31 may optionally include: SoftMax operations in the attention layer of the neural network model definition.

[0209] In Example 33, one or more non-transitory computer-readable media as described in any of the examples 23-32 may optionally include: the SoffMax operation following the scaling operation in the neural network model definition.

[0210] Example 34 includes one or more non-transitory computer-readable media storing instructions that, when executed by one or more processing engines, cause one or more processing engines to: receive an input tensor of a SoffMax operation of a neural network; maintain a running maximum value and a denominator value based on one or more elements of the input tensor; update the denominator value based on the logarithm of the denominator value; determine an operand value based on a first sum of the denominator value and the running maximum value; update at least a first element of the input tensor by raising the base to a first exponent, the first exponent being the first element of the input tensor minus the operand value; and output an output tensor comprising one or more elements of the input tensor that have been transformed into one or more probabilities of a probability distribution.

[0211] In Example 35, one or more non-transitory computer-readable media as described in Example 34 may optionally include one or more values ​​of one or more elements of an output tensor in the range from 0 to 1, such that a second sum of one or more values ​​of one or more elements of the output tensor is equal to 1.

[0212] In Example 36, one or more non-transitory computer-readable media as described in Examples 34 or 35 may optionally include maintaining the running maximum value and the denominator value by: determining whether a first element of the input tensor is greater than the running maximum value.

[0213] In Example 37, one or more non-transitory computer-readable media as described in Example 36 may optionally include maintaining the running maximum value and the denominator value, further including: in response to determining that a first element of the input tensor is greater than the running maximum value; determining the product of the denominator value and a second exponent power of the base, the second exponent being the running maximum value minus the first element of the input tensor; and updating the running maximum value based on the first element of the input tensor.

[0214] In Example 38, one or more non-transitory computer-readable media as described in Example 37 may optionally include maintaining the running maximum value and the denominator value, further including updating the denominator value based on a third sum of the product and the base raised to a third exponent, the third exponent being the first element of the input tensor minus the running maximum value.

[0215] In Example 39, one or more non-transitory computer-readable media as described in any of the examples 34-38 may optionally include a logarithm of the denominator value that is the natural logarithm of the denominator value.

[0216] In Example 40, one or more non-transitory computer-readable media as described in any of the examples 34-39 may optionally include a base that is a natural number e.

[0217] In Example 41, one or more non-transitory computer-readable media as described in any of the examples 34-39 may optionally include powers of 2.

[0218] In Example 42, one or more non-transitory computer-readable media as described in Example 41 may optionally include: the neural network further includes matrix multiplication operations on the output tensor and matrix; and the instructions further cause one or more processing engines to: perform displacement of a plurality of positions, the number of positions being a rounded integer value of a first exponent.

[0219] In Example 43, one or more non-transitory computer-readable media as described in any of the examples 34-42 may optionally include: SoftMax operations in the attention layer of a neural network.

[0220] In Example 44, one or more non-transitory computer-readable media as described in any of the examples 34-43 may optionally include the SoftMax operation following a scaling operation in a neural network.

[0221] Example 45 includes a system comprising: one or more processors; and one or more non-transitory computer-readable memories for storing instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors to: receive a neural network model definition specifying a SoftMax operation being applied to an input tensor; generate one or more first instructions executable by a processing engine based on one or more elements of the input tensor to maintain a running maximum and a denominator value; generate a second instruction executable by the processing engine to update the denominator value based on the logarithm of the denominator value; generate a third instruction executable by the processing engine to determine an operand value based on a first sum of the denominator value and the running maximum; and generate one or more fourth instructions executable by the processing engine to update at least a first element of the input tensor by raising the base to a first exponent, the first exponent being the first element of the input tensor minus the operand value; wherein the one or more first, second, third, and one or more fourth instructions cause the processing engine to perform a SoftMax operation on the input tensor.

[0222] In Example 46, the system as described in Example 45 may optionally include a SoftMax operation on the input tensor that transforms one or more elements of the input tensor into one or more values ​​in the range from 0 to 1 such that a second sum of one or more values ​​of the input tensor equals 1.

[0223] In Example 47, the system described in Examples 45 or 46 may optionally include one or more first instructions to maintain the running maximum value and denominator value by determining whether a first element of the input tensor is greater than the running maximum value.

[0224] In Example 48, the system as described in Example 47 may optionally include one or more first instructions to maintain the running maximum value and the denominator value by: in response to determining that a first element of the input tensor is greater than the running maximum value; determining the product of the denominator value and a second exponent of the base, the second exponent being the running maximum value minus the first element of the input tensor; and updating the running maximum value based on the first element of the input tensor.

[0225] In Example 49, the system as described in Example 48 may optionally include one or more first instructions to maintain the running maximum value and denominator value by updating the denominator value based on a third sum of the product and the third exponent of the base, where the third exponent is the first element of the input tensor minus the running maximum value.

[0226] In Example 50, the system described in any of the examples 45-49 may optionally include a logarithm of the denominator value that is the natural logarithm of the denominator value.

[0227] In Example 51, the system described in any of the examples 45-50 may optionally include a base that is a natural number e.

[0228] In Example 52, the system described in any of the examples 45-50 may optionally include powers of 2.

[0229] In Example 53, the system as described in Example 52 may optionally include: a neural network model definition further specifying matrix multiplication operations on the output tensor and matrix of the SoffMax operation; and instructions further prompting one or more processors to: generate one or more fifth instructions executable by the processing engine to perform displacement of a plurality of positions, the number of positions being a rounded integer value of a first exponent.

[0230] In Example 54, the system described in any of the examples 45-53 may optionally include: a SoftMax operation in the attention layer of the neural network model definition.

[0231] In Example 55, the system described in any of the examples 45-54 may optionally include: the SoftMax operation following the scaling operation in the neural network model definition.

[0232] Example 56 includes a system comprising: one or more processing engines; and one or more non-transitory computer-readable storages for storing instructions, wherein the instructions, when executed by the one or more processing engines, cause the one or more processing engines to: receive an input tensor of a SoftMax operation of a neural network; maintain a running maximum value and a denominator value based on one or more elements of the input tensor; update the denominator value based on the logarithm of the denominator value; determine an operand value based on a first sum of the denominator value and the running maximum value; update at least a first element of the input tensor by raising the base to a first exponent, the first exponent being the first element of the input tensor minus the operand value; and output an output tensor comprising one or more elements of the input tensor that have been transformed into one or more probabilities of a probability distribution.

[0233] In Example 57, the system as described in Example 56 may optionally include one or more values ​​of one or more elements of the output tensor in the range from 0 to 1, such that a second sum of one or more values ​​of one or more elements of the output tensor equals 1.

[0234] In Example 58, the system described in Example 56 or 57 may optionally include maintaining the running maximum value and the denominator value by: determining whether a first element of the input tensor is greater than the running maximum value.

[0235] In Example 59, the system described in Example 58 may optionally include maintaining the running maximum value and the denominator value, further including: in response to determining that a first element of the input tensor is greater than the running maximum value; determining the product of the denominator value and a second exponent of the base, the second exponent being the running maximum value minus the first element of the input tensor; and updating the running maximum value based on the first element of the input tensor.

[0236] In Example 60, the system described in Example 59 may optionally include maintaining the running maximum value and the denominator value, further including updating the denominator value based on a third sum of the product and the base raised to a third exponent, where the third exponent is the first element of the input tensor minus the running maximum value.

[0237] In Example 61, the system described in any of the examples 56-60 may optionally include a denominator logarithm that is the natural logarithm of the denominator value.

[0238] In Example 62, the system described in any of the examples 56-61 may optionally include a base that is a natural number e.

[0239] In Example 63, the system described in any of the examples 56-61 may optionally include powers of 2.

[0240] In Example 64, the system as described in Example 63 may optionally include: the neural network further includes matrix multiplication operations on the output tensor and matrix; and instructions further cause one or more processing engines to: perform displacement of multiple positions, the number of positions being a rounded integer value of a first exponent.

[0241] In Example 65, the system described in any of the examples 56-64 may optionally include: a SoftMax operation in an attention layer of a neural network.

[0242] In Example 66, the system described in any of the examples 56-65 may optionally include: the SoftMax operation following the scaling operation in the neural network.

[0243] Example A is a device that includes components for performing any of the methods described according to Examples 1-22.

[0244] Example B includes a DNN module as described and illustrated in this document.

[0245] Example C includes a DNN module for performing any of the methods described in Examples 1-11.

[0246] Example D includes a compiler as described and illustrated in this document.

[0247] Example E includes a compiler for executing any of the methods described according to Examples 1-11.

[0248] Example F includes a DNN accelerator as described and illustrated in this paper.

[0249] Example G includes a DNN accelerator for performing any of the methods described in Examples 12-22.

[0250] Example H includes one or more data processing units for performing any of the methods described according to Examples 12-22.

[0251] Example I includes one or more processing engines used to execute any of the methods described according to Examples 12-22.

[0252] Example J includes a DNN system that includes DNN modules and a DNN accelerator as described and illustrated herein.

[0253] Example K includes a DNN system comprising a DNN module according to Example C and a DNN accelerator according to Example G. Variations and other annotations

[0254] Despite Figure 8-11 As shown and referenced Figure 8-11 The operations of the example methods described are stated to occur once each and in a specific order; however, it will be appreciated that the operations can be performed in any suitable order and can be repeated as expected. Furthermore, one or more operations can be executed in parallel. Additionally, they can be combined. Figure 8-11 The operation described in the text or Figure 8-11 The operations described may include more or less detail than those described.

[0255] The various implementations described in this article can refer to artificial intelligence, machine learning, and deep learning. Deep learning can be a subset of machine learning. Machine learning can be a subset of artificial intelligence. Where deep learning models are mentioned, machine learning models may be used alternatively if suitable for a particular application. Where deep learning models are mentioned, digital signal processing systems may be used alternatively if suitable for a particular application.

[0256] The above description of the disclosed implementation (including those described in the abstract) is not exhaustive or intended to limit the disclosure to its precise form. While specific implementations and examples of the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as will be recognized by those skilled in the art. These modifications can be made to the disclosure based on the detailed description above.

[0257] For illustrative purposes, specific figures, materials, and configurations have been set forth to provide a thorough understanding of the illustrative implementation. However, it will be apparent to those skilled in the art that this disclosure may be practiced without specific details and / or may be practiced using only some of the aspects described. In other instances, well-known features have been omitted or simplified so as not to obscure the illustrative implementation.

[0258] Furthermore, reference is made to the accompanying drawings, which form part of the present invention, in which illustrative embodiments that may be implemented are shown. It should be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of this disclosure. Therefore, the following detailed description should not be construed in a limiting sense.

[0259] Various operations can be described sequentially as multiple discrete actions or operations in a manner most conducive to understanding the disclosed subject matter. However, the order of description should not be construed as implying that these operations must be sequentially related. In particular, these operations may not be performed in the order presented. The described operations may be performed in an order different from the described embodiments. In additional embodiments, various additional operations may be performed or the described operations may be omitted.

[0260] For the purposes of this disclosure, the phrase "A or B" or the phrase "A and / or B" refers to (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" refers to (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). When used with reference to a measurement range, the term "between" includes the endpoints of the measurement range.

[0261] The description uses the phrases "in embodiments" or "in multiple embodiments," which may each refer to one or more embodiments of the same or different embodiments. The terms "comprising," "including," "having," etc., used with respect to embodiments of this disclosure are synonymous. The disclosure may use perspective-based descriptions (such as "above," "below," "top," "bottom," and "side") to interpret various features of the figures; however, these terms are merely for ease of discussion and do not imply desired or required orientation. The figures are not necessarily drawn to scale. Unless otherwise specified, the use of ordinal adjectives such as "first," "second," and "third" to describe common objects merely indicates that different instances of similar objects are being mentioned and is not intended to imply that the objects so described must be in a given order, either temporally, spatially, in arrangement, or in any other way.

[0262] In the following detailed description, terms commonly used by those skilled in the art will be used to describe various aspects of the illustrative implementation in order to convey the substance of the work to others skilled in the art.

[0263] The terms “substantially,” “near,” “approximately,” “in the vicinity of,” and “about” generally refer to a target value as described herein or as known in the art within + / - 20%. Similarly, terms indicating the orientation of various elements (e.g., “coplanar,” “perpendicular,” “orthogonal,” “parallel,” or any other angle between elements) generally refer to a target value as described herein or as known in the art within + / - 5-20%.

[0264] Furthermore, the terms “comprise,” “comprising,” “include,” “including,” “have,” “having,” or any other variations thereof are intended to cover non-exclusive inclusion. For example, a method, process, or apparatus that includes a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent to such a method, process, or apparatus. Moreover, the term “or” refers to inclusive “or,” not exclusive “or.”

[0265] The systems, methods, and apparatuses disclosed herein each have several innovative aspects, none of which individually is solely responsible for all the desired properties disclosed herein. Details of one or more implementations of the subject matter described herein are set forth in the description and accompanying drawings.

Claims

1. A method comprising: receiving a neural network model definition that specifies a SoftMax operation being applied to an input tensor; generating, based on one or more elements of the input tensor, one or more first instructions executable by a processing engine to maintain a running maximum value and a denominator value; generating a second instruction executable by the processing engine to update the denominator value based on a logarithm of the denominator value; generating a third instruction executable by the processing engine to determine an operand value based on a first sum of the denominator value and the running maximum value; and generating one or more fourth instructions executable by the processing engine to update at least a first element of the input tensor by a first exponential power of a base number, the first exponent being the first element of the input tensor minus the operand value; wherein the one or more first instructions, the second instruction, the third instruction, and the one or more fourth instructions cause the processing engine to perform the SoftMax operation on the input tensor. The SoftMax operation on the output tensor converts one or more elements of the input tensor to one or more values in a range from 0 to 1 such that a second sum of the one or more values of the input tensor equals 1.

2. The method of claim 1, wherein, The one or more first instructions maintain the running maximum value and the denominator value by:

3. The method of claim 1 or 2, wherein, determining whether the first element of the input tensor is greater than the running maximum value. The one or more first instructions further maintain the running maximum value and the denominator value by:

4. The method of claim 3, wherein, in response to determining that the first element of the input tensor is greater than the running maximum value: determining a product of the denominator value and a second exponential power of the base number, the second exponent being the running maximum value minus the first element of the input tensor; and updating the running maximum value based on the first element of the input tensor. The one or more first instructions further maintain the running maximum value and the denominator value by:

5. The method of claim 4, wherein, updating the denominator value based on a third sum of the product and a third exponential power of the base number, the third exponent being the first element of the input tensor minus the running maximum value. The logarithm of the denominator value is a natural logarithm of the denominator value.

6. The method of claim 1 or 2, wherein, The base number is a natural number e.

7. The method of claim 1 or 2, wherein, The base number is a power of 2.

8. The method of claim 1 or 2, wherein, 9. The method of claim 8, wherein: the neural network model definition further specifies a matrix multiplication operation on an output tensor of the SoftMax operation and a matrix; and the method further comprises: generating one or more fifth instructions executable by the processing engine to perform a bit shift by a number of positions, the number of positions being a rounded integer value of the first exponent.

10. The method of claim 1 or 2, wherein: the SoftMax operation is in an attention layer in the neural network model definition.

11. The method of claim 1 or 2, wherein: the SoftMax operation is after a scaling operation in the neural network model definition.

12. A method: ​ ​ receiving, by a processing engine, an input tensor for a SoftMax operation of a neural network; maintaining, by the processing engine, a running maximum and a denominator value based on one or more elements of the input tensor; updating, by the processing engine, the denominator value based on a logarithm of the denominator value; determining, by the processing engine, an operand value based on a first sum of the denominator value and the running maximum; updating, by the processing engine, at least a first element of the input tensor by a first exponentiation of a base number to a first exponent of the first element of the input tensor minus the operand value; and outputting, by the processing engine, the input tensor as an output tensor that includes the one or more elements of the input tensor converted to one or more probabilities of a probability distribution. one or more values of the one or more elements of the output tensor are in a range from 0 to 1 such that a second sum of the one or more values of the one or more elements of the output tensor is equal to 1.

13. The method of claim 12, wherein, maintaining the running maximum and the denominator value includes:

14. The method of claim 12 or 13, wherein, determining whether the first element of the input tensor is greater than the running maximum. maintaining the running maximum and the denominator value further includes:

15. The method of claim 14, wherein, in response to determining that the first element of the input tensor is greater than the running maximum: determining a product of the denominator value and a second exponentiation of the base number to a second exponent of the running maximum minus the first element of the input tensor; and updating the running maximum based on the first element of the input tensor. maintaining the running maximum and the denominator value further includes:

16. The method of claim 15, wherein, updating the denominator value based on a third sum of the product and a third exponentiation of the base number to a third exponent of the first element of the input tensor minus the running maximum. the logarithm of the denominator value is a natural logarithm of the denominator value.

17. The method of claim 12 or 13, wherein, the base number is a natural number e.

18. The method of claim 12 or 13, wherein, the base number is a power of 2.

19. The method of claim 12 or 13, wherein, 20. The method of claim 19, wherein: the neural network further includes a matrix multiplication operation of the output tensor and a matrix; and the method further includes: performing a shift by a number of positions, the number of positions being a rounded integer value of the first exponent.

21. The method of claim 12 or 13, wherein: the SoftMax operation is in an attention layer in the neural network.

22. The method of claim 12 or 13, wherein: the SoftMax operation is after a scaling operation in the neural network.

23. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform any of the methods of any of claims 1-22.

24. An apparatus comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform any of the methods of any of claims 1-22. ​ ​ 25. A computer program product comprising instructions which, when executed by a processor, cause the processor to carry out the method of any one of claims 1 to 22.

Images