Performing fourier transform operations with a deep neural network accelerator

Abstract

Fourier transform operations can be converted to matrix multiplications that can resemble matrix multiplications in convolutions and can be performed by a deep neural network (DNN) accelerator. The DNN accelerator can receive an input tensor representing an input signal of a Fourier transform operation. The input tensor can include activations arranged in one or more rows and one or more columns. The DNN accelerator can receive a weight tensor determined based on one or more rotation factors of the Fourier transform operation. The DNN accelerator can perform two sequences of multiply-accumulate (MAC) operations. The MAC operations in a first sequence can be performed on the weight tensor and rows in the input tensor. The MAC operations in a second sequence can be performed on the weight tensor and columns in the input tensor. Outputs of the MAC operations can represent a Fourier transform of the input signal.

Description

Technical Field

[0001] This disclosure generally relates to neural networks (also known as “deep neural networks” or “DNNs”), and more specifically, to performing Fourier transform operations using a DNN accelerator. Background Technology

[0002] DNNs are widely used in various artificial intelligence (AI) applications, ranging from computer vision to speech recognition and natural language processing, due to their ability to achieve high accuracy. Many AI applications require high-quality data that can be obtained by processing noisy input data. A widely used approach is to use Fourier transform to convert complex and noisy raw data into a more suitable format. Attached Figure Description

[0003] The embodiments will be readily understood from the following detailed description taken in conjunction with the accompanying drawings. For ease of description, similar reference numerals denote similar structural elements. In the accompanying figures, embodiments are shown by way of example rather than limitation.

[0004] Figure 1 Example DNNs according to various embodiments are shown.

[0005] Figure 2 Example convolutions according to various embodiments are shown.

[0006] Figure 3 This is a block diagram of a DNN system according to various embodiments.

[0007] Figure 4 This is a block diagram of a DNN module according to various embodiments.

[0008] Figure 5 Example transformation matrices for Fourier transform operations according to various embodiments are shown.

[0009] Figure 6 Sparse convolution units according to various embodiments are shown.

[0010] Figure 7 Example sparse cell arrays according to various embodiments are shown.

[0011] Figure 8 The diagram illustrates how the Discrete Fourier Transform (DFT) operation is mapped to a sparse cell array according to various embodiments.

[0012] Figure 9 The diagram illustrates how real-valued DFT (RDFT) operations are mapped to sparse cell arrays according to various embodiments.

[0013] Figure 10 The diagram illustrates the mapping of complex DFTs to sparse cell arrays according to various embodiments.

[0014] Figure 11 An example sliding window mode of short-time Fourier transform (STFT) operation according to various embodiments is shown.

[0015] Figure 12 Another example of a sliding window mode for STFT operation according to various embodiments is shown.

[0016] Figure 13 The image illustrates the mapping of frames of STFT operation to a sparse cell array according to various embodiments.

[0017] Figure 14 This is a flowchart illustrating a method for performing a Fourier transform operation according to various embodiments.

[0018] Figure 15 This is a block diagram of an example computing device according to various embodiments. Detailed Implementation

[0019] Overview The past decade has witnessed the rapid development of AI-based data processing, especially DNN-based applications. DNNs are widely used in computer vision, speech recognition, image and video processing primarily due to their ability to achieve accuracy exceeding human levels. The significant improvement in DNN model size and accuracy, combined with the rapid growth in the computing power of execution platforms, has led to the adoption of DNN applications even in resource-constrained mobile and edge devices with limited energy availability.

[0020] A DNN layer can include one or more deep learning operations (also known as "neural network operations"), such as convolution, pooling, element-wise operations, linear operations, non-linear operations, and so on. Deep learning operations in a DNN can be performed on one or more intrinsic parameters (e.g., weights) of the DNN determined during the training phase and during one or more activations. Activations can be data points (also known as "data elements" or "elements"). Activations or weights of a DNN layer can be elements of a tensor of the DNN layer. A tensor is a data structure with multiple elements in one or more dimensions. Example tensors include vectors that are one-dimensional tensors and matrices that are two-dimensional tensors. There can also be three-dimensional tensors, and even tensors of higher dimensions. A DNN layer can have an input tensor (also known as an "input feature map (IFM)") that includes one or more input activations (also known as "input elements") and a weight tensor that includes one or more weights. Weights are elements in the weight tensor. The weight tensor of a convolution can be a kernel, a filter, or a set of filters. The output data of a DNN layer can be an output tensor (also known as an "output feature map (OFM)") that includes one or more output activations (also known as "output elements").

[0021] Many AI applications frequently require Fourier transforms of raw data for image and audio modeling and analysis. For example, noisy and complex raw data is often transformed into the frequency domain, representing the amplitude and phase of different frequency components, for further analysis. This transformation facilitates processing techniques including filtering, normalization, segmentation, feature extraction, and encoding. These techniques help remove noise, extract relevant features, and improve the accuracy of analysis and modeling. Image and audio processing can be crucial for ensuring reliable results and improving overall quality and modeling.

[0022] Many audio and image-based DNN models require the application of the Fourier Transform (DFT). However, these Fourier transform operations can become bottlenecks in end-to-end use cases. The main reason is that these operations often do not fully utilize computational resources. Many solutions implement the kernel in a high-level programming language and run the kernel on a programmable engine, which may be more general-purpose but inefficient. For example, some solutions rely on general-purpose computing systems to perform these steps. Examples of such general-purpose computing systems include central processing units (CPUs), digital signal processors (DSPs), streaming hybrid architecture vector engines, other very long instruction word (VLIW) processors, and so on.

[0023] Many currently available solutions for mapping Fourier transforms for DNNs (which can be part of the network or pre / post-processing operations) involve mapping these layers to a limited number of programmable computations (e.g., DSPs / vector processors), which are also part of the overall DNN accelerator subsystem. However, these programmable computational elements may have limited computational capacity, leading to performance bottlenecks even when normalized to power and area. They can degrade performance because they operate at lower frequencies and lower bandwidths. Furthermore, they can consume significant power. Additionally, general-purpose processors inherently do not support the trigonometric cosine / sine functions required for Fourier transform computations. They typically require Taylor series decomposition to enable these functions, which can lead to power consumption or performance degradation. Another drawback of these solutions is that efficiency depends heavily on the kernel implementation, which varies considerably among different engineers. Poor implementations can lead to deep code loops and further inefficient inference. An alternative approach is to use dedicated hardware to map these layers. However, this is a large-area adder, unsuitable for edge or client devices that typically require a smaller footprint or area.

[0024] Embodiments of this disclosure can improve at least some of the aforementioned challenges and problems by converting Fourier transform operations into matrix multiplications that can be performed by a DNN accelerator. Fourier transform operations that can be performed by a DNN accelerator include DFT, inverse DFT (IDFT), RDFT, inverse RDFT (IRDFT), STFT, etc.

[0025] In various embodiments, the Fourier transform operation can be transformed into a two-dimensional (2D) matrix multiplication between the input signal and the transform matrix. The 2D matrix multiplication can be performed by a DNN accelerator capable of performing matrix multiplication in convolution. The DNN accelerator may include components that can optimize computational efficiency during convolution execution. The input signal of the Fourier transform operation can be represented by a 2D input matrix having data elements arranged in rows and columns. The data elements in the input matrix can be processed in the same or similar manner as the activation of the input tensor for convolution. The transform matrix can be generated from the twist factor of the Fourier transform operation. The data elements in the transform matrix can be processed in the same or similar manner as the convolution weights. The twist factor of the FFT algorithm can be any trigonometric constant coefficient multiplied with the data during the algorithmic process. The twist factor can be any data-independent multiplication constant used in the FFT process.

[0026] Activation vectors can be generated from the input matrix, for example, by a DNN module associated with a DNN accelerator. In some embodiments, the activation vectors can be rows or columns in the input matrix. In other embodiments (e.g., where the Fourier transform is an STFT), the activation vectors can be frames extracted from the input sequence by sliding a window over the input sequence. The DNN module can also generate a transform matrix and divide the transform matrix into weight vectors. Activation vectors can be loaded into a register file designated for storing activations of convolutions in the DNN accelerator, and further loaded into multiply-accumulate (MAC) units associated with the register file. Weight vectors can be loaded into a register file designated for storing convolution weights, and further loaded into MAC units associated with the register file. Examples of DNN accelerators may include MAC units arranged in rows and columns. Activation vectors can be loaded into a row of MAC units and processed therewith. Weight vectors can be loaded into a column of MAC units and processed therewith. DNN modules and DNN accelerators can also facilitate Fourier transform operations with complex input signals or complex rotation factors.

[0027] The Fourier transform operation can be converted into two multiplication-accumulation (MAC) operation sequences. Each MAC operation in the first sequence can be performed on corresponding rows in the weight matrix and the input matrix. Each MAC operation in the second sequence can be performed on corresponding columns in the weight matrix and the input matrix. The input matrix can be transposed after the first sequence, such that each column in the input matrix becomes a row in the transposed input matrix. The second sequence can be performed by repeating the first sequence on the transposed input matrix. The output of the two MAC operation sequences can represent the Fourier transform of the input signal. In some embodiments, the total number of output elements can be less than the total number of elements in the Fourier transform of the input signal. For example, the input signal can have N elements, and the Fourier transform of the input signal can also have N elements. Since the Fourier transform of the input signal is symmetric, the DNN accelerator can compute and store N / 2+1 elements to represent the Fourier transform of the input signal. This can further reduce the power, time, and memory bandwidth required to perform the Fourier transform operation and improve the efficiency of the DNN accelerator.

[0028] This disclosure provides a method that allows DNN accelerators to perform Fourier transform operations, including complex Fourier transforms, by converting Fourier transform operations into matrix multiplication and mapping the input signal and rotation factor matrix as activations and weights to MAC units. This method offers performance and efficiency advantages. Compared to currently available methods, the method in this disclosure requires significantly less time, power, and memory bandwidth to perform Fourier transforms in AI applications.

[0029] For illustrative purposes, specific figures, materials, and configurations have been set forth to provide a thorough understanding of the illustrative implementation. However, it will be apparent to those skilled in the art that this disclosure may be practiced without specific details, or / and may be practiced only in some of the described aspects. In other instances, well-known features have been omitted or simplified so as not to obscure the illustrative implementation.

[0030] Furthermore, reference is made to the accompanying drawings, which form part of this disclosure, in which illustrative embodiments that may be implemented are shown. It should be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of this disclosure. Therefore, the following detailed description should not be construed in a limiting sense.

[0031] Various operations can be described sequentially as a plurality of discrete actions or operations in a manner most conducive to understanding the claimed subject matter. However, the order of description should not be construed as implying that these operations are necessarily sequentially related. In particular, these operations may not be performed in the order presented. The described operations may be performed in a different order than the described embodiments. In other embodiments, various additional operations may be performed, or the described operations may be omitted.

[0032] For the purposes of this disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). The term "between" when used to refer to a measurement range includes the endpoints of the measurement range.

[0033] The description uses the phrases “in one embodiment” or “in an embodiment,” each of which can refer to one or more of the same or different embodiments. The terms “comprising,” “including,” “having,” etc., are synonymous when used with respect to embodiments of this disclosure. This disclosure may use perspective-based descriptions, such as “upper,” “lower,” “top,” “bottom,” and “side,” to interpret various features of the figures; however, these terms are merely for ease of discussion and do not imply an expected or required orientation. The figures are not necessarily drawn to scale. Unless otherwise stated, the use of ordering adjectives such as “first,” “second,” and “third,” etc., describing common objects merely indicates that different instances of similar objects are being referenced and is not intended to mean that the objects thus described must be ranked in time, space, or any other way in a given order.

[0034] In the following detailed description, various aspects of the illustrative implementation will be described using terms commonly used by those skilled in the art to communicate the substance of their work to others skilled in the art.

[0035] The terms “substantially,” “close to,” “approximately,” “near,” and “about” generally refer to within + / - 20% of the target value, or as described herein or as known in the art. Similarly, terms indicating the orientation of various elements, such as “coplanar,” “perpendicular,” “orthogonal,” “parallel,” or any other angle between elements, generally refer to within + / - 5-20% of the target value, as described herein or as known in the art.

[0036] Furthermore, the terms “comprise,” “include,” “have,” or any other variations thereof are intended to cover non-exclusive inclusion. For example, a method, process, apparatus, or DNN accelerator that includes a sequence of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to the method, process, apparatus, or DNN accelerator. Additionally, the term “or” refers to an inclusive “or,” not an exclusive “or.”

[0037] The systems, methods, and apparatuses disclosed herein each possess several innovative aspects, none of which alone is solely responsible for all the desired properties disclosed herein. Details of one or more implementations of the subjects described in this specification are set forth in the following description and figures.

[0038] Example DNN Figure 1 An example DNN 100 according to various embodiments is shown. The DNN 100 may be an example of a teacher network or an example of a student network. For illustrative purposes, Figure 1 DNN 100 in this example is a CNN. In other embodiments, DNN 100 can be other types of DNNs. As an example, DNN 100 is trained to receive an image and output a classification of objects in the image. Figure 1 In one embodiment, DNN 100 receives an input image 105 including objects 115, 125, and 135. DNN 100 includes a series of layers comprising: multiple convolutional layers 110 (collectively referred to as "convolutional layer 110"), multiple pooling layers 120 (collectively referred to as "pooling layer 120"), and multiple fully connected layers 130 (collectively referred to as "fully connected layer 130"). In other embodiments, DNN 100 may include fewer, more, or different layers. For example, DNN 100 may include one or more DFT layers or one or more inverse DFT (IDFT) layers. Furthermore, DNN 100 can be trained to perform tasks other than image classification. In the inference of DNN 100, the layers of DNN 100 perform tensor computations including a variety of tensor operations, such as convolution (e.g., multiplication-accumulation (MAC) operations), pooling operations, element-wise operations (e.g., element-wise addition, element-wise multiplication, etc.), other types of tensor operations, or some combination thereof.

[0039] Convolutional layer 110 summarizes the presence of features in the input image 105. Convolutional layer 110 acts as a feature extractor. The first layer of DNN 100 is convolutional layer 110. In the example, convolutional layer 110 performs convolution on the input tensor 140 (also known as IFM 140) and filter 150. Figure 1As shown, the IFM 140 is represented by a 7×7×3 three-dimensional (3D) matrix. The IFM 140 includes three input channels, each represented by a 7×7 two-dimensional (2D) matrix. The 7×7 2D matrix includes seven input elements (also called input points) in each row and seven input elements in each column. The filter 150 is represented by a 3×3×3 3D matrix. The filter 150 includes three kernels, each corresponding to a different input channel of the IFM 140. A kernel is a 2D matrix of weights, where the weights are arranged by column and row. A kernel can be smaller than the IFM. Figure 1 In this embodiment, each kernel is represented by a 3×3 2D matrix. The 3×3 kernel comprises 3 weights in each row and 3 weights in each column. The weights can be initialized and updated using backpropagation via gradient descent. The magnitude of the weights can indicate the importance of filter 150 in extracting features from IFM 140.

[0040] Convolution involves a MAC operation with the input elements in IFM 140 and the weights in filter 150. The convolution can be a standard convolution 163 or a depth-direction convolution 183. In a standard convolution 163, the entire filter 150 slides through IFM 140. All input channels are combined to produce an output tensor 160 (also called OFM 160). OFM 160 is represented by a 5×5 2D matrix. The 5×5 2D matrix includes 5 output elements (also called output points) in each row and 5 output elements in each column. For illustrative purposes, in Figure 1 In one embodiment, the standard convolution includes a filter. In an embodiment with multiple filters, the standard convolution can produce multiple output channels in the OFM160.

[0041] The multiplication applied between a kernel-sized patch of IFM 140 and a kernel can be a dot product. A dot product is an element-wise multiplication between a kernel-sized patch of IFM 140 and its corresponding kernel, then summed, always resulting in a single value. Because it yields a single value, this operation is often called a "scalar product." Using a kernel smaller than IFM 140 is intentional because it allows the same kernel (set of weights) to be multiplied multiple times at different points on IFM 140. Specifically, the kernels are systematically applied to each overlapping portion or kernel-sized patch of IFM 140 from left to right and top to bottom. The result of multiplying a kernel by IFM 140 once is a single value. Since the kernel is applied multiple times to IFM 140, the result of the multiplication is a 2D matrix of output elements. Thus, the 2D output matrix from a standard convolution 163 (i.e., OFM 160) is called OFM.

[0042] In depthwise convolution 183, the input channels are not combined. Instead, the MAC operation is performed on separate input channels and separate kernels, producing an output channel. For example... Figure 1 As shown, depth-direction convolution 183 produces a depth-direction output tensor 180. The depth-direction output tensor 180 is represented by a 5×5×3 3D matrix. The depth-direction output tensor 180 includes three output channels, each represented by a 5×5 2D matrix. The 5×5 2D matrix includes five output elements in each row and five output elements in each column. Each output channel is the result of a MAC operation between the input channel of IFM 140 and the kernel of filter 150. For example, the first output channel (patterned with dots) is the result of a MAC operation between the first input channel (patterned with dots) and the first kernel (patterned with dots), the second output channel (patterned with horizontal bars) is the result of a MAC operation between the second input channel (patterned with horizontal bars) and the second kernel (patterned with horizontal bars), and the third output channel (patterned with diagonal bars) is the result of a MAC operation between the third input channel (patterned with diagonal bars) and the third kernel (patterned with diagonal bars). In this depthwise convolution, the number of input channels equals the number of output channels, and each output channel corresponds to a different input channel. The input and output channels are collectively referred to as depthwise channels. After the depthwise convolution, pointwise convolution 193 is then performed on the depthwise output tensor 180 and the 1×1×3 tensor 190 to produce OFM 160.

[0043] OFM 160 is then passed to the next layer in the sequence. In some embodiments, OFM 160 is passed via an activation function. An example activation function is Rectified Linear Unit (ReLU). ReLU is a computation that returns the value provided as input directly, or zero if the input is zero or less. Convolutional layer 110 can receive several images as input and compute a convolution of each image with each kernel. This process can be repeated several times. For example, OFM 160 is passed to subsequent convolutional layers 110 (i.e., convolutional layer 110 after the one that generates OFM 160 in the sequence). Subsequent convolutional layers 110 perform convolution on OFM 160 with new kernels and generate new feature maps. The new feature maps can also be normalized and resized. The new feature maps can be kernelized again by further subsequent convolutional layers 110, and so on.

[0044] In some embodiments, the convolutional layer 110 has four hyperparameters: the number of kernels, the kernel size F (e.g., kernel dimensions F×F×D pixels), the S-steps taken to drag the window corresponding to the kernel across the image (e.g., one step means moving the window one pixel at a time), and zero padding P (e.g., adding a black outline of P pixels thickness to the input image of the convolutional layer 110). The convolutional layer 110 can perform various types of convolutions, such as 2D convolution, dilation or contraction convolution, spatially separable convolution, depthwise separable convolution, transposed convolution, and so on. The DNN 100 includes 16 convolutional layers 110. In other embodiments, the DNN 100 may include a different number of convolutional layers.

[0045] Pooling layer 120 downsamples the feature map generated by the convolutional layer, for example, by summarizing the presence of features in blocks of the feature map. Pooling layer 120 is located between two convolutional layers 110: the previous convolutional layer 110 (the convolutional layer 110 before pooling layer 120 in the layer sequence) and the subsequent convolutional layer 110 (the convolutional layer 110 after pooling layer 120 in the layer sequence). In some embodiments, pooling layer 120 is added after convolutional layer 110, for example, after an activation function (e.g., ReLU, etc.) has been applied to OFM 160.

[0046] Pooling layer 120 receives feature maps generated by previous convolutional layer 110 and applies pooling operations to them. Pooling operations reduce the size of the feature maps while preserving their important characteristics. Therefore, pooling operations improve the efficiency of the DNN and avoid overlearning. Pooling layer 120 can perform pooling operations using average pooling (calculating the average value for each block on the feature map), max pooling (calculating the maximum value for each block on the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature map. In various embodiments, the pooling operation is applied with a stride of two pixels at 2×2 pixels, such that the pooling operation reduces the size of the feature map by half, for example, reducing the number of pixels or values ​​in the feature map to one-quarter of that size. In the example, pooling layer 120 applied to a 6×6 feature map yields a 3×3 output pooled feature map. The output of pooling layer 120 is fed into subsequent convolutional layers 110 for further feature extraction. In some embodiments, pooling layer 120 operates on each feature map individually to create a new set of the same number of pooled feature maps.

[0047] Fully connected layer 130 is the last layer of the DNN. Fully connected layer 130 may or may not be convolutional. Fully connected layer 130 receives input operands. The input operands define the outputs of convolutional layers 110 and pooling layers 120, and include the values ​​of the final feature map generated by the last pooling layer 120 in the sequence. Fully connected layer 130 applies a linear combination and activation function to the input operands and generates a vector. The vector can contain as many elements as there are classes: element i represents the probability that the image belongs to class i. Therefore, each element is between 0 and 1, and the sum of all probabilities equals 1. These probabilities are calculated by the last fully connected layer 130 using either a logistic function (for binary classification) or a SoftMax function (for multi-class classification) as the activation function.

[0048] In some embodiments, the fully connected layer 130 classifies the input image 105 and returns an operand of size N, where N is the number of classes in the image classification problem. Figure 1 In this embodiment, N equals 3 because there are 3 objects 115, 125, and 135 in the input image. Each element of the operand indicates the probability that the input image 105 belongs to a category. To calculate the probabilities, the fully connected layer 130 multiplies each input element by a weight, sums them, and then applies an activation function (e.g., logistic if N=2, and SoftMax if N>2). This is equivalent to multiplying the input operand by a matrix containing the weights. In this example, the vector includes 3 probabilities: a first probability indicating that object 115 is a tree, a second probability indicating that object 125 is a car, and a third probability indicating that object 135 is a person. In other embodiments where the input image 105 includes different objects or different numbers of objects, the individual values ​​may be different.

[0049] Example Convolution Figure 2 Example convolutions according to various embodiments are shown. The convolution can be a convolutional layer of a DNN (e.g., ...). Figure 1 The deep learning operation is performed in the convolutional layer 110. Convolution can be performed on the input tensor 210 and the filter 220 (referred to individually as "filter 220"). The filter, a subset of the filters, or a combination of filters can be referred to as the weight tensor of the convolution. The result of the convolution is the output tensor 230. In some embodiments, the convolution is performed by a DNN accelerator. Examples of DNN accelerators could be... Figure 3 The DNN accelerator 302 is used in the DNN accelerator 302. For example, convolution can be performed by the sparse unit array 370 in the DNN accelerator 302.

[0050] exist Figure 2In this embodiment, the input tensor 210 includes activations (also referred to as "input activations," "elements," or "input elements") arranged in a 3D matrix. Input elements are data points in the input tensor 210. The input tensor 210 has a spatial size H. in ×W in ×C in H in W is the height of the 3D matrix (i.e., the length along the Y-axis, which indicates the number of activations in a column of the 3D matrix for each input channel). in It is the width of the 3D matrix (i.e., the length along the X-axis, which indicates the number of activations in a row of the 2D matrix for each input channel), and C in This is the depth of the 3D matrix (i.e., the length along the Z-axis, which indicates the number of input channels). For simplicity and illustration, input tensor 210 has a spatial size of 7×7×3, meaning input tensor 210 includes three input channels, and each input channel has a 7×7 2D matrix. Each input element in input tensor 210 can be represented by (X, Y, Z) coordinates. In other embodiments, the height, width, or depth of input tensor 210 may be different.

[0051] Each filter 220 includes weights arranged in a 3D matrix. The values ​​of the weights can be determined by training a DNN. Filter 220 has a spatial size H. f ×W f ×C f H f It is the height of the filter (i.e., the length along the Y-axis, which indicates the number of weights in each column of each kernel), W f It is the width of the filter (i.e., the length along the X-axis, which indicates the number of weights in each row of each kernel), and C f It is the depth of the filter (i.e., the length along the Z-axis, which indicates the number of channels). In some embodiments, C f Equals C in For the purposes of simplification and explanation, Figure 2 Each filter 220 in the input tensor has a spatial size of 2×3×3, meaning that filter 220 includes two convolutional kernels with a spatial size of 2×3. In other embodiments, the height, width, or depth of filter 220 may be different. The spatial size of the convolutional kernel is smaller than the spatial size of the 2D matrix of each input channel in the input tensor 210.

[0052] An activation or weight may occupy one or more bytes in memory. The number of bytes for an activation or weight may depend on the data format. For example, when an activation or weight is in INT8 format, it occupies one byte. When an activation or weight is in FP16 format, it occupies two bytes. Other data formats can be used for activations or weights.

[0053] In convolution, each filter 220 slides through the input tensor 210 and generates a 2D matrix for the output channel in the output tensor 230. Figure 2 In this embodiment, the 2D matrix has a spatial size of 5×5. The output tensor 230 includes activations (also referred to as "output activations," "elements," or "output elements") arranged in the 3D matrix. Output activations are data points in the output tensor 230. The output tensor 230 has a spatial size H. out ×W out ×C out H out W is the height of the 3D matrix (i.e., the length along the Y-axis, which indicates the number of output activations in a column of the 2D matrix for each output channel). out It is the width of the 3D matrix (i.e., the length along the X-axis, which indicates the number of output activations in a row of the 2D matrix for each output channel), and C out It is the depth of the 3D matrix (i.e., the length along the Z-axis, which indicates the number of output channels). C out It can be equal to the number of filters 220 in the convolution. H out and W out It can depend on the height and weights of each filter 220 and input tensor 210.

[0054] As part of the convolution, a 2×3×3 sub-tensor 215 (which is in each filter 220 and input tensor 210) can be generated. Figure 2 The MAC operation is performed on (highlighted by dashed lines). The result of the MAC operation on subtensor 215 and filter 220 is the output activation. In some embodiments (e.g., embodiments where the convolution is an integral convolution), the output activation may include 8 bits, such as one byte. In other embodiments (e.g., embodiments where the convolution is a floating-point convolution), the output activation may include more than one byte. For example, the output element may include two bytes.

[0055] After completing the MAC operations on subtensor 215 and all filters 220, vector 235 is generated. Vector 235 is... Figure 2 Highlighted with a diagonal line. Vector 235 comprises a sequence of output activations arranged along the Z-axis. The output activations in vector 235 have the same (X, Y) coordinates, but each output activation corresponds to a different output channel and has a different Z-coordinate. The dimension of vector 235 along the Z-axis can be equal to the total number of output channels in output tensor 230. After generating vector 235, further MAC operations are performed to generate additional vectors until output tensor 230 is generated.

[0056] In some embodiments, the MAC operations on the 2×3×3 subtensor (e.g., subtensor 215) and filter 220 can be performed by multiple MAC units. One or more MAC units can receive input operands (e.g., Figure 2 The input operand 217 shown) and the weight operand (e.g., Figure 2 The weight operand 227 is shown in the diagram. Input operand 217 includes a sequence of activations with the same (x, y) coordinates but different z coordinates. Input operand 217 includes activations from each input channel in input tensor 210. Weight operand 227 includes a sequence of weights with the same (x, y) coordinates but different z coordinates. Weight operand 227 includes weights from each channel in filter 220. The activations in input operand 217 and the weights in weight operand 227 can be sequentially fed into a MAC unit. The MAC unit can receive activations and weights (“activation-weight pairs”) at a time and multiply the activations and weights. The position of the activations in input operand 217 can match the position of the weights in weight operand 227. Activations and weights can correspond to the same channel.

[0057] The activation or weight can be a floating-point number. Floating-point numbers may have various data formats, such as FP32, FP16, BF16, etc. Floating-point numbers can be positive or negative with a decimal point. Floating-point numbers can be represented by a bit sequence that includes one or more bits representing the sign (e.g., positive or negative), bits representing the exponent, and bits representing the mantissa. The mantissa is a portion of the floating-point number that represents its significant digits. Multiplying the mantissa by the exponent of the base yields the actual value of the floating-point number.

[0058] In some embodiments, the output activations in output tensor 230 may be further processed based on one or more activation functions before being stored or input to the next layer of the DNN. Processing based on one or more activation functions may be at least part of post-processing of the convolution. In some embodiments, post-processing may include one or more other computations, such as offset computation, bias computation, etc. The result of post-processing may be stored in the local memory of the computation block and used as input to the next DNN layer. In some embodiments, the input activations in input tensor 210 may be the result of post-processing of a previous DNN layer.

[0059] Example DNN system Figure 3 This is a block diagram of a DNN system 300 according to various embodiments. The entire DNN system 300 or a portion thereof may be implemented in one or more computing devices, such as computing device 1900 in FIG. 19. The DNN system 300 can generate and execute DNNs, for example... Figure 1DNN 100 in the example. Figure 3 As shown, the DNN system 300 includes a DNN module 301 and a DNN accelerator 302. In other embodiments, the DNN system 300 may include alternative configurations, different, or additional components. For example, the DNN system 300 may include multiple DNN modules or multiple DNN accelerators. Furthermore, the functionality of the components belonging to the DNN system 300 may be implemented by different components included in the DNN system 300 or different systems. In some embodiments, the DNN module 301 and the DNN accelerator 302 may include different types of processing units. In the example, the DNN module 301 may be implemented by a CPU. The DNN accelerator 302 may also be referred to as an AI accelerator or an AI processor. The DNN module 301 and the DNN accelerator 302 may be implemented on the same chip or on separate chips.

[0060] DNN module 301 facilitates the generation and deployment of DNNs. In some embodiments, DNN module 301 can generate and train DNNs. For example, DNN module 301 can define the hierarchical architecture of the DNN. DNN module 301 can also determine the intrinsic parameters of the DNN through the DNN training process. DNN module 301 can also determine one or more hyperparameters that define how the DNN is trained. An example hyperparameter is the sparsity ratio, which defines the sparsity level of one or more deep learning tensors of the DNN.

[0061] DNN module 301 can also compress the DNN, for example, during or after training. In some embodiments, DNN module 301 can prune weights in one or more layers of the DNN by changing non-zero value weights to zero. DNN module 301 can prune weights based on a target weight sparsity ratio. The weight sparsity ratio can be the ratio of the number of zero-value weights to the total number of weights. In the example where DNN module 301 prunes weights during DNN training, DNN module 301 can prune the layer weights after one or more epochs to achieve the target sparsity ratio. DNN module 301 can prevent the pruned weights from changing their values ​​during the remainder of the training process. Alternatively, DNN module 301 can allow the pruned weights to change their values ​​such that the pruned zero-value weights can have non-zero values ​​after further training. DNN module 301 can prune the layer weights again after one or more additional epochs.

[0062] DNN module 301 can deploy trained, compressed, or validated DNNs for use in deep learning applications. In some embodiments, DNN module 301 can distribute trained, compressed, or validated DNNs to devices or systems that can use the DNNs to perform tasks for which the DNNs were trained (e.g., image classification, motion planning, etc.). In other embodiments, DNN module 301 can use DNN accelerator 302 to facilitate DNN deployment. For example, DNN module 301 can receive data from a device or system coupled to DNN system 300 and input the received data (or data generated by DNN module 301, for example, based on the received data) into the DNN. DNN module 301 can generate instructions (e.g., configuration files) to control the operation of DNN accelerator 302 during DNN execution. DNN module 301 can receive the output of the DNN from DNN accelerator 302. DNN module 301 can transmit the output of the DNN (or the result of processing the output of the DNN by DNN module 301) to a device or system.

[0063] DNN module 301 can control the execution process of a trained, compressed, or validated DNN. In some embodiments, DNN module 301 facilitates the execution of Fourier transform operations by DNN accelerator 302. For example, DNN module 301 can convert the Fourier transform operation into a matrix multiplication that can be performed by DNN accelerator 302. Matrix multiplication can include a MAC operation similar to the MAC operation in convolution. DNN module 301 can store the input signal of the Fourier transform operation as an activation vector. DNN module 301 can also generate a transformation matrix with a rotation factor for the Fourier transform operation and store the transformation matrix as a weight vector. The activation vector and weight vector can be processed by DNN accelerator 302 in the same or similar manner as DNN accelerator 302 processes activation operands and weight operands in convolution. The following is in conjunction with... Figure 4 Some aspects of the DNN module 301 are provided.

[0064] DNN accelerator 302 executes the DNN provided by DNN module 301. For example, DNN accelerator 302 can perform DNN execution, for instance, by running deep learning operations within the DNN, for training the DNN, or for performing tasks using a trained / compressed / validated DNN. Figure 3As shown, the DNN accelerator 302 includes a memory 310, a DMA (Direct Memory Access) engine 320, and a compute block 330 (referred to separately as "compute block 330"). In other embodiments, alternative configurations, different, or additional components may be included in the DNN accelerator 302. For example, the DNN accelerator 302 may include more than one memory 310 or DMA engine 320. As another example, the DNN accelerator 302 may include a single compute block 330. Furthermore, the functionality of the components belonging to the DNN accelerator 302 may be performed by different components included in the DNN accelerator 302 or by different systems. The components of the DNN accelerator 302 may be implemented in hardware, software, firmware, or some combination thereof.

[0065] Memory 310 stores data associated with deep learning operations performed by the DNN accelerator. In some embodiments, memory 310 may store data to be used by computation block 330 for DNN execution. Memory 310 may store weights, such as the weights of convolutional layers determined by training the DNN. Memory 310 may also store transformation matrices of DFT and IDFT operations. Memory 310 may further store inputs to or outputs of DNN layers, such as data generated by computation block 330 by performing deep learning operations in the DNN. Example deep learning operations include convolution (also referred to as a “convolution operation”), DFT operation, IDFT operation, pooling operation, element-wise operation, activation function, other types of deep learning operations, or some combination thereof. Memory 310 may be the main memory of the DNN accelerator 302. In some embodiments, memory 310 includes one or more dynamic random access memories (DRAM).

[0066] DMA engine 320 facilitates data transfer between memory 310 and the local memory of computation block 330. For example, DMA engine 320 can read data from memory 310 and write the data to the local memory of computation block 330. As another example, DMA engine 320 can read data from the local memory of computation block 330 and write the data to memory 310. DMA engine 320 provides DMA features that allow computation block 330 to initiate data transfer between memory 310 and its local memory, and to perform other operations while the data transfer is taking place. In some embodiments, DMA engine 320 can read tensors from memory 310 and modify the tensors in a manner optimized for computation block 330 before writing them to the local memory of computation block 330.

[0067] Computation block 330 can perform deep learning operations within a DNN. For example, computation block 330 can execute a DNN layer by running one or more deep learning operations within the DNN layer. Computation block 330 can execute one layer or a portion of a layer at a time. Computation block 330 may be able to run various types of deep learning operations, such as convolution, pooling, element-wise operations, linear operations, non-linear operations, and so on. In the example, computation block 330 can perform convolution, such as standard convolution or depthwise convolution. In some embodiments, computation block 330 receives an input tensor and one or more convolution kernels and performs a convolution with the input tensor and the convolution kernels. The result of the convolution can be an output tensor, which can be further computed, for example, by computation block 330 or another computation block 330. In some embodiments, the operations of a DNN layer can be run in parallel by multiple computation blocks 330. For example, multiple computation blocks 330 can each perform a portion of the convolution workload. Data can be shared among computation blocks 330. Computation block 330 can also be referred to as a computation tile. In some embodiments, each computation block 330 can be a processing unit.

[0068] exist Figure 3 In some embodiments, each computation block 330 includes a local memory 340, a sparsity pattern module 350, a loading module 360, a sparse cell array 370 (also referred to as a data processing unit), and an exhaust module 380. Some or all of the components of the computation block 330 may be implemented on the same chip. In other embodiments, alternative configurations, different or additional components may be included in the computation block 330. Furthermore, the functionality of the components belonging to the computation block 330 may be implemented by different components included in the computation block 330, different computation blocks 330, another component of the DNN accelerator 302, or different systems. The components of the computation block 330 may be implemented in hardware, software, firmware, or some combination thereof.

[0069] Local memory 340 is located locally within the corresponding computation block 330. Figure 3 In one embodiment, local memory 340 is internal to computation block 330. In other embodiments, local memory 340 may be external to computation block 330. Data in local memory 340 may be transferred to or from memory 310, for example, via DMA engine 320. In some embodiments, data in local memory 340 may be transferred to or from the local memory of another computation block 330. Local memory 340 may store data received, used, or generated by sparse pattern module 350, loading module 360, sparse cell array 370, or drain module 380. Examples of data may include input activations, weights, output activations, sparse bitmaps, etc.

[0070] In some embodiments, local memory 340 may store dense tensors (e.g., dense activation tensors, dense weight tensors, etc.), sparse tensors (e.g., sparse activation tensors, sparse weight tensors, etc.), and so on. A dense tensor may be a tensor from which zero-value elements (if any) have not been removed. A dense tensor can be converted into a sparse tensor by removing one or more zero-value elements from it. A sparse tensor may also be referred to as a compressed tensor or a packed tensor. The process of converting a dense tensor into a sparse tensor may be called sparsity encoding. Sparsity encoding can also generate sparse tensors. Each element in a sparse tensor may correspond to a different element in a dense tensor and indicate whether an element in the dense tensor is zero. A sparse tensor may indicate the position of an element of a sparse tensor within a dense tensor. A sparse tensor may be a sparse bitmap, where each element is a bit. A sparse tensor can be converted into a dense tensor by a densification process in which one or more zeros may be added to the sparse tensor based on the sparse tensor.

[0071] In some embodiments, local memory 340 includes one or more static random access memories (SRAMs). Local memory 340 may be byte-addressable, and each memory address identifies a single byte (eight bits) of storage. In some embodiments, local memory 340 may include a memory bank. The number of data banks in local memory 340 may be 16, 64, 128, 356, 512, 1024, 3048, or other numbers. A memory bank may include multiple memory cells. In the example, a data bank may include 8, 16, 64, or different numbers of memory cells. A memory bank or a memory cell within a memory bank may have a memory address. In the example, a memory cell may store a single byte, and data larger than a single byte may be stored in memory cells with contiguous memory addresses, i.e., adjacent memory cells. For example, a memory cell may store an integer in INT8 format, whereas storing a 16-bit number in FP16 or BF16 format might require two memory cells. In some embodiments, 16 bits can be transferred from local memory 340 in a single read cycle. In other embodiments, 16 bits can be transferred from local memory 340 in multiple read cycles (e.g., two cycles).

[0072] The sparsity mode module 350 determines the sparsity mode employed by the computation block 330 to execute the DNN layer. For example, the sparsity mode module 350 may determine whether to accelerate the layer based on weight sparsity, activation sparsity, or both. The sparsity mode module 350 selects a sparsity mode for the layer from a set of sparsity modes, such as a combined sparsity mode that accelerates the layer based on both weight sparsity and activation sparsity, an activation sparsity mode that accelerates the layer based on activation sparsity but not on weight sparsity, a weight sparsity mode that accelerates the layer based on weight sparsity but not on activation sparsity, and a dense mode that does not accelerate the layer based on sparsity. In some embodiments (e.g., embodiments where the layer is executed by multiple computation blocks 330), the sparsity module 345 may determine the sparsity mode for executing the layer for all computation blocks 330. In some embodiments, the sparsity mode module 350 may receive configuration parameters from the DNN module 301. Configuration parameters can correspond to layers and indicate whether to accelerate the layer based on weight sparsity. The sparsity mode module 350 can determine the sparsity mode of the layer based on the configuration parameters.

[0073] The loading module 360 ​​loads data from local memory 340 into the sparse cell array 370. The loading module 360 ​​can read tensors from local memory 340. Tensors may include sparse activation tensors, sparse weight tensors, activation sparsity tensors, weight sparsity tensors, etc. In some embodiments, the loading module 360 ​​may load data based on a sparsity pattern determined by the sparsity pattern module 350. The loading module 360 ​​may select different data to be transmitted to the sparse cell array 370 in different sparsity patterns. For example, the loading module 360 ​​may transmit the activation sparsity tensor and weight sparsity tensor of the layer to the sparse cell array 370 in a combined sparsity pattern, while in an activation sparsity pattern, it may transmit the activation sparsity tensor instead of the weight sparsity tensor to the sparse cell array 370, and in a weight sparsity pattern, it may transmit the weight sparsity tensor instead of the activation sparsity tensor to the sparse cell array 370. In dense mode, loading module 360 ​​does not transmit active sparsity tensors or weighted sparsity tensors to sparse cell array 370.

[0074] In some embodiments, before providing data to the sparse cell array 370, the loading module 360 ​​may process (e.g., densify) data stored in local memory 340. In an example, when operating in a weighted sparsity mode, the loading module 360 ​​may densify a sparse activation tensor based on the corresponding activation sparsity tensor to generate a dense activation tensor. For example, the loading module 360 ​​may add one or more zeros to the sparse activation tensor based on the activation sparsity tensor associated with the sparse activation tensor to generate a dense activation tensor. The dense activation tensor includes one or more elements compared to the sparse activation tensor. The value of the (one or more) additional elements is zero. The loading module 360 ​​may identify one or more elements in the activation sparsity tensor corresponding to the (one or more) zero-value elements, determine the position of each zero-value element in the dense activation tensor, and insert the (one or more) zero-value elements into the sparse activation tensor based on the determined positions. After densification, the loading module 360 ​​may transfer the dense activation tensor to the sparse cell array 370. The loading module 360 ​​can also transmit the corresponding sparse weight tensor and weight sparsity tensor to the sparse cell array 370. The activation sparsity tensor of the dense activation tensor may not be loaded into the sparse cell array 370.

[0075] In another example, when the operation is in active sparsity mode, the loading module 360 ​​can densify the sparse weight tensor by inserting zeros into the sparse weight tensor based on the corresponding weight sparsity tensor, thereby generating a dense weight tensor. The densification of the sparse weight tensor can be similar to the densification of the sparse activation tensor described above. After densification, the loading module 360 ​​can transfer the dense weight tensor to the sparse cell array 370. The loading module 360 ​​can also transfer the corresponding sparse activation tensor and activation sparsity tensor to the sparse cell array 370. The weight sparsity tensor of the dense weight tensor may not be loaded into the sparse cell array 370.

[0076] In yet another example, when operating in dense mode, loading module 360 ​​can densify the sparse weight tensor and sparse activation tensor. Loading module 360 ​​can generate the input tensor and weight tensor of the layer and pass the tensors to sparse cell array 370 for execution of the layer without sparsity acceleration.

[0077] The sparse unit array 370 may include one or more sparse convolutional units. Each sparse convolutional unit may include one or more MAC units capable of performing MAC operations. The MAC units in the sparse convolutional unit may be arranged in an array including rows and columns. The sparse convolutional units may be arranged in one or more rows and one or more columns in the sparse unit array 370. All MAC units in the sparse unit array 370 may form a larger array including more rows and columns. In some embodiments (e.g., embodiments where computation block 330 performs a convolutional layer), the computation in the MAC unit may be a MAC operation on activation operands and weight operands. The activation operands may be activation tensors, which may include one or more activations in the input tensor of the convolution. Different activations may be in different input channels. The weight operands may be weight tensors, which may include one or more weights in the filters of the convolution. The values ​​of the weights are determined by training the DNN. The weights in the weight operands may be in different input channels.

[0078] In some embodiments, a MAC unit includes one or more multipliers for performing multiplication. A MAC unit may also include one or more accumulators (“adders”) for performing accumulation. The columns of a MAC unit are referred to as MAC columns. A MAC column may be associated with one or more MAC paths. A MAC path is a path used, for example, by a loading module 360 ​​to load data into a MAC column. A MAC path may also be referred to as a data transfer path or a data loading path. A MAC column may have multiple MAC paths. The load bandwidth of a MAC column is the aggregation of the load bandwidth of all MAC paths associated with the MAC column. Using a certain number of MAC paths, data can be fed to the same number of independent PEs simultaneously. In some embodiments where a MAC column has four MAC paths for feeding activations or weights into the MAC column and each MAC path may have 16 bytes of bandwidth, these four MAC paths may have a total load bandwidth of 64 bytes.

[0079] In some embodiments, the sparse unit array 370 may be capable of performing depthwise convolution, standard convolution, or both. In depthwise convolution, the MAC unit may perform a MAC operation comprising a sequence of multiplications including input operands and weight operands. Each multiplication in the sequence (also called a period) is a multiplication of different activations in the input operands with different weights in the weight operands. Activations and weights in the same period may correspond to the same channel. The sequence of multiplications produces a product operand comprising a sequence of product operations. The MAC operation may also include accumulation, where multiple product operands are accumulated to produce the output operand of the MAC unit. The sparse unit array 370 may output multiple output operands at once, each operand generated by a different MAC unit. In standard convolution, the MAC operation may include accumulation across channels. For example, instead of generating output operands, the MAC unit may accumulate products across different channels to generate a single output point.

[0080] In some embodiments, the sparse unit array 370 can perform MAC operations in quantized deep learning operations, such as MAC operations in quantized convolution. In some embodiments, the MAC units in the sparse unit array 370 can receive quantized activations and quantized weights, and compute a quantized MAC result. The quantized MAC result can be a quantized value in integer format and can be the output of the PE. In some embodiments, the MAC unit can also include a quantized multiplier that can multiply the quantized scale by the quantized MAC result, and the output of the MAC unit can be a real value in floating-point format. The MAC unit may not include a quantized subtractor because zero-point offset is not required for MAC operations in quantized deep learning operations.

[0081] In some embodiments, the sparse unit array 370 may include sparsity acceleration logic to facilitate sparsity acceleration. For example, each sparse convolutional unit in the sparse unit array 370 may include one or more sparsity modules. In an example, each MAC column or each MAC row may have a corresponding sparsity module that accelerates MAC operations in the MAC column or MAC row. In some embodiments, the sparsity module accelerates computation in the sparse unit array 370 based on sparsity in activations, sparsity in weights, or both. The sparsity module may include a storage unit for storing sparsity tensors, which may be loaded into the storage unit by a loading module 360. The sparsity tensor may be an activation sparsity tensor, a weight sparsity tensor, or a combined sparsity tensor.

[0082] An activation sparsity tensor can be a sparse tensor of an activation tensor and has the same number of elements as the activation tensor. Elements in the activation sparsity tensor indicate whether the corresponding element in the activation tensor is zero. For example, a zero value in the activation sparsity tensor indicates that the corresponding element in the activation tensor is zero. A one value in the activation sparsity tensor indicates that the corresponding element in the activation tensor is non-zero. A weight sparsity tensor can be a sparse tensor of a weight tensor and has the same number of elements as the weight tensor. Elements in the weight sparsity tensor indicate whether the corresponding element in the weight tensor is zero. For example, a zero value in the weight sparsity tensor indicates that the corresponding element in the weight tensor is zero. A one value in the weight sparsity tensor indicates that the corresponding element in the weight tensor is non-zero. The sparsity module can use activation sparsity tensors and weight sparsity tensors to generate composite sparsity tensors. For example, the sparsity module can multiply the elements of the activation sparsity tensor with the corresponding elements of the weighted sparsity tensor to compute the elements of the combinatorial sparsity tensor. The positions of the three elements in their corresponding sparsity tensors may match. In some embodiments, each element in the sparsity tensor can be a bit, and the sparsity tensor can be referred to as a sparsity bitmap.

[0083] The sparsity module can use a sparse tensor to identify the activations and weights to be used by the MAC unit in MAC operations. In an embodiment where the sparse unit array 370 operates in a combinatorial sparsity mode, the sparsity module can identify the activations and weights corresponding to the non-zero value elements of the combinatorial sparsity tensor. In an embodiment where the sparse unit array 370 operates in an activation sparsity mode, the sparsity module can identify the activations and weights corresponding to the non-zero value elements of the activation sparsity tensor. In an embodiment where the sparse unit array 370 operates in a weighted sparsity mode, the sparsity module can identify the activations and weights corresponding to the non-zero value elements of the weighted sparsity tensor. The sparsity module can be bypassed in dense mode because no sparsity acceleration will be performed.

[0084] The ejection module 380 ejects data from the sparse unit array 370 and writes the data to local memory 340. This data may be the output of a MAC operation performed by a MAC unit in the sparse unit array 370. In some embodiments, the ejection module 380 may eject data at the sparse convolutional unit level. For each sparse convolutional unit, the ejection module 380 may eject the output of the MAC unit in the sparse convolutional unit based on the row or column index of each MAC unit. For example, the ejection module 380 may use a cyclic sequence to eject data from the sparse convolutional unit. The ejection module 380 may eject the output of some MAC units in each cycle. The cyclic sequence may be configured based on configuration parameters indicating the operating mode of the loading module 360.

[0085] In some embodiments, the discharge module 380 can determine whether to discharge the output of the MAC unit based on the column index of the MAC unit when the loading module operation is in active sparsity mode, or based on the row index of the MAC unit when the loading module operation is in weighted sparsity mode. For example, for a MAC operation performed by the loading module 360 ​​in active sparsity mode, the discharge module 380 can discharge the output of different MAC columns in each cycle. The cyclic sequence can begin with the first MAC column (e.g., the MAC column to the left of the sparse convolution unit) and end with the last MAC column (e.g., the MAC column to the right of the sparse convolution unit). For a MAC operation performed by the loading module 360 ​​in weighted sparsity mode, the discharge module 380 can discharge the output of different MAC rows in each cycle. The cyclic sequence can begin with the first MAC row (e.g., the MAC row at the top of the sparse convolution unit) and end with the last MAC row (e.g., the MAC column at the bottom of the sparse convolution unit). In other embodiments, the discharge module 380 may determine whether to discharge the output of the MAC unit based on the row index of the MAC unit when the loading module operation is in the active sparsity mode, or to discharge the output of the MAC unit based on the column index of the MAC unit when the loading module operation is in the weighted sparsity mode.

[0086] The discharge module 380 may also include sparsity encoding logic that can convert the output of the sparse cell array 370 from a dense format to a sparse format. For example, the discharge module 380 may be implemented using one or more sparse encoders. A sparse encoder converts dense data into compressed data based on sparsity in the dense data. For example, a sparse encoder may remove zeros from the activation tensor computed by the sparse cell array 370 to convert the activation tensor into a compressed activation tensor. A sparse encoder may also generate sparse tensors, including activation sparse tensors.

[0087] In some embodiments, the data discharged from the sparse cell array 370 may be the output tensor of a deep learning operation (e.g., Figure 2 The output tensor (230) is at least a portion of the output tensor. A sparse encoder can generate a compressed version of the output tensor. The sparse encoder can identify each zero-value activation in the output tensor and remove these activations from the output tensor to generate a compressed activation tensor (also known as a "sparse activation tensor"). The sparse encoder can also generate one or more sparse tensors for the output tensor. A sparse tensor can correspond to a portion of the output tensor (e.g., ...). Figure 2 (Vector 235 in the vector). A sparse tensor can include sparse elements (e.g., bits), each sparse element corresponding to a different activation in the vector and indicating whether the corresponding activation is zero.

[0088] The discharge module 380 can write the compressed activation tensor and one or more sparse tensors to local memory 340. The sparse activation tensor and one or more sparse tensors can be further loaded into memory 310, for example, via DMA engine 320. Additionally or alternatively, the sparse activation tensor and one or more sparse tensors can be loaded into a sparse cell array by the loading module 360 ​​for further computation, such as for performing deep learning operations in lower layers.

[0089] DNN accelerator 302 can be used to perform Fourier transform operations, such as those in a DNN. Fourier transform operations can be converted into matrix operations similar to convolution. For example, the input signal for a Fourier transform operation can be encoded by an input tensor and processed as such. The transformation matrix of the Fourier transform operation can be processed by DNN accelerator 302 as if it were the weight tensor of a convolution. DNN accelerator 302 (e.g., sparse unit array 370) can perform a MAC operation on the input tensor and the transformation matrix to compute the Fourier transform of the input signal, i.e., the output signal of the Fourier transform operation. The transformation matrix can be determined offline, for example, before performing the Fourier transform operation or even before executing the entire DNN. In some embodiments, the transformation matrix can be determined by DNN module 301.

[0090] Figure 4 This is a block diagram of a DNN module 400 according to various embodiments. The DNN module 400 can be... Figure 3 An example of the DNN module 301. (See example...) Figure 4 As shown, the DNN module 400 includes an interface module 410, a training module 420, a compression module 430, a validation module 440, a Fourier transform module 450, and a data storage module 460. In other embodiments, the DNN module 400 may include alternative configurations, different or additional components. Furthermore, the functionality of the components belonging to the DNN module 400 may be implemented by different components, different modules, or systems included in the DNN module 400.

[0091] Interface module 410 facilitates communication between DNN module 400 and other modules or systems. For example, interface module 410 establishes communication between DNN module 400 and an external database to receive data that can be used to train the DNN or input into the DNN to perform tasks. As another example, interface module 410 supports DNN module 400 in distributing the DNN to other systems, such as computing devices configured to apply the DNN to perform tasks.

[0092] Training module 420 trains the DNN using a training dataset. Training module 420 forms the training dataset. In embodiments where training module 420 trains the DNN to recognize objects in images, the training dataset includes training images and training labels. The training labels describe the ground-truth classification of objects in the training images. In some embodiments, each label in the training dataset corresponds to an object in a training image. In some embodiments, a portion of the training dataset can be used for initial training of the DNN, while the remainder of the training dataset can be reserved as a validation subset used by validation module 440 to validate the performance of the trained DNN. The portion of the training dataset that does not include the tuning and validation subsets can be used to train the DNN.

[0093] The training module 420 also determines the hyperparameters used to train the DNN. Hyperparameters are variables that specify the DNN training process. Hyperparameters are different from parameters inside the DNN (such as filter weights). In some embodiments, hyperparameters include variables that determine the DNN architecture, such as the number of hidden layers. Hyperparameters also include variables that determine how the DNN is trained, such as batch size, number of epochs, etc. Batch size defines the number of training samples to be solved before updating the DNN's parameters. The batch size is equal to or less than the number of samples in the training dataset. The training dataset can be divided into one or more batches. The number of epochs defines the number of times the entire training dataset is passed back and forth throughout the network. The number of epochs defines the number of times the deep learning algorithm solves the entire training dataset. An epoch means that each training sample in the training dataset has a chance to update the parameters inside the DNN. An epoch can include one or more batches. The number of epochs can be 1, 5, 10, 50, 100, 500, 1000, or even greater.

[0094] Training module 420 defines the architecture of a DNN based on some hyperparameters. The DNN architecture includes an input layer, an output layer, and multiple hidden layers. The input layer of a DNN can include tensors (e.g., multidimensional arrays) specifying attributes of the input image, such as the height, width, and depth of the input image (e.g., specifying the number of bits of color in the pixels of the input image). The output layer includes labels for objects in the input layer. Hidden layers are layers between the input and output layers. Hidden layers include one or more convolutional layers and one or more other types of layers, such as pooling layers, fully connected layers, normalization layers, SoftMax, or logistic layers. The convolutional layers of a DNN abstract the input image into feature maps, which are represented by tensors specifying the feature map height, width, and number of channels (e.g., red, green, and blue images include 3 channels). Pooling layers are used to reduce the spatial volume of the input image after convolution. They are used between two convolutional layers. Fully connected layers involve weights, biases, and neurons. They connect neurons in one layer to neurons in another layer. They are used to classify images among different classes through training.

[0095] In defining the architecture of the DNN, the training module 420 also adds activation functions to the hidden or output layers. The activation function of a layer transforms the weighted sum of the layer's inputs into the layer's output. The activation function can be, for example, ReLU, tangent, or other types of activation functions.

[0096] After defining the architecture of the DNN in training module 420, training module 420 inputs a training dataset into the DNN. The training dataset includes multiple training samples. Examples of training samples include objects in images and the ground truth labels of those objects. Training module 420 modifies the parameters internal to the DNN (“DNN internal parameters”) to minimize the error between the labels of the training objects generated by the DNN and the ground truth labels of the objects. Internal parameters include the weights of filters in the convolutional layers of the DNN. In some embodiments, training module 420 uses a cost function to minimize the error.

[0097] The training module 420 can train the DNN for a predetermined number of epochs. The number of epochs is a hyperparameter that defines how many times the deep learning algorithm will solve the entire training dataset. An epoch means that each sample in the training dataset has a chance to update the internal parameters of the DNN. After the training module 420 has completed the predetermined number of epochs, it can stop updating the parameters in the DNN. A DNN with updated parameters is called a trained DNN.

[0098] Compression module 430 compresses the DNN. For example, compression module 430 can add pruning operations to the DNN layers to reduce computational complexity or memory usage. The pruning operation can trim the weight tensor of the DNN layer by changing one or more non-zero value weights to zero. Modifications can be made before, during, or after training. Weights can be pruned during training, during inference, or a combination of both. Compression module 430 can determine the sparsity ratio of the DNN layers. The sparsity ratio can be the ratio of the number of zero-value weights in the layer to the total number of weights. Compression module 430 can perform pruning operations until the sparsity ratio of the DNN layers meets a target sparsity ratio, such as 10%, 20%, 30%, 40%, 50%, etc.

[0099] In some embodiments, the compression module 430 may select one or more layers in the DNN and modify each selected layer with a pruning operation. For example, the compression module 430 may select computationally complex layers, such as layers with large filters. For a pruning operation on a layer or a class of layers, the compression module 430 may determine a weight threshold that will not cause the DNN to lose accuracy beyond the accuracy loss constraint. The pruning operation may modify weights with absolute values ​​higher than the weight threshold to zero, while leaving other weights unchanged. Weight pruning can reduce memory storage because zero-value weights may not be stored. Furthermore, the number of operations in a layer can be reduced because the computation of zero-value weights can be skipped without affecting the layer's output. In some embodiments, the compression module 430 may also measure energy savings, final DNN accuracy, or layer-by-layer sparsity caused by the pruning operation.

[0100] After compressing the DNN, the compression module 430 can fine-tune the DNN, for example, through a retraining process. The compression module 430 can fine-tune the DNN after pruning the weights. In some embodiments, the fine-tuning process is a retraining or further training process. For example, after pruning the weights in the DNN, the compression module 430 can further train the DNN by inputting a training dataset into the DNN. The values ​​of the unpruned weights in the DNN can be modified based on the output of the DNN and the true labels of the training samples in the training dataset. In some embodiments, the values ​​of the pruned weights (i.e., zero) remain unchanged during the fine-tuning process. For example, the compression module 430 can place a mask on the pruned weight block, and the mask can prevent the values ​​in the pruned weight block from being changed during the fine-tuning process. In other embodiments, the values ​​of all weights, including the pruned weights, can be changed during fine-tuning. After one or more epochs of retraining and weight changes by the compression module 430, the compression module 430 can perform a new pruning process, for example, by selecting a weight block and pruning the selected weight block. In some embodiments, the weight pruning process can be repeated multiple times before the fine-tuning process is completed.

[0101] In some embodiments, the number of rounds in the fine-tuning process may differ from the number of rounds in the training process used to determine the pre-pruned weight values. For example, the fine-tuning process may have fewer rounds than the training process. In the examples, the number of rounds in the fine-tuning process may be relatively small, such as 2, 3, 4, 5, etc.

[0102] The validation module 440 verifies the accuracy of the trained or compressed DNN. In some embodiments, the validation module 440 inputs samples from the validation dataset into the trained DNN and uses the output of the DNN to determine the model accuracy. In some embodiments, the validation dataset may be formed from some or all of the samples in the training dataset. Additionally or alternatively, the validation dataset may include additional samples in addition to those in the training set. In some embodiments, the validation module 440 may determine an accuracy score that measures the accuracy, recall, or a combination of accuracy and recall of the DNN. The validation module 440 may use the following metrics to determine the accuracy score: accuracy = TP / (TP+FP) and recall = TP / (TP+FN), where accuracy can be how many of the total number of predictions (TP+FP or false positives) the DNN correctly predicts (TP or true positives), and recall can be how many of the total number of objects with the considered attributes (TP+FN or false negatives) the DNN correctly predicts (TP). Unify accuracy and recall into a single metric.

[0103] The verification module 440 can compare the accuracy score with a threshold score. In an example where the verification module 440 determines that the accuracy score of the DNN is less than the threshold score, the verification module 440 instructs the training module 420 to retrain the DNN. In one embodiment, the training module 420 can iteratively retrain the DNN until a stopping condition is met, such as an accuracy measurement indicating that the DNN may be accurate enough, or multiple training epochs have already occurred.

[0104] The Fourier transform module 450 facilitates the conversion of Fourier transform operations into matrix multiplications that can be performed by a DNN accelerator (e.g., DNN accelerator 302). A Fourier transform operation can be an operation that transforms an input signal into an output signal. The input and output signals can be in different domains. For example, the input signal may be in the time domain, while the output signal may be in the frequency domain. As another example, the input signal may be in the frequency domain, while the output signal may be in the time domain. Fourier transform operations can include DFT operations, IDFT operations, real-valued DFT (RDFT) operations, inverse RDFT (IRDFT) operations, STFT operations, other types of Fourier transform operations, or some combination thereof.

[0105] In some embodiments, the DFT is a method for transforming a finite, equally spaced sequence of samples of a function into a sequence of equally spaced samples of the same length for the Discrete-Time Fourier Transform (DTFT), which is a complex-valued function of frequency. The sampling interval of the DTFT is the reciprocal of the duration of the input sequence. The IDFT is a Fourier series that uses DTFT samples as coefficients of a complex sine curve at the corresponding DTFT frequency. It has the same sample values ​​as the original input sequence. The DFT output can be a frequency domain representation of the input. The DFT can be an invertible linear transform, and its inverse transform is called the IDFT transform. Both the DFT and IDFT can be used in many practical applications, such as digital signal processing and image processing.

[0106] In some embodiments, the DFT operation can be represented as: in, It is the input to the DFT operation, and is a sequence of N complex numbers; and It is the output of the DFT operation, and is another sequence of N complex numbers. Input sequence It can be a signal in the time domain, and the output sequence The signal can be in the frequency domain. The output sequence can be a frequency domain representation of the input sequence. The DFT operation has a corresponding IDFT operation, which converts a signal in the frequency domain to a signal in the time domain. The IDFT operation can be represented as: RDFT is a variant of DFT used when the input sequence is real rather than complex. The input sequence for RDFT can be symmetric. The output sequence of the RDFT operation can be a complex sequence. In some embodiments, the RDFT operation produces two signals: a real output signal and an imaginary output signal. Each of the two output signals can include N / 2+1 data points ranging from 0 to N / 2. The input signal can be in the time domain, and the two output signals can be in the frequency domain. The output signals can constitute a frequency domain representation of the input signal. The RDFT operation can have a corresponding IRDFT operation, which transforms a real sequence in the frequency domain back into an equidistant sample sequence of a function in the time domain.

[0107] The Fourier transform module 450 can generate activation vectors from the input signal of the Fourier transform operation. The input signal of the Fourier transform operation can be a sequence of discrete data elements. The Fourier transform module 450 can transform the input sequence into a 2D matrix. For example, the Fourier transform module 450 can store segments of the input sequence in different storage elements. Each segment can be used as a context or operand by the DNN accelerator 302 and processed later. These segments are called activation vectors and can be stored and processed by the DNN accelerator 302 in the same or similar manner as activation vectors in convolution. In some embodiments, the activation vectors of the Fourier transform operation can be rows or columns of a 2D matrix representing the input signal.

[0108] In some embodiments (e.g., where the Fourier transform is an STFT embodiment), the Fourier transform module 450 can extract frames from the input sequence by sliding a window over the input sequence. Frames can be used as activation vectors. During frame extraction, the Fourier transform module 450 can perform padding to add additional elements to the input sequence.

[0109] The Fourier transform module 450 can also generate the transformation matrix for the Fourier transform operation. The Fourier transform module 450 can generate the transformation matrix for the Fourier transform operation in advance, i.e., before performing the Fourier transform operation or even before starting the execution of the DNN including the Fourier transform operation. The transformation matrix can also be called the twitch factor matrix. The twitch factor of the FFT algorithm can be any trigonometric constant coefficient multiplied with the data during the algorithm. For an FFT operation with an N×N input tensor, the Fourier transform module 450 can generate an N×N transformation matrix, for example... Figure 5 The transformation tensor in the value is 500.

[0110] The Fourier transform module 450 can divide the transformation matrix into weight vectors, which can be stored and processed by the DNN accelerator 302 in the same or similar manner as weight vectors in convolution. The weight vectors can be columns or rows of the transformation matrix. In some embodiments (e.g., embodiments where the transformation matrix has complex elements), the Fourier transform module 450 can divide a single column or row of the transformation matrix into two weight vectors: one weight vector includes the real part of the complex elements in that row or column, and the other weight vector includes the imaginary part of the complex elements in that row or column.

[0111] In some embodiments, the Fourier transform module 450 can map two-dimensional DFT (2D-DFT) operations to matrix multiplication operations. A 2D-DFT operation can have a 2D input tensor, also referred to as the input matrix, and a 2D transformation tensor, also referred to as the transformation matrix. The Fourier transform module 450 can convert the 2D-DFT operation into two sequences of one-dimensional DFT (1D-DFT) operations. The first sequence can include 1D-DFT operations on all rows of the input matrix. Each 1D-DFT operation in the first sequence can be a matrix multiplication operation on a different row of the transformation matrix and the input matrix. The second sequence can include 1D-DFT operations on all columns of the input matrix. Each 1D-DFT operation in the second sequence can be a matrix multiplication operation on a different column of the transformation matrix and the input matrix.

[0112] In some embodiments, the input matrix can be transposed after the first sequence is completed, such that each row of the input matrix becomes a column in the transposed input matrix. The second sequence can be performed on the transposed input matrix in the same manner as the first sequence. In one embodiment, the second sequence can be performed after the first sequence. In another embodiment, the second sequence can be performed before the first sequence. In yet another embodiment, the two sequences can be performed simultaneously, for example, by different sparsity units in a DNN accelerator. The two sequences of matrix multiplication operations can be mapped to MAC units in a DNN accelerator. The output of the MAC unit can constitute the output signal of the Fourier transform operation. The following is combined with... Figure 8 and Figure 9 This describes more details about mapping Fourier transform operations to MAC units.

[0113] In some embodiments, the DFT operation is associated with the IDFT operation. For example, a DNN that includes a DFT operation also includes an IDFT operation. In a DNN, the IDFT operation may be arranged after the DFT operation. The Fourier transform module 450 can also generate a transformation matrix for the IDFT operation in the DNN. In embodiments where the DNN includes an RDFT operation, the DNN may also include an IRDFT operation. The transformation matrix of the IRDFT operation may be the complex conjugate of the transformation matrix of the RDFT operation. The transformation matrix of the IRDFT operation can be scaled using the Nth dimension. Since the input sequence of the RDFT operation can be symmetric, the Fourier transform module 450 can configure the DNN accelerator to compute and store a portion of the output sequence, rather than the entire output sequence. For example, the Fourier transform module 450 can configure the DNN accelerator to compute and store N / 2+1 data points out of all N data points in the output sequence. This can save a significant amount of power and memory for the DNN accelerator. For Nk data points, loading weights can be avoided.

[0114] Data storage 460 stores data received, generated, used, or otherwise associated with DNN module 400. For example, data storage 460 stores datasets used by training module 420 and validation module 440. Data storage 460 may also store data generated by training module 420 and validation module 440, such as hyperparameters used to train the DNN, intrinsic parameters of the trained DNN (e.g., weights), data used for sparsity acceleration (e.g., sparse bitmaps), etc. Data storage 460 may store transformation matrices generated by Fourier transform module 450. Figure 4 In one embodiment, data storage 460 is a component of DNN module 400. In other embodiments, data storage 460 may be external to DNN module 400 and communicate with DNN module 400 via a network.

[0115] Figure 5 Example transform tensor 500 according to various embodiments of Fourier transform operations is shown. The Fourier transform operation can be a DFT operation, an IDFT operation, an RDFT operation, an IRDFT operation, etc. Transform tensor 500 can be derived from... Figure 4 The Fourier transform module 450 performs the calculation. For illustrative purposes, the transform tensor 500 is a 2D tensor. In other embodiments, the transform tensor 500 may have different dimensions.

[0116] like Figure 5 As shown, the transform tensor 500 has an N×N spatial size, that is, the transform tensor 500 has N rows and N columns. Figure 5 In this embodiment, N is an integer greater than 5. In other embodiments, N may have other values. The data points in the transform tensor 500 may be rotation factors of the Fourier transform operation. The data points in the transform tensor 500 may be referred to as weights. During the Fourier transform operation, the transform tensor 500 will be processed by the DNN accelerator in the same or similar manner as the weight tensors processed by the DNN accelerator for convolution.

[0117] Figure 6 An example sparse convolutional unit 600 according to various embodiments is shown. The sparse convolutional unit 600 can be in a sparse unit array, for example... Figure 3The sparse convolutional unit 600 is located in a sparse unit array 370. The sparse convolutional unit 600 includes 16 MAC units 610 (referred to individually as "MAC units 610") arranged in four rows and four columns, 16 weight register files 620 (referred to individually as "weight register files 620"), 16 activation register files 630 (referred to individually as "activation register files 630"), four row buffers 640 (referred to individually as "row buffers 640"), and a sparsity module 660 (referred to individually as "sparse module 660"). In other embodiments, the sparse convolutional unit 600 may include fewer, more, or different components. For example, the sparse convolutional unit 600 may include different numbers of MAC units 610, weight register files 620, activation register files 630, row buffers 640, or sparsity modules 660. As another example, the sparse convolutional unit 600 may include column buffers as an alternative to or supplement to the row buffers 640.

[0118] MAC unit 610 is configured to perform MAC operations. Each MAC unit 610 may include one or more multipliers and one or more adders. The multipliers may compute a product by multiplying the activations by the weights each time. In some embodiments (e.g., embodiments where MAC unit 610 includes multiple multipliers), the multipliers may operate simultaneously to process multiple activation-weight pairs and compute multiple products in one cycle. The adders may accumulate the products computed by the multipliers. Although Figure 6 Not shown, but the sparse convolutional unit may include an adder tree comprising multiple adder levels. A first level may receive the outputs of multiple MAC units 610. The number of adders in the first level may be half the number of MAC units 610, and each adder may accumulate the outputs of two MAC units 610. A second level may receive the outputs of the adders in the first level. The number of adders in the second level may be half the number of adders in the first level, and each adder in the second level may accumulate the outputs of two adders in the first level. The adder tree may include one or more other levels. The final level may include a single adder that accumulates the outputs of the adders in the penultimate level to compute a partial sum of the sparse convolutional unit 600.

[0119] Weight register file 620 stores the weights that will be processed during MAC operations. Figure 6 In one embodiment, four weight register files 620 are grouped into storage groups that store data to be used by a column of MAC units 610. There are four storage groups corresponding to four columns of MAC units 610. In some embodiments, the weight register files 620 may correspond to MAC units 610 and store data to be processed by the MAC units. In some embodiments, all 16 weight register files 620 constitute a weight storage unit, which may be... Figure 5Example of a medium-weighted storage unit 545.

[0120] Activating register file 630 stores the activation that will be handled during MAC operations. Figure 6 In this embodiment, four active register files 630 are grouped into storage groups that store data to be used by a row MAC unit 610. There are four storage groups corresponding to four rows of MAC units 610. In some embodiments, the active register files 630 may correspond to MAC units 610 and store data to be processed by the MAC units. In some embodiments, all 16 active register files 630 constitute active storage units. Row buffers 640 store the outputs of MAC units 610. Each row buffer 640 may discharge the output of a single row of MAC units 610.

[0121] The sparsity module 660 facilitates dynamic, sparsity-based acceleration in the sparse convolutional unit 600. Figure 6 In the embodiments, each sparsity module 660 includes a sparsity tensor storage unit 665 and control logic 667. The sparsity tensor storage unit 665 stores combinatorial sparsity tensors. The combinatorial sparsity tensors stored in the sparsity tensor storage unit 665 can correspond to activation tensors and weight tensors. Non-zero elements in the combinatorial sparsity tensors can correspond to non-zero activation-weight pairs including non-zero activations and non-zero weights. The positions of non-zero activations in the activation tensor can match the positions of non-zero weights in the weight tensor. The product of non-zero activations and non-zero weights will be non-zero.

[0122] Control logic 667 can, based on a sparse tensor, control the transfer of stored activations and weights from the weight register file 620 and the activation register file 630 to the MAC unit 610. For example, control logic 667 can, based on a combinatorial sparse tensor, select a subset of weights stored in the weight register file 620 and a subset of activations stored in the activation register file 630. The selected weights and activations form non-zero activation-weight pairs. Control logic 667 can then transfer the selected weights and activations to the MAC unit 610 for performing a MAC operation. This causes other weights stored in the weight register file 620 and other activations stored in the activation register file 630 to skip computation. Figure 6 In this embodiment, each sparsity module 660 controls the sparsity acceleration in the corresponding MAC unit 610. Since the sparsity acceleration is based on both weight sparsity and activation sparsity, 16 sparsity modules 660 are used for accelerated computation in 16 MAC units 610.

[0123] like Figure 6As shown, the sparse convolutional unit 600 is associated with multiplexers (MUX) 603, 604, 605, and 606. In other embodiments, the sparse convolutional unit 600 may be associated with a different number of MUXs or other devices. MUX 603 facilitates, for example, loading weights from local memory 340 into a weight register file 620. An example of MUX 603 could be... Figure 5 The MUX 530 in the example. The MUX 604 facilitates, for example, loading activation from local memory 340 into the activation register file 630. An example of the MUX 604 could be... Figure 5 MUX 540. MUX 605 facilitates loading sparse tensors into sparse tensor storage unit 665. An example of MUX 605 could be... Figure 5 MUX 550. MUX 606 can be an exhaust MUX, which can facilitate the exhaust of the output of MAC unit 610 to, for example, local memory 340.

[0124] In some embodiments, the sparse convolution unit 600 can also perform matrix multiplications transformed from the Fourier transform operation. For the example Fourier transform operation, the MAC unit 610 can perform a MAC operation on two sequences of matrix multiplications transformed from the Fourier transform operation. The weight register file 620 can be used to store data points in the transform tensor of the Fourier transform operation. The activation register file 630 can be used to store data points in the input tensor of the Fourier transform operation. The row buffer 640 can store data points in the output tensor of the Fourier transform operation.

[0125] Figure 7 A sparse cell array 700 according to various embodiments is shown. The sparse cell array 700 can be... Figure 3 An example of a medium sparse cell array 370. Figure 7 In this configuration, the sparse unit array 700 includes sparse convolutional units 710 (referred to individually as "sparse convolutional units 710") arranged in four columns and four rows, an activation memory 720, and a weight memory 730. The sparse unit array 700 may also be referred to as a data processing unit. In other embodiments, the sparse unit array 700 may include fewer, more, or different components. For example, the sparse unit array 700 may include different numbers of columns, rows, or sparse convolutional units 710.

[0126] Each sparse convolutional unit 710 can perform sparsity-accelerated MAC operations. The sparse convolutional unit 710 can facilitate dynamic sparsity patterns. For example, the sparsity pattern of the sparse convolutional unit 710 can dynamically change between a combinational sparsity pattern, an activation sparsity pattern, a weight sparsity pattern, and a dense pattern. Embodiments of the sparse convolutional unit 710 may include... Figure 6The sparse convolutional unit 710 is configured with sparse convolutional units 600 and activation memory 720 storing activations, such as activations in the input tensor of a deep learning operation. Activations can be loaded from activation memory 720 into sparse convolutional units 710. Weight memory 730 stores weights, such as weights in a filter of a deep learning operation. Weights can be loaded from weight memory 730 into sparse convolutional units 710. Activation memory 720 or weight memory 730 may be buffers. In other embodiments, sparse unit array 700 may include dense data memory and sparse data memory instead of activation memory 720 and weight memory 730. Dense data memory may store dense tensors, such as dense tensors generated by loading module 360. Sparse data memory may store sparse tensors.

[0127] The sparse cell array 700 can also perform matrix multiplication in Fourier transform operations. The activation memory 710 can be used to store the input tensors of the Fourier transform operations. The weight memory 730 can be used to store the transformation matrix of the Fourier transform operations.

[0128] Mapping the Fourier transform operation to a sparse cell array Figure 8 The diagram illustrates mapping DFT operations to a sparse cell array 800 according to various embodiments. The sparse cell array 800 can be... Figure 3 An example of a sparse cell array 370. A spatial cell array 800 can also be referred to as a data processing unit. For illustrative purposes, Figure 8 The sparse unit array 800 includes 256 MAC units arranged in 16 rows and 16 columns. The DFT operation has an input tensor of size 16×16 and a transform tensor of size 16×16. In other embodiments, the sparse unit array 800 may include a different number of MAC units or have different shapes. Furthermore, the input tensor or transform tensor may have different shapes or sizes.

[0129] exist Figure 8 In this embodiment, the input tensor is divided into activation vectors 810A-810P (collectively referred to as "multiple activation vectors 810" or "activation vectors 810"), and each activation vector 810 includes 16 activations. Activation vectors 810 can be rows in the input tensor and can be processed as activation operands. The transformation tensor is divided into weight vectors 820A-820P (collectively referred to as "multiple weight vectors 820" or "weight vectors 820"), and each weight vector 820 includes 16 weights. Weight vectors 820 can be columns in the transformation matrix and can be processed as weight operands.

[0130] In some embodiments (e.g., where the DFT operation is an embodiment of a 2D-DFT operation that can be transformed into two sequential 1D-DFT operations, each 1D-DFT operation comprising a sequence of vector-matrix multiplications), 16 MAC units in the same row of the sparse unit array 800 can perform a single vector-matrix multiplication, i.e., the multiplication of the activation vector 810 with the entire transformation matrix. All 256 MAC units can perform all vector-matrix multiplications in the first 1D-DFT operation. After the first 1D-DFT operation is completed, the sparse unit array 800 can perform a second 1D-DFT operation. For example, the input tensor can be transposed such that each activation vector 810 can become a column in the input tensor. The sparse unit array 800 can perform the second 1D-DFT operation in the same manner as it performed the first 1D-DFT operation.

[0131] Figure 9 The diagram illustrates mapping RDFT operations to a sparse cell array 900 according to various embodiments. The sparse cell array 900 can be... Figure 3 An example of a sparse cell array 370. A sparse cell array 900 can also be referred to as a data processing unit. For illustrative purposes, Figure 9 The sparse unit array 900 includes 256 MAC units arranged in 16 rows and 16 columns. The RDFT operation has an input tensor of size 4×4 and a transform tensor of size 4×4. In other embodiments, the sparse unit array 900 may include a different number of MAC units or have different shapes. Furthermore, the input tensor or transform tensor may have different shapes or sizes.

[0132] exist Figure 9In this embodiment, the input tensor has real numbers and is divided into four activation vectors 910A-910D (collectively referred to as "multiple activation vectors 910" or "activation vectors 910"), and each activation vector 910 includes four activations. Activation vectors 910 can be rows in the input tensor and can be processed as activation operands. The transformation tensor has complex numbers and is divided into eight weight vectors 920A-920H (collectively referred to as "multiple weight vectors 920" or "weight vectors 920"), and each weight vector 920 includes four weights. Weight vectors 920 can be processed as weight operands. Four weight vectors 920A-920D have real elements, and the other four weight vectors 920E-920H have imaginary elements. Weight vectors 920A and 920E can form the first column of the transformation matrix. For example, weight vector 920A can include the real part of a complex number in the first column of the transformation matrix, and weight vector 920B can include the imaginary part of a complex number in a column of the transformation matrix. Similarly, weight vectors 920B and 920F can form the second column of the transformation matrix, where weight vector 920B includes the real part of the complex numbers in the second column, and weight vector 920B also includes the imaginary part of the complex numbers in the second column. Weight vectors 920C and 920G can form the third column of the transformation matrix. Weight vectors 920D and 920H can form the fourth column of the transformation matrix.

[0133] Four activation vectors 910 are loaded into four rows of MAC cells. Eight weight vectors 920 are loaded into eight columns of MAC cells. The 32 MAC cells in the four rows and eight columns can perform RDFT operations. The other MAC cells in the sparse cell array 900 may be idle. Although... Figure 9 The mapping of RDFT operations to sparse cell array 900 is shown, but IRDFT operations can be mapped to sparse cell array 900 in the same or similar way.

[0134] Figure 10 The diagram illustrates mapping a complex number's DFT to a sparse cell array 1000 according to various embodiments. The sparse cell array 1000 can be... Figure 3 An example of a sparse cell array 370. A spatial cell array 1000 can also be referred to as a data processing unit. For illustrative purposes, Figure 10 The sparse cell array 1000 includes 256 MAC cells arranged in 16 rows and 16 columns. In other embodiments, the sparse cell array 1000 may include a different number of MAC cells or have different shapes.

[0135] exist Figure 10In this embodiment, both the input tensor and the transformation tensor contain complex numbers. The activation in the input tensor can be represented as a + ib, where a represents the real part and b represents the imaginary part. The weights in the transformation tensor can be represented as c + id, where c represents the real part and d represents the imaginary part. Multiplying the activation by the weights yields an output element represented as ac - bd + i(bc + ad), where ac - bd is the real part and bc + ad is the imaginary part.

[0136] Because the sparse cell array performs element-wise multiplication of different tensor elements during the DFT operation, the execution of the DFT operation can be divided into four separate workloads for the sparse cell array 1000. For example, the activation vector and weight vector can be loaded into the sparse cell array as four pairs: i) a, c; ii) a, b; iii) a, b; and iv) b, c. Each workload can include matrix multiplication on different pairs. For the pair (b, c), a negative scale can be applied to the post-processing cell array associated with the sparse cell array to accommodate the negative sign. Partial additions can be performed separately.

[0137] In some embodiments, the (a, c) multiplication can be performed before the (b, d) multiplication. Figure 10 As shown, in the same loading cycle, (a, c) are loaded into a row MAC unit and a column MAC unit, respectively, and in subsequent loading cycles, (b, d) are loaded into the same row MAC unit and column MAC unit, respectively. In some embodiments, the (b, d) multiplication can be performed with negative weights. These weights can be set by the DNN module 301 with minimal or even no hardware overhead. The discharge module 380 can output real and imaginary results as different outputs, so that they can be correctly understood by the loading module 360 ​​during the execution of subsequent layers.

[0138] In some embodiments, (a, c) multiplication can be fused with (a, d) multiplication. 'a' can be loaded once as an activation (e.g., as an activation vector) onto the sparse cell array 1000, and 'c' and 'd' can be sequentially loaded as two separate sets of weights to be multiplied sequentially with the activations (e.g., as two separate weight vectors). Similarly, (b, c) multiplication can be fused with (b, d) multiplication. 'b' can be loaded once as an activation (e.g., as an activation vector) onto the sparse cell array 1000, and 'c' and 'd' can be sequentially loaded as two separate sets of weights to be multiplied sequentially with the activations (e.g., as two separate weight vectors). This allows activations to be reused in both sets of multiplications, saving power and memory bandwidth. The output corresponding to the 'c' weight set can be consumed later as the real part in one or more subsequent layers. The output corresponding to the 'd' weight set can be consumed later as the imaginary part in one or more subsequent layers.

[0139] Figure 11 Example sliding window patterns of STFT operation according to various embodiments are shown. STFT is a Fourier transform with a real-valued input signal. In the discrete case, the input signal to be transformed can be decomposed into frames (also called chunks) based on a window. These frames can all have the size of the window. A Fourier transform can be performed on each frame, and the complex result can be added to a matrix that can encode the amplitude and phase at each point in time and frequency. The STFT operation can be represented as: Where x[n] represents the input signal; and w[n] represents the window. In some embodiments, m is discrete, while It is continuous. In other embodiments, m and They are all discrete and quantized.

[0140] Frames can be extracted from an input sequence using a sliding window. The STFT operation in a DNN can have a window length (also called "frame length") and a frame stride (also called "stride"). The window length indicates the number of data elements in the window, i.e., the number of data elements in each frame. The frame stride indicates the number of data elements traversed in each sliding pass. An STFT operation can be translated into a series of matrix multiplication operations. Each matrix multiplication operation can be performed on the corresponding frame. In some embodiments, an STFT operation can be represented as a 1D convolution, where the frame stride is the stride and the window length is the number of input channels. In an example of an STFT operation with 16,000 data elements in the input sequence, a window length of 512, and a frame stride of 128, the STFT operation can be translated into a matrix multiplication operation on a 257×512 weight tensor and a 512×1247 input tensor.

[0141] For the purpose of explanation, Figure 11 The STFT operation in the embodiment has a window length of 8 and a frame step size of 1. Figure 11 An input sequence 1110 comprising 14 data elements and a window 1120 comprising 8 data elements are shown. Each data element is composed of... Figure 11 The boxes in the diagram represent the data. Frames 1130A-1130G (collectively referred to as "multiple frames 1130" or "frames 1130") are extracted from the input sequence 1110 using window 1120. Figure 11 In this diagram, frame 1130 is represented by a box filled with a dashed pattern. Frame 1130A is generated from a first slide of window 1120. Frame 1130B is generated from a second slide of window 1120. This continues until frame 1130G is generated. Each slide traverses one data element. In other embodiments, the input length (i.e., the length of the input sequence), window length, or frame step size may have different values.

[0142] Frame 1130 can be generated by DNN module 301. In some embodiments, DNN module 301 stores frame 1130 as a separate activation vector in, for example, memory 310 or local memory 340. The activation vector can be used as context or operand. Loading module 360 ​​can load frame 1130 into sparse cell array 370 for matrix multiplication. In the example, storage elements can be used to store a space of size [missing information]. The activation vector, where It is an integer. This can be equal to the window length. Storage elements can have storage element pointers, which store the location of the storage element in memory, such as memory 310 or local memory 340. Using storage element pointers, a 1D input sequence can be stored as a 2D matrix without any data shifting operations. For example, an input sequence with 16,000 elements can be stored as a 512×1247 2D matrix. When the storage element is represented as having 128 elements, the input sequence can be represented using 1250 storage element pointers. The following combines... Figure 13 More details are described regarding mapping frames to the sparse cell array 370.

[0143] Figure 12 Another example of a sliding window mode for STFT operation according to various embodiments is shown. Figure 12 The sliding window pattern in the diagram needs to be filled, that is, new data elements are added to the input signal. For illustrative purposes, Figure 12 The STFT operation in the embodiment has an input length of 10, a window length of 8, and a frame step size of 1.

[0144] like Figure 12 As shown, the input sequence 1210 includes 10 data elements, and the window 1220 includes 8 data elements. Each data element consists of... Figure 12 The boxes in the diagram represent the frames. Frames 1230A-1230E (collectively referred to as "multiple frames 1230" or "frames 1230") are extracted from the input sequence 1210 using window 1220. Figure 12 In the image, frame 1230 is represented by a box filled with a dashed pattern. Although Figure 12 Four frames 1230 are shown, but a different number of frames can be extracted from the input sequence 1210. In some embodiments, the total number of frames extracted from the input sequence can be equal to the input length divided by the frame step size. For an input length of 10 and a frame step size of 1, the total number of frames can be 10.

[0145] Frame 1230A is generated from a first slide of window 1220. Frame 1230B is generated from a second slide of window 1220. Frame 1230C is generated from a third slide of window 1220. Each slide traverses one data element. To generate frame 1230D, a new data element is added to the end of the input sequence 1210 such that frame 1230D satisfies the window length. Furthermore, to generate frame 1230E, another new data element is added such that frame 1230E satisfies the window length. In some embodiments, each new element is zero. In other embodiments, the new elements may have other values. Furthermore, further new elements may be added to generate more frames.

[0146] Frame 1230 can be generated by DNN module 301. In some embodiments, DNN module 301 stores frame 1230 as a separate context or a separate operand in, for example, memory 310 or local memory 340. Loading module 360 ​​can load frame 1230 into sparse cell array 370 for matrix multiplication. The following is combined with... Figure 13 More details are described regarding mapping frames to the sparse cell array 370.

[0147] Figure 13 The diagram illustrates mapping an STFT-operated frame 1310 to a sparse cell array 1300 according to various embodiments. The sparse cell array 1300 may be... Figure 3 An example of a medium sparse cell array 370. For illustrative purposes, Figure 13 The STFT operation in the embodiment has an input length of 7, a window length of 4, and a frame step size of 2. STFT operations with different input lengths, window lengths, or frame step sizes can also be mapped to the sparse cell array 1300.

[0148] Figure 13 Four frames 1310, individually referred to as "frame 1310", are shown. Each frame 1310 includes four data elements. The four frames 1310 are loaded into four rows of MAC units, respectively. In some embodiments, the four frames 1310 are stored separately in local memory 340 and loaded as separate contexts or separate operands into the active register file in sparse cell array 1300.

[0149] Four weight vectors 1320 are respectively loaded into four columns of MAC cells in the sparse cell array 1300. For example... Figure 13 As shown, the weights are shifted to map the sliding window pattern of frame 1310. A sparsity tensor can be used to shift the weights. The sparsity tensor can be processed by a sparsity module (e.g., sparsity module 660) in the sparse unit array 1300.

[0150] Methods for performing Fourier transform operations Figure 14 This is a flowchart illustrating a method 1400 for performing a Fourier transform operation according to various embodiments. Method 1400 can be performed by... Figure 3 The DNN accelerator 302 in the reference is executed. Figure 14 The flowchart shown describes method 1400, but many other methods for performing Fourier transform operations can be used alternatively. For example, the method can be changed. Figure 14 The execution order of the steps. As another example, some steps can be changed, deleted, or combined.

[0151] DNN accelerator 302 receives an input tensor 1410 representing an input signal for a DFT operation. In some embodiments, the input tensor is mapped onto a data processing unit as an activation tensor, the activation tensor comprising activations arranged in one or more rows and one or more columns. In some embodiments, DNN accelerator 302 receives the input tensor from a plurality of storage elements. Each of the plurality of storage elements corresponds to a different row in the input tensor and stores activations in that different row. In some embodiments, the input tensor is generated from the input signal. The total number of activations in the input tensor is greater than the total number of data elements in the input signal.

[0152] The DNN accelerator 302 transforms the Discrete Fourier Transform (DFT) operation 1420 into one or more two-dimensional matrix multiplications between the input tensor and the transformation matrix of the DFT operation. In some embodiments, the transformation matrix is ​​mapped onto the data processing unit as a weight tensor including weights. In some embodiments, the weight tensor is determined based on one or more rotation factors of the DFT operation. In some embodiments, some or all elements of the weight tensor are rotation factors of the DFT operation. In some embodiments, the weight tensor is determined offline by the DNN module 301.

[0153] DNN accelerator 302 performs 1430 MAC operations on the input tensor and the transformation matrix to generate an output tensor representing at least a portion of the discrete Fourier transform of the input tensor. In some embodiments, the data processing unit performs MAC operations by performing a first sequence of MAC operations and a second sequence of MAC operations. The MAC operations in the first sequence are performed on rows of the weight tensor and the input tensor. In some embodiments, the first sequence of MAC operations is performed by MAC units arranged by rows and columns. DNN accelerator 302 provides the activations in that row of the input tensor to a row of MAC units. DNN accelerator 302 splits the weight tensor into a weight vector. DNN accelerator 302 provides the weight vectors to MAC units in different columns.

[0154] The second sequence of MAC operations is performed on columns in the weight tensor and the input tensor. In some embodiments, the second sequence of MAC operations is performed by a MAC unit. In some embodiments, the DNN accelerator 302 transposes the input tensor to generate a transposed tensor. After transposing the input tensor, the DNN accelerator 302 performs the second sequence of MAC operations on the transposed tensor and the weight tensor. The activations in that column of the input tensor are arranged as rows in the transposed tensor. The second sequence of MAC operations is performed on that row in the weight tensor and the transposed tensor.

[0155] In some embodiments, the DNN accelerator 302 divides the weight tensor into weight vectors by splitting the columns of the weight tensor into a first weight vector and a second weight vector. Data elements in the first weight vector represent the real parts of the data elements in that column of the weight tensor. Data elements in the second weight vector represent the imaginary parts of the data elements in that column of the weight tensor.

[0156] In some embodiments, the total number of data elements in the output tensor is less than the total number of data elements in the Fourier transform of the input tensor. In some embodiments, the total number of data elements in the output tensor is equal to half the total number of data elements in the Fourier transform of the input tensor plus one. In some embodiments, the DNN accelerator 302 performs a third sequence of MAC operations to compute the output of the inverse DFT operation.

[0157] Example computing device Figure 15 This is a block diagram of an example computing device 1500 according to various embodiments. In some embodiments, the computing device 1500 may be used as at least a portion of a DNN system 300. Multiple components are... Figure 15 The components are shown as included in computing device 1500, but any one or more of these components may be omitted or copied as appropriate for an application. In some embodiments, some or all of the components included in computing device 1500 may be attached to one or more motherboards. In some embodiments, some or all of these components are manufactured onto a single system-on-a-chip (SoC) die. Additionally, in various embodiments, computing device 1500 may not include... Figure 15The computing device 1500 may include one or more components as shown, but may include interface circuitry modules for coupling to one or more components. For example, the computing device 1500 may not include display device 1506, but may include display device interface circuitry modules (e.g., connector and driver circuitry modules) to which display device 1506 may be coupled. In another set of examples, the computing device 1500 may not include audio input device 1518 or audio output device 1508, but may include audio input or output device interface circuitry modules (e.g., connector and support circuitry modules) to which audio input device 1518 or audio output device 1508 may be coupled.

[0158] Computing device 1500 may include processing device 1502 (e.g., one or more processing devices). Processing device 1502 processes electronic data from registers and / or memory to convert the electronic data into other electronic data that can be stored in registers and / or memory. Computing device 1500 may include memory 1504, which itself may include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high-bandwidth memory (HBM), flash memory, solid-state memory, and / or hard disk drive. In some embodiments, memory 1504 may include memory sharing a die with processing device 1502. In some embodiments, memory 1504 includes one or more non-transitory computer-readable media storing instructions executable to perform operations for performing Fourier transform operations (e.g., combining...). Figure 14 The described method 1400) or some operations performed by the DNN system 300. Instructions stored in one or more non-transitory computer-readable media can be executed by the processing device 1502.

[0159] In some embodiments, computing device 1500 may include communication chip 1512 (e.g., one or more communication chips). For example, communication chip 1512 may be configured to manage wireless communication for transmitting data to and from computing device 1500. The term "wireless" and its derivatives can be used to describe circuits, devices, systems, methods, technologies, communication channels, etc., which can transmit data by using modulated electromagnetic radiation through a non-solid medium. This term does not imply that the associated devices do not contain any wiring, although in some embodiments they may not.

[0160] The 1512 communication chip can implement any of the following wireless standards or protocols, including but not limited to: Institute of Electrical and Electronics Engineers (IEEE) standards, including Wi-Fi (IEEE 802.10 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendments), Long Term Evolution (LTE) projects along with any amendments, updates, and / or revisions (e.g., Advanced LTE project, Ultra Mobile Broadband (UMB) project (also known as "3GPP2"), etc.). IEEE 802.16 compliant Broadband Wireless Access (BWA) networks are commonly referred to as WiMAX networks, an acronym for Microwave Access Global Interoperability, and are a certification mark for products that have passed conformance and interoperability testing of the IEEE 802.16 standard. The 1512 communication chip can operate according to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE networks. Communication chip 1512 may operate according to Enhanced GSM Data Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). Communication chip 1512 may operate according to Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolved Data Optimization (EV-DO) and its derivative protocols, as well as any other wireless protocol designated as 3G, 4G, 5G, and above. In other embodiments, communication chip 1512 may operate according to other wireless protocols. Computing device 1500 may include antenna 1522 to facilitate wireless communication and / or receive other wireless communications (e.g., AM or FM radio transmissions).

[0161] In some embodiments, the communication chip 1512 can manage wired communications, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication chip 1512 may include multiple communication chips. For example, a first communication chip 1512 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, while a second communication chip 1512 may be dedicated to longer-range wireless communications such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, etc. In some embodiments, the first communication chip 1512 may be dedicated to wireless communications, while the second communication chip 1512 may be dedicated to wired communications.

[0162] The computing device 1500 may include a battery / power circuit module 1514. The battery / power circuit module 1514 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 1500 to a power source (e.g., AC line power) that is separate from the computing device 1500.

[0163] The computing device 1500 may include a display device 1506 (or a corresponding interface circuit module, as described above). For example, the display device 1506 may include any visual indicator, such as a head-up display, computer monitor, projector, touch screen display, liquid crystal display (LCD), light-emitting diode display, or flat panel display.

[0164] The computing device 1500 may include an audio output device 1508 (or a corresponding interface circuit module, as described above). For example, the audio output device 1508 may include any device that generates auditory indicators, such as a speaker, headphones, or in-ear headphones.

[0165] The computing device 1500 may include an audio input device 1518 (or a corresponding interface circuit, as described above). The audio input device 1518 may include any device that generates a signal representing sound, such as a microphone, microphone array, or digital instrument (e.g., an instrument with a Musical Instrument Digital Interface (MIDI) output).

[0166] The computing device 1500 may include a GPS device 1516 (or a corresponding interface circuit module, as described above). As is known in the art, the GPS device 1516 can communicate with a satellite-based system and can receive the location of the computing device 1500.

[0167] The computing device 1500 may include another output device 1510 (or a corresponding interface circuit, as described above). Examples of other output devices 1510 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0168] The computing device 1500 may include another input device 1520 (or a corresponding interface circuit module, as described above). Examples of the other input device 1520 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a barcode reader, a quick-response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0169] The computing device 1500 can have any desired form factor, such as a handheld or mobile computer system (e.g., a cellular phone, smartphone, mobile internet device, music player, tablet computer, laptop computer, netbook computer, ultrabook computer, personal digital assistant (PDA), ultra-mobile personal computer, etc.), a desktop computer system, a server or other networked computing component, a printer, scanner, monitor, set-top box, entertainment control unit, vehicle control unit, digital camera, digital video recorder, or wearable computer system. In some embodiments, the computing device 1500 can be any other electronic device that processes data.

[0170] Select Example The following paragraphs provide various examples of the embodiments disclosed herein.

[0171] Example 1 provides a method comprising: receiving an input tensor representing an input signal of a discrete Fourier transform operation; converting the discrete Fourier transform operation into one or more two-dimensional matrix multiplications between the input tensor and a transform matrix of the discrete Fourier transform operation; and performing a MAC operation on the input tensor and the transform matrix to generate an output tensor representing at least a portion of the discrete Fourier transform of the input tensor.

[0172] Example 2 provides the method of Example 1, wherein the input tensor is mapped onto a data processing unit as an activation tensor, the activation tensor comprising activations arranged in rows and columns, the transformation matrix is ​​mapped onto the data processing unit as a weight tensor including weights, and the data processing unit performs the MAC operation.

[0173] Example 3 provides the method of Example 2, wherein the data processing unit performs the MAC operation by: performing a first sequence of MAC operations on rows in the weight tensor and the input tensor; and performing a second sequence of MAC operations on columns in the weight tensor and the input tensor.

[0174] Example 4 provides the method of Example 3, wherein the second sequence of performing the MAC operation includes: transposing the input tensor to generate a transposed tensor; and after transposing the input tensor, performing the second sequence of the MAC operation on the transposed tensor and the weight tensor, wherein activations in the columns of the input tensor are arranged as rows in the transposed tensor, and the MAC operation in the second sequence is performed on the rows of the weight tensor and the transposed tensor.

[0175] Example 5 provides the method of Example 3 or 4, wherein the first sequence of the MAC operations is executed by MAC units in the data processing unit, the MAC units being arranged in rows and columns, and the execution of the first sequence of the MAC operations includes: providing activations in the rows of the input tensor to a row MAC unit; dividing the weight tensor into weight vectors; and providing the weight vectors to MAC units in different columns.

[0176] Example 6 provides the method of Example 5, wherein dividing the weight tensor into weight vectors includes: dividing the columns of the weight tensor into a first weight vector and a second weight vector, wherein the data elements in the first weight vector represent the real parts of the data elements in the columns of the weight tensor, and the data elements in the second weight vector represent the imaginary parts of the data elements in the columns of the weight tensor.

[0177] Example 7 provides a method from any of Examples 3-6, further comprising: performing a third sequence of MAC operations to compute the output of the inverse discrete Fourier transform operation.

[0178] Example 8 provides a method of any one of Examples 1-7, wherein the total number of data elements in the output tensor is less than the total number of data elements in the discrete Fourier transform of the input tensor.

[0179] Example 9 provides the method of Example 8, wherein the total number of data elements in the output tensor is equal to half plus one of the total number of data elements in the discrete Fourier transform of the input tensor.

[0180] Example 10 provides a method of any one of Examples 1-9, wherein the input tensor is generated from the input signal, and the total number of elements in the input tensor is greater than the total number of data elements in the input signal.

[0181] Example 11 provides one or more non-transitory computer-readable media storing instructions executable to perform operations including: receiving an input tensor representing an input signal of a discrete Fourier transform operation; converting the discrete Fourier transform operation into one or more two-dimensional matrix multiplications between the input tensor and a transform matrix of the discrete Fourier transform operation; and performing a MAC operation on the input tensor and the transform matrix to generate an output tensor representing at least a portion of the discrete Fourier transform of the input tensor.

[0182] Example 12 provides one or more non-transitory computer-readable media of Example 11, wherein the input tensor is mapped onto a data processing unit as an activation tensor, the activation tensor comprising activations arranged in rows and columns, the transformation matrix is ​​mapped onto the data processing unit as a weight tensor including weights, and the data processing unit performs the MAC operation.

[0183] Example 13 provides one or more non-transitory computer-readable media of Example 12, wherein the data processing unit performs the MAC operation by: performing a first sequence of MAC operations on rows in the weight tensor and the input tensor; and performing a second sequence of MAC operations on columns in the weight tensor and the input tensor.

[0184] Example 14 provides one or more non-transitory computer-readable media of Example 13, wherein a second sequence for performing the MAC operation comprises: transposing the input tensor to generate a transposed tensor; and, after transposing the input tensor, performing a second sequence of the MAC operation on the transposed tensor and the weight tensor, wherein activations in the columns of the input tensor are arranged as rows in the transposed tensor, and the MAC operation in the second sequence is performed on the rows of the weight tensor and the transposed tensor.

[0185] Example 15 provides one or more non-transitory computer-readable media of Example 13 or 14, wherein a first sequence of MAC operations is performed by MAC units in the data processing unit, the MAC units being arranged in rows and columns, and performing the first sequence of MAC operations includes: providing activations in the rows of the input tensor to a row MAC unit; dividing the weight tensor into weight vectors; and providing the weight vectors to MAC units in different columns.

[0186] Example 16 provides one or more non-transitory computer-readable media of Example 15, wherein dividing the weight tensor into weight vectors comprises: dividing columns of the weight tensor into a first weight vector and a second weight vector, wherein data elements in the first weight vector represent the real parts of data elements in the columns of the weight tensor, and data elements in the second weight vector represent the imaginary parts of data elements in the columns of the weight tensor.

[0187] Example 17 provides one or more non-transitory computer-readable media of any one of Examples 11-16, wherein the total number of data elements in the output tensor is less than the total number of data elements in the discrete Fourier transform of the input tensor.

[0188] Example 18 provides an apparatus comprising: a computer processor for executing computer program instructions; and a non-transitory computer-readable storage medium storing computer program instructions executable by the computer processor to perform operations including: receiving an input tensor representing an input signal of a discrete Fourier transform operation; converting the discrete Fourier transform operation into one or more two-dimensional matrix multiplications between the input tensor and a transform matrix of the discrete Fourier transform operation; and performing a MAC operation on the input tensor and the transform matrix to generate an output tensor representing at least a portion of the discrete Fourier transform of the input tensor.

[0189] Example 19 provides the apparatus of Example 18, wherein the input tensor is mapped onto a data processing unit as an activation tensor, the activation tensor comprising activations arranged in rows and columns, the transformation matrix is ​​mapped onto the data processing unit as a weight tensor including weights, and the data processing unit performs the MAC operation.

[0190] Example 20 provides an apparatus of Example 19, wherein the data processing unit performs the MAC operation by: performing a first sequence of MAC operations on rows in the weight tensor and the input tensor; and performing a second sequence of MAC operations on columns in the weight tensor and the input tensor.

[0191] Other options examples The following paragraphs provide various examples of the embodiments disclosed herein.

[0192] Example 1 provides a method comprising: receiving an input tensor representing an input signal of a DFT operation, the input tensor including activations arranged in one or more rows and one or more columns; receiving a weight tensor determined based on one or more rotation factors of the DFT operation; performing a first multiply-accumulate (MAC) operation sequence on rows of the weight tensor and the input tensor; performing a second sequence of MAC operations on columns of the weight tensor and the input tensor; and generating an output tensor representing at least a portion of the DFT of the input tensor.

[0193] Example 2 provides the method of Example 1, wherein the second sequence of performing the MAC operation includes: transposing the input tensor to generate a transposed tensor; and after transposing the input tensor, performing the second sequence of the MAC operation on the transposed tensor and the weight tensor, wherein activations in the columns of the input tensor are arranged as rows in the transposed tensor, and the MAC operation in the second sequence is performed on the rows of the weight tensor and the transposed tensor.

[0194] Example 3 provides the method of Example 1 or 2, wherein a first sequence of MAC operations is performed by MAC units arranged in rows and columns, and performing the first sequence of MAC operations includes: providing activations in the rows of the input tensor to a row MAC unit; dividing the weight tensor into weight vectors; and providing the weight vectors to MAC units in different columns.

[0195] Example 4 provides the method of Example 3, wherein the second sequence of the MAC operation is executed by the MAC unit.

[0196] Example 5 provides the method of Example 3 or 4, wherein dividing the weight tensor into weight vectors includes: dividing the columns of the weight tensor into a first weight vector and a second weight vector, wherein the data elements in the first weight vector represent the real parts of the data elements in the columns of the weight tensor, and the data elements in the second weight vector represent the imaginary parts of the data elements in the columns of the weight tensor.

[0197] Example 6 provides a method of any one of Examples 1-5, further including: performing a third sequence of MAC operations to compute the output of the inverse DFT operation.

[0198] Example 7 provides a method of any one of Examples 1-6, wherein the total number of data elements in the output tensor is less than the total number of data elements in the DFT of the input tensor.

[0199] Example 8 provides the method of Example 7, wherein the total number of data elements in the output tensor is equal to half plus one of the total number of data elements in the DFT of the input tensor.

[0200] Example 9 provides a method of any one of Examples 1-8, wherein receiving an input tensor includes receiving the input tensor from a plurality of storage elements, each of the plurality of storage elements storing a different row corresponding to the input tensor, and storing activations in the different rows.

[0201] Example 10 provides a method of any one of Examples 1-9, wherein the input tensor is generated from the input signal, and the total number of activated elements in the input tensor is greater than the total number of data elements in the input signal.

[0202] Example 11 provides one or more non-transitory computer-readable media storing instructions executable to perform operations, the operations including: receiving an input tensor representing an input signal of a DFT operation, the input tensor including activations arranged in one or more rows and one or more columns; receiving a weight tensor determined based on one or more rotation factors of the DFT operation; performing a first multiply-accumulate (MAC) operation sequence on rows of the weight tensor and the input tensor; performing a second sequence of MAC operations on columns of the weight tensor and the input tensor; and generating an output tensor representing at least a portion of the DFT of the input tensor.

[0203] Example 12 provides one or more non-transitory computer-readable media of Example 11, wherein a second sequence for performing the MAC operation comprises: transposing the input tensor to generate a transposed tensor; and, after transposing the input tensor, performing a second sequence of the MAC operation on the transposed tensor and the weight tensor, wherein activations in the columns of the input tensor are arranged as rows in the transposed tensor, and the MAC operation in the second sequence is performed on the rows of the weight tensor and the transposed tensor.

[0204] Example 13 provides one or more non-transitory computer-readable media of Example 11 or 12, wherein a first sequence of MAC operations is performed by MAC units arranged in rows and columns, and performing the first sequence of MAC operations includes: providing activations in the rows of the input tensor to a row MAC unit; dividing the weight tensor into weight vectors; and providing the weight vectors to MAC units in different columns.

[0205] Example 14 provides one or more non-transitory computer-readable media of any one of Examples 11-13, wherein the operation further includes: performing a third sequence of MAC operations to compute the output of an inverse DFT operation.

[0206] Example 15 provides one or more non-transitory computer-readable media of any one of Examples 11-14, wherein the total number of data elements in the output tensor is less than the total number of data elements in the DFT of the input tensor.

[0207] Example 16 provides one or more non-transitory computer-readable media of any one of Examples 11-15, wherein receiving an input tensor includes receiving the input tensor from a plurality of storage elements, each of the plurality of storage elements storing a different row corresponding to the input tensor and storing activations in the different rows.

[0208] Example 17 provides one or more non-transitory computer-readable media of any one of Examples 11-16, wherein the input tensor is generated from the input signal, and the total number of activated elements in the input tensor is greater than the total number of data elements in the input signal.

[0209] Example 18 provides an apparatus comprising: a computer processor for executing computer program instructions; and a non-transitory computer-readable storage memory storing computer program instructions executable by the computer processor to perform operations including: receiving an input tensor representing an input signal of a DFT operation, the input tensor including activations arranged in one or more rows and one or more columns; receiving a weight tensor determined based on one or more rotation factors of the DFT operation; performing a first multiply-accumulate (MAC) operation sequence on rows of the weight tensor and the input tensor; performing a second sequence of MAC operations on columns of the weight tensor and the input tensor; and generating an output tensor representing at least a portion of a DFT of the input tensor. Example 19 provides an apparatus of Example 18, wherein the second sequence for performing the MAC operation includes: transposing the input tensor to generate a transposed tensor; and, after transposing the input tensor, performing the second sequence of the MAC operation on the transposed tensor and the weight tensor, wherein activations in the columns of the input tensor are arranged as rows in the transposed tensor, and the MAC operation in the second sequence is performed on the rows of the weight tensor and the transposed tensor.

[0210] Example 20 provides an apparatus of Example 18 or 19, wherein a first sequence of MAC operations is performed by MAC units arranged in rows and columns, and performing the first sequence of MAC operations includes: providing activations in the rows of the input tensor to a row MAC unit; dividing the weight tensor into weight vectors; and providing the weight vectors to MAC units in different columns.

[0211] The above description of the illustrated embodiments of this disclosure, including the content described in the abstract, is not intended to be exhaustive or to limit this disclosure to the precise forms disclosed. While specific implementations and examples of this disclosure have been described herein for illustrative purposes, various equivalent modifications are possible within the scope of this disclosure, as will be recognized by those skilled in the art. These modifications can be made to this disclosure based on the detailed description above.

Claims

1. A method comprising: Receives an input tensor representing the input signal of the Discrete Fourier Transform operation; The Discrete Fourier Transform operation is transformed into one or more two-dimensional matrix multiplications between the input tensor and the transformation matrix of the Discrete Fourier Transform operation; as well as A multiply-accumulate (MAC) operation is performed on the input tensor and the transformation matrix to generate an output tensor representing at least a portion of the discrete Fourier transform of the input tensor.

2. The method as described in claim 1, wherein, The input tensor is mapped onto the data processing unit as an activation tensor, the activation tensor including activations arranged by rows and columns, the transformation matrix is ​​mapped onto the data processing unit as a weight tensor including weights, and the data processing unit performs the MAC operation.

3. The method of claim 2, wherein, The data processing unit performs the MAC operation in the following manner: The first sequence of MAC operations is performed on rows of the weight tensor and the input tensor. as well as The second sequence of MAC operations is performed on the weight tensor and the columns in the input tensor.

4. The method of claim 3, wherein, The second sequence for performing the MAC operation includes: Transpose the input tensor to generate a transposed tensor; and After transposing the input tensor, the second sequence of the MAC operation is performed on the transposed tensor and the weight tensor. The activations in the columns of the input tensor are arranged as rows in the transposed tensor, and the MAC operation in the second sequence is performed on the weight tensor and the rows in the transposed tensor.

5. The method of claim 3, wherein, The first sequence of MAC operations is executed by MAC units in the data processing unit, the MAC units being arranged in rows and columns, and the first sequence of MAC operations includes: Provide the activation in the row of the input tensor to a row MAC unit; The weight tensor is divided into weight vectors; and The weight vector is provided to the MAC cells of different columns.

6. The method of claim 5, wherein, Dividing the weight tensor into the weight vector includes: The columns in the weight tensor are divided into a first weight vector and a second weight vector. Wherein, the data elements in the first weight vector represent the real part of the data elements in the column of the weight tensor, and the data elements in the second weight vector represent the imaginary part of the data elements in the column of the weight tensor.

7. The method of claim 3, further comprising: The third sequence of MAC operations is executed to compute the output of the inverse discrete Fourier transform operation.

8. The method of claim 1, wherein, The total number of data elements in the output tensor is less than the total number of data elements in the discrete Fourier transform of the input tensor.

9. The method of claim 8, wherein, The total number of data elements in the output tensor is equal to half plus one of the total number of data elements in the discrete Fourier transform of the input tensor.

10. The method of claim 1, wherein, The input tensor is generated from the input signal, and the total number of elements in the input tensor is greater than the total number of data elements in the input signal.

11. One or more non-transitory computer-readable media storing instructions executable to perform operations, said operations including: Receives an input tensor representing the input signal of the Discrete Fourier Transform operation; The Discrete Fourier Transform operation is transformed into one or more two-dimensional matrix multiplications between the input tensor and the transformation matrix of the Discrete Fourier Transform operation; as well as A multiply-accumulate (MAC) operation is performed on the input tensor and the transformation matrix to generate an output tensor representing at least a portion of the discrete Fourier transform of the input tensor.

12. One or more non-transitory computer-readable media as claimed in claim 11, wherein, The input tensor is mapped onto the data processing unit as an activation tensor, the activation tensor including activations arranged by rows and columns, the transformation matrix is ​​mapped onto the data processing unit as a weight tensor including weights, and the data processing unit performs the MAC operation.

13. One or more non-transitory computer-readable media as claimed in claim 12, wherein, The data processing unit performs the MAC operation in the following manner: The first sequence of MAC operations is performed on rows of the weight tensor and the input tensor. as well as The second sequence of MAC operations is performed on the weight tensor and the columns in the input tensor.

14. One or more non-transitory computer-readable media as claimed in claim 13, wherein, The second sequence for performing the MAC operation includes: Transpose the input tensor to generate a transposed tensor; and After transposing the input tensor, the second sequence of the MAC operation is performed on the transposed tensor and the weight tensor. The activations in the columns of the input tensor are arranged as rows in the transposed tensor, and the MAC operation in the second sequence is performed on the weight tensor and the rows in the transposed tensor.

15. One or more non-transitory computer-readable media as claimed in claim 13, wherein, The first sequence of MAC operations is executed by MAC units in the data processing unit, the MAC units being arranged in rows and columns, and the first sequence of MAC operations includes: Provide the activation in the row of the input tensor to a row MAC unit; The weight tensor is divided into weight vectors; and The weight vector is provided to the MAC cells of different columns.

16. One or more non-transitory computer-readable media as claimed in claim 15, wherein, Dividing the weight tensor into the weight vector includes: The columns in the weight tensor are divided into a first weight vector and a second weight vector. Wherein, the data elements in the first weight vector represent the real part of the data elements in the column of the weight tensor, and the data elements in the second weight vector represent the imaginary part of the data elements in the column of the weight tensor.

17. One or more non-transitory computer-readable media as claimed in claim 11, wherein, The total number of data elements in the output tensor is less than the total number of data elements in the discrete Fourier transform of the input tensor.

18. An apparatus comprising: A computer processor is used to execute computer program instructions; as well as A non-transitory computer-readable storage medium storing computer program instructions executable by the computer processor to perform operations, said operations including: Receives an input tensor representing the input signal of the Discrete Fourier Transform operation; The Discrete Fourier Transform operation is transformed into one or more two-dimensional matrix multiplications between the input tensor and the transformation matrix of the Discrete Fourier Transform operation; and A multiply-accumulate (MAC) operation is performed on the input tensor and the transformation matrix to generate an output tensor representing at least a portion of the discrete Fourier transform of the input tensor.

19. The apparatus of claim 18, wherein, The input tensor is mapped onto the data processing unit as an activation tensor, the activation tensor including activations arranged by rows and columns, the transformation matrix is ​​mapped onto the data processing unit as a weight tensor including weights, and the data processing unit performs the MAC operation.

20. The apparatus of claim 19, wherein, The data processing unit performs the MAC operation in the following manner: The first sequence of MAC operations is performed on rows of the weight tensor and the input tensor. as well as The second sequence of MAC operations is performed on the weight tensor and the columns in the input tensor.

Images