Pipelined winograd based accelerator architecture for convolutional neural networks

Abstract

A method includes transforming, at a processing engine (PE), pixel data into a time variant pixel representation based on receiving the pixel data at the PE. The method also includes transforming, at the PE, kernel data into a time variant kernel representation based on receiving the kernel data at the PE. The method further includes generating, at the PE, transformed pixel data based on the time variant pixel representation. The method still further includes generating, at the PE, transformed kernel data based on the time variant kernel representation. The method also includes generating, at the PE, a transformed output of the pixel data based on an element-wise product of the transformed pixel data and the transformed kernel data.

Description

Qualcomm Ref. No.2403743WO PIPELINED WINOGRAD BASED ACCELERATOR ARCHITECTURE FOR CONVOLUTIONAL NEURAL NETWORKS FIELD OF THE DISCLOSURE

[0001] The present application claims priority to India Patent Application No.202441034042, filed on April 29, 2024, and titled “PIPELINED WINOGRAD BASED ACCELERATOR ARCHITECTURE FOR CONVOLUTIONAL NEURAL NETWORKS,” the disclosure of which is expressly incorporated by reference herein in its entirety. BACKGROUND Field

[0002] Aspects of the present disclosure generally relate to computer hardware forneural networks, and more specifically to a pipelined Winograd based accelerator architecture for convolutional neural networks.

[0003] Artificial neural networks may comprise interconnected groups of artificialneurons (e.g., neuron models). The artificial neural network (ANN) may be a computational device or be represented as a method to be performed by a computational device. Convolutional neural networks (CNNs) are a type of feed-forward ANN. Convolutional neural networks may include collections of neurons that each have a receptive field and that collectively tile an input space. Convolutional neural networks, such as deep convolutional neural networks (DCNs), have numerous applications. In particular, these neural network architectures are used in various technologies, such as image recognition, speech recognition, acoustic scene classification, keyword spotting, autonomous driving, and other classification tasks.

[0004] CNNs may be trained and implemented on software platforms that use high-performance central processing units (CPUs) and graphics processing units (GPUs). Despite their effectiveness, software implementations of CNNs have inherent limitations. Therefore, dedicated hardware architectures or accelerators may be developed for CNNs. In some cases, these hardware architectures or accelerators may be specified for embedded systems. Application-specific integrated circuit (ASIC) andSeyfarth Ref. No. 72178-0067611 310721357v.1Qualcomm Ref. No.2403743WO field programmable gate array (FPGA) implementations are examples of hardware architectures. SUMMARY

[0005] In aspects of the present disclosure, a method includes transforming, at aprocessing engine (PE), pixel data into a time variant pixel representation based on receiving the pixel data at the PE. The method further includes transforming, at the PE, kernel data into a time variant kernel representation based on receiving the kernel data at the PE. The method also includes generating, at the PE, transformed pixel data based on the time variant pixel representation. The method further includes generating, at the PE, transformed kernel data based on the time variant kernel representation. The method still further includes generating, at the PE, a transformed output of the pixel data based on an element-wise product of the transformed pixel data and the transformed kernel data.

[0006] Other aspects of the present disclosure are directed to an apparatus. Theapparatus includes means for transforming, at a PE, pixel data into a time variant pixel representation based on receiving the pixel data at the PE. The apparatus further includes means for transforming, at the PE, kernel data into a time variant kernel representation based on receiving the kernel data at the PE. The apparatus further includes means for generating, at the PE, transformed pixel data based on the time variant pixel representation. The apparatus further includes means for generating, at the PE, transformed kernel data based on the time variant kernel representation. The apparatus further includes means for generating, at the PE, a transformed output of the pixel data based on an element-wise product of the transformed pixel data and the transformed kernel data.

[0007] In other aspects of the present disclosure, a non-transitory computer-readablemedium with program code recorded thereon is disclosed. The program code is executed by a processor and includes program code to transform, at a PE, pixel data into a time variant pixel representation based on receiving the pixel data at the PE. The program code also includes program code to transform, at the PE, kernel data into a time variant kernel representation based on receiving the kernel data at the PE. The program code also includes program code to generate, at the PE, transformed pixel data based onSeyfarth Ref. No. 72178-0067612 310721357v.1Qualcomm Ref. No.2403743WO the time variant pixel representation. The program code further includes program code to generate, at the PE, transformed kernel data based on the time variant kernel representation. The program code still further includes program code to generate, at the PE, a transformed output of the pixel data based on an element-wise product of the transformed pixel data and the transformed kernel data.

[0008] Other aspects of the present disclosure are directed to an apparatus includingone or more processors, and one or more memories coupled with the one or more processors and storing processor-executable code that, when executed by the one or more processors, is configured to cause the apparatus transform, at a PE, pixel data into a time variant pixel representation based on receiving the pixel data at the PE. Execution of the processor-executable code also causes the apparatus to transform, at the PE, kernel data into a time variant kernel representation based on receiving the kernel data at the PE. Execution of the processor-executable code further causes the apparatus to generate, at the PE, transformed pixel data based on the time variant pixel representation. Execution of the processor-executable code still further causes the apparatus to generate, at the PE, transformed kernel data based on the time variant kernel representation. Execution of the processor-executable code also causes the apparatus to generate, at the PE, a transformed output of the pixel data based on an element-wise product of the transformed pixel data and the transformed kernel data.

[0009] Additional features and advantages of the disclosure will be described below.It should be appreciated by those skilled in the art that this disclosure may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. It should also be realized by those skilled in the art that such equivalent constructions do not depart from the teachings of the disclosure as set forth in the appended claims. The novel features, which are believed to be characteristic of the disclosure, both as to its organization and method of operation, together with further objects and advantages, will be better understood from the following description when considered in connection with the accompanying figures. It is to be expressly understood, however, that each of the figures is provided for the purpose of illustration and description only and is not intended as a definition of the limits of the present disclosure.Seyfarth Ref. No. 72178-0067613 310721357v.1Qualcomm Ref. No.2403743WO BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The features, nature, and advantages of the present disclosure will becomemore apparent from the detailed description set forth below when taken in conjunction with the drawings in which like reference characters identify correspondingly throughout.

[0011] FIGURE 1 illustrates an example implementation of a neural network usinga system-on-a-chip (SOC), including a general-purpose processor in accordance with certain aspects of the present disclosure.

[0012] FIGURES 2A, 2B, and 2C are diagrams illustrating a neural network inaccordance with aspects of the present disclosure.

[0013] FIGURE 2D is a diagram illustrating an exemplary deep convolutionalnetwork (DCN) in accordance with aspects of the present disclosure.

[0014] FIGURE 3 is a block diagram illustrating an exemplary deep convolutionalnetwork (DCN) in accordance with aspects of the present disclosure.

[0015] FIGURE 4 is a block diagram illustrating an exemplary software architecturethat may modularize artificial intelligence (AI) functions, in accordance with aspects of the present disclosure.

[0016] FIGURE 5 is a block diagram illustrating an example of a Winogradconvolution processing engine, in accordance with various aspects of the present disclosure.

[0017] FIGURE 6 is a block diagram illustrating an example of a convolutionalneural network (CNN) model, in accordance with various aspects of the present disclosure.

[0018] FIGURE 7A is a block diagram illustrating an example of block randomaccess memory (RAM) data organization for pixels, in accordance with various aspects of the present disclosure.Seyfarth Ref. No. 72178-0067614 310721357v.1Qualcomm Ref. No.2403743WO

[0019] FIGURE 7B is a block diagram illustrating an example of block RAM dataorganization for kernel data, in accordance with various aspects of the present disclosure.

[0020] FIGURE 8 is a block diagram illustrating an example of read addresstranslation tables (RATTs) and write address translation tables (WATTs), in accordance with various aspects of the present disclosure.

[0021] FIGURE 9 is a flow diagram illustrating an example of a process for timevariant Winograd based convolution, in accordance with various aspects of the present disclosure. DETAILED DESCRIPTION

[0022] The detailed description set forth below, in connection with the appendeddrawings, is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described may be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form in order to avoid obscuring such concepts.

[0023] Based on the teachings, one skilled in the art should appreciate that the scopeof the disclosure is intended to cover any aspect of the disclosure, whether implemented independently of or combined with any other aspect of the disclosure. For example, an apparatus may be implemented or a method may be practiced using any number of the aspects set forth. In addition, the scope of the disclosure is intended to cover such an apparatus or method practiced using other structure, functionality, or structure and functionality in addition to or other than the various aspects of the disclosure set forth. It should be understood that any aspect of the disclosure disclosed may be embodied by one or more elements of a claim.

[0024] The word “exemplary” is used to mean “serving as an example, instance, orillustration.” Any aspect described as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.Seyfarth Ref. No. 72178-0067615 310721357v.1Qualcomm Ref. No.2403743WO

[0025] Although particular aspects are described, many variations and permutationsof these aspects fall within the scope of the disclosure. Although some benefits and advantages of the preferred aspects are mentioned, the scope of the disclosure is not intended to be limited to particular benefits, uses, or objectives. Rather, aspects of the disclosure are intended to be broadly applicable to different technologies, system configurations, networks, and protocols, some of which are illustrated by way of examples in the figures and in the following description of the preferred aspects. The detailed description and drawings are merely illustrative of the disclosure rather than limiting, the scope of the disclosure being defined by the appended claims and equivalents thereof.

[0026] As discussed, convolutional neural networks (CNNs) may be trained andimplemented on software platforms that use high-performance central processing units (CPUs) and graphics processing units (GPUs). Because of the increasing complexity of hardware architectures and growing performance specifications, CPUs and GPUs may not be suitable for deploying inference engines. Therefore, dedicated hardware architectures or accelerators may be developed for CNNs. In some cases, these hardware architectures or accelerators may be specified for embedded systems. Application-specific integrated circuit (ASIC) and field programmable gate array (FPGA) implementations are examples of hardware architectures that satisfy performance specifications for inference operations.

[0027] Despite the rapid evolution of CNN models, their basic operations, such asconvolutions, remain unchanged. Convolution operations significantly contribute to computational complexity and consume a considerable amount of resources. Consequently, several techniques have been outlined to develop an efficient hardware architecture for CNN models. Such proposals are directed to reducing an amount of resources used by CNN models during inference. One such proposal is a pipelined accelerator that incorporates the Winograd minimal filtering function.

[0028] The Winograd minimal filtering function, denoted as ^^(^^, ^^), convolves animage of size ^^ + ^^ − 1 with a kernel of size ^^ to produce an output of dimension^^ × ^^. Specifically, the function applies a transformation on input pixels and kernels(e.g., filters), followed by a point-wise multiplication on the transformed matrix (e.g., transformed pixels and kernels). That is, the input pixels and kernels are transformed toSeyfarth Ref. No. 72178-0067616 310721357v.1Qualcomm Ref. No.2403743WO a new domain, where the convolution is reduced to simpler element-wise multiplications. An inverse transformation is then performed to obtain the ^^x^^ output pixels. These transformation operations result in some overhead due to the matrix multiplication.

[0029] Various aspects of the present disclosure are directed to reducing theoverhead associated with hardware specified for CNNs. In some examples, the overhead is a result of the transformation operations associated with the Winograd minimal filtering function. In some examples, to address this overhead, the transformed pixels, kernels, and outputs, which are represented as matrices within the Winograd framework, may be converted into time variant vectors.

[0030] Particular aspects of the subject matter described in this disclosure can beimplemented to realize one or more of the following potential advantages. In some examples, the described techniques of converting matrices to time variant vectors for use in a Winograd based accelerator architecture may simplify transformations, thereby reducing the overhead of the Winograd minimal filtering function (e.g., Winograd function). Reducing the overhead of the Winograd function may reduce overall resource use, such as, but not limited to, memory use, processor use, and / or power consumption. Reducing the overhead may also improve inference speed.

[0031] FIGURE 1 illustrates an example implementation of a system-on-a-chip(SOC) 100, which may include a central processing unit (CPU) 102 or a multi-core CPU configured for a Winograd based accelerator architecture that converts matrices to time variant vectors. The Winograd based accelerator architecture may be used in a CNN. Variables (e.g., neural signals and synaptic weights), system parameters associated with a computational device (e.g., neural network with weights), delays, frequency bin information, and task information may be stored in a memory block associated with a neural processing unit (NPU) 108, in a memory block associated with a CPU 102, in a memory block associated with a graphics processing unit (GPU) 104, in a memory block associated with a digital signal processor (DSP) 106, in a memory block 118, or may be distributed across multiple blocks. Instructions executed at the CPU 102 may be loaded from a program memory associated with the CPU 102 or may be loaded from a memory block 118.Seyfarth Ref. No. 72178-0067617 310721357v.1Qualcomm Ref. No.2403743WO

[0032] The SOC 100 may also include additional processing blocks tailored tospecific functions, such as a GPU 104, a DSP 106, a connectivity block 110, which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, and the like, and a multimedia processor 112 that may, for example, detect and recognize gestures. In one implementation, the NPU 108 is implemented in the CPU 102, DSP 106, and / or GPU 104. The SOC 100 may also include a sensor processor 114, image signal processors (ISPs) 116, and / or navigation module 120, which may include a global positioning system.

[0033] The SOC 100 may be based on an ARM, RISC-V (RISC-five), or anyreduced instruction set computing (RISC) architecture. In aspects of the present disclosure, the instructions loaded into the general-purpose processor 102 may include code to transform, at a PE, pixel data into a time variant pixel representation based on receiving the pixel data at the PE; code to transform, at the PE, kernel data into a time variant kernel representation based on receiving the kernel data at the PE; code to generate, at the PE, transformed pixel data based on the time variant pixel representation; code to generate, at the PE, transformed kernel data based on the time variant kernel representation; and code to generate, at the PE, a transformed output of the pixel data based on an element-wise product of the transformed pixel data and the transformed kernel data.

[0034] In some aspects, the general-purpose processor 102 may include means fortransforming, at a PE, pixel data into a time variant pixel representation based on receiving the pixel data at the PE; means for transforming, at the PE, kernel data into a time variant kernel representation based on receiving the kernel data at the PE; means for generating, at the PE, transformed pixel data based on the time variant pixel representation; means for generating, at the PE, transformed kernel data based on the time variant kernel representation; and means for generating, at the PE, a transformed output of the pixel data based on an element-wise product of the transformed pixel data and the transformed kernel data.

[0035] Deep learning architectures may perform an object recognition task bylearning to represent inputs at successively higher levels of abstraction in each layer, thereby building up a useful feature representation of the input data. In this way, deepSeyfarth Ref. No. 72178-0067618 310721357v.1Qualcomm Ref. No.2403743WO learning addresses a major bottleneck of traditional machine learning. Prior to the advent of deep learning, a machine learning approach to an object recognition problem may have relied heavily on human engineered features, perhaps in combination with a shallow classifier. A shallow classifier may be a two-class linear classifier, for example, in which a weighted sum of the feature vector components may be compared with a threshold to predict to which class the input belongs. Human engineered features may be templates or kernels tailored to a specific problem domain by engineers with domain expertise. Deep learning architectures, in contrast, may learn to represent features that are similar to what a human engineer might design, but through training. Furthermore, a deep network may learn to represent and recognize new types of features that a human might not have considered.

[0036] A deep learning architecture may learn a hierarchy of features. If presentedwith visual data, for example, the first layer may learn to recognize relatively simple features, such as edges, in the input stream. In another example, if presented with auditory data, the first layer may learn to recognize spectral power in specific frequencies. The second layer, taking the output of the first layer as input, may learn to recognize combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For instance, higher layers may learn to represent complex shapes in visual data or words in auditory data. Still higher layers may learn to recognize common visual objects or spoken phrases.

[0037] Deep learning architectures may perform especially well when applied toproblems that have a natural hierarchical structure. For example, the classification of motorized vehicles may benefit from first learning to recognize wheels, windshields, and other features. These features may be combined at higher layers in different ways to recognize cars, trucks, and airplanes.

[0038] Neural networks may be designed with a variety of connectivity patterns. Infeed-forward networks, information is passed from lower to higher layers, with each neuron in a given layer communicating to neurons in higher layers. A hierarchical representation may be built up in successive layers of a feed-forward network, as described above. Neural networks may also have recurrent or feedback (also called top- down) connections. In a recurrent connection, the output from a neuron in a given layer may be communicated to another neuron in the same layer. A recurrent architectureSeyfarth Ref. No. 72178-0067619 310721357v.1Qualcomm Ref. No.2403743WO may be helpful in recognizing patterns that span more than one of the input data chunks that are delivered to the neural network in a sequence. A connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. A network with many feedback connections may be helpful when the recognition of a high-level concept may aid in discriminating the particular low-level features of an input.

[0039] The connections between layers of a neural network may be fully connectedor locally connected. FIGURE 2A illustrates an example of a fully connected neural network 202. In a fully connected neural network 202, a neuron in a first layer may communicate its output to every neuron in a second layer, so that each neuron in the second layer will receive input from every neuron in the first layer. FIGURE 2B illustrates an example of a locally connected neural network 204. In a locally connected neural network 204, a neuron in a first layer may be connected to a limited number of neurons in the second layer. More generally, a locally connected layer of the locally connected neural network 204 may be configured so that each neuron in a layer will have the same or a similar connectivity pattern, but with connections strengths that may have different values (e.g., 210, 212, 214, and 216). The locally connected connectivity pattern may give rise to spatially distinct receptive fields in a higher layer because the higher layer neurons in a given region may receive inputs that are tuned through training to the properties of a restricted portion of the total input to the network.

[0040] One example of a locally connected neural network is a convolutional neuralnetwork. FIGURE 2C illustrates an example of a convolutional neural network 206. The convolutional neural network 206 may be configured such that the connection strengths associated with the inputs for each neuron in the second layer are shared (e.g., 208). Convolutional neural networks may be well suited to problems in which the spatial location of inputs is meaningful.

[0041] One type of convolutional neural network is a deep convolutional network(DCN). FIGURE 2D illustrates a detailed example of a DCN 200 designed to recognize visual features from an image 226 input from an image capturing device 230, such as a car-mounted camera. The DCN 200 of the current example may be trained to identify traffic signs and a number provided on the traffic sign. Of course, the DCN 200 may be trained for other tasks, such as identifying lane markings or identifying traffic lights.Seyfarth Ref. No. 72178-00676110 310721357v.1Qualcomm Ref. No.2403743WO

[0042] The DCN 200 may be trained with supervised learning. During training, theDCN 200 may be presented with an image, such as the image 226 of a speed limit sign, and a forward pass may then be computed to produce an output 222. The DCN 200 may include a feature extraction section and a classification section. Upon receiving the image 226, a convolutional layer 232 may apply convolutional kernels (not shown) to the image 226 to generate a first set of feature maps 218. As an example, the convolutional kernel for the convolutional layer 232 may be a 5x5 kernel that generates 28x28 feature maps. In the present example, because four different feature maps are generated in the first set of feature maps 218, four different convolutional kernels were applied to the image 226 at the convolutional layer 232. The convolutional kernels may also be referred to as filters or convolutional filters.

[0043] The first set of feature maps 218 may be subsampled by a max pooling layer(not shown) to generate a second set of feature maps 220. The max pooling layer reduces the size of the first set of feature maps 218. That is, a size of the second set of feature maps 220, such as 14x14, is less than the size of the first set of feature maps 218, such as 28x28. The reduced size provides similar information to a subsequent layer while reducing memory consumption. The second set of feature maps 220 may be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0044] In the example of FIGURE 2D, the second set of feature maps 220 isconvolved to generate a first feature vector 224. Furthermore, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature of the second feature vector 228 may include a number that corresponds to a possible feature of the image 226, such as “sign,” “60,” and “100.” A softmax function (not shown) may convert the numbers in the second feature vector 228 to a probability. As such, an output 222 of the DCN 200 may be a probability of the image 226 including one or more features.

[0045] In the present example, the probabilities in the output 222 for “sign” and“60” are higher than the probabilities of the others of the output 222, such as “30,” “40,” “50,” “70,” “80,” “90,” and “100”. Before training, the output 222 produced by the DCN 200 may likely be incorrect. Thus, an error may be calculated between the output 222 and a target output. The target output is the ground truth of the image 226 (e.g.,Seyfarth Ref. No. 72178-00676111 310721357v.1Qualcomm Ref. No.2403743WO “sign” and “60”). The weights of the DCN 200 may then be adjusted so the output 222 of the DCN 200 is more closely aligned with the target output.

[0046] To adjust the weights, a learning algorithm may compute a gradient vectorfor the weights. The gradient may indicate an amount that an error would increase or decrease if the weight were adjusted. At the top layer, the gradient may correspond directly to the value of a weight connecting an activated neuron in the penultimate layer and a neuron in the output layer. In lower layers, the gradient may depend on the value of the weights and on the computed error gradients of the higher layers. The weights may then be adjusted to reduce the error. This manner of adjusting the weights may be referred to as “back propagation” as it involves a “backward pass” through the neural network.

[0047] In practice, the error gradient of weights may be calculated over a smallnumber of examples, so that the calculated gradient approximates the true error gradient. This approximation method may be referred to as stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate of the entire system has stopped decreasing or until the error rate has reached a target level. After learning, the DCN 200 may be presented with new images (e.g., the speed limit sign of the image 226) and a forward pass through the DCN 200 may yield an output 222 that may be considered an inference or a prediction of the DCN 200.

[0048] Deep belief networks (DBNs) are probabilistic models comprising multiplelayers of hidden nodes. DBNs may be used to extract a hierarchical representation of training data sets. A DBN may be obtained by stacking up layers of Restricted Boltzmann Machines (RBMs). An RBM is a type of artificial neural network that can learn a probability distribution over a set of inputs. Because RBMs can learn a probability distribution in the absence of information about the class to which each input should be categorized, RBMs are often used in unsupervised learning. Using a hybrid unsupervised and supervised paradigm, the bottom RBMs of a DBN may be trained in an unsupervised manner and may serve as feature extractors, and the top RBM may be trained in a supervised manner (on a joint distribution of inputs from the previous layer and target classes) and may serve as a classifier.Seyfarth Ref. No. 72178-00676112 310721357v.1Qualcomm Ref. No.2403743WO

[0049] DCNs are networks of convolutional networks, configured with additionalpooling and normalization layers. DCNs have achieved state-of-the-art performance on many tasks. DCNs can be trained using supervised learning in which both the input and output targets are known for many exemplars and are used to modify the weights of the network by use of gradient descent methods.

[0050] DCNs may be feed-forward networks. In addition, as described above, theconnections from a neuron in a first layer of a DCN to a group of neurons in the next higher layer are shared across the neurons in the first layer. The feed-forward and shared connections of DCNs may be exploited for fast processing. The computational burden of a DCN may be much less, for example, than that of a similarly sized neural network that comprises recurrent or feedback connections.

[0051] The processing of each layer of a convolutional network may be considereda spatially variant template or basis projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, then the convolutional network trained on that input may be considered three-dimensional, with two spatial dimensions along the axes of the image and a third dimension capturing color information. The outputs of the convolutional connections may be considered to form a feature map in the subsequent layer, with each element of the feature map (e.g., 220) receiving input from a range of neurons in the previous layer (e.g., feature maps 218) and from each of the multiple channels. The values in the feature map may be further processed with a non-linearity, such as a rectification, max(0, x). Values from adjacent neurons may be further pooled, which corresponds to down sampling, and may provide additional local invariance and dimensionality reduction. Normalization, which corresponds to whitening, may also be applied through lateral inhibition between neurons in the feature map.

[0052] FIGURE 3 is a block diagram illustrating a DCN 350. The DCN 350 mayinclude multiple different types of layers based on connectivity and weight sharing. As shown in FIGURE 3, the DCN 350 includes the convolution blocks 354A, 354B. Each of the convolution blocks 354A, 354B may be configured with a convolution layer (CONV) 356, a normalization layer (LNorm) 358, and a max pooling layer (MAX POOL) 360.Seyfarth Ref. No. 72178-00676113 310721357v.1Qualcomm Ref. No.2403743WO

[0053] Although only two of the convolution blocks 354A, 354B are shown, thepresent disclosure is not so limiting, and instead, any number of the convolution blocks 354A, 354B may be included in the DCN 350 according to design preference.

[0054] The convolution layers 356 may include one or more convolutional filters,which may be applied to the input data to generate a feature map. The normalization layer 358 may normalize the output of the convolution filters. For example, the normalization layer 358 may provide whitening or lateral inhibition. The max pooling layer 360 may provide down sampling aggregation over space for local invariance and dimensionality reduction.

[0055] The parallel filter banks, for example, of a deep convolutional network maybe loaded on a CPU 102 or GPU 104 of an SOC 100 (e.g., FIGURE 1) to achieve high performance and low power consumption. In alternative embodiments, the parallel filter banks may be loaded on the DSP 106 or an ISP 116 of an SOC 100. In addition, the DCN 350 may access other processing blocks that may be present on the SOC 100, such as sensor processor 114 and navigation module 120, dedicated, respectively, to sensors and navigation.

[0056] The DCN 350 may also include one or more fully connected layers 362 (FC1and FC2). The DCN 350 may further include a logistic regression (LR) layer 364. Between each layer 356, 358, 360, 362, 364 of the DCN 350 are weights (not shown) that are to be updated. The output of each of the layers (e.g., 356, 358, 360, 362, 364) may serve as an input of a succeeding one of the layers (e.g., 356, 358, 360, 362, 364) in the DCN 350 to learn hierarchical feature representations from input data 352 (e.g., images, audio, video, sensor data and / or other input data) supplied at the first of the convolution blocks 354A. The output of the DCN 350 is a classification score 366 for the input data 352. The classification score 366 may be a set of probabilities, where each probability is the probability of the input data including a feature from a set of features.

[0057] FIGURE 4 is a block diagram illustrating an exemplary software architecture400 that may modularize artificial intelligence (AI) functions. Using the architecture 400, applications may be designed that may cause various processing blocks of an SOC 420 (for example a CPU 422, a DSP 424, a GPU 426 and / or an NPU 428) (which maySeyfarth Ref. No. 72178-00676114 310721357v.1Qualcomm Ref. No.2403743WO be similar to SOC 100 of FIGURE 1) to support time variant Winograd based convolutions for an AI application 402, according to aspects of the present disclosure. The architecture 400 may, for example, be included in a computational device, such as a smartphone.

[0058] The AI application 402 may be configured to call functions defined in a userspace 404 that may, for example, provide for the detection and recognition of a scene indicative of the location at which the computational device including the architecture 400 currently operates. The AI application 402 may, for example, configure a microphone and a camera differently depending on whether the recognized scene is an office, a lecture hall, a restaurant, or an outdoor setting such as a lake. The AI application 402 may make a request to compiled program code associated with a library defined in an AI function application programming interface (API) 406. This request may ultimately rely on the output of a deep neural network configured to provide an inference response based on video and positioning data, for example.

[0059] The run-time engine 408, which may be compiled code of a runtimeframework, may be further accessible to the AI application 402. The AI application 402 may cause the run-time engine 408, for example, to request an inference at a particular time interval or triggered by an event detected by the user interface of the AI application 402. When caused to provide an inference response, the run-time engine 408 may in turn send a signal to an operating system in an operating system (OS) space 410, such as a Kernel 412, running on the SOC 420. In some examples, the Kernel 412 may be a LINUX Kernel. The operating system, in turn, may cause a continuous relaxation of quantization to be performed on the CPU 422, the DSP 424, the GPU 426, the NPU 428, or some combination thereof. The CPU 422 may be accessed directly by the operating system, and other processing blocks may be accessed through a driver, such as a driver 414, 416, or 418 for, respectively, the DSP 424, the GPU 426, or the NPU 428. In the exemplary example, the deep neural network may be configured to run on a combination of processing blocks, such as the CPU 422, the DSP 424, and the GPU 426, or may be run on the NPU 428.

[0060] As discussed, CNNs may be trained and implemented on software platformsthat use high-performance central processing units (CPUs) and graphics processing units (GPUs). Because of the increasing complexity of hardware architectures andSeyfarth Ref. No. 72178-00676115 310721357v.1Qualcomm Ref. No.2403743WO growing performance specifications, CPUs and GPUs may not be suitable for deploying inference engines. Therefore, dedicated hardware architectures or accelerators may be developed for CNNs. In some cases, these hardware architectures or accelerators may be specified for embedded systems. Application-specific integrated circuit (ASIC) and field programmable gate array (FPGA) implementations are examples of hardware architectures that satisfy performance specifications for inference operations.

[0061] Despite the rapid evolution of CNN models, their basic operations, such asconvolutions, remain unchanged. Convolution operations significantly contribute to computational complexity and consume a considerable amount of resources. Consequently, several techniques have been outlined to develop an efficient hardware architecture for CNN models. Such proposals are directed to reducing an amount of resources used by CNN models during inference. One such proposal is a pipelined accelerator that incorporates the Winograd minimal filtering function.

[0062] The Winograd minimal filtering function, denoted as ^^(^^, ^^), convolves animage of size ^^ + ^^ − 1 with a kernel of size ^^ to produce an output of dimension^^ × ^^. Specifically, the function applies a transformation on input pixels andkernels(e.g., filters), followed by a point-wise multiplication on the transformed matrix (e.g., transformed pixels and kernels). That is, the input pixels and kernels are transformed to a new domain, where the convolution is reduced to simpler element-wise multiplications. An inverse transformation is then performed to obtain the ^^x^^ output pixels. A generalized Winograd computation may be represented as: ^^ = ^^^^^^^^; ^^ = ^^^^^^^^; Out = ^^^^(^^. ^^)^^ (1)

[0063] In Equation 1, ^^^^ and ^^ represent transformation matrices. For example, ^^transforms the input data, and ^^^^is the transpose of ^^. Additionally, ^^ represents input data, such as an image or an image patch, that is being transformed. Accordingly, the operation, ^^^^^^^^ transforms the input data ^^ by first applying ^^, then applying ^^^^on the result. This sequence creates a new representation of the input data, denoted as ^^. Additionally, ^^ and ^^^^are transformation matrices used on a kernel ^^, where ^^ is an output of the transformation of the kernel ^^ by ^^ and then ^^^^. ^^^^and ^^ are another pair of transformation matrices. ^^ finalizes an output transformation and ^^^^is thetranspose. ^^. ^^ represents the element-wise multiplication of the transformed input ^^Seyfarth Ref. No. 72178-00676116 310721357v.1Qualcomm Ref. No.2403743WO and the transformed kernel ^^. The actual convolution operation is performed in thistransformed space. The expression Out = ^^^^(^^. ^^)^^ transforms this convolutionresult back into the spatial domain to produce the final output, denoted here as Out. This final transformation ensures that the result is in an appropriate format or layout for further processing by a convolutional neural network.

[0064] An image ^^ of size 4 × 4 and a filter ^^ of size 3 × 3 are shown in Equation2. The transformation matrices ^^, ^^, and ^^ for the image ^^ of size 4 × 4 and the filter^^ of size 3 × 3 are shown in Equation 3.

[0065] The transformation operations, which result in transformed input ^^,transformed kernel ^^, and transformed output Out may result in overhead due to the matrix multiplications with constant transformation matrices. In some examples, anoutput of the respective matrix multiplications ^^^^, ^^^^^^, and ^^^^^^ can be re-written astime variant vectors of size 1x4, 1x4, and 1x2, as follows:^^^^0 ^^^^0 − ^^^^2^^^⃗^ [^^] = [ 0.5(^^^^0 + ^^^^1 + ^^^^2) ^^⃗ ^^^^1 + ^^^^2 ^^⃗ ^^^^0 + ^^^^1 + ^^^^20.5(^^^^0 − ^^^^1 + ^^^^2) ] ^^ ^^ [^^] = [ ^^^^2 − ^^^^1 ] ^^ ^^ [^^] = [ ^^^^1 − ^^^^2 + ^^^^3] (4) ^^^^2 ^^^^3 − ^^^^1

[0066] In Equation 4, the matrix ^^ represents the intermediate computation resultsbased on (^^. ^^). ^^^⃗^ [^^] is computed by transforming the kernel ^^ with the matrix ^^ atspecific times or positions ^^. ^^^^^⃗^ [^^] is computed by transforming an input image ^^with the matrix ^^^^, the operation ^^^^^⃗^ [^^] precomputes certain sums and differences ofpixels that will later interact with the transformed kernel (^^^⃗^ [^^]), minimizing theworkload when these two components are combined. ^^^^^⃗^ [^^] is computed bytransforming the intermediate computation ^^ with the matrix ^^^^, the transformationSeyfarth Ref. No. 72178-00676117 310721357v.1Qualcomm Ref. No.2403743WO^^^^^⃗^ [^^] aligns and adjusts the convolution results to the format specified for the finaloutput. Additionally, in Equation 4, ^^^^0to ^^^^2refers to specific kernel values of an input kernel data, ^^^^0to ^^^^3refers to specific pixel data values of an input pixel data, and ^^^^0to ^^^^3refers to specific intermediate computation result values. As such, time variance does not refer to chronological time, rather, time variance specifies that the computation of each vector may be dependent on a specific portion of data that is being processed at any given stage of the convolution. That is, time represents a different stage in the convolution process, and the values processed by each vector vary based on the segment of the respective input data (e.g., pixel data ^^, kernel data ^^, intermediate computation result values ^^) that is being processed.

[0067] The matrix multiplication ^^^^^^^^ , ^^^^^^^^, and ^^^^^^^^ described with referenceto Equation 1 may be re-written as a time variant vector of size 4 × 1, 4 × 1, and 2 × 1,respectively, as represented in Equations 5, 6, and 7: ^^^^^⃗^^^[^^] = [^^^^0^^, 0.5(^^^^0^^ + ^^^^1^^ + ^^^^2^^), 0.5(^^^^0^^ − ^^^^1^^ + ^^^^2^^), ^^^^2^^] (5)^^^^^⃗^ ^^[^^] = [^^^^^^ ^^ ^^ ^^ ^^ ^^ ^^0^^ − ^^ ^^2^^, ^^ ^^1^^ + ^^ ^^2^^, ^^ ^^2^^ − ^^ ^^1^^, ^^ ^^3^^ − ^^^^^^1^^] (6) ^^^^^⃗^ ^^[^^] = [^^^^^^0^^ + ^^ ^^^^1^^ + ^^ ^^^^2^^, ^^ ^^^^1^^ − ^^ ^^^^2^^ + ^^^^^^3^^] (7)

[0068] As discussed below, Equations 4, 5, 6, and 7 may be realized in hardwareusing the pixel, kernel, and output transformer blocks, respectively. FIGURE 5 is a block diagram illustrating an example of a Winograd convolution processing engine 500, in accordance with various aspects of the present disclosure. The Winograd convolution processing engine 500 may also be referred to as the Winograd processing engine. The Winograd convolution processing engine 500 may be a component of a CNN model that is used for inference or another task. The Winograd convolution processing engine 500 may be implemented by, or work in conjunction with, one or more of the CPU (e.g., 102, 422), the GPU (e.g., 104, 426), or other processing units (e.g., DSP 424, NPU 428).

[0069] In the example of FIGURE 5, block 502 (denoted as ^^^^^^^^) corresponds toEquation 6, block 504 (denoted as ^^^^^^^^) corresponds to Equation 5, and block 506 (denoted as ^^^^^^^^) corresponds to Equation 7. Portions of Equation 4 are represented inSeyfarth Ref. No. 72178-00676118 310721357v.1Qualcomm Ref. No.2403743WO each of blocks 502, 504, and 506, as discussed further. Each block 502, 504, and 506 may be associated with a respective generator finite state machine (FSM) 550, which is an example of a control logic framework for controlling and coordinating data transfers between the blocks 502, 504, and 506 of the Winograd convolution processing engine 500. For brevity, only some FSMs are labeled in the example of FIGURE 5. The FSMs 550 may generate necessary handshake signals and addresses for loading and storing of intermediate results in local line buffers.

[0070] In the example of FIGURE 5, at block 502, pixel data ^^ (shown aspixel_data), may be received via a pixel data mover 510 and transformed by a first pixel transformer 512 (^^^^), resulting in an output ^^^^^^[^^], which may also be denoted as^^^^^⃗^ [^^] (for ease of explanation, the vector representation symbolis not shown in theexample of FIGURE 5). The transformation may be an example of a Winograd transformation. The transformed pixel data ^^^^^^[^^] may be stored in a pixel buffer 514 at an address (addr) provided by the FSM 550. A pixel selector 516 then selects the transformed pixel data ^^^^^^[^^] from the address for processing at the next stage. At the next stage, a second pixel transformer 518 (^^) applies a second transformation to the transformed pixel data ^^^^^^[^^] and outputs transformed pixel data ^^^^^^^^[^^]. The transformed pixel data ^^^^^^^^[^^] may be stored at a first array 520 (e.g., a 4x4 register array).

[0071] Additionally, in the example of FIGURE 5, at block 504, kernel data ^^(shown as kernel_coef), may be received via a coefficient data mover 522 and transformed by a first kernel transformer 524 (^^), resulting in an output ^^^^[^^]. The transformation may be an example of a Winograd transformation. The transformed kernel data ^^^^[^^] may be stored in a kernel buffer 526 at an address (addr) provided by the FSM 550. A coefficient selector 528 then selects the transformed kernel data ^^^^[^^] from the address for processing at the next stage. At the next stage, a second kernel transformer 530 (^^^^) applies a second transformation to the transformed kernel data ^^^^[^^] and outputs transformed kernel data ^^^^^^^^[^^]. The transformed kernel data ^^^^^^^^[^^] may be stored at a second array 532 (e.g., a 4x4 register array).

[0072] In the example of FIGURE 5, a multiplier 534 multiplies the transformedpixel data ^^^^^^^^[^^] and the transformed pixel data ^^^^^^^^[^^] to generate a partialSeyfarth Ref. No. 72178-00676119 310721357v.1Qualcomm Ref. No.2403743WO product (shown as partial_prod). The partial product may be a result of element-wise multiplication, such that the partial product is an example of an element-wise product of all 4x4 inputs. An intermediate computation ^^ may be based on the partial product. The partial product is received at an output selector 536 of block 506 in a time multiplexed manner. The output selector 536 selects the partial product intermediate computation ^^ from the multiplier 534 to be transformed by a first output transformer 538 (^^^^). The first output transformer 538 generates a transformed output ^^^^^^[^^], which is stored at the output buffer 540 at an address supplied by the FSM 550. In some examples, the output selector 536 receives control information from the FSM 536 and selects the partial product from the 4x4 reg array. The output selector 536 selects the transformed output ^^^^^^[^^] from the output buffer 540 to be transformed by a second output transformer 542 (^^), which generates the transformed output ^^^^^^^^[^^]. In the example 500 of FIGURE 5 two different output selectors 536 may be implemented or the same output selector 536 may be used.

[0073] In the example of FIGURE 5, the transformations are time varianttransformations corresponding to the transformations of Equations 4, 5, 6, and 7. Thus, each transformation may be performed over one or more clock cycles to process an entire image. The Winograd convolution processing engine 500 is an example of a hardware application of Equations 4, 5, 6, and 7 for low power image processing. In some examples, the Winograd convolution processing engine 500 reduces a quantity of multiplication operations specified for an image convolution while also reducing the overhead associated with conventional Winograd functions.

[0074] As discussed, in some examples, the Winograd convolution processingengine 500 may be a component of a CNN model. FIGURE 6 is a block diagram illustrating an example of a CNN model 600, in accordance with various aspects of the present disclosure. The CNN model 600 may be implemented by, or work in conjunction with, one or more of the CPU (e.g., 102, 422), the GPU (e.g., 104, 426), or other processing units (e.g., DSP 424, NPU 428). The CNN model 600 includes an array of processing engines (PE) 500 to perform convolution operations. The PEs are examples of the Winograd convolution processing engines 500 described with reference to FIGURE 5. The CNN model 600 also includes one or more pixel data movers 510 and one or more kernel data movers 522. The kernel data movers may also be referredSeyfarth Ref. No. 72178-00676120 310721357v.1Qualcomm Ref. No.2403743WO to as coefficient data movers 522. The data movers 510, 522 re-order data and provide the re-ordered data, such as pixels and / or kernels, to the PEs 500, in a time multiplexed fashion. A length of the PE array (e.g., a number of PEs 500) may be proportional to a quantity of filters in a layer. In some examples, only a specified quantity of instances of PEs 500 per layer may be enabled. Additionally, the PEs 500 may operate on a common 4x4 image tile. Other sizes are also contemplated.

[0075] The image data may be stored in a pixel memory block 602, such as pixelblock random access memory (RAM), or another type of memory. The convolutions may be applied to the image data stored in the pixel block memory 602. The pixel block memory 602 may also store the resulting output pixel data. A filter memory block 604 may store filter coefficients (e.g., kernels), and the filter memory block 604 may be an example of a dedicated memory block. The filter memory block 604 may be an example of RAM, or another type of memory.

[0076] A memory management module (MMM) 606 communicates with the pixelmemory block 602 and the filter memory block 604 using a memory interface, such as a streaming interface. For example, the MMM 606 may serve as a backbone between both the pixel memory block 602 and the filter memory block 604, and the array of PEs 500. Specifically, the MMM 606 may include control and compute logic 620 for controlling and coordinating data transfers between the PEs 500 and memory blocks 602 and 604. As shown in the example of FIGURE 6, the MMM 606 includes a write flow finite state machine (WF-FSM) 608 and a read flow FSM (RF-FSM) 610. The RF- FSM 610 loads the PEs 500 with pixel data of a current tile 650 by calculating the physical address. The tile 650 refers to 4x4 pixel data. The RF-FSM 610 may first calculate the virtual address of the current tile 650 followed by virtual to physical translation of the address by querying a read address translation table (RATT) 612. In some examples, a write address translation table (WATT) 614 is used by the WF-FSM 608 to transfer the processed pixels back to the memory blocks 602 and 604.

[0077] In some examples, the RF-FSM 610 may generate memory addresses used toretrieve pixel data for processing in a PE 500 in accordance with a memory address function shown in TABLE 1. The RF-FSM 610 may be one or more of the FSMs 550 described with reference to FIGURE 5. As shown in TABLE 1, the memory address function of the RF-FSM 610 initiates with two read bursts, each with a length of four, toSeyfarth Ref. No. 72178-00676121 310721357v.1Qualcomm Ref. No.2403743WO gather the initial pixel tile of an image row. Subsequent pixel retrievals specify only two bursts of a length of two. This approach reuses previously fetched pixels, thereby reducing memory bus transactions.

[0078] In the example of TABLE 1, the input is a PE array handshake, which maybe a synchronization signal from that array of PEs 500 indicating the array is ready to receive new data. The input also includes layer information, which includes information about the neural network layer being processed, such as layer dimensions. The output is pixel tile data, which is a segment of the input image (e.g., a tile) that is ready to be processed by the PE array. The function loops through a three-dimensional space defined by rows, channels (ch), and columns (col), which correspond to the dimensions of the input data (e.g., an image or feature map).

[0079] The memory address function includes a row loop (line 1) that iterates overhalf the size of the input (e.g., two rows), followed by a channel loop (line 2) that iterates over the number of channels (ch num). The number of channels corresponds to a depth of an image (e.g., red, green, and blue (RGB) channels). Alternatively, the number of channels may correspond to a number of feature maps in a layer. The memory address function also includes a column loop (line 3) that iterates over the width of the input, incrementing by four each time, based on the 4 x 4 tile size. Line 5 calculates the channel offset (ch off) based on a size of the channel data, line 6Seyfarth Ref. No. 72178-00676122 310721357v.1Qualcomm Ref. No.2403743WO calculates a row offset (row off) based on a size of one row’s data and a number of channels, and line 7 computes the virtual tile address (virt_tile_addr), which is a memory address where the pixel tile data begins. The virtual tile address is the sum of the channel and row offsets, plus the current column. At line 8, the virtual address may be translated to a physical address. At line 9, the neighboring pixel data is retrieved. The neighboring pixel data refers to a block or neighborhood of pixels around the current position so that the convolution operation can be performed. Because the convolution kernel overlaps adjacent pixels as it moves across the image, fetching neighboring pixel data is a necessary step for each computation. At line 10, the fetched pixel data is written to the data mover. Finally, at line 11, the function waits for a handshake from the PE array, signaling that it has successfully received the previous tile data and is ready for more.

[0080] The data fetched by the RF-FSM 610 may be transferred to the data movers510 and 522. In some examples, a single instance of a pixel data mover 510 is used. Additionally, all of the kernels in a current layer operate on the same pixel tile. A quantity of kernel data movers 522 may be directly proportional to a quantity of kernels in the layer. These kernel data movers 522 are responsible for feeding kernel data 652 to the PEs 500 in time-multiplexed format. An output accumulator and data mover 616 stores processed pixels 618 and implements an accumulator, used to accumulate the results over the depth of the image. Once a complete row of the image is processed over the depth, the output data is stored back to the pixel memory block 602 via the MMM 606.

[0081] In some examples, a maximum memory specified to store the pixels (e.g.,features) may be calculated as ^^^^^^ = ^^^^^^(^^^^^^^^ ∗ ^^^^^^^^ ∗ ^^ℎ^^^^^^^^^^). Considering a 16-bit representation, a total memory specified to store pixel values may be ^^^^^^ ∗ 16 bits.In some such examples, 50,176 KB of storage may be specified, using the VGG-16 convolutional neural network (CNN) architecture as a reference model.

[0082] FIGURE 7A is a block diagram illustrating an example 700 of block RAMdata organization for pixels 710, in accordance with various aspects of the present disclosure. In the example 700 of FIGURE 7A, the pixel data may be associated with an image having a dimension of Size x Size, channel depth. As shown in FIGURE 7A, the depth is three, where the depth includes three channels, such as red 702, green 704,Seyfarth Ref. No. 72178-00676123 310721357v.1Qualcomm Ref. No.2403743WO and blue 706. Aspects of the present disclosure are not limited to three channels. Images with other amounts of channels are contemplated. As shown in FIGURE 7A, two blocks of RAM 708 with a width of 32 bits and a depth of 1,048,576 bits may store the pixels 702, 704, and 706. Other sizes for the RAM 708 are contemplated.

[0083] In some examples, a single memory access retrieves two pixels. The storagescheme follows an inverted N-order, with a depth of two pixels, arranged in a raster scan order. This storage pattern interleaves two pixel rows across the different channels of the image. Memory use is improved by applying this process across all image rows. Specifically, image data retrieval may be improved because a total number of memory accesses may be reduced. By storing and accessing two pixels at a time, the system can reduce the time and computational overhead involved in reading each pixel individually. This is particularly beneficial in processing models, such as VGG-16, where large volumes of image data are accessed for analysis.

[0084] In the example 700 of FIGURE 7A, the unused space in the RAM 708 maystore the processed output pixel. An MMM, such as the MMM 606 described with reference to FIGURE 6, handles the address management and is aware of the unused memory locations. Therefore, the MMM may store the output pixels, such as the processed pixels 618 described with reference to FIGURE 6, into the unused memory locations. With this kind of memory arrangement, whole layers of a CNN, such as VGG-16, may be processed.

[0085] FIGURE 7B is a block diagram illustrating an example 750 of block RAMdata organization for kernel data, in accordance with various aspects of the present disclosure. In the example 750 of FIGURE 7B, a block of RAM 754 may have a width of 144 bits and a depth of 8,192 bits. Other sizes for the RAM 754 are also contemplated. Kernels for different layers may be stored in an address space of the RAM 754. For example, a first set of kernels 752 for one layer may be stored in one address space. For ease of explanation, only one set of kernels are labeled in the example 750 of FIGURE 7B.

[0086] FIGURE 8 is a block diagram illustrating an example 800 of read addresstranslation tables (RATTs) and write address translation tables (WATTs), in accordance with various aspects of the present disclosure. As shown in the example 800 ofSeyfarth Ref. No. 72178-00676124 310721357v.1Qualcomm Ref. No.2403743WO FIGURE 8, a first RATT 802 contains a virtual (shown as virt_addr) to physical address (shown as phy_addr) translation look up of a location of the memory containing the image to be processed by the current layer (e.g., layer n-1). After completing a fetch operation (fetch()), a physical location of a processed pixel is pushed back to a first WATT 804. The fetch operation is an example of the FETCH operation in line 9 of TABLE 1. The first WATT 804 may be referenced by a write flow FSM (WF-FSM) of an MMM, such as the WF-FSM 608 described with reference to FIGURE 6, to write the results of convolution. That is, a write operation (write()) may write to the virtual address, which is then translated to the physical address location. In the example 800 of Figure 8, valid data out represents a final output of the convolution for the layer, such as the layer n-1.

[0087] After processing the current layer, such as layer n-1, the WATT and RATTtables are interchanged. For example, in the example 800, at layer n, a second RATT 806 is the first WATT 804 of layer n-1, and a second WATT 808 is the first RATT 802 of layer n-1. However, the virtual addresses may remain the same. In this way, memory can be utilized effectively.

[0088] FIGURE 9 is a flow diagram illustrating an example of a process 900 fortime variant Winograd based convolution, in accordance with various aspects of the present disclosure. The 900 may be performed by one or more processors such as the CPU (e.g., 102, 422), GPU (e.g., 104, 426), other processing units (e.g., DSP 424, NPU 428), Winograd convolution processing engine 500, and / or a CNN model 600, for example. The process 900 begins at block 902 by transforming, at a PE, pixel data into a time variant pixel representation based on receiving the pixel data at the PE. At block 904, the process 900 transforms, at the PE, kernel data into a time variant kernel representation based on receiving the kernel data at the PE. At block 906, the process 900 generates, at the PE, transformed pixel data based on the time variant pixel representation. At block 908, the process 900 generates, at the PE, transformed kernel data based on the time variant kernel representation. At block 910, the process 900 generates, at the PE, a transformed output of the pixel data based on an element-wise product of the transformed pixel data and the transformed kernel data.

[0089] Implementation examples are described in the following numbered clauses:Seyfarth Ref. No. 72178-00676125 310721357v.1Qualcomm Ref. No.2403743WO Clause 1. A method comprising: transforming, at a processing engine (PE), pixel data into a time variant pixel representation based on receiving the pixel data at the PE; transforming, at the PE, kernel data into a time variant kernel representation based on receiving the kernel data at the PE; generating, at the PE, transformed pixel data based on the time variant pixel representation; generating, at the PE, transformed kernel data based on the time variant kernel representation; and generating, at the PE, a transformed output of the pixel data based on an element-wise product of the transformed pixel data and the transformed kernel data. Clause 2. The method of Clause 1, where the PE is a Winograd processing engine. Clause 3. The method of any one of Clauses 1-2, wherein: generating the transformed pixel data comprises transforming the time variant pixel representation by a pixel transformation matrix and a transpose of the pixel transformation matrix; generating the transformed kernel data comprises transforming the time variant kernel representation by a kernel transformation matrix and a transpose of the kernel transformation matrix; and generating the transformed output comprises transforming the element-wise product by an output transformation matrix and a transpose of the output transformation matrix. Clause 4. The method of any one of Clauses 1-3, wherein the PE is one PE of a group of PEs of a convolutional neural network (CNN) model. Clause 5. The method of Clause 4, wherein the CNN model includes a memory management module (MMM) configured to coordinate transfers of pixel data and kernel data between respective memory blocks and the PE. Clause 6. The method of Clause 5, wherein the MMM comprises a read flow finite state machine (RF-FSM) configured to read the pixel data from a pixel memory block and a write flow FSM (WF-FSM) configured to transfer the transformed output to the pixel memory block.Seyfarth Ref. No. 72178-00676126 310721357v.1Qualcomm Ref. No.2403743WO Clause 7. The method of any one of Clauses 1-6, wherein the pixel data is a pixel tile from an input image.

[0090] The various operations of methods described above may be performed byany suitable means capable of performing the corresponding functions. The means may include various hardware and / or software component(s) and / or module(s), including, but not limited to, a circuit, an application specific integrated circuit (ASIC), or processor. Generally, where there are operations illustrated in the figures, those operations may have corresponding counterpart means-plus-function components with similar numbering.

[0091] As used, the term “determining” encompasses a wide variety of actions. Forexample, “determining” may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a database, or another data structure), ascertaining and the like. Additionally, “determining” may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and the like. Furthermore, “determining” may include resolving, selecting, choosing, establishing, and the like.

[0092] As used, a phrase referring to “at least one of” a list of items refers to anycombination of those items, including single members. As an example, “at least one of: a, b, or c” is intended to cover: a, b, c, a-b, a-c, b-c, and a-b-c.

[0093] The various illustrative logical blocks, modules and circuits described inconnection with the present disclosure may be implemented or performed with a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.Seyfarth Ref. No. 72178-00676127 310721357v.1Qualcomm Ref. No.2403743WO

[0094] The steps of a method or algorithm described in connection with the presentdisclosure may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in any form of storage medium that is known in the art. Some examples of storage media that may be used include random access memory (RAM), read only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, a hard disk, a removable disk, a CD-ROM and so forth. A software module may comprise a single instruction, or many instructions, and may be distributed over several different code segments, among different programs, and across multiple storage media. A storage medium may be coupled to a processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor.

[0095] The methods disclosed comprise one or more steps or actions for achievingthe described method. The method steps and / or actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0096] The functions described may be implemented in hardware, software,firmware, or any combination thereof. If implemented in hardware, an example hardware configuration may comprise a processing system in a device. The processing system may be implemented with a bus architecture. The bus may include any number of interconnecting buses and bridges depending on the specific application of the processing system and the overall design constraints. The bus may link together various circuits including a processor, machine-readable media, and a bus interface. The bus interface may be used to connect a network adapter, among other things, to the processing system via the bus. The network adapter may be used to implement signal processing functions. For certain aspects, a user interface (e.g., keypad, display, mouse, joystick, etc.) may also be connected to the bus. The bus may also link various other circuits such as timing sources, peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further.Seyfarth Ref. No. 72178-00676128 310721357v.1Qualcomm Ref. No.2403743WO

[0097] The processor may be responsible for managing the bus and generalprocessing, including the execution of software stored on the machine-readable media. The processor may be implemented with one or more general-purpose and / or special- purpose processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuitry that can execute software. Software shall be construed broadly to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. Machine-readable media may include, by way of example, random access memory (RAM), flash memory, read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable Read-only memory (EEPROM), registers, magnetic disks, optical disks, hard drives, or any other suitable storage medium, or any combination thereof. The machine-readable media may be embodied in a computer-program product. The computer-program product may comprise packaging materials.

[0098] In a hardware implementation, the machine-readable media may be part ofthe processing system separate from the processor. However, as those skilled in the art will readily appreciate, the machine-readable media, or any portion thereof, may be external to the processing system. By way of example, the machine-readable media may include a transmission line, a carrier wave modulated by data, and / or a computer product separate from the device, all which may be accessed by the processor through the bus interface. Alternatively, or in addition, the machine-readable media, or any portion thereof, may be integrated into the processor, such as the case may be with cache and / or general register files. Although the various components discussed may be described as having a specific location, such as a local component, they may also be configured in various ways, such as certain components being configured as part of a distributed computing system.

[0099] The processing system may be configured as a general-purpose processingsystem with one or more microprocessors providing the processor functionality and external memory providing at least a portion of the machine-readable media, all linked together with other supporting circuitry through an external bus architecture. Alternatively, the processing system may comprise one or more neuromorphic processors for implementing the neuron models and models of neural systemsSeyfarth Ref. No. 72178-00676129 310721357v.1Qualcomm Ref. No.2403743WO described. As another alternative, the processing system may be implemented with an application specific integrated circuit (ASIC) with the processor, the bus interface, the user interface, supporting circuitry, and at least a portion of the machine-readable media integrated into a single chip, or with one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic, discrete hardware components, or any other suitable circuitry, or any combination of circuits that can perform the various functionality described throughout this disclosure. Those skilled in the art will recognize how best to implement the described functionality for the processing system depending on the particular application and the overall design constraints imposed on the overall system.

[0100] The machine-readable media may comprise a number of software modules.The software modules include instructions that, when executed by the processor, cause the processing system to perform various functions. The software modules may include a transmission module and a receiving module. Each software module may reside in a single storage device or be distributed across multiple storage devices. By way of example, a software module may be loaded into RAM from a hard drive when a triggering event occurs. During execution of the software module, the processor may load some of the instructions into cache to increase access speed. One or more cache lines may then be loaded into a general register file for execution by the processor. When referring to the functionality of a software module below, it will be understood that such functionality is implemented by the processor when executing instructions from that software module. Furthermore, it should be appreciated that aspects of the present disclosure result in improvements to the functioning of the processor, computer, machine, or other system implementing such aspects.

[0101] If implemented in software, the functions may be stored or transmitted overas one or more instructions or code on a computer-readable medium. Computer- readable media include both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage medium may be any available medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used toSeyfarth Ref. No. 72178-00676130 310721357v.1Qualcomm Ref. No.2403743WO carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Additionally, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared (IR), radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray® disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Thus, in some aspects, computer-readable media may comprise non-transitory computer- readable media (e.g., tangible media). In addition, for other aspects computer-readable media may comprise transitory computer- readable media (e.g., a signal). Combinations of the above should also be included within the scope of computer-readable media.

[0102] Thus, certain aspects may comprise a computer program product forperforming the operations presented. For example, such a computer program product may comprise a computer-readable medium having instructions stored (and / or encoded) thereon, the instructions being executable by one or more processors to perform the operations described. For certain aspects, the computer program product may include packaging material.

[0103] Further, it should be appreciated that modules and / or other appropriatemeans for performing the methods and techniques described can be downloaded and / or otherwise obtained by a user terminal and / or base station as applicable. For example, such a device can be coupled to a server to facilitate the transfer of means for performing the methods described. Alternatively, various methods described can be provided via storage means (e.g., RAM, ROM, a physical storage medium such as a compact disc (CD) or floppy disk, etc.), such that a user terminal and / or base station can obtain the various methods upon coupling or providing the storage means to the device. Moreover, any other suitable technique for providing the methods and techniques described to a device can be utilized.

[0104] It is to be understood that the claims are not limited to the preciseconfiguration and components illustrated above. Various modifications, changes, andSeyfarth Ref. No. 72178-00676131 310721357v.1Qualcomm Ref. No.2403743WO variations may be made in the arrangement, operation, and details of the methods and apparatus described above without departing from the scope of the claims.Seyfarth Ref. No. 72178-00676132 310721357v.1

Claims

Qualcomm Ref. No.2403743WO CLAIMS 1. A method comprising: transforming, at a processing engine (PE), pixel data into a time variant pixel representation based on receiving the pixel data at the PE; transforming, at the PE, kernel data into a time variant kernel representation based on receiving the kernel data at the PE; generating, at the PE, transformed pixel data based on the time variant pixel representation; generating, at the PE, transformed kernel data based on the time variant kernel representation; and generating, at the PE, a transformed output of the pixel data based on an element-wise product of the transformed pixel data and the transformed kernel data.

2. The method of claim 1, where the PE is a Winograd processing engine.

3. The method of claim 1, wherein: generating the transformed pixel data comprises transforming the time variant pixel representation by a pixel transformation matrix and a transpose of the pixel transformation matrix; generating the transformed kernel data comprises transforming the time variant kernel representation by a kernel transformation matrix and a transpose of the kernel transformation matrix; and generating the transformed output comprises transforming the element-wise product by an output transformation matrix and a transpose of the output transformation matrix.

4. The method of claim 1, wherein the PE is one PE of a group of PEs of a convolutional neural network (CNN) model.

5. The method of claim 4, wherein the CNN model includes a memory management module (MMM) configured to coordinate transfers of pixel data and kernel data between respective memory blocks and the PE.Seyfarth Ref. No. 72178-00676133 310721357v.1Qualcomm Ref. No.2403743WO 6. The method of claim 5, wherein the MMM comprises a read flow finite state machine (RF-FSM) configured to read the pixel data from a pixel memory block and a write flow FSM (WF-FSM) configured to transfer the transformed output to the pixel memory block.

7. The method of claim 1, wherein the pixel data is a pixel tile from an input image.

8. A apparatus comprising: one or more processors; and one or more memories coupled with the one or more processors and storing processor-executable code that, when executed by the one or more processors, is configured to cause the apparatus to: transform, at a processing engine (PE), pixel data into a time variant pixel representation based on receiving the pixel data at the PE; transform, at the PE, kernel data into a time variant kernel representation based on receiving the kernel data at the PE; generate, at the PE, transformed pixel data based on the time variant pixel representation; generate, at the PE, transformed kernel data based on the time variant kernel representation; and generate, at the PE, a transformed output of the pixel data based on an element-wise product of the transformed pixel data and the transformed kernel data.

9. The apparatus of claim 8, where the PE is a Winograd processing engine.

10. The apparatus of claim 8, wherein execution of the processor-executable code that causes the apparatus to: generate the transformed pixel data further causes the apparatus to transform the time variant pixel representation by a pixel transformation matrix and a transpose of the pixel transformation matrix;Seyfarth Ref. No. 72178-00676134 310721357v.1Qualcomm Ref. No.2403743WO generate the transformed kernel data further causes the apparatus to transform the time variant kernel representation by a kernel transformation matrix and a transpose of the kernel transformation matrix; and generate the transformed output further causes the apparatus to transform the element-wise product by an output transformation matrix and a transpose of the output transformation matrix.

11. The apparatus of claim 8, wherein the PE is one PE of a group of PEs of a convolutional neural network (CNN) model.

12. The apparatus of claim 11, wherein the CNN model includes a memory management module (MMM) configured to coordinate transfers of pixel data and kernel data between respective memory blocks and the PE.

13. The apparatus of claim 12, wherein the MMM comprises a read flow finite state machine (RF-FSM) configured to read the pixel data from a pixel memory block and a write flow FSM (WF-FSM) configured to transfer the transformed output to the pixel memory block.

14. The apparatus of claim 8, wherein the pixel data is a pixel tile from an input image.

15. A non-transitory computer-readable medium having program code recorded thereon, the program code executed by one or more processors and comprising: program code to transform, at a processing engine (PE), pixel data into a time variant pixel representation based on receiving the pixel data at the PE; program code to transform, at the PE, kernel data into a time variant kernel representation based on receiving the kernel data at the PE; program code to generate, at the PE, transformed pixel data based on the time variant pixel representation; program code to generate, at the PE, transformed kernel data based on the time variant kernel representation; andSeyfarth Ref. No. 72178-00676135 310721357v.1Qualcomm Ref. No.2403743WO program code to generate, at the PE, a transformed output of the pixel data based on an element-wise product of the transformed pixel data and the transformed kernel data.

16. The non-transitory computer-readable medium of claim 15, where the PE is a Winograd processing engine.

17. The non-transitory computer-readable medium of claim 15, wherein: the program code to generate the transformed pixel data further comprises program code to transform the time variant pixel representation by a pixel transformation matrix and a transpose of the pixel transformation matrix; the program code to generate the transformed kernel data further comprises program code to transform the time variant kernel representation by a kernel transformation matrix and a transpose of the kernel transformation matrix; and the program code to generate the transformed output further comprises program code to transform the element-wise product by an output transformation matrix and a transpose of the output transformation matrix.

18. The non-transitory computer-readable medium of claim 15, wherein the PE is one PE of a group of PEs of a convolutional neural network (CNN) model.

19. The non-transitory computer-readable medium of claim 18, wherein the CNN model includes a memory management module (MMM) configured to coordinate transfers of pixel data and kernel data between respective memory blocks and the PE.

20. The non-transitory computer-readable medium of claim 19, wherein the MMM comprises a read flow finite state machine (RF-FSM) configured to read the pixel data from a pixel memory block and a write flow FSM (WF-FSM) configured to transfer the transformed output to the pixel memory block.Seyfarth Ref. No. 72178-00676136 310721357v.1

Images