De-sparse convolution for sparse activation

Abstract

Certain aspects of the present disclosure provide techniques for de-sparse convolution. An activation tensor is received and a convolution output is generated for the activation tensor, including: selecting a subset of weight elements from a weight tensor corresponding to a set of non-zero elements in the activation tensor; and multiplying the set of non-zero elements by the set of weighted elements.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application serial number 17 / 817,552 filed on August 4, 2022, which is hereby incorporated by reference into this application.

[0003] introduction

[0004] Aspects of the present disclosure relate to efficient convolution operations.

[0005] Convolution has become an increasingly important operation for a wide variety of computational solutions, including in convolutional neural networks (which typically involve applying a large number of convolution operations to input data). Convolutional neural networks can be trained for countless tasks, such as computer vision (e.g., image or object recognition), audio processing, and the like. Generally speaking, a single convolution operation involves multiplying one or more portions of an input tensor by one or more weights in a convolution kernel (e.g., where the weights are organized as a set of channels for a K×K kernel), where the weights are learned during the training process. Conventional convolution operations and networks typically require a large number of such multiplications due to a number of factors, including the size of the data tensor (e.g., the number of elements), the number of applications of each kernel, the number of kernels, the number of layers, and the like.

[0006] Typically, convolution operations are paired with activation operations. For example, the output of a given convolution is often processed using an activation function (e.g., ReLU) before being used as the input to a subsequent convolution. Many popular activation functions introduce nonlinearity to the network by setting a subset of elements in the input tensor to zero. For example, in the case of ReLU, all negative values ​​are set to zero. These nonlinear functions are widely used and can significantly improve the accuracy of the model. Notably, they also introduce a lot of sparsity to the intermediate data (in the form of zero values) being processed by the neural network.

[0007] Conventional neural networks typically ignore this sparsity and, for example, convolve all elements of the input tensor, even if the convolution has elements with a value of zero that have no effect on the output of the convolution. This wastes energy, time, and computational resources. Some conventional approaches to handling sparsity have focused on sparse weight tensors, which are known a priori (e.g., before runtime inference) and can therefore be handled using static methods. However, activations are not known until runtime and thus must be handled dynamically.

[0008] Conventional methods for activating sparsity typically require modifying the network itself, retraining the network, and / or performing additional rounds of training and data access, which results in significant additional costs for computation and development. Furthermore, this additional cost is often accompanied by an inevitable degradation in model accuracy and performance.

[0009] Therefore, techniques for improving convolutions on sparse tensors are needed. Summary of the invention

[0010] Certain aspects provide a method comprising: receiving an activation tensor; and generating a first convolution output for the activation tensor, comprising: selecting a first subset of weight elements from a weight tensor corresponding to a first set of non-zero elements in the activation tensor; and multiplying the first set of non-zero elements with the first set of weight elements.

[0011] Other aspects provide processing systems configured to perform the aforementioned methods and those described herein; non-transitory computer-readable media comprising instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the aforementioned methods and those described herein; a computer program product embodied on a computer-readable storage medium, the computer program product comprising code for performing the aforementioned methods and those further described herein; and a processing system comprising components for performing the aforementioned methods and those further described herein.

[0012] The following description and associated drawings set forth in detail certain illustrative features of the one or more aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings depict certain aspects of the one or more aspects and therefore are not to be considered limiting of the scope of the present disclosure.

[0014] Figure 1 Depicts an example workflow for performing desparse convolutions.

[0015] Figure 2 Depicts an example workflow for using desparse convolutions on relatively sparse activation tensors to efficiently generate convolution outputs.

[0016] Figure 3 Depicts an example workflow for pipelined desparse convolutions to improve computational efficiency.

[0017] Figure 4 Depicted is an example architecture for performing sparse lookahead using a set of shifters.

[0018] Figure 5 An example flow chart illustrating a method for generating an output using desparse convolution is depicted.

[0019] Figure 6 An example flow chart illustrating a method for desparse convolution is depicted.

[0020] Figure 7An example flow chart illustrating a method for generating a convolution output using desparse convolution is depicted.

[0021] Figure 8 An example processing system configured to perform various aspects of the present disclosure is depicted.

[0022] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one aspect may be beneficially incorporated in other aspects without further recitation. DETAILED DESCRIPTION

[0023] Aspects of the present disclosure provide techniques for improved convolution (referred to herein as desparse convolution).

[0024] In various aspects, desparse convolutions utilize sparse lookahead to identify sparsity in activations in order to perform mathematically equivalent convolutions with reduced computational complexity. That is, unlike approximating conventional convolution operations, the techniques described herein produce mathematically equivalent outputs, thereby allowing the use of desparse convolution operations to improve computational efficiency and latency without affecting model accuracy.

[0025] Furthermore, the techniques described herein are lightweight in terms of design and computational complexity as well as execution overhead. Additionally, desparse convolutions are easily applied to any model and location (including during training and during inference) without requiring any network modification or retraining. Furthermore, desparse convolution operations can be used to replace multiple convolution types, including standard convolutions and point-wise convolutions, as well as in fully connected layers.

[0026] On the one hand, the desparse convolution operation realizes the dynamic identification of non-zero (or non-empty) activation elements, the dynamic retrieval of the corresponding weights of these non-zero activations, and the calculation using only these non-zero elements. By ignoring the activation elements (and corresponding weights) with a value of zero, the technology described herein can use less computing resources (e.g., reduced memory usage, reduced processing time, reduced energy consumption, etc.) to perform mathematically equivalent convolutions, thereby significantly improving the operation of the underlying system of the model and training and using such models. As used herein, an element may be referred to as "sparse" to indicate that it has a value of zero or a null value. Similarly, an element may be referred to as "non-sparse" to indicate that it has a non-zero value or a non-null value. In various aspects, alternative specific implementations may include other definitions or formats, such as elements with a value less than a defined value are considered to be sparse.

[0027] Example workflow for performing desparse convolution

[0028] Figure 1 An example workflow 100 for performing desparse convolutions for an example layer of a convolutional neural network is depicted.

[0029] In the illustrated workflow, an input tensor 105 is received for processing by the example layer. In one aspect, the input tensor 105 may generally correspond to a tensor having a plurality of data elements, each element corresponding to a certain value. In general, the input tensor 105 may have any dimensions, including, for example, H×W×C in , where H is the height of the input tensor 105 (eg, the number of elements in the vertical orientation), W is the width of the tensor 105 (eg, the number of elements in the horizontal orientation), and C in is the number of channels in tensor 105 (e.g., the number of elements in the depth of the tensor). Generally, height and width may be referred to as spatial dimensions, while the number of channels may be referred to as the depth dimension. In the illustrated example, tensor 105 has dimensions 4×4×8.

[0030] In one aspect, tensor 105 can be referred to as pre-activation data because it has not yet been processed using an activation function of a layer (e.g., activation function 110). For example, tensor 105 can be the output of one or more previous layers of a neural network before activation (e.g., the output of a convolution operation in a previous layer), the original input to a neural network, etc. In some aspects, elements in input tensor 105 can have a wide range of values. In some implementations, input tensor 105 can be considered to be quite dense because it typically has few (or no) elements with values ​​of zero.

[0031] In the illustrated aspect, the input tensor 105 is processed using an activation function 110 to generate an activation tensor 115 (also referred to in some aspects as activation data, an activation set, or simply activations). The specific algorithm or operation of the activation function 110 may vary depending on the specific implementation. For example, the activation function 110 can be a ReLU activation function (e.g., a function that sets negative input elements to zero and leaves positive input elements unchanged), a binary step function, a linear function, a sigmoid function, a hyperbolic tangent (tanh) function, a leaky or parametric ReLU function, a softmax function, a swish function, etc.

[0032] In the illustrated workflow 100, the activation tensor 115 includes a certain amount of sparsity indicated by the pattern of each element. Specifically, sparse elements (e.g., those elements having a value of zero) may be indicated by stippling, while non-sparse elements (e.g., those elements having a non-zero value) do not have stippling. In various aspects, the distribution of sparsity in the activation tensor 115 cannot generally be determined a priori. That is, at least the input tensor 105 must be known in order to determine the specific sparse and non-sparse elements of the activation tensor 115. Since the input tensor 105 itself is not known until runtime (e.g., until it is generated by a previous layer in the network), the system cannot use any static configuration to handle this activation sparsity (compared to weights that are generally determined after training).

[0033] In conventional approaches, the system simply convolves this (potentially sparse) activation tensor 115 with the set of weights (e.g., in weight tensor 125) to be used for each convolution kernel of the layer. This involves multiplying and aggregating multiple (possibly large) sparse elements in the activation tensor 115, even though the multiplication of these elements will inherently result in zero values ​​and thus have no effect on the convolution output. Therefore, conventional systems waste a lot of computational resources and time by processing these sparse elements.

[0034] In some cases, the sparsity may approach or exceed 50% of the activations. That is, half or more of the elements of the activation tensor 115 may be zero-valued. For example, ReLU sets all negative values ​​to zero. Since the values ​​of the input tensor 105 are typically centered around zero (e.g., due to a normalization process within the network), approximately half of the elements in the tensor 105 may be set to zero. Thus, by using the desparse convolution operations described herein, the system can significantly reduce the computational cost of convolutions for a wide variety of models (by about 50%).

[0035] In the illustrated workflow 100, instead of using conventional convolutions, the activation tensor 115 is processed by the convolution bypass 120 using a desparse convolution block 130. Although illustrated as a discrete convolution bypass 120 for conceptual clarity, in various aspects, the desparse convolution block 130 can generally be used to replace any convolution operation. When describing the operations involved in performing the desparse convolution block 130, the convolution bypass 120 indicated by the dashed box is depicted for ease of reference. In general, the convolution bypass 120 involves processing some elements (e.g., non-sparse elements) in the activation tensor 115 while ignoring the convolution of other elements (e.g., sparse elements).

[0036] In general, as discussed above, conventional convolution involves multiplying each element of the activation tensor 115 using one or more elements in the weight tensor 125 to produce an output tensor. In the illustrated example, the convolution bypass 120 instead uses the desparse convolution block 130 to operate only on a subset of the activation tensor 115. Specifically, the convolution bypass 120 can dynamically identify sparse and / or non-sparse elements in the activation tensor 115. For each non-sparse element, the convolution bypass 120 can identify the corresponding weight in the weight tensor 125, and use only these non-sparse elements and corresponding weights to generate the output tensor 135. On the one hand, because the convolution bypass 120 does not operate on sparse elements (e.g., they are skipped or ignored), the computational resources required to generate the output tensor 135 are significantly reduced. Reference is made below to the desparse convolution block 130. Figure 3 One example of the desparse convolution block 130 is described in more detail.

[0037] Although aspects of the present disclosure relate to desparse convolutions for sparse inputs, it should be understood that desparse convolution techniques can also be easily used to convolve relatively non-sparse input tensors. That is, the desparse convolution techniques described herein can be used to process inputs with any amount of sparsity, including high sparsity (e.g., having few non-sparse elements), low sparsity (e.g., having few sparse elements), and no sparsity (e.g., having no sparse elements). Since the resulting output is mathematically equivalent to a conventional convolution, using desparse convolution on dense inputs requires only a conventional amount of computational resources (e.g., the same number of multiplications as a conventional convolution).

[0038] In at least one aspect, desparse convolution block 130 is defined using Equation 1 below, where y = [y hw :h∈{0,…,H-1},w∈{0,…,W-1}] (e.g., where y is a vector with dimensions H×W×C out The output tensor 135), W ij is the weight tensor 125, x hw is the input tensor 105, σ represents the application of the activation function 110 (thereby generating the activation tensor 115), and D(·) and D M (·) is a pair of desparse selection operations, where D M (·) Selects columns of the input matrix based on a mask reflecting the indices of the non-sparse entries of the tensor, as identified by D(·).

[0039] y=∑ i,j D M (W ij )D(σ(x hw,ij ))over(h,w) (1)

[0040] As discussed above, by using the desparse convolution block 130, the convolution bypass 120 is able to perform a convolution that is mathematically identical to a conventional convolution while using significantly reduced computational resources. This can reduce power requirements, improve computational throughput, reduce memory and processing requirements, and maintain the accuracy of the underlying machine learning model. Additionally, since the desparse convolution block 130 can be used whenever a conventional convolution is used, the desparse convolution block 130 can be used during training of the machine learning model (e.g., during a forward pass of training through the model), during inference after the model has been trained, and the like.

[0041] Example workflow for generating convolution output using desparse convolutions

[0042] Figure 2 An example workflow 200 for generating a convolution output using a desparse convolution on a potentially sparse activation tensor is depicted. In at least one aspect, the workflow 200 is performed by a convolution system such as Figure 1 The convolution bypass 120 of is performed. As discussed above, the workflow 200 can generally be used to replace any conventional convolution (including standard and point-by-point convolution) and replace the fully connected layer. In addition, the workflow 200 can be used during training a model, using a model for inference, or both.

[0043] As shown in the figure, the receiving in The input tensor 202 (also referred to in some aspects as an activation tensor) is used to generate an input tensor having a dimension C out ×C in The weight tensor 214 (also referred to in some aspects as a weight matrix) of the input tensor 202 is convolved. In the illustrated example, stippling is used to indicate non-sparse elements (e.g., non-zero values) of the data in the input tensor 202, and white or blank elements indicate sparse values ​​(e.g., zero values) in the input tensor 202. The weight tensor 214 does not include such stippling because the workflow 200 involves identifying sparsity in activations rather than sparsity in weights. In the illustrated example, some stippling is used in the weight values ​​(e.g., at 218 and 222) to indicate which weights are selected for convolution. It should be understood that both the selected and unselected weight values ​​may or may not include sparsity (e.g., zero values) values ​​because the selection is driven based on the sparsity in the input tensor 202.

[0044] The illustrated workflow 200 uses two loops (inner loop 205 and outer loop 229 indicated by dashed rectangles) to generate a outoutput tensor 244. In general, the inner loop 205 corresponds to processing a single spatial element of the input tensor 202 (e.g., an (h, w) index in the spatial dimensions, across all channels in the input tensor 202). This M*N inner loop 205 may be executed to apply a kernel of size M×N once. That is, assume that the convolution kernel used to generate the output tensor 244 is a 3×3 kernel. In the illustrated workflow 200, the inner loop 205 is repeated nine times, once for each spatial value / vector included in the receptive domain of the kernel (e.g., once for each spatial element in the patch of the input tensor 202 being convolved with the kernel). That is, each iteration of the inner loop 205 may correspond to the multiplication between a single spatial element / vector in the input tensor 202 and a corresponding weight in the weight tensor 214. If the convolution is a point-by-point convolution, the inner loop 205 is executed only once (because there is only one spatial element in the receptive domain of the kernel). In general, a kernel (also referred to in some aspects as a filter) corresponds to all or a portion of a weight tensor 214 (e.g., weight tensor 214 stores weights for one or more kernels). By repeating inner loop 205 (where, for each iteration, corresponding elements in a patch of input tensor 202 are multiplied by corresponding elements in the weight tensor), the system can generate an output for a single application of the kernel at a single location (e.g., without striding).

[0045] Additionally, in the illustrated workflow 200, each iteration of the outer loop 229 corresponds to a respective application of the kernel (e.g., for a given receptive field). That is, the first iteration corresponds to the application of the kernel at a first position (covering the first receptive field), while the second iteration may correspond to the application of the kernel at a second position (e.g., covering the second receptive field after striding the kernel). In one aspect, the outer loop 229 may be performed J times, where J is the number of times the kernel is applied during the convolution. In various aspects, this value may vary depending on the size of the input tensor 202 and the output tensor 244, the stride of the convolution, whether padding is used, etc. For example, for a point-by-point convolution using a stride of one (e.g., using a 1×1 kernel), the outer loop 229 may be repeated for up to H*W iterations, where H and W are the height and width of the input tensor 202, respectively.

[0046] In the illustrated workflow 200, for each iteration of the inner loop 205, the input tensor 202 undergoes a slicing operation 204 to produce a vector 206. Although the illustrated example depicts a slicing "operation" 204, in some aspects, the system does not use a discrete operation, but rather accesses or processes the input tensor 202 only in a sliced ​​manner (e.g., where the vector 206 is a slice of the input tensor 202). The vector 206 generally corresponds to a single spatial element across all channels of the input tensor 202 (e.g., a single index in the spatial dimensions of the input tensor 202). For example, for one iteration of the inner loop 205, the vector 206 may correspond to the lower left element (across all channels) of the input tensor 202. For subsequent iterations, the vector 206 may be the lower right element across all channels (any number of iterations occurring between them).

[0047] As illustrated in operation 208, the convolution system applies a sparse lookahead operation to identify sparse and / or non-sparse elements in vector 206. In general, operation 208 may be performed using any suitable technique, including via one or more shifters or switches (see below). Figure 4 Discussed in more detail) using in-place operations (e.g., ignoring sparse elements and applying non-sparse elements directly to convolution hardware, such as multiplication and accumulation circuits), using first-in-first-out (FIFO) circuits (e.g., ignoring sparse elements and inserting non-sparse elements into the FIFO), etc.

[0048] The sparse lookahead operation 208 produces a reduced vector 210 corresponding to the non-sparse elements from the vector 206. As described above, the convolution system can efficiently perform convolutions by operating only on these non-sparse elements, ignoring the processing of sparse elements. As illustrated by operation 209, the sparse lookahead operation 208 is also used to indicate the corresponding set of associated weights for the non-sparse elements. In the illustrated example, the weight tensor 218 includes stippling to indicate these associated weights.

[0049] In various aspects, the convolution system may use any suitable technique to indicate the associated weights. In one aspect, operation 209 indicates a set of offsets or indices for non-sparse elements in vector 206. For example, sparse lookahead operation 208 may generate a set of offsets indicating that non-sparse elements are located at indices [0, 2, 5, and 7] (indicated by stippling) in vector 206 (and / or corresponding weights are located at these indices). In some aspects, operation 209 includes a set of address offsets that the convolution system uses to retrieve the associated weights from a memory (e.g., from a tightly coupled memory (TCM) having a dedicated connection to a processing core of the system for generating the convolution output).

[0050] As illustrated by operation 220, using these offsets or indices, the convolution system extracts the relevant set of weights 222 from the weight tensor 218. In various aspects, the relevant set of weights 222 may be extracted using in-place operations (e.g., directly retrieving the relevant weights and applying these relevant weights to convolution hardware, such as multiplication and accumulation circuits), using first-in-first-out (FIFO) circuits (e.g., inserting the relevant weights into the FIFO and ignoring other weights), etc. At block 226, the non-sparse set of activations (in vector 210) and the relevant set of weights 222 are then multiplied to generate an intermediate tensor 230A. For example, block 226 may include performing an element-by-element multiplication of vector 210 and weight 222, followed by performing a corresponding accumulation (as part of a typical convolution operation). As discussed above, the inner loop 205 corresponds to processing a single spatial element in the input tensor 202. Each iteration of the inner loop 205 produces a corresponding intermediate tensor 230. As indicated by the ellipses, inner loop 205 is repeated until each element in the kernel has been applied to the corresponding element in input tensor 202 (e.g., repeated nine times for a 3×3 kernel), producing a set of intermediate tensors 230 (e.g., nine intermediate tensors 230 for a 3×3 kernel).

[0051] In workflow 200, at block 236, these intermediate tensors 230A-230N are then aggregated (e.g., using element-wise summation) to generate convolution output 240. Although not depicted in the illustrated example for conceptual clarity, in some aspects, prior to this summation, the convolution system expands the intermediate tensor 230 back to the appropriate depth (e.g., based on the index or offset determined in operations 208 and / or 209). That is, as part of the aggregation (or as part of or after the multiplication operation at block 226), the convolution system may generally expand the vectors as needed (by inserting zeros in the appropriate positions indicated via the offset or index) so that the intermediate tensor 230 has the appropriate depth C for the output tensor 244. out .

[0052] In general, the convolution output 240 corresponds to a single application of the convolution kernel. That is, the convolution output 240 is generated by convolving the convolution kernel with the input tensor 202 at a single position. Convolution typically involves striding the convolution kernel across the input tensor 202, thereby generating a corresponding convolution output for each position of the kernel. Therefore, in the illustrated example, the outer loop 229 is repeated for each such kernel position, generating a set of convolution outputs 240. For example, if the convolution is a point-by-point operation using a stride of one, the outer loop 229 is repeated H*W times, generating an equal number of convolution outputs 240. In general, the number of times the outer loop 229 is repeated is a function of the spatial dimensions of the input tensor 202, the dimensions of the kernel, the stride of the convolution operation, and the padding (if any) used during the convolution.

[0053] More generally, the system can compute J convolution outputs 240, where J is the number of times the kernel is applied to the input tensor 202 during the convolution, or the number of (h, w) positions to which the kernel is applied. Typically, for H×W kernels, J=H*W (assuming a stride of one, and appropriate zero padding).

[0054] In general, the number of operations required to perform a given desparse convolution operation may vary depending on the specific type of convolution. For example, for an input tensor H×W×C with a stride of one in , output tensor H×W×C out and has size M×N×C in For a standard convolution of a kernel of (which can be decomposed into M*N point-wise convolutions before summing), workflow 200 can use M*N iterations of inner loop 205 and H*W iterations of outer loop 229.

[0055] For an input tensor H×W×C with a stride of 1 in , output tensor H×W×C out and having a size of 1×1×C in For point-by-point convolution of a kernel of , workflow 200 may use one iteration of inner loop 205 and H*W iterations of outer loop 229 .

[0056] For fully connected operations, the input dimension can be converted to 1×1×C in , and the output data can be converted to 1×1×C out . With size 1×1×C in The core of may be used to replicate this operation (eg, with a value of one). In this regard, the workflow 200 may use 1 iteration of the inner loop 205 and 1 iteration of the outer loop 229.

[0057] In the depicted workflow 200, the convolution outputs 240 are then combined using a stacking operation 242 (or other operation) to generate a final output tensor 244 from the convolution operation. That is, the final output tensor 244 is typically created by aggregating (such as by stacking) the convolution outputs 240. In other words, the values ​​of the output tensor 244 are generated in multiple sets or segments, as reflected by the convolution outputs 240. As discussed above, the output tensor 244 is mathematically equivalent to the output generated using a conventional convolution. However, because the desparse convolution only operates on a subset of the input activations, it can be performed with significantly reduced computational cost and time.

[0058] Example workflow for pipelined desparse convolutions

[0059] Figure 3An example workflow 300 is depicted for pipelined desparse convolutions to further improve computational efficiency. In at least one aspect, workflow 300 is performed by a convolution system such as by using Figure 1 The convolution bypass 120 is performed.

[0060] The illustrated example depicts a set of operations required to perform a desparse convolution, where each box corresponds to a single operation at a single point in time (e.g., during a single clock cycle), and various densities of stippling are used to indicate which boxes are linked via data dependencies. In the illustrated workflow 300, arrows 345 indicate the flow of data through the pipeline for one convolution cycle (e.g., for one iteration of the inner loop 205).

[0061] For example, at block 350, the convolutional system may first use operation 305 (labeled "Sparse Activation Lookahead") to perform sparse lookahead in the activation data. In one aspect, this operation 305 corresponds to Figure 2 Operation 208.

[0062] In the next clock cycle, at block 355, operation 310 (labeled "Update weight addresses") is performed using this sparse information (e.g., a set of indices or offsets) to update the weight addresses to point to the relevant set of weights corresponding to the non-sparse activations. In some aspects, the convolution system may fetch and / or process multiple such weights in parallel (e.g., using single instruction multiple data (SIMD) instructions). In one aspect, operation 310 corresponds to Figure 2 Operation 209.

[0063] In a subsequent clock cycle, at block 360, these weights and non-sparse activations may be processed using operation 315 (labeled "matrix multiply accumulate"), where the non-sparse activations are multiplied by the corresponding weights, as discussed above. In one aspect, this operation 315 corresponds to Figure 2 226 and / or 236 of FIG. 226 and / or 236 of FIG. 236. In the illustrated example, operations 310 and 315 may be repeated one or more times to sweep through all non-sparse elements in the activation tensor (or relevant portions of the activation tensor, such as portions extracted by the slicing operation discussed above). At block 365, the convolution system applies a final operation 320 (labeled “Extract Accumulate”) in which the accumulated products are extracted from the accumulator into one or more registers. In many cases, the accumulator has a wider bit width than the operands of the multiplication (e.g., 40 bits) than ordinary registers (which may be, for example, eight bits wide or sixteen bits wide). In one aspect, operation 320 corresponds to Figure 2 That is, block 365 may represent a stacking operation 242. Figure 2 229 iterations of the outer loop.

[0064] As shown, the operations can be efficiently pipelined to allow the convolution system to efficiently perform desparse convolutions.

[0065] Example architecture for sparse lookahead

[0066] Figure 4 An example architecture 400 is depicted to perform sparse lookahead using a set of shifters. The architecture 400 depicts a de-thinning switch including a set of parallel shifters.

[0067] In the illustrated aspect, activation set 405 is used as input to the desparse switch via blocks 410A-410H. Specifically, input block 410A receives a first activation (labeled "A"), input block 410B receives a second activation (labeled "B"), and so on. As described above, activation 405 generally corresponds to all or a portion of an activation tensor. For example, activation 405 may correspond to Figure 1 A portion of the activation tensor 115 and / or Figure 2 206 in . In the illustrated example, stippling is used to indicate non-sparse activations 405 (e.g., activations having non-zero values), while sparse activations (e.g., those having values ​​of zero) are not stippled. As discussed above, the sparsity of activations 405 is not known a priori and must be determined dynamically.

[0068] Although the illustrated example includes eight input boxes 410, in various aspects, there may be any number of input boxes depending on the specific implementation and architecture. As shown, the desparse switch also includes a set of shifters 420A-420D. In some aspects, these shifters 420 may correspond to hardware components (e.g., shift registers), software components, or a combination of hardware and software. Although four shifters are included for illustration, there may be any number of shifters depending on the specific implementation and architecture. In general, changing the number of input boxes 410 and / or shifters 420 may affect the amount of sparsity handled by the system per cycle. For example, additional input boxes 410 may allow the system to desparse more activation tensors at a time. Additional shifters 420 may allow the system to more easily handle denser (e.g., less sparse) activations.

[0069] In some aspects, the number of shifters 420 is selected to be half the number of input boxes 410. This allows the system to efficiently handle up to fifty percent sparsity (e.g., where approximately half of the values ​​are zero), which enables efficient convolution with very few additional switching components. As discussed above, many common activation functions such as ReLU produce about 50% sparsity. Therefore, by using a 2:1 ratio of input box 410 to shifter 420, the de-sparse switch can efficiently extract non-sparse elements. Although additional shifters 420 can achieve more efficient handling of higher density inputs, they also involve additional costs (e.g., due to the need for additional hardware, higher chip area, increased power requirements, etc.). In various aspects, the specific number and ratio of shifters 420 to input box 410 may vary according to specific implementations and architectures.

[0070] In the illustrated architecture 400, there is a connection set 415 or link from input box 410 to shifter 420. Specifically, each input box 410 can provide input to one or more shifters 420, and each shifter 420 can receive input from multiple input boxes 410. In the depicted example, each shifter 420 covers a look-ahead span of four. That is, each shifter 420 is linked to a set of corresponding four input boxes 410, and can receive activation 405 from this range of four input boxes 410. Specifically, first shifter 420A is linked to input box 410A, 410B, 410C and 410D. Second shifter 420B is linked to input box 410B, 410C, 410D and 410E. As indicated by the ellipse, the remaining connection 415 (to shifters 420C and 420D) is omitted for clarity of description. In some aspects, shifter 420C is linked to input blocks 410C, 410D, 410E, and 410F. In some aspects, shifter 420D is linked to input blocks 410E, 410F, 410G, and 410H. In some aspects, the final shifter (420D) may have a larger look-ahead span (e.g., a span of five). For example, shifter 420D may be connected to input blocks 410D, 410E, 410F, 410G, and 410H. In various aspects, the look-ahead span is a design choice that may be different depending on the underlying architecture, desired sparsity support, etc.

[0071] In general, each shifter 420 is configured to extract or select the leftmost (e.g., with the lowest index) non-sparse activation 405 from its input set and pass the index of the selected non-sparse element to its neighbor. For example, shifter 420A selects the lowest index non-sparse activation from input boxes 410A-410D and passes that index (or the index plus one) to shifter 420B. In one aspect, shifter 420B then selects the lowest index non-sparse activation having an index greater than the index used by shifter 420A.

[0072] In the illustrated example, shifter 420A selects activation "A" from input box 410A and passes this information to shifter 420B. Depending on the particular implementation, shifter 420A may pass an indication that it selected activation from input box 410A (e.g., passing index 0) and / or an indication that shifter 420B should begin searching at input box 410B (e.g., passing index 1). Shifter 420B will then select activation "C" from input box 410C. As shown, using this approach, shifter 420C selects activation "F" from input box 410F, and shifter 420D selects activation "H" from input box 410H.

[0073] As shown, these selected (non-sparse) activations may then be provided to downstream components for processing, such as into time slots 430A-430D (e.g., to form Figure 2 210). For example, the shifter 420 may input the selected activations into a FIFO, or may provide them directly as input to one or more components (e.g., MAC circuits) that perform the convolution. As discussed above, these indices or offsets may also be used to extract the relevant set of weights. In addition, in some aspects, the indices or offsets may be used to expand the intermediate tensor as needed.

[0074] In various aspects, the desparse switch can efficiently handle both dense activations and sparse activations. In the case of particularly dense activations (e.g., in the case where there are more non-sparse activations than shifters 420), the system can simply start the next iteration or cycle at the relevant activation. For example, if activations "A", "B", "C", and "D" are all non-zero, the system can initiate a subsequent iteration by providing the next activation ("E") to input box 410A. In one aspect, if such dense activations are expected, the system can include additional shifters 420.

[0075] Similarly, in the case of particularly sparse activations (e.g., in the case where there are fewer non-zero activations than shifter 420), the shifter may simply select a zero-valued activation if no non-sparse activations are available within the corresponding input range. For example, assume that activations "B," "C," "D," and "E" are all zero. In one aspect, shifter 420B may select any of these zero values, including selecting the leftmost value as the default value (e.g., selecting activation "B" from input box 410B), or selecting the rightmost value as the default value (e.g., selecting activation "E" from input box 410E). Shifter 420B may then indicate that shifter 420C should start at input box 410F. Although this may introduce some computational inefficiencies (due to processing sparse elements), the de-sparse convolution may still proceed as usual. In some aspects, increasing the look-ahead span of the shifter may reduce this inefficiency with particularly sparse activations.

[0076] Example method for generating output using desparse convolution

[0077] Figure 5 An example flow chart illustrating a method 500 for generating an output using desparse convolution is depicted. In at least one aspect, the method 500 is performed by a convolution system such as by using Figure 1 In one aspect, the method 500 is Figure 1 The workflow 100 provides additional details.

[0078] At box 505, the convolution system receives pre-activation data in the form of a pre-activation tensor. In one aspect, this data is referred to as "pre-activation" to indicate that the data will be processed using one or more activation functions before convolution in a given layer (but possibly after convolution in a previous layer). For example, the pre-activation data may be output by a layer (e.g., a convolutional layer) of a neural network, where the model is configured to apply activation functions and then supply the results to subsequent layers. Although method 500 refers to pre-activation data as an example, in some aspects, the input data may be simply input data, regardless of whether an activation function will be applied. For example, method 500 (or a subset thereof) may be performed in the first layer of a neural network, allowing de-sparse convolutions to be applied to data input to the model.

[0079] At block 510, the convolution system applies an activation function to the input data to generate an activation tensor (e.g., a tensor of activation data). For example, the data may be processed using a ReLU activation function to introduce nonlinearity to the machine learning model. As discussed above, such activation functions significantly improve the accuracy of models for a variety of tasks. However, such activation functions typically introduce at least some degree of sparsity (due to setting at least some non-zero values ​​to zero). In general, although these sparse elements have no effect on the output of each convolution operation (because multiplication by zero produces an output of zero), conventional systems still perform this (meaningless) multiplication, thereby incurring significant computational expense in performing irrelevant calculations.

[0080] At block 515, the convolution system applies a desparse convolution operation to the activation tensor instead of performing such a conventional convolution. Figure 2 An example technique for such a desparse convolution is discussed. In general, a desparse convolution involves generating a convolution output based on a non-sparse activation while disregarding multiplication or convolution of one or more sparse activations in a tensor.

[0081] At block 520, the convolution system outputs the convolution output. In various aspects, outputting the convolution output may include providing it as an input to an activation function, providing it as an input to a subsequent layer of a neural network, and the like.

[0082] As discussed above, this de-sparse convolution can significantly improve computational efficiency by reducing the number of operations required to perform the convolution, reducing the required memory footprint, reducing the energy expended, reducing the required computation time, etc.

[0083] Furthermore, because the desparse convolution technique described herein is mathematically equivalent to a regular convolution (because the output generated is the same as the output of a regular convolution), the convolution system can use this desparse convolution at any stage where a regular convolution can be applied, including in ordinary (full) convolutions, in point-by-point convolutions, in depth-separable convolutions, in fully connected layers, etc. Furthermore, the desparse convolution technique can be easily applied both during training (e.g., during forward passing of training data through the model) and during inference (e.g., when processing new data to generate outputs), and has no adverse effect on the accuracy of the model.

[0084] It should be noted that Figure 5 This is merely one example of a method, and other methods including fewer, additional, or alternative steps may also be consistent with the present disclosure.

[0085] Example method for desparse convolution

[0086] Figure 6An example flow chart illustrating a method 600 for desparse convolution is depicted. In at least one aspect, the method 600 is performed by a convolution system such as by using Figure 1 In one aspect, the method 600 is Figure 2 Workflow 200 provides additional details.

[0087] At block 605, the convolution system selects a first kernel position for performing a convolution. As discussed above, convolutions typically involve applying a convolution kernel (e.g., a tensor specifying a set of weights) to an input tensor at multiple positions (e.g., an activation tensor generated using an activation function) to generate corresponding outputs. The use of multiple kernel positions is conventionally referred to as "striding" the kernel across the input, and the kernel stride is typically a configurable hyperparameter of the convolution.

[0088] In the illustrated example, at block 605, the convolution system selects one such position to generate the corresponding output. In general, the specific method used to select this position may vary, as the convolution system will process each such position during the convolution operation. Additionally, although the illustrated example depicts an iterative process for conceptual clarity (e.g., processing each kernel position sequentially), in some aspects, some or all of these kernel positions may actually be processed in parallel.

[0089] At block 610, the convolution system extracts or otherwise accesses a slice from the input tensor. For example, as discussed above, the convolution system may use a slice operation (e.g., Figure 2 slicing operation 204) to extract a vector with a height and width of one and a depth equal to the number of input channels (e.g., Figure 2 That is, the convolution system may extract or otherwise access a set of values ​​corresponding to the elements of each channel at a single index in the height and width dimensions.

[0090] In one aspect, a slice is extracted from within the current extent of the kernel (determined based on the kernel size and the position determined at block 605). In various aspects, the specific method used to select / extract a slice from the extent may vary, as the convolution system will typically iterate over all relevant slices (e.g., all input slices involved in the convolution) during method 600. Although a sequential process is depicted for conceptual clarity, in some aspects, all or a portion of the slices may be extracted and processed in parallel.

[0091] At block 615, the convolution system identifies the set of non-zero elements in the extracted slice. For example, as described above with respect to Figure 4As discussed, the convolution system may use a desparse switch to efficiently identify and extract non-sparse elements, thereby ignoring any sparse (e.g., zero-valued) elements. In some aspects, the convolution system may extract these elements and apply them directly as input to downstream components (e.g., to a MAC array and / or Figure 2 In some aspects, the convolution system inputs the elements into a FIFO or queue.

[0092] Additionally, in at least one aspect, the convolution system also determines or provides data for identifying the locations of these non-sparse elements. For example, the convolution system can determine their index in the slice (e.g., which channel they belong to), etc.

[0093] At block 620, the convolution system identifies the corresponding weight element for the non-sparse element. In one aspect, the convolution system does this using the index or address offset determined at block 615, as discussed above. For example, if the non-sparse activation elements are located in the first and fourth positions of the slice, the convolution system may determine that the corresponding weight is located at a base address (e.g., the beginning of the weight tensor) plus zero (for the first element) and, for the fourth element, plus three (e.g., plus three bytes, words, or other amount depending on the particular implementation).

[0094] In some aspects, the convolution system may identify and extract these weight elements, inputting them directly to downstream components (such as a MAC array). In some aspects, as discussed above, the convolution system may write the relevant weights to a FIFO or queue, as discussed above. They may then be input to downstream processing components.

[0095] At block 625, the convolution system generates an intermediate tensor for the slice based on the non-sparse activation elements and corresponding weights. For example, using components such as Figure 2 At block 226 , the convolution system may multiply the activation elements and the weight elements (eg, using convolution, element-wise multiplication, etc.) to generate an intermediate tensor 230 .

[0096] In some aspects, generating the intermediate tensor further comprises: expanding the resulting output tensor back to the original depth, as discussed above. For example, using the aforementioned indexes or offsets, the convolution system can insert elements with values ​​of "zero" as needed (e.g., skipping zero elements anywhere in the slice) to ensure that non-sparse elements are in the correct locations in the intermediate tensor.

[0097] At block 630, the convolution system determines whether there is at least one additional slice remaining for processing in the current range covered by the kernel. For example, if the kernel is a point-by-point kernel, the convolution system may determine that no additional slices are needed because the kernel only covers a single (spatial) element at a time. If the kernel is a standard convolution (e.g., 3×3) kernel, the convolution system may determine whether each of the (nine) related spatial elements has been processed. If at least one such slice remains, the method 600 returns to block 610, as described above with reference to FIG. Figure 2 and inner loop 205 discussed.

[0098] If all such slices have been processed, the method 600 continues to block 635. At block 635, the convolution system aggregates the intermediate tensor sets (generated for each slice using blocks 615-625) to generate the convolution output tensor. For example, as described above with reference to Figure 2 As discussed, the convolution system may aggregate intermediate tensors using element-wise summation, producing a single tensor (e.g., convolution output 240) corresponding to a single application of the kernel.

[0099] The method 600 then proceeds to block 640, where the convolution system determines whether the convolution is complete. For example, as discussed above, the convolution may include multiple applications of the kernel (e.g., including striding the kernel across the input tensor). If the convolution is not complete (e.g., because there is at least one additional kernel position that must be used to compute the output of the convolution), the method 600 returns to block 605 to select the next kernel position, as described above with reference to Figure 2 and outer loop 229. Although depicted as a sequential process for conceptual clarity, in some aspects, the convolution system can generate multiple convolution outputs (corresponding to multiple kernel positions) at least partially in parallel.

[0100] If the convolution has ended (e.g., if all kernel positions have been processed), the method 600 continues to block 645, where the convolution system aggregates the convolution outputs. For example, as discussed above, the convolution system can stack the convolution outputs based on the kernel position used to generate each convolution output (e.g., placing the convolution outputs generated by applying the kernels in the leftmost and topmost positions into the resulting stack). In this manner, as discussed above, the convolution system is able to perform convolutions that are mathematically equivalent to conventional convolutions while using a reduced number of operations and, thereby, resulting in reduced computational cost.

[0101] At block 650, the convolution system outputs the aggregated (e.g., stacked) convolution output as the total output tensor of the convolution process. In various aspects, as discussed above, this output can be used for a variety of purposes, including as input to subsequent layers of a neural network, as input to an activation function, as output from the last layer of a neural network, etc. Additionally, as discussed above, the desparse convolution process can be generally applied to a wide variety of use cases, including both point-by-point convolution and full convolution, as well as during training and during inference.

[0102] It should be noted that Figure 6 This is merely one example of a method, and other methods including fewer, additional, or alternative steps may also be consistent with the present disclosure.

[0103] Example method for generating convolution output using desparse convolution

[0104] Figure 7 An example flow chart illustrating a method 700 for generating a convolution output using desparse convolution is depicted. In at least one aspect, the method 700 is performed by a convolution system such as by using Figure 1 The convolution bypass 120 is performed.

[0105] At block 705 , an activation tensor is received.

[0106] At block 710, a first subset of weight elements corresponding to a first set of non-zero elements in the activation tensor is selected from the weight tensor. In some aspects, the first subset of weight elements is a proper subset of the weight tensor (e.g., the first subset of weight elements excludes at least one element of the weight tensor).

[0107] At block 715, the first set of non-zero elements is multiplied by the first set of weight elements.

[0108] In some aspects, blocks 710 and 715 correspond to generating a first convolution output for the activation tensor.

[0109] In some aspects, method 700 also includes identifying a set of respective indexes, wherein each respective index in the set of respective indexes corresponds to a respective non-zero element in the first set of non-zero elements.

[0110] In some aspects, retrieving the first set of weight elements based on the corresponding set of indexes includes: determining a set of memory offsets based on the corresponding set of indexes; and retrieving the first set of weight elements from the memory using the set of memory offsets.

[0111] In some aspects, the memory is a tightly coupled memory (TCM) having a dedicated connection to a processing core for generating the first convolution output.

[0112] In some aspects, the first set of non-zero elements is identified within a first one-dimensional vector extracted from the activation tensor, and the first one-dimensional vector corresponds to a single spatial element in the activation tensor across the input channels.

[0113] In some aspects, the first convolution output corresponds to a first application of a kernel having height M and width N, and generating the first convolution output includes: identifying M*N sets of non-zero elements in the activation tensor; selecting M*N sets of weight elements corresponding to the M*N sets of non-zero elements; and generating an intermediate tensor set by multiplying each set of weight elements in the M*N sets of weight elements with a corresponding set of non-zero elements from the M*N sets of non-zero elements.

[0114] In some aspects, generating the first convolution output further comprises summing the set of intermediate tensors to produce the first convolution output.

[0115] In some aspects, method 700 also includes: generating a second convolution output corresponding to a second application of the kernel; and generating a total convolution output for the activation tensor by stacking the first convolution output and the second convolution output.

[0116] In some aspects, H*W convolution outputs are generated for the activation tensor, where H is the height of the activation tensor and W is the width of the activation tensor.

[0117] In some aspects, the first convolution output is generated as part of applying a convolution kernel comprising at least one of: a width dimension greater than one, or a height dimension greater than one.

[0118] In some aspects, the first convolution output is generated as part of applying a convolution kernel comprising: a width dimension equal to one, and a height dimension equal to one.

[0119] In some aspects, the first convolution output is generated as part of a training process of the neural network.

[0120] In some aspects, the first convolution output is generated as part of generating inference using the trained neural network.

[0121] It should be noted that Figure 7 This is merely one example of a method, and other methods including fewer, additional, or alternative steps may also be consistent with the present disclosure.

[0122] Example Processing System for Desparse Convolution

[0123] In some respects, reference Figures 1 to 7 The described workflows, techniques, and methods may be implemented on one or more devices or systems. Figure 8An example processing system 800 is depicted that is configured to perform various aspects of the present disclosure, including, for example, with respect to Figure 1 9. In one aspect, the processing system 800 may correspond to a method for using Figure 1 The convolution bypass 120 of the convolution system, as discussed above. In at least some aspects, as discussed above, the operations described below with respect to the processing system 800 can be distributed across any number of devices. For example, one system can use desparse convolution to train a model, while a second system uses the trained model to perform inference using desparse convolution.

[0124] The processing system 800 includes a central processing unit (CPU) 802, which may be a multi-core CPU in some examples. Instructions executed at the CPU 802 may be loaded, for example, from a program memory associated with the CPU 802, or may be loaded from a memory partition (e.g., from memory 824).

[0125] The processing system 800 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 804 , a digital signal processor (DSP) 806 , a neural processing unit (NPU) 808 , a multimedia processing unit 810 , and wireless connectivity components 812 .

[0126] An NPU, such as 808, is typically a dedicated circuit configured to implement all necessary control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. An NPU is sometimes alternatively referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), a vision processing unit (VPU), or a graphics processing unit.

[0127] NPUs such as 808 are configured to accelerate the execution of common machine learning tasks such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip such as a system on a chip (SoC), while in other examples, multiple NPUs may be part of a dedicated neural network accelerator.

[0128] The NPU can be optimized for training or inference, or in some cases configured to balance performance between the two. For NPUs that can perform both training and inference, the two tasks can generally still be performed independently.

[0129] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing data set (usually labeled or tagged), iterating on the data set, and then adjusting model parameters (such as weights and biases) to improve model performance. Generally speaking, optimization based on error predictions involves passing back through the layers of the model and determining the gradient to reduce the prediction error.

[0130] NPUs designed to accelerate inference are generally configured to operate on complete models. Thus, such NPUs can be configured to input new data slices and quickly process them through the trained model to generate model outputs (e.g., inferences).

[0131] In one specific implementation, the NPU 808 is part of one or more of the CPU 802 , the GPU 804 , and / or the DSP 806 .

[0132] In some examples, the wireless connectivity component 812 may include, for example, subcomponents for third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity component 812 is further connected to one or more antennas 814.

[0133] The processing system 800 may also include one or more sensor processing units 816 associated with any manner of sensors, one or more image signal processors (ISPs) 818 associated with any manner of image sensors, and / or a navigation processor 820, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0134] The processing system 800 may also include one or more input and / or output devices 822, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, speakers, microphones, and the like.

[0135] In some examples, one or more of the processors of processing system 800 may be based on the ARM or RISC-V instruction set.

[0136] The processing system 800 also includes a memory 824, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 824 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 800.

[0137] Specifically, in this example, memory 824 includes a sparse component 824A, a weight component 824B, an aggregation component 824C, a training component 824D, and an inference component 824E. Memory 824 also includes a model parameter set 824F (which may correspond to the parameters (e.g., weights and / or biases) of the machine learning model discussed above). The depicted components, as well as other components not depicted, may be configured to perform various aspects of the techniques described herein. Although for conceptual clarity, the description of FIG. Figure 8 824A, weight component 824B, aggregation component 824C, training component 824D, and inference component 824E may be implemented jointly or separately.

[0138] Processing system 800 also includes sparsification circuitry 826, weight circuitry 827, aggregation circuitry 828, training circuitry 829, and inference circuitry 830. The depicted circuitry, as well as other circuitry not depicted, may be configured to perform various aspects of the techniques described herein.

[0139] For example, the sparse component 824A and the sparse circuit 826 may be used to extract slices of the input tensor and identify non-sparse activation elements (and in some aspects, identify indices or address offsets of these non-sparse elements), as discussed above. The weight component 824B and the weight circuit 827 may be used to extract corresponding weights for these non-sparse elements (e.g., from the memory 824 and / or from the TCM), as discussed above. The aggregation component 824C and the aggregation circuit 828 may generally be used to aggregate various intermediate tensors and convolution outputs (e.g., summing the intermediate tensors 230 and / or stacking the convolution outputs 240, as discussed above with reference to Figure 2 The training component 824D and the training circuit 829 can be used to control the training, refinement and / or fine-tuning of various machine learning models using desparse convolutions, as discussed above. The inference component 824E and the inference circuit 830 can generally use the trained machine learning model to generate inferences or predictions using desparse convolutions, as discussed above.

[0140] Although for the sake of clarity Figure 8 Although depicted as separate components and circuits, the sparse circuit 826, the weight circuit 827, the aggregation circuit 828, the training circuit 829, and the inference circuit 830 may be implemented collectively or individually in other processing devices of the processing system 800, such as in the CPU 802, the GPU 804, the DSP 806, the NPU 808, etc.

[0141] Generally speaking, the processing system 800 and / or its components may be configured to perform the methods described herein.

[0142] It is worth noting that in other aspects, aspects of the processing system 800 may be omitted, such as where the processing system 800 is a server computer, etc. For example, in other aspects, the multimedia processing unit 810, the wireless connectivity component 812, the sensor processing unit 816, the ISP 818, and / or the navigation processor 820 may be omitted. Furthermore, aspects of the processing system 800 may be distributed among multiple devices.

[0143] Sample Clauses

[0144] Clause 1: A method comprising: receiving an activation tensor; and generating a first convolution output for the activation tensor, comprising: identifying a first set of non-zero elements in the activation tensor; selecting a first set of weight elements from a weight tensor corresponding to the first set of non-zero elements; and multiplying the first set of non-zero elements with the first set of weight elements.

[0145] Clause 2: A method comprising: receiving an activation tensor; and generating a first convolution output for the activation tensor, comprising: selecting a first subset of weight elements from a weight tensor corresponding to a first set of non-zero elements in the activation tensor; and multiplying the first set of non-zero elements with the first set of weight elements.

[0146] Clause 3: According to the method described in any one of clauses 1 to 2, the method further includes: identifying a corresponding index set, wherein each corresponding index in the corresponding index set corresponds to a corresponding non-zero element in the first set of non-zero elements, wherein selecting the first set of weight elements includes: retrieving the first set of weight elements based on the corresponding index set.

[0147] Clause 4: A method according to any one of clauses 1 to 3, wherein retrieving the first set of weight elements based on the corresponding index set comprises: determining a memory offset set based on the corresponding index set; and using the memory offset set to retrieve the first set of weight elements from the memory.

[0148] Clause 5: The method of any of clauses 1 to 4, wherein the memory is a tightly coupled memory (TCM) having a dedicated connection to a processing core for generating the first convolution output.

[0149] Clause 6: A method according to any one of clauses 1 to 5, wherein: the first set of non-zero elements is identified within a first one-dimensional vector extracted from the activation tensor, and the first one-dimensional vector corresponds to a single spatial element in the activation tensor across input channels.

[0150] Clause 7: A method according to any one of clauses 1 to 6, wherein: the first convolution output corresponds to a first application of a kernel having a height M and a width N, and generating the first convolution output includes: identifying M*N sets of non-zero elements in the activation tensor; selecting M*N sets of weight elements corresponding to the M*N sets of non-zero elements; and generating an intermediate tensor set by multiplying each set of weight elements in the M*N sets of weight elements with a corresponding set of non-zero elements from the M*N sets of non-zero elements.

[0151] Clause 8: A method according to any one of clauses 1 to 7, wherein generating the first convolution output further comprises: summing the set of intermediate tensors to produce the first convolution output.

[0152] Clause 9: The method of any one of clauses 1 to 8, further comprising: generating a second convolution output corresponding to a second application of the kernel; and generating a total convolution output for the activation tensor by stacking the first convolution output and the second convolution output.

[0153] Clause 10: A method according to any one of clauses 1 to 9, wherein: H*W convolution outputs are generated for the activation tensor, H is the height of the activation tensor, and W is the width of the activation tensor.

[0154] Clause 11: A method according to any of clauses 1 to 10, wherein the first convolution output is generated as part of applying a convolution kernel comprising at least one of: a width dimension greater than one, or a height dimension greater than one.

[0155] Clause 12: A method according to any of clauses 1 to 11, wherein the first convolution output is generated as part of applying a convolution kernel comprising: a width dimension equal to one and a height dimension equal to one.

[0156] Clause 13: A method according to any one of clauses 1 to 12, wherein the first convolution output is generated as part of a training process of a neural network.

[0157] Clause 14: A method as described in any of clauses 1 to 13, wherein the first convolution output is generated as part of generating inference using a trained neural network.

[0158] Clause 15: A system comprising: a memory including computer executable instructions; and one or more processors configured to execute the computer executable instructions and cause the processing system to perform the method according to any one of clauses 1 to 14.

[0159] Clause 16: A system comprising: means for performing the method according to any one of clauses 1 to 14.

[0160] Clause 17: A non-transitory computer readable medium comprising: computer executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method of any one of clauses 1 to 14.

[0161] Clause 18: A computer program product embodied on a computer readable storage medium, comprising: code for performing the method according to any one of clauses 1 to 14.

[0162] Additional considerations

[0163] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein are not limited to the scope, applicability or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, without departing from the scope of the present disclosure, the functions and arrangements of the elements discussed may be changed. Each example may ignore, replace or add each procedure or component as appropriate. For example, the described method may be performed in a different order than described, and various steps may be added, omitted or combined. In addition, the features described with respect to some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover such devices or methods practiced using other structures, functionality, or structures and functionality that are supplemented or substituted for the various aspects of the present disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the present invention.

[0164] As used herein, the word “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0165] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items (including single members). As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple of the same elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0166] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, and the like. Furthermore, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Furthermore, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0167] As used herein, in the context of sharing electronic signals and data between the various elements described herein, the term "connected to" generally refers to data communication between the corresponding elements connected to each other. In some cases, the elements can be directly connected to each other, such as via one or more conductive traces, lines or other conductive carriers that can carry signals and / or data between the corresponding elements directly connected to each other. In other cases, the elements can be indirectly connected to each other, such as via one or more data buses or similar shared circuits and / or integrated circuit elements for conveying signals and data between the corresponding elements indirectly connected to each other.

[0168] The method disclosed herein includes one or more steps or actions for implementing the method. The method steps and / or actions can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions can be modified without departing from the scope of the claims. In addition, the various operations of the method described above can be performed by any appropriate component capable of performing the corresponding function. The component may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors. Generally speaking, in the case of operations illustrated in the accompanying drawings, those operations may have corresponding corresponding components with similar numbers plus functional components.

[0169] The following claims are not intended to be limited to the various aspects shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless otherwise specified, reference to an element in the singular form is not intended to mean "one and only one", but "one or more". Unless otherwise specified, the term "some" refers to one or more. No claim element should be interpreted according to the provisions of 35 USC § 112 (f) unless the phrase "parts for..." is used to explicitly describe the element, or in the case of a method claim, the phrase "step for..." is used to describe the element. All structural and functional equivalents of the elements of the various aspects described throughout the present disclosure that are known or will be known later to a person of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be dedicated to the public, regardless of whether such disclosure is explicitly recorded in the claims.

Claims

1. A processor-implemented method, comprising: Receives an activation tensor; as well as Generating a first convolution output for the activation tensor, including: selecting from the weight tensor a first subset of weight elements corresponding to a first set of non-zero elements in the activation tensor; as well as The first set of non-zero elements is multiplied by the first subset of weight elements.

2. The method according to claim 1, further comprising: Identifies the corresponding index set, where: Each respective index in the set of respective indices corresponds to a respective non-zero element in the first set of non-zero elements, and Selecting the first subset of weight elements includes retrieving the first subset of weight elements based on the corresponding set of indexes.

3. The processor-implemented method of claim 2 , wherein retrieving the first subset of weight elements based on the corresponding set of indexes comprises: determining a set of memory offsets based on the corresponding set of indexes; as well as The first subset of weight elements is retrieved from memory using the set of memory offsets.

4. The processor-implemented method of claim 1 , wherein: The first set of non-zero elements is identified within a first one-dimensional vector extracted from the activation tensor, and The first one-dimensional vector corresponds to a single spatial element in the activation tensor across input channels.

5. The processor-implemented method of claim 4, wherein: The first convolution output corresponds to a first application of a kernel having height M and width N, and Generating the first convolution output includes: Identify a set of M*N non-zero elements in the activation tensor; Selecting a subset of M*N weight elements corresponding to the set of M*N non-zero elements; and A set of intermediate tensors is generated by multiplying each subset of weight elements in the M*N subsets of weight elements with a corresponding set of non-zero elements from the M*N sets of non-zero elements.

6. The processor-implemented method of claim 5, wherein generating the first convolution output further comprises: The set of intermediate tensors are summed to produce the first convolution output.

7. The processor-implemented method of claim 5 , further comprising: generating a second convolution output corresponding to a second application of the kernel; as well as A total convolution output is generated for the activation tensor by stacking the first convolution output and the second convolution output.

8. The processor-implemented method of claim 7, wherein: H*W convolution outputs are generated for the activation tensor, H is the height of the activation tensor, and W is the width of the activation tensor.

9. The processor-implemented method of claim 1 , wherein the first convolution output is generated as part of applying a convolution kernel comprising at least one of: a width dimension greater than one, or A height dimension greater than one.

10. The processor-implemented method of claim 1 , wherein the first convolution output is generated as part of applying a convolution kernel comprising: a width dimension equal to one, and A height dimension equal to one.

11. The processor-implemented method of claim 1 , wherein the first convolution output is generated as part of a training process of a neural network.

12. The processor-implemented method of claim 1, wherein the first convolution output is generated as part of generating inference using a trained neural network.

13. A system comprising: a memory including computer executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the system to perform operations comprising: receives an activation tensor; and Generating a first convolution output for the activation tensor, including: selecting from the weight tensor a first subset of weight elements corresponding to a first set of non-zero elements in the activation tensor; and The first set of non-zero elements is multiplied by the first subset of weight elements.

14. The system of claim 13, wherein the operations further comprise: Identifies the corresponding index set, where: Each respective index in the set of respective indices corresponds to a respective non-zero element in the first set of non-zero elements, and Selecting the first subset of weight elements includes retrieving the first subset of weight elements based on the corresponding set of indexes.

15. The system of claim 14, wherein retrieving the first subset of weight elements based on the corresponding set of indexes comprises: determining a set of memory offsets based on the corresponding set of indexes; as well as The first subset of weight elements is retrieved from memory using the set of memory offsets.

16. The system of claim 15, wherein the memory is a tightly coupled memory (TCM) having a dedicated connection to a processing core for generating the first convolution output.

17. The system of claim 13, wherein: The first set of non-zero elements is identified within a first one-dimensional vector extracted from the activation tensor, and The first one-dimensional vector corresponds to a single spatial element in the activation tensor across input channels.

18. The system of claim 17, wherein: The first convolution output corresponds to a first application of a kernel having height M and width N, and Generating the first convolution output includes: Identify a set of M*N non-zero elements in the activation tensor; Selecting a subset of M*N weight elements corresponding to the set of M*N non-zero elements; and A set of intermediate tensors is generated by multiplying each subset of weight elements in the M*N subsets of weight elements with a corresponding set of non-zero elements from the M*N sets of non-zero elements.

19. The system of claim 18, wherein generating the first convolution output further comprises: The set of intermediate tensors are summed to produce the first convolution output.

20. The system of claim 18, the operations further comprising: generating a second convolution output corresponding to a second application of the kernel; as well as A total convolution output is generated for the activation tensor by stacking the first convolution output and the second convolution output.

21. The system of claim 13, wherein the first convolution output is generated as part of applying a convolution kernel comprising at least one of: a width dimension greater than one, or A height dimension greater than one.

22. The system of claim 13, wherein the first convolution output is generated as part of applying a convolution kernel comprising: a width dimension equal to one, and A height dimension equal to one.

23. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors of a processing system, cause the processing system to perform operations comprising: Receives an activation tensor; as well as Generating a first convolution output for the activation tensor, including: selecting from the weight tensor a first subset of weight elements corresponding to a first set of non-zero elements in the activation tensor; as well as The first set of non-zero elements is multiplied by the first subset of weight elements.

24. The non-transitory computer readable medium of claim 23, the operations further comprising: Identifies the corresponding index set, where: Each respective index in the set of respective indices corresponds to a respective non-zero element in the first set of non-zero elements, and Selecting the first subset of weight elements includes retrieving the first subset of weight elements based on the corresponding set of indexes.

25. The non-transitory computer-readable medium of claim 24, wherein retrieving the first subset of weight elements based on the corresponding set of indexes comprises: determining a set of memory offsets based on the corresponding set of indexes; as well as The first subset of weight elements is retrieved from memory using the set of memory offsets.

26. The non-transitory computer readable medium of claim 23, wherein: The first set of non-zero elements is identified within a first one-dimensional vector extracted from the activation tensor, and The first one-dimensional vector corresponds to a single spatial element in the activation tensor across input channels.

27. The non-transitory computer readable medium of claim 26, wherein: The first convolution output corresponds to a first application of a kernel having height M and width N, and Generating the first convolution output includes: Identify a set of M*N non-zero elements in the activation tensor; Selecting a subset of M*N weight elements corresponding to the set of M*N non-zero elements; and A set of intermediate tensors is generated by multiplying each subset of weight elements in the M*N subsets of weight elements with a corresponding set of non-zero elements from the M*N sets of non-zero elements.

28. The non-transitory computer readable medium of claim 27, wherein generating the first convolution output further comprises: The set of intermediate tensors are summed to produce the first convolution output.

29. The non-transitory computer readable medium of claim 28, wherein generating the first convolution output further comprises: The set of intermediate tensors are summed to produce the first convolution output.

30. A system comprising: A component for receiving activation tensors; and The means for generating a first convolution output for the activation tensor comprises: means for selecting from the weight tensor a first subset of weight elements corresponding to a first set of non-zero elements in the activation tensor; and Means for multiplying said first set of non-zero elements with said first subset of weight elements.

Images