De-sparse convolution for sparse tensor

Abstract

Certain aspects of the present disclosure provide techniques for de-sparse convolution. A weight tensor having unstructured sparsity is accessed, and a densified weight tensor is generated based on the weight tensor by directionally extruding the weight tensor to remove sparse values and generating a sparse map based on the directional extrusion. And outputting the densified weight tensor and the sparse mapping for use in a convolutional neural network.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application serial number 17 / 932,527 filed on September 15, 2022, which is hereby incorporated by reference into this document.

[0003] introduction

[0004] Aspects of the present disclosure relate to efficient convolution operations.

[0005] Convolution has become an increasingly important operation for a wide variety of computational solutions, including in convolutional neural networks (which typically involve applying a large number of convolution operations to input data). Convolutional neural networks can be trained for countless tasks, such as computer vision (e.g., image or object recognition), audio processing, and the like. Generally speaking, a single convolution operation involves multiplying one or more portions of the input tensor by one or more weights in the convolution kernel, where the weights are learned during the training process. Conventional convolution operations and networks typically require a large number of such multiplications due to a number of factors including the size of the data tensor (e.g., the number of elements), the number of applications of each kernel, the number of kernels, the number of layers, and the like.

[0006] In general, training of a neural network involves iteratively refining parameters (e.g., weights) based on training data. In many cases, a significant portion of weights or other parameters reach negligible or nearly negligible values ​​(e.g., equal to or close to zero), and these weights typically have no effect (or very little effect) on the output of the network. In such systems, the weight tensor may be referred to as "sparse" (e.g., if a significant portion of the elements have a value of zero).

[0007] Therefore, techniques for improving convolutions using sparse weight tensors are needed. Summary of the invention

[0008] Certain aspects provide a method comprising: accessing a weight tensor having unstructured sparsity; generating a densified weight tensor based on the weight tensor, comprising: directionally squeezing the weight tensor to remove sparse values; and generating a sparse map based on the directed squeezing; and outputting the densified weight tensor and the sparse map for use in a convolutional neural network.

[0009] Certain aspects provide a method comprising: accessing an activation tensor for processing using a convolution operation; and performing the convolution operation, including: retrieving a densified weight tensor for the convolution operation; retrieving a sparse map associated with the densified weight tensor; generating a set of intermediate tensors by multiplying the densified weight tensor with the activation tensor; and generating an output tensor for the convolution operation by accumulating the set of intermediate tensors based on the sparse map.

[0010] Other aspects provide processing systems configured to perform the aforementioned methods and those described herein; non-transitory computer-readable media comprising instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the aforementioned methods and those described herein; a computer program product embodied on a computer-readable storage medium, the computer program product comprising code for performing the aforementioned methods and those further described herein; and a processing system comprising components for performing the aforementioned methods and those further described herein.

[0011] The following description and associated drawings set forth in detail certain illustrative features of the one or more aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings depict certain aspects of the one or more aspects and therefore are not to be considered limiting of the scope of the present disclosure.

[0013] Figure 1 Depicts an example workflow for densifying weight tensors to improve convolutions.

[0014] Figure 2A Depicted is an example workflow for generating a sparse map using absolute indications.

[0015] Figure 2B Depicted is an example workflow for generating a sparse map using relative indications.

[0016] Figure 3 Depicts an example workflow for performing convolutions and sparse accumulations using densified weight tensors.

[0017] Figure 4 An example flow chart illustrating a method for densifying a weight tensor is depicted.

[0018] Figure 5 An example flow chart illustrating a method for selectively densifying a weight tensor is depicted.

[0019] Figure 6 An example flow chart illustrating a method for performing desparse convolution using a densified weight tensor is depicted.

[0020] Figure 7 An example flow chart illustrating a method for densifying a tensor using directed extrusion is depicted.

[0021] Figure 8 An example flow chart illustrating a method for convolving with a densified weight tensor is depicted.

[0022] Fig. 9 An example processing system configured to perform various aspects of the present disclosure is depicted.

[0023] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one aspect may be beneficially incorporated in other aspects without further recitation. DETAILED DESCRIPTION

[0024] Aspects of the present disclosure provide techniques for improved tensor densification (alternatively referred to as desparseness) and improved convolutions (referred to herein as desparsed convolutions).

[0025] Conventional methods of convolution typically ignore weight sparsity and, for example, convolve all elements of the weight tensor with the input tensor, even if the convolution has elements with zero values ​​that have no effect on the output of the convolution. This wastes energy, time, and computational resources. Some conventional methods of dealing with weight sparsity rely on highly structured or semi-structured sparsity, where the allowable sparsity is strictly controlled (e.g., to perfectly distribute the sparsity to a defined portion of the weight tensor), which can be defined as elements with zero values ​​or within a defined distance from zero). Although this structured sparsity can achieve valuable benefits, it introduces artificial restrictions on sparsity and, therefore, may not provide optimal results and negatively impact model accuracy.

[0026] Additionally, conventional methods for handling weight sparsity often introduce significant overhead at inference time (e.g., due to unstructured model compression). In some cases, the added overhead may mask the benefits of compression, and in many cases, using an uncompressed model (even one with significant sparsity) is better than using conventional sparsity methods.

[0027] In various aspects of the present disclosure, parameter tensors (e.g., weights in a neural network) can be densified using directed squeezing to remove elements that meet a defined sparsity criterion (e.g., equal to a value of zero or within a threshold distance of a value of zero). In some aspects, this directed squeezing can effectively compress the weight tensor, thereby reducing the storage and memory footprint of the tensor, as well as reducing transmission overhead (e.g., if the tensor or model is sent over a network or over a local data bus). In addition, the densified tensor can be effectively used during runtime, thereby enabling fast and efficient convolutions. Notably, various aspects of the present disclosure can operate with fully unstructured sparsity without significant overhead, thereby increasing the potential gain of densification (e.g., allowing the process to be guided by the sparsity rather than forcing the sparsity to fit a defined structure) while maintaining high inference efficiency.

[0028] In some aspects, convolution is discussed as an example process that can be improved using the densification and sparse accumulation processes described herein. However, aspects of the present disclosure can be readily applied to a wide variety of operations, including transformer operations, sparse transformer and transformer-like operations (e.g., in lambda networks), graph transformers, graph neural networks, etc.

[0029] In some aspects of the present disclosure, a squeeze operation is used to densify or compress a tensor by removing zero-value elements and squeezing the remaining elements in the tensor vertically or horizontally to produce a densified or compressed tensor, as discussed in more detail below. As used herein, a data element may be referred to as "sparse" or "zero value" to indicate that its value is equal to zero or within a defined distance from zero. That is, some small values ​​that are technically non-zero may still be referred to as "zero value" or "sparse" elements. Additionally, in some aspects, a mapping (in some aspects referred to as a sparse map, an accumulator map, or a densification map) may be generated based on this squeeze operation. A sparse map generally indicates an association between an element (e.g., a non-zero weight) of a densified parameter tensor and an output element (e.g., an accumulator in the case of a convolution). In at least one aspect, a sparse map has the same size as a densified tensor, so that the convolution product can be quickly mapped to a corresponding accumulator during a convolution operation. This enables efficient, rapid, and accurate densification and use of tensors with unstructured sparsity. In turn, the techniques disclosed herein may reduce computation time, reduce power usage, reduce memory requirements, enable machine learning to be performed on a wider range of (potentially low-power) devices, and the like.

[0030] Example workflow for densifying a weight tensor

[0031] Figure 1 An example workflow 100 for densifying weight tensors to improve convolutions is depicted.

[0032] In the illustrated example, a weight tensor 105 (e.g., a weight tensor for a convolution operation in a neural network) is processed by a squeeze component 110 to produce a densified tensor 115 and a sparse map 120. Although depicted as discrete components for conceptual clarity, in some aspects, the operations of the squeeze component 110 may be implemented as part of other components or processing (e.g., by a compiler at compile time after the network is trained). In general, the operations of the squeeze component 110 may be implemented using hardware, software, or a combination of hardware and software.

[0033] In workflow 100, as indicated by legend 102, sparsity in weight tensor 105 is indicated using stippling. Specifically, stippled elements are zero-valued elements (e.g., element 106), indicating that the value of the element is zero (or in some aspects within a defined distance from zero), while non-stippled element regions are non-zero (e.g., element 108), indicating that the value of the element is not zero (or in some aspects greater than a defined threshold distance from zero). In the illustrated example, for ease of reference, each non-zero element includes a letter. For example, the upper left element of weight tensor 105 is non-zero and has a value represented by "A", while the upper right element is represented by a value of "m".

[0034] As discussed above and used herein, an element may be referred to as "zero-valued" or "sparse" if it satisfies one or more defined sparsity criteria. In one aspect, the sparsity criteria indicate that an element is sparse or zero-valued (if the element has an actual value of zero). In some aspects, an element may be zero-valued or sparse if the corresponding value is within a threshold distance from zero (e.g., plus or minus 0.001). In addition, the thresholds may be defined asymmetrically about a target value such as zero.

[0035] As discussed above, conventional systems typically perform convolutions directly using weight tensor 105, which inherently involves a large number of multiplications using "zero-valued" elements. These multiplications require significant computational overhead, but have no effect (or little effect) on the output of the operation. However, by creating a densified tensor 115, aspects of the present disclosure enable convolutions to be performed efficiently and with reduced computational resources (including memory, storage, processor time, etc.).

[0036] In the illustrated example, the extrusion component 110 generates the densified tensor 115 by vertically extruding the weight tensor 105 from the bottom to the top. That is, the extrusion component 110 can vertically extrude the tensor, removing zero-valued elements, to produce a densified tensor of non-zero elements. Although the illustrated example does not include zero-valued elements in the densified tensor 115, in some aspects, one or more zero-valued elements can be retained (e.g., to allow for more efficient use of relative spacing indications), as discussed in more detail below.

[0037] Furthermore, while the illustrated example depicts squeezing from bottom to top, the system may similarly squeeze from top to bottom. Additionally, while the illustrated workflow 100 includes vertical squeezing, in some aspects, the squeezing component 110 may perform horizontal squeezing (e.g., from left to right, or from right to left). In some aspects, the direction for directional squeezing may differ depending on the specific implementation. In at least one aspect, the squeezing direction may be selected (e.g., selected by the squeezing component 110, or predefined by the user) based on a variety of criteria (e.g., the dimensions of the weight tensor 105, the capabilities of the inference device (e.g., the device that will use the densified weight tensor during operation), etc.). That is, the hardware design that will be used to perform multiplication and accumulation may be better suited to first processing row by row before iterating column by column (or vice versa), such that horizontal squeezing or vertical squeezing may be preferred, such as due to read / write access practices or benefits.

[0038] In general, when using a densified tensor, the directionality of the squeeze can affect the convolution and / or accumulation process. For example, vertical squeezing (as in the illustrated example) can generally enable operand association to be maintained, which can simplify or facilitate efficient multiplication. This is because, in some aspects, convolving the weight tensor 105 (or the densified tensor 115) with the activation tensor involves multiplying each element in the activation tensor with an element in a given column of the weight tensor 105. Therefore, by squeezing the weight tensor 105 vertically, each element in the densified tensor 115 is kept in the same column as its column in the weight tensor 105, thereby facilitating multiplication (e.g., by enabling the use of conventional multiplication logic or hardware).

[0039] However, as discussed in more detail below, this vertical squeezing may cause the elements in the densified tensor 115 to lose their previous association with the appropriate output elements for the convolution (e.g., for the appropriate accumulator). This is because the elements in a given row of the weight tensor 105 are typically used to determine the output elements of that row, and the vertical squeezing may shift the elements to different rows in the densified tensor 115, causing the association to be lost. In some aspects, the horizontal squeezing may maintain this output element association, so that the multiplication process may involve some additional overhead, but the accumulation process is easy and efficient (e.g., using conventional logic or hardware).

[0040] In various aspects, the sparsity of the weight tensor 105 can be relatively random and uniformly distributed. However, as discussed above, aspects of the present invention do not require this random or uniform distribution, and do not impose any structure on sparsity. In order to allow this unstructured sparsity and achieve accurate mapping of products to accumulators (or in the case of horizontal squeezing, operands), in one aspect, the squeezing component 110 can generate a sparse map 120 of the densified tensor 115. As discussed in more detail below, in some aspects, the squeezing map 120 can indicate the association between the elements in the densified tensor 115 and the corresponding output elements of the convolution process. That is, for each element in the densified tensor 115 (and therefore for each product generated by multiplying a value in the activation tensor with an element in the densified tensor 115), the sparse map 120 can indicate (or be used to derive) the corresponding element in the output tensor. In the case of horizontal squeezing, the sparse map 120 can indicate the appropriate mapping of operands (e.g., to the weights of the input activations) to facilitate multiplication.

[0041] In at least one aspect, sparse map 120 can be used to indicate associations in multiple dimensions. For example, if a vertical squeeze operation results in unevenly sized columns (e.g., with more non-zero elements in some columns), the system can shift some sets of elements from larger columns to smaller columns and update sparse map 120 to indicate not only the vertical shift of these elements (caused by the vertical squeeze) but also the horizontal shift (to balance the columns).

[0042] In many systems, the weight tensor 105 has more elements than the convolution hardware (e.g., a multiply and accumulate (MAC) array) can process in a single cycle. Therefore, the weight tensor 105 can be divided into multiple parts for separate processing (e.g., the upper half of the tensor is processed first, and the lower half is processed later). Since the densified tensor 115 is typically smaller in size than the initial weight tensor 105, fewer computation cycles may be required to perform the convolution. This reduces computational latency and improves the efficiency of processing.

[0043] Additionally, while the illustrated example depicts a uniform densified tensor 115 (because each column of the weight tensor 105 has exactly four non-zero values), in various aspects, the unstructured nature of the densification process may allow for more or fewer elements in a given column (or row in the case of horizontal squeezing). In some aspects, these additional elements may require one or more additional cycles to perform the convolution. For example, assume that the system hardware is capable of processing 32 weights per cycle (e.g., a weight tensor or portion thereof having a width of four and a height of eight). If the densified tensor 115 has five weights in one or more columns, these additional weights may be processed in subsequent cycles (after the body of the densified weight tensor 115 is processed).

[0044] However, in some aspects, this overflow is typically limited to the end of the convolution process. That is, in many cases, the weight tensor 105 is larger than the size of the hardware, and is therefore subdivided into a collection of sub-tensors for processing (sequentially or in parallel using separate hardware). If the sparsity is unevenly distributed, so that some parts of the weight tensor 105 are denser than other parts, the densified tensor 115 may similarly have some unevenness (for example, some columns have more elements than other columns). However, since the densified tensor 115 can be depicted as a portion for the actual convolution, the overflow from one portion can typically be added to the subsequent portion, and therefore no additional computational cycles are required for most tensors. In some cases, the last portion of the tensor being processed may result in an overflow that requires an additional computational cycle. However, this extra cycle at the end still results in significantly fewer cycles overall to complete the convolution.

[0045] In some aspects, since the weights are static after training, the densified weight tensor 115 can be generated offline (e.g., before runtime inference). For example, the squeeze component 110 can generate the densified tensor 115 and the sparse map 120 at compile time (e.g., when the entire neural network is being compiled, after training).

[0046] Example workflow for generating a sparse map

[0047] Figure 2A Describes a method for generating a sparse map using absolute indications such as Figure 1 An example workflow 200A of the sparse map 120 is described.

[0048] In the illustrated example, as discussed above, the weight tensor 105 is densified using directional squeezing to produce the densified tensor 115. Additionally, as discussed above, a sparse map 120A is generated to indicate the association between the non-zero elements of the input tensor 105 (e.g., elements in the densified weight tensor 115) and the elements of the output tensor. As shown, the sparse map 120A generally has the same number of elements as the densified tensor 115. In this manner, each element in the densified tensor 115 may have a corresponding element in the sparse map 120A, wherein the value of the element in the sparse map 120A indicates the index of the corresponding output element for the element in the densified tensor 115. In some aspects, the sparse map 120A has the same dimensions as the densified tensor 115 (e.g., the same height and width).

[0049] As discussed above, each row of the weight tensor 105 generally corresponds to a given output element in the output tensor. That is, for a given element in the output tensor, the weight from the corresponding row in the weight tensor 105 is multiplied by one or more elements in the input tensor (e.g., the activation tensor) and then accumulated (e.g., summed) to form the output element.

[0050] For example, in the illustrated example, weights "A", "E", "a", and "m", all of which are in the first row of the input tensor 105, should be used to compute the first output element (e.g., the element in the first row of the output tensor at index 0). If the weight tensor 105 is processed in its original state (e.g., not densified), the products generated using these weights can be easily accumulated using a static mapping. However, as shown, the first row of the densified tensor 115 also includes elements ("I", "M", "e", and "i") that should be used to generate the second row (at index 1) of the output tensor.

[0051] As shown, the sparse map 120A indicates the index of the corresponding output element for each weight in the densified tensor 115, and can therefore be used to determine the corresponding output element for each weight in the densified tensor 115. In the illustrated example, the sparse map 120A uses absolute indicators, where the value of the sparse map 120A directly indicates the corresponding row or element of the output tensor for each element in the densified tensor 115. In the depicted workflow 200A, the sparse map 120A has the same dimensions as the densified tensor 115, so that each value in the sparse map 120A has the same index as its corresponding weight in the densified tensor 115.

[0052] As shown, the absolute indications depicted in the sparse map 120A explicitly indicate the corresponding element of the output tensor (based on their initial row in the weight tensor 105) for each weight in the densified tensor 115. Specifically, the first two elements ("A" and "E") correspond to row 0 of the output (for a zero-indexed map), the next two elements ("I" and "M") correspond to row 1, and so on. During runtime, when the densified weight tensor 115 is convolved with an input tensor (e.g., a set of activations), each product (e.g., the result of multiplying a given weight by a given activation) can be provided to the appropriate accumulator (and therefore to the appropriate output element) based on these absolute mappings.

[0053] In some aspects, if the weight tensor 105 is large, the absolute indication used in the sparse map 120A may be similarly quite large. For example, if the height of the weight tensor 105 is eight, three bits may be used to indicate which row a given weight belongs to. Therefore, for a weight tensor with a height of eight, three bits are required for each element in the sparse map 120A. However, as discussed above, the weight tensor is typically significantly larger. If the height of the weight tensor 105 is 256, 8 bits may be required for each element to represent the absolute indication of the sparse map 120A. Generally speaking, for a height of A (which therefore may require A accumulators), each element in the sparse map 120A requires log2||A|| bits.

[0054] Thus, in some aspects, the sparse map 120A itself can become significantly larger, offsetting some of the efficiencies achieved using the directional extrusion discussed above. Thus, in at least one aspect, the extrusion component 110 can use relative indications in the sparse map, as described below with reference to Figure 2B discussed.

[0055] Figure 2B Described is a method for generating a sparse map using relative indications such as about Figure 1 An example workflow 200B of the sparse map 120 is described.

[0056] In the illustrated example, as discussed above, the weight tensor 105 is densified using directed squeezing to produce the densified tensor 115. Additionally, as discussed above, a sparse map 120B is generated to indicate the association between the non-zero elements of the input tensor 105 (e.g., the elements in the densified weight tensor 115) and the elements of the output tensor. However, unlike the method using absolute mapping, Figure 2A In contrast to sparse map 120A, sparse map 120B uses relative indication.

[0057] As shown in the figure, similar to Figure 2A 105. The sparse map 120A of FIG. 106, sparse map 120B generally has the same number of elements as the densified tensor 115. In some aspects, the sparse map 120B has the same dimensions (e.g., the same height and width) as the densified tensor 115. The sparse map 120B and the densified weight tensor 115 can be used to theoretically reconstruct the initial input weight tensor 105.

[0058] As shown, the sparse map 120B includes values ​​indicating the relative spacing of the weight elements, allowing the association between the weights / products and the corresponding output elements / accumulators to be implicitly derived. In the depicted example, these values ​​correspond to the horizontal gaps between the elements in the weight tensor 105, and the association between the weights (or products) and the accumulators can therefore be sequentially derived by specifying or identifying the gap to the next product.

[0059] That is, in some aspects, the values ​​in the sparse map 120B may indicate, for each element of the densified tensor 115, how many (zero-valued) elements to skip or remove (in the illustrated example, moving from right to left and top to bottom) when squeezing the weight tensor 105 (or how many elements need to be skipped in the densified tensor 115 to identify the correct next weight). For example, the first weight "A" has a gap of 0, the second weight "E" has a gap of 0, and the third weight (the third non-zero entry in the weight tensor 105) "a" has a gap of 2, and so on.

[0060] In some aspects, as discussed above, each non-zero weight in a given row (e.g., the i-th) of the initial tensor 105 is used to generate a product associated with the corresponding accumulator (the i-th accumulator). Thus, by referring to the relative gaps indicated in the sparse map 120B, the system can correctly map the product to the corresponding accumulator.

[0061] It should be understood that although reconstructed tensor 105 is presented as an illustrative example for conceptualizing the information contained in sparse map 120B, in various aspects, the system may simply use the values ​​in sparse map 120B to drive the exact accumulation without explicitly reconstructing initial tensor 105.

[0062] In one aspect, sparse map 120B may be used to implement offline derivation of accumulator mappings. That is, an appropriate mapping for accumulating products of multiplications may be correctly and efficiently derived offline prior to runtime inference. In other words, an accumulator index may be derived for each of the non-zero entries of weight tensor 105 (e.g., each element in densified tensor 115) by appropriately accumulating runtime gap values ​​from sparse map 120B.

[0063] In one aspect, to construct the sparse map 120B using a relative indication of the interval, the i-th Acc requires A i entries, so that ∑ j g j +r=C in , where g j is the continuous gap in the weight tensor 105, r is AccA i Arrival C in The remaining gap before, and C in is the input channel size.

[0064] and Figure 2ABy using relative indications, sparse map 120B is able to use smaller values ​​to indicate weight associations than the absolute indications used by sparse map 120A. In this way, sparse map 120B can generally use fewer bits to represent the mapping. In some aspects, the number of bits used for each such element can be defined offline (e.g., by a user offline, or automatically determined based on the maximum or average gap size in weight tensor 105).

[0065] In some aspects, when the gap value for a given entry in sparse map 120B is larger than the allocated set of bits, squeeze map 110 may dynamically insert a zero-valued element (e.g., a weight of zero) at the corresponding location (e.g., in the middle of the large gap) into densified tensor 115. This may allow sparse map 120B to include relative spacing with respect to the inserted zero value, thereby bridging an otherwise large gap with two smaller gaps (and thus requiring fewer bits per element).

[0066] Example workflow for performing sparse accumulation

[0067] Figure 3 Depicted is an example workflow 300 for performing sparse accumulation during a convolution operation using a densified weight tensor.

[0068] In the illustrated example, the multiplication component 315 receives the densified weight tensor 305 and the activation tensor 310 and generates a set of one or more intermediate tensors 320, also referred to as a product tensor 320. Although the activation tensor 305 is depicted as a single column (e.g., a column vector) for conceptual clarity, in various aspects, the activation tensor 305 can have any dimension.

[0069] In general, the depicted convolution involves multiplying each element in the activation tensor 310 with one or more elements in the densified weight tensor 305, and accumulating the results based on a sparse map 322 corresponding to the densified weight tensor 305. As shown, the multiplication component 320 generates an intermediate tensor 320 by multiplying each element in the activation tensor 310 with the weight in the corresponding column of the densified weight tensor 305.

[0070] Specifically, the first element ("S") in the activation tensor 310 is multiplied with the weights in the first column of the densified weight tensor 305 (weights "A", "B", "C", and "D") to produce the first column of the intermediate tensor 320 (products "AS", "BS", "CS", and "DS"). As discussed above, in conventional systems, these products may then be accumulated (e.g., summed) row by row to produce the output tensor. In some aspects, this conventional row-by-row accumulation may also be used in the case of horizontal squeezing (although the multiplication process used to generate the intermediate tensor 320 may be different to ensure proper association of activation elements with weight elements).

[0071] However, in the illustrated workflow 300, the sparse accumulation 325 is performed using the sparse map 322. Specifically, the sparse map 322 can be used to determine a corresponding row or accumulator of output for each product in the intermediate tensor 320. That is, the sparse accumulation 325 can effectively involve rearranging the products in the intermediate tensor 320 to provide each product to the appropriate accumulator, as shown in block 330.

[0072] As shown, the first output element (labeled "a0" in the depicted example) in the output tensor 335 is generated based on the product "AS", "ET", "aW", and "mZ". Figure 1 , it is noted that the weights (“A”, “E”, “a”, and “m”) used to generate these products are each in the first row of the initial weight tensor 105. In this manner, the “a0” output element is generated by accumulating (e.g., summing) the corresponding products “AS”, “ET”, “aW”, and “mZ”. Using the sparse map 322, each product is thus provided to the appropriate accumulator to be aggregated (e.g., summed) with other products belonging to the corresponding element of the output tensor 335.

[0073] As discussed above, this sparse accumulation results in an output tensor 335 that is equal to the output tensor that would have been generated using the initial weight tensor 105. This is because convolution using the initial tensor (having weights with zero values) will result in multiple products with zero values ​​that do not affect the final output. In this way, the sparse mapping 322 coupled with the directional squeezing of the weight tensor enables efficient accumulation of products (and therefore efficient use of the densified weight tensor 305) without imposing any structure or sparsity requirements on the weight tensor or the compression process. In addition, in some aspects, the techniques described herein do not negatively impact the accuracy of the convolution or model because the output is mathematically equivalent despite requiring fewer operations and computational resources.

[0074] Example method for densifying a weight tensor

[0075] Figure 4 An example flow chart illustrating a method 400 for densifying a weight tensor is depicted.

[0076] At block 405, a weight tensor (e.g., Figure 1 The weight tensor 105 in is used for (e.g., by a squeeze component such as Figure 1As used herein, "accessing" data may generally include receiving (e.g., from a transmitter or from another component or system), requesting, retrieving (e.g., from storage or memory), or otherwise obtaining access to the data. In one aspect, as discussed above, a weight tensor may generally correspond to a set of parameters that have been learned (or are being learned) during the training process of a machine learning model. For example, when training a neural network, a set of weights may be learned for each kernel and / or layer of the network. Although the illustrated examples depict receiving and processing a single weight tensor, method 400 may be used to process any number of weight tensors sequentially and / or in parallel. In general, a weight tensor may be of any size and have any amount and distribution of sparsity.

[0077] At block 410, the squeeze component densifies the weight tensor using directional squeezing. Generally, as discussed above, this directional squeezing includes compressing the weight tensor in a horizontal or vertical direction, removing any zero-valued elements in the process. In some aspects, as discussed above, the squeeze map may optionally insert one or more zero-valued elements (or allow one or more zero-valued elements to remain in the densified tensor) to enable the sparse map to use fewer bits per element (e.g., for relative gap indication).

[0078] At block 415, the extrusion component also generates a sparse map (e.g., Figure 1 Sparse mapping in 120, Figure 2A The sparse map 120A and / or Figure 2B 120B in the sparse map. The sparse map generally indicates the association between the elements in the densified weight tensor (and therefore the products generated using the weights and the input activations) and the elements of the output tensor generated using the weight tensor. As discussed above, the sparse map may include these indications explicitly (e.g., using absolute values) or implicitly (e.g., using relative values).

[0079] At block 420, the densified weight tensor (e.g., Figure 1 In some aspects, method 400 is performed offline (e.g., before runtime inference). For example, method 400 may be performed after the model is trained (e.g., by a compiler as part of model compilation). The (compiled) model including the densified weight tensor and the sparse map may then be provided for inference.

[0080] In some examples, the model may be trained, densified, compiled, and used on one or more systems. That is, a first computing system may be used to train the model, densify the weight tensors, and compile the model into an executable / instantiable model, while a second system may use the compiled data to instantiate a local copy of the model and use this model to generate inferences at runtime. In some aspects, densification and compilation may also be performed on a third system that is distinct from the training system and the inference system. In other aspects, a single system may train, densify, compile, and use the model (e.g., as described below with reference to Fig. 9 discussed).

[0081] Please note that Figure 4 This is merely one example of a method, and other methods including fewer, additional, or alternative steps may also be consistent with the present disclosure.

[0082] Example method for selectively densifying a weight tensor

[0083] Figure 5 An example flow chart illustrating a method 500 for selectively densifying a weight tensor is depicted. In some aspects, the method 500 provides the method described above with reference to Figure 4 Additional details of the densification process are discussed.

[0084] At block 505, a weight tensor (e.g., Figure 1 The weight tensor 105 in ) is used for (e.g., by an extrusion component such as Figure 1 The method 500 may be processed by a squeeze component 110 of the machine learning model. As discussed above, a weight tensor may generally correspond to a set of parameters that have been learned (or are being learned) during the training process of a machine learning model. In general, a weight tensor may be of any size and have any amount and distribution of sparsity. Although the illustrated example depicts receiving and processing a single weight tensor, the method 500 may be used to process any number of weight tensors sequentially or in parallel.

[0085] At block 510, the system (e.g., a squeeze component) evaluates the sparsity of the received weight tensor. In at least one aspect, evaluating the tensor sparsity may include determining which elements are zero-valued (e.g., based on one or more rules specifying that weights that meet predefined criteria are to be considered "sparse" or "zero-valued"). In some aspects, evaluating the sparsity includes determining a percentage of elements that are zero-valued, a distribution of sparsity, etc.

[0086] At block 515, the system determines whether the determined sparsity satisfies one or more defined criteria. Generally, this criterion can include a variety of evaluations. For example, in some aspects, the system determines whether the sparsity meets or exceeds a defined minimum (e.g., specifying that the weight tensor must be at least 50% sparse). This can allow the system to avoid introducing densification and sparse mapping overhead when the sparsity is relatively low.

[0087] If it is determined at block 515 that the sparsity does not meet the criteria (e.g., because the weight tensor is not sufficiently sparse), method 500 continues to block 540, where the machine learning model (including the received weight tensor) is compiled for future use in inference during runtime. For example, if the weight tensor is not sufficiently sparse, the system may determine that the computational overhead required to densify the weight tensor, generate a sparse map, and / or use the densified tensor at runtime is not consistent with the efficiency to be obtained through the densification process.

[0088] Returning to block 515, if the sparsity satisfies one or more criteria, method 500 continues to block 520, where the squeezing component selects a direction to squeeze the tensor. In some aspects, the direction is selected based on a predefined configuration. For example, a user can specify a direction to squeeze the weight tensor based on a variety of factors, including the shape of the weight tensor, the distribution of sparsity in the tensor, the hardware to be used during inference, etc. In other aspects, the system can use a set of rules to evaluate these and other factors in order to select a squeezing direction.

[0089] At block 525, the extrusion component then densifies the weight tensor (e.g., to create a Figure 1 As discussed above, directionally squeezing a tensor generally involves compressing the tensor in a selected direction, removing any zero-valued elements or values ​​in the process. In some aspects, as discussed above, the squeeze map may optionally insert one or more zero-valued elements (or allow one or more zero-valued elements to remain in the densified tensor) to enable the sparse map to use fewer bits per element (e.g., for relative gap indication).

[0090] At block 530, the extrusion component also generates a sparse map based on the extrusion (e.g., Figure 1 Sparse mapping in 120, Figure 2A The sparse map 120A and / or Figure 2B120B in sparse map 120B). As discussed above, this sparse map may generally indicate an association between the densified tensor and the output tensor. In some aspects (such as when horizontal squeezing is used), the sparse map may indicate an association between weight elements in the densified tensor and activation elements in the input tensor. The sparse map may indicate an association using an absolute indication, a relative indication, or any suitable technique.

[0091] At block 535, the system also sets one or more sparsity indicators to indicate the presence of a densified weight tensor. That is, the system may set one or more bits or other indicators to indicate that the machine learning model includes a densified tensor and a sparse map. In this way, at runtime, the inference system may check the sparsity indicators to determine whether to use conventional convolutions and accumulations or sparse convolutions and / or sparse accumulations when processing input data using the weight tensor.

[0092] In at least one aspect, a sparsity indicator may indicate, for each weight tensor in a model (e.g., for each kernel or for each layer of a neural network), whether the tensor or layer has been densified. This may allow a squeeze component to dynamically densify (or avoid densifying) each set of weights in a model based on the individual sparsity of each set, and further allow an inference system to dynamically determine whether a given tensor has been densified and should therefore be processed using sparse convolutions or accumulations, as discussed above.

[0093] At box 540, the system then compiles the machine learning model (including the densified tensors, sparse maps, and sparse indicators) for future use. As discussed above, the model can be trained, densified, compiled, and used on one or more systems. That is, a first computing system can be used to train the model, densify the weight tensors, and compile the model into an executable / instantiable model, while a second system can use the compiled data to instantiate a local copy of the model and use this model to generate inferences at runtime. In some aspects, densification and compilation can also be performed on a third system that is different from the training system and the inference system. In other aspects, a single system can train, densify, compile, and use the model (e.g., as described below with reference to Fig. 9 discussed).

[0094] It should be noted that Figure 5 This is merely one example of a method, and other methods including fewer, additional, or alternative steps may also be consistent with the present disclosure.

[0095] Example method for performing desparse convolution using a densified weight tensor

[0096] Figure 6An example flow chart illustrating a method 600 for performing desparse convolution using a densified weight tensor is depicted. In some aspects, the method 600 is performed by an inference system that generates inferences at runtime using a pre-trained model (a model trained and / or compiled locally, or a model trained and / or compiled on one or more remote systems).

[0097] At block 605, the inference system accesses an activation tensor. For example, the activation tensor may be output from a previous layer in a neural network for processing at the current layer. In some aspects, the received tensor may be an initial input tensor to the model, rather than an activation tensor.

[0098] At block 610, the inference system receives or retrieves a weight tensor to be used to process the received activation. For example, as discussed above, if an activation is received at a given layer of a neural network, the inference system can determine which weight tensor(s) correspond to the given layer and retrieve them (e.g., from memory) for use in convolving the activation. In some examples, the weight tensor is convolved with the received activation, although the techniques described herein can also be readily applied to other model architectures.

[0099] At block 615, the inference system generates one or more intermediate tensors by multiplying the activation tensor with the retrieved weight tensor. For example, as described above with reference to Figure 3 As discussed, each element in the activation tensor can be multiplied by the corresponding set of elements in the weight tensor.

[0100] At block 620, the inference system determines whether one or more sparse indicators associated with the weight tensor are set, indicating that the retrieved weight tensor has been densified by directional squeezing. If not, the method 600 continues to block 635, where conventional techniques are used to accumulate intermediate values. The method 600 then continues to block 640.

[0101] If at block 620, the inference system determines that the weight tensor has been densified, then method 600 proceeds to block 625, where the system retrieves the corresponding sparse map for the densified tensor. At block 630, the system then performs sparse accumulation of the intermediate tensors based on this sparse map. For example, as described above with reference to Figure 3 As discussed, the inference system can use the sparse map to determine which set of intermediate values ​​(eg, which set of weights) corresponds to each respective output element in the output tensor, and accumulate these identified elements. Method 600 then continues to block 640 .

[0102] At block 640, the system outputs the accumulated values. This may include, for example, outputting the accumulation as input to an activation function, as input to a downstream layer of a neural network, as output from a machine learning model, etc. In some aspects, method 600 is repeated for each layer (or some subset of layers) of the neural network, thereby enabling the use of either a densified or conventional weight tensor dynamically at each layer.

[0103] The illustrated example describes the use of a vertically squeezed weight tensor. As discussed above, vertical squeezing can enable multiplication to be performed using a fixed mapping (e.g., where each activation element is multiplied by a specific and fixed set of weights, such as a specific column in the weight tensor). Accumulation is then performed using a specific sparse mapping.

[0104] However, in some aspects, horizontal squeezing may cause some or all of the weights to be in columns different from the columns in which the weights were originally (in the initial weight tensor). Thus, in some aspects, the system may determine whether the tensor is densified, and the directionality of the densification (e.g., indicated using a sparsity indicator of the weight tensor) before performing the multiplication. For example, if the tensor is squeezed horizontally, the inference system may retrieve a sparse map and use the sparsity map to identify the corresponding activation for each weight. As discussed above, although the horizontal squeezing introduces a mapping required for the multiplication process, the accumulation process may be performed using a conventional (fixed) process.

[0105] Please note that Figure 6 This is merely one example of a method, and other methods including fewer, additional, or alternative steps may also be consistent with the present disclosure.

[0106] Example method for densifying a weight tensor using directed extrusion

[0107] Figure 7 An example flow chart illustrating a method 700 for densifying a tensor using directed extrusion is depicted.

[0108] At block 705 , a weight tensor with unstructured sparsity is accessed.

[0109] At block 710 , a densified weight tensor is generated based on the weight tensor, including: directionally squeezing the weight tensor to remove zero values ​​or sparse values, and generating a sparse map based on the directed squeezing.

[0110] At box 715, the densified weight tensor and sparse map are output for use in a convolutional neural network.

[0111] In some aspects, outputting the densified weight tensor and the sparse map includes compiling a convolutional neural network based at least in part on the densified weight tensor.

[0112] In some aspects, compiling the convolutional neural network includes setting a sparsity indicator to indicate that the compiled convolutional neural network includes a densified weight tensor.

[0113] In some aspects, directionally squeezing the weight tensor includes removing zero-valued or sparse elements from the weight tensor and compressing remaining elements of the weight tensor along a dimension in the weight tensor.

[0114] In some aspects, directionally squeezing the weight tensor includes selecting a vertical direction or a horizontal direction and squeezing the weight tensor in the selected direction.

[0115] In some aspects, the dimensions of the sparse map match the dimensions of the densified weight tensor, and the sparse map indicates an association between elements in the densified weight tensor and corresponding output elements for the densified weight tensor during convolution.

[0116] In some aspects, the sparse map indicates these associations using, for each element in the densified weight tensor, an absolute indication of the corresponding output element.

[0117] In some aspects, the sparse map indicates these associations using a relative spacing for each corresponding element in the densified weight tensor that indicates a number of output elements to skip between the corresponding element and a subsequent element in the densified weight tensor.

[0118] In some aspects, generating the densified weight tensor is performed in response to determining that unstructured sparsity in the weight tensor satisfies one or more defined thresholds.

[0119] It should be noted that Figure 7 This is merely one example of a method, and other methods including fewer, additional, or alternative steps may also be consistent with the present disclosure.

[0120] Example method for convolution with densified weight tensor

[0121] Figure 8 An example flow chart illustrating a method 800 for convolving with a densified weight tensor is depicted.

[0122] At block 805, an activation tensor is accessed for processing using a convolution operation.

[0123] At block 810 , a densified weight tensor is retrieved for the convolution operation.

[0124] At block 815 , a sparse map associated with the densified weight tensor is retrieved.

[0125] At block 820 , a set of intermediate tensors is generated by multiplying the densified weight tensor with the activation tensor.

[0126] At block 825, an output tensor for the convolution operation is generated by accumulating a set of intermediate tensors based on a sparse map.

[0127] In some aspects, a densified weight tensor is generated by directionally squeezing a weight tensor with unstructured sparsity to remove zero-valued elements or values, and the sparsity map indicates an association between elements in the densified weight tensor and corresponding output elements in the output tensor.

[0128] In some aspects, the sparse map indicates these associations using, for each element in the densified weight tensor, an absolute indication of the corresponding output element.

[0129] In some aspects, the sparse map indicates these associations using a relative spacing for each corresponding element in the densified weight tensor that indicates a number of output elements to skip between the corresponding element and a subsequent element in the output tensor.

[0130] In some aspects, generating the output tensor includes: for each corresponding output element in the output tensor, identifying a corresponding set of elements corresponding to the corresponding output element from the set of intermediate tensors based on a sparse map; and accumulating the corresponding set of elements.

[0131] In some aspects, accumulating the set of intermediate tensors based on the sparse map is performed in response to determining that a sparsity indicator associated with the convolution operation indicates use of the at least one densified weight tensor in the convolution operation.

[0132] It should be noted that Figure 8 This is merely one example of a method, and other methods including fewer, additional, or alternative steps may also be consistent with the present disclosure.

[0133] Example Processing System for Desparse Convolution

[0134] In some respects, reference Figures 1 to 8 The described workflows, techniques, and methods may be implemented on one or more devices or systems. Fig. 9 An example processing system 900 is depicted that is configured to perform various aspects of the present disclosure, including, for example, Figures 1 to 9 Techniques and methods described herein. In one aspect, processing system 900 may correspond to a training and / or compiler system that uses the techniques discussed above to train a model, densify parameter tensors, generate sparse maps, compile a model, and / or perform runtime inference using a model. In at least some aspects, as discussed above, the operations described below with respect to processing system 900 may be distributed across any number of devices. For example, one system may compile a model using a densified weight tensor, while a second system uses the trained model to perform inference using sparse accumulation and / or multiplication.

[0135] The processing system 900 includes a central processing unit (CPU) 902, which may be a multi-core CPU in some examples. Instructions executed at the CPU 902 may be loaded, for example, from a program memory associated with the CPU 902 or may be loaded from a memory partition 924.

[0136] The processing system 900 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 904 , a digital signal processor (DSP) 906 , a neural processing unit (NPU) 908 , a multimedia processing unit 910 , and a wireless connectivity component 912 .

[0137] An NPU, such as 908, is typically a dedicated circuit configured to implement control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. An NPU is sometimes alternatively referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), a vision processing unit (VPU), or a graphics processing unit.

[0138] The NPU, such as 908, is configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system on a chip (SoC), while in other examples, the NPU may be part of a dedicated neural network accelerator.

[0139] The NPU can be optimized for training or inference, or in some cases configured to balance performance between the two. For NPUs that can perform both training and inference, the two tasks can generally still be performed independently.

[0140] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing data set (usually labeled or tagged), iterating on the data set, and then adjusting model parameters (such as weights and biases) to improve model performance. Generally speaking, optimization based on error predictions involves passing back through the layers of the model and determining the gradient to reduce the prediction error.

[0141] NPUs designed to accelerate inference are generally configured to operate on complete models. Thus, such NPUs can be configured to input new data slices and quickly process them through the trained model to generate model outputs (e.g., inferences).

[0142] In one implementation, the NPU 908 is part of one or more of the CPU 902 , the GPU 904 , and / or the DSP 906 .

[0143] In some examples, the wireless connection component 912 may include, for example, subcomponents for third generation (3G) connections, fourth generation (4G) connections (e.g., 4G LTE), fifth generation connections (e.g., 5G or NR), Wi-Fi connections, Bluetooth connections, and other wireless data transmission standards. The wireless connection processing component 912 is further connected to one or more antennas 914.

[0144] The processing system 900 may also include one or more sensor processing units 916 associated with any manner of sensors, one or more image signal processors (ISPs) 918 associated with any manner of image sensors, and / or a navigation processor 920, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0145] The processing system 900 may also include one or more input and / or output devices 922, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, speakers, microphones, and the like.

[0146] In some examples, one or more of the processors of processing system 900 may be based on the ARM or RISC-V instruction set.

[0147] The processing system 900 also includes a memory 924, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 924 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 900.

[0148] Specifically, in this example, the memory 924 includes a squeeze component 924A, a multiplication component 924B, a sparse component 924C, a training component 924D, and an inference component 924E. The memory 924 also includes a set of model parameters 924F, which may correspond to the initial and / or densification parameters (e.g., weights and / or biases) of the machine learning model discussed above. The depicted components, as well as other components not depicted, may be configured to perform various aspects of the techniques described herein. Although for conceptual clarity, the memory 924 includes a set of model parameters 924F, which may correspond to the initial and / or densification parameters (e.g., weights and / or biases) of the machine learning model discussed above. Fig. 9 924A, multiplication component 924B, coefficient accumulation component 924C, training component 924D, and inference component 924E may be implemented jointly or separately.

[0149] Processing system 900 also includes squeeze circuit 926, multiplication circuit 927, sparse accumulation circuit 928, training circuit 929, and inference circuit 930. The depicted circuits, as well as other circuits not depicted, may be configured to perform various aspects of the techniques described herein.

[0150] For example, the squeezing component 924A and the squeezing circuit 926 may be used to directionally squeeze the parameter tensor and generate a sparse map, as described above with reference to Figure 1 , Figure 2A , Figure 2B , Figure 4 , Figure 5 and / or Figure 6 The multiplication component 924B and the multiplication circuit 927 may be used to generate intermediate tensors based on input data (e.g., activations) and weight tensors during inference, as described above with reference to Figure 3 , Figure 6 and / or Figure 8 The sparse accumulation component 924C and the sparse accumulation circuit 928 may generally be used to perform sparse accumulation of intermediate tensors based on sparse mapping, as described above with reference to Figure 3 , Figure 6 and / or Figure 8 The training component 924D and the training circuit 929 may be used to control the training, refinement, and / or fine-tuning of various machine learning models, which may then be densified using the techniques discussed above, as described above with reference to Figure 1 , Figure 2A , Figure 2B , Figure 4 , Figure 5 and / or Figure 6 Inference component 924E and inference circuit 930 may generally use a trained machine learning model with a densified tensor to generate inferences or predictions, as discussed above with respect to Figure 3 , Figure 6 and / or Figure 8 discussed.

[0151] Although for the sake of clarity Fig. 9 Although depicted as separate components and circuits, the squeeze circuit 926, the multiplication circuit 927, the sparse accumulation circuit 928, the training circuit 929, and the inference circuit 930 may be implemented collectively or individually in other processing devices of the processing system 900, such as in the CPU 902, the GPU 904, the DSP 906, the NPU 908, etc.

[0152] Generally speaking, the processing system 900 and / or its components may be configured to perform the methods described herein.

[0153] It is worth noting that in other aspects, aspects of the processing system 900 may be omitted, such as where the processing system 900 is a server computer, etc. For example, in other aspects, the multimedia component 910, the wireless connection 912, the sensor 916, the ISP 918, and / or the navigation component 920 may be omitted. Furthermore, aspects of the processing system 900 may be distributed among multiple devices.

[0154] Sample Clauses

[0155] Item 1: A method comprising: accessing a weight tensor having unstructured sparsity; generating a densified weight tensor based on the weight tensor, comprising: directionally squeezing the weight tensor to remove zero-valued elements or sparse values; and generating a sparse map based on the directed squeezing; and outputting the densified weight tensor and the sparse map for use in a convolutional neural network.

[0156] Clause 2: The method of clause 1, wherein outputting the densified weight tensor and the sparse map comprises compiling the convolutional neural network based at least in part on the densified weight tensor.

[0157] Clause 3: A method according to any one of clauses 1 to 2, wherein compiling the convolutional neural network comprises: setting a sparsity indicator to indicate that the compiled convolutional neural network includes the densified weight tensor.

[0158] Clause 4: A method according to any one of clauses 1 to 3, wherein directionally squeezing the weight tensor comprises: removing zero-valued elements or sparse elements / values ​​from the weight tensor; and compressing the remaining elements of the weight tensor along a dimension in the weight tensor.

[0159] Clause 5: The method of any one of clauses 1 to 4, wherein directionally squeezing the weight tensor comprises: selecting a vertical direction or a horizontal direction; and squeezing the weight tensor along the selected direction.

[0160] Clause 6: A method according to any one of clauses 1 to 5, wherein: the dimension of the sparse map matches the dimension of the densified weight tensor; and the sparse map indicates the association between elements in the densified weight tensor and corresponding output elements for the densified weight tensor during convolution.

[0161] Clause 7: A method according to any of clauses 1 to 6, wherein the sparse map indicates the association using, for each element in the densified weight tensor, an absolute indication of the corresponding output element.

[0162] Clause 8: A method according to any one of clauses 1 to 7, wherein the sparse mapping indicates the association using a relative spacing for each corresponding element in the densified weight tensor, the relative spacing indicating the number of output elements to be skipped between the corresponding element and a subsequent element in the densified weight tensor.

[0163] Clause 9: A method according to any one of clauses 1 to 8, wherein generating the densified weight tensor is performed in response to determining that the unstructured sparsity in the weight tensor satisfies one or more defined thresholds.

[0164] Clause 10: A method comprising: accessing an activation tensor for processing using a convolution operation; and performing the convolution operation, comprising: retrieving a densified weight tensor for the convolution operation; retrieving a sparse map associated with the densified weight tensor; generating a set of intermediate tensors by multiplying the densified weight tensor with the activation tensor; and generating an output tensor for the convolution operation by accumulating the set of intermediate tensors based on the sparse map.

[0165] Clause 11: A method according to clause 10, wherein: the densified weight tensor is generated by directionally squeezing a weight tensor with unstructured sparsity to remove zero-value elements or sparse values, and the sparse map indicates the association between elements in the densified weight tensor and corresponding output elements in the output tensor.

[0166] Clause 12: A method according to any of clauses 10 to 11, wherein the sparse map indicates the association using, for each element in the densified weight tensor, an absolute indication of the corresponding output element.

[0167] Clause 13: A method according to any one of clauses 10 to 12, wherein the sparse map indicates the association using a relative spacing for each corresponding element in the densified weight tensor, the relative spacing indicating the number of output elements to be skipped between the corresponding element and a subsequent element in the output tensor.

[0168] Clause 14: A method according to any one of clauses 10 to 13, wherein generating the output tensor comprises: for each corresponding output element in the output tensor: identifying a corresponding set of elements corresponding to the corresponding output element from the set of intermediate tensors based on the sparse mapping; and accumulating the corresponding set of elements.

[0169] Clause 15: A method according to any one of clauses 10 to 14, wherein accumulating the set of intermediate tensors based on the sparse map is performed in response to determining that a sparsity indicator associated with the convolution operation indicates that at least one densified weight tensor is used in the convolution operation.

[0170] Clause 16: A system comprising: a memory including computer executable instructions; and one or more processors configured to execute the computer executable instructions and cause the processing system to perform the method according to any one of clauses 1 to 15.

[0171] Clause 17: A system comprising: means for performing the method according to any one of clauses 1 to 15.

[0172] Clause 18: A non-transitory computer readable medium comprising: computer executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method of any one of clauses 1 to 15.

[0173] Clause 19: A computer program product embodied on a computer readable storage medium, the computer program product comprising: code for performing the method according to any one of clauses 1 to 15.

[0174] Additional considerations

[0175] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein are not limited to the scope, applicability or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, without departing from the scope of the present disclosure, the functions and arrangements of the elements discussed may be changed. Various examples may omit, replace or add various procedures or components as appropriate. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted or combined. In addition, the features described for some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover such devices or methods practiced using other structures, functionality, or structures and functionality other than the various aspects of the present disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the claims.

[0176] As used herein, the word “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0177] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items (including single members). As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0178] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, database, or other data structure), ascertaining, and the like. Furthermore, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Furthermore, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0179] As used herein, in the context of sharing electronic signals and data between the various elements described herein, the term "connected to" generally refers to data communication between the corresponding elements connected to each other. In some cases, the elements can be directly connected to each other, such as via one or more conductive traces, lines or other conductive carriers that can carry signals and / or data between the corresponding elements directly connected to each other. In other cases, the elements can be indirectly connected to each other, such as via one or more data buses or similar shared circuits and / or integrated circuit elements for conveying signals and data between the corresponding elements indirectly connected to each other.

[0180] The method disclosed herein includes one or more steps or actions for implementing the method. The steps and / or actions of the method can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions can be modified without departing from the scope of the claims. In addition, the various operations of the method described above can be performed by any suitable component capable of performing the corresponding function. The component may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors. Generally speaking, in the case of operations illustrated in the accompanying drawings, those operations may have corresponding corresponding components with similar numbers plus functional components.

[0181] The following claims are not intended to be limited to the various aspects shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless specifically stated, reference to an element in the singular form is not intended to mean "one and only one", but "one or more". Unless otherwise specifically stated, the term "some" refers to one or more. Any claim element should not be interpreted according to the provisions of 35 U.S.C. § 112 (f), unless the phrase "parts for..." is used to expressly record the element, or in the case of a method claim, the phrase "step for..." is used to record the element. All structural and functional equivalents of the elements of the various aspects described throughout the present disclosure that are known or will be known later to a person of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, regardless of whether such disclosure is explicitly stated in the claims, anything disclosed herein is not intended to be disclosed to the public.

Claims

1. A processor-implemented method, comprising: Access weight tensors with unstructured sparsity; Generating a densified weight tensor based on the weight tensor, including: directionally squeezing the weight tensor to remove zero-valued elements; and generating a sparse map based on the directional extrusion; and The densified weight tensor and the sparse map are output for use in a convolutional neural network.

2. The processor-implemented method of claim 1 , wherein outputting the densified weight tensor and the sparse map comprises: The convolutional neural network is compiled based at least in part on the densified weight tensor.

3. The processor-implemented method of claim 2, wherein compiling the convolutional neural network comprises: A sparsity indicator is set to indicate that the compiled convolutional neural network includes the densified weight tensor.

4. The processor-implemented method of claim 1 , wherein directionally squeezing the weight tensor comprises: removing zero-valued elements from the weight tensor; as well as Compress the remaining elements of the weight tensor along a dimension in the weight tensor.

5. The processor-implemented method of claim 1 , wherein directionally squeezing the weight tensor comprises: Select vertical or horizontal orientation; as well as Squash the weight tensor along the chosen direction.

6. The processor-implemented method of claim 1 , wherein: The dimensionality of the sparse map matches the dimensionality of the densified weight tensor; and The sparse map indicates an association between elements in the densified weight tensor and corresponding output elements for the densified weight tensor during convolution.

7. The processor-implemented method of claim 6, wherein the sparse map indicates the association using, for each element in the densified weight tensor, an absolute indication of the corresponding output element.

8. The processor-implemented method of claim 6, wherein the sparse map indicates the association using a relative spacing for each corresponding element in the densified weight tensor, the relative spacing indicating a number of output elements to be skipped between the corresponding element and a subsequent element in the densified weight tensor.

9. The processor-implemented method of claim 1 , wherein generating the densified weight tensor is performed in response to determining that the unstructured sparsity in the weight tensor satisfies one or more defined thresholds.

10. A processor-implemented method comprising: Access the activation tensor for processing using convolution operations; as well as Performing the convolution operation includes: Retrieving a densified weight tensor for the convolution operation; retrieving a sparse map associated with the densified weight tensor; generating a set of intermediate tensors by multiplying the densified weight tensor and the activation tensor; and An output tensor for the convolution operation is generated by accumulating the set of intermediate tensors based on the sparse map.

11. The processor-implemented method of claim 10, wherein: The densified weight tensor is generated by directionally squeezing the weight tensor with unstructured sparsity to remove zero-valued elements, and The sparse map indicates an association between elements in the densified weight tensor and corresponding output elements in the output tensor.

12. The processor-implemented method of claim 11, wherein the sparse map indicates the association using, for each element in the densified weight tensor, an absolute indication of the corresponding output element.

13. The processor-implemented method of claim 11, wherein the sparse map indicates the association using a relative spacing for each corresponding element in the densified weight tensor, the relative spacing indicating a number of output elements to be skipped between the corresponding element and a subsequent element in the output tensor.

14. The processor-implemented method of claim 10 , wherein generating the output tensor comprises, for each corresponding output element in the output tensor: identifying, from the set of intermediate tensors, a respective set of elements corresponding to the respective output element based on the sparse map; and The corresponding set of elements is accumulated.

15. The processor-implemented method of claim 10, wherein accumulating the set of intermediate tensors based on the sparse map is performed in response to determining that a sparsity indicator associated with the convolution operation indicates use of at least one densified weight tensor in the convolution operation.

16. A system comprising: a memory including computer executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the system to perform operations comprising: Access weight tensors with unstructured sparsity; Generating a densified weight tensor based on the weight tensor, including: directionally squeezing the weight tensor to remove zero-valued elements; and generating a sparse map based on the directional extrusion; and The densified weight tensor and the sparse map are output for use in a convolutional neural network.

17. The system of claim 16, wherein outputting the densified weight tensor and the sparse map comprises: The convolutional neural network is compiled based at least in part on the densified weight tensor.

18. The system of claim 17, wherein compiling the convolutional neural network comprises: A sparsity indicator is set to indicate that the compiled convolutional neural network includes the densified weight tensor.

19. The system of claim 16, wherein directionally squeezing the weight tensor comprises: removing zero-valued elements from the weight tensor; as well as Compress the remaining elements of the weight tensor along a dimension in the weight tensor.

20. The system of claim 16, wherein directionally squeezing the weight tensor comprises: Select vertical or horizontal orientation; as well as Squash the weight tensor along the chosen direction.

21. The system of claim 16, wherein: The dimensionality of the sparse map matches the dimensionality of the densified weight tensor; and The sparse map indicates an association between elements in the densified weight tensor and corresponding output elements for the densified weight tensor during convolution.

22. The system of claim 21, wherein the sparse map indicates the association using, for each element in the densified weight tensor, an absolute indication of the corresponding output element.

23. The system of claim 21, wherein the sparse map indicates the association using a relative spacing for each corresponding element in the densified weight tensor, the relative spacing indicating a number of output elements to be skipped between the corresponding element and a subsequent element in the densified weight tensor.

24. The system of claim 16, wherein generating the densified weight tensor is performed in response to determining that the unstructured sparsity in the weight tensor satisfies one or more defined thresholds.

25. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors of a processing system, cause the processing system to perform operations comprising: Access the activation tensor for processing using convolution operations; as well as Performing the convolution operation includes: Retrieving a densified weight tensor for the convolution operation; retrieving a sparse map associated with the densified weight tensor; generating a set of intermediate tensors by multiplying the densified weight tensor and the activation tensor; and An output tensor for the convolution operation is generated by accumulating the set of intermediate tensors based on the sparse map.

26. The one or more non-transitory computer readable media of claim 25, wherein: The densified weight tensor is generated by directionally squeezing the weight tensor with unstructured sparsity to remove zero-valued elements, and The sparse map indicates an association between elements in the densified weight tensor and corresponding output elements in the output tensor.

27. The one or more non-transitory computer-readable media of claim 26, wherein the sparse map indicates the association using, for each element in the densified weight tensor, an absolute indication of the corresponding output element.

28. One or more non-transitory computer-readable media according to claim 26, wherein the sparse map indicates the association using a relative spacing for each corresponding element in the densified weight tensor, the relative spacing indicating a number of output elements to be skipped between the corresponding element and a subsequent element in the output tensor.

29. The one or more non-transitory computer-readable media of claim 25, wherein generating the output tensor comprises, for each corresponding output element in the output tensor: identifying, from the set of intermediate tensors, a respective set of elements corresponding to the respective output element based on the sparse map; and The corresponding set of elements is accumulated.

30. One or more non-transitory computer-readable media according to claim 25, wherein accumulating the set of intermediate tensors based on the sparse map is performed in response to determining that a sparsity indicator associated with the convolution operation indicates that at least one densified weight tensor is used in the convolution operation.

Images