Heterogeneous architecture for neural network computation acceleration based on depthwise separable convolution

Abstract

The present invention relates to methods and systems for performing depthwise separable convolutions on input data in a convolutional neural network. The present invention utilizes a heterogeneous architecture with several MAC arrays, including 1D MAC arrays and 2D MAC arrays for performing depthwise separable convolutions and Winograd conversion logic. Depthwise separable convolutions use fewer weight parameters, and thereby fewer multiplications, while it achieves the same computational results as traditional convolutions.

Description

Technical Field

[0001] This invention generally relates to artificial neural networks. More specifically, this invention relates to efficient convolutional neural network architectures for reducing the computational costs and memory requirements associated with convolutional neural networks. Background Technology

[0002] Convolution is a linear operation involving the multiplication of a set of weights with the input, much like a traditional neural network. Convolutional Neural Networks (CNNs) are a type of neural network model specifically designed for processing two-dimensional image data, but they can also be used for one-dimensional and three-dimensional data.

[0003] Convolution is a simple application of applying filters to an input to generate activation. Repeatedly applying the same filter to the input produces an activation map (called a feature map) that indicates the location and intensity of detected features in the input (e.g., an image).

[0004] In convolution operations, multiplication is performed between an array of input data and an array of two-dimensional weights called filters or kernels. The filters are smaller than the input data, and the type of multiplication applied between a small patch of the input (the size of the filter) and the filter is called a dot product. The dot product is an element-wise multiplication between a small patch of the input (the size of the filter) and the filter, then summed, always resulting in a single value.

[0005] Neural network-based perception requires massive computation. Real-time perception applications, such as autonomous driving, require tens or hundreds of tera operations. To meet these computational demands, traditional methods for accelerating neural networks tend to increase computing power by adding more hardware resources to the accelerator. These can include graphics processing units (GPUs), tensor processing units (TPUs) / neural network processing units (NPUs), digital signal processing units (DSPs), and field-programmable gate arrays (FPGAs).

[0006] Another trend is to reduce computational complexity in neural network architecture by using depthwise separable convolutions instead of traditional convolutions. Unlike typical convolutional neural networks, the main computational unit of a depthwise separable neural network consists of a depthwise convolution and one or more pointwise convolutions.

[0007] Depthwise separable convolutions use fewer weight parameters and thus fewer multiplications, while achieving the same computational results as traditional convolutions. However, leveraging depthwise separable convolutions with conventional accelerator hardware presents new challenges.

[0008] A single, uniform design, such as a 2D / 3D systolic array, cannot efficiently map the heterogeneous computation of depthwise separable convolutions. Depthwise separable convolutions are mostly input / output (I / O) constrained, requiring double data rate (DDR) bandwidth, which means that expensive hardware computing resources may not even improve performance, i.e., computational throughput. By analyzing the algorithmic characteristics of depthwise separable convolutions, typical neural network architectures use them as the computational backbone.

[0009] A highly efficient and power-efficient heterogeneous architecture for accelerating computation of depthwise separable convolution-based neural networks is proposed, comprising one or more 2D MAC arrays (multiplier and accumulator arrays) and one (or more) 1D MAC arrays, tilting logic for mapping computation, shared / dedicated memory for the two arrays, and Winograd transformation logic.

[0010] It is now apparent that numerous methods and systems have been developed in the prior art sufficient for a wide range of purposes. Furthermore, even if these inventions may be suitable for the specific purpose they address, they may correspondingly be unsuitable for the purposes of the aforementioned invention. Summary of the Invention

[0011] Convolution is a crucial mathematical operation in artificial neural networks (ANNs). Convolutional neural networks (CNNs) can be used to learn features and classify data using image frames. There are many types of CNNs. One type is the deep separable convolutional neural network. These types of CNNs are widely used for two reasons: a) they have fewer parameters to tune compared to standard CNNs, reducing overfitting; and b) they are computationally cheaper, making them suitable for mobile vision applications.

[0012] Depthwise separable convolution is named as such because it involves not only spatial dimensions but also depth dimensions and the number of channels. Suppose we have input data of size Df×Df×M, where Df×Df can be the image size and M is the number of channels (3 for RGB images). Suppose we have N filters / kernels of size Dk×Dk×M. If we perform a regular convolution operation, the output size will be Dp×Dp×N.

[0013] In depthwise computation, convolution is applied to a single channel at a time, unlike standard CNNs where convolution is performed on all M channels. Therefore, the filter / kernel size here will be Dk×Dk×1. Given that the input data has M channels, M such filters are needed. The output size will be Dp×Dp×M. In pointwise computation, a 1×1 convolution operation is applied to the M channels. Therefore, the filter size for this operation will be 1×1×M. If N such filters are used, the output size becomes Dp×Dp×N.

[0014] By analyzing the algorithmic characteristics of depthwise separable convolution and a typical neural network architecture that uses depthwise separable convolution as the computational backbone, this invention discloses a highly utilized and power-efficient heterogeneous architecture for accelerating neural network computation based on depthwise separable convolution.

[0015] The architecture includes one or more 2D MAC arrays (multiplier and accumulator arrays) and one (or more) 1D MAC arrays, skew logic for mapping computation, shared / dedicated memory for the two arrays, and Winograd transformation logic.

[0016] Pointwise convolution is a special type of convolution where the kernel of one output channel is a scale value. Therefore, convolution can be directly mapped to matrix-to-matrix multiplication without the overhead of repeating weights or activations, as can be seen by transforming ordinary convolution using the im2col algorithm. Any typical 2D MAC array architecture, such as a 2D systolic array for matrix-to-matrix multiplication or, in more general cases, a General Matrix Multiplication (GEMM) module, can be computed with near 100% hardware utilization.

[0017] Winograd convolutions are computed using a 1D MAC array, providing computational equivalence to depthwise convolutions. Depthwise convolutions differ from typical convolutions in that each input channel uses a separate kernel to compute the corresponding output channel. Depthwise convolutions do not change the number of channels. Furthermore, there is no cross-channel accumulation. In other words, kernel reuse in depthwise convolutions is very low.

[0018] Therefore, traditional hardware acceleration using systolic arrays with high dimensionality (>2) cannot be used to improve convolution performance. Using two-dimensional systolic arrays to compute depthwise convolutions is either unscalable or has low hardware utilization. The Winograd algorithm transforms convolutions in the spatial domain into dot products in the Winograd domain, making computation feasible using a 1-D MAC array.

[0019] The most efficient computation order for pointwise convolution is to process data along the channel direction, while the most efficient computation order for depth calculation is to process one channel at a time.

[0020] Other objects and aspects of the invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings, which illustrate by way of example features according to embodiments of the invention.

[0021] To achieve the above and related objectives, the present invention may be implemented in the form shown in the accompanying drawings. However, it should be noted that the drawings are merely illustrative and that changes may be made to the specific construction shown and described within the scope of the appended claims.

[0022] Although the invention has been described above with reference to various exemplary embodiments and implementations, it should be understood that the various features, aspects, and functions described in one or more of the various embodiments are not limited in their applicability to the specific embodiments in which they are described, but can be applied individually or in various combinations to one or more other embodiments of the invention, whether or not such embodiments are described, and whether such features are presented as part of the described embodiments. Therefore, the breadth and scope of the invention should not be limited by any of the exemplary embodiments described above.

[0023] The presence of words and phrases with broadened meanings, such as “one or more,” “at least,” “but not limited to,” or other similar phrases, should not, in some instances, be interpreted as indicating an intention or demand for the use of a narrower meaning in the absence of such a broadened phrase. Attached Figure Description

[0024] The objects and features of the invention will become more apparent from the accompanying drawings, the following description, and the appended claims. It should be understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. The invention will be described and illustrated with additional specificity and detail using the drawings, in which:

[0025] Figure 1 A method for achieving high utilization of neural network-based computation using depthwise separable convolutions is shown;

[0026] Figure 2A heterogeneous architecture for neural network computation based on depthwise separable convolution is shown;

[0027] Figure 3 A heterogeneous architecture for neural network-based computation using depthwise separable convolutions according to the present invention is illustrated; and

[0028] Figure 4 A pseudo-pipeline process according to the present invention is shown. Detailed Implementation

[0029] Convolution is a crucial mathematical operation in Artificial Neural Networks (ANNs). Convolutional Neural Networks (CNNs) can be used to learn features and classify data using image frames. There are many types of CNNs. One type is the Deeply Separable Convolutional Neural Network. These types of CNNs are widely used for two reasons: a) they have fewer parameters to tune compared to standard CNNs, reducing overfitting; and b) they are computationally less expensive, making them suitable for mobile vision applications.

[0030] Depthwise separable convolution is named as such because it involves not only spatial dimensions but also depth dimensions and the number of channels. Suppose we have input data of size Df×Df×M, where Df×Df can be the image size and M is the number of channels (3 for RGB images). Suppose we have N filters / kernels of size Dk×Dk×M. If we perform a regular convolution operation, the output size will be Dp×Dp×N.

[0031] In depthwise computation, convolution is applied to a single channel at a time, unlike standard CNNs where convolution is performed on all M channels. Therefore, the filter / kernel size here will be Dk×Dk×1. Given that the input data has M channels, M such filters are needed. The output size will be Dp×Dp×M. In pointwise computation, a 1×1 convolution operation is applied to the M channels. Therefore, the filter size for this operation will be 1×1×M. If N such filters are used, the output size becomes Dp×Dp×N.

[0032] Figure 1 A method for achieving high utilization of neural network-based computation using depthwise separable convolutions is illustrated, as depicted in flowchart 100. The method includes step 102, which involves performing pointwise convolutions on the input data using a two-dimensional MAC array to generate a first output in the spatial domain. The next step includes performing a depthwise convolution on the first output using a one-dimensional MAC array to generate a second output in the Winograd domain, 104.

[0033] The Winograd algorithm transforms depthwise convolutions in the spatial domain into dot products in the Winograd domain, making it feasible to implement the computation using a one-dimensional MAC array. The method also includes performing depthwise convolutions on a second output using a two-dimensional MAC array to generate a final output 106 in the spatial domain from the Winograd domain. A configurable adder tree structure is introduced to support multiple kernel sizes for implementing the Winograd domain to / from the spatial domain transformation.

[0034] The next step involves the processing unit segmenting the final output into multiple units 108. This is done by a dedicated tile processing unit that can segment spatial activations into tiles, resulting in larger kernel sizes or larger convolutions. Kernel sizes with a stride > 1 can be decomposed into multiple depthwise convolutions with smaller kernel sizes (less than 3×3). The next step includes the processing unit stripping one or more units 110 from the multiple units, and then the accumulator in the processing unit accumulating one or more units to compute a depthwise separable convolution 112.

[0035] Figure 2 A heterogeneous architecture 200 for accelerating neural network computation based on depthwise separable convolution is illustrated. The heterogeneous architecture includes multiple MAC arrays performing depthwise separable convolution. The MAC arrays include one or more 1D MAC arrays 202 and 2D MAC arrays 204. Depthwise separable convolution includes depthwise convolution and pointwise convolution. The MAC arrays include one or more 2D MAC arrays 204 for performing pointwise convolution in the spatial domain. One or more 2D MAC arrays 204 perform pointwise convolution on the input data to generate a first output.

[0036] The MAC array also includes one or more 1D MAC arrays 202 for performing depthwise convolutions on the first output, wherein the one or more 1D MAC arrays perform depthwise convolutions on the obtained first output. The one or more 1D MAC arrays 202 compute the depthwise convolution of the first output in the Winograd domain by transforming the domain from the spatial domain to the Winograd domain using an adder tree structure. The one or more 1D MAC arrays 202 perform depthwise convolutions on the first output and generate a second output. The Winograd algorithm transforms convolutions in the spatial domain into dot products in the Winograd domain, making it feasible to perform the computation using 1D MAC arrays 202.

[0037] One or more 2D MAC arrays 204 further perform pointwise convolutions on the second output to generate the final output. The final output is obtained in the spatial domain, and the domain transformation is performed by the Winograd adder structure tree.

[0038] The heterogeneous architecture also includes a processing unit 206 containing multiple functional elements. The processing unit 206 includes a segmentation unit 206a as a dedicated tile processing unit, which can segment the obtained final output into tiles (with overlap) so that a large kernel size or convolution with stride > 1 can be decomposed into multiple depthwise convolutions with smaller kernel sizes (less than 3×3).

[0039] The system includes a stripping mechanism configured within the stripping unit 206b (deployed in a Black Sesame Technology (BST) chip), which processes the entire activation (stripping) in smaller units. Therefore, multiple depthwise separable convolutions can be processed with a single DDR load and a single DDR memory, further reducing DDR accesses. The BST chip can be A500, A2000, or A100.

[0040] Furthermore, an accumulator 206c is introduced in the Winograd-to-spatial-domain transformation unit to support tile-based accumulation. The 1D deep convolutional unit and the transformation adder tree can be absorbed into a more general element-wise processing unit to achieve a 1D array, allowing the same hardware resources to serve the different computational needs of the convolutional neural network.

[0041] Compared to 3D MAC array architectures that utilize data reuse across channel directions, the disclosed architecture uses less area to achieve the same computational requirements because it achieves the highest possible hardware utilization. It consumes less power to compute the same inference for a neural network based on depthwise separable convolutions. When compared to a unified MAC array architecture using only a 2D accelerator architecture, i.e., a CPU with a hardware GEMM, the disclosed architecture uses slightly more area, namely a one-dimensional MAC array 202 for depthwise convolutions. For example, a system with both N×N 2D and N×1 1D arrays can achieve more than twice the throughput compared to a system with only an N×N 2D array.

[0042] When comparing the typical architecture to a 1-D array (i.e., a DSP-like vector machine), the disclosed architecture uses more area, namely the two-dimensional MAC array 204 used for pointwise convolution. The hybrid architecture reduces DDR access because data can now be shared locally between / within the MAC arrays, reducing power computation. Furthermore, it enables the shaping of the 2D array 204 and the 1D array 202 to best suit the system's DDR bandwidth based on the distribution of different computational needs (i.e., depthwise and pointwise convolutions), further improving the system's hardware utilization.

[0043] Figure 3An architecture for neural network-based computation using depthwise separable convolutions according to the present invention is illustrated. An additive accumulator within the processing unit in the Winograd-to-spatial-domain transformation unit is introduced to support tile-based accumulation. The 1D depthwise convolutional unit and the transformation adder tree can be absorbed into a more general element-wise processing unit to achieve a 1D array, allowing the same hardware resources to serve the different computational needs of the convolutional neural network.

[0044] Example model structure 300 places almost all computations into dense 1×1 convolutions. DDR / main MEM 302 is distributed via two branches employing element-wise data processing (EDP) 306 (Electronic Data Processing Technique) and GEMM weight MEM 308 as a highly optimized general matrix multiplication. The tiling convolutional network of the present invention uses a novel weight binding scheme (“tiling”), namely activating tiling unit 304, which simultaneously enjoys the benefit of significantly reducing the number of learnable parameters. As shown, in both branches, just after activating tiling unit 304, a configurable Winograd input transform adder tree 310 is introduced to implement Winograd to or from the spatial domain transformation to support multiple kernel sizes. In both branches, after the transformation from the spatial domain to the Winograd domain, partial outputs are obtained in the form of element-wise data processing (EDP) tile buffer 312 and GEMM tile buffer 314, which are fed to element-wise data processing (EDP) unit 318 and GEMM unit 320, respectively.

[0045] The outputs of element-wise data processing (EDP) unit 318 and GEMM unit 320 are provided to Winograd input transform adder tree 322, which receives the outputs of the two units and performs a spatial-to-Winograd domain transformation on the received outputs. The final output is then taken from Winograd input transform adder tree 322 to active tiling unit 304, which divides the output into multiple tiles.

[0046] The three deep Winograd convolution data scheduling schemes proposed in this invention are "weight broadcasting and activation unicast," "weight unicast and activation unicast," and "hybrid data scheduling." The weight broadcasting and activation unicast data scheduling scheme is simple in design and requires one data cycle for weight loading. However, one of its drawbacks is that it generates output channels one by one. Therefore, the output cannot be used by hardware that computes pointwise convolutions using a 2D MAC array.

[0047] As a second scheduling scheme, "weight unicast and activation unicast" simultaneously generates output tiles from multiple channels. The output can be directly fed to pointwise convolutional units, which simplifies memory control. However, this requires D data / cycle memory bandwidth for weight loading, where D is the number of MAC units in the array.

[0048] The third data scheduling scheme, as a hybrid data scheduling approach, organizes the 1D array into groups. Within a group, it uses a weighted broadcast and active unicast scheme, generating one output channel at a time. Each group uses a different core for computation. Therefore, weights are unicast to each group. The weight loading bandwidth is D / G, where G is the number of groups.

[0049] For integer-based inference, a direct Winograd implementation requires a wide range of data precision to maintain identical results. This invention proposes three quantization mechanisms to meet different precision requirements.

[0050] 1. Direct Winograd implementation: The advantage of this implementation is that it perfectly matches the result. However, there is no energy saving in this implementation.

[0051] 2. Quantizing activations in the Winograd domain: This implementation has the advantage of requiring a small hardware area and low power consumption. However, it cannot guarantee a perfect match with the corresponding depthwise convolution, thus reducing the accuracy of the neural network.

[0052] 3. Mixed precision Winograd implementation: The advantage of this implementation is that it uses the optimal precision that balances power and area (basically supported by element-wise processing units).

[0053] Figure 4 This illustrates pseudo-pipeline processing. The most efficient computation order for pointwise convolution is to process data along the channel direction, while the most efficient computation order for depthwise computation is to process one channel at a time. The activation of pointwise convolution is partitioned in the spatial domain, allowing a 2D MAC array to produce partial outputs to feed 1-D depthwise convolution units. The activation of depthwise convolution is partitioned in the spatial domain. The output is then ready for processing via pointwise convolution.

[0054] ReLU6 can be used as a source of nonlinearity due to its robustness when used with low-precision computation. A 3×3 kernel size can be used, which is standard for modern networks. During training, as shown in boxes 402 and 404, dropout and batch normalization can be utilized. Batch normalization is a layer that allows each layer of the network to learn more independently. It is used to normalize the output of the preceding layers. In box 402, batch normalization is used on a 1×1 extended layer, while in box 404, it is used on a 3×3 depthwise convolution. Furthermore, in box 406, batch normalization is applied to a 1×1 projected layer. Adding boxes 402, 404, and 406 to the input received by box 402 gives the bottleneck residual block.

[0055] Deploying a small buffer between pointwise convolutional units and depthwise convolutional units allows for data processing in a pseudo-pipeline manner, such as... Figure 4 As shown in 408 and 410. Given that N is the number of pointwise convolutional units and depthwise convolutional units, pseudo-pipelined processing increases throughput by a factor of N.

[0056] Intermediate results are distributed and stored in the pipeline buffer, which reduces the number of DDR accesses and data loading cycles, and speeds up processing. Pseudo-pipelined processing reduces the system's DDR bandwidth requirements and lowers the power consumption of DDR accesses.

[0057] While various embodiments of the invention have been described above, it should be understood that they are presented by way of example only and not as limiting. Similarly, the figures may depict exemplary architectures or other configurations of the invention, serving to aid in understanding the features and functions that may be included in the invention. The invention is not limited to the exemplary architectures or configurations shown, but various alternative architectures and configurations can be used to implement the desired features.

[0058] Although the invention has been described above with reference to various exemplary embodiments and implementations, it should be understood that the various features, aspects, and functions described in one or more individual embodiments are not limited in their applicability to the specific embodiments in which they are described, but can be applied individually or in various combinations to one or more other embodiments of the invention, whether or not such embodiments are described, and whether or not such features are presented as part of the described embodiments. Therefore, the breadth and scope of the invention should not be limited by any of the exemplary embodiments described above.

[0059] The presence of words and phrases with broadened meanings, such as “one or more,” “at least,” “but not limited to,” or other similar phrases, should not, in some instances, be interpreted as indicating an intention or demand for the use of a narrower meaning in the absence of such a broadened phrase.

Claims

1. A method for achieving high utilization of neural network based computation using depthwise separable convolution, characterized by, The method comprises: performing a point-wise convolution on input data using a two-dimensional MAC array to generate a first output in a spatial domain; performing a depth convolution on the first output using a one-dimensional MAC array to generate a second output in a Winograd domain; performing a point-wise convolution on the second output using the two-dimensional MAC array to generate a final output from the Winograd domain in the spatial domain; segmenting, by a processing unit, the final output into a plurality of units; peeling, by the processing unit, one or more units of the plurality of units; and accumulating the one or more units of the plurality of units to compute the depth separable convolution.

2. The method of claim 1, wherein, The first output is distributed and stored in a pipeline buffer.

3. The method of claim 1, wherein, The processing from the point-wise convolution to the depth convolution of the first output to the second output is performed by using several buffers.

4. The method of claim 3, wherein, The several buffers form a pseudo-pipeline.

5. The method of claim 1, wherein, The conversion from the spatial domain to the Winograd domain is performed by an adder tree structure.

6. The method of claim 1, wherein, The conversion from the Winograd domain to the spatial domain is performed by an adder tree structure.

7. The method according to any one of claims 5 and 6, characterized in that, The adder tree structure supports different kernel sizes.

8. The method of claim 1, wherein, Peeling the one or more units allows processing of the depth separable convolution by a single DDR load and a single DDR store.

9. The method of claim 8, wherein, Processing of the depth separable convolution by the single DDR load and the single DDR store limits access to the DDR.

10. The method of claim 1, wherein, The neural network architecture is a heterogeneous architecture.

11. The method of claim 1, wherein, The depth separable convolution reduces computational complexity and power requirements.

12. A heterogeneous architecture for neural network computation acceleration based on depthwise separable convolution, characterized in that, The heterogeneous architecture comprises: a plurality of MAC arrays for performing a depth separable convolution, wherein the plurality of MAC arrays comprises: one or more two-dimensional MAC arrays for performing a point-wise convolution in a spatial domain, wherein the one or more two-dimensional MAC arrays perform the point-wise convolution on input data to generate a first output; and one or more one-dimensional MAC arrays for performing a depth convolution in a Winograd domain, wherein the one or more one-dimensional MAC arrays perform a Winograd convolution on the first output to generate a second output, and further wherein the one or more two-dimensional MAC arrays perform the point-wise convolution on the second output using an adder tree structure to generate a final output; a processing unit, wherein the processing unit comprises: a segmentation unit, wherein the segmentation unit segments the final output into a plurality of tiles; a peeling unit, wherein the peeling unit peels one or more units of the plurality of tiles; and an accumulator, wherein the accumulator accumulates the one or more units of the plurality of tiles to compute the depth separable convolution.

13. A computer readable medium characterized by The computer readable medium stores computer program logic for enabling at least one processor in a computer system to achieve high utilization of neural network based computations using depth separable convolution, the computer program logic comprising: performing a point-wise convolution on input data using a two-dimensional MAC array to generate a first output in a spatial domain; performing a depthwise convolution on the first output with a one-dimensional MAC array to generate a second output within a Winograd domain; performing a pointwise convolution on the second output with the two-dimensional MAC array to generate a final output within the spatial domain from the Winograd domain; segmenting, by a processing unit, the final output into a plurality of units; peeling, by the processing unit, one or more units of the plurality of units; and accumulating the one or more units of the plurality of units to compute the depthwise separable convolution.

Images