Efficient inference with fast point-wise convolution

Abstract

Embodiments described herein relate to a method comprising: receiving input data at a convolutional neural network (CNN) model; generating a decomposition computing network comprising multiple connections between a first layer of the CNN model and a second layer of the CNN model, wherein: the decomposition computing network comprises N inputs, the decomposition computing network comprises M outputs, and the decomposition computing network comprises at least one path from each of the N inputs to each of the M outputs; setting connection weights of multiple connections in the decomposition computing network to 1 so that a weight density of the decomposition computing network is <100%; performing fast point-by-point convolution using the decomposition computing network to generate fast point-by-point convolution output; and providing the fast point-by-point convolution output to the second layer of the CNN model.

Description

[0001] Priority claim under 35 USC § 119

[0002] This patent application claims priority to non-provisional application No. 16 / 571,760, filed on September 16, 2019, entitled “EFFICIENT INFERENCING WITH FAST POINTWISE CONVOLUTION,” which has been assigned to its assignee and is hereby expressly incorporated herein by reference. Technical Field

[0003] Various aspects of the present disclosure relate to optimizing machine learning model performance, and in particular to improving the efficiency of point-wise convolution. Background Art

[0004] Machine learning can produce a trained model, such as an artificial neural network, a classification tree, etc., which represents a generalized fit to a labeled training data set. Applying the trained model to new data produces inferences, which can be used to gain insights about the new data. In some cases, applying a trained model to new data is referred to as "running inferences" on the new data.

[0005] Creating inferences is computationally intensive, especially as model complexity increases. Accordingly, methods are needed to improve the performance of machine learning models, such as by using neural network models to make inferences more efficient. Summary of the invention

[0006] Certain embodiments provide a method, comprising: receiving input data at a convolutional neural network (CNN) model; generating a decomposition computing network comprising multiple connections between a first layer of the CNN model and a second layer of the CNN model, wherein: the decomposition computing network comprises N inputs, the decomposition computing network comprises M outputs, and the decomposition computing network comprises at least one path from each of the N inputs to each of the M outputs; setting connection weights of multiple connections in the decomposition computing network to 1 so that a weight density of the decomposition computing network is <100%; performing fast point-by-point convolution using the decomposition computing network to generate fast point-by-point convolution output; and providing the fast point-by-point convolution output to the second layer of the CNN model.

[0007] Certain embodiments further provide a processing system comprising: a memory comprising computer-executable instructions; and a first processor configured to execute the computer-executable instructions and cause the processing system to perform the following operations: receive input data at a convolutional neural network (CNN) model; generate a decomposition computing network comprising multiple connections between a first layer of the CNN model and a second layer of the CNN model, wherein: the decomposition computing network comprises N inputs, the decomposition computing network comprises M outputs, and the decomposition computing network comprises at least one path from each of the N inputs to each of the M outputs; set connection weights of multiple connections in the decomposition computing network to 1 so that a weight density of the decomposition computing network is <100%; perform fast point-by-point convolution using the decomposition computing network to generate fast point-by-point convolution output; and provide the fast point-by-point convolution output to the second layer of the CNN model.

[0008] Certain embodiments further provide a non-transitory computer-readable medium comprising instructions that, when executed by a first processor of a processing system, cause the processing system to perform a method comprising: receiving input data at a convolutional neural network (CNN) model; generating a decomposed computational network comprising a plurality of connections between a first layer of the CNN model and a second layer of the CNN model, wherein: the decomposed computational network comprises N inputs, the decomposed computational network comprises M outputs, and the decomposed computational network comprises at least one path from each of the N inputs to each of the M outputs; setting connection weights of a plurality of connections in the decomposed computational network to 1 so that a weight density of the decomposed computational network is <100%; performing fast point-by-point convolution using the decomposed computational network to generate a fast point-by-point convolution output; and providing the fast point-by-point convolution output to the second layer of the CNN model.

[0009] The following description and related drawings set forth in detail certain illustrative features of the one or more embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings depict certain aspects of the one or more embodiments and therefore are not to be considered limiting the scope of the disclosure.

[0011] Figures 1A-1D Depicted are examples of various types of neural networks.

[0012] Figure 2 Depicted is an example of a point-wise convolution operation.

[0013] Figure 3 Depicted is an example network with input and output activations.

[0014] Figure 4 Depicts an example of a fast point-wise convolution method in a bottom-up approach.

[0015] Figure 5 Depicted is an example of a fast point-wise convolution method with a top-down approach.

[0016] Figure 6 Depicted is an example of fast point-wise convolution using a form of variable weight density reduction, where stages are either fully weighted or fully unweighted.

[0017] Fig. 7A and 7B Depicted are example calculations that illustrate the performance improvement of fast point-by-point convolution over conventional point-by-point convolution and butterfly transform point-by-point convolution in terms of both complexity and number of parameters.

[0018] Figure 8 An example method 800 for performing fast point-wise convolution is depicted.

[0019] Fig. 9 Depicted are example implementations of a system on a chip (SOC) that may be implemented using the embodiments described herein.

[0020] Fig.10 Depicted is an example schematic diagram of a multi-processor processing system that may be implemented using embodiments described herein.

[0021] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without specific recitation. DETAILED DESCRIPTION

[0022] Various aspects of the present disclosure provide apparatus, methods, processing systems, and computer-readable media for optimizing the performance of machine learning models, and in particular for improving the efficiency of point-by-point convolution.

[0023] Deep Neural Networks and Deep Learning

[0024] Deep learning architectures can perform complex tasks (such as object recognition) by learning to represent input at successively higher levels of abstraction in each layer, thereby building useful feature representations of the input data. In this way, deep learning solves the main bottleneck of conventional machine learning.

[0025] Before the advent of deep learning, machine learning approaches to tasks might rely heavily on human-engineered features, perhaps combined with shallow classifiers. A shallow classifier could be a two-class linear classifier, for example, where a weighted sum of input values ​​(e.g., input vector components) is compared to a threshold to predict which class the input belongs to. Human-engineered features could be templates or kernels that are customized for a specific problem domain by engineers with domain expertise.

[0026] In contrast, a deep learning architecture can learn to represent features similar to what a human engineer might design, but it learns through training. In addition, deep networks (e.g., deep neural networks) can learn to represent and recognize new types of features that humans may not have considered.

[0027] A deep learning architecture can be configured to learn a hierarchy of features. For example, if a first layer is presented with visual data, the first layer can learn to recognize relatively simple features (such as edges) in the input stream. In another example, if a first layer is presented with auditory data, the first layer can learn to recognize spectral power in specific frequencies. A second layer that takes the output of the first layer as input can learn to recognize combinations of features, such as recognizing simple shapes for visual data or recognizing combinations of sounds for auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0028] Deep learning architectures can perform particularly well when applied to problems that have a natural hierarchical structure. For example, the classification of motor vehicles can benefit from first learning to recognize wheels, windshields, and other features. These features can be combined in different ways at higher levels to recognize cars, trucks, and airplanes.

[0029] Neural networks can be designed with a variety of connectivity patterns. For example, in a feed-forward network, information is passed from lower layers to higher layers, with each neuron in a given layer communicating to neurons in higher layers.

[0030] Neural networks may also have reflow or feedback (also known as top-down) connections. A connection from the output of a neuron in a given layer to another neuron in the same layer is called a reflow connection. The reflow architecture may help identify patterns that span more than one chunk of input data delivered sequentially to the neural network. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. A network with many feedback connections may be helpful when recognition of high-level concepts can assist in discerning specific low-level features of the input.

[0031] The connections between layers of a neural network can be fully connected or partially connected.

[0032] Figure 1A An example of a fully connected neural network 102 is illustrated. In a fully connected neural network 102, a neuron in a first layer may communicate its output to each neuron in a second layer, such that each neuron in the second layer will receive input from each neuron in the first layer.

[0033] Figure 1BAn example of a locally connected neural network 104 is illustrated. In the locally connected neural network 104, neurons in a first layer may be connected to a limited number of neurons in a second layer. More generally, the locally connected layers of the locally connected neural network 104 may be configured such that each neuron in a layer will have the same or similar connectivity pattern, but their connection strengths may have different values ​​(e.g., 110, 112, 114, and 116). The connectivity patterns of the local connections may produce spatially distinct receptive fields in higher layers because higher layer neurons in a given area may receive inputs that are tuned through training to the properties of a limited portion of the total input to the network.

[0034] An example of a locally connected neural network is a convolutional neural network. Figure 1C An example of a convolutional neural network 106 is illustrated. The convolutional neural network 106 may be configured such that the connection strengths associated with the inputs to each neuron in the second layer are shared (e.g., 108). Convolutional neural networks may be well suited for problems where the spatial location of the inputs is meaningful.

[0035] The processing of each layer of the convolutional network can be thought of as a spatially invariant template or basis projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, then the convolutional network trained on this input can be thought of as three-dimensional, with two spatial dimensions along the axes of the image and a third dimension that captures color information. The output of the convolutional connection can be thought of as forming a feature map in the subsequent layer, each element of which receives input from a range of neurons in the previous layer and from each of the multiple channels. The values ​​in the feature map can be further processed with nonlinearities (such as rectified max(0,x)). The values ​​from adjacent neurons can be further pooled (which corresponds to downsampling) and can provide additional local invariance and dimensionality reduction.

[0036] One type of convolutional neural network is a deep convolutional network (DCN). A deep convolutional network (DCN) is a network of convolutional layers configured with additional pooling and normalization layers. A DCN can be trained using supervised learning, where both the input and output targets are known for many examples and are used to modify the weights of the network by using gradient descent.

[0037] The DCN may be a feed-forward network. In addition, as described above, connections from neurons in the first layer of the DCN to groups of neurons in the next higher layer are shared across neurons in the first layer. The feed-forward and shared connections of the DCN may be utilized for fast processing. The computational burden of the DCN may be much smaller than, for example, a neural network of similar size that includes recurrent or feedback connections.

[0038] Figure 1DA detailed example of a DCN 100 designed to recognize visual features of an image 126 input from an image capture device 130 (such as an onboard camera) is illustrated. The DCN 100 of the current example can be trained to identify traffic signs and numbers provided on traffic signs. Of course, the DCN 200 can be trained for other tasks, such as identifying lane markings or identifying traffic lights. These are just some example tasks, and many other tasks are possible.

[0039] The DCN 100 may be trained using supervised learning. During training, the DCN 100 may be presented with an image, such as an image 126 of a speed limit sign, and then a “forward pass” may be computed to produce an output 122 .

[0040] In this example, the DCN 100 includes a feature extraction section and a classification section. Upon receiving the image 126, the convolution layer 132 applies a convolution kernel (not shown) to the image 126 to generate a first set of feature maps 118. In general, a kernel includes a two-dimensional weight array specified to emphasize different aspects of the input data channel, while a filter includes a three-dimensional structure containing multiple kernels stacked together. Three-dimensional filters are frequently used in deep learning.

[0041] In one example, the convolution kernel of the convolution layer 132 may be a 5x5 kernel that generates a 28x28 feature map. Further in this example, since four different feature maps are generated in the first set of feature maps 118, four different convolution kernels are applied to the image 126 at the convolution layer 132. A convolution kernel may also be referred to as a filter or a convolution filter.

[0042] The first set of feature maps 118 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 120. The max pooling layer reduces the size of the first set of feature maps 118. That is, the size of the second set of feature maps 120 (such as 14x14) is smaller than the size of the first set of feature maps 118 (such as 28x28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 120 may be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent feature map sets (not shown).

[0043] exist Figure 1DIn the example of , the second set of feature maps 120 is convolved to generate a first feature vector 124. In addition, the first feature vector 124 is further convolved to generate a second feature vector 128. Each feature of the second feature vector 128 may include a number corresponding to a possible feature of the image 126 (such as "sign", "60", and "100"). A softmax function (not shown) may convert the numerical values ​​in the second feature vector 128 into probabilities. In this way, the output 122 of the DCN 100 is the probability that the image 126 includes one or more features.

[0044] In this example, the probabilities of "flag" and "60" in output 122 are higher than the probabilities of other numbers of output 122, such as "30", "40", "50", "70", "80", "90", and "100".

[0045] Before training DCN 100, output 122 generated by DCN 100 is likely to be incorrect. Thus, the error between output 122 and a target output can be calculated. The target output is the truth of image 126 (e.g., "sign" and "60"). The weights of DCN 100 can then be adjusted to more closely align output 122 of DCN 100 with the target output.

[0046] To adjust the weights of DCN 100, the learning algorithm may calculate a gradient vector for the weights. The gradient may indicate the amount by which the error will increase or decrease if the weights are adjusted. At the top layer, the gradient may directly correspond to the value of the weight connecting the activated neurons in the penultimate layer to the neurons in the output layer. In the lower layers, the gradient may depend on the value of the weights and the calculated error gradient of the higher layers. The weights may then be adjusted to reduce the error. This way of adjusting the weights may be referred to as "backward propagation" because it involves a "backward pass" in a neural network.

[0047] In practice, the error gradient of the weights may be calculated over a small number of examples so that the calculated gradient approximates the true error gradient. This approximation method may be referred to as stochastic gradient descent. Stochastic gradient descent may be repeated until the error rate achievable by the entire system has stopped decreasing or until the error rate has reached a target level. After learning, the DCN 100 may be presented with new images and a forward pass through the network may produce an output 122, which may be considered an inference or prediction of the DCN.

[0048] Convolution in Convolutional Neural Networks

[0049] Convolution is a data analysis technique used in, for example, signal processing, image processing, machine learning, and other technical fields. In deep learning, convolution is used to extract useful features from an input data set. For example, in a convolutional neural network, such as described above, convolution enables the use of filters to extract different features, and the weights of the filters are automatically learned during training. The extracted features are then combined to make inferences.

[0050] One approach to reducing the computational burden (e.g., measured in floating-point operations per second (FLOPs)) and the number of parameters associated with convolutional neural networks is to decompose the convolutional layer into two components using separable depthwise convolutions: (1) spatial fusion, where each spatial channel is independently convolved via depthwise convolutions; and (2) channel fusion, where all spatial channels are linearly combined via 1x1 convolutions, known as pointwise convolutions.

[0051] During spatial fusion, the network learns features from spatial planes, while during channel fusion, the network learns the relationship between these features across channels. Spatial fusion is sometimes implemented using 3x3 filters, and channel fusion is implemented using 1x1 filters. In general, channel fusion via point-wise convolution is useful for computationally efficient dimensionality reduction, efficient low-dimensional embedding or feature pooling, and reapplying nonlinearity after convolution, to name just a few examples.

[0052] For example, a point-wise convolution may use a 1x1xd kernel that iterates over every single point in the input image with a depth of d. The depth d of the point-wise convolution kernel typically matches the number of channels of the input data, such as an image with multiple channels for color data.

[0053] In general, an activation layer (such as ReLU, PReLU, Softmax or others) can be applied after each layer of a neural network. Applying a 1x1xd kernel and adding an activation layer after the kernel may increase the depth of the network, which may improve its performance.

[0054] Figure 2 An example of a point-wise convolution operation is depicted, where a 1x1x5 kernel 204 iterates over an 8x8x5 input image 202 to generate an 8x8x1 output image 206. As depicted in this example, the output image 206 has a reduced dimensionality (1 channel versus 8 channels), which allows for more efficient computation using the output image 206.

[0055] Although not in Figure 2, but multiple (e.g., m) point-wise convolution kernels (e.g., in filters) can be used to increase the number of output channels of the input data. Thus, for example, m=256 1x1x5 kernels 204 can be generated, each of which outputs an 8x8x1 output image (e.g., 206), and these output images can be stacked to obtain an 8x8x256 resulting image, i.e., an image of 8 pixels wide by 8 pixels high with a depth of 256 channels. The resulting increase in output channels provides more parameters for training, which can improve the model's ability to identify features (e.g., in input image 202).

[0056] Examining the computational profile of the decomposed convolutional neural network at inference time reveals that the computational burden of spatial fusion is much smaller than that of channel fusion. In fact, the computational complexity of the point-by-point convolution in channel fusion is quadratic in the number of channels (O(n 2 ), where n is the number of channels).

[0057] Figure 3 An example of a fully connected network 300 with N=4 inputs and M=4 outputs is depicted. To determine the complexity of performing a conventional point-wise convolution on the network 300, consider Figure X A point-wise (1x1) convolution that takes N activations and produces M activations as the output feature map Y. Therefore:

[0058] Y=WX,where X∈R N , W∈R N*M , and Y∈R M .

[0059] In a regular point-wise convolution operation, there are N*M multiplications between activations (X) and weights (W); hence, the complexity order is O(N 2 ).

[0060] Fast point-by-point convolution method

[0061] Described in this paper is a fast point-by-point convolution method that uses decomposition and reduced weight density rather than strict point-by-point multiplication to achieve significantly reduced operational complexity and reduced number of weights (parameters) using the operation Y = WX from the feature Figure X A similar transformation to the feature map Y.

[0062] For example, assume that X has shape [x0, x1, x2, x3], and Y has shape [y0, y1, y2, y3], along the x1 and y1 dimensions of X and Y, respectively, and further assume that size(x1) = N and size(y1) = M. Then, the conventional point-wise convolution is operated on the weights W with a complexity of N*M as defined below:

[0063] W M×N=[[w 01 , w 02 ,…,w 0(N-1) ],[w 11 , w 12 , ..., w 1(N-1) ],...,[w (M-1)1 , w (M-1)2 , ..., w (M-1)(N-1) ]]

[0064] In contrast, the method described herein uses decomposition and reduced weight density to reduce the number of weights and thus multiplications required for point-by-point convolution. Weight density generally refers to the percentage of weighted connections or edges in a network to all connections or edges in the network. In conventional point-by-point convolution, since all connections in the decomposition calculation network are weighted, the weight density is 100%. In contrast, the decomposition and weight density reduction described herein results in a required number of parameters and complexity of N*log2N+N, which is a significant improvement over N*M in both complexity and number of required parameters (e.g., Fig. 7A and 7B as depicted in ).

[0065] In general, the higher the weight density of a given network, the more trainable weights the network has, and the more expressive the resulting model based on the network is. However, as further described herein, the weight density can be varied to strike a suitable balance between trained model performance and training complexity.

[0066] The fast point-by-point convolution method described herein results in many improvements over conventional methods. For example, reducing the number of multiplications required to perform point-by-point convolution reduces power usage and latency in a processing system that performs point-by-point convolution. In addition, reducing the number of trainable weights reduces model size, which advantageously reduces memory usage (e.g., fewer memory transactions); required memory size (e.g., for holding smaller activations); and latency (because of fewer memory transactions). Reducing memory (e.g., on-chip memory) size further reduces power usage and cost. Therefore, fast point-by-point convolution is particularly advantageous for mobile devices and resource-constrained edge devices.

[0067] Fast point-wise convolutions are beneficial in any machine learning context where a certain accuracy can be traded for performance. For example, fast point-wise convolutions can be used to perform model compression before releasing the "run-time" weights in the actual product. The compressed weights can beneficially reduce the memory and processing requirements of the product.

[0068] Figure 4 An example of a fast point-wise convolution method in a bottom-up approach is depicted on a network 400. The network 400 may be referred to as a factorized computation network or graph.

[0069] In this example, the following conventions are used: ij is the weight of the jth downward connection in the i-th stage; g ij is the weight of the jth upward connection in the i-th stage; d ij is the weight of the jth direct connection in the i-th stage; and the weights of all other direct connections are equal to 1 (alternatively referred to as "unity"). The direct connection weights may alternatively be referred to as horizontal connection weights, and the upward and downward connection weights may alternatively be referred to as cross-connection weights. Furthermore, connections may alternatively be referred to as edges.

[0070] In general, computation network 400 includes a plurality of stages (402 and 404) having inter-stage connections (e.g., d) connecting between inputs (e.g., [x0, x1, x2, x3]), addition nodes (e.g., 408) (which may also be referred to as accumulator nodes), and outputs (e.g., [y0, y1, y2, y3]). ij 、f ij , and g ij The number of stages n can be determined by log2 N, where N is the number of inputs (eg, the number of input channels). Thus, in this example, n=log2 4=2.

[0071] In this example, in each stage of the decomposition, the inputs are organized into different pairs, which results in the input [x0, x1, x2, x3] being organized into pairs [x0, x1] and [x2, x3] in the first stage 402, and into [x0, x2] and [x1, x3] in the second stage 404.

[0072] Then, all direct connection weights (d ij ) and cross-connection weights (f ij and g ij ), which in this example includes d 00 、f 00 , g 00 d 01 d 02 、f 01 , g 01 , and d 03 However, in the second stage 404, only the cross-connect weights are calculated, which in this example include f 10 , g 10 、f 11 , and g 11 In this example, the second stage direct connection weights are set equal to 1 (or "unity"), which reduces the weight density of network 400 and therefore reduces its complexity.

[0073] Note that in this example, since the index origin of the input and output nodes is 0, the phase index numbers are 1 less than their ordinal references. For example, here, the phase index of "Phase 1" is 0, and the phase index of "Phase 2" is 1. These could alternatively be referred to as Phase 0 and Phase 1, consistent with the indexing.

[0074] It is noteworthy that network 400 is fully connected, in that every input x is connected to every output y, as Figure 3 , but the reduced number of trainable weights reduces the number of necessary multiplications when computing outputs based on fully connected inputs. Specifically, in this example, the direct connection weights 406 in the second stage 404 are all equal to 1, and therefore do not need to be trained or multiplied. Instead, the weights from the first stage 402 can be carried right along with the direct connection weights 404. Therefore, the decomposition of the network 400 in this example yields the following equation:

[0075] y0=d 00 x0+g 00 x1+d 02 g 10 x2+g 01 g 10 x3

[0076] y1=f 00 x0+d 01 x1+f 01 g 11 x2+d 03 g 11 x3

[0077] y2=d 00 f 10 x0+g 00 f 10 x1+d 02 x2+g 01 x3

[0078] y3=f 00 f 11 x0+d 01 f 11 x1+f 01 x2+d 03 x3

[0079] As depicted in the above set of equations, compared to conventional point-wise convolution (e.g., Figure 3Compared to the N*M=16 parameters required in the example shown in , the fast point-by-point convolution method has only 12 parameters. Therefore, in this example, the weight density of network 400 is 12 / 16=3 / 4 or 75%. Despite the significant reduction in complexity, the performance of the model obtained using the fast point-by-point convolution method is very similar to that of conventional point-by-point convolution. Therefore, in the context where performance priority outweighs absolute performance, fast point-by-point convolution is beneficial.

[0080] Figure 5 An example of a fast point-by-point convolution method with a top-down approach on network 500 is depicted. Computationally, this top-down replacement approach is equivalent to referring to Figure 4 The bottom-up approach described and depicted. However, implementation considerations may lead to a preference for one approach over another.

[0081] Here, again, in each stage of the decomposition, the inputs are organized into different pairs, and the number of stages n is again determined by n=log2N, where N is the number of inputs (e.g., input channels). Thus, in this example, the number of stages of the decomposition is determined as n=log2 4=2. As described above, the inputs [x0, x1, x2, x3] may again be organized into pairs [x0, x1] and [x2, x3] in the first stage 502, and into pairs [x0, x2] and [x1, x3] in the second stage 504.

[0082] In this example, only the first stage 502 cross-connect weights (f ij and g ij ), which in this example includes f 00 、f 01 , g 00 and g 01 Then, in the second stage 504, the direct connection weight (d ij ) and cross-connection weights (f ij and g ij ), which in this example includes d 10 、f 10 , g 10 d 11 d 12 、f 11 , g 11 and d 13 .

[0083] as Figure 4, where network 500 is fully connected because each input x is connected to each output y. And here, again, the use of variable weighted density across multiple stages (502 and 504) allows for a reduced number of trainable weights, and thereby a reduced number of multiplications. Specifically, in this example, the direct connection weights 506 in the first stage 504 are all equal to 1 (or identity), and therefore do not need to be trained or multiplied. Therefore, the decomposition of network 400 in this example yields the following equation:

[0084] y0=d 10 x0+g 10 x1+g 00 d 10 x2+g 01 g 10 x3

[0085] y1=f 10 x0+d 11 x1+g 00 f 10 x2+g 01 d 11 x3

[0086] y2=f 00 d 12 x0+f 01 g 11 x1+d 12 x2+g 11 x3

[0087] y3=f 00 f 11 x0+f 01 d 13 x1+f 11 x2+d 13 x3

[0088] Here, again, the same problem as would be encountered in a regular point-wise convolution (e.g., Figure 3 Compared to the N*M=16 parameters required in the example shown in FIG. 5 ), the fast point-by-point convolution method has only 12 parameters in the above set of equations. Therefore, here again, the weight density of network 500 is 3 / 4 or 75%.

[0089] A general mathematical model for fast point-wise convolution

[0090] According to the following Figure 3 and 4 With the described example, the general mathematical model for fast point-wise convolution can be described as follows.

[0091] Starting from the simple case, N = 2 input direct connection weights (D N ) can be expressed as follows:

[0092]

[0093] Similarly, the downward cross-connect weights (F N ) can be expressed as follows:

[0094]

[0095] And finally, the upward cross-connection weights (G N ) can be expressed as follows:

[0096]

[0097] Therefore, for M = 2 outputs, the following expression can be used:

[0098] Y2=(D2⊙F2⊙G2)X2

[0099] In the above equation, ⊙ represents an element-wise multiplication operation.

[0100] The above equations for the simple case of N=2 can be generalized to any N inputs, where N=2 n (or equivalently, n=log2N) and n is a positive integer, as shown below.

[0101] First, let represents the Kronecker multiplication operation. Then, from N = 2 n →2 n+1 , the weight matrix can be expressed as follows:

[0102]

[0103] Similarly, the downward cross-connect weights can be expressed as follows:

[0104]

[0105] And finally, the upward cross-connect weights can be expressed as follows:

[0106]

[0107] Therefore, the following general expression:

[0108] Y N =(D N ⊙F N ⊙G N )X N

[0109] Thus, for example, in another example with N=4 inputs and M=4 outputs, the following expression may be used:

[0110] Y4=(D4⊙F4⊙G4)X4

[0111] Furthermore, when N=4 inputs and M=4 outputs, the following equation can be derived:

[0112]

[0113] It is worth noting that the above formula regarding N=4 is only an example, and the same deduction can be made step by step at each stage of the network with different numbers N of input nodes.

[0114] In one example, the weight density of the computation graph may be reduced by setting some direct connection weights to 1, such as in the above reference. Figure 4 and 5 So, for example, for the example of N=4, the modified version of the direct connection weight matrix can be configured as (where one or more of the indicated d' weights are set to 1):

[0115]

[0116] It is worth noting that this is just one example and many others are possible.

[0117] Fast point-by-point convolution linear scaling method

[0118] Because, as mentioned above, when creating a decomposition computation network, the inputs are organized in pairs, the number of inputs N is preferably 2 n , where n is a positive integer. When N is not the number 2 n When , n can be derived as n = Ceil[log2(N)], where Ceil[] is an upper limit function that rounds real numbers toward positive infinity. Having such a definition for n ensures that the computation graph is always equal to or larger than the size required for any N that is a positive integer.

[0119] Furthermore, the above decomposition scheme is not directly applicable if the number of inputs N does not match the number of outputs M. In either case, linear scaling with empty activation skipping can be applied to the fast point-wise convolution computation network so that the fast point-wise convolution will work as described above.

[0120] The following is an example algorithm for applying linear scaling with null activation capability. Initially, N and M are assumed as the number of channels for input and output features, respectively. Subsequently, the following algorithm can be applied to fast pointwise convolution (FPC):

[0121] 1: Set numStages = int(ceil(log2(max(N, M))))) (set numStages to the smallest integer value greater than or equal to the log2 of the larger of N and M);

[0122] 2: Set N new =M new =2 numStages ; (Note that N new ≥ N and M new ≥M);

[0123] 3: N new -N zeros are appended to the input tensor along the channel dimension;

[0124] 4: Execute FPC(N new , M new ), instead of FPC(N,M), but skipping all multiplications with “0” values ​​as input activations; and

[0125] 5: At the output, return the first M output channels in a tensor along the channel dimension.

[0126] Assuming FPC(N new , M new ) is C. So the new complexity (or number of parameters) obtained by using the proposed linear scaling method becomes CNM / (N new M new ).

[0127] For example, the linear scaling method with empty activation skipping can be implemented on an ASIC neural network accelerator or other programmable core with dedicated instruction / accelerator extensions.

[0128] Fast point-wise convolution with skip connections

[0129] Conventional point-wise convolutions do not utilize "skip connections," which are connections that skip layers, and can therefore also be referred to as "shortcut" connections. The motivation for skipping layers is to simplify the network, using fewer layers in the initial training phase. This speeds up learning by reducing the effects of vanishing gradients, since there are fewer layers to propagate through. The network then gradually recovers the skipped layers as it learns the feature space.

[0130] In reference Figure 4 and 5 In the examples discussed, there are no “skip connections”. However, these examples can be adapted to use skip connections (ie, in conjunction with fast point-wise convolutions). The general form of fast point-wise convolutional skip connections can be expressed in the form of Y=WX defined as follows.

[0131] about Figure 4An example of bottom-up fast point-wise convolution in:

[0132] y0=(d 00 +I0)x0+g 00 x1+d 02 g 10 x2+g 01 g 10 x3

[0133] y1=f 00 x0+(d 01 +I1)x1+f 01 g 11 x2+d 03 g 11 x3

[0134] y2=d 00 f 10 x0+g 00 f 10 x1+(d 02 +I2)x2+g 01 x3

[0135] y3=f 00 f 11 x0+d 01 f 11 x1+f 01 x2+(d 03 +I3)x3

[0136] and about Figure 5 Example of fast top-down point-wise convolution in:

[0137] y0=(d 10 +I0)x0+g 10 x1+g 00 d 10 x2+g 01 g 10 x3

[0138] y1=f 10 x0+(d 11 +I1)x1+g 00 f 10 x2+g 01 d 11 x3

[0139] y2=f 00 d 12 x0+f 01 g 11 x1+(d 12 +I2)x2+g 11 x3

[0140] y3=f 00 f 11 x0+f 01 d 13 x1+f 11 x2+(d 13 +I3)x3

[0141] Although I0=I1=I2=I3 is individually configurable, simulations have shown that I0=I1=I2=I3=1 is effective. For example, skip connections equal to 1 are applied in combination with fast point-by-point convolutions to achieve equal or better accuracy performance with reduced computational complexity and using a reduced number of weights. Other choices of these values ​​are also possible.

[0142] Fast point-wise convolution with variable stage weighting

[0143] Figure 6 Depicted is an example of fast point-wise convolution using a form of variable weight density reduction, where each stage is either fully weighted or fully unweighted. This stage-based weight reduction may be referred to as structured variable weight density reduction. In the case of structured variable weight density reduction, the weight density may be more simply calculated as the percentage of weighted stages to all stages in the decomposed computation network.

[0144] exist Figure 6 In the depicted example, N=8, which results in n=log2N=3 stages. Furthermore, in this example, the weight density is 2 / 3 because the first stage 602 and the third stage 606 are weighted in a total of three stages (602, 604, and 606), which means that 2 / 3 of the possible connection weights include weights different from 1 (i.e., identity). In contrast, in Figure 4 and 5 In the equation, the weight density is 12 / 16=3 / 4=75%.

[0145] Furthermore, in this example: Stage 0 is a "weighted stage" that computes weights; Stage 1 is a "non-weighted stage" that does not compute weights (i.e., no direct or cross-connect weights); and Stage 2 is another weighted stage. It is worth noting that this is another example of a bottom-up approach, where the first stage (Stage 0) includes trainable direct connection weights (d ij weights) and cross-connection weights (f ij and g ij weights), while the final stage (stage 2) includes only trainable cross-connection weights.

[0146] The decomposition of network 600 in this example yields the following equation:

[0147] y0=d 00 d 20 x0+g00 d 20 x1+d 02 d 20 x2+g 01 d 20 x3+d 04 g 20 x4+g 02 g 20 x5+d 06 g 20 x6+g 03 g 20 x7

[0148] y1=f 00 d 21 x0+d 01 d 21 x1+f 01 d 21 x2+d 03 d 21 x3+f 02 g 21 x4+d 05 g 21 x5+f 03 g 21 x6+d 07 g 21 x7

[0149] y2=d 00 d 22 x0+g 00 d 22 x1+d 02 d 22 x2+g 01 d 22 x3+d 04 g 22 x4+g 02 g 22 x5+d 06 g 22 x6+g 03 g 22 x7

[0150] y3=f 00 d 23 x0+d 01 d 23 x1+f 01 d 23 x2+d 03 d 23 x3+f 02 g 23 x4+d 05 g 23 x5+f 03 g23 x6+d 07 g 23 x7

[0151] y4=d 00 f 20 x0+g 00 f 20 x1+d 02 f 20 x2+g 01 f 20 x3+d 04 d 24 x4+g 02 d 24 x5+d 06 d 24 x6+g 03 d 24 x7

[0152] y5=f 00 f 21 x0+d 01 f 21 x1+f 01 f 21 x2+d 03 f 21 x3+f 02 d 25 x4+d 05 d 25 x5+f 03 d 25 x6+d 07 d 25 x7

[0153] y6=d 00 f 22 x0+g 00 f 22 x1+d 02 f 22 x2+g 01 f 22 x3+d 04 d 26 x4+g 02 d 26 x5+d 06 d 26 x6+g 03 d 26 x7

[0154] y7=f 00 f 23 x0+d 01 f 23 x1+f 01 f 23 x2+d03 f 23 x3+f 02 d 27 x4+d 05 d 27 x5+f 03 d 27 x6+d 07 d 27 x7

[0155] In some cases, a sensitivity analysis may be performed to determine the impact (eg, on trained model performance) of treating a particular stage as a non-weighted stage or a weighted stage.

[0156] Example performance improvement using fast point-wise convolution

[0157] Fig. 7A and 7B Depicted are example calculations that illustrate the performance improvement of fast point-by-point convolution over conventional point-by-point convolution and butterfly transform point-by-point convolution, both in terms of complexity and number of parameters.

[0158] from Fig. 7A It is clear from the table in that there is a dramatic reduction in the number of parameters (e.g., weights) used for N→N point-by-point convolutions when comparing fast point-by-point convolutions to conventional point-by-point convolutions, and even to point-by-point convolutions with improved performance (such as point-by-point convolutions via a butterfly transform). As discussed above, this parameter reduction translates directly into performance improvements for processing systems that perform point-by-point convolutions. Furthermore, as discussed above, the reduction in parameter count means that the hardware requirements for supporting models that rely on point-by-point convolutions can be beneficially reduced, which further saves chip area, power usage, and cost, to name just a few advantages.

[0159] Similarly, Figure 7B The graph depicted in depicts the complexity reduction of the fast point-by-point convolution as described herein compared to conventional point-by-point convolution and butterfly transform point-by-point convolution in terms of the number of multiply-accumulate operations. In a processing system performing point-by-point convolution, the reduction in multiply-accumulate operations beneficially reduces latency and power consumption.

[0160] Example method for performing fast point-wise convolution

[0161] Figure 8 An example method 800 for performing fast point-wise convolution is depicted.

[0162] Method 800 begins at step 802 by receiving input data at a convolutional neural network (CNN) model.

[0163] The method 800 then proceeds to step 804 to generate a decomposed computation network including a plurality of connections between the first layer of the CNN model and the second layer of the CNN model. In some embodiments, the decomposed computation network includes N inputs. In some embodiments, the decomposed computation network includes M outputs. In some embodiments, the decomposed computation network includes at least one path from each of the N inputs to each of the M outputs.

[0164] The method 800 then proceeds to step 806 to set the connection weights of the plurality of connections in the decomposition computing network to 1, so that the weight density of the decomposition computing network is <100%.

[0165] The method 800 then proceeds to step 808 to perform fast point-by-point convolution using the decomposed computation network to generate a fast point-by-point convolution output.

[0166] Method 800 then proceeds to step 810 to provide the fast point-by-point convolution output to the second layer of the CNN model.

[0167] In some embodiments of method 800, the decomposition computation network includes a number, n=log2N, of stages.

[0168] In some embodiments of method 800, the plurality of connections include a plurality of weighted connections connected to an addition node between a first layer of the CNN model and a second layer of the CNN model, such as described above in one embodiment with reference to Figure 4-6 as described.

[0169] In some embodiments, setting the connection weights of the plurality of connections in the decomposed computation network to 1 includes setting all connection weights of at least one stage in the plurality of stages to 1, such as described above with reference to Figure 6 In some embodiments, setting the connection weights of the plurality of connections in the decomposed computation network to 1 includes setting a subset of the connection weights of at least one of the plurality of stages to 1, such as described above with reference to Figure 4 and 5 as described.

[0170] In some embodiments, method 800 further includes: performing linear scaling with empty activation skipping on the decomposed computation network, such as when N≠2 n and n is a positive integer, or when N≠M.

[0171] In some embodiments, such as those performing linear scaling with null activations, providing the fast point-wise convolution output to the second layer of the CNN model includes providing only the first M channels of the fast point-wise convolution output to the second layer of the CNN model.

[0172] In some embodiments, method 800 further includes adding a plurality of skip connections to the decomposed computation network.

[0173] In some embodiments, method 800 further includes: generating a model output from the CNN model; and providing the generated model output to a low power application running on the mobile electronic device.

[0174] In some embodiments, method 800 further includes: performing a first inference based on the fast point-by-point convolution output by the first processor; performing a second fast point-by-point convolution using the decomposition computation network by the second processor to generate a second fast point-by-point convolution output; and providing the second fast point-by-point convolution output to a second layer of the CNN model. In other words, in a multi-processor processing system, the fast point-by-point convolution can be parallelized across hardware resources. For example, the first processor and the second processor can be the following reference Fig. 9 and / or one of the processors described in 10.

[0175] In some embodiments of method 800, from a sensor in a mobile electronic device (e.g., Fig. 9 In some embodiments, the input data is associated with an application running on the mobile electronic device and is configured for at least one of: biometric detection, human presence detection, environmental condition detection, object detection, or object classification.

[0176] In some embodiments of method 800, the input data is one of image data, video data, audio data, or signal data. In other embodiments, the input data may represent other "features" that have been quantized.

[0177] Example processing system for optimizing machine learning model performance

[0178] Fig. 9 An example implementation of a system on a chip (SOC) 900 according to certain aspects of the present disclosure is illustrated, which may include a central processing unit (CPU) 902 or a multi-core CPU configured to perform a parallel Monte Carlo dropout function. Variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency bin information, and task information may be stored in a memory block associated with a neural processing unit (NPU) 908, a memory block associated with a CPU 902, a memory block associated with a graphics processing unit (GPU) 904, a memory block associated with a digital signal processor (DSP) 906, a memory block 918, or may be distributed across multiple blocks. Instructions executed at the CPU 902 may be loaded from a program memory associated with the CPU 902 or may be loaded from a memory block 918.

[0179] The SOC 900 may also include additional processing blocks tailored for specific functions, such as a GPU 904, a DSP 906, a connectivity block 910, which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc., and a multimedia processor 912 that may detect and recognize gestures, for example. In one implementation, the NPU is implemented in the CPU 902, the DSP 906, and / or the GPU 904. The SOC 900 may also include one or more sensor processors 914 associated with any kind of sensor, one or more image signal processors (ISPs) 916, and / or a navigation module 920 (which may include a global positioning system).

[0180] SOC 900 may be based on the ARM instruction set. In one aspect of the present disclosure, the instructions loaded into CPU 902 may include code for searching a stored multiplication result in a lookup table (LUT) corresponding to the product of the input value and the filter weight. The instructions loaded into CPU 902 may also include code for disabling a multiplier during a multiplication operation of the multiplication product when a lookup table hit of the multiplication product is detected. In addition, the instructions loaded into CPU 902 may include code for storing the calculated product of the input value and the filter weight when a lookup table miss of the product is detected.

[0181] SOC 900 and / or its components may be configured to perform the methods described herein.

[0182] Fig.10 Depicted is an example schematic diagram of a multi-processor processing system 1000 that may be implemented using embodiments described herein.

[0183] In this example, system 1000 includes processors 1001, 1003, and 1005, but in other examples, any number of individual processors may be used. In addition, although similarly described, processors 1001, 1003, and 1005 may represent various different types of processors in electronic devices, such as CPUs, GPUs, DSPs, NPUs, etc. described herein.

[0184] Each of the processors 1001, 1003, and 1005 includes an instruction scheduler, various hardware subcomponents (e.g., hardware X, hardware Y, and hardware Z), and a local memory. In some embodiments, the local memory may be a tightly coupled memory (TCM). Note that the components of each of the processors 1001, 1003, and 1005 are shown as being the same in this example, and each processor may have a different hardware configuration, different hardware elements, etc.

[0185] Each of the processors 1001 , 1003 , and 1005 is also in data communication with a global memory, such as a DDR memory or other type of volatile working memory.

[0186] In some implementations, in a multi-processor processing system such as 1000, one of the processors may act as a master processor. For example, in this example, processor 1001 may be the master processor. The master processor may include a compiler that, when executed, may determine how a model (such as a neural network model) will be processed by various components of the processing system 1000. For example, hardware parallelism may be achieved by mapping processing portions of a model to various hardware (e.g., hardware X, hardware Y, and hardware Z) within a given processor (e.g., processor 1001), and mapping processing portions of the model to other processors (e.g., processors 1003 and 1005) and their associated hardware.

[0187] The previous description is provided to enable any person skilled in the art to practice the various embodiments described herein. The examples discussed herein are not limitations on the scope, applicability, or embodiments set forth in the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the universal principles defined herein may be applied to other embodiments. For example, changes may be made to the functions and arrangements of the elements discussed without departing from the scope of the present disclosure. Various examples may appropriately omit, replace, or add various procedures or components. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Moreover, the features described with reference to some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement a device or practice a method. In addition, the scope of the present disclosure is intended to cover such devices or methods practiced using other structures, functionality, or structures and functionality that are supplementary to or different from the various aspects of the present disclosure set forth herein. It should be understood that any aspect of the present disclosure disclosed herein may be implemented by one or more elements of the claims.

[0188] As used herein, the term “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0189] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0190] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a database, or another data structure), ascertaining, and the like. Also, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. "Determining" may also include resolving, selecting, choosing, establishing, and the like.

[0191] Each method disclosed herein includes one or more steps or actions for implementing the method. These method steps and / or actions can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions can be changed without departing from the scope of the claims. In addition, the various operations of the above-mentioned methods can be performed by any suitable device capable of performing the corresponding functions. These devices may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs), or processors. Generally, where there are operations illustrated in the accompanying drawings, these operations may have corresponding paired devices plus functional components with similar numbers.

[0192] The following claims are not intended to be limited to the embodiments shown herein, but should be granted the full scope consistent with the language of the claims. Within the claims, references to singular elements are not intended to mean "there is and only one" (unless specifically stated), but "one or more". Unless otherwise specifically stated, the term "some" means one or more. No element of a claim should be interpreted under the provisions of 35 USC § 112 (f) unless the element is explicitly described using the phrase "device for..." or in the case of a method claim, the element is described using the phrase "step for...". The elements of the various aspects described throughout the present disclosure are all structural and functional equivalents currently or hereafter known to those of ordinary skill in the art, and are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, regardless of whether such disclosure is explicitly recorded in the claims, the content disclosed herein is not intended to be donated to the public.

Claims

1. A computer-implemented method comprising: Receiving input data at a convolutional neural network (CNN) model, the input data being one of image data, video data, audio data, or signal data; Generate a decomposed computational network including a plurality of connections between a first layer of the CNN model and a second layer of the CNN model, wherein: The decomposition computing network includes N inputs, The decomposition computation network includes M outputs, and The decomposed computation network includes at least one path from each of the N inputs to each of the M outputs; The connection weights for the plurality of connections include a direct connection weight matrix D N , downward cross connection weight matrix F N and the upward cross-connection weight matrix G N , these connection weight matrices are defined as iteration N = 2 n , n is a positive integer; performing fast point-wise convolution using the decomposed computation network to generate a fast point-wise convolution output; and The fast point-by-point convolution output is provided to the second layer of the CNN model.

2. The method of claim 1, further comprising: The connection weights of multiple connections in the decomposed computing network are set to 1, so that the weight density of the decomposed computing network is less than 100%.

3. The method of claim 1, wherein: The decomposition computation network includes a number of stages n=log2N, and The multiple connections include multiple weighted connections connected to an addition node between the first layer of the CNN model and the second layer of the CNN model. 4 . The method of claim 3 , wherein setting the connection weights of the plurality of connections in the decomposed computation network to 1 comprises setting all connection weights of at least one stage in the plurality of stages to 1. 5 . The method of claim 3 , wherein setting the connection weights of the plurality of connections in the decomposed computation network to 1 comprises setting a subset of the connection weights of at least one stage in the plurality of stages to 1. 6 .

6. The method of claim 1, wherein: N≠2 n Or N ≠ M, and The method further comprises: performing linear scaling with empty activation skipping on the factorized computation network, and Providing the fast point-by-point convolution output to the second layer of the CNN model includes providing only the first M channels of the fast point-by-point convolution output to the second layer of the CNN model.

7. The method of claim 1, further comprising: A plurality of skip connections are added to the decomposed computation network.

8. The method of claim 1, further comprising: Generating a model output from the CNN model; as well as The model output is provided to a low power application running on the mobile electronic device.

9. The method of claim 1, further comprising: performing, by a first processor, a first inference based on the fast point-by-point convolution output; performing, by a second processor, a second fast point-wise convolution using the decomposition computation network to generate a second fast point-wise convolution output; as well as The second fast point-by-point convolution output is provided to the second layer of the CNN model.

10. The method of claim 1, wherein: The input data is received from a sensor in the mobile electronic device, and The input data is associated with an application running on the mobile electronic device and is configured for at least one of: biometric detection, human presence detection, environmental condition detection, object detection, or object classification.

11. A processing system comprising: a memory including computer executable instructions; as well as A first processor is configured to execute the computer executable instructions and cause the processing system to: Receiving input data at a convolutional neural network (CNN) model, the input data being one of image data, video data, audio data, or signal data; Generate a decomposed computational network including a plurality of connections between a first layer of the CNN model and a second layer of the CNN model, wherein: The decomposition computing network includes N inputs, The decomposition computation network includes M outputs, and The decomposed computation network includes at least one path from each of the N inputs to each of the M outputs; The connection weights for the plurality of connections include a direct connection weight matrix D N , downward cross connection weight matrix F N and the upward cross-connection weight matrix G N , these connection weight matrices are defined as iteration N = 2 n , n is a positive integer; performing fast point-wise convolution using the decomposed computation network to generate a fast point-wise convolution output; and The fast point-by-point convolution output is provided to the second layer of the CNN model.

12. The processing system of claim 11, wherein the first processor is further configured to cause the processing system to: set connection weights of a plurality of connections in the decomposed computing network to 1 so that a weight density of the decomposed computing network is <100%.

13. The processing system of claim 11, wherein: The decomposition computation network includes a number of stages n=log2N, and The multiple connections include multiple weighted connections connected to an addition node between the first layer of the CNN model and the second layer of the CNN model.

14. The processing system of claim 13, wherein the plurality of connections set to 1 in the decomposed computation network include all connection weights of at least one stage in the plurality of stages.

15. The processing system of claim 13, wherein the plurality of connections set to 1 in the decomposed computation network comprises a subset of connection weights of at least one stage in the plurality of stages.

16. The processing system of claim 11, wherein: N≠2 n Or N ≠ M, and The first processor is further configured to cause the processing system to: performing linear scaling with null activation skipping on the factorized computation network, and As the fast point-by-point convolution output, only the first M channels of the fast point-by-point convolution output are provided to the second layer of the CNN model.

17. The processing system of claim 11, further comprising: A plurality of skip connections are added to the decomposed computation network.

18. The processing system of claim 11, wherein the first processor is further configured to cause the processing system to: Generating a model output from the CNN model; and The model output is provided to a low power application running on the mobile electronic device.

19. The processing system of claim 11, wherein: The first processor is further configured to cause the processing system to perform a first inference based on the fast point-by-point convolution output, and The processing system further includes a second processor configured to cause the processing system to: performing a second fast point-by-point convolution using the decomposition computation network to generate a second fast point-by-point convolution output; and The second fast point-by-point convolution output is provided to the second layer of the CNN model.

20. The processing system of claim 11, wherein: The input data is received from a sensor in the mobile electronic device, and The input data is associated with an application running on the mobile electronic device and is configured for at least one of: biometric detection, human presence detection, environmental condition detection, object detection, or object classification.

21. A non-transitory computer readable medium comprising instructions that, when executed by a first processor of a processing system, cause the processing system to perform a method comprising: Receiving input data at a convolutional neural network (CNN) model, the input data being one of image data, video data, audio data, or signal data; Generate a decomposed computational network including a plurality of connections between a first layer of the CNN model and a second layer of the CNN model, wherein: The decomposition computing network includes N inputs, The decomposition computation network includes M outputs, and The decomposed computation network includes at least one path from each of the N inputs to each of the M outputs; The connection weights for the plurality of connections include a direct connection weight matrix D N , downward cross connection weight matrix F N and the upward cross-connection weight matrix G N , these connection weight matrices are defined as iteration N = 2 n , n is a positive integer; performing fast point-wise convolution using the decomposed computation network to generate a fast point-wise convolution output; and The fast point-by-point convolution output is provided to the second layer of the CNN model.

22. The non-transitory computer readable medium of claim 21, wherein the method further comprises: The connection weights of multiple connections in the decomposed computing network are set to 1, so that the weight density of the decomposed computing network is less than 100%.

23. The non-transitory computer readable medium of claim 21, wherein: The decomposition computation network includes a number of stages n=log2N, and The multiple connections include multiple weighted connections connected to an addition node between the first layer of the CNN model and the second layer of the CNN model.

24. The non-transitory computer-readable medium of claim 23, wherein setting the connection weights of the plurality of connections in the decomposed computation network to 1 comprises: All connection weights of at least one stage among the plurality of stages are set to 1.

25. The non-transitory computer-readable medium of claim 23, wherein setting the connection weights of the plurality of connections in the decomposed computation network to 1 comprises: A subset of connection weights of at least one stage in the plurality of stages is set to 1.

26. The non-transitory computer readable medium of claim 21, wherein: N≠2 n Or N ≠ M, and The method further comprises: performing linear scaling with empty activation skipping on the factorized computation network, and Providing the fast point-by-point convolution output to the second layer of the CNN model includes providing only the first M channels of the fast point-by-point convolution output to the second layer of the CNN model.

27. The non-transitory computer readable medium of claim 21, wherein the method further comprises: A plurality of skip connections are added to the decomposed computation network.

28. The non-transitory computer readable medium of claim 21, wherein the method further comprises: Generating a model output from the CNN model; as well as The model output is provided to a low power application running on the mobile electronic device.

29. The non-transitory computer readable medium of claim 21, wherein the method further comprises: performing, by a first processor, a first inference based on the fast point-by-point convolution output; performing, by a second processor, a second fast point-wise convolution using the decomposition computation network to generate a second fast point-wise convolution output; as well as The second fast point-by-point convolution output is provided to the second layer of the CNN model.

30. The non-transitory computer readable medium of claim 21, wherein: The input data is received from a sensor in the mobile electronic device, and The input data is associated with an application running on the mobile electronic device and is configured for at least one of: biometric detection, human presence detection, environmental condition detection, object detection, or object classification.

Images