Bit width selection for fixed-point neural networks

Abstract

The present application relates to bit width selection for fixed-point neural networks. A method for selecting the bit width of a fixed-point machine learning model includes evaluating the sensitivity of model accuracy to bit width at each computational level of the model. The method also includes selecting the bit width of parameters and / or intermediate calculations in the computational levels of the model. The bit width of the parameter and the bit width of the intermediate calculation can be different. The selected bit width can be determined based on the sensitivity evaluation.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 159,097, filed on May 8, 2015, entitled “BIT WIDTH SELECTION FOR FIXEDPOINT NEURAL NETWORKS,” the disclosure of which is expressly incorporated herein by reference in its entirety. background

[0003] field

[0004] Certain aspects of the present disclosure relate generally to machine learning, and more particularly to systems and methods for improving bitwidth selection for fixed-point machine learning models, such as artificial neural networks.

[0005] background

[0006] An artificial neural network, which may include a group of interconnected artificial neurons (eg, a neuron model), is a computing device or represents a method to be performed by a computing device.

[0007] A convolutional neural network is a feed-forward artificial neural network. A convolutional neural network may include a collection of neurons, each of which has a receptive field and collectively spells out an input space. Convolutional neural networks (CNNs) have many applications. In particular, CNNs have been widely used in the fields of pattern recognition and classification.

[0008] Deep learning architectures, such as deep belief networks and deep convolutional networks, are layered neural network architectures where the outputs of the first layer of neurons become the inputs of the second layer of neurons, the outputs of the second layer of neurons become the inputs of the third layer of neurons, and so on. Deep neural networks can be trained to recognize hierarchies of features and therefore they have been increasingly used in object recognition applications. Similar to convolutional neural networks, computations in these deep learning architectures can be distributed across a population of processing nodes, which can be configured in one or more computational chains. These multi-layer architectures can be trained one layer at a time and can involve fine-tuning using backpropagation.

[0009] Other models can also be used for object recognition. For example, support vector machine (SVM) is a learning tool that can be applied to classification. Support vector machine includes a separating hyperplane (e.g., decision boundary) that classifies data. The hyperplane is defined by supervised learning. The desired hyperplane increases the margin of training data. In other words, the hyperplane should have the maximum minimum distance to the training examples.

[0010] Although these solutions achieve excellent results on several classification benchmarks, their computational complexity can be extremely high. In addition, training the models can be challenging.

[0011] Overview

[0012] In one aspect of the present disclosure, a method for selecting the bit width of a fixed-point machine learning model is provided. The method includes evaluating the sensitivity of the model accuracy at each computational level of the machine learning model to the bit width. The method also includes selecting the bit width of parameters and / or intermediate calculations in the computational level of the model based on the sensitivity evaluation. The bit width of the parameter and the bit width of the intermediate calculation can be different.

[0013] In another aspect of the present disclosure, a device for selecting a bit width of a fixed-point machine learning model is provided. The device includes a memory and at least one processor coupled to the memory. The one or more processors are configured to evaluate the sensitivity of the model accuracy at each computational level of the machine learning model to the bit width. The processor(s) are further configured to select the bit width of parameters and / or intermediate calculations in the computational level of the model based on the sensitivity evaluation. The bit width of the parameter and the bit width of the intermediate calculation can be different.

[0014] In yet another aspect of the present disclosure, an apparatus for selecting a bit width of a fixed-point machine learning model is provided. The apparatus includes means for evaluating the sensitivity of the model accuracy at each computational stage of the machine learning model to the bit width. The apparatus further includes means for selecting the bit width of parameters and / or intermediate calculations in the computational stages of the model based on the sensitivity evaluation. The bit width of the parameter and the bit width of the intermediate calculation may be different.

[0015] According to another aspect of the present disclosure, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium has program code encoded thereon for selecting the bit width of a fixed-point machine learning model. The program code is executed by a processor and includes program code for evaluating the sensitivity of the model accuracy at each computational level of the machine learning model to the bit width. The program code also includes program code for selecting the bit width of parameters and / or intermediate calculations in the computational level of the model based on the sensitivity evaluation. The bit width of the parameter and the bit width of the intermediate calculation can be different.

[0016] Additional features and advantages of the present disclosure will be described below. Those skilled in the art will appreciate that the present disclosure can be easily used as a basis for modifying or designing other structures for implementing the same purpose as the present disclosure. Those skilled in the art will also recognize that such equivalent constructions do not depart from the teachings of the present disclosure as set forth in the appended claims. The novel features that are considered to be characteristic of the present disclosure, both in terms of its organization and method of operation, together with further objects and advantages, will be better understood when the following description is considered in conjunction with the accompanying drawings. However, it is to be clearly understood that each of the drawings is provided for illustration and description purposes only and is not intended as a definition of limitations of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The features, nature and advantages of the present disclosure will become more apparent when the detailed description set forth below is read in conjunction with the accompanying drawings, in which like reference numerals designate correspondingly throughout.

[0018] Figure 1 An example implementation of designing a neural network using a system on a chip (SOC) including a general-purpose processor in accordance with certain aspects of the present disclosure is illustrated.

[0019] Figure 2 Example implementations of systems according to aspects of the present disclosure are illustrated.

[0020] Figure 3A is a diagram illustrating a neural network in accordance with aspects of the present disclosure.

[0021] Figure 3B is a block diagram illustrating an exemplary deep convolutional network (DCN) in accordance with aspects of the present disclosure.

[0022] Figure 4 and 5 Methods for selecting the bitwidth of a fixed-point machine learning model according to aspects of the present disclosure are explained. Detailed Description

[0023] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. This detailed description includes specific details in order to provide a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0024] Based on this teaching, it will be appreciated by those skilled in the art that the scope of the present disclosure is intended to cover any aspect of the present disclosure, whether it is implemented independently or in combination with any other aspect of the present disclosure. For example, any number of aspects described can be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover such devices or methods that are practiced using other structures, functionality, or structures and functionality that are supplemented or different from the various aspects of the present disclosure described. It should be understood that any aspect of the present disclosure disclosed can be implemented by one or more elements of the claims.

[0025] The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0026] Although specific aspects are described herein, numerous variations and permutations of these aspects fall within the scope of the present disclosure. Although some benefits and advantages of preferred aspects are mentioned, the scope of the present disclosure is not intended to be limited to specific benefits, uses or goals. On the contrary, various aspects of the present disclosure are intended to be broadly applicable to different technologies, system configurations, networks and protocols, some of which are illustrated as examples in the accompanying drawings and the following description of preferred aspects. The detailed description and drawings merely illustrate the present disclosure and do not limit the present disclosure, and the scope of the present disclosure is defined by the attached claims and their equivalent technical solutions.

[0027] Bit width selection for fixed-point neural networks

[0028] Machine learning models can be configured using fixed-point implementation or floating-point implementation. In floating-point implementation, elements and parameters of the model and input data can be represented using floating-point numbers. On the other hand, in fixed-point representation, a fixed position of the decimal point is selected so that there is a fixed number of bits to the right and / or left of the decimal point, and the fixed position of the decimal point is used to represent the elements, parameters, and inputs of the model. Floating-point representation can provide greater precision. On the other hand, fixed-point implementation can be desirable for machine learning models such as artificial neural networks (ANNs). Specifically, fixed-point implementation can reduce model size, shorten processing time, reduce memory bandwidth, and / or reduce power consumption.

[0029] One challenge of fixed-point implementation is determining the bit widths for different components in a representation model (e.g., a neural network). In some cases, a specific bit width may be used throughout the network. However, it may be more desirable to use different bit widths in different components of the network. For example, different bit widths may be selected for bias values, activation values, and / or weights for each layer of the neural network. Aspects of the present disclosure relate to selecting bit widths for different layers of the ANN and / or different components of each layer of the ANN. Additionally, aspects of the present disclosure relate to changing the bit width based on performance specifications and system resources.

[0030] In some aspects, the signal-to-quantization noise ratio (SQNR) can be used to evaluate model performance. That is, in a machine learning model (such as a deep convolutional network), the effect of quantizing weights and / or activations is to introduce quantization noise. Similar to other communication systems, when quantization noise increases, model performance decreases. Accordingly, the SQNR observed at the output can provide an indication of model performance or accuracy.

[0031] In some aspects, the model quantizer can have a fixed efficiency of K dB / bit. In other words, each bit in the fixed-point representation contributes an SQNR of K dB. Thus, the SQNR can be used to select an improved or optimized bit width. The bit width can be selected for the entire model or can be selected on a computational level basis (e.g., a layer of a deep convolutional network (DCN)). For example, given an SQNR target at the output of a DCN (with N layers) of Y dB (where noise is introduced due to quantization), the total bit width in the model can be selected by minimizing the following formula:

[0032]

[0033] where K represents the quantizer efficiency, and x i represents the quantization noise variance (or power) introduced at each computational level (e.g., layer) of the model. In some aspects, for simplicity of notation, the signal variance (or power) at each level can be assumed to be normalized to 1. The bit width selection may be subject to certain constraints. For example, in some aspects, the bit width selection may be subject to a threshold value of the SQNR at the output of the model, which can be expressed as:

[0034]

[0035] Therefore, Equations 1 and 2 can be considered as the SQNR budget of the machine learning model.

[0036] Equation 2 may represent the allowable quantization noise injected. Because the model may be linear in the region of interest, the noise may be additive from one computational level to the next. Thus, the objective function of Equation 1 is the total number of bits or bit width at each computational level or layer.

[0037] In some ways, the bit width selection can be simplified to:

[0038] min-∑ρ i log(x i ),st∑α i x i G, (3)

[0039] where α iis the noise amplification or reduction factor from layer i to the output, C is a constant constraining the α factor, and ρ i is the scaling factor of the bit width at layer i. In one exemplary aspect, i represents the number of parameters when the goal is to reduce the model size. In another example, ρ i represents the number of multiply-accumulates (MACs) per layer when the goal is to reduce computation. In some aspects, the constant C may be calculated based on the SQNR limit.

[0040] The noise that can be introduced for each layer can be proportional to the bit width scaling factor and inversely proportional to the amplification factor. If a particular layer has a high scaling factor (e.g., because there are many neurons), the quantizer can increase the bit width reduction if the goal is to reduce the overall model size. Accordingly, the bit width selection can be similar to the water injection equation and is given by:

[0041]

[0042] The bit width selection equation also recognizes and accounts for differences in the layers of the neural network. That is, the bit width selection can be more influential in some layers than in other layers. For example, the fully connected layers of the ANN are more robust to quantization noise than the convolutional layers. That is, the fan-in at the fully connected layers can be larger, thereby resulting in suppression of quantization noise. Additionally, if it is desired to reduce the model size, it is desirable to have a larger quantization for the fully connected layers compared to the quantization of the convolutional layers. It is worth noting that the number of parameters in the fully connected layers can be greater than the number of parameters of the convolutional layers. Thus, quantizing the fully connected layers can result in a greater reduction in model size than quantizing the convolutional layers.

[0043] In another example, comparing a layer with one million weights and a layer with one thousand weights, for the one million weight layer, a smaller reduction in bit width can produce a significant change in the overall model size. On the other hand, in a layer with one thousand weights, a larger bit width may have a smaller impact.

[0044] In one example, for the purpose of simplicity and ease of understanding, the scaling factor and noise amplification factor can be assumed to be the same across layers. i can be equal, and the SQNR budget for each layer can be the same. In addition, if N layers are assumed, the SQNR specification at each layer can be SQNR 最小 +10log N. Specifically, in one example, where SQNR 最小 =26dB and N=10, the SQNR specification for each layer is 36dB (or 12 bits if 3dB / bit is assumed).

[0045] In some aspects, additional safety factors may be added to account for non-Gaussian distributions of activations and weights and / or variants for different training and test sets. For example, assume a quantization efficiency of K = 3dB / bit. With a bit width of 12, the SQNR specification may be 36dB SQNR per layer. However, in reality, the quantization efficiency may be 2.5dB / bit because, for example, the input distribution at test time may be different. Thus, the actual SQNR per layer may be 30dB SQNR per layer. The resulting SQNR at the output will be 20dB, which is lower than the SQNR 最小 One way to prevent this from happening is to add a safety factor (e.g., +12dB) to the SQNR specification at each layer. That is, instead of a 36dB SQNR per layer, a target of 48SQNR per layer can be used. This is equivalent to a bit width of 16. Thus, even if the actual quantization efficiency is 2.5dB / bit, a 40dB SQNR per layer can be achieved, resulting in a 30dB SQNR at the output, which is better than the SQNR. 最小 high.

[0046] The bit width selection techniques described above can also be applied to components of a computational level (e.g., layer). For example, in some cases, a layer of an ANN (such as a DCN) can refer to biases, weights, and / or activation values. Still, the quantization operation that introduces quantization noise can be considered to affect the layer for the purpose of selecting the bit width. That is, each bias, weight, and activation value can be considered a layer. In other words, in the above SQNR calculation, the process of quantizing weights and biases can be considered a "layer", and quantizing activations can be considered another "layer."

[0047] For example, for the purpose of SQNR calculation, a DCN with 3 convolutional layers can have 6 quantization "layers", or the following steps:

[0048] - quantize the weights and biases of convolutional layer (conv) 1,

[0049] - quantize the activations of conv1,

[0050] - quantize the weights and biases of conv2,

[0051] - quantize the activations of conv2,

[0052] - quantize the weights and biases of conv3,

[0053] - Quantize the activations of conv3.

[0054] Each quantization layer or step may introduce a certain amount of quantization noise.

[0055] In one example, the weight w is multiplied by the activation a. Both the weight and the activation may include inherent quantization noise nw and n a . Because (w+n w )×(α+n a )≈w·a+w·n a +n w a, so the new SQNR due to the noise phenomenon can satisfy:

[0056]

[0057] It follows that introducing quantization noise into weights and activations independently can be analogous to adding total noise after the product operation in a normalized system.

[0058] In some cases, the bit width of the activation values ​​can be smaller than the bit width of the weights (or vice versa). In addition, the bit width selection can be based on the number of weights per layer in the model, the number of activation values ​​per layer, the filter size per layer, the filter stride per layer, and / or the number of filters per layer.

[0059] Transfer Learning

[0060] In some aspects, transfer learning can be adopted to perform learning tasks. In this case, a machine learning model (e.g., DCN) can be used to extract features from an input (e.g., an image) and generate corresponding feature vectors. For classification tasks, features can be used to train a classifier (e.g., a support vector machine (SVM)). Because the classifier can be retrained after quantization, the model can be further enhanced and can be less sensitive to quantization noise.

[0061] Random weights

[0062] In some aspects, the model can be initialized with random weights. For example, random weights can be used for fully connected layers. Because the classifier (e.g., SVM) can be retrained (as indicated above), the model can be stored without storing weights. Thus, for the fully connected layers, the bit width can be further reduced. Therefore, the model size can be greatly reduced.

[0063] In some aspects, the bit width of intermediate calculations (e.g., activations) can be reduced online. For example, the activation bit width can be selected online based on model performance, device capabilities, latency requirements, etc. In some aspects, the bit width of intermediate calculations at different computation levels (e.g., layers) can also be determined based on priorities, for example, the priorities are determined by the cost function of Equation 1 or another cost function. For example, the cost function can be based on user feedback collected during or after applying the model. For example, user feedback can indicate that the model has made a classification error or has taken too long to calculate a tag.

[0064] In some aspects, the SQNR budget may be determined based on an empirical approach. For a given data set, noise may be injected layer by layer. Injecting noise may, for example, include injecting random noise or performing quantization according to a specific bit width.

[0065] Model performance (e.g., classification accuracy and / or speed) can then be evaluated to determine whether performance remains acceptable relative to a threshold. Due to the structure of the data (as opposed to the structure of the model or network), certain layers may have very sparse activations and therefore may have larger noise budgets (be less sensitive to noise).

[0066] Additionally, in some aspects, the bit width can be dynamically adjusted based on user feedback. For example, a user can provide an indication of the performance of the classifier (e.g., indicating a classification error). The number of instances indicated by the user can be compared to a threshold and used to adjust the bit width (e.g., triggering a reload of the model with a finer quantization).

[0067] Figure 1 An example implementation of the aforementioned bit width selection for a fixed-point machine learning model using a system on a chip (SOC) 100 is illustrated, according to certain aspects of the present disclosure, and the SOC 100 may include a general purpose processor (CPU) or a multi-core general purpose processor (CPU) 102. Variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency bin information, and task information may be stored in a memory block associated with a neural processing unit (NPU) 108, a memory block associated with the CPU 102, a memory block associated with a graphics processing unit (GPU) 104, a memory block associated with a digital signal processor (DSP) 106, a dedicated memory block 118, or may be distributed across multiple blocks. Instructions executed at the general purpose processor 102 may be loaded from a program memory associated with the CPU 102 or may be loaded from a dedicated memory block 118.

[0068] The SOC 100 may also include additional processing blocks tailored for specific functions, such as a GPU 104, a DSP 106, a connectivity block 110 (which may include fourth generation long term evolution (4G LTE) connectivity, unlicensed Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and a multimedia processor 112 that may, for example, detect and recognize gestures. In one implementation, the NPU is implemented in the CPU, DSP, and / or GPU. The SOC 100 may also include a sensor 114, an image signal processor (ISP), and / or a navigation 120 (which may include a global positioning system).

[0069] The SOC 100 may be based on the ARM instruction set. In one aspect of the present disclosure, the instructions loaded into the general purpose processor 102 may include code for evaluating the sensitivity of the model accuracy to the bit width at each calculation stage of the model. The instructions loaded into the general purpose processor 102 may also include code for selecting the bit width of one or more of the parameters and intermediate calculations in each calculation stage of the model based on the evaluation of the model accuracy sensitivity.

[0070] Figure 2 An example implementation of a system 200 according to certain aspects of the present disclosure is illustrated. Figure 2 As illustrated in , the system 200 may have a plurality of local processing units 202 that may perform various operations of the methods described herein. Each local processing unit 202 may include a local state memory 204 and a local parameter memory 206 that may store parameters of a neural network. In addition, the local processing unit 202 may have a local (neuron) model program (LMP) memory 208 for storing a local model program, a local learning program (LLP) memory 210 for storing a local learning program, and a local connection memory 212. In addition, as Figure 2 As illustrated in , each local processing unit 202 may interface with a configuration processor unit 214 for providing configuration for each local memory of the local processing unit, and with a routing connection processor unit 216 for providing routing between each local processing unit 202 .

[0071] Deep learning architectures can perform object recognition tasks by learning to represent the input at successively higher levels of abstraction in each layer, thereby building useful feature representations of the input data. In this way, deep learning addresses a major bottleneck of traditional machine learning. Before the advent of deep learning, machine learning approaches to object recognition problems might rely heavily on human-engineered features, perhaps combined with shallow classifiers. A shallow classifier could be a two-class linear classifier, for example, where a weighted sum of the components of a feature vector is compared to a threshold to predict which class the input belongs to. A human-engineered feature could be a template or kernel that is customized for a specific problem domain by an engineer with domain expertise. In contrast, a deep learning architecture can learn to represent features similar to what a human engineer might design, but it does so through training. In addition, deep networks can learn to represent and recognize new types of features that humans might not have considered.

[0072] A deep learning architecture can learn a hierarchy of features. For example, if a first layer is presented with visual data, the first layer can learn to recognize relatively simple features (such as edges) in the input stream. In another example, if a first layer is presented with auditory data, the first layer can learn to recognize spectral power in specific frequencies. A second layer that takes the output of the first layer as input can learn to recognize combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0073] Deep learning architectures can perform particularly well when applied to problems that have a natural hierarchical structure. For example, the classification of motor vehicles can benefit from first learning to recognize wheels, windshields, and other features. These features can be combined in different ways at higher levels to recognize cars, trucks, and airplanes.

[0074] Neural networks can be designed to have various connectivity patterns. In a feedforward network, information is passed from lower layers to higher layers, with each neuron in a given layer communicating to neurons in a higher layer. As described above, hierarchical representations can be constructed in successive layers of a feedforward network. Neural networks can also have reflow or feedback (also known as top-down) connections. In a reflow connection, the output from a neuron in a given layer can be communicated to another neuron in the same layer. The reflow architecture can help identify patterns that span more than one block of input data delivered sequentially to the neural network. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. A network with many feedback connections may be helpful when the recognition of high-level concepts can assist in discerning specific low-level features of the input.

[0075] Reference Figure 3A, the connections between the layers of the neural network can be fully connected (302) or locally connected (304). In the fully connected network 302, a neuron in the first layer can communicate its output to each neuron in the second layer, so that each neuron in the second layer will receive input from each neuron in the first layer. Alternatively, in the locally connected network 304, the neurons in the first layer can be connected to a limited number of neurons in the second layer. The convolutional network 306 can be locally connected and further configured so that the connection strengths associated with the input to each neuron in the second layer are shared (e.g., 308). More generally, the locally connected layers of the network can be configured so that each neuron in a layer will have the same or similar connectivity pattern, but its connection strengths may have different values ​​(e.g., 310, 312, 314, and 316). The connectivity patterns of local connections may produce spatially distinct receptive fields in higher layers because higher layer neurons in a given area may receive inputs that are tuned through training to be a restricted portion of the total input to the network.

[0076] Locally connected neural networks may be well suited for problems where the spatial location of the input is meaningful. For example, a network 300 designed to recognize visual features from an onboard camera may develop high-level neurons with different properties depending on whether they are associated with the lower portion of the image or the upper portion of the image. For example, neurons associated with the lower portion of the image may learn to recognize lane markings, while neurons associated with the upper portion of the image may learn to recognize traffic lights, traffic signs, etc.

[0077] The DCN can be trained using supervised learning. During training, an image (such as a cropped image 326 of a speed limit sign) can be presented to the DCN, and a "forward pass" can then be calculated to produce an output 322. The output 322 can be a vector of values ​​corresponding to features (such as "sign", "60", and "100"). The network designer may want the DCN to output high scores for some of the neurons in the output feature vector, such as those neurons corresponding to "sign" and "60" shown in the output 322 of the trained network 300. Prior to training, the output produced by the DCN is likely to be incorrect, and the error between the actual output and the target output can be calculated. The weights of the DCN can then be adjusted to align the DCN's output scores more closely with the target.

[0078] To adjust the weights, the learning algorithm may calculate a gradient vector for the weights. The gradient may indicate the amount by which the error would increase or decrease if the weights were adjusted slightly. At the top layer, the gradient may correspond directly to the value of the weights connecting the activated neurons in the penultimate layer to the neurons in the output layer. In lower layers, the gradient may depend on the value of the weights and the calculated error gradient for the higher layers. The weights may then be adjusted to reduce the error. This way of adjusting weights may be referred to as "backward propagation" because it involves a "backward pass" through a neural network.

[0079] In practice, the error gradient of the weights may be computed on a small number of examples so that the computed gradient approximates the true error gradient. This approximation may be called stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate of the entire system has stopped decreasing or until the error rate has reached a target level.

[0080] After learning, the DCN may be presented with a new image 326 and a forward pass through the network may produce an output 322, which may be considered an inference or prediction of the DCN.

[0081] Deep belief network (DBN) is a probabilistic model including multiple layers of hidden nodes. DBN can be used to extract the hierarchical representation of training data set. DBN can be obtained by stacking multiple layers of restricted Boltzmann machine (RBM). RBM is a kind of artificial neural network that can learn probability distribution on input set. Since RBM can learn probability distribution when there is no information about which class each input should be classified into, RBM is often used for unsupervised learning. Using mixed unsupervised and supervised paradigm, the bottom RBM of DBN can be trained in an unsupervised manner and can be used as a feature extractor, while the top RBM can be trained in a supervised manner (on the joint distribution of input and target class from previous layer) and can be used as a classifier.

[0082] A deep convolutional network (DCN) is a network of convolutional networks configured with additional pooling and normalization layers. DCN has achieved state-of-the-art performance on many tasks. DCN can be trained using supervised learning, where both the input and output targets are known for many examples and are used to modify the weights of the network by using gradient descent.

[0083] The DCN can be a feed-forward network. In addition, as described above, the connections from the neurons in the first layer of the DCN to the neuron groups in the next higher layer are shared across the neurons in the first layer. The feed-forward and shared connections of the DCN can be utilized for fast processing. The computational burden of the DCN can be much smaller than, for example, a neural network of similar size that includes recurrent or feedback connections.

[0084] The processing of each layer of the convolutional network can be considered as a spatially invariant template or basis projection. If the input is first decomposed into multiple channels, such as the red, green and blue channels of a color image, the convolutional network trained on the input can be considered to be three-dimensional, with two spatial dimensions along the axis of the image and a third dimension that captures color information. The output of the convolutional connection can be considered to form a feature map in subsequent layers 318 and 320, each element in the feature map (e.g., 320) receives input from a certain range of neurons in the previous layer (e.g., 318) and from each of the multiple channels. The values ​​in the feature map can be further processed with nonlinearity (such as correction) max (0, x). The values ​​from adjacent neurons can be further pooled (which corresponds to downsampling) and can provide additional local invariance and dimensionality reduction. Normalization can also be applied by lateral inhibition between neurons in the feature map, which corresponds to whitening.

[0085] The performance of deep learning architectures can improve as more labeled data points become available or as computing power increases. Modern deep neural networks are routinely trained with thousands of times more computing resources than were available to a typical researcher just fifteen years ago. New architectures and training paradigms can further boost the performance of deep learning. Rectified linear units can reduce the training problem known as vanishing gradients. New training techniques can reduce over-fitting and thus enable larger models to achieve better generalization. Encapsulation techniques can abstract the data within a given receptive field and further improve overall performance.

[0086] Figure 3B is a block diagram illustrating an exemplary deep convolutional network 350. The deep convolutional network 350 may include multiple layers of different types based on connectivity and weight sharing. Figure 3B As shown, the exemplary deep convolutional network 350 includes multiple convolutional blocks (e.g., C1 and C2). Each convolutional block may be configured with a convolutional layer, a normalization layer (LNorm), and a pooling layer. The convolutional layer may include one or more convolutional filters, which may be applied to the input data to generate a feature map. Although only two convolutional blocks are shown, the present disclosure is not limited to this, but, depending on design preferences, any number of convolutional blocks may be included in the deep convolutional network 350. The normalization layer can be used to normalize the output of the convolutional filter. For example, the normalization layer can provide whitening or lateral suppression. The pooling layer can provide downsampling aggregation in space to achieve local invariance and dimensionality reduction.

[0087] For example, the parallel filter banks of the deep convolutional network may be optionally loaded onto the CPU 102 or GPU 104 of the SOC 100 based on the ARM instruction set to achieve high performance and low power consumption. In alternative embodiments, the parallel filter banks may be loaded onto the DSP 106 or ISP 116 of the SOC 100. In addition, the DCN may access other processing blocks that may exist on the SOC, such as processing blocks dedicated to sensors 114 and navigation 120.

[0088] The deep convolutional network 350 may also include one or more fully connected layers (e.g., FC1 and FC2). The deep convolutional network 350 may further include a logistic regression (LR) layer. Between each layer of the deep convolutional network 350 are weights (not shown) to be updated. The output of each layer can be used as an input to subsequent layers in the deep convolutional network 350 to learn a hierarchical feature representation from the input data (e.g., image, audio, video, sensor data, and / or other input data) provided at the first convolutional block C1.

[0089] In one configuration, the machine learning model is configured to evaluate the sensitivity of the model accuracy to the bit width at each computational stage of the model, and to select the bit width of one or more of the parameters and intermediate calculations in each computational stage of the model based on the evaluation. The model includes an evaluation device and / or a selection device. In one aspect, the evaluation device and / or the selection device can be a general-purpose processor 102 configured to perform the described functions, a program memory associated with the general-purpose processor 102, a memory block 118, a local processing unit 202, and / or a routing connection processing unit 216. In another configuration, the aforementioned means can be any module or any device configured to perform the functions described by the aforementioned means.

[0090] According to certain aspects of the present disclosure, each local processing unit 202 may be configured to determine parameters of the model based on one or more desired functional characteristics of the model, and to develop the one or more functional characteristics toward the desired functional characteristics as the determined parameters are further adapted, tuned, and updated.

[0091] Figure 4 A method 400 for selecting a bit width for a fixed-point machine learning model is illustrated. At block 402, the process evaluates the sensitivity of the model accuracy at each computational stage of the model to the bit width. The model accuracy may include a measure of the signal-to-quantization noise ratio (SQNR) at the output of the model or classification accuracy. In some aspects, the output may use a floating point format.

[0092] At block 404, the process selects the bit width of parameters and intermediate calculations in each computation stage of the model based on the evaluation. In some aspects, the bit width of intermediate calculations of a computation stage may be smaller than the bit width of parameters in the computation stage, or vice versa.

[0093] In some configurations, for example, the machine learning model may be a neural network, such as a DCN. In this configuration, the computational levels may include layers of the neural network. Furthermore, the parameters may include bias values ​​and / or weights. Furthermore, the intermediate computations may include activation values.

[0094] Thus, in some aspects, the bit width can be selected based on the connectivity of the neural network. For example, the connectivity can include whether the layers of the network are in a fully connected configuration, a convolutional configuration, or a configuration with a particular sparsity. In some aspects, the bit width of the fully connected layer can be less than the bit width of the convolutional layer of the neural network. Additionally, the weights and / or bias values ​​of the fully connected layers and the convolutional layers can be random in the transfer learning arrangement.

[0095] In addition, the selection of the bit width can be based on whether the bit width is for bias values, weights, or activation values. For example, in some aspects, the bit width of the bias values, weights, and / or activation values ​​can be based on the number of weights per layer in the neural network, the number of activation values ​​per layer, the filter size per layer, the filter stride per layer, and / or the number of filters per layer.

[0096] The neural network can also be fine-tuned after selecting the bit width for the bias values, activation values, and / or weights of each layer.

[0097] In some aspects, the process may also inject noise into one or more computational levels of the model. Model accuracy may be determined for each computational level of injected noise. The process may further select an injected noise level that provides a desired level of model accuracy.

[0098] In some aspects, the process may further dynamically update the bit width based on performance specifications or system resources or user input.

[0099] Figure 5 5 is a block diagram illustrating a method for selecting a bit width of a fixed-point machine learning model in accordance with aspects of the present disclosure. At block 502, the process injects noise into a computational stage of the machine learning model.

[0100] At block 504, the process determines model performance. In some aspects, model performance may include classification accuracy, classification speed, SQNR, other model performance metrics, or combinations thereof. At block 506, model performance may be evaluated by comparing performance to a threshold. The threshold may include a minimum acceptable performance level. If the performance is above the threshold, the process may inject more noise at block 502 and re-evaluate model performance. On the other hand, if the model performance is below the threshold, then at block 508, the bit width may be selected based on the final acceptable noise level.

[0101] In some aspects, when the model performance is below a threshold, the noise level can be reduced and the model performance can be re-evaluated. Furthermore, the increments of increasing or decreasing the noise level can be adjusted for more refined selections of bit widths.

[0102] The various operations of the methods described above may be performed by any suitable device capable of performing the corresponding functions. These devices may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs), or processors. In general, where there are operations illustrated in the accompanying drawings, those operations may have corresponding paired device-plus-function components with similar numbers.

[0103] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, database, or other data structure), ascertaining, and the like. Additionally, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Furthermore, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0104] As used herein, a phrase referring to "at least one" of a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc.

[0105] The various illustrative logical blocks, modules, and circuits described in conjunction with the present disclosure may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0106] The steps of the method or algorithm described in conjunction with the present disclosure may be implemented directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in any form of storage medium known in the art. Some examples of usable storage media include random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, CD-ROMs, and the like. The software module may include a single instruction, or many instructions, and may be distributed over several different code segments, distributed between different programs, and distributed across multiple storage media. A storage medium may be coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor.

[0107] The methods disclosed herein include one or more steps or actions for achieving the described methods. These method steps and / or actions may be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be changed without departing from the scope of the claims.

[0108] The functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in hardware, an example hardware configuration may include a processing system in a device. The processing system may be implemented with a bus architecture. Depending on the specific application and overall design constraints of the processing system, the bus may include any number of interconnecting buses and bridges. The bus may link together various circuits including a processor, a machine-readable medium, and a bus interface. The bus interface may be used to connect a network adapter, etc., to the processing system via the bus, in particular. The network adapter may be used to implement signal processing functions. For some aspects, a user interface (e.g., a keypad, a display, a mouse, a joystick, etc.) may also be connected to the bus. The bus may also link various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, and similar circuits, which are well known in the art and will not be described further.

[0109] The processor may be responsible for managing the bus and general processing, including executing software stored on a machine-readable medium. The processor may be implemented with one or more general and / or special processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuit systems that can execute software. Software should be broadly interpreted as meaning instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or other. As an example, a machine-readable medium may include a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a disk, an optical disk, a hard drive, or any other suitable storage medium, or any combination thereof. The machine-readable medium may be implemented in a computer program product. The computer program product may include packaging material.

[0110] In hardware implementation, the machine-readable medium can be a part separated from the processor in the processing system. However, as those skilled in the art will readily appreciate, the machine-readable medium or any part thereof can be outside the processing system. As an example, the machine-readable medium can include a transmission line, a carrier wave modulated by data, and / or a computer product separated from the device, all of which can be accessed by the processor through a bus interface. Alternatively or in addition, the machine-readable medium or any part thereof can be integrated into the processor, such as a cache and / or a general register file may be this case. Although the various components discussed can be described as having specific locations, such as local components, they can also be configured in various ways, such as some components are configured as a part of a distributed computing system.

[0111] The processing system can be configured as a general processing system having one or more microprocessors providing processor functionality and external memory providing at least a portion of the machine-readable medium, all linked together with other supporting circuitry via an external bus architecture. Alternatively, the processing system can include one or more neuromorphic processors for implementing the neuron model and neural system model described herein. As another alternative, the processing system can be implemented with an application specific integrated circuit (ASIC) with a processor, bus interface, user interface, supporting circuitry, and at least a portion of the machine-readable medium integrated in a single chip, or with one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic, discrete hardware components, or any other suitable circuitry, or any combination of circuits capable of performing the various functionalities described throughout this disclosure. Depending on the specific application and the overall design constraints imposed on the overall system, those skilled in the art will recognize how to best implement the functionality described with respect to the processing system.

[0112] The machine-readable medium may include several software modules. These software modules include instructions that cause the processing system to perform various functions when executed by the processor. These software modules may include a transmission module and a receiving module. Each software module may reside in a single storage device or be distributed across multiple storage devices. As an example, when a triggering event occurs, the software module can be loaded into the RAM from the hard drive. During the execution of the software module, the processor can load some instructions into the cache to increase the access speed. Subsequently, one or more cache lines can be loaded into the general register file for execution by the processor. When the functionality of the software module is described below, it will be understood that such functionality is implemented by the processor when the processor executes the instructions from the software module. In addition, it should be appreciated that various aspects of the present disclosure produce improvements in the functionality of processors, computers, machines, or other systems that implement such aspects.

[0113] If implemented in software, each function can be stored on or transmitted by one or more instructions or codes on a computer-readable medium. Computer-readable media include both computer storage media and communication media, and these media include any media that facilitate the transfer of a computer program from one place to another. Storage media can be any available media that can be accessed by a computer. As an example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other media that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer. In addition, any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, a server, or other remote sources using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL), or a wireless technology (such as infrared (IR), radio, and microwaves), the coaxial cable, the fiber optic cable, the twisted pair, the DSL, or the wireless technology (such as infrared, radio, and microwaves) are included in the definition of the medium. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Disks, where disks often reproduce data magnetically, and discs reproduce data optically with lasers. Thus, in some aspects, computer-readable media may include non-transitory computer-readable media (e.g., tangible media). Additionally, for other aspects, computer-readable media may include transient computer-readable media (e.g., signals). Combinations of the above should also be included within the scope of computer-readable media.

[0114] Thus, some aspects may include a computer program product for performing the operations presented herein. For example, such a computer program product may include a computer-readable medium having stored (and / or encoded) thereon instructions that can be executed by one or more processors to perform the operations described herein. For some aspects, the computer program product may include packaging materials.

[0115] In addition, it should be appreciated that the modules and / or other appropriate means for performing the methods and techniques described herein can be downloaded and / or otherwise obtained by the user terminal and / or base station where applicable. For example, such a device can be coupled to a server to facilitate the transfer of the means for performing the methods described herein. Alternatively, the various methods described herein can be provided via a storage device (e.g., RAM, ROM, a physical storage medium such as a compact disc (CD) or a floppy disk, etc.) so that once the storage device is coupled to or provided to the user terminal and / or base station, the device can obtain the various methods. In addition, any other suitable technology suitable for providing the methods and techniques described herein to the device can be utilized.

[0116] It will be understood that the claims are not limited to the precise configuration and components illustrated above. Various changes, substitutions and variations may be made in the arrangement, operation and details of the methods and apparatus described above without departing from the scope of the claims.

Claims

1. A computer-implemented method for selecting a bit width for a machine learning model for input classification, the machine learning model comprising a neural network, wherein the input comprises images, audio, video, sensor data, and other input data, the method comprising: Receiving a first input, the first input comprising a first set of activation values ​​for a first layer of the neural network, the first layer comprising a first set of corresponding weights; processing to perform at least one of: quantizing the first set of activation values ​​to a first layer activation bit width and quantizing the first set of weights to a first layer weight bit width to introduce quantization noise into at least one of the activation values ​​or the weights; determining a model performance metric for the machine learning model, including at least a signal-to-quantization-noise ratio (SQNR), wherein the determining comprises multiplying a first set of processed activation values ​​with a first set of processed weights; as well as At least one of the first layer activation bit width and the first layer weight bit width is changed based on the determined model performance metric.

2. The method of claim 1, wherein the first layer is a convolutional layer.

3. The method of claim 2, wherein the step of receiving the first input comprises first decomposing the first input into a plurality of channels. The method of claim 1 , wherein the first layer is a fully connected layer.

5. The method of any one of claims 1-4, wherein the processing comprises quantizing both the first set of activation values ​​and the first set of weights.

6. The method of claim 5, wherein the first layer activation bit width is different from the first layer weight bit width.

7. The method of claim 1, further comprising: receiving a second input comprising a second set of activation values ​​for a second layer of the neural network, the second layer comprising a second set of corresponding weights different from the first set of weights; Processing to perform at least one of the following: quantizing the second set of activation values ​​to a second layer activation bit width, and quantizing the second set of weights to a second layer weight bit width; and Determine a second model performance metric for the machine learning model, wherein the determining comprises multiplying a second set of processed activation values ​​with a second set of processed weights.

8. The method of claim 7, further comprising changing at least one of the second layer activation bit width and the second layer weight bit width based on the determined second model performance metric.

9. The method of claim 7, further comprising maintaining the second layer activation bit width the same and maintaining the second layer weight bit width the same based on the determined second model performance metric.

10. The method of claim 7, wherein: multiplying the first set of activation values ​​by the first set of weights to generate a result, the result being used to generate a first set of intermediate calculations; The second set of activation values ​​includes the first set of intermediate calculations.

11. The method of claim 1, wherein: The first layer further comprises a first set of corresponding bias values; The processing includes quantizing the first set of bias values ​​to a first layer bias bit width.

12. The method of claim 1, wherein the first layer comprises different components, and the method further comprises selecting different bit widths for different components of the first layer.

13. The method of claim 1, wherein the model performance metric further comprises at least one of classification accuracy and classification speed.

14. An apparatus for selecting a bit width for a fixed-point implementation of a machine learning model for input classification, wherein the input includes images, audio, video, sensor data, and other input data, comprising: Memory; as well as At least one processor is coupled to the memory and is configured to perform the method of any one of claims 1-13.

15. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method of any one of claims 1-13.

Images