Dynamic Adaptation of Deep Neural Networks

Abstract

To provide techniques for training a deep neural network (DNN) for reduced computational resource requirements.SOLUTION: A computing system includes memory for storing a set of weights of a DNN. The DNN comprises a plurality of layers. For each of the plurality of layers, the set of weights includes weights of the layer and a set of bit precision values includes a bit precision value of the layer. The weights of a layer are represented in the memory using values having bit precisions that equal the bit precision value of the layer. The weights of a layer are associated with inputs to neurons of the layer. Additionally, the computing system includes processing circuitry for executing a machine learning system configured to train the DNN. Training the DNN comprises optimizing the set of weights and the set of bit precision values.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 644715, filed March 19, 2018, U.S. Provisional Patent Application No. 62 / 645358, filed March 20, 2018, and U.S. Patent Application No. 16 / 133446, filed September 17, 2018. The entire contents of U.S. Provisional Patent Application No. 62 / 645358 and U.S. Patent Application No. 16 / 133446 are incorporated herein by reference.

[0002] Government Rights This invention was made with Government support under Contract No. N00014-17-C-1011 awarded by the Office of Naval Research and support under Grant No. 1526399 awarded by the National Science Foundation. The Government has certain rights in this invention.

[0003] The present disclosure relates generally to machine learning systems. [Background technology]

[0004] Machine learning algorithms have made rapid progress in recent years through the use of deep neural networks (DNNs). DNNs are artificial neural networks with multiple hidden layers between the input and output layers. Exemplary types of DNNs include recurrent neural networks (RNNs) and convolutional neural networks (CNNs). DNNs have wide applications in fields such as artificial intelligence, computer vision, automatic speech recognition, and language translation. Training time, memory requirements, processor availability, battery power consumption, and energy efficiency are challenges associated with DNNs. Summary of the Invention [Problem to be solved by the invention]

[0005] Generally, the disclosure describes techniques related to machine learning algorithms. For example, the disclosure discloses the use of low-precision methods (i.e., methods using low-precision weights) for training deep neural networks (DNNs). For example, the disclosure describes training methods that determine a set of precision-optimized weights for a DNN and a set of fixed-precision weights for the same DNN. The methods for training DNNs are considered low-precision because the precision-optimized weights may be different from or lower than the fixed-precision weights. Computational resources, such as memory storage locations, power, processing cycles, and other computer resources, are important considerations regarding the performance of DNNs and their suitability for use with certain types of devices that have relatively low amounts of available computational resources. Because the precision-optimized weights may be lower than the fixed-precision weights, the low-precision methods of the disclosure enable low memory and computational requirements for DNN processing. In some instances, low precision methods may have adverse effects on microprocessor design in the sense that computational resources may be used by low precision methods and in the sense that microprocessors may be designed to efficiently use precision-optimized weights.

[0006] Furthermore, low-precision training methods enable artificial intelligence (AI) systems to continue learning after an initial training phase. Additionally, aspects of the present disclosure relate to lifelong learning methods for DNNs and to hardware architectures that can efficiently support learning and inference simultaneously. The techniques of the present disclosure enable AI systems to select and configure their own learning requirements (e.g., configure the bit precision of their weights) to meet real-time computation constraints at the edge. In some examples, the techniques of the present disclosure do not assume a probability distribution for optimal parameters. [Means for solving the problem]

[0007] In one example, the disclosure describes a computer system for training a deep neural network (DNN) to reduce computational resource requirements, the computer system comprising: a memory that stores a set of weights for the DNN, the DNN having a plurality of layers, and for each of the plurality of layers, the set of weights comprises a layer weight, and the set of bit-precision values comprises a layer bit-precision value, the layer weights being represented in the memory using values having a bit-precision equal to the layer bit-precision value, the layer weights being associated with inputs to neurons of the layer; and a processing circuitry that executes a machine learning system configured to train the DNN, wherein training the DNN comprises optimizing the set of weights and the set of bit-precision values.

[0008] In another example, the present disclosure describes a method of training a deep neural network (DNN) to reduce computational resource requirements, the method comprising: storing a set of weights for the DNN, the DNN having a plurality of layers, and for each of the plurality of layers, the set of weights comprising a layer weight and the set of bit-precision values comprising a layer bit-precision value, the layer weights being represented in memory using values having a bit-precision equal to the layer bit-precision value, the layer weights being associated with inputs to neurons of the layer; and running a machine learning system configured to train the DNN, wherein training the DNN comprises optimizing the set of weights and the set of bit-precision values.

[0009] In another example, the disclosure describes a computer-readable data storage medium having stored thereon instructions that, when executed, cause one or more processors to: store a set of weights for a deep neural network (DNN), the DNN having a plurality of layers, and for each of the plurality of layers, the set of weights comprises a layer weight, and the set of bit-precision values comprises a layer bit-precision value, the layer weights being represented in memory using values having a bit-precision equal to the layer bit-precision value, the layer weights being associated with inputs to neurons of the layer; and execute a machine learning system configured to train the DNN, wherein training the DNN comprises optimizing the set of weights and the set of bit-precision values.

[0010] The details of one or more examples of the techniques of this disclosure are set forth in the accompanying drawings and the description that follows. Other features, objects, and advantages of the techniques will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram illustrating an example of a computer system according to the disclosed technology. [Figure 2] 1 is a flowchart illustrating an example of the operation of a BiNet deep neural network (DNN) according to the techniques of the present disclosure. [Figure 3] 1 is a flowchart illustrating an example of an operation for training a DNN according to the techniques of the present disclosure. [Figure 4] 10 is a flowchart illustrating an example of an operation for determining a loss function according to the technique of the present disclosure. [Figure 5] FIG. 1 is a block diagram illustrating an example of a heterogeneous neural architecture. [Figure 6A] An example of the performance of BiNet and LeNet-FP32 is shown below. [Figure 6B] An example of the performance of BiNet and LeNet-FP32 is shown below. [Figure 7]The number of bits used to represent the parameters of each layer of the CNN is shown. [Figure 8A] We demonstrate the performance of the MNIST database and the impact of compression. [Figure 8B] We show the performance and compression impact of the CIFAR-10 database. [Figure 9] We show the validation error rate for BiNet iterations compared to LeNet-FP32 across mini-batches. [Figure 10] We show an example of the performance of BiNet for various coefficients of linear bit penalty on the MNIST and CIFAR-10 databases. [Figure 11] It is a layered neural network software architecture. [Figure 12] 1 is an example of a system architecture with a heterogeneous set of processors and an example of a mapping of a neural network software architecture to the system architecture resources. [Figure 13] 1 is a flowchart illustrating an example of the operation of a machine learning system according to the technique of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] Like reference characters refer to like elements throughout the drawings and description.

[0013] 1 is a block diagram illustrating an example of a computer system 100. As shown, the computer system 100 includes processing circuitry for executing a machine learning system 104 having a deep neural network (DNN) 106 including multiple layers 108A-108N (collectively "layers 108"). The DNN 106 may include various types of deep neural networks (DNNs), such as recurrent neural networks (RNNs) and convolutional neural networks (CNNs).

[0014] In some examples, the processing circuitry of computer system 100 includes one or more of a microprocessor, a controller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), equivalent discrete logic circuitry, equivalent integrated logic circuitry, or other types of processing circuitry. In other examples, computer system 100 comprises any suitable computer system, such as a desktop computer, a laptop computer, a game console, a multifunction television, a handheld device, a tablet, a mobile phone, a smartphone, etc. In some examples, at least a portion of system 100 is distributed across a network, such as a cloud computing system, a data center, or the Internet or a public or private communications network, e.g., broadband, cellular, Wi-Fi, and / or other types of communications networks that transmit data between computer systems, servers, and computing devices.

[0015] In some examples, computer system 100 is implemented in circuitry, for example, with one or more processors and memory 102. Memory 102 may include one or more storage devices. One or more components of computer system 100 (e.g., processor, memory 102, etc.) may be interconnected to enable communication between the components (physical, communication, and / or operational). In some examples, such connections may be provided by a system bus, a network connection, an inter-process communication data structure, a local area network, a wide area network, or other data communication methods. One or more processors of computer system 100 may perform functionality and / or execute instructions related to computer system 100. Examples of processors include a microprocessor, an application processor, a display controller, an auxiliary processor, one or more sensor hubs, and any other hardware configured to function as a processor, processing unit, or processing device. Computer system 100 may use one or more processors to perform operations according to one or more aspects of the present disclosure using software, hardware, firmware, or a combination thereof residing on and / or executing on computer system 100. One or more storage devices of memory 102 may be distributed among multiple devices.

[0016] The memory 102 may store information for processing during operation of the computer system 100. In some examples, the memory 102 comprises temporary memory, meaning that the primary purpose of one or more storage devices in the memory 102 is not long-term storage. The memory 102 may also be configured as volatile memory, providing short-term storage of information and not retaining the stored content when operation is stopped. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art. In some examples, the memory 102 includes one or more computer-readable storage media. The memory 102 may be configured to store a greater amount of information than volatile memory. The memory 102 may also be configured as non-volatile memory space, providing long-term storage of information and retaining information after on-off cycles. Examples of non-volatile memory include magnetic hard disks, optical disks, flash memory, or forms of electrically programmable memory (EPROM) or electrically erasable programmable memory (EEPROM). The memory 102 may store program instructions and / or data associated with one or more of the modules described in accordance with one or more aspects of the present disclosure.

[0017] The one or more processors and memory 102 may provide an operating environment or platform for one or more modules or units, which may be implemented as software, but in some examples may include any combination of hardware, firmware, and software. The one or more processors may execute instructions, and the one or more storage devices may store instructions and / or data for one or more modules. The combination of processor and memory 102 may retrieve, store, or execute instructions and / or data for one or more applications, modules, or software. The processor and / or memory 102 may be operatively coupled to one or more other software and / or hardware components, including, but not limited to, one or more of the components shown in FIG. 1 or other figures of this disclosure.

[0018] In the example of FIG. 1 , DNN 106 receives input data from input dataset 110 and generates output data 112. Input dataset 110 and output data 112 may include various types of information. For example, input dataset 110 may include image data, video data, audio data, source text data, numerical data, speech data, etc. Output data 112 may include classification data, translated text data, image classification data, robot control data, transcription data, etc. Additionally, as described elsewhere in this disclosure, output data 112 may include a mapping of the DNN of the neural network software architecture to a processor of the neural network software architecture and hardware architecture.

[0019] As described above, the DNN 106 includes multiple layers 108. Each layer 108 may include a respective set of artificial neurons. The layers 108 include an input layer 108A, an output layer 108N, and one or more hidden layers (e.g., layers 108B-108M). The layers 108 may include fully connected layers, convolutional layers, pooling layers, and / or other types of layers. In fully connected layers, the output of each neuron in the previous layer forms the input of each neuron in the fully connected layer. In convolutional layers, each neuron in the convolutional layer processes input from neurons associated with each region of neurons. A pooling layer combines the outputs of neuron clusters in one layer into a single neuron in the next layer.

[0020] Each input of each artificial neuron in layer 108 is associated with a corresponding weight in high precision weights 114 and low precision weights 116 in accordance with the techniques of this disclosure. The output of the kth artificial neuron in DNN 106 is defined as:

[0021]

number

[0022] In formula (1), y k is the output of the kth artificial neuron, φ(·) is the activation function, and W k is a vector of weights (e.g., high-precision weights 114 or low-precision weights 116) of the k-th artificial neuron, and X k is a vector of input values for the kth artificial neuron. In some examples, one or more inputs to the kth artificial neuron are bias terms that are not based on the output values of other artificial neurons or on source data. Various activation functions are known in the art, such as rectified linear units (ReLU), TanH, Sigmoid, etc.

[0023] In accordance with the techniques of this disclosure, the memory 102 stores a set of low-precision weights 116 of the DNN 106 (referred to herein as a first set of weights), a set of high-precision weights 114 (referred to herein as a second set of weights), and a set of bit-precision values 118. This disclosure will denote the high-precision weights 114 by W and the low-precision weights 116 by

[0024]

number

[0025] As will be explained in more detail below, both the high-precision weights 114 and the low-precision weights 116 may be used as weights in equation (1). The bit-precision values 118 have a bit-precision value for each layer 108 of the DNN 106. The bit-precision value for a layer represents the bit depth of the weights of the low-precision weights 116. Furthermore, as will be explained in more detail below, the training process may simultaneously determine the values of the high-precision weights 114, the values of the low-precision weights 116, and the bit-precision values 118. Use of a DNN 106 having high-precision weights 114 and low-precision weights 116 can produce comparable output data. However, because the low-precision weights 116 have fewer bits than the high-precision weights 114, fewer operations may be required to read the low-precision weights 116 from memory 102 than to read the high-precision weights 114 from memory 102. Furthermore, in some examples, because the low-precision weights 116 have fewer bits than the high-precision weights 114, the machine learning system 104 can store the low-precision weights 116 using a data type that occupies even fewer bits. For example, the machine learning system 104 may use 8-bit integers for the low-precision weights 116 and 32-bits for each of the high-precision weights 114. Thus, memory requirements can be reduced after training using only the low-precision weights 116. Furthermore, the number of read operations can be reduced, resulting in lower latency and reduced electrical consumption.

[0026] As described in more detail below, as part of performing a training process, the machine learning system 104 may perform a feedforward stage in which the machine learning system 104 uses the high-precision weights 114 of the DNN 106 to determine output data 112 based on input data from the input dataset 110. Additionally, the machine learning system 104 may perform backpropagation to calculate the gradient of a loss function. The loss function calculates a cost value based on the output data. In accordance with the techniques of this disclosure, the machine learning system 104 may update the high-precision weights 114, the low-precision weights 116, and the bit-precision values 118 based on the gradient of the loss function. The machine learning system 104 may perform the feedforward and backpropagation processes multiple times using different input data. During or after the training process is complete, the machine learning system 104 or other device may use the low-precision weights 116 in an evaluation process to generate output data based on non-training input data.

[0027] In the example of FIG. 1 , the memory 102 also stores a set of one or more hyperparameters 120. The hyperparameters 120 may include a hyperparameter that controls a learning rate. Additionally, in accordance with the techniques of this disclosure, the hyperparameters 120 may include a hyperparameter (labeled λ2 in this disclosure) that controls the severity of a bit precision penalty term in the loss function. The bit precision value 118 (i.e., the number of bits used in the low-precision bit weights 116 in each layer 108 of the DNN 106) may be based on the value of the bit precision penalty term. Thus, different values of the hyperparameters result in a loss function that penalizes different degrees of when the DNN 106 uses high-bit precision weights.

[0028] DNNs become increasingly larger and deeper in layers to learn higher levels of semantic and spatiotemporal dependencies in application data. In fact, over the past decade, DNN parameter sizes have grown exponentially, from the remarkably simple LeNet-5 convolutional neural network (CNN) with only 1 million parameters for classifying handwritten digits to emerging DNNs like AlexNetCNN, which used 60 million parameters to win the 2012 ImageNet image classification competition, and Deeperface, which uses 120 million parameters for human face recognition. Even networks with 10 billion parameters exist.

[0029] Because the memory 102 stores the parameters (e.g., high-precision weights 114, low-precision weights 116) of the DNN 106, how the DNN 106 uses the memory 102 is important. For example, the more storage locations in the memory 102, the more parameters the DNN 106 requires. Furthermore, memory access (i.e., reading and writing from memory) requires electrical energy. Therefore, the size of the storage locations in the memory 102 available for storing parameters may reflect the learning capacity of the DNN 106 (i.e., the capacity of the DNN 106 to learn), and at the same time, the size of the storage locations in the memory 102 available for storing parameters may affect the computational efficiency and processing time of the DNN 106. For example, 630 million synaptic connections consume approximately 3 teraflops / second (assuming 512x512 images at 100 gigaflops / frame). Furthermore, based on rough estimates, AlexNetCNN consumes 8W for DRAM access alone, which far exceeds the power budget for a typical mobile device. Therefore, the computation and run-time memory footprint required by these recent DNNs in inference mode can exceed the power budget and memory size of a typical device. Large DNNs can reach 10 billion parameters, and there is a trend toward large and deep networks.

[0030] Returning to the issue of power and memory size, it is well understood that power density no longer decreases according to Moore's Law and computational throughput must scale with energy efficiency (e.g., joules / operation). First, the energy of an operation consists of (1) the energy for the operation, e.g., floating-point operations, (2) the energy for moving data between storage devices and processing cores, and (3) the energy required to store the data. It is well documented that the energy for data movement (e.g., memory read and write operations) is likely to dominate the energy cost of a computation. This effect can be amplified by DNN computational characteristics, which have a low operations / byte ratio and poor locality behaviors.

[0031] Memory performance continues to lag computation, which is evident in the slow growth in package pins, bandwidth, and frequency, and more importantly, in memory architectures that have remained unchanged for decades. Latency hiding using adaptive cache and memory hierarchies is no longer beneficial because energy usage cannot be hidden. Data movement and storage energy incurs costs and therefore reduces overall efficiency and scalability if not managed.

[0032] Regularization is a technique used to solve the overfitting problem in machine learning. Generally, regularization techniques work by adding a penalty term to the loss function used to train a DNN, as shown in the following equation (2):

[0033]

number

[0034] In equation (2), L(·) is the loss function, λ is a hyperparameter, and N(w) is the norm of the weight vector w. Traditional approaches to regularization in machine learning have been shown to be ineffective for DNNs. Well-studied regularization techniques such as LASSO and L-2 control the magnitude of parameters but not their precision. Previous work known as Binary Connect associates 1-bit precision parameters with a specific form of regularization. In particular, Binary Connect is a method for training DNNs using binary weights during forward and backward propagation while preserving the precision of the stored weights over which gradients are accumulated.

[0035] Historically, bit-level precision of parameters has been studied under post-processing for DNN compression. Recent approaches have required a certain bit-level precision upfront or separated the steps into training the DNN parameters and compressing them. The techniques herein may balance both criteria, providing a flexible way to specify parameters that fit the resource constraints of the target device, while also being able to find the optimal bit precision per DNN layer, from a coarse binary representation of the parameters to a fine-grained 32-bit representation of the parameters.

[0036] One goal of these techniques is to directly address the exponential growth in memory size of large DNNs by investigating new methods for adapting low-precision weights. These techniques may balance energy-saving algorithmic performance, recognizing that these methods cannot employ the discretionary application of simple truncation, such as rounding or recalculating floating-point numbers to low-precision integers. DNNs are typically over-parameterized due to significant redundancy in the learned model. In contrast, sparsity of the learned representation provides high algorithmic performance for DNNs, but at the same time, it is easy to arrive at DNNs that are wasteful in both computation and memory. In this context, sparsity refers to the use of low-precision weights that can only take on a limited (small) number of values. Therefore, in some examples of this disclosure, we aim to find a balance where DNN sparsity (and therefore robustness and algorithmic performance) is maintained without over-parameterizing the model.

[0037] There are several advantageous results from the techniques described in this disclosure. First, the memory size of the DNN 106 can be optimized. Second, the learning rate can be increased because the training process described in this disclosure can better guide the learning goal. Second, the resulting DNN can achieve high performance and low power consumption.

[0038] Regarding the learning rate, note that one of the most difficult unsolved challenges in training DNNs is training them within a limited training time. From a learning perspective, a reasonable question is what is the optimal number of bits for each layer of the DNN and what is a gradient descent algorithm that optimizes the number of bits using training data from the input dataset 110 relative to previous work. The variable number of bits as the number of unique values is captured by a parameter, which is directly used as a regularizer for the classification operation of convolutional networks or other types of DNNs. There is no equivalent matrix norm used as a regularizer in the machine learning literature that forms a direct relationship between low-level bit precision and our formulation, which uses regularization derived from the minimum description length principle. Compared to networks with full-precision weights, our technique can demonstrate improved on-the-fly learning using a varying number of bits during training epochs.

[0039] The rapid increase in neural network capabilities has been driven primarily by the transition to better hardware. DNNs are fundamentally based on matrix multiplication (more specifically, the computation of multiple weighted additions, one per neuron) as their most common and most central operation. Because matrix multiplication is highly parallelizable, GPUs allow DNNs to scale, execute, and train significantly faster. This reduces training times for very large datasets from the years required on serial hardware to weeks or even days, which explains the dramatic performance improvements we've seen in AI recently.

[0040] At the same time, it may be desirable for computer architects to consider new ways to make DNN processing more adaptable for use with mobile platforms that have additional constraints on speed or energy consumption. While GPUs provide speedup from massive parallelism, scaling behavior is limited because they do not fully exploit the weighted addition simplifications used in DNNs. New hardware accelerators are being released, most of which provide integer-based support for DNN processing. Furthermore, advanced nanoelectronics are prone to manufacturing variance, which can increase device behavior variability and adversely affect production yields. Examples implementing one or more of the techniques of the present disclosure can simplify the complexity of the underlying logic and address manufacturability with potentially optimal memory sizes and configurations.

[0041] The number of unique values allowed for a parameter can be considered a measure of regularization. Thus, as will be explained in more detail below, instead of using N(w) as the penalty term as shown in Equation (2), the penalty term may be or be based on the number of unique values that a parameter of the DNN 106 can have. The expressive power of the DNN 106 may be gradually controlled by restricting the set of allowed values of the parameter. Thus, techniques herein may use the concept of bit-level precision to regularize the training of parameters of the DNN 106. These techniques may combine the concept of bit-level precision with the regularization of the training of parameters of the DNN 106.

[0042] The techniques herein may generalize and encompass previous work using the two stages described above (training and compressing DNN parameters). Other approaches addressing memory size, power consumption, and computational speed (including compression, quantization, and approximation of DNNs) are insufficient because they (1) assume an optimally trained DNN is given as input, (2) are agnostic to the performance of the compressed DNN on labeled ground truth data, (3) provide lower algorithmic performance for any compression target, and (4) do not exhibit rapid learning growth during training. The ability to apply the techniques herein to the fundamental ability to both train and compress learned concepts into a small form factor makes them applicable to many areas of deep learning applications and systems.

[0043] The techniques herein can be used to design and utilize hardware accelerators with multi-bit precision that can more efficiently support DNN processing. The techniques herein can take into account the specifications of a target field device (e.g., available memory capacity) when training a DNN 106 and can guide learning to specific training data for a particular target field device. The techniques herein can be a valuable tool for practitioners deploying DNN-based machine learning solutions to real-world applications, including but not limited to mobile platforms and smartphones. The techniques herein enable powerful DNNs for resource-constrained environments. The techniques herein can potentially support enhanced returns in advanced nanotechnology by finding optimal bit precision and weights. The techniques herein can support rapid online learning and other on-board systems where resources (power, size, etc.) limit current deployments (e.g., robots in the field).

[0044] This disclosure describes techniques for training DNNs such that bit precision is part of the parameters of the training procedure. Current training methods consist of model selection (e.g., learning rate, network size, network depth) and a training phase (e.g., using a backpropagation algorithm to set the network weights for multiple iterations). This disclosure describes a method that includes optimizing bit precision as part of the training phase. Accordingly, this disclosure describes a method for training a DNN 106 with bit precision itself as a parameter. The resulting DNN may have different bit precisions for each of the layers 108 of the DNN 106. Such an approach can exhibit fast learning (i.e., can quickly move to solution) and can reach a superior target DNN (e.g., higher algorithmic performance with respect to ground truth) compared to standard approaches for training DNNs.

[0045] Although the techniques can be applied generally to any gradient-based parameter learning for classification or regression, we will limit our attention in this section of this disclosure to CNNs. In this disclosure, uppercase symbols represent tensors and lowercase symbols represent vectors. X (l) represents the input of the lth layer. X (l+1) represents the output of the lth layer. W (l) represents the parameters of the l-th layer. W is the set of all CNN parameters with N layers, denoted by (1), (2),...,(N), {W (1) ,W (2) ,...,W (N) In this disclosure,

[0046]

number

[0047] denotes the dot product, φ denotes the smooth non-linearity applied pointwise to the output for filtering, y denotes the input data set 110, X(l) are the labels for the example mini-batches corresponding to

[0048] In the example of FIG. 1, the DNN 106 may include a CNN that implements feed-forward convolutions over multiple filter banks called layers (i.e., layer 108). The outputs of the convolutional layers are typically connected to a pooling layer that outputs the maximum of the activations within each pixel's neighborhood. The output of a fully connected layer l is simply the dot product of the layer's inputs and parameters, as shown in equation (3).

[0049]

number

[0050] In equation (3), φ is W (l) and X (l) Convolutional layers implement a special form of weight sharing by replicating weights across one of the dimensions of the input. In this case, the dot product in (3) is transformed into a kernel W, as shown in (4) below. (l) This is a convolution operation using

[0051]

number

[0052] In this disclosure, * denotes a convolution operation. Convolution layers are typically connected to pooling layers that output the maximum activation within each pixel's neighborhood. The neighborhoods may be referred to as "patches." For classification, the patches generated as described above may be used as features in a logistic regression layer, as shown in equation (5).

[0053]

number

[0054] In equation (5), any vector

[0055]

number

[0056] For v, softmax(v) is the input

[0057]

number

[0058] The output of the CNN is a vector of size D × 1 that assigns a score to each of the D labels.

[0059] To complete the description of the CNN, we use X (l) In some examples, the predicted labels (i.e., the labels generated as output by the DNN 106) are set as arg maxX for a CNN with N layers, denoted as (1), (2), ..., (N). (l) In other words, the machine learning system 104 may

[0060]

number

[0061] The predicted label may be calculated as the index i with the largest value of

[0062]

number

[0063] Maximum likelihood estimation is the most common method for learning filters from training data.

[0064]

number

[0065] B independent and identically distributed (IID) input label pairs (X (l) Given a batch of (x, y), the loss function described below captures the average log-likelihood. Due to the IID assumption, the likelihood is decomposed into a product of the likelihoods over the individual examples. Then, using properties of logarithms, the logarithm of the product becomes a sum of logarithmic terms.

[0066]

number

[0067] The techniques herein may use a range preserving linear transformation, which uniformly discretizes the range into fixed steps δ. Specifically, W (l) represents the floating-point representation of the high-precision weights 114 of layer l,

[0068]

number

[0069] is the quantized form of the weights of layer l using b bits. The quantization using b bits is as follows:

[0070]

number

[0071] Equations (11) and (12) can be rewritten as follows:

[0072]

number

[0073] In the above formula, min x (W (l)) is W (l) Returns the minimum weight of max x (W (l) ) is W (l) returns the maximum weight of v, round(v) returns the nearest integer,

[0074]

number

[0075] Any W (l) and for b,

[0076]

number

[0077] is W (l) is a step function over a range of values of

[0078]

number

[0079] teeth,

[0080]

number

[0081] Corresponding to each of

[0082]

number

[0083] For constant W and b and varying α≦w≦β,

[0084]

number

[0085] is a step function with discontinuities that are multiples of δ / 2. This problem can be solved using a quantized gradient descent

[0086]

number

[0087] This prevents direct optimization of the loss as a function of

[0088] In some cases, quantization is both efficient and deterministic with closed-form output. In such cases, the technique uses the same error criterion as K-means clustering to calculate the error due to quantization: the sum of squared rounding errors is:

[0089]

number

[0090] In equation (14) and elsewhere in this disclosure:

[0091]

number

[0092] represents the square of the two-dimensional norm of the vector v (i.e.,

[0093]

number

[0094] Similar to the objective function of K-means clustering, the sum of squared roundoff errors specified in (14) is a convex and differentiable measure due to quantization. For a given b, q(w,b) is a scalar

[0095]

number

[0096] A series of parabolas may be formed over q(w,b), the values of which are bounded such that 0≦q(w,b)≦δ / 2.

[0097] In some examples, the low-precision weights 116 are limited to integer powers of 2. For example, the low-precision weights 116 are limited to integer powers of 2. -n ...2 0 ...2 n where the possible values of n are integers. More specifically, the low precision weights 116 are defined as follows:

[0098]

number

[0099] In equation (15),

[0100]

number

[0101] is a set of integers. In other words, the low precision weights 116 may be 0, ±1, ±2, ±4, ..., and ±½, ±¼, ±½, ...

[0102] In examples where the low-precision weights 116 are limited to integer powers of two, logical shift operations are used instead of multiplication operations to calculate the output values of the artificial neurons of the DNN 106 when the DNN 106 operates in inference mode. This allows the DNN 106 to operate more efficiently and with lower latency in inference mode because shift operations are less complex than multiplication operations. The selection of values that are integer powers of two represents a quantization of the high-precision weights 114. It should be understood that other mappings besides integer powers of two can be selected, for example, to prefer a multiplication operation based on the mapping or an approximation of the multiplication operation when the hardware has these mappings available. For example, an analog circuit may use a ternary encoding of analog values (low, medium, high) to prefer a circuit operation (e.g., a voltage divider or current divider) that is equivalent to a multiplication operation. To better match the distribution of values between the high-precision weights 114 and the low-precision weights 116, the machine learning system 104 may select a quantization function that optimally preserves the coded distribution of the high-precision weights 114 during training of the DNN 106, even if the quantization is not differentiable. For example, the machine learning system 104 may use the following quantization function:

[0103]

number

[0104] In equation (16), w is one of the high-precision weights 114,

[0105]

number

[0106] are the corresponding low-precision weights 116. In equation (16), sign(w) is a function that returns the sign of the weight w (e.g., within a predetermined range of values with 0 in the middle) or returns zero. round is a rounding function. In some examples, round is implemented using stochastic rounding. Stochastic rounding refers to a stochastic rounding method for numbers such that 0.3 has a 70% chance of rounding to 0 and a 30% chance of rounding to 1. A value of 0.5 has a 50 / 50 chance of rounding to 0 or 1.

[0107] In some examples where the low precision weights 116 are restricted to integer powers of two, the machine learning system 104 may use the following quantization function:

[0108]

number

[0109] In equation (17), q(w;θ (l) ) may be a transformation of the parameter w (i.e., the high-precision weight 114). Furthermore, in equation (17), θ (l) are parameters for each layer l. In some cases, q(w;θ (l) ) may be defined as follows:

[0110]

number

[0111] In equation (18), θ1 and θ2 are parameters. (l) and θ2 (l) There may be various values of θ and θ for each layer, denoted by . Thus, in equation (18), the symbol θ without a subscript may represent one or more parameters, such as θ, θ, etc. Thus, in the example where the machine learning system 104 uses equation (18) for the quantization function of equation (17), the machine learning system 104 may

[0112]

number

[0113] may be determined as follows:

[0114]

number

[0115] Thus, in this example, for each weight in the set of low-precision weights 116, the machine learning system 104 may be configured such that, as part of updating the low-precision weights 116, the machine learning system 104 determines the weight in the set of low-precision weights 116 to be equal to a sign value multiplied by a power of two exponent value, where the sign value represents the sign of the corresponding weight in the set of high-precision weights 114 and the exponent value is based on the base-2 logarithm of the corresponding weight in the set of high-precision weights 114.

[0116] In equations (18) and (19), θ1 and θ2 may have various values. For example, when considering an identity transform, θ1=0 and θ2=1. When θ1=θ2=0, it is a ternary quantization based on the sign of the weight w. When θ1<0;θ2<0, it gives a family of hyperbolic functions whose focus and decentering are controlled by θ. When θ1>0, it gives a family of hyperbolic tangent functions. In one example, W is the following matrix:

[0117]

number

[0118] In this case, θ1=−1 and θ2=3.5. In this example, the machine learning system 104

[0119]

number

[0120] may be determined to be as follows:

[0121]

number

[0122] 2 where the value of 2.5 has a large quantization error -6 = 0.015625. Some experiments have shown that the quantization error is not very important for high accuracy and that the learned value of θ is important.

[0123] In some examples where each of the low-precision weights 116 is an integer power of two, the machine learning system 104 may store the low-precision weights 116 by storing only the exponent and sign instead of the full integer value or floating-point value. This can save storage space. For example, in examples where the maximum value of the exponent is 127, the machine learning system 104 can use 8-bit integers for 4X model compression compared to storing floating-point parameters.

[0124] There are a variety of motivations for parameter quantization. First, it is desirable to reduce the memory footprint of a trained CNN model so that it can be deployed in small, embedded, and portable devices. Second, these devices do not meet the high computational demands necessary for inference using a CNN model. Second, it is desirable to reduce the energy and power consumption of a trained CNN model. Second, the processors of devices embedded with advanced microelectronics may have manufacturing variations, which may require different accuracies for inference using a CNN model. The techniques herein can address these concerns during CNN training without post-hoc optimization and analysis. The techniques herein can limit the number of unique values taken on by parameters. First, the range of values taken on by CNN parameters is small. Therefore, the techniques herein can use significantly fewer bits to represent a small number of unique values within a small range.

[0125] As described elsewhere in this disclosure, previous work has been done on more complex quantization methods, including stochastic rounding, density-based quantization, and K-means clustering. Examples of this disclosure may use a simple rounding method that simultaneously accepts bits learned by backpropagation as parameters for the CNN. In some examples of this disclosure, the machine learning system 104 uses deterministic rounding. In some examples, the machine learning system 104 uses stochastic rounding with a straight-through estimator. Using a straight-through estimator means that the machine learning system 104 ignores discrete functions while training using backpropagation. Quantization reduces a range of values to a single value called a "bin." Note that a uniform permutation of "bins" is asymptotically optimal for minimizing mean squared error regardless of source distribution. Various examples of this disclosure can work with any differentiable transformation that converts bin indices to real values, such as logarithmic scale quantization. Recall that native floating point precision is also a non-linear quantization.

[0126] Previous work has been done on non-uniform binning schemes, for example, using parameter density, Fisher information, and K-means clustering. Although K-means minimizes squared error, it requires a more complex algorithm with two separate steps for clustering and backpropagation. Empirically, uniform binning schemes are superior to density-based binning because they do not necessarily guide the peaks of the initial parameter estimate toward optimal parameters. K-means clustering also minimizes squared error, but it requires alternating steps of backpropagation and generally non-differentiable (hard) cluster assignment.

[0127] The techniques of this disclosure may work with any differentiable transform, and a linear scale for quantization is a design choice, using equidistant bins between α and β. For example, logarithmic quantization can be used, as shown in "Convolutional Neural Networks using Logarithmic Data Representation," arXiv:1603.01025, by Daisuke Miyazaki et al. (available at https: / / arxiv.org / pdf / 1603.01025.pdf) for post-hoc compression algorithms. Recall that native floating-point precision, using a small number of bits to represent the exponent and extra bits to store the fraction, is also a nonlinear quantization. In "Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization, and Huffman Coding," arXiv:1510.00149 (available at https: / / arxiv.org / pdf / 1510.00149.pdf), Song Han et al. demonstrated that linear quantization outperforms density-based quantization, which focuses on the peak of the W distribution via inverse cumulative distribution function (CDF) sampling. Inverse CDF sampling is a method of sampling random numbers from a probability distribution. For example, if the probability of a coin landing on heads is 0.3, how do you sample which random event is likely to occur? You sample a random number between 0 and 1, and if the random number is less than 0.3, it's heads. Inverse CDF sampling is a technical term in probability distributions. This technique uses stochastic gradient descent to simultaneously optimize the values of W and b. This technique allows each layer to learn different quantizations, which has been shown to be beneficial.

[0128] In some cases, the goal of training is to arrive at a CNN with parameters W that have a small total number of bits. For example, the goal is to learn the number of bits in conjunction with the network parameters via backpropagation. Therefore, in some cases, layer-by-layer quantization is performed by using W for each layer l of the CNN. (l) and b (l) However, the loss function

[0129]

number

[0130] is not continuous and differentiable over the range of parameter values. Furthermore, small changes in W

[0131]

number

[0132] Since the loss is not reflected in W, the loss remains constant for small changes in W, and the stochastic gradient descent remains in plateaus. Thus, according to one or more examples, the machine learning system 104 updates the high precision weights 114 (W) such that the quantization error q(w, b) is small. Intuitively, W and

[0133]

number

[0134] When is close, the machine learning system 104 may use l(W) instead. The machine learning system 104 may minimize a loss function that is the negative log-likelihood regularized by a quantization function q, where q may be defined according to any of the examples of this disclosure, such as Equation (14) or Equation (18).

[0135] When b is not constant (i.e., when b is determined using backpropagation), the value of b is maximized (e.g., 32 bits) to minimize quantization error. Thus, according to the techniques of this disclosure, a penalty term is added to the number of bits, i.e., the number of unique values extracted by the network parameters. A uniformly spaced quantization model allows for a simple function for the number of unique values. According to the techniques of this disclosure, the machine learning system 104 calculates the number of unique values for each layer l of the CNN.

[0136]

number

[0137] and b (l) For example, the machine learning system 104 may employ layer-wise quantization to learn the loss function defined in Equation (20) as a function of l(W) defined in Equation (7) and Equation (8) above and q(w, b) defined in Equation (14) above.

[0138]

number

[0139] may also be used.

[0140]

number

[0141] In equation (20), λ1 and λ2 are hyperparameters used to adjust the tradeoff between the objectives of minimizing quantum error and minimizing bit depth. When λ1 = 0 and λ2 = 1, the CNN uses 1 bit per layer for bit penalty. When λ1 = 1 and λ2 = 0, the CNN uses the maximum allowed number of bits per layer (e.g., 32 bits) to minimize quantum error. The parameters λ1 and λ2 allow flexibility in specifying the cost of bits for their impact on quantum error and classification error.

[0142] In some examples, such as those in which the low precision weights 116 are restricted to integer powers of two, the machine learning system 104 may use a loss function defined in Equation (21) below:

[0143]

number

[0144] may also be used.

[0145]

number

[0146] In equation (21), l(W), λ, λ, and b may have the same meaning as in equation (20). However, in equation (21), D(W, θ) refers to the distillation loss. The distillation loss is calculated when the machine learning system 104 uses the high-precision weights 114 (W) and the low-precision weights 116 (W).

[0147]

number

[0148] , which indicates the difference between the outputs produced by the DNN 106 when running the DNN 106 on the same input using . As shown in connection with equation (19), the low-precision weights 116 may be calculated from the high-precision weights 114 (W) and the parameters θ. Thus, D may be parameterized by W and θ as shown in equation (21).

[0149] In some examples, the values of the hyperparameters may be selected (e.g., by the machine learning system 104 or an engineer) based on the availability of one or more resources (e.g., FPGA fabric or GPU cores), the quality of the data based on the algorithm task (e.g., blurry images may require a high-precision network), or other factors. In some examples, the values of the hyperparameters may be selected based on manufacturing variations in advanced nanotechnology. Processing latency and energy consumption can be reduced due to the close coupling of bit precision, storage, and energy consumption.

[0150] Meanwhile, consider updating b given a fixed W. We consider two choices for the update, corresponding to halving or doubling the number of bins. When W is optimal and λ2 = 0, this corresponds to finding the bin with the smallest quantum error. In almost all cases, this corresponds to using all 32 bits within W. Therefore, a penalty of λ2 ≠ 0 is added to each bin, so there is a trade-off between quantum error and number of bins. When λ1 = 0, this results in using one bit per layer for the penalty.

[0151] In an example where the machine learning system 104 calculates the loss function according to equation (20) during training, the machine learning system 104 may update W and b using the following rules expressed in equations (22) and (23), where W is updated by preserving high floating-point precision:

[0152]

number

[0153] In Equation (22) and Equation (23), μ is a hyperparameter representing the learning rate. The updated value of W in Equation (22) is projected to W using quantization as shown in Equation (14). The machine learning system 104 may use automatic differentiation provided in Bergstra et al., "Theamo: A CPU and GPU Math Compiler in Python," In Proc. 9th Python in Science Conf., pages 1-7, 2010 (hereinafter "Theano"), to calculate gradients with respect to Equation (22) and Equation (23).

[0154] As shown in equation (24), the sign function in equation (23) returns the sign operand if the operand is not close to 0, and if the operand is close to 0, the sign function returns 0. This allows the number of bits to converge when the learning rate and gradient become 0. In some examples, ε=10 -9 is.

[0155]

number

[0156] In some examples, such as examples where weights are restricted to powers of two and the machine learning system 104 calculates a loss function as shown in equation (21), for each layer l, the machine learning system 104 determines the bit depth of layer l (i.e., b (l) ) may be updated.

[0157]

number

[0158] In equations (27), (28), and (29), q(w, b) may be defined as shown in equation (18). Note that equation (18) depends on the values of θ1 and θ2. Thus, in this example, for each of the multiple layers, the machine learning system 104 may be configured to determine a set of quantized values for the layer by rounding values generated by applying a quantization function (e.g., θ1+θ2log2|w|) to weights in a second set of weights associated with the layer (i.e., high-precision weights 114). Further, the machine learning system 104 may determine a maximum value of the set of quantized values for the layer and a minimum value of the set of quantized values for the layer. In this case, the machine learning system 104 calculates the bit-precision value (b) of the layer based on the base-2 logarithm of the range defined by the maximum value of the set of quantized values and the minimum value of the set of quantized values. (l) ) can be set. Therefore, in this example, the low-precision weights of layer l can be set as b (l) The value b may be stored in memory 102 as a bit code value and an integer. (l) Instead of storing it in memory 102, memory 102 stores θ1 (l) The value of θ2 (l) The value of θ1 is stored and is calculated as shown in the above equations (25) to (29). (l) and θ2 (l) From b (l) may be calculated.

[0159] The loss function in equation (20) encourages small and / or large updates to W and discourages intermediate-sized updates. For illustration, consider two bins of {0, -1, +1} (i.e., ±1) and consider some weights equal to 0. While typical gradient descent updates weights in all directions, updates of size less than 1 / 2 are

[0160]

number

[0161] (W) + q(w,b)), which does not change the weights and therefore does not improve the classification loss. Furthermore, updates introduce quantization errors when rounding weights to zero. Similarly, updates of magnitude between 1 / 2 and 1 incur a quantization penalty. The optimal update using a loss as defined in equation (20) (or as defined by l(W) + q(w,b)) may be ±1, either way improving the likelihood estimate without introducing quantization errors.

[0162]

number

[0163] Note that {right arrow over (x)} is convex and a differentiable relaxation of the negative log-likelihood for the quantized parameters.

[0164]

number

[0165] It is clear that is an upper bound on l(W), the Lagrangian corresponding to the small quantization error limit using a small number of bits. The uniformly spaced quantization of equation (14) allows for such a simple functional form for the number of eigenvalues.

[0166] The quantization penalty combined with the classification loss can have a significant impact on the learning curve, and in some cases, as empirically shown, can speed up learning when the learning rate is constant. This phenomenon was noted in previous work on binary neural networks, where quantization acts as a regularizer by ignoring small changes in parameters and amplifying significant changes to the parameters. Furthermore, previous work has shown that the final parameters of CNNs have a bimodal distribution. Most previous work on DNN approximation and compression does not take advantage of this benefit because it assumes that DNNs decouple learning from post-quantization or provide pre-trained networks.

[0167] In some examples, once training is complete, the machine learning system 104 discards the high precision weights 114 (W) and

[0168]

number

[0169] Alternatively, only α and δ may be stored. In this case, the machine learning system 104 stores the low-precision weights of the DNN 106.

[0170]

number

[0171] The machine learning system 104 may encode all parameters of a layer as integers corresponding to bin indices, thereby significantly reducing the storage requirements of the low-precision weights 116. The form of parameters shown in the following equation (30) may be used to design a hardware accelerator (e.g., using an FPGA) for fast computation of convolutions and dot products.

[0172]

number

[0173] In equation (30), z is the bin index. Thus, for each layer, the machine learning system 104 may store in memory 102 only one value of α and one value of δ for each layer, as well as a separate integer value z for each input of the layer.

[0174] In some examples where the low-precision weights 116 are limited to integer powers of two, upon completing training, the machine learning system 104 discards the high-precision weights 114 (W) and reduces the low-precision weights 116

[0175]

number

[0176] may be stored as an index value. Therefore, the machine learning system 104 may store the low precision weight

[0177]

number

[0178] Store the value z for

[0179]

number

[0180] may be rearranged as follows:

[0181]

number

[0182] The memory required to store the final model can be significantly reduced because our approach allows the selection of bit precision as part of the training process. The machine learning system 104 may use and / or store only two floating-point numbers per layer, and encode all parameters of the CNN as integers. Conversely, inference or prediction requires only two floating-point operations in addition to operations between integers. Therefore, the DNN 106 can be easily deployed with the computing power of most field devices, FPGAs and complex programmable logic devices (CPLDs), mobile devices, etc.

[0183] This is a general method for training DNNs that can take into account the specifics of the target device (e.g., available memory space) and direct learning to specific training data and devices. Exemplary types of target devices may include mobile phones, tablet computers, laptop computers, desktop computers, server computers, Internet of Things (IoT) devices, autonomous vehicles, robots, etc. This disclosure may focus on DNNs trained according to the techniques of this disclosure, such as the BitNet DNN. The BitNet DNN may have different bit precision values for different layers. The BitNet DNN can be applied to many domains, from image recognition to natural language processing. The BitNet DNN can be beneficial to practitioners deploying DNN-based machine learning solutions for real-world applications, including, but not limited to, mobile platforms and smartphones. The techniques of this disclosure enable effective DNNs for resource-constrained environments. It would be advantageous to find concatenations and new forms of parallelism to simplify basic DNN processing to reduce memory size. As described in detail elsewhere in this disclosure, the machine learning system 104 may use the DNN 106 to map other DNNs of neural network software architectures to hardware architectures. Additionally, in some examples, the machine learning system 104 may train a BitNet DNN for operation on various hardware architectures.

[0184] FIG. 2 is a flowchart illustrating an exemplary operation of a BitNet DNN according to the techniques of this disclosure. In the example of FIG. 2, memory 102 stores (200) a set of weights for a DNN 106 and a set of precision values for the DNN 106. The DNN 106 has multiple layers 108. For each of the multiple layers 108, the set of weights includes a layer weight, and the set of precision values includes a layer bit precision value. The layer weights are represented in memory 102 using values having a bit precision equal to the layer bit precision value. For example, if the layer bit precision is equal to 5, each layer weight may be represented using a 5-bit index (e.g., as an integer having 5 bits), an offset value α, and a quantization step size value δ, as shown in equation (30) above. In this example, memory 102 may store one offset value α and one quantization step size value δ for each layer. In an example where weights are limited to integer powers of 2, the weights for each layer may be represented in memory 102 using exponent values having a bit precision equal to the layer bit precision value. The weights of a layer relate the inputs to the neurons of the layer.

[0185] 2, the machine learning system 104 may train 202 the DNN 106. Training the DNN 106 comprises optimizing a set of weights and a set of bit precision values. Accordingly, the bit precision values are updated during training of the DNN 106. As a result, two or more of the layers 108 of the DNN 106 may have different bit precisions from one another. In some examples, all of the layers 108 of the DNN 106 have different bit precisions from one another.

[0186] The machine learning system 104 may apply a backpropagation algorithm over multiple iterations as part of training the DNN 106. Each iteration of the backpropagation algorithm may update a set of weights and optimize a set of bit-precision values. Exemplary details of the backpropagation algorithm and bit-precision optimization are described in connection with FIG. 3.

[0187] 3 is a flowchart illustrating exemplary operations for training a DNN 106 according to the techniques of this disclosure. As described above, the machine learning system 104 may perform multiple iterations for training the DNN 106. As described below, the machine learning system 104 may perform operations (300) through (308) of FIG. 3 for each of the multiple iterations.

[0188] With reference to FIG. 3, the set of low precision weights 116 (FIG. 1) discussed above is weight

[0189]

number

[0190] During training, the memory 102 may store a second set of weights (i.e., high-precision weights 114 (FIG. 1)) having a constant precision set of weights for each of the multiple layers. Each weight in the set of high-precision weights 114 may have a bit precision equal to a predefined highest bit precision value (e.g., 32 bits, 16 bits, etc.).

[0191]

number

[0192] The set of bit precision values (b) (i.e., low precision weights 116) comprises a precision-optimized set of weights for each of the multiple layers. In some examples, such as in equation (20), each weight in the set of low precision weights 116 is an integer. In some examples, each weight in the set of low precision weights 116 is a power of 2. The set of bit precision values (b) (i.e., bit precision values 118 (FIG. 1)) comprises a bit precision value for each of the multiple layers. For each of the multiple layers, the weight

[0193]

number

[0194] Each weight in the precision-optimized set of layers may be represented in memory 102 with a value having a bit precision equal to the bit precision value for the layer. As part of performing multiple iterations to train DNN 106, machine learning system 104 may perform operations 300-308 for each of multiple iterations.

[0195] 3, the machine learning system 104 may use 300 the set of high-precision weights as input weights for neurons of the DNN 106 to calculate a first output data set based on a first input data set. For example, the machine learning system 104 may use a second set of weights as weights w to calculate output values for each of the artificial neurons of the DNN 106 according to Equation (1) or another activation function. The first output data set may be the output y of the output layer 108N of the DNN 106.

[0196] Additionally, the machine learning system 104 may determine 302 a loss function. For example, the machine learning system 104 may determine the loss function based on the data label pair, the first output dataset, the set of bit-precision values 118, the set of high-precision weights 114, and the set of hyperparameters 120. The data label pair comprises the first input dataset and the label. FIG. 4 is a flowchart illustrating exemplary operations for determining the loss function, as described in more detail below. In other examples, the loss function may be determined in various ways. For example, the loss function may include one or more additional coefficients, as described elsewhere in this disclosure.

[0197] 3, the machine learning system 104 updates 304 the set of high-precision weights 114 based on the loss function. For example, the machine learning system 104 may update the set of high-precision weights 114 (W) as shown in equation (22). Accordingly, the machine learning system 104 may determine the updated set of high-precision weights 114 such that the updated set of high-precision weights 114 is:

[0198]

number

[0199] where W is the set of high precision weights 114, μ is the learning rate,

[0200]

number

[0201] is the set of low-precision weights,116,

[0202]

number

[0203] is the partial derivative of the loss function with respect to the set of high-precision weights 114.

[0204] Additionally, the machine learning system 104 may update 306 the set of precision values. For example, the machine learning system 104 may update 306 the set of bit precision values 118 based on a loss function, such as the loss function shown in equation (23). Thus, the machine learning system 104 may update 306 the set of bit precision values 118 based on a loss function, such as the loss function shown in equation (23).

[0205]

number

[0206] where b is the set of bit precision values 118, μ is a learning rate, and

[0207]

number

[0208] is the set of low-precision weights,116,

[0209]

number

[0210] is the partial derivative of the loss function with respect to the set of bit-precision values 118, and sign(·) is a function that returns the sign of the argument of the function if the absolute value of the argument of the function is not less than a predetermined threshold and returns 0 if the absolute value of the argument of the function is less than a predetermined threshold.

[0211] In some examples, such as examples where the low-precision weights 116 are restricted to integer powers of two, the machine learning system 104 may update the set of bit-precision values 118 based on equations (25)-(29). Thus, the machine learning system 104 may determine that the updated first parameters for the layer are

[0212]

number

[0213] where θ is the first parameter for the layer and μ is the learning rate;

[0214]

number

[0215] is the partial derivative of the loss function with respect to θ. Additionally, the machine learning system 104 may determine that the updated second parameter for the layer is

[0216]

number

[0217] where θ is the second parameter for the layer and μ is the learning rate;

[0218]

number

[0219] is the partial derivative of the loss function with respect to θ. In this example, the machine learning system 104 may determine a set of quantized weights for a layer by applying a quantization function θ+θlog|w|, where w represents a weight in the second set of weights associated with the layer. The machine learning system 104 may determine a maximum weight in the set of quantized weights for the layer and a minimum weight in the set of quantized weights for the layer. The machine learning system 104 may set the bit-precision value of the layer based on the base 2 logarithm of the range defined by the maximum weight in the set of quantized weights and the minimum weight in the set of quantized weights (e.g., as shown in equation (29)).

[0220] After updating the set of high-precision weights 114(W) and after updating the set of bit-precision values 118, the machine learning system 104 may update the low-precision weights 116(W) based on the updated set of high-precision weights 114(W) and the updated set of bit-precision values 118.

[0221]

number

[0222] For example, the machine learning system 104 may update 308 the set of low-precision weights 116 as shown in equation (13):

[0223]

number

[0224] Thus, for each of the multiple layers, the machine learning system 104 may update the set of updated accuracy-optimized weights for the layer.

[0225]

number

[0226] The updated set of low-precision weights 116 may be updated to be equal to α, where α is the minimum weight in the constant-precision set of weights for the layer (i.e., the set of high-precision weights 114), W is the constant-precision set of weights for the layer, δ is the total number of constant steps in the discretized range from the minimum weight in the constant-precision set of weights for the layer to the maximum weight in a second constant-precision set of weights for the layer, and round(·) is a rounding function. In this example, δ is

[0227]

number

[0228] where β is the maximum weight of a constant precision set of weights for a layer and b is the bit precision value for the layer.

[0229] In some examples where the set of low-precision weights 116 is limited to integer powers of two, the machine learning system 104 may determine, for each weight in the set of low-precision weights 116, a set weight for the low-precision weights 116 that is equal to a sign value multiplied by a power of two exponent value. In this example, the sign value represents the sign of the corresponding weight in the set of high-precision weights 114. In this example, the exponent value is based on the base 2 logarithm of the corresponding weight in the set of high-precision weights 114. For example, the machine learning system 104 may determine an updated set of low-precision weights 116 as shown in equation (19).

[0230] Apart from the training process, the machine learning system 104 may use low precision weights 116 as input weights for neurons of the DNN 106 to calculate a second output dataset based on a second input dataset.

[0231]

number

[0232] The machine learning system 104 may use the set of low-precision weights 116 as input weights for neurons of the DNN 106 during the evaluation mode to calculate the output data based on the input data (310). In other words, the machine learning system 104 may use the second input data to generate the output data. For example, the machine learning system 104 may use the set of low-precision weights 116 as input weights for neurons of the DNN 106 during the evaluation mode to calculate the output data based on the input data.

[0233] In some examples of the present disclosure, referring to FIG. 2, neural network training (202) may result in the DNN 106 being equal to a power of two. In examples where the low-precision weights 116 are equal to a power of two, calculations during inference mode can be simplified by using logical shift operations instead of multiplication operations. As a result, the DNN 106 operates more efficiently and potentially with lower latency during inference mode because shift operations are less complex than multiplications. In one example, referring to FIG. 3, BitNet training can limit the weights to integer powers of two during operations (300)-(308). For example, during operation (308), the machine learning system 104 may update the low-precision weights 116 so that their values are integer powers of two, and during operation (310), the set of low-precision weights 116 is used as input weights for neurons in the neural network.

[0234] For example, the selection of values that are integer powers of two in operation 308 represents the quantization of the selected weights. To achieve a better match between high accuracy and the distribution of values between the BitNet-trained neural network, the BitNet training can select a quantization function that best preserves the coded distribution of the learned weight parameters, even when the quantization is non-differentiable. For example, in operation 308, the machine learning system 104 may select a quantization function sign(w)*2 (round(log|w|) may be used, where w is the original floating-point weight constructed in operation (306).

[0235] 4 is a flowchart illustrating example operations for determining a loss function according to the techniques of this disclosure. In the example of FIG. 4, the machine learning system 104 determines 400 a first operand l(W). The first operand l(W) is an intermediate loss function. In some examples, the intermediate loss function is a function of the data label pair (X (l) ,y), the first output data (X (l) ) and a second set of weights (W). Equations (7) and (8) show examples of intermediate loss functions. Thus, the first input dataset comprises a batch of training data-label pairs, and the machine learning system 104, as part of determining the first operand, determines whether the first operand is

[0236]

number

[0237] where B is the total number of data label pairs in the batch of data label pairs, each label in the batch of data label pairs is an element of the set of labels having B labels, i is an index, log(·) is a logarithmic function, N is the total number of layers, and y i is the i-th label in the set of labels,

[0238]

number

[0239] is the output of the Nth layer of the multiple layers when data of the i-th data label pair of the batch of data label pairs is provided as input to the DNN 106 and the DNN 106 uses the second set of weights. In this example, the data label pairs of the batch of data label pairs may be independent, uniformly distributed data label pairs. In some examples, the intermediate loss function may be any standard supervised or unsupervised loss function, such as cross-entropy (or negative log-likelihood) for supervised classification or reconstruction error for unsupervised autoencoders. In one example where the intermediate loss function is a loss function, the machine learning system 104 may calculate the first operand as shown in Equation (33).

[0240]

number

[0241] In equation (32), x is the input data,

[0242]

number

[0243] is the output of the DNN 106 using high precision weights 144.

[0244] Furthermore, in the example of FIG. 4, the machine learning system 104 calculates the second computation target as a hyperparameter (λ1) and a quantization error q(w (l) ,b (l) ) (402). For example, the machine learning system 104 determines the second operand to be equal to the product of

[0245]

number

[0246] For each of the multiple layers, the machine learning system 104 may determine a quantization error for the layer based on the difference between the set of high-precision weights for the layer (i.e., the second set of weights) and the set of low-precision weights for the layer (i.e., the first set of weights), as shown in Equation (14). In some examples, such as examples in which the low-precision weights 116 are restricted to integer powers of 2 and the machine learning system 104 uses the loss function defined in Equation (21), instead of determining the second operand as the product of the hyperparameter and the quantization error, the machine learning system 104 may determine the second operand to be equal to the value of the hyperparameter (λ) and the distillation loss, as described above.

[0247] Additionally, in the example of FIG. 4, the machine learning system 104 determines whether the third operand is a hyperparameter (λ2) or a

[0248]

number

[0249] a third operand may be determined to be equal to the product of i and b, where i is an index, N is a total number of layers, and b i is the bit precision value of the ith layer of the multiple layers 404. The machine learning system 104 may determine the loss function as the sum of the first operand, the second operand, and the third operand.

[0250] This disclosure describes techniques that can manage DNN memory size while simultaneously providing new opportunities for neural network composition resulting from the low-precision techniques described herein. FIG. 5 is a block diagram illustrating an exemplary heterogeneous neural architecture. The heterogeneous neural architecture of FIG. 5 highlights new areas of neural network composition resulting from the low-precision techniques of this disclosure. In the example of FIG. 5, system 500 includes a subband decomposition unit 502, a binary neural network (BNN) 504, a BNN 506, a DNN 508, and a fusion unit 510. The subband decomposition unit 502 receives input data. The BNN 504, the BNN 506, and the DNN 508 receive the output of the subband decomposition unit 502 as input. The fusion unit 510 receives the outputs of the BNN 504, the BNN 506, and the DNN 508 as input. The fusion unit 510 generates an output. The machine learning system 104 (FIG. 1) may implement each of the subband decomposition unit 502, the BNN 504, the BNN 506, the DNN 508, and the fusion unit 510.

[0251] DNNs are neural networks that use only single-bit precision to represent the Synapse weights and Synapse activations. This represents a significant savings in processing because the computational architecture does not require multiplications and memory usage is significantly reduced. BNNs have previously been applied for object detection and classification. In inference mode, BNNs perform in less than 1 / 32 the runtime memory footprint without requiring multiply-accumulate hardware. To give a perspective, AlexNetCNN uses only 0.25W while being 23x faster by using bitwise operations.

[0252] The subband decomposition unit 502 can decompose an image into distinct frequency bands so that each frequency band can be processed by a low-precision DNN, such as the BNN 504, the BNN 506, and the DNN 508. Separating the image into high- and low-frequency bands allows the DNN to process edges and textures separately. The decomposition relies on preprocessing the input data into distinct subbands in a process that separates image content, much like wavelet decomposition. This process may also include other forms of data preprocessing, such as data augmentation to rotate, mirror, and adjust contrast of the image.

[0253] The subband decomposition process allows for the assembly of neural networks in which each subband can be processed in parallel by a different DNN. This approach allows the subband decomposition unit 502 to decompose the input data into multiple parallel streams. According to the techniques of this disclosure, the machine learning system 104 may select each subband to best suit storage and computational requirements based on the basic premise that each subband is "optimal" from a learning perspective. Each subband may be optimized from a bit precision perspective. There is a cost to preprocessing the input data with savings when reducing the precision of each processed subband.

[0254] As described elsewhere in this disclosure, training a DNN using the bit precision itself as a hyperparameter can be advantageous from an algorithm performance perspective. The resulting DNN 508 may have different bit precisions for each layer of the DNN 508. Other approaches, such as quantizing and rounding the DNN weights, not only degrade algorithm performance but also treat all weight values with the same global precision (e.g., 32 bits, 16 bits, 8 bits, etc.). Similarly, a BNN may be trained such that the minimum bit setting is applied to a uniform setting for the DNN layers. The amount of memory size savings may depend on the algorithm task (e.g., the number of features and object classes).

[0255] In the example of FIG. 5, the fusion unit 510 may generate output data based on output data generated by one or more of the BNN 504, the BNN 506, and the DNN 508. In some examples, the fusion unit 510 may be another DNN. In some examples, the fusion unit 510 may be a program that does not use a DNN. Thus, FIG. 5 illustrates an exemplary embodiment comprised of neural networks that can optimize each neural network (BNN 504, BNN 506, DNN 508, fusion unit 510) from a training perspective and a resource usage perspective (e.g., controlling overall memory retention and bit precision for hardware computations).

[0256] The following sections of this disclosure demonstrate that a BitNet DNN, such as DNN 106 (FIG. 1), can support a higher learning rate (e.g., converge faster) and arrive at DNN weight settings that can have higher performance than standard DNNs. Using bit precision allows the machine learning system 104 to provide training guidance with a more clearly directed goal (e.g., within the range of values that can be selected for the weights). For example, the machine learning system 104 can better regularize the training process because BitNet training can better guide the acceptable range of values for the neural network weights. In contrast, standard high-precision approaches have a much wider range of values that can be selected for the weights and therefore may require a longer training process to arrive at appropriate values. In some examples, the machine learning system 104 starts with low precision and gradually increases the bit precision to quickly approach a solution, thereby reducing overall training time. For example, the machine learning system 104 first uses a value of λ2 higher than the value of λ1, and then gradually decreases the value of λ2 relative to the value of λ1.

[0257] The BitNet DNN has been evaluated on two well-known benchmarks for image recognition and classification: MNIST and CIFAR-10. For illustrative purposes, a simple neural architecture based on LeNet-5 was used without training for many epochs, so it was not necessary for the BitNet DNN to provide conventional performance. Instead, emphasis is placed on the comparison of the BiNet DNN to corresponding high-precision realizations, specifically the same CNN with 32-bit parameters, referred to in this disclosure as "LeNet EP32." For the same reasons, no preprocessing or data augmentation was performed. Batch normalization (also known as centering) was performed on the inputs to avoid covariate shifts between batches. Automatic differentiation, as described in Theano, was used to calculate the gradients associated with equations (22) and (23).

[0258] The MNIST database of handwritten digits contains a total of 70,000 grayscale images, each 28x28 in size. Each image consists of one digit: 0, 1,..., 9. The data is divided into 50,000 training, 10,000 test, and 10,000 validation examples. The digits are size-normalized and centered in images of a fixed size. The training data is divided into batches of 250 images. The baseline architecture for this database consists of two convolutional layers, each consisting of 30 5x5 filters followed by 4x4 pooling and 50 5x5 filters followed by 4x4 pooling. The filtered images are fed into a hidden layer of 500 hidden parts (i.e., artificial neurons) followed by a softmax layer to output scores across 10 labels.

[0259] The CIF AR-10 dataset consists of 60,000 32x32 color images in 10 categories, with 6,000 images per category corresponding to object prototypes such as "cat," "dog," "airplane," and "bird." 40,000 images were used for training, and 10,000 images were used for testing and validation, respectively. The training data was divided into batches of 250 images. The baseline architecture for this dataset consisted of two convolutional layers, each consisting of 30 5x5 filters followed by 4x4 pooling and 50 5x5 filters followed by 4x4 pooling. The filtered images were fed into a hidden layer of 500 hidden parts (i.e., artificial neurons) followed by a softmax layer to output scores across the 10 labels.

[0260] Figures 6A and 6B show exemplary performance of BiNet and LeNet-FP32. That is, Figures 6A and 6B show the performance of BiNet compared to LeNet-FP32 on the MNIST and CIFAR-10 datasets. The left panel of each of Figures 6A and 6B shows the validation error % over training iterations, and the right panel of each of Figures 6A and 6B shows the negative log-likelihood over training iterations. In Figures 6A and 6B, the final validation error is shown in parentheses. Furthermore, in Figures 6A and 6B, the learning rate μ is halved after each epoch, which is 250 iterations for MNIST and 200 iterations for CIFAR-10.

[0261] As shown in Figures 6A and 6B, regularization in BiNet results in significantly faster training. In the left panels of Figures 6A and 6B, BiNet's validation error decreases more rapidly than LeNet-FP32. The resulting validation error for BiNet after 100 epochs is 2% lower than LeNet-FP32. Similarly, BitNet achieves a 5.25% error on the test set, compared to a 7.3% error for LeNet-FP32. For a given performance, BiNet requires roughly half the number of iterations of the baseline. The right panels of Figures 6A and 6B show that the training error for high-precision parameters for BiNet decreases more quickly than LeNet-FP32, indicating that the low validation error is not caused by quantization alone. In addition to superior performance, BiNet uses an average of only 6 bits per layer, corresponding to a 5.33x compression increase relative to LeNet-FP32.

[0262] Figure 7 shows the number of bits used to represent the parameters of each layer of a CNN. That is, Figure 7 shows the change in the number of bits over the training iterations. We can see that the number of bits is concentrated within the first five epochs. We can also see that the gradient with respect to the bits quickly becomes zero.

[0263] In one experiment, we demonstrate the impact of the hyperparameters (i.e., λ1 and λ2) in Equation (20). In this experiment, each CNN is trained for only 30 epochs. Figures 8A and 8B show the impact of performance and compression on MNIST and CIFAR-10 data, respectively. In other words, Figures 8A and 8B show the sensitivity of BiNet's hyperparameters to the test error and compression ratio for the MNIST and CIFAR-10 datasets, respectively. For Figures 8A and 8B, we define the compression ratio as the ratio of the total number of bits used by LeNet-FP32 (=32 × 4) to the total number of bits used by BiNet. For both datasets, when λ2 = 0 and λ1 = 1, BiNet uses 32 bits uniformly spaced across the range of parameter values, and we find that it achieves significantly better test error than LeNet-FP32, which uses 32 bits to maintain the linear transformation in Equation (11), which is nonlinear and less sensitive to range. On MNIST, in the left panel of Figure 8A, BiNet with λ2 = 0 and λ1 = 1 and using 32 bits achieves a test error of 11.18% compared to 19.95% error for LeNet-FP32, and -7 and λ2=10 -3 At the optimal setting, BiNet achieves an error of 11%.,The same observation holds for the CIFAR-10 dataset, as shown,in Figure 8B.

[0264] In contrast, when λ1 = 0 and λ2 = 1, BiNet uses only 2 bits per layer and achieves a test error of 13.09% on NMIST, a small degradation at the cost of 16x compression. This approach offers some flexibility in limiting the bit width of the parameters and provides an alternative way to arrive at the binary or ternary networks of previous work.

[0265] For a constant value of λ1, increasing the value of λ2 reduces the number of bits, further increases compression, and slightly degrades performance. For a constant value of λ2, increasing the value of λ1 increases the number of bits and decreases compression. The variation in compression ratio with test error is significant. In practice, most of the settings tested resulted in similar test errors but very different numbers of bits per layer. The optimal setting was found by grid search to maximize both compression and accuracy. For MNIST and CIFAR-10, this was found for λ1=10 -7 and λ2=10 -3 is.

[0266] In one experiment, we evaluated the impact of bit regularization on performance and compression while adding more layers to a CNN. There are two main challenges in training a DNN. First, a large number of parameters requires increased data volume and training time. A second problem with very deep neural networks is overfitting to the training data, which can be seen by the increase in test error with the addition of more layers. We show that bit regularization helps train deep networks without overfitting. The space of possible architectures and layers that can be added is intractable. We present sample arrangements of layers that can be gradually added to improve performance. These layers were selected manually using knowledge and some experimentation.

[0267] Table 1 shows the results on MNIST after 30 epochs.

[0268] [Table 1]

[0269] Table 1 shows the performance of BiNet after 30 epochs on MNIST and after 100 epochs on CIFAR-10 as the complexity of the neural architecture increases. The first column (#) in Table 1 represents the total number of layers. Test error was evaluated on the test set (i.e., data not seen by the DNN106 during training), and the error measure is the percentage of incorrect answers. The compression ratio (Compr. Ratio) is a percentage of the average number of bits used by BiNet. The column to the right of the compression ratio specifies the architecture and number of bits of the final BiNet model. In each table, the last row contains all architectures, and the columns read from left to right are neural architectures. In the rows above the last row, some of these layers are omitted to train a smaller DNN. The heads have the format PQR, where P is the number of convolutional filters, Q is the size of each filter, and R is the size of max pooling performed after filtering. In the case of a dense layer (i.e., a fully connected layer), it means the number of neurons. The first column of the convolutional layer specifies the number of filters, the spatial domain, and the pooling size. Here, λ1 = 10 -7 and λ2=10 -4 is.

[0270] We first start with BiNet with four layers and demonstrate its performance in the previous section. First, we observe that the test error gradually decreases without any signs of overfitting. Second, we observe that the number of bits and compression rate are not significantly affected by the architecture and appear to be strong functions of the data and hyperparameters. Second, we observe that the test error decreases with additional convolutional and dense layers. The addition of 1x1 filters (corresponding to global scaling) can reduce the test error without increasing the number of parameters relative to the addition of dense layers.

[0271] A similar comparison on the CIFAR-10 dataset is shown in Table 1 after 100 epochs of training. There is no sign of overfitting, as both the training and test errors decrease with each experiment. As the architecture becomes more complex, there is a slight decrease in performance, due to the reduction in the total number of parameters and the training for a fixed 100 epochs. As in the previous experiments, we can see that the compression ratio is not significantly affected by the architecture. There are no prior results for CIFAR-10. However, BiNet is focused on performance at any given time; each of these experiments takes approximately 1 hour compared to approximately 20 hours for prior results.

[0272] In one experiment, BiNet's accelerated learning property is indirectly related to the learning rate. To this end, a linear penalty is used for the number of bits instead of the exponential penalty (the third term) in Equation (20). Figure 9 shows an example of BitNet's performance on the MNIST dataset. In particular, Figure 9 shows the validation error rate for BitNet versus LeNet-FP32 over minibatches. The final validation error is shown in parentheses in Figure 9. In the example in Figure 9, the learning rate μ is multiplied by 0.1 after each epoch. The left panel of Figure 9 shows that BiNet exhibits similar fast learning as using the exponential penalty. The right panel of Figure 9 shows that the baseline LeNet-FP32 can also learn at a rate similar to BiNet when the learning rate is increased. This point is further illustrated in Figure 10, where the linear coefficient for the number of bits shows a direct relationship with the learning rate for various values of λ2. Figure 10 shows exemplary performance of BitNet for various linear bit penalty factors on the MNIST and CIFAR-10 datasets. In particular, the right panel of Figure 10 shows that as the value of λ increases, the performance becomes unstable and degrades, whereas as the value of λ decreases, the learning curve becomes smoother.

[0273] However, increasing the learning rate globally for all parameters in LeNet-FP32 CNNs does not produce the same stability as the accuracy obtained for each parameter in BiNet. Furthermore, learning fluctuates, especially when the learning rate is further increased. This establishes an interesting connection between sophisticated gradient descent algorithms such as AdaGrad, which also address the issues of low-precision training, momentum, or "static" learning rates. AdaGrad is described by Duchi et al., "Adaptive Subgradient Methods for Online Learning and Stochastic Optimization," Journal of Machine Learning Research, pages 2121–2159, July 12, 2011. Alternatively, BiNet can be evaluated as training with noisy gradients, demonstrating a method for promoting stable learning with little noise probability distribution. The noise incorporated into BiNet gradients resembles a disjoint set of Gaussian distributions with uniformly spaced centers determined by W and b, the number of which is determined by b and the variance of the range of values of W.

[0274] The deployment of deep neural networks in real-world applications is significantly limited by their computational and storage requirements. In summary, this disclosure describes a flexible tool for training compact DNNs given an indirect specification of the total number of bits available on the target device. This disclosure provides a formulation that incorporates constraints, such as regularization, in a traditional classification loss function. This formulation is based on controlling the expressive power of the network by dynamically quantizing the range and set of values that parameters can take. Experiments described here demonstrate superior learning in terms of training and test error over comparable unregularized networks. The robustness of our approach was demonstrated with increasing neural network depth and various hyperparameters. Our experiments showed that BiNet may have an indirect relationship with the overall learning rate. BiNet can be interpreted as having a dynamic learning rate per parameter that depends on the number of bits. In that sense, bit regularization is related to dynamic learning rate schedulers such as AdaGrad. In some examples, the machine learning system 104 may perform constraint annealing to take advantage of fast initial learning combined with high-precision fine-tuning. In that sense, BiNet can be interpreted as having a joint optimization for learning a representation of the training data and the system architecture.

[0275] Previous sections of this disclosure described methods for training DNNs such that bit precision is part of the parameters of the training procedure. In other words, previous sections of this disclosure described optimizing bit precision as part of the training phase. The resulting DNNs may have different bit precisions for different DNN layers. Benefits may include a smaller memory footprint, faster learning, and potentially higher algorithm performance.

[0276] The next section of this disclosure provides additional details related to processing embodiments and their relationship to training and inference. In particular, this disclosure describes (1) how system architecture parameters are used within BiNet selection for DNN assembly, and (2) how system architecture parameters are used to distribute processing for trained DNNs.

[0277] This disclosure uses the term "system architecture" to mean a set of processing hardware (and associated software stack) for training one or more DNNs and / or executing one or more DNNs in inference mode. In some examples, the processing hardware may include a virtual machine that operates like physical hardware. In some examples, the system architecture includes one or more processors (e.g., CPU, GPU, FPAG, DSP, or virtual representations thereof) that support training of BiNet DNNs. In some examples, the same system architecture may be used to execute the trained DNNs in inference mode. In some examples, alternative system architectures may be used. For example, one or more BiNet DNNs may be trained in a first system architecture (e.g., a cloud computing system) and then used in inference mode in a second, different system architecture (e.g., a mobile device). This disclosure uses the term "neural network software architecture" to mean a system consisting of one or more DNNs and the configuration of these DNNs. For example, a neural network software architecture may have multiple individual DNNs that can interact with each other. In addition to the techniques described above, this disclosure describes techniques related to methods of DNN training in which system architecture inputs are used to select the configuration of the DNN model. Additional techniques of this disclosure include methods for distributing DNN processing (both training and inference) among processors of a system architecture.

[0278] FIG. 11 illustrates an exemplary neural network software architecture composed of a hierarchy of layers, each of which is a DNN. That is, FIG. 11 illustrates an exemplary neural network software architecture composed of a hierarchy of neural networks. In the example of FIG. 11, sensor inputs to neural network software architecture 1100 are video data, audio data, and a depth map. The output of neural network software architecture 1100 is a classification of activity detected from an analysis of the video, audio, and depth map. In one example, an application of neural network software architecture 1100 can be human gesture recognition. A human developer or a computer system may select neural network software architecture 1100.

[0279] Further, in the example of FIG. 11 , an 8-bit CNN 1102 is selected for video data. In this example, the high-precision weights of the CNN 1102 are 8 bits each. During training of the CNN 1102, the precision-optimized weights of the CNN 1102 may be updated to have less than 8-bit precision. A 4-bit CNN 1104 followed by a 32-bit LSTM (Long Short-Term Memory Neural Network) 1106 is selected for audio data. In this example, the high-precision weights of the CNN 1104 are 4 bits each, and the high-precision weights of the LSTM 1106 are 32 bits each. During training of the CNN 1104 and the LSTM 1106, the precision-optimized weights of the CNN 1104 and the LSTM 1106 may be updated to have less than 4-bit precision and less than 32-bit precision, respectively. A 1-bit BNN 1108 is selected for depth maps. 11, the three streams generated by the CNN 1102, LSTM 1106, and BNN 1108 are fed to an 8-bit MLP (Multi-Layer Perceptron) 1110 to generate an activity classification output. The high-precision weights of the MLP 1110 are 8 bits each. During training of the MLP 1110, the precision-optimized weights of the MLP 1110 may be updated to have less than 8-bit precision.

[0280] Video data may comprise two-dimensional images with distinct color planes (red, green, blue), typically 8 bits per pixel. Audio data may comprise one-dimensional streams, typically 16 bits per sample. Audio processing may include feature extraction (e.g., by a CNN 1104) and subsequent audio analysis (e.g., by an LSTM 1106). Depth maps may comprise two-dimensional masks with pixel values representing distance from a sensor (e.g., a camera).

[0281] FIG. 12 illustrates an exemplary system architecture 1200 with a heterogeneous set of processors. In the example of FIG. 12, the processors include a CPU 1202, a GPU 1204, a GPU 1206, an FPGA 1208, and a DSP 1210. Each processor has different bit-precision support. Exemplary supported bit-precision may include 8-bit and 1-bit precision (integer) and 32-bit and 64-bit (floating point). Different processor hardware may handle DNN computations differently (e.g., with different levels of parallelism and memory organization) to support different size, weight, and power (SWaP) offerings. The processors may be connected to a network. In the example of FIG. 12, each network connection has different bandwidth availability, e.g., 10 Mbps, 100 Mbps, and 10 Gbps. In the example of FIG. 12, there is a router 1212 that manages traffic but does not perform computations for DNNs. Network bandwidth availability sets communication limits between processors and can therefore affect how DNNs are trained and processed during inference.

[0282] This disclosure describes a technique for selecting bit precision based on system architecture capabilities. This disclosure uses the examples of FIGS. 11 and 12 to illustrate this technique. Given a sensor data type that needs to be analyzed, the machine learning system 104 may first examine the system architecture that requires the neural network software architecture 1100 to operate in inference mode. For example, the machine learning system 104 may determine that the optimal processor for a 1-bit BNN is an FPGA because it has fine-grain programmable units that can support binary operations. In contrast, an LSTM may require higher precision for time series analysis. The machine learning system 104 may also consider the network bandwidth needed to support communication between various layers of the DNN. For example, video processing requires more communication bandwidth than audio processing. Another example of a system architecture parameter may include memory footprint (e.g., a 1-bit BNN has less memory requirements than an 8-bit CNN).

[0283] The system architecture parameters are used to map the neural network software architecture to an appropriate processor in the system architecture. For example, the machine learning system 104 of the computer system 100 (FIG. 1) maps the neural network software architecture to an appropriate processor in the system architecture. The machine learning system 104 may use a cost function to select the optimal mapping (i.e., a best-fit method may be used). The cost function may be one of size, weight, power, and cost (SWaPC). For example, in the case of selecting an 8-bit CNN, the machine learning system 104 may use a cost function to select a mapping that provides a lower system performance. For example, the machine learning system 104 may evaluate various potential mappings of the neural network software architecture to the processors in the hardware architecture. The machine learning system 104 may use the mapping cost function to select a mapping of the neural network software architecture to the processors in the hardware architecture.

[0284] FIG. 12 shows an example mapping of neural network software architecture to a system architecture. More specifically, the machine learning system 104 may map the 8-bit CNN 1102 of FIG. 11 to the 64-bit floating-point GPU 1206 of FIG. 12. The same 8-bit CNN, when mapped to an FPGA, may incur greater computational resources (e.g., greater use of memory and FPGA fabric to support floating-point calculations). Furthermore, in the example of FIG. 12, the machine learning system 104 may map the 1-bit BNN 1108 to the 1-bit FPGA 1208, the 8-bit MLP 1110 to the 16-bit CPU 1202, the 32-bit LSTM 1106 to the 64-bit floating-point GPU 1204, and the 4-bit CNN 1104 to the 8-bit DSP 1210. After the machine learning system 104 maps the DNN to the processor, the processor may execute the DNN. For example, in the example of FIG. 12, the GPU 1206 may execute the CNN 1102.

[0285] One aspect of the present disclosure is directed to a BiNet training method. For example, in some examples, system architecture parameters serve as input to a BiNet DNN (e.g., DNN 106 (FIG. 1)) to select an appropriate neural network software architecture and a mapping of the DNN of the neural network software to a processor of the hardware architecture. For example, DNN 106 may be a BiNet DNN that receives a description of a hardware architecture and a description of a problem that the neural network software architecture is deployed to solve. In this example, the output of DNN 106 may be an appropriate neural network software architecture for the problem that the neural network software architecture is deployed to solve and a mapping of the DNN of the neural network software architecture to a processor of the hardware architecture. In this example, DNN 106 may be trained using existing examples of hardware architectures and existing examples of problem descriptions.

[0286] Furthermore, the ability of BiNet DNNs to target multi-bit precision allows for efficient mapping to available hardware resources and is particularly useful for heterogeneous sets of processors. In other words, because DNNs can be trained to represent weights using more or fewer bits, the machine learning system 104 may be able to train multiple versions of the same BiNet DNN based on the same input data, such that different versions of the same BiNet DNN have different bit depths.

[0287] The cost functions described in this section can be integrated into the BiNet training algorithm to find the potentially optimal DNN algorithm performance. For example, the machine learning system 104 may use a modified version of Equation (20), as shown in the following Equation (33):

[0288]

number

[0289] In the example of Equation (33), P represents a set of hardware parameters of the hardware architecture, and r(b,P) is a function, referred to herein as a resource function, that takes as parameters the precision-optimized bit depth b of the BiNet DNN and the set of hardware parameters P. In some examples, the resource function r generates a large value when the precision-optimized bit depth of any layer of the BiNet DNN exceeds the limit represented by the set of hardware parameters P. For example, the resource function r may be a step function that generates a value of 0 when the bit depth of each layer of the BiNet DNN is below the limit and a value of 1 when the bit depth of any layer of the BiNet DNN is above the limit. In other examples, the resource function r may generate values that gradually increase as the degree to which the precision-optimized bit depth exceeds the limit increases. In some examples, the limit is memory requirements. For example, in one example, the limit may be the total amount of memory required to store the precision-optimized weights b (e.g., the total amount of memory required to store b needs to be less than 32 kilobytes). In another example, the limit may be the total amount of memory required to store b that can be used to store the precision-optimized weights of a single layer of the BiNet DNN (e.g., a layer cannot have more than 4 bits of precision-optimized weights). In another example, the resource function r may return a value based on whether the weights aligned with a read boundary. For example, if a single memory read operation returns 8 bits, it would require two read operations to return a 9-bit weight. In this example, the resource function r returns a low value if the precision-optimized weights aligned with a read boundary. Thus, in this example, the priority is to minimize the number of read operations, thereby reducing the potential energy consumption and delays associated with the read operations.

[0290] In another example, the machine learning system 104 may use a loss function expressed by the following equation (34):

[0291]

number

[0292] In Equation (34), λ3 is a third hyperparameter, and r(b, P) may be defined as described above. The loss function in Equation (34) may be able to consider bit depth separately from the hardware parameter p. In other examples, the loss function may have one or more additional coefficients associated with additional hyperparameters and additional functions based on b and various hardware coefficients. In this manner, the loss function may be able to consider various hardware parameters when training the BiNet DNN. Thus, in some examples, the techniques herein enable the machine learning system 104 to find optimal DNN performance given a set of hardware resources, such as the processor and communication links of FIG. 12. Computations during the training phase may also be similarly mapped to appropriate hardware when testing on appropriate hardware that supports the selected bit precision.

[0293] The techniques described herein may be applicable to hardware resources within a processor. For example, in a system-on-chip (SoC) having multiple processor cores, hardware accelerators, and reconfigurable fabric, the techniques may similarly be used with system architecture parameters (where the system architecture parameters are related to the SoC) that map the neural network software architecture to SoC resources. In another example, if the system architecture includes or consists of a bank of FPGAs, the machine learning system 104 may use the DNN 106 to select appropriate FPGA resources (e.g., allocate FPGA lookup tables (LUTs) and memory) to obtain the highest and optimal DNN performance based on the available hardware.

[0294] As mentioned above, FIG. 12 illustrates an example mapping of a neural network software architecture to system architecture resources. Previously, this disclosure described an AI system capable of managing and predicting resource requirements. In other words, the AI system may predict resource requirements by determining which portions of the neural network software architecture will execute most effectively on available processors in a given system architecture. For example, an engineer or computer system may identify multiple distinct scenarios. In some examples, the distinct scenarios may be different system architectures. In some examples, the distinct scenarios may include the same set of processors with differences in parameters such as available bandwidth, remaining available battery life, remaining allocable memory space, processor workload, etc. In this example, the machine learning system 104 may train multiple versions of the same neural network software architecture using the techniques of this disclosure based on the distinct scenarios. Thus, each version of the neural network software architecture may be tailored for use in a different predicted scenario.

[0295] 12 , if one of the hardware resources becomes unavailable (e.g., power loss, loss of network connection, etc.), the BiNet training method can trigger a mapping of the neural network software architecture to a new set of system architecture parameters. In particular, the new mapping is performed on the neural network software architecture (e.g., neural network software architecture 1100 of FIG. 11 ) to the new system architecture (e.g., a subset of processors in FIG. 12 ). For example, in the example of FIG. 12 , if GPU 1206 becomes unavailable, mapping module 122 may place a version of neural network software architecture 1100 tailored for use in situations where GPU 1206 is unavailable on the remaining processors of system architecture 1200 (i.e., the hardware architecture). For example, in the version of neural network software architecture 1100 tailored for use in situations where GPU 1206 is unavailable, both CNN 1102 and LSTM 1106 may be mapped to GPU 1204. One aspect here is the combination of BiNet training with a dynamic set of system architecture parameters. One aspect here is that the cost function for BitNet may be based on a dynamic set of system architecture parameters (e.g., a loss function). In this example, the cost function may be the loss function previously described in Equation (34), where λ1 and λ2 are set based on optimizing one of the size, weight, power, and cost (SWaPC) of the hardware architecture 1200.

[0296] In another example, BitNet training can be used to arrive at multiple neural network software architectures for a set of typical settings. For example, referring to FIG. 12, router 1212 has periodic bandwidth reductions (e.g., during different periods when network congestion exists). BitNet may generate two neural network software architectures during training: one that maps the DNN to the system using router 1212 and one that maps the DNN to the system without using router 1212. Thus, this process is similar to fat binaries, where multiple statically compiled programs can be selected at runtime based on dynamic conditions. In our case, BiNet DNN selects the appropriate neural network software architecture for the available hardware. In other words, DNN 106 may select an appropriate previously prepared neural network software architecture from multiple previously prepared neural network software architectures. Each DNN in the previously prepared neural network software architectures has already been trained.

[0297] For the machine learning system 104, the weights selected in the annealing constraint step are selected based on the joint optimization of DNN performance and hardware resources. Regarding loss function selection, the λ parameters (λ, λ, and λ) in Equation (34) are based on hardware parameters P, and this disclosure describes hardware parameters P that may be updated during the training phase to influence the selection of annealing constraints. The selection of hardware parameters P during training enables Pareto-optimal selection of hardware resources, e.g., with respect to size, weight, and power. Such an approach to dynamic hardware parameter selection serves as a design tradeoff method for designers using loss functions as a joint optimization of machine learning performance and hardware resources. This disclosure describes loss functions using Equation (34) and the correspondence between bit precision and hardware parameters for r(b,P). It should be understood that the correspondence can be a similar mapping of machine learning hyperparameters (e.g., accuracy, learning rate, DNN depth, fusion layer, activation, learning algorithm) to hardware parameters to enable joint optimization. It should be understood that the correspondence may also be other similar mappings of machine learning parameters to other physical constraints that affect DNN training (e.g., environmental, data biases), in which case joint optimization can be performed in a manner similar to equation (34).

[0298] In this section, this disclosure describes an example of a spaceborne application. This example uses a spaceborne application because there are strict limitations on available SWaP and communication bandwidth. Small satellites such as CubeSats are only 10 x 10 x 10 cm and weigh 1.33 kg. CubeSats typically operate as a distributed swarm, each with a different set of sensors. There is also a limited power budget and limited communication bandwidth to the ground station and between the CubeSats. Referring to FIG. 12, the machine learning system 104 (FIG. 1) may map a neural network software architecture to the CubeSat swarm. BiNet DNN may be used to distribute and map DNN inference computations among the CubeSats. In this example, if the communication link is lost with a subset of the CubeSats, BiNet DNN can be used to retrain and distribute the DNN computations to currently available CubeSat hardware. One aspect here is that the BiNet DNN (e.g., DNN 106) may simultaneously distribute computation and select neural network software architectures. Because the BiNet DNN can target specific bit precision, it can optimally map other DNNs to system resources. By changing the set of system resources, the DNN 106 can dynamically reselect different neural network software architectures to suit the available hardware. In another example, the BitNet cost function for mapping can be one of resilience. If the CubeSat network topology is allowed to change, the BiNet DNN can be used to find the optimal network shape among the available CubeSat hardware. For example, point-to-point network communication between CubeSats can be performed in a manner that supports optimal DNN performance.

[0299] AI systems have been demonstrated in countless applications, from search engines to autonomous vehicles. AI systems may use DNN algorithms, which have recently shown superior performance in learning complex relationships from multidimensional data. However, current AI systems typically lack the ability to learn beyond what the AI system has been trained to do. In situations where the full details are often unknown in advance, current AI systems are unable to rapidly respond and adapt to dynamic environments. Furthermore, traditional approaches may require taking the system offline for retraining / reprogramming. This process is not only disruptive or infeasible for a given task, but is also expensive and time-consuming.

[0300] Therefore, it may be advantageous to have an AI system that continues to perform learning substantially simultaneously with inference. More particularly, it may be advantageous for an AI system to support learning and inference simultaneously in an efficient manner (e.g., from a learning perspective and from a power / performance perspective). Such an AI system may autonomously self-reconfigure on the fly, including adjusting hyperparameters such as the learning rate.

[0301] The present disclosure provides an AI system that can (a) predict learning needs and (b) dynamically and proactively manage operations / resources. The AI system may select or search for optimal DNN parameters without training for full convergence. The techniques herein decouple the training and DNN structure optimization tasks, thereby moving away from traditional approaches that rely on iterative model selection and training to find a set of hyperparameters that leads to optimal performance for a given dataset.

[0302] The technology herein can be applied in many fields because it applies to the fundamental ability of AI systems to select infrastructure (learning methods, software / hardware, etc.). The technology allows AI systems to adapt at the edge by providing a way to trade off algorithm performance and hardware SWaP (size, weight, and power). The technology herein can be useful in applications such as self-driving cars and robots, where the ability to self-reconfigure improves autonomous aspects while improving learning capabilities. The technology herein enables powerful DNNs in resource-limited environments such as mobile smartphone devices. The technology herein can be a useful tool in other fields such as cybersecurity, where prior information about cyberattacks is unknown (and taking the system offline for retraining is not a viable option). The technology herein can be useful for user customization / personalization, where AI systems self-reconfigure based on user input.

[0303] This disclosure describes a system that senses, learns, and responds. The system (e.g., DNN 106) collects input from sensors from the external environment and internal operations. The sensors may generate various types of data, such as data representing processor availability, communication link up / down status, electrical energy availability, temperature, battery level, remaining memory space, real-time performance requirements, etc. The system may learn from these input sensor data. This disclosure extends to a "response" aspect in which the system reconfigures itself (algorithm, software, hardware, etc.) based on what it senses and the learning it performs. In this case, the system (a) predicts what the system needs in terms of learning and (b) dynamically and proactively manages operations / resources to achieve the learning goals. For example, the DNN 106 may be implemented as a BiNet DNN, which takes hardware parameters as input and outputs a neural network software architecture with one or more DNNs that are mapped to the processors of the hardware architecture. One or more DNNs in the neural network software architecture may be the BiNet DNN itself. In some examples, one or more DNNs in the neural network software architecture may be pre-trained. In some examples, the machine learning system 104 may train one or more of the DNNs of the neural network software architecture to fit the requirements of a particular processor after the DNN 106 selects the neural network software architecture and mapping. In some examples, the DNN 106 may continue to update the neural network software architecture as the DNN 106 receives new inputs from sensors.

[0304] Systems exist that can adjust system settings based on computational demands and resource limitations (e.g., a process can adjust operating frequency levels based on remaining battery power). According to the techniques of the present disclosure, the machine learning system 104 may reconfigure the DNN 106 based on selecting or searching for optimal learning parameters. More specifically, the techniques involve selecting appropriate DNN parameters (e.g., weights) rather than hardware / software parameters (e.g., processor operating frequency, software processing demands, remaining battery power). With regard to DNN training, the techniques for selecting or searching for optimal DNN parameters may be performed without full convergence training. This is because, in one example, a DNN with low bit precision can be an alternative neural network that approaches the computational performance of a high-precision neural network. The present disclosure combines (a) system reconfiguration, (b) DNN learning parameter selection and optimization, and (c) skipping sufficient training of the DNN, thereby essentially ceasing training when a BitNet-trained neural network reaches a predefined performance that approaches that of a neural network trained with high bit precision.

[0305] There are several options for low-SWaP hardware and circuit implementation of BitNet-trained neural networks. For example, in multi-core processors, specific processors or hardware accelerators can be turned on when needed to support various DNN computational demands. For FPGAs, specific circuit configurations can be turned on (configured) when needed. In other architectures with stacked 3D memory, specific banks of memory can be utilized to provide high bandwidth for stored parameters. Other time / energy tradeoffs for data storage and movement can be configured to support rapid gradient evaluation (local and global). Additional circuitry (neuromorphic, analog, and memristor) can be used. Analog processors can choose different bit precisions to address manufacturing variations in advanced nanoelectronics.

[0306] In some examples, the computer system 100 enables autonomous changes in implementation optimized for SWaPC (size, weight, power, cost) and effectiveness (throughput, latency, and functional optimality). The BitNet DNN technology of the present disclosure enables lifelong learning with architectural evolution. In some examples, adjustments to both the system (hardware and software) and learning (learning rate, DNN depth, fused layers) may be made dynamically based on system requirements. The hardware may support both online learning and near-concurrent inference, supported by our DNN selection algorithm, which does not require training for convergence. That is, the computer system 100 (e.g., DNN 106) may deterministically select the appropriate neural network software architecture (using the selection algorithm described in this disclosure), configure the hardware appropriately, and continue operation without taking it offline for training. Much like a context switch from a software perspective, this disclosure describes a method for selecting / switching DNN contexts.

[0307] The capabilities provided by AI systems such as computer system 100 can be important for agile systems. New AI systems may simultaneously adapt to context from various types of data (e.g., EO, IR, radar, etc.). It is desirable for these platforms to support online training in areas where power and size budgets are limited. The techniques herein are particularly useful for network structure optimization for diverse multimodal data. Agile systems may use unsupervised and continuous learning techniques, and thus the techniques disclosed herein may continuously adapt to respond to changing environmental cues. Computer system 100 may address the problem of catastrophic learning loss by enabling rapid structure (e.g., neural network software architecture) selection and optimization using multimodal configurations of DNNs using the above-described techniques with dynamic parameters without training for full convergence, for example, as described in FIG. 12.

[0308] Such an AI system may be more resilient to adversarial attacks. For example, computer system 100 may prevent an adversary from learning the AI behavior of computer system 100 by allowing computer system 100 to self-reconfigure its learning technique (e.g., by using different hyperparameters of the loss function described in equation (16) to arrive at new sets of DNN parameters with different bit precisions and value ranges). From a cybersecurity perspective, such a learning technique may be resilient to adversarial targeting. An exemplary application example for cybersecurity is as follows. Causative attacks (e.g., attacks that are missed because an AI vulnerability is introduced during training and therefore grows over time) - An AI system can modify its learning method in a way that makes the AI vulnerability difficult to detect. An adversary attempts to cause an AI system to incorrectly classify some sets of inputs by manipulating the training data. For example, if an AI system is used to detect suspicious credit card transactions, an adversary planning to commit theft using certain types of credit card transactions may manipulate the training data so that the AI system does not recognize certain types of credit card transactions as suspicious. However, such an adversary is less likely to succeed if multiple versions of the neural network software architecture exist and if any version of the neural network software architecture is subject to deployed changes. Exploratory attacks (e.g., rule-based triggers can be inferred by sufficient sampling of system outputs, exploiting post-training vulnerabilities)—Our AI systems can be reconfigured to change the underlying response and reward functions (e.g., of reinforcement learning techniques) to make it difficult for adversaries to learn protective measures. For example, an adversary can predict how an AI system will classify input data by observing a sufficient number of system outputs. In this example, if an AI system is designed to detect suspicious credit card transactions, an adversary can identify specific types of credit card transactions that the AI system inaccurately classifies as innocuous. Thus, in this example, the adversary can use specific types of credit card transactions to initiate the commission of a crime. However, by modifying the response and reward functions (e.g., different versions of a neural network software architecture are deployed while changing hardware parameters), an adversary can make it significantly more difficult to identify sets of input data that the AI system misclassifies in a way that makes it attractive to the adversary. Evasion attacks (e.g., attack signals are below the detection threshold and evade through obfuscation) - Our AI can improve its detection capabilities to detect adversaries (e.g., by changing the location and complexity of honeypots). Poisoning attacks (e.g., attacks in which an adversary corrupts training data to weaken the distribution of input data, thereby resulting in misclassification)—AI systems may be resilient to catastrophic forgetting because the AI system does not require the entire DNN for retention. Instead, the AI system can reconfigure the neural network software architecture with new DNN layers to learn new training data without changing the previous distribution (e.g., substitute a trained DNN into the neural network software architecture). For example, computer system 100 may deploy a neural network software architecture with one or more DNNs that can continue to learn based on incoming training data. Because incoming training data may be intentionally corrupted to impair the neural network software architecture's ability to accurately classify legitimate input data, computer system 100 may retain and deploy a previously trained version of the same neural network software architecture if the deployed neural network software architecture begins misclassifying inputs due to a poisoning attack.

[0309] An enabler of the technology of this disclosure is for AI systems to support system adaptation to machine speed. In other words, the AI system may automatically change the DNN version to be effective in real time (e.g., based on hardware architecture parameters, cybersecurity risk indicators, etc.). This enables more effective detection, learning, and response (e.g., sensing, learning, and reacting) with low latency. Furthermore, this is important for autonomous systems and cybersecurity systems, where detection and prevention measures must be effective and rapid.

[0310] 13 is a flowchart illustrating an example operation of a machine learning system according to the techniques of this disclosure. In the example of FIG. 13, the machine learning system 104 may obtain system architecture parameters (1300). The system architecture parameters may include data describing processors of the system architecture. In some examples, the system architecture parameters include data describing communication links equivalent to the processors of the system architecture. Furthermore, in some examples, the system architecture parameters include data representing the current state of computational resources of the system architecture. In some examples, the machine learning system 104 may obtain the system architecture parameters from the machine learning system 104 (FIG. 1).

[0311] 13 , the machine learning system 104 may determine 1302 a neural network software architecture and a mapping of the DNN of the neural network software architecture to a processor of the system architecture based on the system architecture parameters. For example, the machine learning system 104 may use the DNN 106 to determine the neural network software architecture and the mapping based on the system architecture parameters. Thus, the mapping of the DNN of the neural network software architecture to the neural network software architecture and the processor of the system architecture is an output of the DNN 106. In this example, the machine learning system 104 may use a cost function during training of the DNN 106 to assign cost values to potential neural network software architectures and mappings of the DNN based on one or more of size, weight, power, and cost, as specified using the system architecture parameters.

[0312] To further validate the process of determining the neural network software architecture, the machine learning system 104 may use a pre-trained DNN of the neural network software architecture. In some examples, the machine learning system 104 may perform additional training iterations on the pre-trained DNN using different values of hyperparameters (e.g., λ1, λ2, λ3, etc.) of the loss function used to generate the pre-trained DNN to optimize the pre-trained DNN for the processor of the hardware architecture to which the DNN is mapped.

[0313] The machine learning system 104 may repeat operations 1300-1302 multiple times based on various system architecture parameters to determine multiple versions of the neural network software architecture represented by the system architecture parameters and various potential mappings of the DNN to processors of the system architecture.

[0314] In the example of Figure 13, a processor mapped to the DNN of the neural network software architecture may execute the DNN of the neural network software architecture (1304). In some examples, the processor may execute the DNN in an evaluation mode. In some examples, the processor may continue training the DNN.

[0315] The following paragraphs provide a non-limiting list of examples of the present disclosure.

[0316] Example 1. A computer system for training a deep neural network (DNN) to reduce computational resource requirements, comprising: a memory that stores a set of weights for the DNN, the DNN having a plurality of layers, and for each of the plurality of layers, the set of weights comprises layer weights and the set of bit-precision values comprises layer bit-precision values, the layer weights being represented in the memory using values having bit-precision equal to the layer bit-precision values, and the layer weights being associated with inputs to neurons of the layer; and a processing circuitry that executes a machine learning system configured to train the DNN, wherein training the DNN comprises optimizing the set of weights and the set of bit-precision values.

[0317] Example 2. The computer system of Example 1, wherein the machine learning system is configured to apply a backpropagation algorithm over multiple iterations as part of training the DNN, each iteration of the backpropagation algorithm updating the set of weights and optimizing the set of bit precision values.

[0318] Example 3. The computer system of example 1, wherein two or more of the layers of the DNN have different bit precision values from each other.

[0319] Example 4. The computer system of Example 1, wherein the set of weights is a first set of weights, the memory stores a second set of weights having a set of constant precision weights for each layer of the plurality of layers, each weight in the second set of weights having a bit-precision value equal to a predetermined bit-precision value, and the machine learning system is configured, as part of training the DNN, to perform a plurality of iterations to train the DNN, and as part of performing the plurality of iterations, the machine learning system is configured, for each iteration of the plurality of iterations, to use the second set of weights as input weights for neurons of the DNN to calculate a first output dataset based on a first input dataset, calculate a loss function, update the second set of weights based on the loss function, update the set of bit-precision values based on the loss function, and after updating the second set of weights and the set of bit-precision values, update the first set of weights based on the updated second set of weights and the updated set of bit-precision values, and the machine learning system is further configured to use the first set of weights as input weights for neurons of the DNN to calculate a second output dataset based on the second input dataset.

[0320] Example 5. A machine learning system may, as part of determining a loss function, determine a first operand that is an intermediate loss function, determine a second operand that is equal to the product of the value of a first hyperparameter and the sum of the quantization errors of each layer of a plurality of layers, and determine a second operand that is equal to the value of the second hyperparameter and

[0321]

number

[0322] Determine a third operand such that the third operand is equal to the product of i, where i is an index, N is the total number of layers, and b iis the bit-precision value of the ith layer of the plurality of layers, and the loss function is determined as a summation of the first operand, the second operand, and the third operand.

[0323] Example 6. The computer system of Example 5, wherein the machine learning system is further configured to, for each layer of the plurality of layers, determine a quantization error for the layer based on a difference between a layer weight from the first set of weights and a layer weight from the second set of weights.

[0324] Example 7. The first input dataset comprises a batch of training data-label pairs, and the machine learning system determines, as part of determining the first operand,

[0325]

number

[0326] Determine a first operand such that y is equal to B, where B is the total number of data label pairs in the batch of training data label pairs, each label in the batch of training data label pairs is an element of a label set having B labels, i is an index, log(·) is a logarithm function, N is the total number of layers, and y i is the i-th label in the set of labels,

[0327]

number

[0328] is the output of the n-th layer of the plurality of layers when the DNN is given as an input the ith data label pair of a batch of training data label pairs, and the data label pairs of the batch of training data label pairs are independent uniformly distributed data label pairs.

[0329] As part of updating the set of bit precision values, the machine learning system:

[0330]

number

[0331] determine an updated set of bit precision values to be equal to

[0332]

number

[0333] is the first set of weights,

[0334]

number

[0335] is the partial derivative of the loss function with respect to the set of bit-precision values, and sign(·) is a function that returns 0 when the absolute value of the argument of the function is less than a threshold, and returns the sign of the argument of the function otherwise.

[0336] Example 9. As part of updating the second set of weights, the machine learning system:

[0337]

number

[0338] where W is the second set of weights, μ is a learning rate, and

[0339]

number

[0340] is the first set of weights,

[0341]

number

[0342] is the partial derivative of the loss function with respect to the second set of weights.

[0343] Example 10. As part of updating a first set of weights, the machine learning system may determine for each layer of a plurality of layers that the updated optimal accuracy weights for each layer are

[0344]

number

[0345] where α is the minimum weight in the set of constant precision weights for the layer, W is the set of constant precision weights for the layer, δ is the total number of constant steps in the discretized range from the minimum weight in the set of constant precision weights for the layer to the maximum weight in the set of constant precision weights for the layer, and round( ) is a rounding function.

[0346] Example 11.δ is

[0347]

number

[0348] where β is the maximum weight of the set of constant precision weights of the layer, and b is the bit precision of the layer.

[0349] Example 12. For each of a plurality of layers, the machine learning system determines whether the updated first parameter of the layer is

[0350]

number

[0351] Determine the updated first parameter of the layer to be set equal to θ1, where θ1 is the first parameter of the layer, μ is the learning rate, and

[0352]

number

[0353] is the partial derivative of the loss function with respect to θ1, and the updated second parameter of the layer is

[0354]

number

[0355] Determine the updated second parameter of the layer to be set equal to θ1, where θ1 is the second parameter of the layer, μ is the learning rate, and

[0356]

number

[0357] is a partial derivative of the loss function with respect to θ, and is configured to determine a set of quantized values for the layer by rounding values produced by applying a quantization function θ+θlog|w|, where w represents a weight of a second set of weights associated with the layer, determine a maximum value of the set of quantized values for the layer and a minimum value of the set of quantized values for the layer, and set a bit precision value for the layer based on the base 2 logarithm of a range defined by the maximum value of the set of quantized values and the minimum value of the set of quantized values.

[0358] Example 13. The computer system of Example 4, wherein the machine learning system is configured such that, for each weight in the first set of weights, as part of updating the first set of weights, the machine learning system determines a weight in the first set of weights equal to a sign value multiplied by a power of 2 to an exponent value, where the sign value represents the sign of a corresponding weight in the second set of weights, and the exponent value is based on the logarithm base 2 of the corresponding weight in the second set of weights.

[0359] Example 14. The computer system of example 1, wherein each weight in the set of weights is equal to a power of two.

[0360] Example 15. The computer system of example 1, wherein each weight is represented in memory using an integer value having a bit precision equal to the bit precision value of the layer, an offset value, and a quantization step width value.

[0361] Example 16. The computer system of Example 1, wherein the machine learning system is configured to train the DNN based on system architecture parameters that describe the system architecture.

[0362] Example 17. The computer system of Example 1, wherein the machine learning system is configured to obtain system architecture parameters having data describing a processor of the system architecture, and determine, based on the system architecture parameters, a neural network software architecture having a DNN and a mapping of the DNN of the neural network software architecture to a processor of the system architecture, wherein the processor of the system architecture mapped to the DNN of the neural network software architecture is configured to execute the DNN of the neural network software architecture.

[0363] Example 18. A method of training a deep neural network (DNN) to reduce computational resource requirements, comprising: storing a set of weights for the DNN, the DNN having a plurality of layers, and for each of the plurality of layers, the set of weights comprising layer weights and the set of bit-precision values comprising layer bit-precision values, the layer weights being represented in memory using values having bit-precision equal to the layer bit-precision values, the layer weights being associated with inputs to neurons of the layer; and running a machine learning system configured to train the DNN, wherein training the DNN comprises optimizing the set of weights and the set of bit-precision values.

[0364] Example 19. The method of Example 18, wherein training the DNN comprises applying a backpropagation algorithm over multiple iterations, each iteration of the backpropagation algorithm updating the set of weights and optimizing the set of bit precision values.

[0365] Example 20. The method of example 18, wherein two or more layers of the DNN have different bit precision values from each other.

[0366] Example 21. The method of Example 18, further comprising: storing a second set of weights having a set of constant precision weights for each layer of the plurality of layers, each weight in the second set of weights having a bit-precision value equal to a predetermined bit-precision value; training the DNN further comprising: performing a plurality of iterations to train the DNN, performing the plurality of iterations comprising, for each iteration of the plurality of iterations, using the second set of weights as input weights for neurons of the DNN to calculate a first output dataset based on a first input dataset; calculating a loss function; updating the second set of weights based on the loss function; updating the set of bit-precision values based on the loss function; after updating the second set of weights and after updating the set of bit-precision values, updating the first set of weights based on the updated second set of weights and the updated set of bit-precision values; and using the first set of weights as input weights for neurons of the DNN to calculate a second output dataset based on a second input dataset.

[0367] Example 22. Determining a loss function includes determining a first operand that is an intermediate loss function, determining a second operand that is equal to the product of the value of the first hyperparameter and the sum of the quantization errors of each of the multiple layers, and determining a second hyperparameter that is equal to the product of the value of the second hyperparameter and the sum of the quantization errors of each of the multiple layers.

[0368]

number

[0369] determining a third operand such that the third operand is equal to the product of i, where i is an index, N is the total number of layers, and b i is the bit precision value of the ith layer of the plurality of layers; and determining the loss function as a sum of the first operand, the second operand, and the third operand.

[0370] Example 23. The method of Example 22, further comprising, for each layer of the plurality of layers, determining a quantization error for the layer based on a difference between a layer weight of the first set of weights and a layer weight of the second set of weights.

[0371] Example 24. The first input dataset comprises a batch of training data label pairs, and determining the first operand comprises:

[0372]

number

[0373] Determine a first operand such that y is equal to B, where B is the total number of data label pairs in the batch of training data label pairs, each label in the batch of training data label pairs is an element of a label set having B labels, i is an index, log(·) is a logarithm function, N is the total number of layers, and y i is the i-th label in the set of labels,

[0374]

number

[0375] is the output of the N-th layer of the plurality of layers when the DNN is given as input data for the i-th data label pair of the batch of training data label pairs, wherein the data label pairs of the batch of training data label pairs are independent, uniformly distributed data label pairs.

[0376] Example 25. Updating a set of bit precision values

[0377]

number

[0378] determine an updated set of bit precision values to be equal to

[0379]

number

[0380] is the first set of weights,

[0381]

number

[0382] is the partial derivative of the loss function with respect to the set of bit-precision values, and sign(·) is a function that returns 0 when the absolute value of the argument of the function is less than a threshold, and returns the sign of the argument of the function otherwise.

[0383] Example 26. Updating the second set of weights

[0384]

number

[0385] where W is the second set of weights, μ is a learning rate, and

[0386]

number

[0387] is the first set of weights,

[0388]

number

[0389] 22. The method of example 21, wherein:

[0390] Example 27. Updating a first set of weights is performed by a machine learning system for each layer of multiple layers, where the updated optimal accuracy weights for each layer are

[0391]

number

[0392] , where α is a minimum weight in the set of constant precision weights of the layer, W is the set of constant precision weights of the layer, δ is a total number of constant steps in a discretized range from the minimum weight in the set of constant precision weights of the layer to the maximum weight in the set of constant precision weights of the layer, and round( ) is a rounding function.

[0393] Example 28.δ is

[0394]

number

[0395] , β is the maximum weight of the set of constant precision weights of the layer, and b is the bit precision of the layer.

[0396] Example 29. Updating the set of bit precision values for each of a plurality of layers,

[0397]

number

[0398] Determine the updated first parameter of the layer to be set equal to θ1, where θ1 is the first parameter of the layer, μ is the learning rate, and

[0399]

number

[0400] is the partial derivative of the loss function with respect to θ1, and the updated second parameter of the layer is

[0401]

number

[0402] Determine the updated second parameter of the layer to be set equal to θ1, where θ1 is the second parameter of the layer, μ is the learning rate, and

[0403]

number

[0404] 2. The method of claim 21, comprising: |w|, where |w| is a partial derivative of the loss function with respect to θ; determining a set of quantized values for the layer by rounding values produced by applying a quantization function θ+θlog|w|, where w represents a weight of a second set of weights associated with the layer; determining a maximum value of the set of quantized values for the layer and a minimum value of the set of quantized values for the layer; and setting a bit precision value for the layer based on the base 2 logarithm of a range defined by the maximum value of the set of quantized values and the minimum value of the set of quantized values.

[0405] Example 30. The method of example 21, wherein updating the first set of weights comprises: for each weight in the first set of weights, determining a weight in the first set of weights equal to a sign value multiplied by a power of 2 exponent value, the sign value representing the sign of a corresponding weight in the second set of weights, and the exponent value based on the logarithm base 2 of the corresponding weight in the second set of weights.

[0406] Example 31. The method of example 18, wherein each weight in the set of weights is equal to a power of two.

[0407] Example 32. The method of example 18, wherein each weight is represented in memory by using an integer value having a bit precision, an offset value, and a quantization step width value equal to the bit precision value of the layer.

[0408] Example 33. The method of example 18, wherein the machine learning system is configured to train the DNN based on system architecture parameters that describe the system architecture.

[0409] Example 34. The method of Example 18, further comprising: obtaining system architecture parameters having data describing a processor of the system architecture; determining a neural network software architecture having a DNN and a mapping of the DNN of the neural network software architecture to a processor of the system architecture based on the system architecture parameters; and executing the DNN of the neural network software architecture by the processor of the system architecture that is mapped to the DNN of the neural network software architecture.

[0410] Example 35. A computer-readable data storage medium having stored thereon instructions that, when executed, cause one or more processors to: store a set of weights for a deep neural network (DNN), the DNN having a plurality of layers, and for each of the plurality of layers, the set of weights comprises a layer weight, and the set of bit-precision values comprises a layer bit-precision value, the layer weights being represented in memory using values having a bit-precision equal to the layer bit-precision value, the layer weights being associated with inputs to neurons of the layer; and execute a machine learning system configured to train the DNN, wherein training the DNN comprises optimizing the set of weights and the set of bit-precision values.

[0411] The techniques described in this disclosure may be implemented, at least in part, by hardware, software, firmware, and any combination thereof. For example, various aspects of the techniques described may be implemented in one or more processors, including one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or any other equivalent integrated or discrete logic circuitry, and any combination of such components. The term "processor" or "processing circuitry" generally refers to the aforementioned logic circuitry alone, a combination of the aforementioned logic circuitry with other logic circuitry, or any other equivalent circuitry. A controller comprising hardware may perform one or more of the techniques described in this disclosure.

[0412] Such hardware, software, and firmware may be implemented within the same circuitry or within separate circuits to support the various operations and functions described in this disclosure. Additionally, any of the described units, modules, or components may be implemented individually or together as separate but interoperable logic devices. The depiction of various features as modules or units is intended to emphasize various functional aspects and does not necessarily imply that such modules or units must be implemented by separate hardware or software components. Rather, functionality associated with one or more modules or units may be implemented by separate hardware or software components or integrated within common or separate hardware or software components.

[0413] The techniques described in this disclosure may be embodied in or encoded on a computer-readable medium, such as a computer-readable storage medium containing instructions. The instructions embodied in or encoded on the computer-readable storage medium may cause a programmable processor or other processor to perform a method, for example, when the instructions are executed. The computer-readable medium may include a computer-readable storage medium that corresponds to a tangible expression medium, such as a data storage medium, or a communication medium, including, for example, any medium that facilitates transfer of a computer program from one place to another according to a communication protocol. In this manner, computer-readable medium generally corresponds to (1) a non-transitory tangible expression computer-readable storage medium or (2) a communication medium, such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures of the techniques described in this disclosure. The computer readable storage medium may include random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), flash memory, hard disk, CD-ROM, floppy disk, cassette, magnetic medium, optical medium, or other computer readable medium.

[0414] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. 1. A computer system for training a deep neural network (DNN) to reduce computational resource requirements, comprising: a memory that stores a first set of weights of the DNN, a second set of weights of the DNN, and a set of bit precision values, the DNN having a plurality of layers, and for each layer of the plurality of layers, the first set of weights comprises a first weight of the layer, the second set of weights comprises a second weight of the layer, the set of bit precision values comprises a bit precision value of the layer, the first weight of the layer is represented in the memory using a value having a bit precision equal to the bit precision value of the layer, and the second weight of the layer is represented in the memory using a value having a bit precision equal to a predefined highest bit precision value, and the first weight of the layer and the second weight of the layer are associated with inputs to neurons of the layer; a processing circuit for executing a machine learning system configured to train the DNN, wherein training the DNN comprises updating the first set of weights, the second set of weights, and the set of bit precision values based on a gradient of a loss function; Equipped with the machine learning system updates the first set of weights based on the updated second set of weights and the updated bit precision value; the loss function is determined as a sum of a first operand, a second operand, and a third operand; the first computation target is an intermediate loss function, the second operand is equal to the product of a value of a first hyperparameter and a distillation loss representing the difference between the output produced by the DNN when the machine learning system runs the DNN on a first input data set using the first set of weights and the second set of weights; the third operand is equal to the product of the value of the second hyperparameter and the sum of terms corresponding to each layer of the plurality of layers, and for each layer of the plurality of layers, the term corresponding to each layer is equal to a power of 2 of the bit precision value for each layer; each weight in the first set of weights is equal to a power of two; The machine learning system updates the set of bit precision values for each of the plurality of layers as part of the update. determining updated parameters for the layer based on a gradient of the loss function with respect to parameters for the layer; determining a set of quantized values for the layer by rounding values produced by applying a quantization function based on the updated parameters for the layer to weights in the second set of weights associated with the layer; determining a maximum value of a set of quantized values for the layer and a minimum value of a set of quantized values for the layer; a computer system configured to set a bit precision value for the layer based on the base 2 logarithm of the maximum value of the set of quantized values minus the minimum value of the set of quantized values plus one.

2. 2. The computer system of claim 1, wherein two or more layers of the DNN have different bit precision values.

3. the second set of weights comprises a set of constant precision weights for each layer of the plurality of layers; The machine learning system is configured, as part of training the DNN, to perform a plurality of iterations for training the DNN, and, as part of performing the plurality of iterations, the machine learning system performs, for each iteration of the plurality of iterations: using the second set of weights as input weights for neurons of the DNN to calculate a first output data set based on a first input data set; Calculating the loss function; updating the second set of weights based on the loss function to determine updated second set of weights; updating the set of bit precision values; configured to update the first set of weights based on the updated second set of weights and the updated set of bit precision values after updating the second set of weights and the updated set of bit precision values; 2. The computer system of claim 1, wherein the machine learning system is further configured to use the first set of weights as input weights for neurons of the DNN to calculate a second output data set based on a second input data set.

4. For each layer of the plurality of layers, the parameters for the layer include a first parameter for the layer and a second parameter for the layer, and the updated parameters for the layer include an updated first parameter for the layer and an updated second parameter for the layer, and the machine learning system: The updated first parameters for the layer are [Equation 1] determining updated first parameters for the layer to be equal to θ 1 is the first parameter for the layer, μ is the learning rate, [Equation 2] is θ 1 is the partial derivative of the loss function with respect to The updated second parameter for the layer is [Equation 3] determining an updated second parameter for the layer to be equal to θ 2 is the second parameter for the layer, μ is the learning rate, [Equation 4] is θ 2 is the partial derivative of the loss function with respect to quantization function θ 1 +θ 2 4. The computer system of claim 3, configured to determine the set of quantized weights for the layer by rounding values produced by applying log2 |w|.

5. 2. The computer system of claim 1, wherein the machine learning system sets the bit precision value of the layer based on one plus a rounded value, the rounded value equal to the logarithm to the base 2 of one plus the maximum value of the set of quantized values minus the minimum value of the set of quantized values.

6. 4. The computer system of claim 3, wherein, for each weight in the first set of weights, as part of updating the first set of weights, the machine learning system is configured to determine a weight in the first set of weights equal to a sign value multiplied by a power of 2 to an exponent value, the sign value representing the sign of a corresponding weight in the second set of weights, and the exponent value being based on the logarithm base 2 of the corresponding weight in the second set of weights.

7. 2. The computer system of claim 1, wherein each weight in the first set of weights is represented in the memory by using an exponent value and a sign value.

8. The computer system of claim 1 , wherein the machine learning system is configured to train the DNN based on system architecture parameters that describe a system architecture.

9. The machine learning system includes: obtaining system architecture parameters having data describing a processor of the system architecture; configured to determine, based on the system architecture parameters, a neural network software architecture having the DNN and a mapping of the DNN of the neural network software architecture to the processors of the system architecture; 2. The computer system of claim 1, wherein the processor of the system architecture that is mapped to the DNN of the neural network software architecture is configured to execute the DNN of the neural network software architecture.

10. 1. A method for training a deep neural network (DNN) to reduce computational resource requirements, comprising: storing a first set of weights of the DNN, a second set of weights of the DNN, and a set of bit precision values, the DNN having a plurality of layers, and for each of the plurality of layers, the first set of weights comprises a first weight of the layer, the second set of weights comprises a second weight of the layer, the set of bit precision values comprises a bit precision value of the layer, the first weight of the layer being represented in memory using a value having a bit precision equal to the bit precision value of the layer, and the second weight of the layer being represented in memory using a value having a bit precision equal to a predefined highest bit precision value, and the first weight of the layer and the second weight of the layer being associated with inputs to neurons of the layer; running a machine learning system configured to train the DNN, wherein training the DNN comprises updating the first set of weights, the second set of weights, and the set of bit precision values based on a gradient of a loss function; Equipped with the machine learning system updates the first set of weights based on the updated second set of weights and the updated bit precision value; the loss function is determined as a sum of a first operand, a second operand, and a third operand; the first computation target is an intermediate loss function, the second operand is equal to the product of a value of a first hyperparameter and a distillation loss representing the difference between the output produced by the DNN when the machine learning system runs the DNN on a first input data set using the first set of weights and the second set of weights; the third operand is equal to the product of the value of the second hyperparameter and the sum of terms corresponding to each layer of the plurality of layers, and for each layer of the plurality of layers, the term corresponding to each layer is equal to a power of 2 of the bit precision value of each layer; each weight in the first set of weights is equal to a power of two; The machine learning system updates the set of bit precision values for each of the plurality of layers as part of the update. determining updated parameters for the layer based on a gradient of the loss function with respect to parameters for the layer; determining a set of quantized values for the layer by rounding values produced by applying a quantization function based on the updated parameters for the layer to weights in the second set of weights associated with the layer; determining a maximum value of a set of quantized values for the layer and a minimum value of a set of quantized values for the layer; 4. A method configured to set a bit precision value for the layer based on the base 2 logarithm of the maximum value of the set of quantized values minus the minimum value of the set of quantized values plus one.

11. The method of claim 10 , wherein two or more layers of the DNN have different bit precision values.

12. the second set of weights comprises a set of constant precision weights for each layer of the plurality of layers; Training the DNN further comprises performing a plurality of iterations for training the DNN, wherein performing the plurality of iterations includes, for each iteration of the plurality of iterations: using the second set of weights as input weights for neurons of the DNN to calculate a first output data set based on a first input data set; Calculating a loss function; and updating the second set of weights based on the loss function to determine updated second set of weights; updating the set of bit precision values based on the loss function; after updating the second set of weights and after updating the set of bit precision values, updating the first set of weights based on the updated second set of weights and the updated set of bit precision values; 11. The method of claim 10, further comprising using the first set of weights as input weights for neurons of the DNN to calculate a second output data set based on a second input data set.

13. For each layer of the plurality of layers, the parameters for the layer include a first parameter for the layer and a second parameter for the layer, and the updated parameters for the layer include an updated first parameter for the layer and an updated second parameter for the layer, and updating the set of bit precision values includes: The updated first parameters for the layer are [Equation 5] determining updated first parameters for the layer to be equal to θ 1 is the first parameter for the layer, μ is the learning rate, [Equation 6] is θ 1 It is the partial derivative of the loss function with respect to The updated second parameter for the layer is [Equation 7] determining an updated second parameter for the layer to be equal to θ 2 is the second parameter for the layer, μ is the learning rate, [Equation 8] is θ 2 It is the partial derivative of the loss function with respect to quantization function θ 1 +θ 2 determining a set of quantized weights for the layer by rounding the values produced by applying log2 |w|; The method of claim 12 comprising:

14. 11. The method of claim 10, wherein the machine learning system sets the bit precision value of the layer based on one plus a rounded value, the rounded value equal to the logarithm to the base 2 of one plus the maximum value of the set of quantized values minus the minimum value of the set of quantized values.

15. When executing, Storing in memory a first set of weights of a deep neural network (DNN), a second set of weights of the DNN, and a set of bit precision values, the DNN having a plurality of layers, and for each of the plurality of layers, the first set of weights comprises a first weight of the layer, the second set of weights comprises a second weight of the layer, the set of bit precision values comprises a bit precision value of the layer, the first weight of the layer is represented in the memory using a value having a bit precision equal to the bit precision value of the layer, and the second weight of the layer is represented in the memory using a value having a bit precision equal to a predefined highest bit precision value, and the first weight of the layer and the second weight of the layer are associated with inputs to neurons of the layer; running a machine learning system configured to train the DNN, wherein training the DNN comprises updating the first set of weights, the second set of weights, and the set of bit precision values based on a gradient of a loss function; storing instructions for execution by one or more processors; the machine learning system updates the first set of weights based on the updated second set of weights and the updated bit precision value; the loss function is determined as a sum of a first operand, a second operand, and a third operand; the first computation target is an intermediate loss function, the second operand is equal to the product of a value of a first hyperparameter and a distillation loss representing the difference between the output produced by the DNN when the machine learning system runs the DNN on a first input data set using the first set of weights and the second set of weights; the third operand is equal to the product of the value of the second hyperparameter and the sum of terms corresponding to each layer of the plurality of layers, and for each layer of the plurality of layers, the term corresponding to each layer is equal to a power of 2 of the bit precision value of each layer; each weight in the first set of weights is equal to a power of two; The machine learning system updates the set of bit precision values for each of the plurality of layers as part of the update. determining updated parameters for the layer based on a gradient of the loss function with respect to parameters for the layer; determining a set of quantized values for the layer by rounding values produced by applying a quantization function based on the updated parameters for the layer to weights in the second set of weights associated with the layer; determining a maximum value of a set of quantized values for the layer and a minimum value of a set of quantized values for the layer; 10. A computer-readable data storage medium configured to set a bit precision value for the layer based on the base 2 logarithm of the maximum value of the set of quantized values minus the minimum value of the set of quantized values plus one.

Images