Scaling learning for training DNNs

Abstract

Methods and apparatus for tuning hyperparameters of a neural network to compensate for noise, such as noise introduced by quantization of one or more parameters of the neural network, are disclosed. In some examples, tuning may include scaling the hyperparameters based on at least one metric representing the noise present in the neural network. The at least one metric may include the ratio of the noise signal to the weights of the neural network (such as edge weights and activation weights). In a quantized neural network, the learning rate hyperparameter used to compute gradient updates for layers during backpropagation may be scaled based on at least one metric. In some examples, the same scaled learning rate may be used when computing gradient updates for other layers.

Description

Background Technology

[0001] Machine learning (ML) and artificial intelligence (AI) technologies can be used to solve many complex computational problems, such as recognizing images and speech, analyzing and classifying information, and performing various classification tasks. Machine learning is a field of computer science that uses statistical techniques to enable computer systems to extract higher-level features from a set of training data. Specifically, features can be extracted by training models such as artificial neural networks or deep neural networks. Traditionally, deep neural networks have been trained and deployed using values ​​in single-precision floating-point formats (e.g., float32). Recent research has shown that lower-precision quantization formats (such as float16 or fixed-point) can be used for inference with an acceptable loss of accuracy. However, as precision decreases, error (also known as "noise") increases. Summary of the Invention

[0002] Methods and apparatus for compensating for quantization noise during training of a neural network implemented using a quantization-enabled system are disclosed. In some examples, a method for training a neural network includes acquiring a tensor comprising values ​​of one or more parameters of the neural network represented in a quantization precision format, and generating at least one metric (e.g., at least one noise signal metric) representing the quantization noise present in the tensor. For example, the parameters may include edge weights and activation weights of the neural network. The learning rate can then be scaled using the at least one metric for use during backpropagation phases of one or more subsequent training epochs of the neural network.

[0003] As used herein, a "noise-signal" metric refers to a quantitative relationship between a portion of the signal considered as "noise" (e.g., a signal representing parameter values) and the signal itself. For example, the quantization of the values ​​of parameters (e.g., activation weights or edge weights of a neural network) may introduce noise because the values ​​are represented in a lower-precision quantization format. In such an example, a noise-signal metric could include the ratio of the portion of the quantized value that constitutes noise to the value of the parameter before quantization.

[0004] Those skilled in the art who benefit from this disclosure will readily understand that, in the example where the noise-signal ratio is a ratio, it is not limited to the ratio of scalar noise to signal values. Rather, it can also include ratios where the numerator and denominator are not scalar values. For example, a noise-signal ratio metric could represent the ratio of a noise vector containing multiple noise values ​​to a signal vector containing multiple signal values ​​(e.g., the values ​​of parameters of a single layer in a neural network), where each noise value in the noise vector represents a corresponding signal portion in the signal vector considered as "noise," and each corresponding signal value in the signal vector represents the signal itself. As another example, a noise-signal ratio metric could represent the ratio of a noise matrix containing multiple noise values ​​to a signal matrix containing multiple signal values ​​(e.g., the values ​​of parameters of multiple layers in a neural network), where each noise value in the noise matrix represents a corresponding signal portion in the signal matrix considered as "noise," and each corresponding signal value in the signal matrix represents the signal itself. Thus, if the noise-signal ratio is conceived as a fraction, the numerator and denominator can be scalar values, vectors, or matrices. Alternatively, the numerator and denominator of the noise-signal ratio can take another form without departing from the scope of this disclosure. In another implementation, the noise signal metric has a form other than a ratio.

[0005] During the subsequent training phase of a quantized neural network, a scaling factor calculated based on at least one noise signal metric can be used to scale the learning rate used to compute gradient updates for the parameters of the neural network. Those skilled in the art who benefit from this disclosure will readily understand that by adjusting the hyperparameters of the neural network, such as the learning rate, based on the ratio of the quantization noise signal, errors arising during gradient update computation due to aggregated quantization noise can be mitigated. This noise compensation advantageously allows for the use of lower-precision computations when training the neural network, while still achieving accuracy similar to higher-precision computations. A certain amount of noise can be beneficial for training a neural network because it reduces the risk of the neural network overfitting the data. In fact, for each neural network, there may be an optimal amount of random fluctuation in the dynamics. However, when backpropagation is performed in a neural network with values ​​in a lower-precision quantization format, quantization noise from different layers aggregates. Due to this aggregation of noise, the error in the computation of gradient updates during backpropagation can reach unacceptable levels.

[0006] In some examples of the disclosed techniques, the hyperparameters of a neural network can be tuned to compensate for noise originating from sources other than quantization. For example, a method for compensating for noise during the training of a neural network may include calculating at least one noise-to-signal ratio representing the noise present in the neural network. The calculated noise-to-signal ratio can then be used to tune the hyperparameters of the neural network, such as the learning rate, learning rate scheduling, bias, stochastic gradient descent batch size, the number of neurons in the neural network, the number of layers in the neural network, etc. The tuned hyperparameters can then be used to train the neural network. For example, the tuned hyperparameters can be added as factors to the gradient update calculation during the backpropagation phase of a subsequent training period of the neural network. Therefore, techniques that introduce noise but improve the training efficiency of the neural network can be utilized without compromising the accuracy of the training results.

[0007] This "Summary" is provided to introduce some concepts in a simplified form, which will be further described in the "Detailed Description" below. This "Summary" is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0008] The foregoing and other objects, features and advantages of the disclosed subject matter will become more apparent from the following detailed description with reference to the accompanying drawings. Attached Figure Description

[0009] Figure 1 It is a block diagram of a quantization-enabled system that can be implemented in some examples of the disclosed technology.

[0010] Figure 2 It is a diagram depicting a deep neural network that can be modeled using some of the example methods and apparatuses disclosed in this paper.

[0011] Figure 3 This is a flowchart outlining example methods for scaling the learning rate used to train a quantized neural network, which can be performed in some examples of the disclosed techniques.

[0012] Figure 4 This is a flowchart outlining example methods for adjusting hyperparameters to compensate for noise during the training of a neural network, which can be implemented in some examples of the disclosed techniques.

[0013] Figure 5 This is a diagram illustrating an example computing environment in which some examples of the disclosed techniques can be implemented.

[0014] Figures 6-9 It is a graph illustrating experimental results that can be observed when performing certain examples of the disclosed techniques. Detailed Implementation

[0015] I. General Considerations The invention is set forth in the context of representative embodiments, which are not intended to be limiting in any way.

[0016] Unless the context clearly specifies otherwise, as used herein, the singular forms “a,” “an,” and “the” include the plural forms. Additionally, the term “include” means “comprise.” Furthermore, the term “couple” includes mechanical, electrical, magnetic, optical, and other practical means of coupling or linking articles together, and does not exclude the presence of intermediate elements between the coupled items. Moreover, as used herein, the term “and / or” refers to any one item or combination of items in the phrase.

[0017] The systems, methods, and apparatuses described herein should not be construed as limiting in any way. Rather, this disclosure relates individually and in various combinations and sub-combinations with each other to all novel and non-obvious features and aspects of the various disclosed embodiments. The disclosed systems, methods, and apparatuses are not limited to any particular aspect or feature or combination thereof, nor are the disclosed things and methods required to have any one or more particular advantages or problems solved. Furthermore, any feature or aspect of the disclosed embodiments may be used in various combinations and sub-combinations with each other.

[0018] Although some operations of the disclosed methods are described in a specific order for ease of presentation, it should be understood that this descriptive approach includes rearrangement unless the specific language described below requires a particular order. For example, in some cases, operations described sequentially may be rearranged or performed simultaneously. Furthermore, for simplicity, the accompanying drawings may not show the various ways in which the disclosed content and methods can be combined with other things and methods. Additionally, this specification sometimes uses terms such as “produce,” “generate,” “perform,” “select,” “receive,” “emit,” “verify,” and “convert” to describe the disclosed methods. These terms are high-level descriptions of the actual operations performed. The actual operations corresponding to these terms will vary depending on the specific order and will be readily discernible to those skilled in the art who benefit from this disclosure.

[0019] The operational theories, scientific principles, or other theoretical descriptions presented herein with reference to the apparatuses or methods of this disclosure have been provided for ease of understanding and are not intended to be limiting. The apparatuses and methods in the appended claims are not limited to those that operate in a manner described by such operational theories.

[0020] Any of the disclosed methods can be implemented as computer-executable instructions stored on one or more computer-readable media (e.g., computer-readable media such as one or more optical media disks, volatile memory components such as DRAM or SRAM, or non-volatile memory components such as hard disk drives) and executed on a computer (e.g., any commercial computer, including smartphones or other mobile devices that include computing hardware). Any computer-executable instructions used to implement the disclosed technology, and any data created and used during the implementation of the disclosed embodiments, can be stored on one or more computer-readable media (e.g., computer-readable storage media). The computer-executable instructions can be, for example, a dedicated software application or a software application or other software application (such as a remote computing application) accessed or downloaded via a web browser. Such software can be executed, for example, on a single local computer (e.g., on a general-purpose and / or dedicated processor executing on any suitable commercial computer), or in a networked environment using one or more networked computers (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network), or other such networks).

[0021] For clarity, only selected aspects of the software-based implementation are described. Other details well-known in the art are omitted. For example, it should be understood that the disclosed techniques are not limited to any particular computer language or program. For instance, the disclosed techniques can be implemented by software written in C, C++, Java, or any other suitable programming language. Similarly, the disclosed techniques are not limited to any particular computer or hardware type. Certain details of suitable computers and hardware are well-known and do not need to be elaborated in this disclosure.

[0022] Furthermore, any software-based implementation (including, for example, computer-executable instructions for causing a computer to perform any of the disclosed methods) can be uploaded, downloaded, or remotely accessed via suitable communication devices. Such suitable communication devices include, for example, the Internet, the World Wide Web, intranets, software applications, cable (including fiber optic cables), magnetic communication, electromagnetic communication (including RF, microwave, and infrared communication), electronic communication, or other such communication devices.

[0023] II. Introduction to Neural Networks and Quantization Formats Artificial neural networks (ANNs, or "NNs" throughout this article) are used in many applications of artificial intelligence and machine learning, including image recognition, speech recognition, search engines, and other suitable applications. Processing in these applications can be performed on individual devices such as personal computers or mobile phones, but also in large data centers. Meanwhile, hardware accelerators that can be used with NNs include dedicated NN processing units, such as Tensor Processing Units (TPUs) and Field-Programmable Gate Arrays (FPGAs) programmed to accelerate neural network processing. Due to their flexibility and low power consumption per unit of computation, such hardware is being deployed in consumer devices and data centers.

[0024] Traditionally, neural networks (NNs) have been trained and deployed using single-precision floating-point (32-bit floating-point or float32 format). However, it has been shown that inference operations with minimal accuracy loss can be performed using lower-precision floating-point formats (such as 16-bit floating-point (float16)) or fixed-point. On dedicated hardware such as FPGAs, reduced-precision formats can significantly improve the latency and throughput of DNN processing.

[0025] Numbers represented in ordinary precision floating-point format (e.g., floating-point numbers represented in 16-bit, 32-bit, 64-bit, or 80-bit floating-point format (also referred to herein as ordinary precision floating-point format)) can be converted to quantized precision format numbers, thereby gaining performance advantages when performing operations. In particular, NN weights and activation values ​​can be represented in a lower precision quantization format, where an acceptable level of error is introduced. Examples of lower precision quantization formats include formats with reduced bit width (including by reducing the number of bits used to represent the mantissa or exponent) and block floating-point formats, in which two or more numbers share the same single exponent.

[0026] One characteristic of computation on FPGA devices is their typically lack of hardware floating-point support. While floating-point operations can be performed with penalties using flexible logic, the amount of logic required to support floating-point in FPGA implementations is often prohibitively large. Some newer FPGAs have been developed that support floating-point computation, but even on these FPGAs, the same device produces twice the computational output per unit time compared to when used in integer mode. Typically, neural networks (NNs) are created with floating-point computation in mind, but when using FPGAs for NN processing, it would be beneficial if neural networks could be represented using integer arithmetic. Examples of disclosed techniques include hardware implementations of block floating-point (BFP), including the use of BFP in NN, FPGA, and other hardware environments.

[0027] Neural network operations are used in many artificial intelligence operations. Typically, much of the processing performed when implementing a neural network involves matrix-matrix or matrix-vector multiplication. Such operations are computationally and memory bandwidth intensive, where the matrices can be, for example, 1000×1000 elements (e.g., 1000×1000 numbers, each including a sign, mantissa, and exponent) or larger, and many matrices are used. As discussed in this paper, BFP techniques can be applied to such operations to reduce the computational and memory bandwidth requirements of a given system, whether it is an FPGA, CPU, or other hardware platform. As used herein, the term "element" refers to a member of such a matrix or vector.

[0028] As used herein, the term "tensor" refers to a multidimensional array matrix that can be used to represent the properties of a neural network, and includes one-dimensional vectors as well as two-dimensional, three-dimensional, four-dimensional, or larger dimensions. As used in this disclosure, unless otherwise specified, tensors do not require any other mathematical properties.

[0029] As used herein, the term "normal precision floating-point" refers to a floating-point number format that has a mantissa, exponent, and optional sign and is natively supported by a native or virtual CPU. Examples of normal precision floating-point formats include, but are not limited to, IEEE 754 standard formats (such as 16-bit, 32-bit, and 64-bit) or other processor formats supported by the processor (such as Intel AVX, AVX2, IA32, and x86_64 80-bit floating-point formats).

[0030] As used herein, the term "quantized precision floating-point" refers to a floating-point format in which two or more values ​​of a tensor have been modified to simulate neural network hardware. Specifically, many examples of quantized precision floating-point representations include block floating-point formats, where a common exponent is referenced to represent two or more values ​​of the tensor. Quantized precision floating-point numbers can be generated by selecting a common exponent for two, more, or all elements of a tensor and shifting the mantissa of individual elements to match the shared common exponent. In some examples, the grouping of elements within a tensor can share a common exponent on, for example, per row, per column, per block, or other basis.

[0031] III. Introduction to the disclosed technology Figure 1 This is a block diagram outlining some examples of an exemplary quantization-enabled system 110 that can implement the disclosed techniques. Figure 1As shown, the quantization-enabled system 110 may include multiple hardware resources, including a general-purpose processor 120 and a dedicated processor 122 such as a graphics processing unit. The processors are coupled to memory 125 and storage device 127, which may include volatile or non-volatile memory devices. Processors 120 and 122 execute instructions stored in the memory or storage device to provide a general-precision neural network module 130. The general-precision neural network module 130 includes a software interface that allows the system to be programmed to implement various types of neural networks. For example, software functions that allow applications to define neural networks, including weights, activation values, and interconnections between layers of the neural network, may be provided. The general-precision neural network module 130 may also provide utilities that allow training and retraining of neural networks implemented using the module. The values ​​of the neural network module are stored in memory or storage device and manipulated by instructions executed by one of the processors.

[0032] In some examples, proprietary or open-source libraries or frameworks are provided to programmers to create, train, and evaluate neural networks. Examples of such libraries include TensorFlow, Microsoft Cognitive Toolkit (CNTK), Caffe, Theano, and Keras. In other examples, programming tools such as integrated development environments (IDEs) support programmers and users in defining, compiling, and evaluating neural networks.

[0033] The quantized-enabled system 110 also includes a quantization domain 140. The quantization domain 140 provides functionality for converting data represented in full-precision floating-point format in the normal-precision neural network module 130 into quantized values. In some examples, the quantization domain is implemented as a software simulator that models NN operations performed in quantized format. In some examples, the quantization domain includes a hardware accelerator that can be used to accelerate inference and / or training operations in quantized-precision digital format. In some examples, the conversion with the quantization domain 140 can be performed in the normal-precision software domain or by additional hardware within the quantization domain (such as...). Figure 1 (As shown). This feature will be discussed in more detail below.

[0034] The standard-precision neural network module 130 can be used to specify, train, and evaluate neural network models using a toolflow that includes a hardware-independent modeling framework 131 (also known as a native framework or machine learning execution engine), a neural network compiler 132, and a neural network runtime environment 133. The memory includes computer-executable instructions for the toolflow, including the modeling framework 131, the neural network compiler 132, and the neural network runtime environment 133. The toolflow can be used to generate neural network data 200, which represents all or part of the neural network model, such as the following regarding… Figure 2The neural network model discussed. It should be noted that although the toolflow is described as having three separate tools (131, 132, and 133), in various examples, the toolflow may have fewer or more tools. For example, the functionality of different tools (131, 132, and 133) can be combined into a single modeling and execution environment.

[0035] Neural network data 200 can be stored in memory 125. Neural network data 200 can be represented in one or more formats. For example, neural network data 200 corresponding to a given neural network model can have a different format associated with each corresponding tool in the toolflow. Typically, neural network data 200 can include descriptions of nodes, edges, groups, weights, biases, activation functions, and / or tensor values. As a specific example, neural network data 200 can include source code, executable code, metadata, configuration data, data structures, and / or files representing the neural network model.

[0036] Modeling framework 131 can be used to define and use neural network models. As an example, modeling framework 131 may include predefined APIs and / or programming primitives that can be used to specify one or more aspects of a neural network model. Predefined APIs may include both lower-level APIs (e.g., activation functions, cost or error functions, nodes, edges, and tensors) and higher-level APIs (e.g., layers, convolutional neural networks, recurrent neural networks, linear classifiers, etc.). "Source code" can be used as input to modeling framework 131 to define the topology of the graph for a given neural network model. Specifically, the APIs of modeling framework 131 can be instantiated and interconnected within the source code to specify complex neural network models. Data scientists can create different neural network models by using different APIs, different numbers of APIs, and APIs interconnected in different ways.

[0037] In addition to source code, memory 125 can also store training data. The training data includes a set of input data applied to neural network model 200 and the expected output from the neural network model for each corresponding dataset of input data. Modeling framework 131 can be used to train the neural network model using the training data. The output of the training is the weights and biases associated with each node of the neural network model. After training the neural network model, modeling framework 131 can be used to classify new data applied to the trained neural network model. Specifically, the trained neural network model uses the trained weights and biases to perform classification and recognition tasks on data not used to train the neural network model. Modeling framework 131 typically uses only CPU 120 to execute the neural network model, therefore real-time performance may not be achievable for some classification tasks. Modeling framework 131 can also support the use of GPU 122 to execute the neural network model, but performance may still not reach real-time levels.

[0038] Compiler 132 analyzes the source code and data provided to the neural network model (e.g., weights and biases learned from the trained model) and converts the model into a format that can be accelerated in quantization domain 140 and / or optional neural network accelerator 180, as will be described in further detail below. Specifically, compiler 132 converts the source code into executable code, metadata, configuration data, and / or data structures to represent the neural network model and memory as neural network data 200. In some examples, compiler 132 may divide the neural network model into parts that use CPU 120 and / or GPU 122 (e.g., neural network 200) and other parts that can be executed on neural network accelerator 180 (e.g., neural network subgraphs). Compiler 132 may generate executable code (e.g., runtime modules) to execute the subgraph assigned to CPU 120 and communicate with the subgraph assigned to optional accelerator 180. Compiler 132 may generate configuration data for accelerator 180 to configure accelerator resources for evaluating the subgraph assigned to optional accelerator 180. Compiler 132 can create data structures for storing values ​​generated by the neural network model during execution and / or tracking and for communication between CPU 120 and accelerator 180. Compiler 132 can generate metadata that can be used to identify subgraphs, edge groups, training data, and various other information about the neural network model during runtime. For example, the metadata may include information for docking between different subgraphs of the neural network model.

[0039] Runtime environment 133 provides an executable environment or interpreter that can be used to train a neural network model during training mode and to evaluate the neural network model in training, inference, or classification modes. During inference mode, input data can be applied to the neural network model input, and the input data can be classified based on the training of the neural network model. The input data can be archived data or real-time data.

[0040] The runtime environment 133 may include deployment tools that, during deployment mode, can be used to deploy or install all or part of the neural network onto the quantization domain 140. The runtime environment 133 may also include a scheduler that manages the execution of different runtime modules and communication between the runtime modules and the quantization domain 140. Therefore, the runtime environment 133 can be used to control the data flow between nodes modeled on the normal-precision neural network module 130 and the quantization domain 140.

[0041] The quantization field 140 receives a normal-precision value 150 from the normal-precision neural network module 130. The normal-precision value can be represented in 16-bit, 32-bit, 64-bit, or other suitable floating-point format. For example, it can receive values ​​representing a portion of the neural network, including edge weights, activation values, or other suitable quantization parameters. The normal-precision value 150 is provided to a normal-precision floating-point to quantization floating-point converter 152, which converts the normal-precision value to a quantized value. A quantization floating-point operation 154 can then be performed on the quantized value. The quantized value can then be converted back to normal-precision floating-point format using the quantization floating-point to normal-precision floating-point converter that produced the normal-precision floating-point value.

[0042] The conversions between ordinary floating-point and quantized floating-point numbers performed by converters 152 and 156 are typically performed on a set of numbers represented as vectors or multidimensional matrices. In some examples, additional ordinary-precision operations 158 (including operations that may be necessary in a particular neural network implementation) may be performed based on the ordinary-precision format, including adding biases to one or more nodes of the neural network, applying a hyperbolic tangent function or other such sigmoid function, or a rectification function (e.g., ReLU operation) to the ordinary-precision values ​​converted back from the quantized floating-point format.

[0043] In some examples, the quantized values ​​are actually stored in memory as ordinary floating-point values. In other words, the quantization field 140 quantizes the inputs, weights, and activations of a neural network model, but the underlying operations are performed in regular floating-point. In other examples, the quantization field provides a full simulation of the quantization, including storing only a copy of the shared exponent and operating with a reduced mantissa width. Some results may differ in the version that performs the underlying operations in ordinary floating-point. For example, the full simulation version can examine underflow or overflow cases with a limited quantization bit width (e.g., a 3, 4, or 5-bit mantissa).

[0044] The majority of the computational cost of a DNN is matrix-vector and matrix-matrix multiplication. These operations are quadratic in terms of the input size, while operations such as bias addition and activation functions are linear in terms of the input size. Therefore, in some examples, quantization is only applied to matrix-vector multiplication operations, which will ultimately be implemented on NN hardware accelerators such as TPUs or FPGAs. In such examples, all other operations are performed in a normal-precision format, such as float16. Thus, from the user's or programmer's perspective, a quantized-enabled system 110 accepts and outputs normal-precision float16 values ​​to / from a normal-precision neural network module 130, and outputs float16 formatted values. All conversions between this and block floating-point formats can be hidden from the programmer or user. In some examples, the programmer or user can specify certain parameters for the quantization operation. In other examples, the quantization operation can utilize the block floating-point format to reduce computational complexity.

[0045] In some examples, an optional neural network accelerator 180 is used to accelerate the evaluation and / or training of neural network subgraphs, typically with increased speed and reduced latency, which would not be achieved when evaluating the subgraph only in the quantization domain 140. In the illustrated example, the accelerator includes a tensor processing unit 182 and / or a reconfigurable logic device 184 (e.g., contained within one or more FPGAs or programmable circuit structures), but any suitable hardware accelerator can be used to simulate the neural network. Accelerator 180 may include configuration logic that provides a soft CPU. The soft CPU supervises the operation of the accelerated subgraphs on accelerator 180 and can manage communication with the normal-precision neural network module 130 and / or the quantization domain 140. The soft CPU can also be used for configuration logic and to control the loading and storage of data from RAM on the accelerator, such as in Block RAM within the FPGA.

[0046] In some examples, quantization domain 140 is used to prototype all or part of the training, inference, or classification of neural network model 200. For example, quantization parameters can be selected based on accuracy or performance results obtained by prototyping the network within quantization domain 140. After selecting a desired set of quantization parameters, the quantized model can be programmed into accelerator 180 to perform further operations. In some examples, the final quantized model implemented using quantization domain 140 is the same as the quantized model to be programmed into accelerator 180. In other examples, the model programmed into the accelerator may differ in some respects.

[0047] The compiler 132 and runtime 133 provide a fast interface between the normal-precision neural network module 130, the quantization domain 140, and the (optional) accelerator 180. In practice, the user of the neural network model may not be aware that a portion of the model is being accelerated on the provided accelerator. For example, node values ​​are typically propagated throughout the model by writing tensor values ​​to a data structure that includes identifiers. The runtime 133 associates subgraph identifiers with the accelerator and provides the accelerator with logic for transforming messages to transparently write the values ​​of weights, biases, and / or tensors to the quantization domain 140 and / or (optional) accelerator 180 without program intervention. Similarly, the values ​​output by the quantization domain 140 and (optional) accelerator 180 can be transparently sent back to the normal-precision neural network module 130 with a message including the identifier of the receiving node at the server and a payload containing the values ​​of weights, biases, and / or tensors sent back to the entire neural network model.

[0048] IV. Example Deep Neural Network Topology Figure 2A simplified topology of a deep neural network (DNN) 200 that can be used to perform enhanced image processing is shown. One or more processing layers can be implemented using quantization and BFP matrix / vector operations, including one or more of the multiple (210) neural network cores in the quantization-enabled system 110 described above. It should be noted that the application of the neural network implementations disclosed herein is not limited to DNNs, but can also be used with other types of neural networks, such as convolutional neural networks (CNNs) (including implementations with long short-term memory (LSTM) or gated recurrent units (GRUs)) or other suitable artificial neural networks that can be adapted to use the BFP methods and apparatus disclosed herein.

[0049] like Figure 2 As shown, a first set of nodes 210 (including nodes 215 and 216) forms the input layer. Each node in set 210 is connected to each node in a first hidden layer formed by a second set of nodes 220 (including nodes 225 and 226). The second hidden layer is formed by a third set of nodes 230, including node 235. The output layer is formed by a fourth set of nodes 240 (including node 245). In Example 200, the nodes of a given layer are fully interconnected with the nodes of its neighboring layers. In other words, a layer may include nodes that share a common input with other nodes in that layer and / or provide outputs to a common destination for other nodes in that layer. In other examples, a layer may include nodes that share a common input with other nodes in that layer and / or provide outputs to a subset of the common destination for other nodes in that layer.

[0050] Each node produces an output by applying weights to each input generated from the previous node and collecting the weights to produce an output value. In some examples, each individual node may have an activation function and / or bias applied. For example, any properly programmed processor or FPGA can be configured to implement the nodes in the depicted neural network 200. In some example neural networks, the activation function f() of the hidden combined node n can produce an output mathematically expressed as follows:

[0051] Where wi is the weight applied (multiplied) to the input edge xi, plus the bias value bi. In some examples, the activation function produces a continuous value (represented as a floating-point number) between 0 and 1. In some examples, the activation function produces a binary 1 or 0 value, depending on whether the sum is above or below a threshold.

[0052] Neural networks can be trained and retrained by adjusting the component values ​​of the activation function. For example, by adjusting the weights *wi* or biases *bi* of nodes, the behavior of the neural network is adjusted through corresponding changes in the network's output tensor values. For instance, the cost function *C(w, b)* can be used to find suitable weights and biases for the network and is mathematically described as:

[0053] Where w and b represent all weights and biases, n is the number of training inputs, and a is a vector of the network's output values ​​from the input vector for the training input x. By adjusting the network weights and biases, various search techniques (e.g., stochastic gradient descent) can be used to drive the cost function C toward a target value (e.g., zero (0)).

[0054] In techniques such as stochastic gradient descent, various parameters can be tuned to adjust the performance of the neural network (NN) during training. These parameters, referred to as "hyperparameters" in this paper, include the learning rate parameter, which affects the rate at which the cost function C is driven to the target value. As discussed further below, hyperparameters such as the learning rate can be tuned to compensate for noise introduced by the quantization of the NN parameters. Such tuning can result in the same or better training accuracy for quantized NNs compared to unquantized NNs. Furthermore, such tuning can cause the cost function C to converge to the target value more quickly (e.g., after fewer training epochs).

[0055] Based on certain aspects of the disclosed techniques, the performance of neural network (NN) training and inference can be improved. For example, by using certain disclosed examples of adjusting the learning rate based on at least one noise signal metric, training of a quantized NN can be achieved faster with less memory and / or higher accuracy, depending on the specific example, despite the noise introduced by quantization. In particular, the duration of any particular training period can be reduced by decreasing the time spent on training, including backpropagation. Furthermore, the number of training periods can be reduced by using certain disclosed examples of adjusting the learning rate.

[0056] Examples of suitable applications for such neural network BFP implementations include, but are not limited to: performing image recognition, performing speech recognition, classifying images, converting speech to text and / or other languages, facial or other biometrics, natural language processing, automatic language conversion, query processing in search engines, automatic content selection, analyzing emails and other electronic documents, relationship management, biomedical informatics, identifying candidate biomolecules, providing suggestions or other classification and artificial intelligence tasks.

[0057] In some examples, a set of parallel multiply-accumulate (MAC) units within each convolutional layer can be used to accelerate computation. Similarly, parallel multiplier units can be used in fully connected and dense matrix multiplication stages. A set of parallel classifiers can also be used. This parallelization approach can further accelerate computation at the cost of increased control complexity.

[0058] Those skilled in the art who benefit from this disclosure will readily understand that the application of the neural network implementation can be used to utilize different aspects of the neural network, either alone or in combination or sub-combination with other neural networks. For example, the disclosed implementation can be used to train a neural network via gradient descent and / or the backpropagation operation of the neural network.

[0059] V. Example methods for scaling the learning rate Figure 3 This is a flowchart that outlines example methods for scaling the learning rate to train a neural network (e.g., a DNN) in a quantization-enabled system, as can be used in some examples of the disclosed techniques. For example, Figure 1 The system can be used to combine Figure 2 The DNN topology shown is used to implement the method shown.

[0060] At procedure block 310, obtain a first tensor having one or more NN parameter values ​​represented in normal precision floating-point format. The first tensor may include values ​​of one or more or all parameters of one or more layers of the NN. For example, this may include values ​​of activation weights, edge weights, etc. For example, the first tensor may be in matrix form.

[0061] At procedure block 320, a second tensor of the neural network is obtained. This second tensor has the same values ​​as the first tensor, but with values ​​represented in a quantization precision format, which introduces noise. In some examples, the second tensor is obtained by converting (e.g., by a processor) the values ​​of the first tensor to a quantization precision format. The quantization precision format can be a format where the bit width representing the exponent or mantissa is adjusted relative to a normal precision floating-point format. Alternatively, the quantization precision format can be a block floating-point format. The same quantization precision format can be used for all parameters of the neural network. However, in other examples, different quantization precision formats can be used for different parameters within the neural network.

[0062] At process block 330, at least one noise signal metric is generated for the NN. For example, at least one noise signal metric may include one or more noise-to-signal ratios. In such an example, as further discussed below in Section VI, the quantized noise-to-signal ratio of the activation weights X of the l-th layer of the NN can be calculated in the following manner. First, calculate the difference between the (quantized) activation weights of the second tensor and the (unquantized) activation weights of the first tensor. ,in This represents the quantization noise in the quantization activation weights, and then the difference... Divide by the absolute value of the activation weights of the first tensor. The activation weights X of the l-th layer can be represented as a vector, in which case, the difference... Sum and ratio It can also be represented as a vector. Furthermore, the quantization-to-noise ratio of the edge weights w in each of multiple layers k can be calculated in the following way: : Calculate the difference between the (quantized) edge weights of the second tensor and the (unquantized) edge weights of the first tensor. ,in This represents the quantization noise in the quantized edge weights, and the difference is divided by the absolute value of the edge weights in the first tensor. The edge weights of the k-th layer. It can be represented as a matrix, in which case, the difference Sum and ratio It can also be represented as a matrix. In some examples, at least one noise signal metric includes the quantized noise ratio of the layers after the l-th layer (e.g., the l+1-th layer) and all other layers of the NN after the l+1-th layer.

[0063] Without departing from the scope of this disclosure, other noise signal metrics may also be calculated at process block 330. For example, the noise-to-signal ratio of any quantization parameter of the NN or any vector or matrix of the quantization parameters of the NN may be calculated.

[0064] At process block 340, a scaling factor is calculated based on at least one noise signal metric. For example, as further discussed below in Section VI, the scaling factor g can be calculated in the context of the DNN using the following equation: , in This represents the average noise-to-signal ratio vector of the l-th layer of the NNN with respect to batch size and tensor elements. This represents the average noise-to-signal ratio of each sample across the tensor elements. This represents the average value of layers l+1 to L of the neural network (e.g., all layers in the neural network after layer l). The sum of these. This formula takes into account a first-order approximation of the noise-to-signal ratio and can be modified as needed to include higher-order noise levels. Alternatively, the scaling factor g can be calculated in the context of an RNN via different equations (e.g., equation (34) as illustrated below in Section VII).

[0065] At procedure block 350, the learning rate of the neural network (NN) is scaled using the scaling factor calculated at procedure block 340. The scaled learning rate can be a predetermined learning rate for the NN (e.g., a "global" learning rate used to compute gradient updates for all layers of the NN during the backpropagation phase of training). For example, as discussed further below in Section VI, the scaled learning rate can be calculated as the product of the scaling factor and the predetermined learning rate of the neural network. The scaled learning rate can vary for each layer of the neural network.

[0066] At process box 360, the neural network (NN) is trained. This may include performing one or more training periods. In some examples, training may continue until convergence of the NN's output is achieved. Stochastic gradient descent training is one example of a suitable technique that can be used to train the NN; however, other techniques may be used to train the NN without departing from the scope of this disclosure.

[0067] As shown in the figure, training a quantized neural network at process box 360 involves determining gradient updates using a scaled learning rate (e.g., as determined at process box 350). Gradient updates can be applied to one or more parameters (e.g., weights) of one or more layers of the neural network. In some examples, the scaling factor computed at 340 is for a single layer of the neural network and is used to determine gradient updates only for the parameters of that layer. However, in other examples, the scaling factor computed for a single layer may also be used to determine gradient updates for the parameters of other layers (e.g., for each layer of the neural network).

[0068] Alternatively, the scaled learning rate can be used to determine gradient updates only for parameters of a neural network whose values ​​have the same quantization precision format as those used to compute at least one noise signal metric at process block 330. Experimental results show that the variance of the noise signal ratio computed according to this disclosure is lower for parameters with the same quantization precision format (e.g., the same bit width). Therefore, scaling factors computed according to this disclosure for different quantization precision formats and neural network architectures / topologies can be stored in memory, such as in a lookup table. In such an example, determining gradient updates for a given neural network parameter may include accessing an entry in the lookup table corresponding to a specific quantization precision format (e.g., bit width) for that parameter or the parameters of that layer to obtain a scaling factor previously determined for that format.

[0069] Scaling the learning rate used to determine gradient updates during the training of a neural network (NN) can advantageously improve the accuracy of the training results. For example, experimental results discussed below in Section X show that the accuracy of training a quantized NN can be improved to match or even exceed the accuracy achieved when training an equivalent non-quantized NN. Therefore, according to Figure 3The method of scaling the learning rate can facilitate the use of lower precision quantization formats when training the neural network, and thus improve the efficiency of the hardware used to implement the training.

[0070] VI. Calculate the adjusted learning rate of the DNN This section will describe the theoretical basis of the computations discussed in Section V above for a DNN model with L layers. In such a model, the gradient update of the l-th layer using stochastic gradient descent can be expressed as follows: (1) in Where N is the learning rate, and N is the total size of the training dataset. It is the actual / true gradient update of the l-th layer relative to the quantization weights. This is a gradient update relative to the estimated quantized weights, evaluated on a mini-batch of size B. Specifically, ,and (2) Assuming the underlying DNN model is designed based on the Corrected Linear Unit (ReLU) as a non-linearity, the gradient update value of the hidden layer l of the DNN can be calculated as follows: (3)

[0071] here, It is the activation vector of input sample i in the l-th layer. This is noise introduced into the activation vector due to quantization. This represents the output of the DNN after the Softmax layer of input sample i. This represents the net value in the last layer before the Softmax layer. (Symbol) This represents the weight matrix of the j-th layer. It is its corresponding quantization noise.

[0072] same: (4) Given gradient error Defined as The difference between them, so the mean and variance of the gradient error can be calculated as: (5) The first step in equation (5) is derived using the linear properties of the desired operation. The variance of the gradient noise is: (6) Each term in equation (6) can be calculated as follows: (7) The terms in equation (7) It can be ignored because it is and The product of values ​​and several orders of magnitude smaller than other terms. In equation (7), the quantized noise signal is the product of the vector and matrix ( and It is approximated by its corresponding expected value. Therefore: (8) value This represents the average quantization noise ratio in the l-th activation layer of input sample i. This value is generally less than 1 and can be replaced by the expected value on the sample; if the encoding method is well chosen, the quantization noise ratio of different data samples i will vary. The variance of [the variable] is much smaller than the variance of the other terms in the quantization coefficients in equation (8). Therefore, (9) (10) The matrix describing the average gradient covariance can be represented as: It is a function of the current parameter / weight values. Specifically, (11) Therefore, we obtain the following formula: (12) Using the central limit theorem and Gaussian random noise to account for gradient error To model, in equation (6) Equivalent to: (13) Equation (13) is an approximation, where given And given two partial sums (YB and YN) for independent random variables, where B <N), Therefore, the variance of the gradient noise in equation (6) is equivalent to: (14) Continuing, interpreting equation (1) as a discrete update of a stochastic differential equation yields: (15) Where t is a continuous variable. This represents gradient noise with an expected value of 0, and The constant g (instead referred to as the scaling factor in this paper) controls the scale of random fluctuations in the dynamics.

[0073] Equations 1 and 15 are related to each other because To keep the scaling factor g constant, the variance in this gradient update can be equal to the variance in equation (1), as follows: (16) (17) As long as the denominator remains positive, equation (17) holds true.

[0074] VII. Calculate the adjusted learning rate of the recurrent neural network. The theoretical basis of the computations discussed in Part V above will now be described in the context of recurrent neural networks (RNNs). An RNN is a type of neural network composed of long short-term memory (LSTM) units.

[0075] Considering the LSTM layer, the forward pass activation / state calculation is as follows: (18) By defining LSTM variables as , , and The backpropagation gradient for each variable can be calculated as follows: (19) here, It is the output difference calculated by any subsequent layer.

[0076] The update of the internal parameters can be evaluated according to the following formula: (20) The gradient update for each set of weights using stochastic gradient descent can be represented as follows: (twenty one) in Where N is the learning rate, and N is the total size of the training dataset. It is the actual / true gradient value relative to the quantization weights. This is the estimated gradient evaluated on a mini-batch of size B. Specifically, ,and (twenty two) In this way, the gradient with respect to each weight matrix can be calculated accordingly. For example,

[0077] Each part of the above equation can be calculated as follows. In order to find a solution in closed form, we must adhere to a set of assumptions: (i) quantization noise on the output gate is absorbed when calculating the loss value (e.g., L2 difference); (ii) the variance of the noise-to-signal ratio within the layer is relatively small compared to the corresponding average, and therefore, the noise-to-signal ratio per neuron / activation can be replaced by the corresponding average in that layer; (iii) second-order quantization noise is negligible; and (iv) quantization noise (the difference between the quantized value and the floating-point value) is small enough to lie within the linear region of tanh.

[0078] (twenty three)

[0079] Under the first-order assumption, the dominant factor is equivalent to: (twenty four) This means: (25) The average value is calculated based on the number of data samples ( N ) and timestamp ( T ) is calculated on.

[0080] Assume the gradient error α is defined as The difference between them allows us to calculate the mean and variance of the gradient error as follows: (26) The first step in equation (26) is derived using the linear properties of the desired operation. The variance of the gradient noise is: (27) The matrix describing the average gradient covariance can be derived from... express, It is a function of the current parameter / weight values. Specifically, (28) therefore: (29) Here, η is defined according to equation (25). By adopting the central limit theorem and modeling the gradient error α using Gaussian random noise, we obtain equation (27). Equivalent to: (30) Equation (30) is approximate, where given And for the sum of two parts of the independent random variable (YB and YN, where B... <N), Therefore, the variance of the gradient noise in equation (27) is equivalent to: (31) Continuing, equation (18) can be interpreted as a discrete update of the stochastic differential equation: (32) Where t is a continuous variable. It is gradient noise with an expected value of 0, and Here, again, the constant g controls the scale of the random fluctuations in the dynamics, and is instead called the scaling factor.

[0081] Equations (18) and (32) are related to each other because: .

[0082] In order to keep the scaling factor g constant, the variance of the gradient update can be equal to the variance in equation (18).

[0083] (33) (34) here, It is the learning rate of the quantization model. This is the learning rate used to train the floating-point network. A similar approach can be applied to other parameters / weights in LSTM layers.

[0084] VIII. Compensation for other types of noise Similar methods to those described above can be used to compensate for any type of noise that can be measured in a neural network. For example, in addition to quantization noise, other types of noise can be introduced into a neural network to improve efficiency. This includes noise introduced by training the neural network in a low-voltage mode, such as a mode where the voltage applied to the hardware implementing the neural network is below the hardware's rated voltage. As another example, the parameters of the neural network can be stored using a relatively lossy medium (such as DRAM) during training, resulting in bit flips caused by noise. Again, some parameter values ​​of the neural network can be set to 0 or otherwise ignored, introducing noise; this process is called "pruning." Noise can also be introduced via block sparse training, where selected parameter values ​​of the neural network are pruned for one or more, but not all, training periods or iterations. As another example, noise can be introduced by converting some or all of the neural network's parameters to a different data type that does not have a lower bit width than in quantization, but is otherwise noisier. Alternatively, neural network training can be performed via a simulation-based training system, which also introduces noise.

[0085] Figure 4 This is a flowchart 400 outlining example methods for adjusting hyperparameters to compensate for noise during the training of a neural network, as can be used in some examples of the disclosed techniques. For example, Figure 1 The system can be used to implement the method shown.

[0086] At process block 410, noise in the NN is measured. The noise can be any type of noise associated with the measurable neural network. For example, as referenced above... Figure 3 The noise discussed here can be quantization noise generated by quantizing the values ​​of one or more parameters of the neural network. As another example, the noise measured can be noise introduced by training the neural network in low-voltage mode, noise introduced by storing the parameters of the neural network in a relatively lossy medium (such as DRAM) during training, noise introduced by pruning or block sparse training, noise introduced by converting some or all of the parameters of the neural network to a non-quantized but noisy data type, noise introduced by using a simulation-based training system for the neural network, etc.

[0087] At process block 420, at least one noise-to-signal ratio is calculated for the NN using the measured noise. In some examples, this may include acquiring a first set of values ​​for one or more signals before introducing noise, acquiring a second set of values ​​for one or more signals after introducing noise, and calculating the ratio of the difference between the first set of values ​​and the second set of values ​​to the first set of values.

[0088] At process block 430, the hyperparameters of the NN are adjusted based on the noise-to-signal ratio calculated at process block 420. As used herein, “hyperparameter” refers to a variable that determines the learning rate or the structure of the neural network. For example, the hyperparameters adjusted at process block 430 could be the learning rate, learning rate scheduling, bias, stochastic gradient descent batch size, the number of neurons in the neural network, the number of layers in the neural network, the sparsity of the network, or parameters related to data privacy (e.g., differential privacy protocols).

[0089] At process block 440, the neural network (NN) is trained using adjusted hyperparameters. In the example where the adjusted hyperparameter is the learning rate, training the NN using adjusted hyperparameters may optionally include computing gradient updates using the adjusted learning rate during backpropagation. In the example where the adjusted hyperparameter is the learning rate schedule, training the NN using adjusted hyperparameters may optionally include computing gradient updates during backpropagation according to the adjusted learning rate schedule. In the example where the adjusted hyperparameter is the bias, training the NN using adjusted hyperparameters may optionally include computing node outputs using the adjusted bias during forward propagation. In the example where the adjusted hyperparameter is the stochastic gradient descent batch size, training the NN using adjusted hyperparameters may optionally include increasing the stochastic gradient descent batch size as the noise-to-signal ratio increases. In the example where the adjusted hyperparameter is the number of neurons in the neural network, training the NN using adjusted hyperparameters may optionally include increasing the number of neurons in the neural network as the noise-to-signal ratio increases. In the example where the adjusted hyperparameter is the number of layers in the neural network, training the NN using adjusted hyperparameters may optionally include increasing the number of layers in the neural network as the noise-to-signal ratio increases. In other non-limiting examples, training a neural network using tuned hyperparameters may include using the tuned hyperparameters to compute the values ​​used during forward or backward propagation.

[0090] IX. Example Computing Environment Figure 5 A generalized example of a suitable computing environment 500 in which the described embodiments, techniques, and sciences can be implemented is shown. For example, computing environment 500 can implement the disclosed techniques to configure a processor to implement the disclosed software architecture and neural networks, and / or compile code into computer-executable instructions and / or configure bitstreams to perform these operations, including neural networks, as described herein.

[0091] The Computing Environment 500 is not intended to impose any limitations on the scope or functionality of the technology, as it can be implemented in various general-purpose or special-purpose computing environments. For example, the disclosed technology can be implemented using other computer system configurations, including handheld devices, multiprocessor systems, programmable consumer electronics, network PCs, minicomputers, mainframes, etc. The disclosed technology can also be implemented in distributed computing environments, where tasks are performed by remote processing devices linked via a communication network. In a distributed computing environment, program modules can reside in both local and remote memory storage devices.

[0092] refer to Figure 5 The computing environment 500 includes at least one processing unit 510, an optional neural network accelerator 515, and memory 520. Figure 5In this diagram, the most basic configuration 530 is included within the dashed lines. Processing unit 510 executes computer-executable instructions and can be a real or virtual processor. In a multiprocessor system, multiple processing units execute computer-executable instructions to increase processing power; therefore, multiple processors can operate simultaneously. Accelerator 515 may include a Tensor Processing Unit (TPU) and / or reconfigurable logic devices, such as those contained in an FPGA or programmable circuit architecture. Memory 520 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of both. Memory 520 stores, for example, software 580, images, and video that can implement the techniques described herein. The computing environment may have additional features. For example, computing environment 500 includes storage device 540, one or more input devices 550, one or more output devices 560, and one or more communication connections 570. Interconnect mechanisms (not shown) (such as buses, controllers, or networks) interconnect the components of computing environment 500. Typically, operating system software (not shown) provides an operating environment for other software running in computing environment 500 and coordinates the activities of the components of computing environment 500.

[0093] Storage device 540 may be removable or non-removable and includes a magnetic disk, magnetic tape or cassette tape, CD-ROM, CD-RW, DVD, or any other medium that can be used to store information and can be accessed within computing environment 500. Storage device 540 stores instructions for software 580 that can be used to implement the techniques described herein.

[0094] Input device 550 may be a touch input device that provides input to computing environment 500, such as a keyboard, keypad, mouse, touchscreen display, pen or trackball, voice input device, scanning device, or other device. For audio, input device 550 may be a sound card or similar device that accepts analog or digital audio input, or a CD-ROM reader that provides audio samples to computing environment 500. Output device 560 may be a monitor, printer, speaker, CD burner, or another device that provides output from computing environment 500.

[0095] Communication connection 570 enables communication with another computing entity via a communication medium (e.g., a connected network). The communication medium transmits information such as computer-executable instructions, compressed graphics information, video, or other data in the form of modulated data signals. Communication connection 570 is not limited to wired connections (e.g., megabit or gigabit Ethernet, Infiniband, Fibre Channel, or Fibre Channel connections), but also includes wireless technologies (e.g., via Bluetooth, WiFi (IEEE 802.11a / b / n), WiMax, cellular, satellite, laser, infrared RF connections) and other suitable communication connections for providing network connectivity for the disclosed methods. In a virtual hosting environment, the communication connection can be a virtualized network connection provided by the virtual host.

[0096] Some embodiments of the disclosed methods can be executed using computer-executable instructions that implement all or part of the disclosed techniques in a computing cloud 590. For example, the disclosed compiler, processor, and / or neural network is implemented using a server located in the computing environment, or the disclosed compiler, processor, and / or neural network can be implemented on a server located in the computing cloud 590. In some examples, the disclosed compiler executes on a conventional central processing unit (e.g., a RISC or CISC processor) that is extended to include vector processing instructions or a vector processor.

[0097] Computer-readable media is any available medium that can be accessed within computing environment 500. By way of example and not limitation, using computing environment 500, computer-readable media includes memory 520 and / or storage device 540. As used herein, the term computer-readable storage medium includes all tangible media used for data storage, such as memory 520 and storage device 540, but excludes transmission media such as modulated data signals.

[0098] X. Experimental Results Figures 6 to 9 It is a series of graphs showing the experimental results observed when training a quantized neural network with and without learning rate scaling to compensate for quantization noise.

[0099] Figure 6This is a graph 600 illustrating the accuracy measured against a test neural network model using various quantization levels with and without learning rate scaling, according to this disclosure. The test neural network model used in this experiment is a 16-layer VGG-16 neural network, and training was performed using CIFAR-10 data. In the baseline neural network, the values ​​of the neural network weights are represented in plain-precision float32 format. The results of the baseline float32 model (represented by a graph labeled "baseline") and models with parameters in four different quantization precision formats are plotted: 8-bit shared exponent format with one mantissa and one sign bit (represented by graphs labeled "1M_1S_8Exp_bk16 (scaled)" and "1M_1S_8Exp_bk16 (unscaled)"); 8-bit shared exponent format with two mantissas and one sign bit (represented by graphs labeled "2M_1S_8Exp_bk16 (scaled)" and "2M_1S_8Exp_bk16 (unscaled)"); and 8-bit shared exponent format with two mantissas and one sign bit (represented by graphs labeled "2M_1S_8Exp_bk16 (scaled)" and "2M_1S_8Exp_bk16 (unscaled)"). The experiment used two graph representations: one for "3M_1S_8Exp_bk16 (unscaled)" and the other for "4M_1S_8Exp_bk16 (unscaled)". The graphs also included a shared exponent format with three mantissa bits and a sign bit (represented by graphs labeled "3M_1S_8Exp_bk16 (scaled)" and "3M_1S_8Exp_bk16 (unscaled)"). Other experimental parameters included batch normalization, where gradients were quantized before or after calculating the average for each batch, and the use of common deep learning libraries, including PyTorch, TensorFlow, and Keras.

[0100] As shown in the figure, scaling the learning rate in the disclosed manner improves training accuracy compared to using the same learning rate as the baseline float32 model. For example, the accuracy achieved during training of a model quantized in an 8-bit shared exponent format with four mantissa bits and one sign bit is higher than that achieved using the baseline float32 model. Furthermore, this quantized model converges faster than the baseline model, thus requiring fewer training epochs. As shown in the figure, similar improvements in accuracy and the number of training epochs for convergence are achieved for other quantization formats when scaling the learning rate in the disclosed manner. Therefore, scaling the learning rate in the disclosed manner can advantageously improve the accuracy and efficiency of training quantized neural networks, thereby improving the operation of the computational hardware used to implement the training.

[0101] Figure 7This is a graph 700 showing the accuracy measured according to this disclosure for a test neural network model with parameters quantized in an 8-bit shared exponent format having five mantissa bits and one sign bit. The test neural network model used in this experiment is a ResNet50 neural network with 16 layers, and training was performed using the ImageNet database. In the baseline neural network, the values ​​of the neural network weights are represented in ordinary precision float32 format. The results for the following items were plotted: the baseline float32 model (represented by a plot labeled "Baseline"); the model with 8-bit shared exponential parameters (five mantissa bits and one sign bit) without learning rate scaling (represented by a plot labeled "M5S1t16Exp8_NoScaling"); the same model with learning rate scaling and a first learning rate schedule (represented by a plot labeled "M5S1t16Exp8_Scaled"); and the same model with learning rate scaling and a second learning rate schedule different from the first learning rate schedule (represented by a plot labeled "M5S1t16Exp8_Scaled (Different lr schedule)"). Other experimental parameters included batch normalization, post-quantization gradients, and the use of the Keras library.

[0102] As shown in the figure, the accuracy and number of convergence epochs of the quantization model without learning rate scaling are similar to those of the baseline model. Learning rate scaling can improve the accuracy and reduce the number of convergence epochs of both quantization models. For example, scaling the learning rate improves the accuracy of the quantization model by approximately 1.24% compared to the accuracy of the quantization model without learning rate scaling. Furthermore, the number of convergence epochs of the quantization model can be significantly reduced by using different learning rate schedules. Specifically, as shown in the figure, when training the quantization model using learning rate scaling, using the second learning rate schedule reduces the number of convergence epochs by almost 50%. These results further demonstrate that scaling the learning rate in the disclosed manner can advantageously improve the accuracy and efficiency of training quantization neural networks, thereby improving the operation of the computational hardware used for training. Moreover, these results indicate that adjusting the learning rate schedule can provide further improvements by reducing the number of convergence training epochs, thereby enabling more efficient operation of the computational hardware used for training.

[0103] Figure 8This is a graph 800 showing the accuracy measured for a test neural network model with parameters quantized in an 8-bit shared exponent format having one sign bit and different numbers of mantissa bits. In this experiment, according to this disclosure, all results except the baseline were obtained using a scaled learning rate. The test neural network model used in this experiment is a 16-layer ResNet50 neural network, and training was performed using the ImageNet database. In the baseline neural network, the values ​​of the neural network weights are represented in plain-precision float32 format. The results are plotted as follows: the baseline float32 model (represented by a graph labeled "baseline"), and quantized models with a proportional learning rate and parameters in an 8-bit shared exponent format having one sign bit and three, four, five, six, seven, or eight mantissa bits (represented by graphs labeled "M3S1t16Exp8", "M4S1t16Exp8", "M5S1t16Exp8", "M6S1t16Exp8", "M7S1t16Exp8", and "M8S1t16Exp8", respectively). Other parameters of the experiment included batch normalization, post-quantization gradient, and the use of the TensorFlow library.

[0104] like Figure 8 As shown, all models achieved higher accuracy compared to the baseline model, except for one of the quantized models with a scaled learning rate. Specifically, the quantized model with a scaled learning rate and parameters in an 8-bit shared exponent format with one sign bit and 4, 5, 6, 7, or 8 mantissa bits achieved higher accuracy than the non-quantized baseline model, while the quantized model with a scaled learning rate and parameters in an 8-bit shared exponent format with one sign bit and three mantissa bits was less accurate than the baseline model. As the results of this experiment demonstrate, the baseline model still needs modification to match state-of-the-art accuracy.

[0105] Figure 9Figure 900 shows the average accuracy improvement achieved relative to the non-quantized baseline model for experiments using a scaled learning rate for parameters with different quantization formats. The test neural network model used in this experiment is a ResNet50 neural network with 128 layers, and training was performed using the ImageNet database. In the baseline neural network, the values ​​of the neural network weights are represented in plain-precision float32 format. For quantized models with parameters in an 8-bit shared exponent format (with 3, 4, 5, 6, 7, or 8 mantissa bits), the average accuracy improvement achieved relative to the baseline model by using a scaled learning rate is shown (represented by regions labeled "M3_T128", "M4_T128", "M5_T128", "M6_T128", "M7_T128", and "M8_T128", respectively). Other parameters in the experiment include the post-quantization gradient and the use of the TensorFlow library.

[0106] like Figure 9 As shown, the average percentage improvement in accuracy achieved by using a scaled learning rate depends on the precision level of the quantization format. It should be understood that the M8_T128 model is more accurate than the M7_T128 model, which is more accurate than the M6_T128 model, and so on, due to the relative number of mantissas in the model. In this experiment, models with 4, 5, 6, 7, and 8 mantissas achieved accuracy improvements, with the model with 7 mantissas (M7_T128) achieving the highest accuracy improvement (greater than 1.5%).

[0107] However, the accuracy improvement for the model with 8 mantissas is minimal; the accuracy achieved by this model with a scaled learning rate is roughly equivalent to that achieved by the same model without a scaled learning rate. Therefore, for quantization formats with a relatively high noise-to-signal ratio (e.g., models with 8 mantissas), training with a scaled learning rate can provide accuracy comparable to that achieved by training without a scaled learning rate. Furthermore, the accuracy achieved by the model with 3 mantissas is lower than that achieved by training without a scaled learning rate. These results suggest that for quantization formats with a relatively low noise-to-signal ratio (e.g., models with 3 mantissas), scaling the learning rate may not improve accuracy because the amount of quantization noise dominates.

[0108] Therefore, performance improvements can be achieved by scaling the learning rate used in training neural network tensors of certain types of quantized formats. For example, such performance gains would not be readily achievable without the methods and apparatus disclosed herein, given the reduced programmer productivity and effort required to achieve acceptable levels of accuracy when training quantized neural networks. Thus, improved programmer productivity and ultimately improved hardware acceleration (through the use of quantized precision formats) can be achieved using certain disclosed methods and apparatus.

[0109] XI. Other examples of the disclosed technology Additional examples of the disclosed techniques are provided based on the examples above.

[0110] In some examples of the disclosed techniques, a neural network implemented using a quantization-enabled system can be trained by: acquiring a tensor using the quantization-enabled system, the tensor including values ​​of one or more parameters of the neural network represented in a quantization precision format; generating at least one noise signal metric representing quantization noise present in the tensor; and generating a scaled learning rate based on the at least one noise signal metric. The method may also include performing a training phase of the neural network using the values ​​of the tensor, including computing one or more gradient updates using the scaled learning rate.

[0111] In some examples, the tensor is a second tensor obtained by converting the values ​​of the first tensor from ordinary precision floating-point format to quantized precision format, and one or more parameters are weights used during the forward propagation phase of the neural network's training. Furthermore, in some examples, one or more parameters represent the edge weights and activation weights of the neural network, and generating at least one noise signal metric includes: for each of the multiple layers of the neural network, generating the noise signal ratio for the activation weights of that layer and generating the noise signal ratio for the edge weights of that layer.

[0112] In some examples, generating the noise-to-signal ratio for the activation weights of each of the multiple layers includes: calculating the difference between the activation weights of the second tensor for that layer and the activation weights of the first tensor for that layer, and dividing that difference by the absolute value of the activation weights of the first tensor for that layer. Similarly, in some examples, generating the noise-to-signal ratio for the edge weights of each of the multiple layers includes: calculating the difference between the edge weights of the second tensor for that layer and the edge weights of the first tensor for that layer, and dividing the calculated difference by the absolute value of the edge weights of the first tensor for that layer.

[0113] In some examples, the method further includes generating a scaling factor based on at least one noise metric. For example, for a neural network comprising a total of L layers, the scaling factor for the l-th layer of the neural network can be generated based on the sum of the average noise ratio of the activation weights of the l-th layer and the average noise ratio of the edge weights from the (l+1)-th to the L-th layers of the neural network. Furthermore, in some examples, training the neural network includes training the neural network via stochastic gradient descent. In such examples, the scaled learning rate of the l-th layer of the neural network is calculated using the following equation:

[0114] in This indicates that the learning rate has been scaled. This represents the predetermined learning rate of the neural network. The average signal-to-noise ratio of the activation weights of the l-th layer over the stochastic gradient descent batch size is represented in vector form. The average noise-to-signal ratio of the edge weights of the k-th layer of the neural network for each sample is represented in matrix form.

[0115] In some examples, computing one or more gradient updates using a scaled learning rate includes computing gradient updates using a scaled learning rate against one or more parameters of layer l. Alternatively or additionally, computing one or more gradient updates using a scaled learning rate may include computing gradient updates using the same scaled learning rate generated for layer l against one or more parameters of one or more other layers of the neural network.

[0116] In some examples, the method further includes generating a scaling factor based on at least one noise signal metric. In such examples, a normal-precision floating-point format may represent a value with a first bit width, a quantized-precision format may represent a value with a second bit width, the second bit width being less than the first bit width, and the method may further include storing the scaling factor in an entry for the second bit width in a lookup table; obtaining the scaling factor for the second bit width by accessing the entry for the second bit width in the lookup table; computing gradient updates for one or more other parameters of the neural network, the one or more other parameters being represented in terms of the second bit width; and computing the gradient updates for the one or more other parameters using the scaling factor for the second bit width.

[0117] In some examples, the training period of the neural network is a second period performed after the first training period of the neural network. In such examples, the method may further include performing the first training period using the values ​​of a tensor before generating a scaled learning rate, including computing one or more gradient updates using a predetermined learning rate of the neural network. Furthermore, generating the scaled learning rate based on at least one noise signal metric may include scaling the predetermined learning rate based on at least one noise signal metric.

[0118] In some examples of the disclosed techniques, a system for training a neural network implemented using a quantization-enabled system may include a memory; one or more processors coupled to the memory and adapted to perform quantization precision operations; and one or more computer-readable storage media storing computer-readable instructions that, when executed by the one or more processors, cause the system to perform a method for training the neural network. For example, the one or more processors may include a neural network accelerator with tensor processing units.

[0119] The method may include instructions to cause the system to represent one or more parameter values ​​of a neural network in a quantization precision format; instructions to cause the system to calculate at least one metric representing the quantization noise present in the values ​​represented in the quantization precision format; and instructions to cause the system to adjust the learning rate of the neural network based on the at least one metric.

[0120] In some examples, one or more parameters of the neural network include multiple weights of a layer of the neural network; and at least one metric includes the noise-to-signal ratio. This noise-to-signal ratio can be calculated by dividing the difference between the weight values ​​expressed in quantized precision format and the weight values ​​expressed in ordinary precision floating-point format by the absolute value of the weight values ​​expressed in ordinary precision floating-point format.

[0121] In some examples, one or more parameters may include activation weights and edge weights of a first layer of the neural network. In such examples, calculating at least one metric may include calculating a first noise-to-signal ratio for the activation weights of the first layer and a second noise-to-signal ratio for the edge weights of the first layer, and the system may also include instructions to train the neural network using at least some values ​​of the parameters expressed in quantized precision format, including instructions to compute gradient updates using an adjusted learning rate for the first layer and at least one other layer of the neural network.

[0122] In some examples, one or more parameters may include the weights of a first layer and the weights of a second layer of the neural network. In such examples, instructions causing the system to compute at least one metric may include instructions causing the system to compute a first noise-to-signal ratio for the weights of the first layer and a second noise-to-signal ratio for the weights of the second layer. Furthermore, instructions causing the system to adjust the learning rate based on at least one metric may include instructions causing the system to: compute a first scaling factor for the first layer based on the first noise-to-signal ratio; compute a scaled learning rate for the first layer by scaling the global learning rate of the neural network using the first scaling factor; compute a second scaling factor for the second layer based on the second noise-to-signal ratio; and compute a scaled learning rate for the first layer by scaling the global learning rate of the neural network using the first scaling factor. The system may also include instructions causing the system to train the neural network using the weights of the first and second layers, expressed in quantized precision format, including computing a first gradient update for the weights of the first layer using the scaled learning rate of the first layer, and computing a second gradient update for the weights of the second layer using the scaled learning rate of the second layer.

[0123] In some examples of the disclosed techniques, a method for compensating for noise during the training of a neural network may include calculating at least one noise-to-signal ratio representing the noise present in the neural network; adjusting the hyperparameters of the neural network based on the at least one noise-to-signal ratio; and training the neural network using the adjusted hyperparameters. The hyperparameters may include at least one of the following: learning rate, learning rate scheduling, bias, stochastic gradient descent batch size, the number of neurons in the neural network, or the number of layers in the neural network. Alternatively, other hyperparameters may be used.

[0124] In some examples, calculating at least one noise-to-signal ratio includes obtaining a first tensor containing the values ​​of one or more parameters of the neural network before introducing noise into the neural network; introducing noise into the neural network; obtaining a second tensor containing the values ​​of one or more parameters after introducing noise into the neural network; calculating the difference between one or more values ​​of the second tensor and one or more corresponding values ​​of the first tensor; and dividing the difference by the absolute value of one or more corresponding values ​​of the first tensor.

[0125] In some examples, introducing noise into a neural network may include one or more of the following: changing the data type of the values ​​of one or more parameters of the neural network, reducing the batch size of stochastic gradient descent of one or more layers of the neural network, reducing the voltage supplied to the hardware implementing the neural network, enabling simulation-based training of the neural network, or storing the values ​​of one or more parameters of the neural network in DRAM.

[0126] In some examples, adjusting hyperparameters based on at least one noise-to-signal ratio includes: calculating a scaling factor based on at least one noise-to-signal ratio; and scaling the hyperparameters using the scaling factor. As discussed herein, hyperparameters can be adjusted to compensate for the impact of noise present in the neural network on the accuracy of gradient updates computed during the training of the neural network.

[0127] Given the many possible embodiments in which the principles of the disclosed subject matter can be applied, it should be recognized that the illustrated embodiments are merely preferred examples and should not be considered as limiting the scope of the claimed subject matter. Rather, the scope of the claimed subject matter is defined by the appended claims. Therefore, we claim all that falls within the scope of these claims as our invention.

Claims

1. A method for training a neural network implemented using a quantization-enabled computing system, the computing system including at least one processor, at least one memory, and a storage device, the computing system further including a quantization domain providing functionality for converting data represented in full-precision floating-point format in a normal-precision neural network module into quantized format values, the method comprising: Using the aforementioned quantization-enabled system: Obtain a tensor, the tensor including the values ​​of one or more parameters of the neural network in a quantized precision format; Generate at least one noise-to-signal ratio representing the quantization noise present in the tensor; Based on the at least one noise signal-to-speech ratio, a scaled learning rate is generated, along with at least one tuned hyperparameter among learning rate scheduling, bias, stochastic gradient descent batch size, number of neurons in the neural network, and number of layers in the neural network. as well as The training phase of the neural network is performed using the values ​​of the tensor, including computing one or more gradient updates using the scaled learning rate to compensate for the quantization noise. The neural network implemented using a quantization-enabled computing system is applied to at least one of the following technical fields: Image recognition, Speech recognition, Image classification, Convert speech to text and / or other languages; Natural Language Processing, or Automatic language switching.

2. The method according to claim 1, wherein: The tensor is a second tensor obtained by converting the value of the first tensor from ordinary precision floating-point format to the quantization precision format, and The one or more parameters are weights used during the forward propagation phase of the training period of the neural network.

3. The method according to claim 2, wherein: The one or more parameters represent the edge weights and activation weights of the neural network, and Generating the noise signal metric includes: for each of the plurality of layers of the neural network, generating a noise signal ratio for the activation weights of the layer and generating a noise signal ratio for the edge weights of the layer.

4. The method according to claim 3, wherein: Generating the noise-to-signal ratio for the activation weights of each of the plurality of layers includes: calculating the difference between the activation weights of the second tensor for the layer and the activation weights of the first tensor for the layer, and dividing the difference by the absolute value of the activation weights of the first tensor for the layer; and Generating the noise-to-signal ratio for the edge weights of each of the plurality of layers includes: calculating the difference between the edge weights of the second tensor for the layer and the edge weights of the first tensor for the layer, and dividing the difference by the absolute value of the edge weights of the first tensor for the layer.

5. The method of claim 3, further comprising generating a scaling factor based on the at least one noise signal metric, wherein: The neural network comprises a total of L layers; and The scaling factor for the l-th layer in the L-th layer of the neural network is generated based on the sum of the average noise-to-signal ratio of the activation weights for the l-th layer and the average noise-to-signal ratio of the edge weights for the (l+1)-th to L-th layers of the neural network.

6. The method according to claim 5, wherein: Training the neural network includes training the neural network via stochastic gradient descent; and The scaled learning rate of the l-th layer of the neural network is calculated using the following formula: Where εq represents the scaled learning rate, ε represents the predetermined learning rate of the neural network, the average noise-to-signal ratio of the activation weights of the l-th layer over the stochastic gradient descent batch size is represented in vector form, and the average noise-to-signal ratio of the edge weights of the k-th layer of the neural network for each sample is represented in matrix form.

7. The method according to claim 6, further comprising: The scaled learning rate is used to compute gradient updates for one or more parameters of the l-th layer.

8. The method of claim 7, wherein calculating the one or more gradient updates using the scaled learning rate further comprises: Using the same scaled learning rate generated for the l-th layer, gradient updates are computed for one or more parameters of one or more other layers of the neural network.

9. The method of claim 2, further comprising generating a scaling factor based on the at least one noise signal metric, wherein: The ordinary precision floating-point format represents the value having a first bit width; The quantization precision format represents the value having a second bit width, which is lower than the first bit width; as well as The method further includes: The scaling factor is stored in the lookup table in the entry for the second bit width; By accessing the entry in the lookup table for the second bit width to obtain the scaling factor for the second bit width, gradient updates are computed for one or more other parameters of the neural network, said one or more other parameters being represented by the second bit width; and The gradient update is calculated using the scaling factor for the second bit width for the one or more other parameters.

10. The method of claim 1, wherein the training period of the neural network is during the training phase of the neural network. The method also includes a second phase, executed after the first training phase, which is further described in the following: Before generating the scaled learning rate, the first training period is performed using the value of the tensor, including making One or more gradient updates are computed using a predetermined learning rate of the neural network. Generating the scaled learning rate based on the at least one noise signal metric includes scaling the predetermined learning rate based on the at least one noise signal metric.

11. The method according to claim 1, further comprising: The number of layers in the neural network is adjusted based on the scaled learning rate.

12. A computational system for training a neural network implemented using a quantization-enabled system, the system comprising: At least one memory; One or more processors are coupled to the memory; A quantization domain is provided to convert data represented in full-precision floating-point format in a normal-precision neural network module into quantized format values. as well as One or more computer-readable storage media storing computer-readable instructions that, when executed, cause the system to perform operations, the operations including: Obtain a tensor, the tensor including the values ​​of one or more parameters of the neural network in a quantized precision format; Generate at least one noise-to-signal ratio representing the quantization noise present in the tensor; Based on the at least one noise signal-to-speech ratio, a scaled learning rate is generated, along with at least one adjusted hyperparameter from the following: learning rate scheduling, bias, stochastic gradient descent batch size, number of neurons in the neural network, and number of layers in the neural network; and The training phase of the neural network is performed using the value of the tensor, including computing one or more gradient updates using the scaled learning rate to compensate for the quantization noise; The neural network implemented using a quantization-enabled computing system is applied to at least one of the following technical fields: Image recognition, Speech recognition, Image classification, Convert speech to text and / or other languages; Natural Language Processing, or Automatic language switching.

13. The computing system according to claim 12, wherein: The one or more parameters of the neural network include multiple weights of the layers of the neural network; and The noise signal metric is calculated by calculating the difference between the value of the weight expressed in the quantization precision format and the value of the weight expressed in ordinary precision floating-point format, and dividing the difference by the absolute value of the weight expressed in the ordinary precision floating-point format.

14. The computing system according to claim 12, wherein: The one or more parameters include the activation weights and edge weights of the first layer of the neural network; The noise signal metric includes calculating a first noise signal ratio for the activation weights of the first layer and a second noise signal ratio for the edge weights of the first layer; and The operation also includes instructions to train the neural network using at least some values ​​of the parameters expressed in the quantization precision format, including instructions to compute gradient updates for the first layer and at least one other layer of the neural network using the adjusted learning rate.

15. The computing system of claim 12, wherein the one or more processors include a neural network accelerator having tensor processing units.

16. The computing system of claim 12, wherein the learning rate is adjusted to compensate for the impact of the quantization noise on the accuracy of gradient updates computed during subsequent training of the neural network.

Images