Tuning activation compression for neural network training

Abstract

Disclosed are apparatus and methods for training a neural network accelerator using a quantized precision data format, and in particular, apparatus and methods for adjusting a floating point format used to store activation values ​​during training. In certain examples of the disclosed technology, a computing system includes a processor, a memory, and a floating point compressor in communication with the memory. The computing system is configured to generate a neural network including activation values ​​expressed in a first floating point format, select a second floating point format for the neural network based on a performance metric, convert at least one of the activation values ​​to a second floating point format, and store the compressed activation value in the memory. Aspects of the second floating point format that can be adjusted include the number of bits used to express the mantissa, the exponential format, the use of non-uniform mantissas, and / or the use of outliers used to express some of the mantissas.

Description

Background Art

[0001] Machine learning (ML) and artificial intelligence (AI) techniques may be useful for solving many complex computing problems, such as recognizing images and voices, analyzing and classifying information, and performing various classification tasks. Machine learning is a field of computer science that uses statistical techniques to enable computer systems to extract higher-level features from a training data set. Specifically, features can be extracted by training models such as artificial neural networks (NN) or deep neural networks (DNN). After the model is trained, new data can be applied to the model, and the trained model can be used to classify the new data (e.g., higher-level features can be extracted). Machine learning models are usually executed on general-purpose processors (also referred to as central processing units (CPUs)). However, training models and / or using models may be computationally expensive, and therefore such techniques may not be used in real time using general-purpose processors. Therefore, there are a lot of opportunities to improve computer hardware and software to implement neural networks. Summary of the invention

[0002] Disclosed are apparatus and methods for storing activation values ​​from a neural network in a compressed format for use during forward and backward propagation training of the neural network. Computing systems suitable for employing such neural networks include computers with general purpose processors, neural network accelerators, or reconfigurable logic devices such as field programmable gate arrays (FPGAs). Activation values ​​generated during forward propagation can be "hidden" (temporarily stored in a mass storage device) in a compressed format and can be retrieved for use during backward propagation. Activation values ​​used during training can be expressed in normal precision, or in quantized or block floating point format (BFP). The truncated activation values ​​can be expressed in a format that is further compressed than the format used during training. In some examples, the compressed format includes lossy or non-uniform mantissas for the compressed values. In some examples, the compressed format includes storing outliers for some but not all mantissa values. As training progresses, the parameters of the compressed format can be adjusted, for example by increasing the precision of the compressed format used to store the activation values.

[0003] In some examples of the disclosed technology, a computer system includes a general and / or special purpose neural network processor, a mass storage including a computer readable storage device or memory, and a floating point compressor in communication with the mass storage. As forward propagation occurs during neural network training, activation values ​​are generated in a first block floating point format. Block floating point is used to convert the activation values ​​to a digital format having a numerical precision less than the precision of the first block floating point format. The compressed activation values ​​are stored in the mass storage for use during back propagation.

[0004] This summary is provided to introduce a selection of concepts that are further described below in the detailed description in a simplified form. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. The foregoing and other objects, features, and advantages of the disclosed subject matter will become more apparent from the detailed description that follows with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 is a block diagram of a quantization-enabled system for performing activation compression, as may be implemented in certain examples of the disclosed technology.

[0006] Figure 2 is a diagram depicting an example of a deep neural network, as may be modeled using certain example methods and apparatus disclosed herein.

[0007] Figure 3 is a diagram depicting certain aspects of converting a normal floating point format to a quantized floating point format, as may be performed in certain examples of the disclosed technology.

[0008] Figure 4 Depicted are several example block floating point formats that may be used to represent quantized neural network models, as may be used in certain examples of the disclosed technology.

[0009] Figure 5 Depicted are several example block floating point formats that may be used to represent quantized neural network models, as may be used in certain examples of the disclosed technology.

[0010] Figure 6 is a flow chart depicting an example method of training a neural network for use with a quantized model, as may be implemented in certain examples of the disclosed technology.

[0011] Figure 7 is a block diagram depicting an example environment for implementing activation compression, as may be implemented in certain examples of the disclosed technology.

[0012] Figure 8 is a block diagram depicting a more detailed example of activation compression, as may be implemented in certain examples of the disclosed technology.

[0013] Fig. 9 Graph illustrating an example of a performance metric that may be used to select an adjusted precision parameter for compressed activation values, as may be used in certain examples of the disclosed technology.

[0014] Fig.10is a diagram illustrating the conversion of a uniform three-value mantissa format to a non-uniform four-value lossy mantissa format, as may be implemented in certain examples of the disclosed technology.

[0015] Fig.11 is a diagram illustrating conversion of a uniform three-value mantissa format to a non-uniform three-value lossy mantissa format, as may be implemented in certain examples of the disclosed technology.

[0016] Fig.12 is a diagram illustrating conversion of a uniform three-valued mantissa format to a non-uniform two-valued lossy mantissa format, as may be implemented in certain examples of the disclosed technology.

[0017] Fig.13 is a diagram illustrating conversion of a sign value and three-value mantissa format to a non-uniform five-value lossy mantissa format, as may be implemented in certain examples of the disclosed technology.

[0018] Fig.14 is a block diagram depicting an example environment for implementing activation compression using outliers, as may be implemented in certain examples of the disclosed technology.

[0019] Fig.15 is a diagram illustrating an example of converting activation values ​​to a second block floating point format with outliers, as may be implemented in certain examples of the disclosed technology.

[0020] Fig.16 is a block diagram depicting a more detailed example of activation compression using outliers, as may be implemented in certain examples of the disclosed technology.

[0021] Fig.17 is a block diagram depicting an example outlier quantizer that may be implemented in certain examples of the disclosed technology.

[0022] Fig.18 Flowchart outlining an example method of storing compressed activation values, as may be performed in certain examples of the disclosed technology.

[0023] Fig.19 As a flow chart outlining an example method for storing compressed activation values ​​in a selected floating point format, as may be performed in certain examples of the disclosed technology.

[0024] Fig. 20 is a flow chart outlining an example method for adjusting activation compression parameters while training a neural network, as may be performed in certain examples of the disclosed technology.

[0025] Fig.21 is a block diagram illustrating a suitable computing environment for implementing certain examples of the disclosed technology. DETAILED DESCRIPTION

[0026] one, General considerations

[0027] The present disclosure is described in the context of representative embodiments, which are not intended to be limiting in any way.

[0028] As used in this application, the singular forms "a", "an", and "the" include the plural forms unless the context clearly indicates otherwise. In addition, the term "including" means "comprising". In addition, the term "coupled" covers mechanical, electrical, magnetic, optical, and other practical ways of coupling or linking items together, and does not exclude the presence of intermediate elements between the coupled items. In addition, as used herein, the term "and / or" means any one or more combinations of the phrases.

[0029] The systems, methods, and devices described herein should not be construed as limiting in any way. Instead, the present disclosure is directed to all novel and non-obvious features and aspects of the various disclosed embodiments, individually and in various combinations and sub-combinations with each other. The disclosed systems, methods, and devices are not limited to any particular aspect or feature or combination thereof, nor do the disclosed things and methods require the presence of any one or more specific advantages or solutions to problems. In addition, any feature or aspect of the disclosed embodiments may be used in various combinations and sub-combinations with each other.

[0030] Although the operations of some disclosed methods are described in a specific, continuous order for ease of presentation, it should be understood that unless the specific language set forth below requires a specific ordering, this description includes rearrangement. For example, the operations described in sequence may be rearranged or performed simultaneously in some cases. In addition, for simplicity, the accompanying drawings may not show the various ways in which the disclosed things and methods can be used in combination with other things and methods. In addition, the description sometimes uses terms such as "produce", "generate", "display", "receive", "verify", "implement", "execute", "convert" and "initiate" to describe the disclosed methods. These terms are high-level descriptions of the actual operations performed. The actual operations corresponding to these terms will vary depending on the specific implementation, and are easily discernible by those of ordinary skill in the art.

[0031] The theories of operation, scientific principles or other theoretical descriptions presented herein with reference to the devices or methods of the present disclosure are provided for the purpose of better understanding and are not intended to limit the scope. The devices and methods in the appended claims are not limited to those devices and methods that function in the manner described by these theories of operation.

[0032] Any disclosed method may be implemented as computer executable instructions stored on one or more computer readable media (e.g., computer readable media such as one or more optical disks, volatile memory components (such as DRAM or SRAM), or non-volatile memory components (such as hard disk drives)) and executed on a computer (e.g., any commercial computer, including a smart phone or other mobile device including computing hardware). Any computer executable instructions for implementing the disclosed technology, as well as any data created and used during the implementation of the disclosed embodiments, may be stored on one or more computer readable media (e.g., computer readable storage media). The computer executable instructions may be part of a software application or a dedicated software application that is accessed or downloaded, for example, via a web browser or other software application (such as a remote computing application). Such software may be executed, for example, on a single local computer or in a network environment (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network), or other such networks) using one or more network computers.

[0033] For clarity, only certain selected aspects of software-based implementations are described. Other details known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any particular computer language or program. For example, the disclosed technology can be implemented by software written in C, C++, Java, or any other suitable programming language. Certain details of suitable computers and hardware are known and need not be elaborated in this disclosure.

[0034] Furthermore, any software-based embodiments (including, for example, computer-executable instructions for causing a computer to perform any disclosed method) may be uploaded, downloaded, or remotely accessed via appropriate communication means, including, for example, the Internet, the World Wide Web, an intranet, a software application, cables (including fiber optic cables), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communications means.

[0035] two, Overview of Quantized Artificial Neural Networks

[0036] Artificial neural networks (ANNs or "NNs" as used throughout this article) are used in many applications in artificial intelligence and machine learning, including image recognition, speech recognition, search engines, and other appropriate applications. Processing for these applications can occur on individual devices such as personal computers or cell phones, but can also be performed in large data centers. At the same time, hardware accelerators that can be used with NNs include specialized NN processing units, such as tensor processing units (TPUs) and field programmable gate arrays (FPGAs) that are programmed to accelerate neural network processing. Due to their flexible nature and low power consumption per unit of computation, such hardware devices are being deployed in consumer devices and data centers.

[0037] Traditionally, NNs are trained and deployed using single-precision floating point (32-bit floating point or float32 format). However, it has been shown that lower-precision floating point formats such as 16-bit floating point (float16) or fixed-point formats can be used to perform inference operations with minimal loss of accuracy. On dedicated hardware such as FPGAs, reduced-precision formats can significantly improve latency and throughput of DNN processing.

[0038] Numbers represented in a normal precision floating point format (e.g., floating point numbers expressed in a 16-bit floating point format, a 32-bit floating point format, a 64-bit floating point format, or an 80-bit floating point format) can be converted to numbers in a quantized precision format, which can allow for performance benefits in performing operations. In particular, NN weights and activation values ​​can be represented in a lower precision quantized format with an acceptable level of error introduced. Examples of lower precision quantized formats include formats with reduced bit widths (including by reducing the number of bits used to represent the mantissa or exponent of a number) and block floating point formats, in which two or more numbers share the same single exponent.

[0039] One of the characteristics of computation on FPGA devices is that it typically lacks hardware floating point support. Floating point operations can be performed with a penalty using flexible logic, but the amount of logic required to support floating point is often prohibitive in FPGA implementations. Some newer FPGAs have been developed that support floating point computations, but even on these FPGAs, the same device produces twice as much computational output per unit time as when it is used in integer mode. Typically, NNs are created with floating point computations in mind, but when FPGAs are targeted for NN processing, it would be beneficial if the neural network could be expressed using integer arithmetic. Examples of the disclosed technology include hardware implementations of block floating point (BFP), including the use of BFP in NNs, FPGAs, and other hardware environments.

[0040] A typical floating point representation in a computer system consists of three parts: the sign (s), the exponent (e), and the mantissa (m). The sign indicates whether the number is positive or negative. The exponent and mantissa are used in scientific notation as follows:

[0041] Value = s × m × 2 e (Formula 1)

[0042] Any number can be represented within the limits of the precision of the mantissa. Since the exponent scales the mantissa by powers of 2, just as the exponent scales by powers of 10 in scientific notation, very large numbers of magnitude can be represented. The precision of the representation is determined by the precision of the mantissa. Typical floating point representations use mantissas that are 10 (float16), 24 (float32), or 53 (float64) bits wide. Magnitude greater than 2 53 An integer of 2 can be approximated in the float64 floating point format, but it will not be represented exactly because there are not enough bits in the mantissa. Similar effects can occur for arbitrary fractions, where the fraction is represented by the bits of the mantissa raised to a negative power of 2. There are many fractions that cannot be represented exactly because they are irrational numbers in the binary number system. More exact representations are possible in both cases, but they may require more bits in the mantissa. Finally, an infinite number of mantissa bits are required to represent some numbers exactly (e.g., ). The 10-bit (half-precision floating point), 24-bit (single-precision floating point), and 53-bit (double-precision floating point) mantissa limits are common trade-offs between mantissa storage requirements and representation accuracy in general-purpose computers.

[0043] With a block floating point format, a set of two or more numbers uses a single shared exponent, and each number still has its own sign and mantissa. In some examples, the shared exponent is selected to be the largest exponent of the original floating point value. For the purposes of this disclosure, the term block floating point (BFP) means a digital system in which a single exponent is shared between two or more values, each of which is represented by a sign and mantissa pair (whether or not there is an explicit sign bit, or whether the mantissa itself is signed). In some examples, all values ​​of one or more rows or columns of a matrix or vector, or all values ​​of a matrix or vector, can share a common exponent. In other examples, the BFP representation can be unsigned. In some examples, some but not all elements in a matrix or vector BFP representation may include numbers represented as integers, floating point numbers, fixed point numbers, signs, or other data formats mixed with numbers represented using signs, mantissas, and exponents. In some examples, some or all elements in a matrix or vector BFP representation may include complex elements with two or more parts, for example: a complex number with an imaginary part (a+bi, where ); a fraction with a numerator and denominator in polar coordinates (r, θ), or other multi-component element.

[0044] The parameters of a specific BFP format can be selected for a specific implementation to balance precision and storage requirements. For example, instead of storing an exponent with each floating point number, a group of numbers can share the same exponent. In order to share the exponent while maintaining a high level of accuracy, the numbers should have nearly the same magnitude, because the difference in magnitude is expressed in the mantissa. If the difference in magnitude is too large, the mantissa for large values ​​will overflow, or the mantissa for small values ​​may be zero ("underflow"). Depending on the specific application, a certain amount of overflow and / or underflow may be acceptable.

[0045] The size of the mantissa can be adjusted to suit a particular application. This may affect the precision of the number represented, but the potential gain is realized through the reduced representation size. For example, a normal single-precision floating point number has a size of four bytes, but for some implementations of the disclosed technology, only two bytes are used to represent the sign and mantissa of each value. In some implementations, the sign and mantissa of each value can be represented with one byte or less.

[0046] In certain examples of the disclosed technology, the representation expressed above is used to derive the original number from the representation, but only a single exponent is stored for a set of numbers, each of which is represented by a signed mantissa. Each signed mantissa can be represented with two bytes or less, so the memory storage savings are about 2 times compared to four-byte floating point numbers. In addition, the memory bandwidth requirements for loading and storing these values ​​are also about one-half of normal floating point numbers.

[0047] Neural network operations are used in many artificial intelligence operations. Typically, most of the processing operations performed when implementing a neural network are to perform matrix × matrix or matrix × vector multiplication or convolution operations. Such operations are computational bandwidth and memory bandwidth intensive, where the size of the matrix can be, for example, 1000 × 1000 elements (e.g., 1000 × 1000 numbers, each number including a sign, a mantissa, and an exponent) or larger, and many matrices are used. As discussed herein, BFP techniques can be applied to such operations to reduce memory bandwidth and computational requirements in a given system, regardless of whether the system is an FPGA, a CPU, or other hardware platform. As used herein, the term "element" is used herein to refer to a member of such a matrix or vector.

[0048] As used herein, the term "tensor" refers to a multidimensional array matrix that can be used to represent the attributes of a NN, and includes one-dimensional vectors and two-dimensional, three-dimensional, four-dimensional or higher dimensions. As used in this disclosure, unless otherwise specified, a tensor does not require any other mathematical properties.

[0049] As used herein, the term "normal precision floating point" refers to a floating point number format having a mantissa, an exponent, and an optional sign and supported by a native or virtual CPU. Examples of normal precision floating point formats include, but are not limited to, IEEE 754 standard formats, such as 16-bit, 32-bit, 64-bit, or other processors supported by a processor, such as Intel AVX, AVX2, IA32, x86_64, or 80-bit floating point formats.

[0050] As used herein, the term "lossy mantissa" refers to a mantissa that represents a higher precision mantissa as a discrete set of lower precision mantissa values. For example, for a four-bit number with a sign bit and three mantissa bits, a three-digit mantissa (which can represent 8 values, such as the set {0,1,2,3,4,5,6,7}) can be converted into a lossy mantissa with a discrete set of mantissa values, such as any one of the following sets: {0,1,3,7}, {0,1,7}, or {0,7}, depending on the selected lossy mantissa scheme. The underlying representation of the lossy mantissa may vary. For the first three sets, example sets of corresponding binary representations are {00,01,10,11}; {00,10,11}; and {0,1}.

[0051] As used herein, the term "non-uniform mantissa" refers to a property of certain lossy mantissas where at least one of the values ​​represented is not uniformly distributed. For the example in the previous paragraph, the three-digit mantissa is uniform because each value is one unit apart. The three sets of lossy mantissas are also non-uniform, being distributed differently: for the first set, 1, 2, and 4 units apart; for the second set, 1 or 6 units apart, and for the third set, 7 units apart.

[0052] Different precision (e.g., different quantization precision) formats can be used to represent a given number. For example, a number can be represented in a format of higher precision (e.g., float32) and a format of lower precision (e.g., float16). Reducing the precision of a number can include reducing the number of digits used to represent the mantissa or exponent of a number. In addition, reducing the precision of a number can include narrowing the range of values ​​that can be used to represent the exponent of a number, such as when multiple numbers share a common exponent. Similarly, increasing the precision of a number can include increasing the number of digits that can be used to represent the mantissa or exponent of a number. In addition, increasing the precision of a number can include increasing the range of values ​​that can be used to represent the exponent of a number, such as when a number is separated from a group of numbers that share a common exponent. As used herein, converting a number from a higher precision format to a lower precision format can be referred to as down-converting or quantizing the number. Converting a number from a lower precision format to a higher precision format can be referred to as up-converting or dequantizing the number.

[0053] As used herein, the term "quantized precision floating point" refers to a floating point number format in which two or more values ​​of a tensor have been modified to have lower precision than when the values ​​are represented in normal precision floating point. Specifically, many examples of quantized precision floating point representations include block floating point formats, in which two or more values ​​of a tensor are represented with reference to a common exponent. Quantized precision floating point numbers can be generated by selecting a common exponent for two, more, or all elements of a tensor and offsetting the mantissas of individual elements to match the shared common exponent. In some examples, groups of elements within a tensor can share a common exponent, for example, on a per-row, per-column, per-block, or other basis.

[0054] In one example of the disclosed technology, a neural network accelerator is configured to perform training operations on a neural network layer, including forward propagation and back propagation. The values ​​of one or more neural network layers may be expressed in a quantized format having a lower precision than a normal precision floating point format. For example, a block floating point format may be used to accelerate calculations performed in training and reasoning operations using a neural network accelerator. The use of a quantized format may improve neural network processing, for example by allowing faster hardware, reduced memory overhead, simpler hardware design, reduced energy use, reduced integrated circuit area, cost savings, and other technical improvements. It is often desirable to perform operations to mitigate noise or other inaccuracies introduced by using a lower precision quantized format. In addition, various parts of neural network training, such as temporary storage of activation values, may be improved by compressing a portion of these values ​​(e.g., for an input layer, hidden layer, or output layer of a neural network), either from a normal precision floating point or from a first block floating point number to a lower precision digital format with a lossy or non-uniform mantissa. The activation values ​​may be retrieved and dequantized later for use during back propagation, for example, during a training phase.

[0055] The input tensor for a given layer can be converted from a normal precision floating point format to a quantized precision floating point format. Tensor operations can be performed using the converted input tensor with lossy or non-uniform mantissas. The result of the tensor operation can be converted from a block floating point format to a normal precision floating point format. Tensor operations can be performed during a forward propagation mode or a reverse propagation mode of a neural network. For example, during the reverse propagation mode, the input tensor can be an output error term from a layer adjacent to (e.g., after) a given layer or a weight of a given layer. As another example, during the forward propagation mode, the input tensor can be an output term from a layer adjacent to (e.g., before) a given layer or a weight of a given layer. The converted result can be used to generate an output tensor of a neural network layer, where the output tensor is in a normal precision floating point format. In this way, the neural network accelerator can potentially become smaller and more efficient than a comparable accelerator using only a normal precision floating point format. Smaller and more efficient accelerators may have enhanced computing performance and / or enhanced energy efficiency. In addition, the neural network accelerator can potentially have higher accuracy than an accelerator that uses only quantized precision floating point formats. By improving the accuracy of the accelerator, the convergence time for training can be reduced and the accelerator can be more accurate when classifying the input of the neural network. Reducing the computational complexity of the model used can potentially reduce the time to extract features during inference, reduce the time used for adjustment during training, and / or reduce energy consumption during training and / or inference. Example architecture for implementing activation compression with non-uniform mantissa block floating point

[0056] Figure 1 1 is a block diagram 100 outlining an example quantization-enabled system 110 that may be implemented in certain examples of the disclosed technology, including for use in compression of activation values ​​during training of a neural network. Figure 1As shown in , the quantization-enabled system 110 may include multiple hardware resources, including a general-purpose processor 120 and a special-purpose processor such as a graphics processing unit (GPU) 122 and a neural network accelerator 180. The processor is coupled to a memory 125 and a storage device 129, which may include volatile or non-volatile memory devices. The processor 120 and the GPU 122 execute instructions stored in the memory or storage device to provide a neural network module 130. The neural network module 130 includes a software interface that allows the system to be programmed to implement various types of neural networks. For example, a software function may be provided that allows an application to define a neural network, including weights, biases, activation functions, node values, and interconnections between neural network layers. In addition, the software function may be used to define state elements for a recurrent neural network. The neural network module 130 may also provide utilities to allow training and retraining of a neural network implemented using the module. Values ​​representing the neural network module are stored in the memory or storage device and are operated on by instructions executed by one of the processors. Values ​​stored in memory or storage may be represented using normal precision floating point and / or quantized floating point values ​​(including floating point values ​​with lossy or non-uniform mantissas or outlier mantissas).

[0057] In some examples, programmers are provided with proprietary or open source libraries or frameworks to enable neural network creation, training, and evaluation. Examples of such libraries include TensorFlow, Microsoft Cognitive Toolkit (CNTK), Caffe, Theano, and Keras. In some examples, programming tools such as integrated development environments provide support for programmers and users to define, compile, and evaluate neural networks.

[0058] The neural network accelerator 180 may be implemented as a custom or application specific integrated circuit (e.g., including a system on chip (SoC) integrated circuit), as a field programmable gate array (FPGA) or other reconfigurable logic, or as a soft processor virtual machine hosted by a physical general purpose processor. The neural network accelerator 180 may include a tensor processing unit 182, a reconfigurable logic device 184, and / or one or more neural processing cores (such as a quantization accelerator 186). The quantization accelerator 186 may be configured with hardware, software, or a combination of hardware and software. As an example, the quantization accelerator 186 may be configured and / or executed using instructions executable on the tensor processing unit 182. As another example, the quantization accelerator 186 may be configured by programming the reconfigurable logic device 184. As another example, the quantization accelerator 186 may be configured using hardwired logic gates of the neural network accelerator 180.

[0059] The quantization accelerator 186 can be programmed to execute a subgraph, an individual layer, or multiple layers of a neural network. For example, the quantization accelerator 186 can be programmed to perform operations on all or a portion of a layer of the NN. The quantization accelerator 186 can access local memory used to store weights, biases, input values, output values, forgetting values, state values, etc. The quantization accelerator 186 can have many inputs, each of which can be weighted by a different weight value. For example, the quantization accelerator 186 can generate a dot product of an input tensor and a programmed input weight for the quantization accelerator 186. In some examples, the dot product can be adjusted by a bias value before being used as an input to an activation function. For example, the output of the quantization accelerator 186 can be stored in a local memory, where the output value can be accessed and sent to different NN processor cores and / or neural network modules 130 or memory 125. Intermediate values ​​in a quantized accelerator can often be stored in smaller or more local memory, while values ​​that may not be needed until later in the training process can be stored in "bulk memory", i.e., larger, more non-local memory (or storage device, such as on an SSD (solid state drive) or hard drive). For example, during the forward propagation of training, once the activation values ​​for the next layer in the NN have been calculated, those values ​​may not be accessible until the propagation through all layers is complete. Such activation values ​​can be stored in such bulk memory.

[0060] The neural network accelerator 180 may include a plurality of 110 quantized accelerators 186 connected to each other via an interconnect (not shown). The interconnect may carry data and control signals between individual quantized accelerators 186, memory interfaces (not shown), and input / output (I / O) interfaces (not shown). The interconnect may transmit and receive signals using electrical, optical, magnetic, or other appropriate communication technologies, and may provide communication connections arranged according to a plurality of different topologies depending on a particular desired configuration. For example, the interconnect may have a crossbar switch, a bus, a point-to-point bus, or other appropriate topology. In some examples, any of the plurality of quantized accelerators 186 may be connected to any other core, while in other examples, some cores are only connected to a subset of other cores. For example, each core may be connected only to the nearest 4, 8, or 10 neighboring cores. The interconnect may be used to transmit input / output data to and from the quantized accelerator 186, as well as to transmit control signals and other information signals to and from the quantized accelerator 186. For example, each of the quantization accelerators 186 may receive and transmit a semaphore indicating the execution status of an operation currently being performed by each of the corresponding quantization accelerators 186. In addition, matrix and vector values ​​may be shared between the quantization accelerators 186 via the interconnect. In some examples, the interconnect is implemented as a line connecting the quantization accelerator 186 and a memory system, while in other examples, the core interconnect may include circuit devices, switches and / or routing components (including active signal drivers and repeaters), or other appropriate circuit devices for multiplexing data signals on (multiple) interconnect lines. In some examples of the disclosed technology, signals transmitted within and to / from the neural network accelerator 180 are not limited to full-swing electrical digital signals, but the neural network accelerator 180 may be configured to include differential signals, pulse signals, or other appropriate signals for transmitting data and control signals.

[0061] In some examples, the quantization-enabled system 110 may include an optional quantization emulator that emulates the functionality of the neural network accelerator 180. The neural network accelerator 180 provides functionality that can be used to convert data represented in full-precision floating point format in the neural network module 130 to quantized format values. The neural network accelerator 180 can also perform operations using the quantized format values. This functionality will be discussed in more detail below.

[0062] The neural network module 130 can be used to specify, train, and evaluate a neural network model using a tool flow that includes a hardware-agnostic modeling framework 131 (also referred to as a native framework or machine learning execution engine), a neural network compiler 132, and a neural network runtime environment 133. The memory includes computer executable instructions for the tool flow that includes the modeling framework 131, the neural network compiler 132, and the neural network runtime environment 133. The tool flow can be used to generate neural network data representing all or part of a neural network model, such as the following about Figure 2 Neural network model 200 discussed. It should be noted that although the tool flow is described as having three separate tools (131, 132, and 133), in various examples, the tool flow may have fewer or more tools. For example, the functionality of different tools (131, 132, and 133) may be combined into a single modeling and execution environment. In other examples of deploying a neural network accelerator, such a modeling framework may not be included.

[0063] Neural network data may be stored in memory 125, which may include: local memory 126, which is typically implemented as static read-only memory (SRAM), embedded dynamic random access memory (eDRAM), implemented in latches or flip-flops in a register file, block RAM, or other suitable structures; and mass storage 127, which is typically implemented in a memory structure that supports larger but often slower access than local memory 126. For example, the mass storage may be off-chip DRAM, network accessible RAM, an SSD drive, a hard drive, or a network accessible storage device. Depending on the specific memory technology available, other memory structures (including the aforementioned structures described for local memory) may be used to implement the mass storage. Neural network data may be represented in one or more formats. For example, neural network data corresponding to a given neural network model may have a different format associated with each corresponding tool of a tool flow. In general, neural network data may include descriptions of nodes, edges, groups, weights, biases, activation functions, and / or tensor values. As a specific example, neural network data may include source code, executable code, metadata, configuration data, data structures, and / or files for representing a neural network model.

[0064] The modeling framework 131 can be used to define and use neural network models. As an example, the modeling framework 131 may include predefined APIs and / or programming primitives that can be used to specify one or more aspects of a neural network model. The predefined APIs may include lower-level APIs (e.g., activation functions, cost or error functions, nodes, edges, and tensors) and higher-level APIs (e.g., layers, convolutional neural networks, recurrent neural networks, linear classifiers, etc.). "Source code" can be used as input to the modeling framework 131 to define the topology of the graph of a given neural network model. In particular, the APIs of the modeling framework 131 can be instantiated and interconnected within the source code to specify complex neural network models. Data scientists can create different neural network models by using different APIs, different numbers of APIs, and interconnecting APIs in different ways.

[0065] In addition to the source code, the memory 125 may also store training data. The training data includes a set of input data for application to the neural network model 200 and an expected output from the neural network model for each corresponding data set of the input data. The modeling framework 131 may be used to train the neural network model using the training data. The output of the training is the weights and biases associated with each node of the neural network model. After the neural network model is trained, the modeling framework 131 may be used to classify new data applied to the trained neural network model. Specifically, the trained neural network model uses the weights and biases obtained from the training to perform classification and recognition tasks on data that was not used to train the neural network model. The modeling framework 131 may use the CPU 120 and a dedicated processor (e.g., GPU 122 and / or neural network accelerator 180) to execute the neural network model with improved performance compared to using only the CPU 120. In some examples, the performance may potentially achieve real-time performance for certain classification tasks.

[0066] The compiler 132 analyzes the source code and data provided for the neural network model (e.g., examples used to train the model) and transforms the model into a format that can be accelerated on the neural network accelerator 180, which will be described in more detail below. Specifically, the compiler 132 transforms the source code into executable code, metadata, configuration data, and / or data structures for representing the neural network model and memory as neural network data. In some examples, the compiler 132 can divide the neural network model into a portion that uses the CPU 120 and / or GPU 122 (e.g., the neural network model 200) and other portions that can be executed on the neural network accelerator 180 (e.g., a subgraph, an individual layer, or multiple layers of the neural network). The compiler 132 can generate executable code (e.g., a runtime module) for executing the NN assigned to the CPU 120 and for communicating with the subgraph, individual layer, or multiple layers of the neural network assigned to the accelerator 180. The compiler 132 can generate configuration data for the accelerator 180, which is used to configure accelerator resources to evaluate the subgraph assigned to the optional accelerator 180. Compiler 132 may create data structures for storing values ​​generated by the neural network model during execution and / or training and for communication between CPU 120 and accelerator 180. Compiler 132 may generate metadata that may be used to identify subgraphs, edge groupings, training data, and various other information about the neural network model during runtime. For example, metadata may include information for interfacing between different subgraphs or other parts of the neural network model.

[0067] The runtime environment 133 provides an executable environment or interpreter that can be used to train the neural network model during training mode and can be used to evaluate the neural network model in training, reasoning or classification mode. During reasoning mode, input data can be applied to the neural network model input and the input data can be classified according to the training of the neural network model. The input data can be archived data or real-time data.

[0068] Runtime environment 133 may include a deployment tool that can be used to deploy or install all or part of a neural network to neural network accelerator 180 during deployment mode. Runtime environment 133 may also include a scheduler that manages the execution of different runtime modules and the communication between the runtime modules and neural network accelerator 180. Thus, runtime environment 133 can be used to control the flow of data between nodes modeled on neural network module 130 and neural network accelerator 180.

[0069] In one example, the neural network accelerator 180 receives and returns the normal precision value 150 from the neural network module 130. Figure 1As illustrated in , the quantization accelerator 186 can use quantized floating points to perform a large number of its operations, and the interface between the quantization accelerator 186 and the neural network module 130 can use full precision values ​​to convey information between modules. Normal precision values ​​can be represented in 16-bit, 32-bit, 64-bit or other appropriate floating point formats. For example, a value representing a portion of a neural network can be received, including edge weights, activation values, or other appropriate parameters for quantization. The normal precision value 150 is provided to a normal precision floating point to quantized floating point converter 152, which converts the normal precision value to a quantized value. Then a quantized floating point operation 154 can be performed on the quantized value. Then the quantized floating point to normal floating point converter 156 that produces a normal precision floating point value can be used to convert the quantized value back to a normal floating point format. As a specific example, the quantization accelerator 186 can be used to accelerate a given layer of a neural network, and can use quantized floating point operations to perform vector-vector, matrix-vector, matrix-matrix, and convolution operations, and can use normal floating point precision operations to perform less computationally intensive operations (such as adding bias values ​​or calculating activation functions). Other examples of conversions that may be performed are conversions of quantized and normal precision floating point values ​​to normal floating point or block floating point values ​​with lossy or non-uniform mantissa values ​​or outlier mantissa values. For example, conversion to a block floating point format with non-uniform mantissa values ​​or outlier mantissa values ​​may be performed to compress activation values, and conversion from a block floating point format with non-uniform mantissa values ​​or outlier mantissa values ​​to normal precision floating point or block floating point values ​​may be performed when retrieving the compressed activation values ​​for later use, such as in back-propagation.

[0070] The conversion between normal floating point and quantized floating point performed by converters 152 and 156 is typically performed on a collection of numbers represented as a vector or multi-dimensional matrix. In some examples, additional normal precision operations 158 (including operations that may be desired in a particular neural network implementation) can be performed based on the normal precision format, including adding a bias to one or more nodes of the neural network, applying a hyperbolic tangent function or other such sigmoid function, or a rectification function (e.g., a ReLU operation) to the normal precision value converted back from the quantized floating point format.

[0071] In some examples, quantized values ​​are used and stored only in the logic gates and internal memory of the neural network accelerator 180, and the memory 125 and the storage device 129 store only normal floating point values. For example, the neural network accelerator 180 can quantize the inputs, weights, and activations for the neural network model received from the neural network module 130, and can dequantize the results of the operations performed on the neural network accelerator 180 before passing the values ​​back to the neural network module 130. The values ​​can be passed between the neural network module 130 and the neural network accelerator 180 using the memory 125, the storage device 129, or the input / output interface (not shown). In other examples, the simulator provides a full simulation of quantization, including storing only one copy of the shared exponent and operating with a reduced mantissa width. Some results may be different on versions that perform basic operations with normal floating point numbers. For example, some examples can check for underflow or overflow conditions for limited quantization bit widths (e.g., three, four, or five-bit wide mantissas).

[0072] A large portion of the computational cost of a DNN is vector-vector, matrix-vector, and matrix-matrix multiplications and / or convolutions. These operations are quadratic in the input size, while operations such as bias addition and activation functions are linear in the input size. Therefore, in some examples, quantization is only applied to matrix-vector multiplication operations, which are implemented on the neural network accelerator 180. In these examples, all other operations are done in a normal precision format, such as float16. Therefore, from the user or programmer's perspective, the quantization-enabled system 110 accepts and outputs normal precision float16 values ​​from / to the neural network module 130 and outputs float16 format values. All conversions to and from block floating point formats can be hidden from the programmer or user. In some examples, the programmer or user can specify certain parameters for the quantization operation. In other examples, the quantization operation can utilize the block floating point format to reduce computational complexity, as described below with respect to Figure 3 discussed.

[0073] Neural network accelerator 180 is used to accelerate the evaluation and / or training of a neural network graph or subgraph, typically with increased speed and reduced latency that is not achievable when evaluating the subgraph using only CPU 120 and / or GPU 122. In the illustrated example, the accelerator includes a tensor processing unit (TPU) 182, a reconfigurable logic device 184 (e.g., contained in one or more FPGAs or programmable circuit structures), and / or a quantization accelerator 186, but any suitable hardware accelerator for modeling a neural network may be used. Accelerator 180 may include configuration logic that provides a soft CPU. The soft CPU oversees operations of the graph or subgraph accelerated on accelerator 180 and may manage communications with neural network module 130. The soft CPU may also be used to configure logic and control loading and storing data from RAM on the accelerator (e.g., in block RAM within an FPGA).

[0074] In some examples, parameters of neural network accelerator 180 may be programmable. Neural network accelerator 180 may be used to prototype training, inference, or classification of all or a portion of neural network model 200. For example, quantization parameters may be selected based on accuracy or performance results obtained by prototyping a network within neural network accelerator 180. After selecting a desired set of quantization parameters, the quantization model may be programmed into accelerator 180 to perform further operations.

[0075] The compiler 132 and runtime environment 133 provide a fast interface between the neural network module 130 and the neural network accelerator 180. In practice, the user of the neural network model may not be aware that a portion of the model is being accelerated on the provided accelerator. For example, node values ​​are typically propagated in the model by writing tensor values ​​to a data structure that includes an identifier. The runtime environment 133 associates a subgraph identifier with the accelerator and provides logic for transforming messages to the accelerator to transparently write values ​​for weights, biases, and / or tensors to the neural network accelerator 180 without program intervention. Similarly, the values ​​output by the neural network accelerator 180 can be transparently sent back to the neural network module 130 along with a message that includes an identifier of the receiving node at the server and a payload that includes values ​​such as weights, biases, and / or tensors that are sent back to the entire neural network model.

[0076] Figure 2The figure shows a simplified topology of a deep neural network (DNN) that can be used to perform neural network applications, such as enhanced image processing, using the disclosed BFP implementation. One or more processing layers can be implemented using the disclosed techniques for quantization and BFP matrix / vector operations, including being implemented using one or more of the multiple neural network quantization accelerators 186 in the above-mentioned quantization-enabled system 110. It should be noted that the application of the neural network implementation disclosed herein is not limited to DNNs, but can also be used with other types of neural networks such as convolutional neural networks (CNNs), including implementations of artificial neural networks with long short-term memory (LSTM) or gated recurrent units (GRU) or other suitable artificial neural networks that can be suitable for use with the BFP methods and apparatus disclosed herein.

[0077] DNN can operate in at least two different modes. Initially, DNN can be trained in training mode and then used as a classifier in inference mode. During training mode, a training data set can be applied to the input of DNN, and various parameters of DNN can be adjusted so that when training is completed, DNN can be used as a classifier. Training includes: performing forward propagation of training input data, calculating loss (e.g., determining the difference between the output of DNN and the expected output of DNN), and performing back propagation through DNN to adjust the parameters of DNN (e.g., weights and biases). When the architecture of DNN is suitable for classifying training data, the parameters of DNN will converge and training can be completed. After training, DNN can be used in inference mode. Specifically, training or non-training data can be applied to the input of DNN and forward propagated through DNN so that input data can be classified by DNN.

[0078] like Figure 2 As shown in , a first node set 210 (including nodes 215 and 216) forms an input layer. Each node of set 210 is connected to each node in a first hidden layer formed by a second node set 220 (including nodes 225 and 226). The second hidden layer is formed by a third node set 230, including node 235. The output layer is formed by a fourth node set 240 (including node 245). In the example neural network model 200, the nodes of a given layer are fully interconnected with the nodes of its adjacent (multiple) layers. In other words, a layer may include nodes that have common inputs with other nodes of the layer and / or provide outputs to a common destination of other nodes of the layer. In other examples, a layer may include a subset of nodes that have common inputs with other nodes of the layer and / or provide outputs to a subset of common destinations of other nodes of the layer.

[0079] During forward propagation, each of the nodes produces an output by applying a weight to each input generated from the previous node and collecting the weights to produce an output value. In some examples, each individual node may apply an activation function (σ) and / or a bias (b). In general, a suitably programmed processor or FPGA may be configured to implement the nodes in the depicted neural network model 200. In some example neural networks, the output function f(n) of the hidden combined node n may produce an output mathematically expressed as follows:

[0080]

[0081] where w i is applied to (multiplied by) the input edge x i , b is the bias value for node n, σ is the activation function for node n, and E is the number of input edges to node n. In some examples, the activation function produces a continuous activation value (represented as a floating point number) between 0 and 1. In some examples, the activation function produces an activation value of binary 1 or 0, depending on whether the sum is above or below a threshold.

[0082] A given neural network may include thousands of individual nodes, and therefore performing all calculations for the nodes in normal precision floating point may be computationally expensive. Implementations for computationally more expensive solutions may include hardware that is larger and consumes more energy than implementations for computationally less expensive solutions. However, performing operations using quantized floating point can potentially reduce the computational complexity of the neural network. A simple implementation using only quantized floating point may significantly reduce the computational complexity, but due to errors introduced by quantization, the implementation may have difficulty converging during training and / or correctly classifying input data. However, the quantized floating point implementation disclosed herein can potentially improve the accuracy of some calculations while also providing the benefit of reducing the complexity associated with quantized floating point.

[0083] DNNs may include nodes that perform operations in quantized floating point. As a specific example, the output function f(n) of a hidden combination node n may produce an output (activation value) that is mathematically expressed as follows:

[0084]

[0085] where w i is applied to (multiplied by) the input edge x i The weight of Q(w i ) is the quantized floating point value of the weight, Q(x i ) is derived from the input edge x i The quantized floating point value of the input, Q -1() is the dequantized representation of the quantized floating point value of the dot product of vectors w and x, b is the bias value for node n, σ is the activation function of node n, and E is the number of input edges of node n. By using quantized floating point values ​​to perform the dot product, the computational complexity can be potentially reduced (compared to using only normal precision floating point values), and by using normal precision floating point values ​​for other operations of the output function, the accuracy of the output function can be potentially improved (compared to using only quantized floating point values).

[0086] A neural network can be trained and retrained by adjusting the values ​​of the components of the output function f(n). For example, by adjusting the weights w for the nodes i Or bias value b, adjusts the behavior of the neural network through corresponding changes in the network output tensor values. For example, the cost function C(w,b) can be used during backpropagation to find appropriate weights and biases for the network, where the cost function can be mathematically described as:

[0087]

[0088] Where w and b represent all weights and biases, n is the number of training inputs, and a is a vector of output values ​​from the network for the input vector of training inputs x. By adjusting the network weights and biases, the cost function C can be driven to a target value (e.g., zero (0)) using various search techniques such as stochastic gradient descent. When the cost function C is driven to the target value, the neural network is said to converge. Similar to the output function f(n), the cost function can be implemented using a quantized precision computer algorithm. For example, vector operations can be performed using quantized floating point values ​​and operations, and non-vector operations can be performed using normal precision floating point values.

[0089] Examples of suitable practical applications for such neural network BFP implementations include, but are not limited to: performing image recognition, performing speech recognition, image classification, translating speech into text and / or other languages, facial or other biometric recognition, natural language processing, automatic language translation, query processing in search engines, automatic content selection, analyzing emails and other electronic documents, relationship management, biomedical informatics, identifying candidate biomolecules, providing recommendations, or other classification and artificial intelligence tasks.

[0090] Network accelerators (such as Figure 1The network accelerator 180 in the DNN can be used to accelerate the calculation of the DNN. As an example, the DNN can be divided into different subgraphs or network layers, which can be accelerated individually. As a specific example, each layer in layers 210, 220, 230 and 240 can be a subgraph or layer accelerated using the same or different accelerators. Quantized floating points can be used to perform computationally expensive calculations of the layer, and normal precision floating points can be used to perform computationally inexpensive calculations of the layer. Normal precision floating points can be used to pass values ​​from one layer to another. By accelerating a set of calculations for all nodes within a layer, some calculations can be reused and the calculations performed by the layer can be reduced compared to accelerating individual nodes.

[0091] In some examples, a set of parallel multiply-accumulate (MAC) units in each convolutional layer can be used to speed up the computation. In addition, parallel multiplier units can be used in the fully connected and dense matrix multiplication stages. A set of parallel classifiers can also be used. This parallelization approach has the potential to further speed up the computation at the expense of increased control complexity.

[0092] Those of ordinary skill in the art who have the benefit of this disclosure will readily appreciate that practical applications of neural network implementations can be used to use different aspects of neural networks, either individually or in various combinations and sub-combinations of one another. For example, the disclosed implementations can be used to implement practical applications of neural network training via gradient descent and / or back-propagation operations for neural networks. Furthermore, the disclosed implementations can be used for evaluation of neural networks.

[0093] Four, Example quantization block floating point format

[0094] Figure 3 300 is a diagram illustrating an example of converting a normal floating point format to a quantized block floating point format, as may be used in certain examples of the disclosed technology. For example, an input tensor of a neural network represented as a normal floating point number (e.g., in a 32-bit or 16-bit floating point format) may be converted to the illustrated block floating point format.

[0095] As shown, many normal floating point format numbers 310 are represented so that each number, such as number 315 or number 316, includes a sign, an exponent, and a mantissa. For example, for the IEEE 754 half-precision floating point format, the sign is represented using one bit, the exponent is represented using five bits, and the mantissa is represented using ten bits. When the floating point format number 310 in the neural network model 200 is converted to a quantized precision, block floating point format number set, there is an exponent value shared by all the numbers of the illustrated set. Therefore, as shown, the block floating point number set 320 is represented by a single exponent value 330, and each number in the number set includes a sign and a mantissa. However, since the illustrated number set has different exponent values ​​in the floating point format, the mantissa of each number can be shifted so that the same or similar numbers are represented in the quantized format (e.g., shifted mantissas 345 and 346).

[0096] In addition, if Figure 3 As shown in , using block floating point format can reduce the computing resources used to perform certain common operations. In the illustrated example, the dot product of two floating point vectors is illustrated in a formal floating point format (350) and a block floating point format (360). For the numbers represented by the normal precision floating point format operation 350, floating point addition is required to perform the dot product operation. In the dot product of floating point vectors, the summation is performed in floating point, which may require shifting to align values ​​with different exponents. On the other hand, for the block floating point dot product operation 360, integer arithmetic can be used to calculate the product to combine the mantissa elements shown. In other words, since the exponent part can be taken into account in the block floating point representation, the multiplication and addition of the mantissa can be completed completely with fixed point or integer representation. As a result, a large dynamic range for a set of numbers can be maintained with a shared exponent while reducing the computational cost by using more integer arithmetic rather than floating point arithmetic. In some examples, the operations performed by the system 110 enabled with quantization can be optimized to utilize the block floating point format.

[0097] In some examples, shared exponent 330 is selected as the largest exponent in the original normal precision numbers in neural network model 200. In other examples, the shared exponent can be selected in a different manner, such as by selecting an exponent that is the mean or median of the normal floating point exponents, or by selecting an exponent that maximizes the dynamic range of values ​​stored in the mantissa when their numbers are converted to a quantized number format. It should be noted that if the shared exponent and the original floating point exponent of the value are different, some bits of the quantized mantissa may be lost. This occurs because the mantissa is shifted to correspond to the new shared exponent.

[0098] There are several possible choices for which values ​​in a block floating-point tensor will share an exponent. The simplest choice is to have the entire matrix or vector share an exponent. However, sharing the exponent at a finer granularity can reduce errors because it increases the likelihood that BFP numbers use a shared exponent that is closer to their original normal floating-point format exponent. Thus, the loss of precision due to discarding mantissa bits (when shifting the mantissa to correspond to the shared exponent) can be reduced.

[0099] For example, consider multiplying a row vector x by a matrix W: y = xW. If each column of W shares exponents, then each dot product xW j (W j is the jth column of W) involves only one shared index for x and one shared index for W j A shared index.

[0100] Figure 4 and Figure 5 The figure shows an alternative block floating point format that can be used for neural network calculations. In the context of neural networks, the core operation is to perform dot products. For example, dot products are the core calculations of matrix multiplication and convolution. Matrix multiplication involves the dot product of the rows / columns of the matrix and the input vector. Convolution involves the dot product of the filter and the input window. In the context of quantized floating points, selecting a set of values ​​for shared exponents may have an impact on the complexity of the computer arithmetic logic used to calculate the dot product. The value of the shared exponent can be referred to as the value within the bounding box. The shape of the bounding box may potentially affect quantization error and computational cost. Although clustering similar amplitude values ​​to create bounding boxes can reduce quantization errors, tracking scaling factors for arbitrary bounding box shapes may be expensive. In contrast, matrices and filters can be divided into bounding boxes that may be more efficient for the operations performed by the neural network. Specifically, the appropriate selection of bounding boxes can reduce the complexity of computer arithmetic circuits used to implement neural network operations. Figure 4 The figure shows a block floating point format that may be well suited for matrices, and Figure 5 The figure shows a block floating point format that may be well suited for convolution filters.

[0101] Figure 4 is a diagram 400 illustrating four alternative block floating point formats that may be used in certain examples of the disclosed technology. As shown, a first format 410 represents an entire array 420 of values ​​that share a single exponent 425. In other words, the entire array 420 of values ​​is packed within a single bounding box.

[0102] In the second format 430, common indices are shared on a per-column basis. In other words, the columns of the matrix are bounding boxes of values. Thus, in this particular example, the block floating point values ​​stored in the even columns 431 of the matrix each share a first single index 432. The block floating point values ​​stored in the odd columns 435 each share a second single index 437. In other examples, each column of the array may be associated with a different shared index. For an eleven-column block of the alternative format, there may be eleven corresponding shared indices, one for each column. In other examples, each row of the array may be associated with a different shared index, or odd and even rows may be associated with a shared common index.

[0103] A third format 450 is shown in which groups of elements in an array share common indices. For example, if a 15×15 value matrix shares indices according to the third format 450, a first set of 5×5 element groups 455 and 456 share a single shared index 458. Similarly, a second 5×5 element group of elements in arrays 460 and 461 may each share a second single index 468. In other examples, each block in the block may be associated with its corresponding shared index. In the example format 450, a 15×15 matrix may have nine shared indices.

[0104] A fourth format 470 is shown in which two shared indices are shared on a tile-plus-per-column basis. Thus, a first set of numbers including numbers 480, 481, and 485 all share a single common exponent 488. Similarly, a second set of numbers including sets 490 and 491 each share a different second single exponent 495. In an alternative example, each of the groups shown may have its own shared exponent.

[0105] In some examples, the computational cost of matrix-vector multiplication can be further reduced by reducing the mantissa width. A large range of values ​​that share a common exponent can be expressed with only a few bits of mantissa. For example, in a representation with a four-bit mantissa and a five-bit exponent, a value can be expressed in [2 -14 0.0012,2 15 1.1112] or approximately [2 -17 , 2 16 ]. In contrast, four-bit fixed-point numbers can only be represented in the range [00012, 11112] or approximately [2 0 ,2 4 ] values ​​in the range.

[0106] Figure 55 is a diagram illustrating three alternative block floating point formats that may be used in certain examples of the disclosed technology. These formats may be useful for two-dimensional convolutions, but these formats may also be generalized to higher-dimensional convolutions. As shown, a first format 510 represents an entire convolution filter 512 of values ​​sharing a single exponent 514. Different convolution filters 516 of values ​​may share a single exponent 518. Thus, the format 510 illustrates that the entire convolution filter may be a bounding box of values.

[0107] In the second format 520, each spatial pixel may be a bounding box, thereby sharing a common index on a per spatial pixel basis along the channel dimension. As shown, spatial pixel values ​​522 share a single index 524 and spatial pixel values ​​526 share a single index 528. For example, for a pixel with dimensions [x, y, c i ], each spatial dimension x and y can be defined with c i The bounding box of the value. Similarly, for dimensions [f x , f y , c i ]c o Convolution filter, for each c o Each pixel of the filter (f x , f y ) can be a i The bounding box size for this method is c i .

[0108] In the third format 530, each spatial pixel may be subdivided along the channel dimension such that the bounding box comprises the subdivisions of the spatial pixel. As shown, the subdivided spatial pixel values ​​532 share a single index 534 and the subdivided spatial pixel values ​​536 share a single index 538. For small c i , the cost of handling scaling factors can be significant. For example, the input image at the first layer of a deep convolutional neural network may have c i =3, corresponding to 3 color channels. Tracking the scaling factor for each triplet can be expensive. In this case, the convolution can be reshaped into a matrix-matrix multiplication to increase the bounding box and reduce the overhead of tracking the bounding box. For example, each convolution filter can be flattened to create a filter with c o Column and f x , *f y , *c i The input matrix X can be created where each row is f x , *f y , *c ivector, corresponding to the input window swept over by the convolution filter. The result Y = XW is a matrix that can be reshaped to match the output of the convolution operation. By reformulating the convolution as a matrix multiplication, the above reference Figure 4 The discussed bounding box strategy can be applied.

[0109] five, Example method for neural network training

[0110] Figure 6 6 is a flow chart depicting a method 600 for training a neural network using a quantized model, as may be implemented in certain examples of the disclosed technology. For example, training the neural network may include iterating through a set of training data, wherein the method 600 is used to update parameters of the neural network during a given iteration of the training data. As an example, the method 600 may be implemented by a quantization-enabled system such as Figure 1 quantization-enabled system 110).

[0111] At process block 610, parameters such as weights and biases of the neural network may be initialized. As one example, the weights and biases may be initialized to random normal precision floating point values. As another example, the weights and biases may be initialized to normal precision floating point values ​​calculated from an earlier training set. The initial parameters may be stored in a memory or storage device of a quantization-enabled system. In one example, the parameters may be stored as quantized floating point values, which may reduce the amount of storage used to store the initial parameters.

[0112] At process block 620, the input values ​​of the neural network may be forward propagated through the neural network. The input values ​​of a given layer of the neural network may be the outputs of another layer of the neural network. These values ​​may be passed between layers from the outputs of one layer to the inputs of the next layer using normal precision floating point numbers. The output function of the i-th layer may include an activation value term that is mathematically described as follows:

[0113] y i =Q -1 (f(Q(y i-1 ), Q(W i ))) (Formula 5)

[0114] where y i-1 is the output from the layer that provides input to the i-th layer, W i is the weight tensor for the i-th layer, f() is the forward function for the layer, Q() is the quantization function, and Q -1() is the dequantization function. The output function of this layer can be a dequantized representation of f(), or alternatively, the output function can include additional terms, such as the addition of an activation function or a bias, which are performed using normal precision floating point (after dequantization) or using quantized floating point (before dequantization). Typically, the inputs, outputs, and parameters of each layer are tensors. Typically, the inputs, outputs, and parameters of each layer will be vectors or matrices. The quantization function Q() converts normal precision floating point values ​​to quantized floating point values. The quantization function can be chosen to take into account the type of input data and the type of operation performed by the i-th layer. For example, when y i and W i is a two-dimensional matrix and the output function consists of taking y i-1 and W i When the cross product of i-1 The quantization function can use the bounding box y i-1 The bounding box of a row or part of a row and used for W i The quantization function can be used including W i The bounding box of a column or a portion of a column. When the bounding box is selected to follow the operator's process, the calculation can be more efficient, making the hardware implementation smaller, faster, faster and more energy-efficient. The dequantization function Q -1 () Converts a quantized floating-point value to a normal-precision floating-point value.

[0115] Also at process block 620, a performance metric may be determined for the neural network. In some examples, the performance metric indicates the accuracy of the neural network, e.g., based on a training data set. In some examples, the performance metric is based on at least one of the following metrics: the number of true positives, the number of true negatives, the number of false positives, or the number of false negatives generated by the neural network. In some examples, the performance metric is based on entropy of one or more layers of the neural network. In some examples, the performance metric is based on a rate-distortion function.

[0116] Also at process block 620, adjusted parameters can be selected for the neural network based at least in part on the performance metric determined at process block 650. For example, any parameters initially selected at process block 610 can be adjusted. For example, the numerical precision of the digital format used to represent activation values ​​of the neural network can be adjusted. For example, the number of bits used to represent the mantissa of the activation values ​​can be increased based on the performance metric.

[0117] At process block 630, a portion of the neural network, such as a layer that has just been forward propagated to the next layer of the neural network, can be compressed and stored in memory. For example, as discussed above in process block 620, the activation values ​​calculated as part of the forward propagation can be compressed and stored in memory. This compression can be mathematically expressed as:

[0118] y ci =C(Q2(y i )) (Formula 6a)

[0119] or

[0120] y ci =C(Q2(f(Q(y i-1 ), Q(W i )))) (Formula 6b)

[0121] where y i is the value generated for the layer by the forward propagation at process block 620, C() is an optional additional compression function (which may include multiple compression operations), Q2() is a quantization function for further compressing the value to a second floating point format (e.g., a format with fewer mantissa bits, a format with fewer exponent bits, a different exponent sharing scheme, a lossy or non-uniform mantissa, or an outlier mantissa value for certain values), and y ci Is a compressed value to be stored in a memory. At least one parameter of the quantization function Q2() can be adjusted based on a performance metric. For example, the output of the layer can be compared with the expected value (e.g., using a confusion matrix or its individual components) to measure the performance of the neural network. In other examples, one or more of the following can be used: mean square error, perplexity, gradient signal-to-noise ratio, or entropy of the neural network. Examples of adjustable activation compression parameters include the number of mantissa bits, the number of exponent bits, exponent sharing schemes, lossy or non-uniform mantissa schemes, the use of outliers for a subset of activation values, or any other publicly adjustable activation compression parameters. Typically, the accuracy of the stored compressed activation values ​​increases as the neural network training proceeds, but in some cases, for partial training, it is possible that the accuracy is reduced from a previous training batch or epoch. In some examples, the Q2() quantization function converts a value from a normal precision format to a smaller quantization format than that used in the quantization layer (as in Formula 6a). In other examples, the Q2() quantization function converts a value directly from the first block floating point format used in the quantization layer to a second block floating point format (as in Formula 6b).

[0122] The compressed activation values ​​are expressed in a second block floating point format that may be different from the first block floating point format used to perform the forward propagation calculations and in at least one of the following ways: having a different mantissa format, having a different exponent format, or having a different exponent sharing scheme. For example, if forward propagation is performed using activation values ​​expressed in an eight-bit format, the values ​​may be converted to a four-bit format by converting the mantissa to a lossy or non-uniform mantissa. As another example, an activation value exponent including a shared exponent in a BBFP format may be converted from a seven-bit format to a five-bit format. As another example, for a four-bit number with one sign bit and three mantissa bits, a three-digit mantissa (which can represent eight values, such as the set {0, 1, 2, 3, 4, 5, 6, 7}) can be converted to a lossy mantissa with a discrete set of mantissa values, such as any of the following sets: {0, 1, 3, 7}, {0, 1, 7}, or {0, 7}, depending on the selected lossy mantissa scheme. The underlying representation of the lossy mantissa may vary. For the first three sets, example sets of corresponding binary representations are {00, 01, 10, 11}; {00, 10, 11}; and {0, 1}, respectively. In some examples, some mantissas can be converted to an outlier mantissa format, where the selected value has additional mantissa bits stored, as will be further described below. In some examples, a multi-value (non-binary) representation can be used for the underlying representation of the lossy mantissa. For example, a multi-level EEPROM or flash memory device can use multi-level storage cells to store the lossy mantissa.

[0123] Values ​​can be converted between two quantization formats used by any appropriate technology. For example, a lookup table, logic gate, arithmetic unit or other circuit device can be used to convert values ​​from normal precision floating point or block floating point format to a floating point format with lower precision. For example, the mantissa can be truncated or rounded to have fewer bits. The mantissa can be converted to a mantissa format with a lossy or non-uniform mantissa. Some mantissa values ​​can be converted to a mantissa format in which additional bits of mantissa information are stored as outliers. In some examples, the floating point format is a block floating point format in which an exponent is shared between two or more values. In other examples, a normal floating point format is used, but with a lossy or non-uniform mantissa and / or outliers.

[0124] In some examples, the compressed block floating point format shares common exponents in a different manner than the format used when performing the forward pass. Aspects of the shared format that can be modified include how the exponents are shared on a per-row, per-column, or per-block basis. In some examples, additional compression can be applied to the compressed block floating point format before storage in memory. Examples of suitable techniques for further compressing activation values ​​in the compressed quantized format include entropy compression (e.g., Huffman coding), zero compression, run-length compression, compressed sparse row compression, or compressed sparse column compression.

[0125] At process block 640, the loss of the neural network can be calculated. For example, the output y of the neural network can be compared with the expected output of the neural network The difference between the output and the expected output can be the input to a cost function that is used to update the parameters of the neural network.

[0126] At process block 650, the activation values ​​stored in the memory are decompressed for backpropagation, and in particular for computing the output error term used in backpropagation for a particular layer. The method may iterate over each layer and decompress the activation values ​​of each layer, perform backpropagation on the layer, and then decompress the activation values ​​of the previous layer. This decompression may be expressed mathematically as:

[0127] y i =Q2 -1 (C -1 (y ci )) (Formula 7a)

[0128] or

[0129] y i =C -1 (y ci ) (Formula 7b)

[0130] where y ci is the compressed activation value retrieved from memory, C -1 () is the decompression function (which may include multiple compression operations), which is the inverse function of the compression function C(), Q2 -1 () is a function that converts the quantized value from the second block floating point format to the first block floating point format, and y i is the value generated for the layer by forward propagation at process block 620. For example, after forward propagation is completed for each layer and neural network as discussed above with respect to process blocks 620 and 630, and the loss calculations as discussed above at process block 640, the values ​​are typically back-propagated through the neural network starting at the output layer of the neural network. Thus, depending on how the compressed quantization format differs from the format used for back-propagation, appropriate transformations of the activation value mantissas, exponents, and / or exponent sharing schemes may be performed. Furthermore, if additional compression such as entropy compression, zero compression, run-length encoding, compressed sparse row compression, or compressed sparse column compression is applied prior to storage in memory, these operations may be reversed prior to performing back-propagation on the layer at process block 660.

[0131] A variety of different techniques can be used to dequantize the mantissa of lower precision. For example, the additional bits in the original (first) floating point format can be filled with ones or zeros (e.g., filled with all zeros, all ones, all selected values) or filled with random values ​​in a deterministic manner. This technique for replacing the additional bits of the original floating point format can be used when dequantizing the mantissa of an outlier or an outlier that has no associated value.

[0132] In the case of dequantizing lossy or non-uniform mantissas, a variety of different techniques can be used. For example, when the non-uniform mantissa set is {0,1,3,7} (e.g., stored in a two-bit mantissa format), those same values ​​can be restored to dequantized values ​​of the first normal precision (e.g., 8, 16, or 32 bits) or block floating point (e.g., 3, 4, 5, or 6 bits) mantissas. In some examples, the dequantized mantissas can be approximate values. For example, for the set {0,1,3,7}, the dequantized mantissas can be {0,1,3,5}. In some examples, the dequantized values ​​are selected in a deterministic manner. For example, for the set {0,1,3,7}, the value 3 can be dequantized to 2 the first time the value is encountered, 3 the second time, 4 the third time, and then back to 2 the fourth time the value is encountered. In some examples, a randomly selected dequantized mantissa can be used. For example, for the set {0,1,3,7}, the non-uniform mantissa value 3 can be transformed into a random value selected from the set {2,3,5,6}, and the non-uniform mantissa value 7 can be transformed into a random value selected from the set {5,4,6,7}. In some examples, the random values ​​are selected according to a uniform distribution, while in other examples, the dequantized mantissa is selected using a normal (Gaussian), Poisson, or other probability distribution.

[0133] At process block 660, the loss of the neural network can be back-propagated through the neural network. During back-propagation, the output error term can be calculated and the weighted error term The output error term can be mathematically described as:

[0134]

[0135] in is the output error term from the layer after the i-th layer, W i is the weight tensor of the i-th layer, g() is the inverse function of the layer, Q() is the quantization function, and Q -1 () is the dequantization function. The inverse function g() can be the inverse function of f(), for i-1 The output error term of this layer can be the dequantized representation of g(), or the output error term can include additional terms performed using normal precision floating point (after dequantization) or using quantized floating point (before dequantization).

[0136] Weight error term Mathematically it can be described as:

[0137]

[0138] in is the weight error term of the i-th layer, is the output error term of the i-th layer, y i is the output of the i-th layer, h() is the inverse function of the layer, Q() is the quantization function, and Q -1 () is the inverse quantization function. The inverse function h() can be the inverse function of f(), for W i-1 The gradient or weight error of h is part of Equation 9. The weight error term for this layer can be a dequantized representation of h(), or the weight error term can include additional terms performed using normal precision floating point (after dequantization) or using quantized floating point (before dequantization). The weight error term can include additional terms performed using normal precision floating point.

[0139] At process block 670, the parameters of each layer may be updated. For example, the weights of each layer may be updated by calculating new weights based on training iterations. As an example, the weight update function may be mathematically described as:

[0140]

[0141] in is the weight error term of the i-th layer, η is the learning rate of the i-th layer of the neural network, W i is the weight tensor of the i-th layer. In one example, the weight update function can be performed using normal precision floating points.

[0142] six, Example environment for performing activation compression

[0143] Figure 7700 is a block diagram depicting an example of a suitable environment for performing activation compression and associated floating point operations between a normal precision floating point domain, a quantized floating point domain, and a floating point domain with a further compressed floating point format. The floating point domain may have an exponent for all values, or a block floating point format with two or more shared exponents. As described above, computationally expensive operations such as vector-vector, vector-matrix, matrix-matrix, and convolution operations may be performed by the quantization layer 710 in the quantized floating point domain. Computationally less expensive operations such as scalar addition and scalar multiplication may be performed outside of the quantization layer 710 in the normal precision floating point domain. With respect to a neural network, the neural network may be divided into layers (such as the quantized layer 710). For the back-propagation portion of training, an output error component 720 may receive activation values ​​and use an additional function such as an error function or an objective function to calculate an output error term In some examples, the output error term is calculated in the normal precision floating point domain This is shown as a solid line by the output error component 720. In other examples, the output error component 720 calculates the error output term in a block floating point format.

[0144] In some examples, the bulk of the computational work within a layer can be performed in the quantized floating point domain, and computationally less expensive operations of the layer, such as adding bias values ​​or calculating activation functions, can be performed in the normal precision floating point domain. Values ​​that interface between layers can be passed from one layer to another in the normal precision floating point domain. By quantizing the input specifically for a given layer, the quantization can be targeted to the operation of that layer to make the operation of that layer more efficient. Specifically, the bounding box of the quantized floating point format can be selected to reduce the complexity of the computer arithmetic circuitry, thereby making the computer logic potentially faster and / or more energy efficient.

[0145] As an example, for the output value y of a given layer i , output error term Weight W i and the weighted error term can be stored in the normal precision floating point domain. During the forward propagation flow, the output values ​​from the earlier layers can be transferred from the normal precision floating point domain to the quantized floating point domain by the quantizer 722 that converts from the normal precision floating point to the quantized floating point. The output values ​​from a given layer can be transferred from the quantized floating point domain to the normal precision floating point domain by the dequantizer 724 that converts from the quantized floating point to the normal precision floating point. The weights for a given layer can be transferred from the normal precision floating point domain to the quantized floating point domain by the quantizer 742. The dequantized value y i Or quantization value Q -1 (y i ) may be sent to a compressor 760 which compresses the value before storing it in mass storage 770.

[0146] The compressor 760 may be a block floating point compressor, or provide other forms of compression to reduce the amount of data stored in the mass storage 770. Typically, the first block floating point format used to represent values ​​during the operation of the quantization layer 710 has a uniform mantissa (e.g., in a normal precision floating point or block floating point format). For example, the first block floating point format used in the quantization layer 710 may have a mantissa with more bits than the second block floating point format, such as: four, five, six, seven, or eight bits; and the second block floating point format may have a non-uniform mantissa with fewer bits than the first block floating point format, such as: three, four, four or five, five or six, or four to six bits, respectively. A person of ordinary skill in the relevant field who has the benefit of the present disclosure should readily understand that the foregoing combination of specific bits of the first block floating point format and the second block floating point format is merely a preferred example, but other precision combinations of mantissa formats may be used in other examples. In some examples, the second block floating point format has a lower precision exponent than the first block floating point format. In some examples, the first block floating point format uses a shared format that is different from the shared format of the public exponent of the second block floating point format. For example, the shared format may differ based on per-row, per-column, or per-block sharing of common exponents for compressed activation values. The precision of the second block floating point format may be adjusted in a number of different ways, including rounding, truncation, and / or shifting of the mantissa value.

[0147] In the illustrated example, the activation values ​​are dequantized 724 to a normal precision format before being converted to the second block floating point format and stored in the mass storage 770. In some examples, the compressor 760 is configured to further compress the activation values ​​in the second block floating point format to provide additional compression by performing at least one or more of the following compression operations: entropy compression, zero compression, run length encoding, compressed sparse row compression, or compressed sparse column compression.

[0148] The bulk memory 770 can be implemented using any suitable memory or storage technology. In some examples, the memory storing temporary values ​​in the quantization layer 710 is typically implemented as static RAM (SRAM), embedded dynamic RAM (eDRAM), implemented register files, block RAM, or other suitable structures, while the bulk memory 770 is typically implemented in a memory structure that supports larger but often slower access, such as off-chip DRAM, network accessible RAM, SSD drive, hard drive, or network accessible storage. A person of ordinary skill in the relevant art having the benefit of the present disclosure will readily appreciate that the type and arrangement of memory used to implement the quantization layer 710 and the bulk memory 770 can be selected depending on desired performance attributes such as cost, energy, and speed.

[0149] The decompressor 780 reads the compressed activation values ​​from the bulk storage 770 and reverses the operations performed by the compressor 760. In examples where additional compression is applied to the quantized values, the values ​​are decompressed prior to dequantization. The values ​​may then be converted from the second block floating point format to a normal precision floating point format (dequantized). The second block floating point format values ​​may be dequantized in a number of different ways, such as described above at processing block 650 for Figure 6 For example, the dequantized non-uniform mantissa value can be converted to the same value in uniform mantissa format. Alternatively, the dequantized mantissa value can be converted to an approximation of the uniform mantissa format, an alternative value selected in a deterministic manner, or a random value according to a distribution selected for a particular non-uniform mantissa value. In an alternative example, the decompressor 780 outputs the value in a quantized block floating point format, as indicated by the dashed line.

[0150] Output error component 720 can receive activation values ​​in normal precision floating point format (as in equation 6a). In an alternative example, output error component 720 can receive activation values ​​in quantized floating point format (as in equation 6b), such as a second block floating point format, as indicated by a dotted line. Output error component 720 calculates the derivative of the activation value used in back propagation. Back propagation is performed using quantization layer 710, as indicated in Figure 700. Output error component 720 can be used to generate performance metrics 725, which are used to select the second block floating point format used by the compressor. Examples of measures that can be used to generate performance metrics 725 include: metrics generated by comparing the output of a neural network with an expected output value. Such metrics can include one or more of the following: true positive rate, true negative rate, positive prediction rate, negative predictive value, false negative rate, false positive rate, false discovery rate, false omission rate, or accuracy rate.

[0151] Those skilled in the relevant art who have benefited from the present disclosure will readily appreciate that other metrics may be used to supplement or add to the expected output value metrics discussed in the previous paragraphs. For example, one or more of the following may be used: mean square error, perplexity, gradient signal-to-noise ratio, or entropy of a neural network.

[0152] As neural network training progresses, the accuracy of the second block floating point format can be adjusted (and often increased). The accuracy can be adjusted in a number of different ways, such as by increasing the number of bits used to express the mantissa value, by adjusting the number of compressed values ​​that have associated outliers, by adjusting the manner and number of bits used to express the exponent, or by adjusting other parameters of the compressed activation values ​​stored in mass storage 740.

[0153] The dashed arrows in diagram 700 depict alternative paths for compressing and decompressing activation values. As shown, as successive layers of the neural network are forward-propagated, generating first activation values ​​in a first block of floating point format, the compressor can convert the quantized activation values ​​directly from the first block of floating point format to a second block of floating point format without converting to a normal precision format, thereby generating compressed activation values.

[0154] During the back propagation flow 730, the output error term from a subsequent layer may be transferred from the normal precision floating point domain to the quantized floating point domain by the quantizer 732. The output error term from a given layer may be transferred from the quantized floating point domain to the normal precision floating point domain by the dequantizer 734. The weights for a given layer may be transferred from the normal precision floating point domain to the quantized floating point domain by the quantizer 742. The weight error term from a given layer may be transferred from the quantized floating point domain to the normal precision floating point domain by the dequantizer 754. As the back propagation proceeds, the quantized activation values ​​in the second block floating point format are either converted to the normal precision format by the quantizer 732 and converted to the first block floating point format, or alternatively, converted from the second block floating point format used to store the activation values ​​in the mass storage 770 to the first block floating point format used by the quantized layer 710. In examples where additional compression (e.g., entropy encoding, zero encoding, or other additional compression schemes) is applied before the data is stored in the mass storage 770, the data may be further decompressed before quantization.

[0155] Figure 8 is to further elaborate on the above Figure 7 800 of the operation of the compressor 760 discussed. As shown in the diagram 800, the activation value y is received from the dequantizer 724. i The dequantizer 724 may receive activation values ​​from a local memory used to implement the quantization layer 710 .

[0156] like Figure 8 As shown in , a set of normal precision floating point values ​​810 is received by a compressor 760, for example from a dequantizer 724. These sets of normal precision floating point values ​​810 are provided to a quantizer 820 that implements a quantization function Q2(), which converts the normal precision values ​​into a second block floating point format that is different from the block floating point format used by the quantization layer 710. The second block floating point format may include non-uniform or lossy mantissas or mantissas with outliers. In the illustrated example, the normal precision floating point values ​​810 each have (for each value) a 10-bit mantissa, a five-bit exponent, and a one-bit sign bit. These normal precision floating point values ​​810 are converted into a set of values ​​Q2(y2) of a second block floating point format 830. i). In the illustrated example, the values ​​expressed in the second block floating point format 830 each have a non-uniform three-bit mantissa, a one-bit sign bit, and all share a four-bit exponent. In some examples, the exponent is not shared by all values ​​in the value set, but can be shared on a per-row, per-column, or per-block basis. Those of ordinary skill in the relevant art having the benefit of the present disclosure will readily appreciate that any suitable floating point format using a lossy or non-uniform mantissa can be used for the second block floating point format 830. For example, for a four-bit number with a sign bit and three mantissa bits, a three-digit mantissa (which can represent eight values, such as the set {0,1,2,3,4,5,6,7}) can be converted into a lossy mantissa with a discrete set of mantissa values, such as any of the following sets: {0,1,3,7}, {0,1,7}, or {0,7}, depending on the selected lossy mantissa scheme. The underlying representation of the lossy mantissa may vary. For the previous three sets, example sets of corresponding binary representations are {00,01,10,11}; {00,10,11}; and {0,1}, respectively. In some examples, a multi-value (non-binary) representation can be used for the underlying representation of the lossy mantissa. For example, a multi-level EEPROM or flash memory device can use multi-level storage cells to store the lossy mantissa.

[0157] In some examples, the set of values ​​in the second block floating point format 830 may be provided to an additional compression unit 840 to be further compressed prior to storage in the mass storage 770. Examples of suitable techniques for further compressing the activation values ​​in the compressed quantized format include entropy compression, zero compression, run length compression, compressed sparse row compression, or compressed sparse column compression.

[0158] Regardless of whether the quantized values ​​in the second block floating point format 830 are subjected to additional compression, the compressed activation value y ci are all stored in the mass storage 770. The mass storage 770 is typically implemented in a memory structure that supports larger but often slower access, such as off-chip DRAM, network accessible RAM, SSD drive, hard drive, or network accessible storage. Those of ordinary skill in the art who have the benefit of this disclosure will readily appreciate that the type and arrangement of memory used to implement the mass storage 770 can be selected depending on desired performance attributes such as cost, energy, and speed.

[0159] Figure 8 Also illustrated is an alternative example apparatus 860 for storing values ​​in a second block of floating point format, in which, instead of receiving values ​​in a normal precision floating point format, quantized values ​​are received directly from the quantization layer 710. In this alternative example apparatus 860, the quantized values ​​f(Q(y)) are received directly from the quantization layer of the first block of floating point format 870. i-1)), where each value in the value set has a seven-bit mantissa, a sign bit, and a shared six-bit exponent. In some examples, the shared exponent is not shared by all values ​​in the set, but can be shared with only certain values ​​on a per-row, per-column, or per-block basis. These values ​​in the first block floating point format can be directly converted to the second block floating point format Q2(f(Q(y i-l Before being stored in the mass storage 770 , the set of values ​​of the second block floating point format 830 may also be provided to an additional compression unit 840 .

[0160] Those of ordinary skill in the relevant art having the benefit of this disclosure will readily appreciate that the illustrated normal precision and block floating point formats are not limited to the particular selections of mantissa, exponent, and sign illustrated, and that the scheme for sharing the exponent between values ​​may also vary depending on the desired aspects for a particular implementation of compressor 760. Furthermore, values ​​stored in mass storage 770 may be decompressed by reversing the operations shown in diagram 800.

[0161] Many techniques can be used to dequantize lossy or non-uniform mantissas. For example, the additional bits in the original (first) floating point format can be filled with ones or zeros in a deterministic manner (e.g., filled with all zeros, all ones, all values ​​selected) or filled with random values. When dequantizing outliers or mantissas without associated outliers, this technique for replacing the additional bits of the original floating point format can be used.

[0162] Many techniques can be used to dequantize lossy or non-uniform mantissas. For example, when the set of non-uniform mantissas is {0,1,3,7} (e.g., stored in a two-bit mantissa format), those same values ​​can be restored to dequantized values ​​of the first normal precision (e.g., 8, 16, or 32 bits) or block floating point (e.g., 3, 4, 5, or 6 bits) mantissas. In some examples, the dequantized mantissas can be approximate values. For example, for the set {0,1,3,7}, the dequantized mantissas can be {0,1,3,5}. In some examples, the dequantized values ​​are selected in a deterministic manner. For example, for the set {0,1,3,7}, the value 3 can be dequantized to 2 the first time the value is encountered, 3 the second time, 4 the third time, and then back to 2 the fourth time the value is encountered. In some examples, a randomly selected dequantized mantissa can be used. For example, for the set {0,1,3,7}, the non-uniform mantissa value 3 can be transformed into a random value selected from the set {2,3,5,6}, and the non-uniform mantissa value 7 can be transformed into a random value selected from the set {5,4,6,7}. In some examples, the random values ​​are selected according to a uniform distribution, while in other examples, the dequantized mantissa is selected using a normal (Gaussian), Poisson, or other probability distribution.

[0163] seven, Example method for determining and using performance metrics to tune activation compression

[0164] As neural network training progresses, performance metrics can be used to determine when to adjust the compression format used to store activation values. For example, as training progresses and the accuracy of the neural network improves, the second floating point format used to store activation values ​​before backpropagation can be adjusted to have higher precision to help the neural network converge as training progresses. An example of a performance metric that can be used to determine when to adjust neural network parameters is entropy. Fig. 9 Graph 900 of FIG. 9 illustrates an example of using entropy to calculate a performance metric. In graph 900 , the x-axis is a square of the variance σ for a given distortion D. 2 The ratio of , and the y-axis is the rate distortion function R(D). The rate distortion function indicates the minimum number of bits that should be used to represent the data for the relevant layer of the neural network. An example of a suitable rate distortion function R(D) for a memoryless Gaussian source with squared error distortion is provided by Equation 11:

[0165]

[0166] The first solid line 910 in the graph indicates the boundary at which, for data with a certain level of variance, reducing the number of bits will result in information loss according to Shannon's theorem. As training progresses and the variance of the values ​​in the neural network increases, the boundary will move, as indicated by the second dashed line 920 in the graph. Therefore, as shown, as the variance and entropy of the data increase, more bits should be used to represent the values ​​of the neural network with sufficient accuracy.

[0167] Those of ordinary skill in the relevant art who have the benefit of the present disclosure will readily appreciate that other suitable rate-distortion functions or other performance metrics may be used. For example, a function based on entropy or entropy change for one or more layers or all layers of a neural network may be used as a performance metric. In other examples, an accuracy-based metric may be used as a performance metric.

[0168] Accuracy-based metrics use known results for a training data set and compare the known results with the values ​​obtained when using the neural network being trained. The values ​​in the confusion matrix (also known as the error matrix) are used to form various accuracy functions. The values ​​generated by the neural network are compared with the known results to generate the following conditions: true positive, where the output of the trained neural network corresponds to a positive result in the known output; true negative, where the output of the trained neural network matches the negative result in the known output; false positive, where the output of the trained neural network indicates a positive result and the corresponding known output indicates a negative result; and false negative, where the output of the trained neural network indicates a negative result and the corresponding known output indicates a positive result. As a simple example, if the known output matches for an image of a cat or dog, when the trained neural network indicates that the image is a cat, but the known output indicates that the image is a dog, this is a false positive. If the trained neural network indicates that the image is not a cat, but the node output indicates that the image is indeed a cat, this is a false negative. Similarly, in the case where both the trained neural network and the known output indicate that the image is a cat or the image is not a cat, it is considered a true positive and a true negative, respectively.

[0169] Examples of accuracy metrics that can be generated from these for confusion matrix values ​​include the following:

[0170] True positive rate, which is the ratio of true positives in the test data to the total number of true positives, also known as sensitivity, recall, or hit rate;

[0171] The true negative rate, which is the ratio of true negatives to this total number, may also be called specificity or selectivity;

[0172] Positive predictive value, which is the ratio of true positives to the sum of all true positives and false positives, may also be called precision;

[0173] Negative predictive value, which is the ratio of true negatives to the sum of all true negatives and false negatives, may also be called negative precision;

[0174] The false negative rate, which is the ratio of false negatives to the total number of true positive examples in the data, can also be called the missed detection rate;

[0175] The false positive rate, which is the ratio of false positives to the total number of true negative examples in the data, may also be referred to as false positives;

[0176] The false discovery rate, which is the ratio of false positives to the sum of all false positives and true positives;

[0177] The false miss rate, which is the ratio of false negatives to the sum of all false negatives and all true negatives; and

[0178] • Precision, which is the ratio of the sum of all true positives and all true negatives in the data to the sum of all positive examples and all negative examples.

[0179] Perplexity is another metric that can be used as a performance measure. As used herein, the term "perplexity" refers to a metric that indicates how "surprising" the output of a neural network is relative to other outputs for a particular training batch or training round. In other words, the perplexity metric indicates how difficult the associated sample is to classify using a neural network. Perplexity can be described by the following formula, where z i is the input sample, p(z i ) is a one-hot vector generated based on the data label, q(z i ) is the prediction of the DNN model, and C is the total number of classifications in a given application:

[0180]

[0181] In some examples, the logarithm of the perplexity value can be used for simplification. At a given training round, a low log perplexity value means that the sample is a typical sample and the neural network model is not "surprised" by the particular sample. In other words, the sample has a relatively low loss value. A high perplexity value indicates that the input sample is difficult to classify using the current DNN model. A person of ordinary skill in the relevant arts having the benefit of the present disclosure will readily appreciate that other appropriate perplexity metrics other than those expressed in Equation 12 may be used.

[0182] A person of ordinary skill in the relevant art having the benefit of this disclosure will readily appreciate that other metrics may be used to determine accuracy and therefore a performance metric for determining when to adjust the parameters of a neural network (including neural network topology and floating point precision). Examples of suitable metrics include, but are not limited to: mean square error, gradient signal-to-noise ratio, or entropy.

[0183] The accuracy rate can be compared to a predetermined value to determine whether to adjust the parameters of the neural network. For example, based on which specific rounds of the training process and whether the accuracy is improving or the rate at which the accuracy is improving, the performance metric can indicate whether to adjust the parameters of the neural network.

[0184] eight, Example non-uniform, lossy mantissa format for activation compression

[0185] Figure 10-13 The figures illustrate examples of non-uniform, lossy mantissa formats that can be used to store compressed activation values ​​in certain examples of the disclosed technology. A person of ordinary skill in the relevant art having the benefit of the present disclosure will readily appreciate that these examples are provided for illustrative purposes, but other lossy non-uniform formats may be used in other examples.

[0186] Fig.10 1 is a diagram 1000 illustrating an example of converting a uniform three-bit mantissa format to a non-uniform four-value lossy mantissa format, which may be referred to as a "mildly lossy format". As shown in the illustrated example, a uniform value set 1010 of a uniform three-bit mantissa format is mapped to a value set 1020 of a non-uniform four-value lossy mantissa format in a non-uniform manner. When the uniform mantissa value is 0, it is converted to a lossy mantissa value 0. When the uniform mantissa value is 1 or 2, the value is converted to a lossy mantissa value 1. When the uniform mantissa value is any one of 3, 4, 5, or 6, the value is converted to a lossy mantissa value 3. When the uniform mantissa value is 7, the value is converted to a lossy mantissa value 7. The base representation may be a four-bit representation, in other words, 0, 1, 3, and 7 may be represented as binary values ​​00, 01, 10, or 11, respectively. Other base representations may also be used.

[0187] Fig.11 1 is a diagram 1100 illustrating an example of converting a uniform three-digit mantissa format to a non-uniform three-value lossy mantissa format, which may be referred to as a "normal lossy format". As shown in the illustrated example, a uniform value set 1110 of a uniform three-digit mantissa format is mapped to a value set 1120 of a non-uniform three-value lossy mantissa format in a non-uniform manner. When the uniform mantissa value is 0, it is converted to a lossy mantissa value 0. When the uniform mantissa value is any one of 1, 2, 3, 4, 5, or 6, the value is converted to a lossy mantissa value 1. When the uniform mantissa value is 7, the value is converted to a lossy mantissa value 7. The base representation may be a two-digit representation, in other words, 0, 1, 7 may be represented as binary values ​​00, 01, 10, or 11, respectively. In other examples, other representations with different number of bits may be used, for example, using 0, 10, and 11 for 0, 1, and 7, respectively. Other base representations may also be used.

[0188] Fig.12 1 is a diagram 1200 illustrating an example of converting a uniform three-digit mantissa format to a non-uniform three-value lossy mantissa format, which may be referred to as a "heavily lossy format". As shown in the illustrated example, a uniform set of values ​​1210 of a uniform three-digit mantissa format is mapped to a set of values ​​1220 of a non-uniform two-value lossy mantissa format in a non-uniform manner. When a uniform mantissa value is 0, it is converted to a lossy mantissa value of 0. When a uniform mantissa value is any one of 1, 2, 3, 4, 5, 6, or 7, the value is converted to a lossy mantissa value of 7. The base representation may be a one-digit representation. Other base representations may also be used.

[0189] Fig.131300 is a diagram illustrating an example of converting uniform values ​​expressed as sign / mantissa values ​​(as a sign bit and a three-digit mantissa) to non-uniform values, which may be referred to as a "full range lossy format". As shown in the illustrated example, a uniform value set 1310 of a uniform format is mapped to a value set 1320 of a non-uniform five-value lossy mantissa format in a non-uniform manner. When the sign / mantissa value is -7, it is converted to a lossy mantissa value of -7. When the sign / mantissa value is any one of -6, -5, -4, -3, or -2, it is converted to a lossy mantissa value of -3. When the sign / mantissa value is any one of -1, 0, or 1, it is converted to a lossy mantissa value of 0. When the sign / mantissa value is any one of 2, 3, 4, 5, or 6, it is converted to a lossy mantissa value of 3. When the sign / mantissa value is 7, it is converted to a lossy mantissa value of 7. The base representation for the lossy sign / mantissa value may be, for example, a three-digit representation. Other base representation formats may also be used.

[0190] Nine, Example of outlier format for activation compression

[0191] Fig.14 1400 is a block diagram depicting an example of a suitable environment for performing activation compression and associated floating point operations between a normal precision floating point domain, a quantized floating point domain, and a compressed quantized floating point domain including outliers. Figure 7 In the described environment, computationally expensive operations such as vector-vector, vector-matrix, matrix-matrix, and convolution operations may be performed by the quantization layer 1410 in the quantized floating point domain. Computationally less expensive operations such as scalar addition and scalar multiplication may be performed outside the quantization layer 1410 in the normal precision floating point domain. With respect to a neural network, the neural network may be divided into layers such as the quantization layer 1410. For the back-propagation portion of training, the output error component 1420 may receive the activation values ​​and use an additional function such as an error function or an objective function to calculate an output error term In some examples, the output error term is calculated in the normal precision floating point domain This is shown as a solid line by the output error component 1420. In other examples, the output error component 1420 computes the error output term in a block floating point format.

[0192] In some examples, the bulk of the computational work within a layer can be performed in the quantized floating point domain, and computationally less expensive operations of the layer, such as adding bias values ​​or calculating activation functions, can be performed in the normal precision floating point domain. Values ​​that interface between layers can be passed from one layer to another in the normal precision floating point domain. By quantizing the input specifically for a given layer, the quantization can be targeted to the operation of that layer to make the operation of that layer more efficient. Specifically, the bounding box of the quantized floating point format can be selected to reduce the complexity of the computer arithmetic circuitry, thereby making the computer logic potentially faster and / or more energy efficient.

[0193] As an example, for the output value y of a given layer i , output error term Weight W i and the weighted error term can be stored in the normal precision floating point domain. During the forward propagation flow, the output values ​​from the earlier layers can be transferred from the normal precision floating point domain to the quantized floating point domain by the quantizer 1422 that converts from the normal precision floating point to the quantized floating point. The output values ​​from a given layer can be transferred from the quantized floating point domain to the normal precision floating point domain by the dequantizer 1424 that converts from the quantized floating point to the normal precision floating point. The weights for a given layer can be transferred from the normal precision floating point domain to the quantized floating point domain by the quantizer 1442. The dequantized value y i Or quantization value Q -1 (y i ) can be sent to a compressor 1460 which compresses the value before storing it in mass storage 1470.

[0194] The compressor 1460 may be a block floating point compressor, or may provide other forms of compression to reduce the amount of data stored in the mass storage 1470. In such an example, the second block floating point format has a mantissa of lower precision than the first block floating point format used to represent values ​​during the operation of the quantization layer 1410. For example, the first block floating point format used in the quantization layer 1410 may have a mantissa with more bits than the second block floating point format, such as: four, five, six, seven, or eight bits; and the second block floating point format may have a mantissa with fewer bits than the first block floating point format, such as: three, four, four or five, five or six, or four to six bits, respectively. In the example discussed below, the first block floating point format has a six-bit mantissa, while the second block floating point format has a four-bit mantissa. It should be readily understood by those of ordinary skill in the relevant art having the benefit of the present disclosure that the aforementioned combination of specific bits of the first block floating point format and the second block floating point format is merely a preferred example, but other precision combinations of mantissa formats may be used in other examples.

[0195] The compressor 1460 includes an outlier quantizer 1465 that identifies some but not all activation values ​​as outliers. The outlier quantizer 1465 may include an index selector that determines a shared index for the compressed activation values. For example, the selector may identify the shared index by determining at least one of a mean (average), a median, and / or a mode for at least a portion of the activation values. In some examples, the selector may identify the shared index by identifying the largest outlier group in the activation value set, and for the remaining groups of activation values ​​that are not identified as being in the largest outlier group, the shared index is determined by selecting the largest index of the remaining groups. In some examples, the index used by the largest number of activation values ​​is selected as the shared index. The outlier quantizer 1465 also identifies a limited number of activation values ​​as having outliers. Those activation values ​​with outliers will store additional data to improve accuracy and / or allow larger or more accurate values ​​to be stored for those specific activation values. For example, a predetermined number of maximum values ​​in the activation value set may be determined as an outlier. As a non-limiting example, the largest eight values ​​of the 256 values ​​may be selected as outlier activation values. The selected largest eight values ​​will have second outlier mantissas and associated exponents that can be combined when the compressed outlier activation values ​​are decompressed.

[0196] The number of outliers can be selected based on performance or hardware properties. In general, storing fewer activation values ​​with outliers reduces storage costs, while storing more activation values ​​with outliers improves the accuracy of the neural network. In some examples, a predetermined number (e.g., one or several) of activation values ​​with a shared index are stored with the outliers. In some examples, the number of activation values ​​stored with the outliers is determined based on each memory access. For example, if 128 activation values ​​are stored per memory access, then each memory access stores a predetermined number (e.g., one or several) of activation values ​​with outliers. In some examples, the number of activation values ​​stored with outliers is predetermined based on the computational granularity. For example, if 16 activation values ​​are generated for a given computational unit (e.g., a single clock cycle or a predetermined number of clock cycles), a predetermined number (e.g., one or several) of activation values ​​with outliers are stored per computational cycle.

[0197] In some examples, the second block floating point format has the same or higher precision exponent as the first block floating point format. In some examples, the second block floating point format has a lower precision exponent than the first block floating point format. In some examples, the first block floating point format uses a shared format that is different from the shared format for the public exponent of the second block floating point format. For example, the shared format can be different based on per-row, per-column, or per-block sharing of the public exponent for the compressed activation values. The precision of the second block floating point format can be adjusted in a variety of different ways, including rounding, truncation, and / or shifting of the mantissa value.

[0198] In the illustrated example, the activation values ​​may alternatively be dequantized 1424 to a normal precision format before being converted to the second block floating point format and stored in the mass storage 1470. In some examples, the compressor 1460 is configured to further compress the activation values ​​in the second block floating point format by performing at least one or more of the following compression operations: entropy compression, zero compression, run length encoding, compressed sparse row compression, or compressed sparse column compression. In some examples, for the activation values ​​y ci The block floating point data is stored in mass storage 1470 with the outlier data o ci In other examples, different memory cells and / or different types of memory cells are used to store the activation value data y ci and outlier data o ci . Outlier data ci The outlier may further include index information indicating which of the activation values ​​the outlier is associated with, and in some examples, an individual index for each of the outliers. In other examples, the outlier may be associated with a particular activation value. ci The exponent can be shared, similar to other block floating point formats.

[0199] The mass memory 1470 can be implemented using any suitable memory or storage technology. In some examples, the memory storing temporary values ​​in the quantization layer 1410 is typically implemented as static RAM (SRAM), embedded dynamic RAM (eDRAM), implemented in a register file, in a block RAM, or in other suitable structures, while the mass memory 1470 is typically implemented in a memory structure that supports larger but often slower access, such as off-chip DRAM, network accessible RAM, SSD drive, hard drive, or network accessible storage. A person of ordinary skill in the art who has the benefit of this disclosure will readily appreciate that the type and arrangement of memory used to implement the quantization layer 1410 and the mass memory 1470 can be selected depending on the desired performance attributes such as cost, energy, and speed.

[0200] The decompressor 1480 reads the compressed activation values ​​from the mass storage 1470 and reverses the operations performed by the compressor 1460. In examples where additional compression is applied to the quantized values, the values ​​are decompressed. The values ​​can then be converted from the second block floating point format to a normal precision floating point format. The decompressor 1480 includes an outlier dequantizer 1485 that converts the values ​​from the second block floating point format back to the first block floating point format or the normal precision floating point format. For those activation values ​​that have associated outliers, the outlier dequantizer 1485 uses the outlier index to identify such values ​​and converts the stored outlier index back to the first block floating point format or the normal precision floating point format. ci With block floating point data y ci The outlier dequantizer 1485 may fill one or more bits of the decompressed mantissa with all-ones or all-zeros values. In other examples, the outlier dequantizer 1485 selects a random value or picks a median value (e.g., a value of three (0011) or a value of four (0100) to replace the discarded four-bit mantissa) to replace the lost mantissa bits of the value that has no associated outliers.

[0201] In an alternative arrangement, the decompressor 1480 outputs values ​​in a quantized block floating point format, as indicated by the dashed line.

[0202] Output error component 1420 can receive activation values ​​in normal precision floating point format (as in equation 6a). In an alternative example, output error component 1420 can receive activation values ​​in quantized floating point format (as in equation 6b), such as a second block floating point format, as indicated by a dotted line. Output error component 1420 calculates the derivative of the activation value used in back propagation. Back propagation is performed using quantization layer 1410, as shown in Figure 1400. Output error component 1420 can be used to generate performance metric 1425, which is used to select the second block floating point format used by the compressor. Examples of measures that can be used to generate performance metric 1425 include metrics generated by comparing the output of a neural network with an expected output value. Such metrics can include one or more of the following: true positive rate, true negative rate, positive prediction rate, negative predictive value, false negative rate, false positive rate, false discovery rate, false omission rate, or accuracy rate.

[0203] Those skilled in the relevant art who have benefited from the present disclosure will readily appreciate that other metrics may be used to supplement or add to the expected output value metrics discussed in the previous paragraphs. For example, one or more of the following may be used: mean square error, perplexity, gradient signal-to-noise ratio, or entropy of a neural network.

[0204] As neural network training progresses, the accuracy parameter of the second block floating point format can be adjusted (and typically increased). The accuracy parameter can be adjusted in a variety of different ways, such as by increasing the number of bits used to express the mantissa value, by adjusting the number of compressed values ​​with associated outliers, by adjusting the manner and number of bits used to express the exponent, or by adjusting other parameters of the compressed activation values ​​stored in mass storage 1470.

[0205] The dashed arrows in diagram 1400 depict alternative paths for compressing and decompressing activation values. As shown, as successive layers of the neural network are forward-propagated, a first block of first activation values ​​in floating point format is generated, and a compressor can convert the quantized activation values ​​to normal precision floating point format before converting to a second block of floating point format, thereby generating compressed activation values.

[0206] During the back propagation flow 1430, the output error term from the next layer can be transferred from the normal precision floating point domain to the quantized floating point domain by the quantizer 1432. The output error term from the given layer can be transferred from the quantized floating point domain to the normal precision floating point domain by the dequantizer 1434. The weight for the given layer can be transferred from the normal precision floating point domain to the quantized floating point domain by the quantizer 1442. The weight error term from the given layer can be transferred from the quantized floating point domain to the normal precision floating point domain by the dequantizer 1454. As the back propagation proceeds, the quantized activation values ​​in the second block floating point format are either converted from the second block floating point format used to store the activation values ​​in the mass storage 1470 to the first block floating point format used by the quantized layer 1410, or alternatively, converted to the normal precision format by the quantizer 1432 and converted to the first block floating point format. In addition, in examples where additional compression (e.g., entropy encoding, zero encoding, or other additional compression schemes) is applied before the data is stored in the mass storage 1470, the data can be further decompressed before quantization.

[0207] Fig.15 1500 is a diagram illustrating an example of converting activation values ​​to a second block floating point format with outliers for selected activation values. For example, the quantization system 110 as discussed above includes Fig.14 Using the compression environment shown in FIG1400 , the quantization system 110 may be used to perform the operations illustrated.

[0208] like Fig.15 As shown in , receiving the activation value set y i 1510. Each activation value in the activation value set has an individual exponent and a six-digit mantissa. The shared exponent is selected by analyzing the activation value set 2 5 In this specific example, it is determined that the selected sharing index 2 51520 will allow the maximum number of activation values ​​1510 to be expressed without losing an unacceptable amount of data. Two outlier activation values ​​1530 and 1531 are also identified that are substantially larger than the other values ​​in the set of activation values. These two outlier activation values ​​will store auxiliary outlier data, allowing larger values ​​than can be expressed using the shared index 1520.

[0209] The mantissas for all activation values ​​1510 in the activation values ​​1510 are shifted based on the shared exponent 1520. For all compressed activation values, a truncated N-bit block floating point mantissa set y is generated for the portion of the shifted mantissa 1540 associated with the shared exponent 1520 for both non-outlier and outlier activation values. ci 1550 (four bits in the illustrated example). For two outlier activation values, for each outlier o ci Generate a mantissa value 1560 of another N additional bits (four bits in the illustrated example) and an outlier index from a plurality of outlier indices 1570. The index associated with each outlier can be determined in a variety of different ways. In some examples, the outlier mantissa 1560 is simply a predetermined number of mantissa bits to the left of the mantissa bits for all activation values ​​in the activation value. In other words, the next N significant bits of the mantissa are selected as the outlier mantissa. In some examples, the outlier mantissa is associated with a shared index for two or more outliers. In some examples, each outlier bit number in the outlier mantissa 1560 is associated with an individual index. In the illustrated example, outliers share the same outlier index 2 1 1575. Each outlier index in outlier index 1570 indicates which activation value the corresponding outlier mantissa is associated with. In the illustrated example, the outliers are associated with activation values ​​from rows 0 and 3. Thus, by storing an index for each outlier, memory is only used to store additional mantissa bits for those activation values ​​that have outliers.

[0210] Fig.16 is to further elaborate on the above Fig.14 1600 of the operation of the compressor 1460 discussed. As shown in the diagram 1600, the quantized activation value f(Q(y i-1 ) 1610. Alternatively, the dequantizer 1424 may generate normal precision floating point values ​​1620 based on activation values ​​received from local memory, which are used to implement the quantization layer 1410.

[0211] like Fig.16As shown in , the quantized values ​​are expressed in a first block floating point format 1610, including a six-bit shared exponent, a sign bit for each individual value, and a six-bit mantissa for each individual value. In some examples, the exponent is not shared by all values ​​in the value set, but can be shared on a per-row, per-column, or per-block basis.

[0212] In an alternative example where a set of normal precision floating point values ​​1620 are received by the compressor 1460, these values ​​are provided to a quantizer 1625, which implements a quantization function Q2() that converts the normal precision values ​​to a second block floating point format that is different from the block floating point format used by the quantization layer 1410. In the illustrated alternative example, the normal precision floating point values ​​1620 each have a 10-bit mantissa, a five-bit exponent (for each value), and a one-bit sign bit. These normal precision floating point values ​​1610 are converted to a set of values ​​Q2(y) in a second block floating point format 1630. i ), each value has a six-bit mantissa, a one-bit sign bit, and all share a six-bit exponent. In some examples, the exponent is not shared by all values ​​in a value set, but can be shared on a per-row, per-column, or per-block basis.

[0213] The outlier quantizer 1465 receives the quantized values ​​of the first block floating point format 1610 and identifies the shared index to be shared by the mantissas of all activation values ​​in the activation value. As shown, the values ​​from the first block floating point format 1610 are converted to the second block floating point format 1620 with shared indexes, and each value has a one-bit sign and a three-bit mantissa. The shared index can be selected in a variety of different ways. For example, the most common index for the activation value set can be selected as the shared index. In some examples, the mean, median, or mode is used to select the shared index. In some examples, the shared index is selected based on the available storage for the outlier. In other words, the selected number of outliers is identified, and the shared index suitable for storing the remaining activation values ​​is selected. In some examples, the index is not shared by all values ​​in the value set, but can be shared on a per-row, per-column, or per-block basis. The second block floating point format also includes data for multiple outliers. For each identified outlier, there is an outlier mantissa. In some examples, there is a shared outlier index for all outliers or a portion of outliers selected on a per row, per column, or per block basis. In some examples, each outlier in the outliers has its own individual index. In addition, outliers are associated with an outlier index, which can be used to identify which activation value the outlier is associated with. The outlier index can be used when decompressing the activated outliers.

[0214] In some examples, the set of values ​​in the second block floating point format 1630 may be provided to additional compression units 1660 and 1661. Examples of suitable techniques for further compressing the activation values ​​in the compressed quantized format include entropy compression, zero compression, run length compression, compressed sparse row compression, or compressed sparse column compression.

[0215] Regardless of whether the quantized values ​​in the second block floating point format 1630 are subjected to additional compression, the compressed values ​​y ci and outliers o ci are stored in mass storage 1470. Mass storage 1470 is typically implemented in a memory structure that supports larger but often slower access, such as off-chip DRAM, network accessible RAM, SSD drive, hard drive, or network accessible storage. One of ordinary skill in the art with the benefit of this disclosure will readily appreciate that the type and arrangement of memory used to implement mass storage 1470 may be selected depending on desired performance attributes such as cost, energy, and speed. In some examples, outliers are stored in different memories or in different portions of mass storage 1470.

[0216] Those of ordinary skill in the relevant art having the benefit of the present disclosure will readily appreciate that the illustrated normal precision and block floating point formats are not limited to a specific choice of mantissa, exponent, and sign bits, and that the scheme for sharing the exponent between values ​​may also vary depending on the desired aspects of a specific implementation of the compressor 1460. Furthermore, the values ​​stored in the mass storage 1470 may be decompressed by reversing the operations shown in the diagram 1600.

[0217] ten, Example Outlier Quantification Device

[0218] Fig.17 1700 is a block diagram outlining an example outlier quantizer 1710 that may be implemented in certain examples of the disclosed technology. For example, the illustrated outlier quantizer may be used, for example, in the quantization-enabled system 110 discussed above to perform the same operations as described above with respect to Figure 14-16 Discuss the operations associated with the outlier quantizer.

[0219] like Fig.17As shown in , an outlier quantizer 1710 receives a plurality of activation values ​​expressed in a floating point format. The floating point format may be a normal precision floating point or a first block floating point format. An outlier selector 1725 analyzes the activation values ​​to determine a sharing index 1730. For each activation value in the activation values ​​1720, a comparator 1740 is used to compare the selected sharing index 1730 with each activation value 1745 in the activation values. If a particular activation value 1745 has a different exponent than the sharing index 1730, a shift controller 1750 selects a shift amount and a shifter 1755 is used to shift the mantissa of the activation value to the left or right depending on the selected sharing index 1730.

[0220] In the example shown, each activation value y ci The four bits of the mantissa are stored in the block floating point memory 1760 as the activation value mantissa 1761. The shift controller 1750 may be used to configure the shifter 1755 to align the four bits of the activation value mantissa to be stored in the block floating point memory 1760 by shifting the mantissa left or right 1757 based on the determination made by the comparator 1740 when comparing the selected shared exponent to the exponent of the particular activation value 1745.

[0221] Outlier selector 1725 also determines which of activation values ​​1720 will be selected to store additional bits of the mantissa as outliers. ci . The shift controller 1750 configures the shifter 1755 so that other bits of the mantissa can be selected as outlier mantissa values. These outlier mantissa values ​​1771 are stored in the outlier memory 1770. The outlier quantizer 1710 also includes an address register 1775 used to select an activation value. The current address of the selected specific activation value 1745 is used to store an indicator 1777, also known as an outlier index, in the outlier memory 1770. In some examples, as shown by the dotted line, the outlier memory 1770 also stores an individual exponent 1779 for each outlier in the outlier. In other examples, the outliers are associated with a shared exponent, or are assumed to be more significant bits of the mantissa of the first block floating point format. For example, in the illustrated example, the three digits to the left of the decimal point can be simply stored in the outlier memory 1770 without explicitly storing the shared exponent. In this case, it may be necessary to additionally shift the specific outlier mantissa according to the exponent of a given activation value.

[0222] Thus, the illustrated outlier quantizer 1710 can maintain higher precision for the selected outliers while still allowing the mantissa of a large number of activation values ​​to be stored in compression. In some examples, the quantizer selects a fixed number of outliers for a set of activation values. For example, every four values ​​out of 64 can be selected as outliers. In other examples, a data structure (e.g., a stack or queue) that allows any number of outliers can be used to store outliers and associated indexes and / or indices. A person of ordinary skill in the relevant field who has the benefit of the present disclosure will readily appreciate that different configurations and numbers of outliers and compressed activation values ​​can be selected. In some examples, the outlier memory 1770 and the block floating point memory 1760 are formed by the same storage device or memory. In other examples, the block floating point memory 1760 can be formed by a memory structure and / or type different from the memory used to form the outlier memory 1770.

[0223] eleven, Example method for storing compressed activation values

[0224] Fig.18 is a flowchart 1800 outlining an example method for storing compressed activation values, as may be performed in certain examples of the disclosed technology. Figure 1 The discussed quantization-enabled system can be used to implement the illustrated method. In some examples, including Figure 7 , Figure 8 , Fig.14 and / or Fig.15 A quantization-enabled system of an example environment of may be used. In some examples, the compressed activation values ​​are stored in a normal precision floating point format. In some examples, the compressed activation values ​​are in a block floating point format. In some examples, the compressed activation values ​​include lossy or non-uniform mantissas. In some examples, the compressed activation values ​​include outlier mantissas.

[0225] At process block 1810, an activation compression format is selected based on a training performance metric for the neural network. In some examples, the training performance metric is generated by calculating the accuracy or change in accuracy of at least one layer of the neural network. In some examples, the training performance metric is evaluated after compressing the activation values ​​in the first activation compression format, continuing the training of the neural network, evaluating the training performance metric for the further trained neural network, and selecting a second, adjusted activation compression format with higher accuracy. In some examples, the adjusted activation compression format has more mantissa bits than the first activation compression format. Some examples, the adjusted activation compression format has an exponential format different from the first activation compression format. In some examples, the first activation compression format includes a mantissa with an outlier, and the adjusted activation compression format does not include a mantissa with an outlier. Some examples, the first activation compression format includes a non-uniform or lossy mantissa, and the adjusted activation compression format does not have a non-uniform or lossy mantissa. In some examples, the first activation compression format activates the compression format as a block floating point format, and the adjusted activation compression format is a normal precision floating point format with fewer or no mantissas sharing a common exponent. In some examples, the performance metric of training is based on comparing the accuracy of the trained neural network to an expected value of an output of the neural network. In some examples, the performance metric of training is based at least in part on one or more of: mean squared error, perplexity, gradient signal-to-noise ratio, or entropy of the neural network.

[0226] At process block 1820, the activation values ​​are stored in a computer readable memory or storage device in an adjusted activation compression format. For example, as the neural network layers are progressively trained, the activation values ​​for a particular layer can be transformed from the format used when performing training to an adjusted activation compression format and stored in memory. In some examples, the activation values ​​are stored in a mass storage device. The stored activation values ​​can then be retrieved after the forward propagation has been performed and decompressed for use in a backpropagation operation. In some examples, additional compression operations, such as entropy compression, are performed before storing the compressed activation values.

[0227] twelve, Example method for storing compressed activation values ​​in a selected floating point format

[0228] Fig.19 is a flowchart 1900 outlining an example method for storing compressed activation values, as may be performed in certain examples of the disclosed technology. Figure 1 The discussed quantization-enabled system can be used to implement the illustrated method. In some examples, including Figure 7 , Figure 8 , Fig.14 and / or Fig.15A quantization-enabled system of an example environment of may be used. In some examples, the compressed activation values ​​are stored in a normal precision floating point format. In some examples, the compressed activation values ​​are in a block floating point format. In some examples, the compressed activation values ​​include lossy or non-uniform mantissas. In some examples, the compressed activation values ​​include outlier mantissas.

[0229] At process block 1910, a neural network is generated that includes activation values ​​expressed in a first floating point format. For example, the floating point format may express the activation values ​​in a normal precision floating point or a block floating point format. In some examples, the neural network is provided by receiving values ​​for a previously trained neural network. In other examples, the neural network is provided by performing training on an untrained neural network. As training of successive layers of the neural network proceeds, the generated activation values ​​may be provided for compression and storage, where they may be retrieved for use during a back-propagation operation. In some examples, a hardware circuit device is configured to provide a floating point compressor. The floating point compressor may be formed by a processor coupled to a memory, a dedicated processor coupled to a memory, a programmed reconfigurable logic device (such as an FPGA), or a device implemented in an application specific integrated circuit (ASIC).

[0230] At process block 1920, a second floating point format is selected based on a performance metric for the neural network. For example, the accuracy of the neural network can be assessed by comparing the output of the trained neural network to the expected output value to determine one or more accuracy metrics. The accuracy metrics can indicate that the accuracy of the neural network has stabilized or diminishing returns are realized with further training of the neural network, so the accuracy of the second floating point format can be increased to further fine-tune the accuracy of the neural network. Thus, early training rounds can be conducted at a faster rate using fewer computer resources, and the accuracy of the compressed activation values ​​can be increased based on the accuracy metrics as training converges the neural network during later training rounds.

[0231] At process block 1930, the activation values ​​are converted to the second floating point format selected at process block 1920. For example, before storing the compressed activation values ​​for later use during back propagation, the mantissa values ​​of the format used to express the activation values ​​during training can be shifted, truncated, or rounded to a lower precision mantissa format. As another example, the mantissa of the first floating point format used for the neural network can be converted to a lossy or non-uniform mantissa format. As another example, the mantissa used for the neural network and the first floating point format can be converted so that some mantissas have associated outliers that are stored with the compressed activation values. As another example, the values ​​of the first floating point format can be in a normal precision floating point format, and the compressed activation values ​​can be in a block floating point format. As another example, a scheme used to share exponents between activation values ​​can be adjusted in a second floating point format.

[0232] At process block 1940, the activation values ​​converted to the second floating point format are stored in a computer readable memory or storage device. For example, these compressed activation values ​​can be retrieved later for use during back propagation. For example, as the neural network layers are progressively trained, the activation values ​​for a particular layer can be transformed from the format used when performing training to an adjusted activation compression format and stored in a memory. In some examples, the activation values ​​are stored in a large capacity memory. The stored activation values ​​can then be retrieved after the forward propagation has been performed and decompressed for use in the back propagation operation. In some examples, additional compression operations, such as entropy compression, are performed before storing the compressed activation values.

[0233] Thirteen, Example method to adjust activation compression parameters

[0234] Fig. 20 2000 is a flowchart outlining an example method of training a neural network, including selecting adjusted activation compression parameters by evaluating the neural network as training progresses. Figure 1 The discussed quantization-enabled system can be used to implement the illustrated method. In some examples, including Figure 7 , Figure 8 , Fig.14 and / or Fig.15 A quantization-enabled system of an example environment of may be used. In some examples, the compressed activation values ​​are stored in a normal precision floating point format. In some examples, the compressed activation values ​​are in a block floating point format. In some examples, the compressed activation values ​​include lossy or non-uniform mantissas. In some examples, the compressed activation values ​​include outlier mantissas.

[0235] At process block 2010, training of the neural network begins by performing forward propagation and / or back propagation for at least one training round. As training proceeds, activation values ​​for one or more layers of the neural network will be determined. These activation values ​​can be stored and then used later during back propagation. In some examples, the neural network is at least partially trained before being received by the quantization-enabled system. In other examples, training begins with an untrained neural network.

[0236] At process block 2020, the accuracy and / or performance attributes of the neural network are evaluated. For example, the accuracy based on the number of true positives, the number of true negatives, the number of false positives, or the number of false negatives can be calculated. In other examples, a combination of these values ​​can be evaluated, such as using a ratio. In some examples, other metrics (such as false discovery rate, false submission rate, or accuracy) are used to evaluate the performance of the neural network. In some examples, the performance of the neural network can be evaluated using performance attributes of the neural network (such as a metric based on mean squared error, perplexity, gradient signal-to-noise ratio, or entropy).

[0237] At process block 2030, an adjusted activation compression parameter is selected based on the evaluation performed at process block 2020. For example, if the accuracy or other performance attribute of the neural network is not improving compared to a predetermined sufficient rate, then the accuracy of the selected, adjusted activation compression parameter can be increased. On the other hand, if the accuracy or other performance attribute of the neural network is still improving with training, then the activation compression parameter can be maintained for an additional number of training cycles or rounds.

[0238] For example, after multiple training batches have been executed or multiple training rounds have been executed, the performance metrics can be evaluated and a determination can be made whether to adjust the compression format used to store activation values. Typically, it is desirable to allow a large search of the neural network solution surface at the beginning of training, and to reduce the neural network solution surface by adjusting the neural network precision parameters. For example, a lower precision numeric format, such as a block floating point format with two or three digits of precision, can be used early in training, and the precision of the numbers used to represent activation values ​​can be increased for successive training operations, as indicated by the performance metrics.

[0239] One or more aspects of the activation compression parameters may be adjusted. For example, one of the operations described below with respect to process blocks 2040-2043 may be used when adjusting the activation compression parameters.

[0240] At process block 2040, the mantissa width used to store the compressed activation value is adjusted. For example, the number of mantissa bits used to store the compressed activation value can be increased to improve the accuracy of the stored activation value. For example, a four-bit, six-bit, or eight-bit mantissa value previously activated in a compressed format can be increased to an eight-bit, 16-bit, or 32-bit mantissa value. Those of ordinary skill in the related art having the benefit of the present disclosure will readily appreciate that the number of mantissa bits can be selected in any suitable manner.

[0241] At process block 2041, the exponent sharing scheme may be adjusted. For example, activation compressed values ​​previously stored in block floating point format may be stored in normal precision floating point format, where no exponents are shared. In other examples, an increased number of exponents are shared by a smaller number of mantissa values ​​by adjusting the shared block of mantissas of the block floating point array.

[0242] At process block 2042, the lossy mantissa format used to store activation values ​​can be adjusted. For example, based on the accuracy or performance attributes of the neural network, a lossy or non-uniform mantissa format can be used in early training cycles of the neural network, while a less severe mantissa format can be used for subsequent training cycles. For example, as training proceeds, the initially used severe lossy format can be adjusted to a normal lossy format, which is used to store activation values ​​in later training cycles, and then a "light" lossy format can be used to store mantissa values ​​for later training. The lossy mantissa format can also be adjusted by storing compressed activation values ​​without a lossy mantissa in subsequent training cycles after previously storing mantissa values ​​in a lossy mantissa format.

[0243] At process block 2043, the outlier format used to store the mantissa of the compressed activation values ​​may be adjusted. For example, the outlier format may be adjusted prior to storing the compressed activation values ​​in the adjusted activation compression format by adjusting the number of outliers stored for the compressed activation value set, adjusting the number of bits used to store the outliers, or adjusting the scheme used to select the outliers.

[0244] At process block 2050, the activation values ​​are compressed to a floating point format having one or more of the adjusted activation compression parameters selected at process blocks 2040-2043. For example, activation values ​​expressed in a first floating point format during training can be converted to an adjusted activation compression format after performing forward propagation for all or a portion of the layers of the neural network. In some examples, the modification to the floating point format can be performed when the values ​​are transmitted as part of the training. For example, activation values ​​transmitted from a first layer of the neural network to a subsequent layer of the neural network can be converted as part of the transmission process.

[0245] At optional process block 2060, additional compression is applied to the activation values ​​converted to the adjusted activation compressed format prior to storage in memory. Examples of suitable techniques for further compressing the activation values ​​in the compressed quantized format include entropy compression (e.g., Huffman encoding), zero compression, run-length compression, compressed sparse row compression, or compressed sparse column compression.

[0246] At process block 2070, the activation values ​​converted to the adjusted activation compression format are stored in a computer-readable memory or storage device. For example, these compressed activation values ​​can be retrieved later for use during back propagation. For example, as the neural network layer is gradually trained, the activation values ​​for a particular layer can be converted from the format used when performing training to the adjusted activation compression format and stored in the memory. In some examples, the activation values ​​are stored in a large capacity memory. The large capacity memory can be implemented with any appropriate storage technology, including, for example, on-chip or off-chip DRAM, network accessible RAM, SSD drive, hard drive, or network accessible storage. In some examples, the large capacity memory is located on an integrated circuit different from the hardware accelerator for processing block floating point values. In some examples, the large capacity memory is located on the same integrated circuit as the hardware accelerator for processing block floating point values. After storing the activation values ​​in the large capacity memory, the calculation for the neural network can proceed to the next layer. For example, once the activation values ​​of a layer have been stored in the large capacity memory, forward propagation can continue for many different layers in the neural network. The stored activation values ​​can be retrieved after the forward pass has been performed and decompressed for use in the backward pass.

[0247] As additional training cycles are performed, the stored compressed activation values ​​can be decompressed and used for the additional training cycles during back-propagation. As the neural network is trained, the activation values ​​can be compressed and decompressed any desired number of times.

[0248] fourteen, Sample computing environment

[0249] Fig.21 The figure illustrates a generalized example of a suitable computing environment 2100 in which the described embodiments, techniques, and technologies may be implemented, including performing activation compression based on performance metrics.

[0250] The computing environment 2100 is not intended to impose any limitations on the scope of use or functionality of the technology, as the technology can be implemented in different general-purpose or special-purpose computing environments. For example, the disclosed technology can be implemented with other computer system configurations, including handheld devices, multiprocessor systems, programmable consumer electronics, network PCs, minicomputers, mainframe computers, etc. The disclosed technology can also be practiced in a distributed computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed computing environment, program modules can be located in local and remote memory storage devices.

[0251] refer to Fig.21 , the computing environment 2100 includes at least one processing unit 2110 and a memory 2120. Fig.21 , this most basic configuration 2130 is included in the dashed line. Processing unit 2110 executes computer executable instructions and can be a real or virtual processor. In a multi-processing system, multiple processing units execute computer executable instructions to increase processing power, and therefore, multiple processors can run simultaneously. Memory 2120 can be a volatile memory (e.g., registers, cache, RAM), a non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two. Memory 2120 stores software 2180, images, and videos that can, for example, implement the technology described herein. The computing environment may have additional features. For example, the computing environment 2100 includes a storage device 2140, one or more input devices 2150, one or more output devices 2160, and one or more communication connections 2170. An interconnection mechanism (not shown) such as a bus, a controller, or a network interconnects the components of the computing environment 2100. Typically, operating system software (not shown) provides an operating environment for other software executed in the computing environment 2100 and coordinates the activities of the components of the computing environment 2100.

[0252] Storage 2140 may be removable or non-removable and include disks, tapes or cassettes, CD-ROMs, CD-RWs, DVDs, or any other medium that can be used to store information and accessed within computing environment 2100. Storage 2140 stores instructions for software 2180, which can be used to implement the techniques described herein.

[0253] The input device(s) 2150 may be a touch input device such as a keyboard, keypad, mouse, touch screen display, pen or trackball, voice input device, scanning device, or other device that provides input to the computing environment 2100. For audio, the input device(s) 2150 may be a sound card or similar device that accepts audio input in analog or digital form, or a CD-ROM reader that provides audio samples to the computing environment 2100. The output device(s) 2160 may be a display, printer, speaker, CD recorder, or other device that provides output from the computing environment 2100.

[0254] The communication connection(s) 2170 enable communication to another computing entity over a communication medium (e.g., a connection network). The communication medium transmits information such as computer executable instructions, compressed graphics information, video, or other data in a modulated data signal. The communication connection(s) 2170 are not limited to wired connections (e.g., Megabit or Gigabit Ethernet, Infiniband, Fiber Channel via electrical or optical fiber connections) but also include wireless technologies (e.g., RF connections via Bluetooth, WiFi (IEEE 802.11a / b / n), WiMax, cellular, satellite, laser, infrared) and other appropriate communication connections for providing network connectivity for the disclosed quantization-enabled computing system. In a virtual host environment, the communication connection(s) may be a virtualized network connection provided by the virtual host.

[0255] Some embodiments of the disclosed methods may be performed using computer executable instructions that implement all or part of the disclosed techniques in the computing cloud 2190. For example, the disclosed methods may be performed on a processing unit 2110 located in the computing environment 2130, or the disclosed methods may be performed on a server located in the computing cloud 2190.

[0256] Computer-readable media is any available media that can be accessed within the computing environment 2100. By way of example and not limitation, for the computing environment 2100, computer-readable media include memory 2120 and / or storage device 2140. It should be readily understood that the term computer-readable storage media includes media for data storage, such as memory 2120 and storage device 2140, but does not include transmission media, such as modulated data signals.

[0257] fifteen, Other Examples of the Disclosed Technology

[0258] Additional examples of the disclosed subject matter are discussed herein based on the examples discussed above.

[0259] A system of one or more computers may be configured to perform a specific operation or action by means of software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform an action when in operation. One or more computer programs may be configured to perform a specific operation or action by means of including instructions that, when executed by a data processing device, cause the device to perform an action. A general aspect includes a computing system comprising: one or more processors. The computing system also includes a memory (mass storage or other forms of memory), including a computer-readable storage device and / or memory. The computing system also includes a floating point compressor formed by at least one processor, the floating point compressor communicating with the mass storage; and the computing system is configured to generate a neural network including activation values ​​expressed in a first floating point format. The computing system may also be configured to select a second floating point format for the neural network based on a performance metric for the neural network. The computing system may also be configured to convert at least one of the activation values ​​to a second floating point format, thereby generating a compressed activation value. The computing system may also be configured to store the compressed activation value in the mass storage using at least one of the processors. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method. The corresponding computer systems, apparatuses, and computer programs can be used to perform any practical application disclosed herein.

[0260] Implementations may include one or more of the following features. The computing system, wherein the second floating point format has a lower precision mantissa than the first floating point format. The computing system is further configured to convert at least one of the compressed activation values ​​to the first floating point format to produce an uncompressed activation value. The computing system may also be configured to perform back propagation on at least one layer of the neural network using the uncompressed activation values ​​to produce a further trained neural network. The computing system may also be configured to perform forward propagation on the further trained neural network to produce an updated activation value. The computing system may also be configured to determine an updated performance metric based on the updated activation value. The computing system may also be configured to select a third floating point format different from the second floating point format based on the updated performance metric. The computing system may also be configured to convert at least one of the updated activation values ​​to the third floating point format to produce a second compressed activation value. The computing system may also be configured to store the second compressed activation value in a mass storage device. In some examples of the computing system, the third floating point format has a higher precision mantissa than the second floating point format. In some examples of the computing system, the second floating point format has at least one mantissa expressed in an outlier mantissa format or a non-uniform mantissa format. In some examples, the computing system is further configured to: determine a difference between an output of a layer of the neural network and an expected output. The computing system may also be configured to select a third floating point format based on the determined difference by increasing the number of mantissa bits used to store the compressed activation value. In some examples of the computing system, the performance metric is based on at least one of the following for the neural network layer: the number of true positives, the number of true negatives, the number of false positives, or the number of false negatives. In some examples of the computing system, the first and second floating point formats are block floating point formats, the second floating point format has a shared format of a public exponent different from the first floating point format, and the shared format is different based on each row, column, or block sharing of the public exponent of the compressed activation value. In some examples of the computing system, the compressor is further configured to further compress the compressed activation value before storage by performing one or more of the following: entropy compression, zero compression, run length encoding, compressed sparse row compression, or compressed sparse column compression. In some examples, the computing system is further configured to perform back propagation for a layer of the neural network by converting the stored compressed activation values ​​into activation values ​​in a first floating point format to produce uncompressed activation values. The computing system may also include performing gradient operations using the uncompressed activation values. In some examples of the computing system, the processor includes at least one of the following: a tensor processing unit, a neural network accelerator, a graphics processing unit, or a processor implemented in a reconfigurable logic array. The computing system may also include a large capacity memory located on an integrated circuit different from the processor.In some examples, the mass memory includes dynamic random access memory (DRAM) or embedded DRAM, and the system also includes a hardware accelerator including a memory that temporarily stores a first activation value of at least a portion of only one layer of the neural network, the hardware accelerator memory including a static RAM (SRAM) or a register file. Implementations of the described techniques may include hardware, methods or processes, or computer software on a computer-accessible medium. Any practical application disclosed herein may be performed using corresponding hardware, methods or processes, or computer software.

[0261] One general aspect includes a method of operating a computing system implementing a neural network, the method comprising: selecting, using the computing system, an activation compression format based on a training performance metric for the neural network. The method also includes: storing compressed activation values ​​for the neural network expressed in the activation compression format in a computer-readable memory or storage device. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method. The method can be used to perform any one or more of the disclosed practical applications.

[0262] Implementations may include one or more of the following features. The method also includes generating a training performance metric by calculating an accuracy or a change in accuracy of at least one layer of the neural network. The method also includes decompressing the stored compressed activation values ​​and training the neural network. The method may also include evaluating the training performance metric for the trained neural network. The method may also include adjusting the activation compression format to an increased precision based on the training performance metric. The method may also include storing the activation values ​​for at least one layer of the trained neural network in an adjusted activation compression form. In an example of the method, the adjusted activation compression format has more mantissa bits than the selected activation compression format. In some examples of the method, the adjusted activation compression format has an exponential format different from the selected activation compression format. In some examples of the method, the selected activation compression format includes an outlier mantissa and the adjusted activation compression format does not include an outlier mantissa. In some examples of the method, the selected activation compression format includes a non-uniform mantissa. In some examples, the selected activation compression format is a block floating point format and the adjusted activation compression format is a normal precision floating point format. In some examples, the training performance metric is based at least in part on one or more of the following for the neural network: true positive rate, true negative rate, positive prediction rate, negative predictive value, false negative rate, false positive rate, false discovery rate, false omission rate, or accuracy rate. In some examples of the method, the training performance metric is based at least in part on one or more of the following for at least one layer of the neural network: mean square error, perplexity, gradient signal-to-noise ratio, or entropy. In some examples, the activation compression format has one of the following mantissa formats: slightly lossy format, normal lossy format, or severely lossy format. One or more computer-readable storage devices or media storing computer executable instructions, which, when executed by a computer, cause the computer to execute the method. Implementations of the described techniques may include hardware, methods or processes, or computer software on a computer-accessible medium. Any one or more of the disclosed practical applications may be performed using corresponding hardware, methods or processes, or computer software.

[0263] One general aspect includes a quantization-enabled system, the system comprising: at least one processor. In some examples, the quantization-enabled system also includes a dedicated processor configured to implement neural network operations in a quantized digital format. The quantization-enabled system also includes a memory (e.g., a mass storage or other suitable memory) configured to receive compressed activation values ​​from the dedicated processor. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0264] One general aspect includes a quantization-enabled system, the system comprising: means for compressing neural network activation values ​​generated during neural network training according to an accuracy parameter. The quantization-enabled system also includes means for storing the compressed neural network activation values. The quantization-enabled system also includes means for evaluating the neural network during neural network training and adjusting the accuracy parameter based on the evaluation. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0265] Implementations may include one or more of the following features. A quantization-enabled system, wherein the means for compressing includes: means for converting activation values ​​from a first floating point format to a second floating point format based on an accuracy parameter. A quantization-enabled system, wherein the means for compressing includes: means for expressing compressed neural network activation values ​​in a lossy mantissa format. A quantization-enabled system, wherein the means for compressing includes: means for expressing compressed neural network activation values ​​in an outlier quantization format. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0266] In view of the many possible embodiments to which the principles of the disclosed subject matter can be applied, it should be recognized that the illustrated embodiments are only preferred examples and should not be considered to limit the scope of the claims to those preferred examples. Instead, the scope of the claimed subject matter is defined by the following claims. We therefore claim as our invention all that falls within the scope of these claims.

Claims

1. A computing system comprising: at least one hardware processor; at least one memory or storage device coupled to the at least one hardware processor; as well as Computer readable instructions or hardware logic that, when executed, cause the computing system to perform operations during a plurality of sets of training epochs for a neural network, the operations comprising: providing a first activation compression format for the neural network, the neural network comprising at least one thousand interconnected nodes, each node comprising an activation function and a set of weights; performing a first set of one or more training rounds of the plurality of sets of training rounds for the neural network using the first activation compressed format, wherein a training round comprises submitting an input of a set of inputs to the neural network, determining an error term, and back-propagating the error term to the nodes of the neural network to update a corresponding set of weights of the neural network; determining, without user intervention, a first value for a training performance metric based on an analysis of outputs obtained in the rounds from corresponding inputs in a set of training data, wherein the training performance metric assesses a response characteristic of the neural network with respect to the corresponding inputs; determining, by the computing system without user intervention, based at least in part on the first value for the training performance metric, that an accuracy of the neural network as indicated by a first training performance metric has not increased or is improving by less than a determined amount; selecting, by the computing system without user intervention, a second activated compression format having a higher accuracy than the first activated compression format based at least in part on a determination that the accuracy of the neural network has not increased, or is improving by less than a determined amount; performing a second set of one or more training rounds of the plurality of sets of training rounds for the neural network; and During execution of the second set of one or more training rounds, compressed activation values ​​for the neural network expressed in the second activation compression format are stored to provide stored compressed activation values.

2. The computing system of claim 1, wherein the operations further comprise: A value for the training performance metric is generated by calculating the accuracy or change in accuracy of at least one layer of the neural network.

3. The computing system of claim 1 , wherein the operations further comprise: decompressing the stored compressed activation values ​​and training the neural network; evaluating the training performance metric for the trained neural network; adjusting the activation compression format to improved accuracy based on the training performance metric; as well as Activation values ​​for at least one layer of the trained neural network are stored in an adjusted compressed form of the activations.

4. The computing system of claim 1, wherein: the second active compressed format has more mantissa bits than the first active compressed format; and The first active compressed format is a block floating point format and the second active compressed format is a normal precision floating point format.

5. The computing system of claim 1, wherein the selected active compression format includes an outlier mantissa and the adjusted active compression format does not include an outlier mantissa.

6. The computing system of claim 1 , wherein the training performance metric is based at least in part on one or more of the following for the neural network: true positive rate, true negative rate, positive predictive rate, negative predictive value, false negative rate, false positive rate, false discovery rate, false omission rate, or accuracy, mean squared error, perplexity, gradient signal-to-noise ratio, or entropy.

7. The computing system of claim 1, the operations further comprising: generating a neural network comprising uncompressed activation values ​​expressed in a first floating point format, the first floating point format having a higher precision than a compressed format of the activations; selecting a second floating point format for the neural network based on a value of the training performance metric; as well as At least one of the activation values ​​is converted to the activation compression format, thereby producing the compressed activation value.

8. A method implemented in a computing system comprising at least one hardware processor and at least one memory or storage device coupled to the at least one hardware processor, the method comprising, during a plurality of training rounds for a neural network: providing a first activation compression format for the neural network, the neural network comprising at least one thousand interconnected nodes, each node comprising an activation function and a set of weights; performing a first set of one or more training rounds of the plurality of sets of training rounds for the neural network using the first activation compressed format, wherein a training round comprises submitting an input of a set of inputs to the neural network, determining an error term, and back-propagating the error term to the nodes of the neural network to update a corresponding set of weights of the neural network; determining, without user intervention, a first value for a training performance metric based on an analysis of outputs obtained in the rounds from corresponding inputs in a set of training data, wherein the training performance metric assesses a response characteristic of the neural network with respect to the corresponding inputs; determining, by the computing system without user intervention, based at least in part on the first value for the training performance metric, that an accuracy of the neural network as indicated by a first training performance metric has not increased or is improving by less than a determined amount; selecting, by the computing system without user intervention, a second activated compression format having a higher accuracy than the first activated compression format based at least in part on a determination that the accuracy of the neural network has not increased, or is improving by less than a determined amount; performing a second set of one or more training rounds of the plurality of sets of training rounds for the neural network; as well as During execution of the second set of one or more training rounds, compressed activation values ​​for the neural network expressed in the second activation compression format are stored to provide stored compressed activation values.

9. The method of claim 8, wherein determining the first value for the training performance metric comprises: An accuracy or a change in accuracy of at least one layer of the neural network is calculated.

10. The method according to claim 8, further comprising: After the second set of one or more training rounds, determining a second value for the training performance metric; evaluating the training performance metric for the trained neural network; adjusting the activation compression format to an adjusted activation compression format that provides improved accuracy based on the second value for the training performance metric; as well as Activation values ​​for at least one layer of the neural network are stored in an adjusted compressed form of the activations.

11. The method according to claim 8, wherein: the second active compressed format has more mantissa bits than the first active compressed format; or The first active compressed format is a block floating point format and the second active compressed format is a normal precision floating point format.

12. The method of claim 8, wherein the first activated compressed format includes an outlier mantissa and the second activated compressed format does not include an outlier mantissa.

13. The method of claim 8, wherein the first activated compressed format includes a non-uniform mantissa.

14. The method of claim 8, wherein the training performance metric is based at least in part on one or more of the following for the neural network: true positive rate, true negative rate, positive predictive rate, negative predictive value, false negative rate, false positive rate, false discovery rate, false omission rate, or accuracy rate.

15. The method of claim 8, wherein the training performance metric is based at least in part on one or more of the following for at least one layer of the neural network: mean squared error, perplexity, gradient signal-to-noise ratio, or entropy.

16. One or more computer-readable storage media or hardware logic comprising: computer readable instructions or hardware logic that, when executed by a computing system comprising at least one hardware processor and at least one memory or storage device coupled to the at least one hardware processor, cause the computing system to set a first activation compression format for a neural network during a plurality of sets of training epochs for the neural network, the neural network comprising at least one thousand interconnected nodes, each node comprising an activation function and a set of weights; computer readable instructions or hardware logic that, when executed by the computing system, cause the computing system to: perform a first set of one or more training rounds of the plurality of sets of training rounds for the neural network using the first activation compressed format, wherein a training round comprises submitting an input of a set of inputs to the neural network, determining an error term, and back-propagating the error term to the nodes of the neural network to update a corresponding set of weights of the neural network; Computer readable instructions or hardware logic that, when executed, cause the computing system to: determine, without user intervention, a first value for a training performance metric based on an analysis of outputs obtained in an episode from corresponding inputs in a set of training data, wherein the training performance metric evaluates a response characteristic of the neural network with respect to the corresponding inputs; Computer readable instructions or hardware logic that, when executed, cause the computing system to: determine, based at least in part on the first value for the training performance metric, that the accuracy of the neural network as indicated by the first training performance metric has not increased, or is improving by less than a determined amount, without user intervention; Computer readable instructions or hardware logic that, when executed, cause the computing system to: based at least in part on determining that the accuracy of the neural network has not increased, or is improving by less than a determined amount, select, without user intervention, a second activated compression format having a higher accuracy than the first activated compression format; Computer readable instructions or hardware logic that, when executed, cause the computing system to: perform a second set of one or more training rounds of the plurality of sets of training rounds for the neural network; as well as Computer readable instructions or hardware logic that, when executed, cause the computing system to: store compressed activation values ​​expressed in the second activation compression format for the neural network during execution of the second set of one or more training rounds to provide stored compressed activation values.

17. The one or more computer-readable storage media or hardware logic of claim 16, comprising: Computer readable instructions or hardware logic that, when executed by the computing system, cause the computing system to: determine a second value for the training performance metric after the second set of one or more training rounds; Computer readable instructions or hardware logic that, when executed by the computing system, cause the computing system to: adjust the activation compression format based on the second value for the training performance metric to provide an adjusted activation compression format, the adjusted activation compression format providing increased accuracy; as well as Computer readable instructions or hardware logic, when executed by the computing system, cause the computing system to: store activation values ​​for at least one layer of the neural network in an adjusted compressed form of the activations.

18. One or more computer-readable storage media or hardware logic according to claim 16, wherein the second active compressed format has more mantissa bits than the first active compressed format; or The first active compressed format is a block floating point format and the second active compressed format is a normal precision floating point format.

19. The one or more computer-readable storage media or hardware logic of claim 16, wherein the first activated compressed format includes an outlier mantissa and the second activated compressed format does not include an outlier mantissa.

20. The one or more computer-readable storage media or hardware logic of claim 16, wherein the first activated compressed format includes a non-uniform mantissa.

21. The one or more computer-readable storage media or hardware logic of claim 16, wherein the training performance metric is based at least in part on: (1) one or more of the following for the neural network: true positive rate, true negative rate, positive predictive value, negative predictive value, false negative rate, false positive rate, false discovery rate, false omission rate, or accuracy; or (2) one or more of the following for at least one layer of the neural network: mean square error, perplexity, gradient signal-to-noise ratio, or entropy.

Images