Neural network activation compression with outlier block floating point

Abstract

Disclosed are apparatus and methods for training a neural network accelerator using a quantized precision data format with outliers, and in particular, disclosed are methods for storing activation values ​​from a neural network in a compressed format for use during forward and backward propagation training of the neural network. In certain examples of the disclosed technology, a computing system is configured to perform a forward propagation for a layer of a neural network to produce a first activation value in a first block of floating point format. In some examples, the activation values ​​generated by the forward propagation are converted by a compressor to a second block of floating point format having a narrower numerical precision than the first block of floating point format. Outliers including additional bits of mantissa and / or exponent are stored in an auxiliary storage device for a subset of the activation values. The compressed activation values ​​are stored in a memory, where the compressed activation values ​​can be retrieved for use during backward propagation.

Description

Background Art

[0001] Machine learning (ML) and artificial intelligence (AI) techniques can be used to solve many complex computing problems, such as recognizing images and voices, analyzing and classifying information, and performing various classification tasks. Machine learning is a field of computer science that uses statistical techniques to enable computer systems to extract higher-level features from training data sets. Specifically, features can be extracted by training models (such as artificial neural networks (NN) or deep neural networks (DNN)). After the model is trained, new data can be applied to the model and the new data can be classified using the trained model (e.g., higher-level features can be extracted). Machine learning models are usually executed on general-purpose processors (also referred to as central processing units (CPUs)). However, training models and / or using models may be computationally expensive, and therefore may not be able to perform feature extraction in real time using general-purpose processors. Therefore, there are a lot of opportunities to improve computer hardware and software to implement neural networks. Summary of the invention

[0002] Apparatus and methods for storing activation values ​​from a neural network in a compressed format for use during forward and backward propagation training of the neural network are disclosed. In certain examples of the compressed format, all activation values ​​are associated with a first mantissa of block floating point having a shared exponent, and some but not all of the activation values ​​are designated as having associated outliers and having a second mantissa associated with the exponent. Thus, activation values ​​associated with outliers can be retrieved from the compressed format by appropriately combining the first mantissa and the second mantissa and the corresponding exponent. Computing systems suitable for employing such neural networks include computers with general purpose processors, neural network accelerators, or reconfigured two logic devices such as field programmable gate arrays (FPGAs). Activation values ​​generated during forward propagation can be "stashed" (temporarily stored in a mass storage device) in a compressed format and retrieved for use during backward propagation. Activation values ​​used during training can be expressed in quantized or block floating point format (BFP). The stored activation values ​​can be in a format that is further compressed than the format used during training.

[0003] In some examples of the disclosed technology, a computer system includes a general and / or special purpose neural network processor, a mass storage device including a computer readable storage device or a computer readable memory, and a block floating point compressor in communication with the mass storage device. As forward propagation occurs during training of the neural network, activation values ​​are generated in a first floating point format. The block floating point is used to convert the activation values ​​to a block floating point format, wherein the converted values ​​have an associated first mantissa and a shared exponent, and a selected number of converted values ​​are exceptions that also have an associated second mantissa and another exponent. The compressed activation values ​​are stored in the mass storage device for use during backward propagation. Each selected exception value can be retrieved by combining a value defined by the first mantissa and the shared exponent with a value defined by the second mantissa and its corresponding exponent.

[0004] The present invention summary is provided to introduce a selection of concepts in a simplified form, which are further described in the detailed description below. The present invention summary is neither intended to identify the key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. The foregoing and other objects, features and advantages of the disclosed subject matter will become more apparent through the following detailed description made with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 is a block diagram of a quantization-enabled system for performing activation compression with exception block floating point values, as may be implemented in certain examples of the disclosed technology.

[0006] Figure 2 is a diagram depicting an example of a deep neural network, as may be modeled using certain example methods and apparatus disclosed herein.

[0007] Figure 3 is a diagram depicting certain aspects of converting a normal floating point format to a quantized floating point format, as may be performed in certain examples of the disclosed technology.

[0008] Figure 4 Depicted are some example block floating point formats that may be used to represent quantized neural network models, as may be used in certain examples of the disclosed technology.

[0009] Figure 5 Depicted are some example block floating point formats that may be used to represent quantized neural network models, as may be used in certain examples of the disclosed technology.

[0010] Figure 6 is a flow chart depicting an example method of training a neural network for use with a quantized model with outliers, as may be implemented in certain examples of the disclosed technology.

[0011] Figure 7 is a block diagram depicting an example environment for implementing activation compression using a narrow block floating point format with outliers, as may be implemented in certain examples of the disclosed technology.

[0012] Figure 8 is a block diagram depicting an example of a subset of data used in activation compression with outliers, as may be implemented in certain examples of the disclosed technology.

[0013] Fig. 9 is a block diagram depicting an example of performing activation compression with outliers, as may be performed in certain examples of the disclosed technology.

[0014] Fig.10 is a block diagram depicting a high-level example of an anomaly quantizer, as may be implemented in certain examples of the disclosed technology.

[0015] Fig.11 is a flow chart outlining an example method of generating activation values ​​having outliers, as may be performed in certain examples of the disclosed technology.

[0016] Fig.12 is a flow chart outlining an example method of generating outliers, as may be performed in certain examples of the disclosed technology.

[0017] Fig.13 is a block diagram illustrating a suitable computing environment for implementing some embodiments of the disclosed technology.

[0018] Figures 14 to 17 is a graph plotting experimental data obtained when training an artificial neural network using activation compression with outliers, as may be performed in certain examples of the disclosed technology. DETAILED DESCRIPTION

[0019] I. General considerations

[0020] The present disclosure is set forth in the context of representative embodiments, which are not intended to be limiting in any way.

[0021] As used in this application, the singular forms "a", "an", and "the" include the plural forms unless the context clearly dictates otherwise. Additionally, the term "includes" means "comprises". Further, the term "coupled" encompasses mechanical, electrical, magnetic, optical, and other practical ways of coupling or linking items together, and does not exclude the presence of intermediate elements between coupled items. Moreover, as used herein, the term "and / or" means any one or combination of items in the phrase.

[0022] The systems, methods, and devices described herein should not be construed as limiting in any way. On the contrary, the present disclosure relates to all novel and non-obvious features and aspects of the various disclosed embodiments, both individually and in various combinations and sub-combinations with each other. The disclosed systems, methods, and devices are not limited to any particular aspect or feature or combination thereof, nor do the disclosed things and methods require that any one or more specific advantages should exist or that problems should be solved. In addition, any feature or aspect of the disclosed embodiments may be used in various combinations and sub-combinations with each other.

[0023] Although for the convenience of presentation, the operation of some methods in the disclosed method is described in a specific sequential order, it should be understood that unless the specific language set forth below requires a specific order, the description mode covers rearrangement. For example, in some cases, the operations described in sequence can be rearranged or performed in parallel. In addition, for simplicity, the accompanying drawings may not show the various ways in which the disclosed things and methods can be used in combination with other things and methods. Additionally, the description sometimes uses terms such as "generate", "generate", "display", "receive", "verify", "execute", "perform", "convert" and "start" to describe the disclosed method. These terms are high-level descriptions of the actual operations performed. The actual operations corresponding to these terms will vary depending on the specific implementation and those skilled in the art can easily distinguish.

[0024] The theories of operation, scientific principles, or other theoretical descriptions proposed herein with reference to the devices or methods of the present disclosure are provided for the purpose of better understanding and are not intended to limit the scope. The devices and methods in the appended claims are not limited to those devices and methods that function in the manner described by such theories of operation.

[0025] Any disclosed method may be implemented as computer executable instructions stored on one or more computer readable media (e.g., computer readable media, such as one or more optical media disks, volatile memory components (such as DRAM or SRAM), or non-volatile memory components (such as hard disk drives)) and executed on a computer (e.g., any commercial computer, including smart phones or other mobile devices including computing hardware). Any computer executable instructions for implementing the disclosed technology and any data created and used during the implementation of the disclosed embodiments may be stored on one or more computer readable media (e.g., computer readable storage media). The computer executable instructions may be, for example, a dedicated software application or part of a software application accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software may be executed, for example, on a single local computer or in a network environment (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network), or other such network) using one or more network computers.

[0026] For clarity, only certain selected aspects of software-based implementations are described. Other details known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any particular computer language or program. For example, the disclosed technology can be implemented by software written in C, C++, Java, or any other suitable programming language. Certain details of suitable computers and hardware are known and need not be elaborated in this disclosure.

[0027] In addition, any software-based embodiment (including, for example, computer-executable instructions for causing a computer to perform any disclosed method) can be uploaded, downloaded, or remotely accessed via suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, an intranet, a software application, cables (including fiber optic cables), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communication means.

[0028] II. Overview of Quantized Artificial Neural Networks

[0029] Artificial neural networks (ANN or "NN" as used throughout this article) are applied to many applications in artificial intelligence and machine learning, including image recognition, speech recognition, search engines, and other suitable applications. Processing of these applications can be performed on a single device (such as a personal computer or mobile phone), but can also be performed in large data centers. At the same time, hardware accelerators that can be used with NN include specialized NN processing units, such as tensor processing units (TPUs) and field programmable gate arrays (FPGAs) that are programmed to accelerate neural network processing. Such hardware devices are deployed in consumer devices as well as in data centers due to their flexible nature and low power consumption per unit of computation.

[0030] Traditionally, NNs are trained and deployed using single-precision floating point (32-bit floating point or float32 format). However, it has been shown that lower precision floating point formats (such as 16-bit floating point (float16) or fixed point formats) can be used to perform inference operations with minimal loss of accuracy. On dedicated hardware (such as FPGAs), reduced precision formats can greatly improve latency and throughput of DNN processing.

[0031] Numbers represented in a normal precision floating point format (e.g., floating point numbers expressed in a 16-bit floating point format, a 32-bit floating point format, a 64-bit floating point format, or an 80-bit floating point format) may be converted to a quantized precision format number, potentially allowing for performance benefits when performing operations. Specifically, NN weights and activation values ​​may be represented in a lower precision quantized format with acceptable levels of error introduced. Examples of lower precision quantized formats include formats with reduced bit widths (including by reducing the number of bits used to represent the mantissa or exponent of a number) and block floating point formats in which two or more numbers share the same single exponent.

[0032] One of the characteristics of computation on FPGA devices is that it typically lacks hardware floating point support. Floating point operations can be performed in a penalty-free manner using flexible logic, but in FPGA implementations, the amount of logic required to support floating point is often prohibitive. Some newer FPGAs have been developed that do support floating point computations, but even on these FPGAs, the same device can produce twice the computational output per unit time when it is used in integer mode. Typically, NNs are created with floating point computations in mind, but when FPGAs are targeted for NN processing, it would be beneficial if the neural network could be expressed using integer arithmetic. Examples of the disclosed technology include hardware implementations of block floating point (BFP), including use of BFP in NN, FPGA, and other hardware environments.

[0033] A typical floating point representation in a computer system consists of three parts: the sign (s), the exponent (e), and the mantissa (m). The sign indicates whether the number is positive or negative. The exponent and mantissa are used in scientific notation as:

[0034] Value = s × m × 2 e (Formula 1)

[0035] Any number can be represented within the limits of the precision of the mantissa. Since the exponent scales the mantissa by powers of 2, just as the exponent does by powers of 10 in scientific notation, extremely large numbers of magnitude can be represented. The precision of the representation is determined by the precision of the mantissa. Typical floating point representations use mantissas that are 10 (float 16), 24 (float 32), or 53 (float 64) bits wide. Magnitude greater than 2 53 An integer of 2 can be approximated using the float64 floating point format, but cannot be represented exactly because there are not enough bits in the mantissa. A similar effect can occur for arbitrary fractions, where the fraction is represented by bits in the mantissa that take values ​​that are negative powers of 2. There are many fractions that cannot be represented exactly because they are irrational numbers in the binary number system. More exact representations are possible in both cases, but they may require the mantissa to contain more bits. Finally, an infinite number of mantissa bits are required to represent some numbers exactly (e.g. ). The 10-bit (half-precision float), 24-bit (single-precision float), and 53-bit (double-precision float) mantissa limits are common tradeoffs between mantissa storage requirements and representation accuracy in general-purpose computers.

[0036] In the case of a block floating point format, a set of two or more numbers uses a single shared exponent, where each number still has its own sign and mantissa. In some examples, the shared exponent is selected as the maximum exponent of the original floating point value. For the purposes of this disclosure, the term block floating point (BFP) means a number system in which a single exponent is shared between two or more values, each value being represented by a sign and mantissa pair (whether there is an explicit sign bit or the mantissa itself is signed). In some examples, all values ​​of one or more rows or columns of a matrix or vector or all values ​​of a matrix or vector can share a common exponent. In some other examples, the BFP representation can be unsigned. In some examples, some but not all of the elements in the matrix or vector BFP representation may include numbers represented as integers, floating point numbers, fixed point numbers, symbols, or other data formats mixed with numbers represented by signs, mantissas, and exponents. In some examples, some or all of the elements in the matrix or vector BFP representation may include complex elements with two or more parts, such as: complex numbers with imaginary parts (a+bi, where ); fractions in polar coordinates (r, θ) or other multi-component elements, including numerators and denominators.

[0037] The parameters of a specific BFP format can be selected to balance precision and storage requirements for a specific implementation. For example, instead of storing an exponent with each floating point number, a group of numbers can share the same exponent. In order to share the exponent while maintaining a high level of accuracy, since the difference in magnitude is expressed in the mantissa, the number should have a nearly identical magnitude. If the difference in magnitude is too large, the mantissa will overflow for a large value, or may be zero ("underflow") for a smaller value. Depending on the specific application, a certain amount of overflow and / or underflow may be acceptable.

[0038] The size of the mantissa can be adjusted to suit a particular application. Such a situation can affect the precision of the number being represented, but potential gains are realized from the reduced representation size. For example, a common single-precision floating point has a size of four bytes, but for some embodiments of the disclosed technology, only two bytes are used to represent the sign and mantissa of each value. In some embodiments, the sign and mantissa of each value can be represented with one byte or less.

[0039] In certain examples of the disclosed technology, the representation expressed above is used to derive the original number from the representation, but only a single exponent is stored for a set of numbers, each of which is represented by a signed mantissa. Each signed mantissa can be represented by two bytes or less, so the memory storage savings is about 2 times compared to a four-byte floating point. In addition, the memory bandwidth requirements for loading and storing these values ​​are also about half of the memory bandwidth requirements of ordinary floating points.

[0040] Neural network operations are used in many artificial intelligence operations. Typically, most of the processing operations performed when implementing a neural network are to perform matrix × matrix or matrix × vector multiplication or convolution operations. Such operations are computational bandwidth and memory bandwidth intensive, where the size of the matrix can be, for example, 1000 × 1000 elements (e.g., 1000 × 1000 numbers, each including a sign, mantissa, and exponent) or larger and there are many matrices used. As discussed herein, BFP technology can be applied to such operations to reduce the computation and memory bandwidth requirements in a given system, whether it is an FPGA, CPU, or another hardware platform. As used herein, the term "element" used in this article refers to a member of such a matrix or vector.

[0041] As used herein, the term "tensor" refers to a multidimensional array that can be used to represent properties of a NN and includes one-dimensional vectors and two-dimensional, three-dimensional, four-dimensional or larger dimensional matrices. As used in this disclosure, unless otherwise stated, a tensor does not require any other mathematical properties.

[0042] As used herein, the term "normal precision floating point" refers to a floating point format with a mantissa, an exponent, and an optional sign and supported natively by a native or virtual CPU. Examples of normal precision floating point formats include, but are not limited to, IEEE 754 standard formats (such as 16-bit, 32-bit, 64-bit) or other processors supported by a processor, such as Intel AVX, AVX2, IA32, x86_64, or 80-bit floating point formats.

[0043] A given number can be represented using different precision (e.g., different quantization precision) formats. For example, a number can be represented in a higher precision format (e.g., float32) and a lower precision format (e.g., float16). Reducing the precision of a number can include reducing the number of digits used to represent the mantissa or exponent of a number. Additionally, reducing the precision of a number can include reducing the range of values ​​of the exponent that can be used to represent a number when multiple numbers share a common exponent. Similarly, increasing the precision of a number can include increasing the number of digits used to represent the mantissa or exponent of a number. Additionally, increasing the precision of a number can include increasing the range of values ​​of the exponent that can be used to represent a number when a group of numbers sharing a common exponent are separated. As used herein, converting a number from a higher precision format to a lower precision format can be referred to as down-conversion or quantization. Converting a number from a lower precision format to a higher precision format can be referred to as up-conversion or dequantization.

[0044] As used herein, the term "quantized precision floating point" refers to a floating point number format in which two or more values ​​of a tensor have been modified to have a lower precision than when the values ​​are represented in ordinary precision floating point. Specifically, many examples of quantized precision floating point representations include block floating point formats in which two or more values ​​of a tensor are represented with reference to a common exponent. Quantized precision floating point numbers can be generated by selecting a common exponent for two, more, or all elements of a tensor and shifting the mantissa of individual elements to match the shared common exponent. In some examples, groups of elements within a tensor can share a common exponent, for example, on a per-row, per-column, per-block, or other basis.

[0045] In one example of the disclosed technology, a neural network accelerator is configured to perform training operations for a layer of a neural network, including forward propagation and backward propagation. The values ​​of one or more neural network layers may be expressed in a quantized format having a lower precision than a normal precision floating point format. For example, a block floating point format may be used to accelerate calculations performed in training and reasoning operations using a neural network accelerator. The use of a quantized format may improve neural network processing by, for example, allowing faster hardware, reduced memory overhead, simpler hardware design, reduced energy usage, reduced integrated circuit area, cost savings, and other technical improvements. It is generally desirable that operations be performed to mitigate noise or other inaccuracies introduced by using a lower precision quantized format. Further, portions of neural network training (such as temporary storage of activation values) may be improved by compressing a portion of these values ​​(e.g., for an input layer, hidden layer, or output layer of a neural network) from a normal precision floating point or from a first block floating point to a lower precision number format. The activation value may be retrieved later for use during, for example, backpropagation during a training phase.

[0046] The input tensor of a given layer can be converted from a normal precision floating point format to a quantized precision floating point format. Tensor operations can be performed using the converted input tensors. The results of the tensor operations can be converted from a block floating point format to a normal precision floating point format. The tensor operations can be performed during a forward propagation mode or a backward propagation mode of a neural network. For example, during the backward propagation mode, the input tensor can be an output error term from a layer adjacent to (e.g., after) a given layer or a weight of a given layer. As another example, during the forward propagation mode, the input tensor can be an output error term from a layer adjacent to (e.g., before) a given layer or a weight of a given layer. The converted result can be used to generate an output tensor of a layer of a neural network, wherein the output tensor is in a normal precision floating point format. In this way, a neural network accelerator can potentially be made smaller and more efficient than a comparable accelerator that uses only a normal precision floating point format. A smaller and more efficient accelerator may have increased computational performance and / or increased energy efficiency. Additionally, a neural network accelerator can potentially have increased accuracy compared to an accelerator that uses only a quantized precision floating point format. By improving the accuracy of the accelerator, the convergence time for training can be reduced and the accelerator can be more accurate in classifying inputs to the neural network. Reducing the computational complexity of the model used can potentially reduce the time to extract features during inference, reduce the time to make adjustments during training, and / or reduce energy consumption during training and / or inference.

[0047] III. Example architecture for activation compression using narrow block floating point

[0048] Figure 11 is a block diagram 100 outlining an example quantization enabling system 110, as may be implemented in certain examples of the disclosed technology, including for use in activation compression utilizing narrow block floating point. Figure 1 As shown in , the quantization-enabled system 110 may include several hardware resources, including a general purpose processor 120 and a special purpose processor (such as a graphics processing unit 122 and a neural network accelerator 180). The processor is coupled to a memory 125 and a storage device 129, which may include volatile or non-volatile memory devices. The processor 120 and the processor 122 execute instructions stored in the memory or storage device to provide a neural network module 130. The neural network module 130 includes a software interface that allows the system to be programmed to implement various types of neural networks. For example, software functions that allow an application to define a neural network, including weights, biases, activation functions, node values, and interconnections between layers of the neural network, may be provided. Additionally, the software functions may be used to define state elements of a recursive neural network. The neural network module 130 may also provide utilities to allow training and retraining of a neural network implemented with the module. Values ​​representing the neural network module are stored in the memory or storage device and are operated on by instructions executed by one of the processors. The values ​​stored in the memory or storage device may be represented using normal precision floating point and / or quantized floating point values.

[0049] In some examples, proprietary or open source libraries or frameworks are provided to programmers to implement neural network creation, training, and evaluation. Examples of such libraries include TensorFlow, Microsoft Cognitive Toolkit (CNTK), Caffe, Theano, and Keras. In some examples, programming tools (such as integrated development environments) provide support for programmers and users to define, compile, and evaluate NNs.

[0050] The neural network accelerator 180 may be implemented as a custom or application-specific integrated circuit (e.g., including a system-on-chip (SoC) integrated circuit), as a field programmable gate array (FPGA) or other reconfigurable logic, or as a soft processor virtual machine hosted by a physical general-purpose processor. The neural network accelerator 180 may include a tensor processing unit 182, a reconfigurable logic device 184, and / or one or more neural processing cores (such as a quantization accelerator 186). The quantization accelerator 186 may be configured in hardware, software, or a combination of hardware and software. As an example, the quantization accelerator 186 may be configured and / or executed using instructions executable on the tensor processing unit 182. As another example, the quantization accelerator 186 may be configured by programming the reconfigurable logic block 184. As another example, the quantization accelerator 186 may be configured using hardwired logic gates of the neural network accelerator 180.

[0051] The quantization accelerator 186 can be programmed to execute a subgraph, an individual layer, or multiple layers of a neural network. For example, the quantization accelerator 186 can be programmed to perform operations on all or part of the layers of the NN. The quantization accelerator 186 can access local memory used to store weights, biases, input values, output values, forgetting values, state values, etc. The quantization accelerator 186 can have many inputs, each of which can be weighted by a different weight value. For example, the quantization accelerator 186 can generate a dot product of an input tensor and a programmed input weight of the quantization accelerator 186. In some examples, the dot product can be adjusted by a bias value before it is used as an input to an activation function. The output of the quantization accelerator 186 can be stored in a local memory, for example, where the output value can be accessed and sent to different NN processor cores and / or neural network modules 130 or memory 125. Intermediate values ​​in the quantization accelerator can typically be stored in a smaller or larger local memory, while values ​​that may not be needed until later in the training process can be stored in a "mass memory", a larger or smaller local memory (or storage device, such as an SSD (solid state drive) or a hard drive). For example, during the forward propagation of training, once the activation values ​​for the next layer in the NN have been calculated, these values ​​cannot be accessed until the propagation through all layers is complete. Such activation values ​​can be stored in such a large capacity memory.

[0052] The neural network accelerator 180 may include a plurality of 110 quantized accelerators 186 connected to each other via an interconnect (not shown). The interconnect may carry data and control signals between individual quantized accelerators 186, memory interfaces (not shown), and input / output (I / O) interfaces (not shown). The interconnect may transmit and receive signals using electrical, optical, magnetic, or other suitable communication technologies, and may provide communication connections arranged according to a plurality of different topologies depending on a particular desired configuration. For example, the interconnect may have a crossbar switch, a bus, a point-to-point bus, or other suitable topology. In some examples, any of the plurality of quantized accelerators 186 may be connected to any other core, while in other examples, some cores are connected only to a subset of other cores. For example, each core may be connected only to the nearest 4, 8, or 10 neighbor cores. The interconnect may be used to transmit input / output data to and from the quantized accelerator 186, as well as to transmit control signals and other information signals to and from the quantized accelerator 186. For example, each of the quantization accelerators 186 may receive and transmit semaphores that indicate the execution status of an operation currently being performed by each of the corresponding quantization accelerators 186. Further, matrix and vector values ​​may be shared between the quantization accelerators 186 via interconnects. In some examples, the interconnects are implemented as wires connecting the quantization accelerators 186 and the memory system, while in other examples, the core interconnects may include circuitry, switches, and / or routing components for multiplexing data signals on (multiple) interconnect lines, including active signal drivers and repeaters, or other suitable circuitry. In some examples of the disclosed technology, the signals transmitted within and to / from the neural network accelerator 180 are not limited to full-swing electrical digital signals, but the neural network accelerator 180 may be configured to include differential signals, pulse signals, or other suitable signals for transmitting data and control signals.

[0053] In some examples, quantization enabling system 110 may include an optional quantization simulator that simulates the functionality of neural network accelerator 180. Neural network accelerator 180 provides functionality that can be used to convert data represented in full precision floating point format in neural network module 130 to quantized format values. Neural network accelerator 180 can also perform operations using quantized format values. Such functionality will be discussed in further detail below.

[0054] The neural network module 130 can be used to specify, train, and evaluate a neural network model using a tool flow that includes a cross-hardware modeling framework 131 (also referred to as a native framework or machine learning execution engine), a neural network compiler 132, and a neural network runtime environment 133. The memory includes computer executable instructions for the tool flow that includes the modeling framework 131, the neural network compiler 132, and the neural network runtime environment 133. The tool flow can be used to generate neural network data 200 representing all or part of a neural network model, such as described below with respect to Figure 2 The neural network model discussed. It should be noted that although the tool flow is described as having three separate tools (131, 132, and 133), in various examples the tool flow may have fewer or more tools. For example, the functionality of the different tools (131, 132, and 133) may be combined into a single modeling and execution environment. In other examples where a neural network accelerator is deployed, such a modeling framework may not be included.

[0055] Neural network data 200 may be stored in memory 125, which may include local memory 126 (typically implemented as static read-only memory (SRAM), embedded dynamic random access memory (eDRAM), implemented in latches or flip-flops in a register file, block RAM, or other suitable structures) and bulk memory 127, which is typically implemented in a memory structure that supports larger but typically slower access than local memory 126. For example, bulk memory may be off-chip DRAM, network accessible RAM, an SSD drive, a hard drive, or a network accessible storage device. Other memory structures (including those listed above for local memory) may be used to implement bulk memory, depending on the specific memory technology available. Neural network data 200 may be represented in one or more formats. For example, neural network data 200 corresponding to a given neural network model may have a different format associated with each respective tool of a tool flow. In general, neural network data 200 may include descriptions of nodes, edges, groups, weights, biases, activation functions, and / or tensor values. As a specific example, neural network data 200 may include source code, executable code, metadata, configuration data, data structures, and / or files for representing a neural network model.

[0056] The modeling framework 131 can be used to define and use neural network models. As an example, the modeling framework 131 may include predefined APIs and / or programming primitives that can be used to specify one or more aspects of a neural network model. The predefined APIs may include both lower-level APIs (e.g., activation functions, cost or error functions, nodes, edges, and tensors) and higher-level APIs (e.g., layers, convolutional neural networks, recursive neural networks, linear classifiers, etc.). "Source code" can be used as input to the modeling framework 131 to define the topology of the graph of a given neural network model. Specifically, the APIs of the modeling framework 131 can be instantiated and interconnected within the source code to specify complex neural network models. Data scientists can create different neural network models by using different APIs, different numbers of APIs, and interconnecting APIs in different ways.

[0057] In addition to the source code, the memory 125 may also store training data. The training data includes an input data set for application to the neural network model 200 and the expected output of the neural network model for each corresponding data set from the input data. The modeling framework 131 may be used to train the neural network model using the training data. The output of the training is the weights and biases associated with each node of the neural network model. After the neural network model is trained, the modeling framework 131 may be used to classify new data applied to the trained neural network model. Specifically, the trained neural network model uses the weights and biases obtained from the training to perform classification and recognition tasks on data that has not yet been used to train the neural network model. The modeling framework 131 may use the CPU 120 and a dedicated processor (e.g., a GPU 122 and / or a neural network accelerator 180) to execute the neural network model with increased performance compared to using only the CPU 120. In some examples, the performance may potentially achieve real-time performance for some classification tasks.

[0058] The compiler 132 analyzes the source code and data provided for the neural network model (e.g., an example used to train the model) and transforms the model into a format that can be accelerated on the neural network accelerator 180, which will be described in further detail below. Specifically, the compiler 132 transforms the source code into executable code, metadata, configuration data, and / or data structures for representing the neural network model and memory as neural network data 200. In some examples, the compiler 132 can divide the neural network model into a portion (e.g., neural network 200) that uses the CPU 120 and / or GPU 122) and other portions that can be executed on the neural network accelerator 180 (e.g., a subgraph, individual layer, or multiple layers of the neural network). The compiler 132 can generate executable code (e.g., a runtime module) for executing the NN assigned to the CPU 120 and for communicating with the subgraph, individual layer, or multiple layers of the neural network assigned to the accelerator 180. The compiler 132 can generate configuration data for the accelerator 180, which is used to configure accelerator resources to evaluate the subgraph assigned to the optional accelerator 180. Compiler 132 may create data structures for storing values ​​generated by the neural network model during execution and / or training and for communication between CPU 120 and accelerator 180. Compiler 132 may generate metadata that may be used to identify subgraphs, edge groupings, training data, and various other information about the neural network model during runtime. For example, metadata may include information for interfacing between different subgraphs or other parts of the neural network model.

[0059] The runtime environment 133 provides an executable environment or interpreter that can be used to train the neural network model during training mode and can be used to evaluate the neural network model in training, reasoning or classification mode. During reasoning mode, input data can be applied to the neural network model input, and the input data can be classified according to the training of the neural network model. The input data can be archived data or real-time data.

[0060] Runtime environment 133 may include a deployment tool that can be used to deploy or install all or part of a neural network to neural network accelerator 180 during deployment mode. Runtime environment 133 may also include a scheduler that manages the execution of different runtime modules and the communication between the runtime modules and neural network accelerator 180. Thus, runtime environment 133 can be used to control the flow of data between nodes modeled on neural network module 130 and neural network accelerator 180.

[0061] In one example, the neural network accelerator 180 receives and returns the normal precision value 150 from the neural network module 130. Figure 1As shown in , the quantized accelerator 186 can use quantized floating points to perform most of its operations, and the interface between the quantized accelerator 186 and the neural network module 130 can use full precision values ​​for conveying information between modules. Ordinary precision values ​​can be represented in 16-bit, 32-bit, 64-bit or other suitable floating point formats. For example, a portion of a value representing a neural network can be received, including edge weights, activation values, or other suitable quantization parameters. Ordinary precision values ​​150 are provided to ordinary precision floating point to quantized floating point converter 152, which converts ordinary precision values ​​to quantized floating point converters. Then, quantized floating point operations 154 can be performed on the quantized values. Then, the quantized values ​​can be converted back to ordinary floating point format using quantized floating point to ordinary floating point converter 156 that produces ordinary precision floating point values. As a specific example, the quantization accelerator 186 can be used to accelerate a given layer of a neural network, and vector-vector, matrix-vector, matrix-matrix, and convolution operations can be performed using quantized floating-point operations, and less computationally intensive operations (such as adding bias values ​​or calculating activation functions) can be performed using normal floating-point precision operations.

[0062] The conversions between normal floating point and quantized floating point performed by converter 152 and converter 156 are typically performed on a collection of numbers represented as vectors or multi-dimensional matrices. In some examples, additional normal precision operations 158 (including operations that may be required in a particular neural network implementation) may be performed based on the normal precision format, including adding a bias to one or more nodes of the neural network, applying a hyperbolic tangent function or other such sigmoid function, or a rectification function (e.g., a ReLU operation) to normal precision values ​​converted back from the quantized floating point format.

[0063] In some examples, quantized values ​​are only used and stored in the logic gates and internal memory of the neural network accelerator 180, and the memory 125 and the storage device 129 store only ordinary floating point values. For example, the neural network accelerator 180 can quantize the inputs, weights, and activations for the neural network model received from the neural network model 130, and can dequantize the results of the operations performed on the neural network accelerator 180 before passing the values ​​back to the neural network model 130. The values ​​can be passed between the neural network model 130 and the neural network accelerator 180 using the memory 125, the storage device 129, or the input / output interface (not shown). In some other examples, the simulator provides a full simulation of quantization, including storing only one copy of the shared exponent and operating with a reduced mantissa width. Some results may be different in versions where the basic operations are performed in ordinary floating point. For example, some examples can check for underflow or overflow conditions for limited quantized bit widths (e.g., 3-bit, 4-bit, or 5-bit wide mantissas).

[0064] Most of the computational cost of a DNN is vector-vector, matrix-vector, and matrix-matrix multiplications and / or convolutions. These operations are quadratic in the input size, while operations (such as bias addition and activation functions) are linear in the input size. Therefore, in some examples, quantization is only applied to matrix-vector multiplication operations, which are implemented on the neural network accelerator 180. In such examples, all other operations are performed in a normal precision format (such as float16). Therefore, from the perspective of a user or programmer, the quantization enabling system 110 accepts and outputs normal precision float16 values ​​from / to the neural network module 130 and outputs float16 format values. All conversions to and from block floating point formats can be hidden from the programmer or user. In some examples, the programmer or user can specify certain parameters for the quantization operation. In some other examples, the quantization operation can utilize the block floating point format to reduce computational complexity, as described below with respect to Figure 3 discussed.

[0065] A neural network accelerator 180 is used to accelerate the evaluation and / or training of a neural network graph or subgraph, generally at increased speed and reduced latency that is not achieved when evaluating a subgraph using only a CPU 120 and / or GPU 122. In the illustrated example, the accelerator includes a tensor processing unit (TPU) 182, a reconfigurable logic device 184 (e.g., contained in one or more FPGAs or programmable circuit structures), and / or a quantization accelerator 186, however, any suitable hardware accelerator that models a neural network may be used. The accelerator 180 may include configuration logic that provides a soft CPU. The soft CPU oversees the operation of the accelerated graph or subgraph on the accelerator 180 and may manage communications with the neural network module 130. The soft CPU may also be used to configure logic and control the loading and storing of data from RAM on the accelerator, such as in block RAM within an FPGA.

[0066] In some examples, parameters of neural network accelerator 180 may be programmable. Neural network accelerator 180 may be used to prototype training, inference, or classification of all or part of neural network model 200. For example, quantization parameters may be selected based on accuracy or performance results obtained by prototyping a network within neural network accelerator 180. After the desired set of quantization parameters is selected, the quantization model may be programmed into accelerator 180 for performing further operations.

[0067] The compiler 132 and runtime 133 provide a fast interface between the neural network module 130 and the neural network accelerator 180. In practice, the user of the neural network model may not be aware that a portion of the model is being accelerated on the provided accelerator. For example, node values ​​are typically propagated in the model by writing tensor values ​​to a data structure that includes an identifier. The runtime 133 associates a subgraph identifier with an accelerator and provides logic for translating messages to the accelerator to transparently write weights, biases, and / or tensor values ​​to the neural network accelerator 180 without program intervention. Similarly, values ​​output by the neural network accelerator 180 can be transparently sent back to the neural network module 130, where the message includes an identifier of the receiving node at the server and a payload that includes values ​​such as weights, biases, and / or tensors that are sent back to the overall neural network model.

[0068] Figure 2 A simplified topology of a deep neural network (DNN) 200 is illustrated that can be used to perform enhanced image processing using the disclosed BFP implementations. One or more processing layers can be implemented using the disclosed techniques for quantization and BFP matrix / vector operations, including using one or more of the multiple neural network quantization accelerators 186 in the quantization-enabled system 110 described above. It should be noted that the application of the neural network implementations disclosed herein is not limited to DNNs, but can also be used with other types of neural networks (such as convolutional neural networks (CNNs), including implementations with long short-term memory (LSTM) or gated recurrent units (GRUs), or other suitable artificial neural networks that can be adapted to use the BFP methods and apparatus disclosed herein).

[0069] DNN 200 can operate in at least two different modes. Initially, DNN 200 can be trained in training mode and then used as a classifier in inference mode. During training mode, a training data set can be applied to the input of DNN 200 and various parameters of DNN 200 can be adjusted so that when training is completed, DNN 200 can be used as a classifier. Training includes performing forward propagation of training input data, calculating losses (e.g., determining the difference between the output of DNN and the expected output of DNN), and performing backward propagation through DNN to adjust the parameters (e.g., weights and biases) of DNN 200. When the architecture of DNN 200 is suitable for classifying training data, the parameters of DNN 200 will converge and training can be completed. After training, DNN 200 can be used in inference mode. Specifically, training or non-training data can be applied to the input of DNN 200 and forward propagated through DNN 200 so that the input data can be classified by DNN 200.

[0070] like Figure 2 As shown in , a first set 210 of nodes (including nodes 215 and 216) forms an input layer. Each node of set 210 is connected to each node in a first hidden layer formed by a second set 220 of nodes (including nodes 225 and 226). The second hidden layer is formed by a third set 230 of nodes (including nodes 235). The output layer is formed by a fourth set 240 of nodes (including nodes 245). In example 200, the nodes of a given layer are fully interconnected with the nodes of its (multiple) adjacent layers. In other words, a layer may include nodes that have a common input with other nodes of the layer and / or provide outputs to a common destination of other nodes of the layer. In some other examples, a layer may include nodes that have a common input subset with other nodes of the layer and / or provide outputs to a common destination subset of other nodes of the layer.

[0071] During forward propagation, each node produces an output by applying a weight to each input generated from the previous node and collecting the weights to produce an output value. In some examples, each individual node may have an activation function (σ) and / or a bias (b) applied. In general, a suitably programmed processor or FPGA may be configured to implement the nodes in the depicted neural network 200. In some example neural networks, the output function f(n) of a hidden combined node n may produce an output mathematically expressed as follows:

[0072]

[0073] where w i is applied to (multiplied by) the input edge x i , b is the bias value for node n, σ is the activation function for node n, and E is the number of input edges of node n. In some examples, the activation function produces a continuous value (represented as a floating point number) between 0 and 1. In some examples, the activation function produces a binary 1 or 0 value depending on whether the sum is above or below a threshold.

[0074] A given neural network may include thousands of individual nodes, and therefore performing all calculations for the nodes in normal precision floating point may be computationally expensive. The implementation of a computationally more expensive solution may include hardware that is larger and consumes more energy than the implementation of a computationally less expensive solution. However, using quantized floating point to perform operations can potentially reduce the computational complexity of the neural network. A simple implementation using only quantized floating point can significantly reduce the computational complexity, but due to the errors introduced by quantization, the implementation may have difficulty converging during training and / or correctly classifying input data. However, the quantized floating point implementation disclosed herein can potentially increase the accuracy of some calculations while also providing the benefit of reducing the complexity associated with quantized floating point.

[0075] DNN 200 may include nodes that perform operations in quantized floating point. As a specific example, the output function f(n) of the hidden combination node n may produce an output mathematically expressed as:

[0076]

[0077] where w i is applied to (multiplied by) the input edge x i The weight of Q(w i ) is the quantized floating point value of the weight, Q(w i ) is derived from the input edge x i The quantized floating point value of the input, Q -1 ( ) is a dequantized representation of the quantized floating point value of the dot product of vector w and x, b is the bias value for node n, σ is the activation function of node n, and E is the number of input edges of node n. The computational complexity can be potentially reduced by performing the dot product using quantized floating point values ​​(compared to using only normal precision floating point values), and the accuracy of the output function can be potentially increased by other operations of the output function using normal precision floating point values ​​(compared to using only quantized floating point values).

[0078] Neural networks can be trained and retrained by adjusting the values ​​of the components of the output function f(n). For example, by adjusting the weights w of the nodes i Or bias value b, the behavior of the neural network is adjusted by the corresponding change of the network output tensor value. For example, the cost function C(w,b) can be used during backpropagation to find appropriate weights and biases for the network, where the cost function can be mathematically described as:

[0079]

[0080] Where w and b represent all weights and biases, n is the number of training inputs, and a is a vector of output values ​​from the network for the input vector of training inputs x. By adjusting the network weights and biases, the cost function C can be driven to a target value (e.g., to zero (0)) using various search techniques (e.g., stochastic gradient descent). When the cost function C is driven to the target value, the neural network is said to converge. Similar to the output function f(n), the cost function can be implemented using a quantized precision computer algorithm. For example, vector operations can be performed using quantized floating point values ​​and operations, and non-vector operations can be performed using normal precision floating point values.

[0081] Examples of suitable applications of such neural network BFP embodiments include, but are not limited to: performing image recognition, performing speech recognition, classifying images, translating speech into text and / or other languages, facial or other biometric recognition, natural language processing, automatic language translation, query processing in search engines, automatic content selection, analyzing email and other electronic documents, relationship management, biomedical informatics, identifying candidate biomolecules, providing recommendations, or other classification and artificial intelligence tasks.

[0082] Network accelerators (such as Figure 1 The network accelerator 180 in the DNN 200 can be used to accelerate the calculations of the DNN 200. As an example, the DNN 200 can be divided into different subgraphs or network layers that can be accelerated separately. As a specific example, each of the layers 210, 220, 230, and 240 can be a subgraph or layer accelerated with the same or different accelerators. The computationally expensive calculations of the layer can be performed using quantized floating points, and the less expensive calculations of the layer can be performed using normal precision floating points. Values ​​can be passed from one layer to another using normal precision floating points. By accelerating a set of calculations for all nodes within a layer, some calculations can be reused and the calculations performed by the layer can be reduced compared to accelerating individual nodes.

[0083] In some examples, a set of parallel multiply-accumulate (MAC) units in each convolutional layer can be used to speed up the computation. In addition, parallel multiplier units can be used for fully connected and dense matrix multiplication stages. A set of parallel classifiers can also be used. Such parallelization methods have the potential to further speed up the computation even at the expense of added control complexity.

[0084] As will be readily appreciated by one of ordinary skill in the art having the benefit of this disclosure, application of the neural network embodiments can be used to employ different aspects of the neural network, either alone or in combination or sub-combination with one another. For example, the disclosed embodiments can be used to implement neural network training via gradient descent and / or backpropagation operations of the neural network. Further, the disclosed embodiments can be used to evaluate the neural network.

[0085] IV. Example quantized block floating point format

[0086] Figure 3 300 is a diagram illustrating an example of converting a normal floating point format to a quantized block floating point format, as may be used in certain examples of the disclosed technology. For example, an input tensor of a neural network represented as a normal floating point number (e.g., in a 32-bit or 16-bit floating point format) may be converted to the illustrated block floating point format.

[0087] As shown, many common floating point format numbers 310 are represented so that each number (e.g., number 315 or number 316) includes a sign, an exponent, and a mantissa. For example, for the IEEE 754 half-precision floating point format, the sign is represented using one bit, the exponent is represented using 5 bits, and the mantissa is represented using 10 bits. When the floating point format numbers 310 in the neural network model 200 are converted to a set of block floating point format numbers of quantized precision, there is an exponent value shared by all the numbers in the illustrated set. Therefore, as shown, the set of block floating point numbers 320 is represented by a single exponent value 330, and each number in the set of numbers includes a sign and a mantissa. However, since the illustrated set of numbers has different exponent values ​​in floating point format, the corresponding mantissa of each number can be shifted so that the same or close numbers are represented in a quantized format (e.g., shifted mantissa 345 and shifted mantissa 346).

[0088] Furthermore, if Figure 3As shown in, using block floating point format can reduce the computing resources required for some common operations. In the illustrated example, the dot product of two floating point vectors is illustrated in formal floating point format (350) and block floating point format (360). For the number represented by the ordinary precision floating point format operation 350, it is necessary to perform floating point addition to perform the dot product operation. In the dot product of floating point vectors, the summation is performed with floating point, which may require shifting to align the value with different exponents. On the other hand, for the block floating point dot product operation 360, the product can be calculated using integer arithmetic to combine the mantissa elements as shown. In other words, since the exponent part can be decomposed in the block floating point representation, the multiplication and addition of the mantissa can be completed completely with fixed point or integer representation. Therefore, the large dynamic range of the set of numbers can be maintained using shared exponents while reducing the computational cost by using more integer arithmetic rather than floating point arithmetic. In some examples, the operation performed by the quantization enabling system 110 can be optimized to utilize the block floating point format.

[0089] In some examples, shared exponent 330 is selected as the largest exponent among the original common precision numbers from neural network model 200. In other examples, shared exponent may be selected in different ways, such as by selecting an exponent that is the average or median of common floating point exponents, or by selecting an exponent to maximize the dynamic range of values ​​stored in the mantissa when the numbers of these values ​​are converted to a quantized number format. It should be noted that if the shared exponent and the original floating point exponent of the values ​​are different, some bits of the quantized mantissa may be lost. Such a situation occurs because the mantissa is shifted to correspond to the new shared exponent.

[0090] There are several possible choices for which values ​​in a block floating point tensor will share an exponent. The simplest choice is for the entire matrix or vector to share an exponent. However, sharing exponents at a finer granularity can reduce errors because it increases the likelihood that a BFP number uses a shared exponent that is closer to its original normal floating point format exponent. Therefore, the precision loss caused by discarding mantissa bits (when shifting the mantissa to correspond to the shared exponent) can be reduced.

[0091] For example, consider multiplying a row vector x by a matrix W: y = xW. If the exponent is shared for each column of W, then each dot product xW j (W j is the jth column of W) involves only one shared index for x and one shared index for W j A shared index.

[0092] Figure 4 and Figure 5An alternative block floating point format that can be used for neural network calculations is illustrated. In the context of neural networks, the core operation is to perform dot products. For example, dot products are the core calculations of matrix multiplication and convolution. Matrix multiplication involves the dot product of the rows / columns of a matrix with an input vector. Convolution involves the dot product of a filter with an input window. In the context of quantized floating points, the value group selected to share the exponent can have an impact on the complexity of the computer algorithm logic used to calculate the dot product. The value of the shared exponent can be referred to as the value within the bounding box. The shape of the bounding box may potentially affect quantization error and computational cost. Although clustering similar magnitude values ​​to create a bounding box can reduce quantization error, tracking the scaling factor of an arbitrary bounding box shape may be expensive. In contrast, matrices and filters can be divided into bounding boxes, which may be more efficient for operations performed by a neural network. Specifically, the appropriate selection of bounding boxes can reduce the complexity of computer algorithm circuits used to implement the operation of a neural network. Figure 4 illustrates a block floating-point format that may be well suited for matrices, and Figure 5 Illustration of a block floating point format that may be well suited for convolution filters.

[0093] Figure 4 4 is a diagram 400 illustrating four alternative block floating point formats, as may be used in certain examples of the disclosed technology. As shown, a first format 410 represents an entire array 420 of values ​​sharing a single exponent 425. In other words, the entire array 420 of values ​​is packed within a single bounding box.

[0094] In the second format 430, public indices are shared on a per-column basis. In other words, the columns of the matrix are bounding boxes for values. Therefore, in this particular example, the block floating point values ​​stored in the even columns 431 of the matrix each share a first single index 432. The block floating point values ​​stored in the odd columns 435 each share a second single index 437. In some other examples, each column of the array may be associated with a different shared index. For eleven columns of small blocks in the alternative format, there may be eleven corresponding shared indices, one shared index per column. In some other examples, each row of the array may be associated with a different shared index, or odd and even rows may be associated with a shared public index.

[0095] A third format 450 is shown in which groups of elements in an array share common indices. For example, if a 15×15 matrix of values ​​shares indices according to the third format 450, then a first set of 5×5 element groups 455 and 456 share a single shared index 458. Similarly, a second 5×5 element group of elements in arrays 460 and 461 may each share a second single index 468. In some other examples, each tile may be associated with its own corresponding shared index. In the example format 450, there may be nine shared indices for the 15×15 matrix.

[0096] A fourth format 470 is shown in which two shared indices are shared on a tile-plus-per-column basis. Thus, a first set of numbers including number 480, number 481, and number 485 all share a single common exponent 488. Similarly, a second set of numbers including sets 490 and 491 each share a second different single exponent 495. In an alternative example, each group shown may have its own shared exponent.

[0097] In some examples, the computational cost of matrix-vector multiplication can be further reduced by reducing the mantissa width. A large range of values ​​that share a common exponent can be expressed using only a few bits of the mantissa. For example, in a representation with a 4-bit mantissa and a 5-bit exponent, a value can be in [2 -14 0.0012,2 15 1.1112] or approximately [2 -17 , 2 16 ] can be expressed in the range of ]. In contrast, 4-bit fixed-point numbers can only represent [10012, 11112] or approximately [2 0 , 2 4 ] is a value in the range of ].

[0098] Figure 5 5 is a diagram illustrating three alternative block floating point formats, as may be used in certain examples of the disclosed technology. These formats may be useful for two-dimensional convolutions, but these formats may also be generalized to higher dimensional convolutions. As shown, a first format 510 represents an entire convolution filter 512 of values ​​sharing a single exponent 514. Different convolution filters 516 of values ​​may share a single exponent 518. Thus, format 510 illustrates that the entire convolution filter may be a bounding box of values.

[0099] In the second format 520, each spatial pixel may be a bounding box such that a common index is shared on a per-spatial pixel basis along the channel dimension. As shown, spatial pixel values ​​522 share a single index 524 and spatial pixel values ​​526 share a single index 528. For example, for a pixel with dimensions [x, y, c i ], each spatial dimension x and y can be defined with c i Similarly, for dimensions [f x , f y , c i ]c o Convolution filter, for each c o Each pixel of the filter (f x , f y ) can be a i The bounding box size of this method is c i .

[0100] In the third format 530, each spatial pixel can be subdivided along the channel dimension so that the bounding box includes the subdivision of the spatial pixel. As shown, the subdivided spatial pixel values ​​532 share a single index 534 and the subdivided spatial pixel values ​​536 share a single index 538. For small c i , the cost of handling scaling factors can be large. For example, the input image of the first layer of a deep convolutional neural network may have c corresponding to 3 color channels. i =3. Tracking the scaling factor for each triplet may be expensive. In this case, the convolution can be reshaped as a matrix-matrix multiplication to increase the bounding box and reduce the cost of tracking the bounding box. For example, each convolution filter can be flattened to create a triple with c o Column and f x , *f y , *c i A matrix W of rows can be created where each row is a window f corresponding to the input that the convolution filter sweeps over. x , *f y , *c i vector. The result Y = XW is a matrix that can be reshaped to match the output of the convolution operation. In the case of convolution reformulated as a matrix multiplication, the above reference Figure 4 The bounding box strategy discussed for matrix multiplication can be applied.

[0101] V. Example method for neural network training

[0102] Figure 6 6 is a flow chart depicting a method 600 of training a neural network using a quantized model with an abnormal block floating point format, as may be implemented in certain examples of the disclosed technology. For example, training the neural network may include iterating through a training data set, wherein the method 600 is used to update the parameters of the neural network during a given iteration of the training data. As an example, the method 600 may be implemented by a quantization-enabled system such as Figure 1 quantization enabling system 110) is executed.

[0103] At process block 610, parameters of the neural network, such as weights and biases, may be initialized. As one example, the weights and biases may be initialized to random normal precision floating point values. As another example, the weights and biases may be initialized to normal precision floating point values ​​calculated from an earlier training set. The initial parameters may be stored in a memory or storage device of a quantization enabled system. In one example, the parameters may be stored as quantized floating point values, which may reduce the amount of storage used to store the initial parameters.

[0104] At process block 620, the input values ​​of the neural network may be forward propagated through the neural network. The input values ​​of a given layer of the neural network may be the outputs of another layer of the neural network. Values ​​may be passed between layers from the outputs of one layer to the inputs of the next layer using normal precision floating points. The output function of layer i may include terms mathematically described as follows:

[0105] y i =Q -1 (f(Q(y i-1 ),Q(W i ))) (Formula 5)

[0106] where y i-1 is the output from the layer that provides input to the i-th layer, W i is the weight tensor of the i-th layer, f() is the forward function of the layer, Q() is the quantization function, and Q -1 () is the dequantization function. The output function of the layer can be a dequantized representation of f(), or alternatively, the output function can include additional terms, such as the addition of activation functions or biases, which are performed using normal precision floating point (after dequantization) or using quantized floating point (before dequantization). In general, the inputs, outputs, and parameters of a layer are tensors. Typically, the inputs, outputs, and parameters of a layer will be vectors or matrices. The quantization function converts normal precision floating point values ​​to quantized floating point values. The quantization function can be chosen to take into account the type of input data and the type of operation performed by the i-th layer. For example, when y i and W i is a two-dimensional matrix and the output function consists of taking y i-1 and W i When the cross product of i-1 The quantization function can use the i-1 The bounding box of a row or part of a row of i The quantization function can be used including W i quantize function converts quantized floating point values ​​to normal precision floating point values. In some examples, the quantization function Q() and the dequantization function Q -1 () are omitted and the forward pass is formed in normal precision floating point format.

[0107] At process block 630, a portion of the neural network (such as a layer that has just been forward propagated to the next layer of the neural network) can be compressed and stored in memory. For example, the activation values ​​calculated as part of the forward propagation of process block 620 as discussed above can be compressed and stored in memory. This compression can be mathematically expressed as:

[0108] y ci [+O ci ]=C(Q2(y i )) (Formula 6a)

[0109] or

[0110] y ci [+O ci ]=C(Q2(f(Q(y i-1 ), Q(W i )))) (Formula 6b)

[0111] where y i is the value generated by the forward pass of the layer at process block 620, C() is an optional additional compression function (possibly including multiple compression operations), Q2() is a quantization function to a second block floating point format, and y ci is the compressed value to be stored in memory, and O ci are outliers for some (but not all) of the activation values. As discussed below, outliers are generated for a selected subset of all activation values ​​and stored in a secondary storage device that compresses the activation values. In block floating point format, the compressed value y ci share a common index, and the outliers y ci More significant bits (increasing the expressible range of those activation values) or precision may be added than for non-outlier compressed values. For example, when some activation values ​​are significantly larger than other activation values, these values ​​may be selected as outliers to enable additional information to be stored in auxiliary storage when compressed.

[0112] In some examples, the Q2() quantization function converts a value from a normal precision format to a smaller quantized format than that used in the quantized layer (as in Equation 6a). In some other examples, the Q2() quantization function converts a value directly from a first block floating point format used in the quantized layer to a second block floating point format (as in Equation 6b). The compressed activation value is expressed in a second block floating point format that may be different from the first block floating point format used to perform the forward propagation calculation and at least one of the following ways: having a different mantissa format, having a different exponent format, or having a different exponent sharing scheme. For example, if the forward propagation is performed using activation values ​​expressed in an 8-bit format, then these values ​​can be transformed into a 4-bit format by truncating or rounding the mantissa. As another example, the activation value exponent (including a shared exponent in BBFP format) can be transformed from a 7-bit format to a 5-bit format. Values ​​can be converted between two quantized formats used by any suitable technique. For example, truncation or rounding of the exponent and any mantissa shift performed to compensate for the adjusted exponent can be performed. In some examples, a table lookup or other technique may be used to perform the conversion.

[0113] In some examples, the compressed block floating point format value y ci The common exponents are shared in a different format than that used when performing the forward pass. Aspects of the sharing format that may be changed include how the exponents are shared on a per-row, per-column, or per-tile basis. In some examples, outliers O ci It can also be expressed in a block floating point format with at least one shared exponent. In some other examples, each outlier value O ci has its own individual exponent. In some examples, additional compression may be applied to the compressed block floating point format prior to storage in memory. Examples of suitable techniques for further compressing the activation values ​​in the compressed quantized format include entropy compression (e.g., Huffman encoding), zero compression, run-length compression, compressed sparse row compression, or compressed sparse column compression.

[0114] At process block 640, the loss of the neural network may be calculated. For example, the output y of the neural network may be compared to the expected output y of the neural network. The difference between the output and the expected output can be an input to a cost function that is used to update the parameters of the neural network.

[0115] At process block 650, the activation values ​​stored in the memory are decompressed for use in the backpropagation, and specifically, for use in computing the output error term used in the backpropagation of a particular layer. The method may iterate over each layer and decompress the activation values ​​of each layer, perform backpropagation on the layer, and then decompress the activation values ​​of the previous layer. The decompression may be expressed mathematically as:

[0116] y i =Q2 -1 (C -1 (y ci ))[+Q2 -1 (C -1 (O ci ))] (Formula 7a)

[0117] or

[0118] y i =C -1 (y ci )[+C -1 (O ci )] (Formula 7b)

[0119] where y ci is the compressed value retrieved from memory, O ci is the exception value retrieved from memory, C -1( ) is an optional decompression function (which may include multiple compression operations), which is the inverse function of the optional compression function C(). Q2 -1 ( ) is a function that converts the quantized value from the second block floating point format to the first block floating point format, and y i is the value generated for the layer by forward propagation at process block 620. For example, after forward propagation is completed for each layer and neural network as discussed above with respect to process blocks 620 and process blocks 630 and the loss is calculated as discussed above at process block 640, the value is back-propagated through the neural network, typically starting at the output layer of the neural network. Thus, depending on how the compressed quantized format differs from the format used for back-propagation, appropriate transformations of the activation value mantissa, exponent, and / or exponent sharing scheme may be performed. Further, if additional compression is applied prior to storage in memory, such as entropy compression, zero compression, run-length encoding, compressed sparse row compression, or compressed sparse column compression, then these operations may be reversed prior to performing back-propagation for the layer at process block 660.

[0120] At process block 660, the loss of the neural network may be back-propagated through the neural network. During back-propagation, the output error term and the weighted error term can be calculated. The output error term can be mathematically described as:

[0121]

[0122] in is the output error term from the layer after the i-th layer, W i is the weight tensor of the i-th layer, g() is the backward function of the layer, Q() is the quantization function, and Q -1 ( ) is the dequantization function. The backward function g() can be i-1 The backward function of f() is the gradient of g() or a portion of the gradient function. The output error term of this layer can be a dequantized representation of g(), or the output error term can include additional terms that are performed using normal precision floating point (after dequantization) or using quantized floating point (before dequantization).

[0123] Weight error term It can be described mathematically as:

[0124]

[0125] in is the weight error term of the i-th layer, is the output error term of the i-th layer, y iis the output of the i-th layer, h() is the backward function of the layer, Q() is the quantization function, and Q -1 () is the inverse quantization function. The backward function h() can be for W i-1 The gradient or weight error term of the layer may be the backward function of f() as part of Equation 9. The weight error term for this layer may be a dequantized representation of h(), or the weight error term may include additional terms that are performed using normal precision floating point (after dequantization) or using quantized floating point (before dequantization). The weight error term may include additional terms that are performed using normal precision floating point.

[0126] At process block 670, the parameters for each layer may be updated. For example, the weights of each layer may be updated by calculating new weights based on iterations of training. As an example, the weight update function may be mathematically described as:

[0127]

[0128] in is the weight error term of the i-th layer, η is the learning rate of the i-th layer of the neural network, W i is the weight tensor of the i-th layer. In one example, the weight update function can be executed using normal precision floating point.

[0129] VI. Example environment for performing activation compression using narrow block floating point

[0130] Figure 7 700 is a block diagram depicting an example of a suitable environment for performing activation compression and associated floating point operations between a normal precision floating point domain, a quantized floating point domain, and a compressed quantized floating point domain including outliers. As described above, computationally more expensive operations (such as vector-vector, vector-matrix, matrix-matrix, and convolution operations) can be performed by the quantized layer 710 in the quantized floating point domain. Computationally less expensive operations (such as scalar addition and scalar multiplication) can be performed outside the quantized layer 710 in the normal precision floating point domain. With respect to a neural network, the neural network can be divided into layers (such as the quantized layer 710). For the back-propagation portion of training, the error output component 720 can receive the activation values ​​and use an additional function (such as an error function or an objective function) to calculate an output error term In some examples, the output error term is is computed in the normal precision floating point domain, as shown by the solid line of the output error component 720. In some other examples, the output error component 720 computes the error output term in a block floating point format.

[0131] Most of the computational work within a layer can be performed in the quantized floating point domain, and computationally less expensive operations of the layer (such as adding bias values ​​or calculating activation functions) can be performed in the normal precision floating point domain. Values ​​that interface between layers can be passed from one layer to another in the normal precision floating point domain. By quantizing the input specifically for a given layer, the quantization can be targeted to the operation of the layer to make the operation of the layer more efficient. In particular, the bounding box of the quantized floating point format can be selected to reduce the complexity of the computer algorithm circuitry to make the computer logic potentially faster and / or more energy efficient.

[0132] As an example, given the output value yi of a layer, the output error term Weight Wi and weight error term can be stored in the normal precision floating point domain. During the forward propagation flow, the output values ​​from the earlier layers can be communicated from the normal precision floating point domain to the quantized floating point domain by the quantizer 722, which converts the normal precision floating point to the quantized floating point. The output values ​​from a given layer can be communicated from the quantized floating point domain to the normal precision floating point domain by the dequantizer 724, which converts the quantized floating point to the normal precision floating point. The weights of a given layer can be communicated from the normal precision floating point domain to the quantized floating point domain by the quantizer 742. The dequantized value y i Or the quantized value Q -1 (y i ) may be sent to a compressor 760 which compresses the values ​​before they are stored in a mass storage device 770.

[0133] The compressor 760 may be a block floating point compressor, or provide other forms of compression to reduce the amount of data stored in the mass storage 770. In such an example, the second block floating point format has a lower precision mantissa than the first block floating point format used to represent values ​​during the operation of the quantized layer 710. For example, the first block floating point format used in the quantized layer 710 may have a mantissa with more bits than the second block floating point format, such as: four bits, five bits, six bits, seven bits, or eight bits; and the second block floating point format may have a mantissa with fewer bits than the first block floating point format, such as: three bits, four bits, four bits or five bits, five bits or six bits, or four bits to six bits, respectively. In the example discussed below, the first block floating point format has a six-bit mantissa and the second block floating point format has a four-bit mantissa. It should be readily understood by those of ordinary skill in the relevant art having the benefit of the present disclosure that the above combinations of specific bits of the first block floating point format and the second block floating point format are only preferred examples, but other precision combinations of mantissa formats may be used in other examples.

[0134] The compressor 760 includes an outlier quantizer 765 that identifies some but not all activation values ​​as outliers. The outlier quantizer 765 may include an index selector that determines a shared index for the compressed activation values. For example, the selector may identify a shared index by determining at least one of a mean (average), a median, and / or a mode of at least a portion of the activation values. In some examples, the selector may identify a shared index by identifying a set of maximum outliers in the activation value set and identifying a shared index for the remaining group activation values ​​that are not identified as being in the group with the maximum anomaly, thereby determining the shared index by selecting the maximum index of the remaining group. In some examples, the index used by the maximum number of activation values ​​is selected as the shared index. The outlier quantizer 765 also identifies a limited number of activation values ​​as having outliers. Those activation values ​​with outliers will have additional data stored to improve accuracy and / or allow for larger or more accurate values ​​to be stored for those specific activation values. For example, a predetermined number of maximum values ​​in the activation value set may be determined as anomalies. As a non-limiting example, the largest 8 of 256 values ​​may be selected as outlier activation values. The largest eight values ​​selected will have a second outlier mantissa and associated exponent that can be combined when the compressed outlier activation values ​​are decompressed.

[0135] The number of outliers can be selected based on performance or hardware properties. In general, storing fewer activation values ​​with outliers will reduce storage costs, while storing more activation values ​​with outliers will improve the accuracy of the neural network. In some examples, a predetermined number (e.g., one or several) of activation values ​​with a shared index are stored with the outliers. In some examples, the number of activation values ​​stored with the outliers is determined per memory access. For example, if 128 activation values ​​are stored per memory access, then a predetermined number (e.g., one or several) of activation values ​​with outliers are stored per memory access. In some examples, the number of activation values ​​stored with outliers is predetermined based on the computational granularity. For example, if 16 activation values ​​are generated for a given computational unit (e.g., a single clock cycle or a predetermined number of clock cycles), then a predetermined number (e.g., one or several) of activation values ​​with outliers are stored per computational cycle.

[0136] In some examples, the second block floating point format has the same or higher precision index than the first block floating point format. In some examples, the second block floating point format has a lower precision index than the first block floating point format. In some examples, the first block floating point format uses a shared format that is different from the shared format of the public exponent used for the second block floating point format. For example, the shared format can be different based on each row, column, or block of public exponents of compressed activation values. The precision of the second block floating point format can be adjusted in a variety of different ways, including rounding, truncating, and / or shifting the mantissa value.

[0137] In the illustrated example, the activation values ​​may alternatively be dequantized 724 to a common precision format prior to conversion to the second block floating point format and storage in the mass storage 770. In some examples, the compressor 760 is configured to further compress the activation values ​​in the second block floating point format by performing at least one or more of the following compression operations: entropy compression, zero compression, run-length encoding, compressed sparse row compression, or compressed sparse column compression. In some examples, the activation values ​​y ci The block floating point data is stored in the mass storage 770 with the exception data O ci In some other examples, different memory cells and / or different types of memory cells are used to store the activation value data y ci and outlier data O ci . Abnormal data O ci Index information indicating which activation values ​​the outliers are associated with may also be included, and in some examples, individual indices for each outlier are included. In other examples, similar to other block floating point formats, outlier values ​​0 ci Indices can be shared.

[0138] The bulk memory 770 can be implemented using any suitable memory or storage technology. In some examples, the memory storing temporary values ​​in the quantized layer 710 is typically implemented as static RAM (SRAM), embedded dynamic RAM (eDRAM), implemented in a register file, in a block RAM, or other suitable structure, while the bulk memory 770 is typically implemented in a memory structure that supports larger but generally slower access, such as off-chip DRAM, network accessible RAM, SSD drive, hard disk drive, or network accessible storage. As will be readily appreciated by those of ordinary skill in the art having the benefit of this disclosure, the type and arrangement of memory used to implement the quantized layer 710 and the bulk memory 770 can be selected depending on desired performance attributes such as cost, energy, and speed.

[0139] The decompressor 780 reads the compressed activation values ​​from the bulk storage 770 and reverses the operations performed by the compressor 760. In examples where additional compression is applied to the quantized values, the values ​​are decompressed. The values ​​can then be converted from the second block floating point format to a normal precision floating point format. The decompressor 780 includes an outlier dequantizer 785 that converts the values ​​from the second block floating point format back to the first block floating point format or the normal precision floating point format. For those activation values ​​that have associated outlier values, the outlier dequantizer 785 identifies such values ​​using an outlier index and stores the outlier value 0. ci With block floating point data y ci The two values ​​are combined (eg, by adding them) to recover the corresponding activation value. In an alternative example, the values ​​are output by the decompressor 780 in a quantized block floating point format, as indicated by the dashed line.

[0140] The output error component 720 may receive the activation value in a normal precision floating point format (as in Equation 6a). In an alternative example, the output error component 720 may receive the activation value in a quantized floating point format (as in Equation 6b) (such as a second block floating point format), as indicated by the dashed line. The output error component 720 calculates the derivative of the activation value used in the back propagation. As indicated in the diagram 700, the back propagation is performed using the quantized layer 710.

[0141] The dashed arrows in diagram 700 depict alternative paths for compressing and decompressing activation values. As shown, as successive layers of the neural network are forward propagated, thereby producing a first block of first activation values ​​in floating point format, the compressor can convert the quantized activation values ​​to normal precision floating point format before converting to a second block of floating point format, thereby producing compressed activation values.

[0142] During the back propagation process 730, the output error term from the subsequent layer can be communicated from the normal precision floating point domain to the quantized floating point domain by the quantizer 732. The output error term from a given layer can be communicated from the quantized floating point domain to the normal precision floating point domain by the dequantizer 734. The weights of a given layer can be communicated from the normal precision floating point domain to the quantized floating point domain by the quantizer 742. The weight error term from the given layer can be communicated from the quantized floating point domain to the normal precision floating point domain by the dequantizer 754. As the back propagation proceeds, the activation values ​​of the quantized second block floating point format are converted from the second block floating point format used to store the activation values ​​in the mass storage 770 to the first block floating point format, which is used by the quantized layer 710, or alternatively, converted to the normal precision format and converted to the first block floating point format by the quantizer 732. Further, in examples where additional compression (e.g., entropy coding, zero coding, or other additional compression schemes) is applied before storage in the mass storage 770, the data can be further decompressed before quantization.

[0143] Figure 8 is a diagram 800 illustrating an example of converting activation values ​​to a second block floating point format with outliers for selected activation values. For example, the quantization system 110 including use of the compression environment 700 as discussed above may be used to perform the illustrated operations.

[0144] like Figure 8 As shown in i A set of 810 is received. Each activation value in the set of activation values ​​has an individual exponent and a six-digit mantissa. Shared exponent 2 5 The set of activation values ​​is selected by analyzing. In this particular example, it is determined that the selected sharing index 2 5 820 will allow the maximum number of activation values ​​810 to be expressed without losing an unacceptable amount of data. Two abnormal activation values ​​830 and 831 are also identified that are substantially larger than the other values ​​in the set of activation values. These two abnormal activation values ​​will have auxiliary abnormal data stored, allowing values ​​larger than can be expressed using the shared index 820.

[0145] The mantissas for all activation values ​​810 are shifted 840 based on the shared exponent 820. For all compressed activation values, due to the truncated N-bit block floating point mantissa y ci A set of N additional bits 850 (four bits in the illustrated example) are generated for the portion of the shifted mantissa 840 associated with the shared exponent 820 for both the non-exception activation value and the exception activation value. For both exception activation values, another N additional bits (four bits in the illustrated example) of the mantissa value 860 and one of a plurality of exception indices 870 are generated for each exception value. ci is generated. The exponent associated with each outlier can be determined in a number of different ways. In some examples, the outlier mantissa 860 is simply a predetermined number of mantissa bits to the left of the mantissa bits of all activated values. In other words, the next N more significant bits of the mantissa are selected as the outlier mantissa. In some examples, the outlier mantissa is associated with a shared exponent for two or more outliers. In some examples, each outlier mantissa in the outlier mantissa 860 is associated with an individual exponent. In the illustrated example, the outliers share the same outlier exponent 2 1 875. Each of the exception indices 870 indicates which activation values ​​the corresponding exception mantissa is associated with. In the illustrated example, the exception values ​​are associated with activation values ​​from rows 0 and 3. Thus, by storing an index for each exception value, memory is only used to store additional bits of the mantissa for those activation values ​​that have exception values.

[0146] Fig. 9 is to further elaborate on the above Figure 7900 of the operation of the compressor 760 in question. As shown in the diagram 900, the quantized activation value f(Q(y i-1 ) 910 is received from the quantized layer 710. Alternatively, the dequantizer 724 may generate a normal precision floating point value 920 based on the activation value received from the local memory used to implement the quantized layer 710.

[0147] like Fig. 9 As shown in , the quantized values ​​are expressed in a first block floating point format 910, including a six-bit shared exponent, a sign bit for each individual value, and a six-bit mantissa for each individual value. In some examples, the exponent is not shared by all values ​​in the value set, but can be shared on a per-row, per-column, or per-small-block basis.

[0148] In an alternative example where a set of normal precision floating point values ​​920 are received by the compressor 760, these values ​​are provided to the quantizer 820, which implements a quantization function Q2() that converts the normal precision values ​​to a second block floating point format that is different from the block floating point format used by the quantized layer 710. In the illustrated alternative example, the normal precision floating point values ​​920 each have a 10-bit mantissa, a 5-bit exponent (for each value), and a 1-bit sign bit. These normal precision floating point values ​​910 are converted to values ​​Q2(y) in the first block floating point format 930. i ), each value has a six-bit mantissa, 1-bit sign bit, and all values ​​share a six-bit exponent. In some examples, the exponent is not shared by all values ​​in the value set, but can be shared on a per-row, per-column, or per-block basis.

[0149] The outlier quantizer 765 receives the quantized values ​​in the first block floating point format 910 and identifies the shared index that will be shared by the mantissas of all activation values. As shown, the values ​​from the first block floating point format 910 are converted to the second block floating point format 920 with shared indexes, and each value has a one-bit sign and a three-bit mantissa. The shared index can be selected in a variety of different ways. For example, the most commonly used index of the activation value set can be selected as the shared index. In some examples, the mean, median, or mode is used to select the shared index. In some examples, the shared index is selected based on the available storage device of the outlier value. In other words, the selected number of outliers is identified, and the shared index suitable for storing the remaining activation values ​​is selected. In some examples, the index is not shared by all values ​​in the value set, but can be shared on a per-row, per-column, or per-small-block basis. The second block floating point format also includes data for multiple outliers. For each identified outlier, there is an outlier mantissa. In some examples, there is a shared outlier index for all outliers or a portion of outliers selected on a per-row, per-column, or per-small-block basis. In some examples, each outlier has its own individual index. Further, the outlier is associated with an outlier index, which can be used to identify which activation values ​​the outlier is associated with. The outlier index can be used when decompressing the activation outlier.

[0150] In some examples, the set of values ​​in the second block floating point format 930 may be provided to additional compression units 960 and 961. Examples of suitable techniques for further compressing activation values ​​in the compressed quantized format include entropy compression, zero compression, run-length compression, compressed sparse row compression, or compressed sparse column compression.

[0151] Regardless of whether the quantized values ​​in the second block floating point format 930 are subjected to additional compression, the compressed values ​​y ci and outliers O ci All are stored in the mass storage 770. The mass storage 770 is typically implemented in a memory structure that supports larger but generally slower accesses, such as off-chip DRAM, network accessible RAM, SSD drive, hard disk drive, or network accessible storage. As will be readily appreciated by those of ordinary skill in the art having the benefit of this disclosure, the type and arrangement of memory used to implement the memory (mass storage 770) can be selected depending on desired performance attributes such as cost, energy, and speed. In some examples, outliers are stored in different memories or different portions of the mass storage 770.

[0152] As one of ordinary skill in the related arts having the benefit of this disclosure will readily appreciate, the illustrated common precision and block floating point formats are not limited to the particular selection of mantissa, exponent, and sign bits illustrated, and the scheme for sharing the exponent between values ​​may also vary depending on the desired aspects of a particular implementation of the compressor 760. Further, values ​​stored in the mass storage 770 may be decompressed by reversing the operations illustrated in diagram 900.

[0153] VII. Example Anomaly Quantification Device

[0154] Fig.10 1 is a block diagram 1000 outlining an example anomaly quantizer 1010, as may be implemented in certain examples of the disclosed technology. For example, the illustrated anomaly quantizer may be used to perform operations similar to those described above with respect to Figures 7 to 9 For example, the outlier quantizer discussed in the quantization enabling system 110 discussed above is associated.

[0155] like Fig.10 As shown in , an exception quantizer 1010 receives a plurality of activation values ​​expressed in a floating point format. The floating point format may be a neural precision floating point or a first block floating point format. An exception selector 1025 analyzes the activation values ​​to determine a sharing index 1030. For each activation value 1020, the selected sharing index 1030 is compared to each activation value 1045 in the activation values ​​using a comparator 1040. If a particular activation value 1045 has a different exponent than the sharing index 1030, a shift controller 1050 selects a shift amount and a shifter 1055 is used to shift the mantissa of the activation value to the left or right depending on the selected sharing index 1030.

[0156] In the example shown, each activation value ends with y ci The four bits of the activation value mantissa 1061 are stored in the block floating point memory 1060 as the activation value mantissa 1061. The shift controller 1050 can be used to configure the shifter 1055 to align the four bits of the activation value mantissa to be stored in the block floating point memory 1060 by shifting the mantissa left or right 1057 based on the determination made by the comparator 1040 when comparing the selected shared exponent and the exponent of the particular activation value 1045.

[0157] The exception selector 1025 also determines which activation values ​​1020 will be selected to store the additional bits of the mantissa as exception values. ci. The shift controller 1050 configures the shifter 1055 so that other bits of the mantissa can be selected as abnormal mantissa values. These abnormal mantissa values ​​1071 are stored in the abnormal memory 1070. The abnormal quantizer 1010 also includes an address register 1075 used to select an activation value. The current address of the selected specific activation value 1045 is used to store an indicator 1077, also known as an abnormal index, in the abnormal memory 1070. In some examples, the abnormal memory 1070 also stores an individual exponent 1079 for each abnormal value, as shown in the dotted line. In some other examples, the abnormal value is associated with a shared exponent, or is assumed to be a more significant bit of the mantissa in the first block floating point format. In the illustrated example, for example, the four bits stored on the left side of the block floating point memory 1060 and to be submitted can be stored in the abnormal memory 1070 without explicitly storing the shared exponent. In such a case, the specific abnormal value mantissa may need to be additionally shifted according to the exponent of the given activation value.

[0158] Therefore, the illustrated outlier quantizer 1010 can maintain higher precision for the selected outliers, while still allowing the mantissa of a large number of activation values ​​to be stored in compression. In some examples, the quantizer selects a fixed number of outliers for a set of activation values. For example, every four values ​​in 64 can be selected as outliers. In some other examples, outliers and associated indexes and / or indices can be stored using a data structure (e.g., a stack or queue) that allows any number of outliers. As will be readily understood by those of ordinary skill in the relevant art having the benefit of the present disclosure, different configurations and numbers of outliers and compressed activation values ​​can be selected. In some examples, the outlier memory 1070 and the block floating point memory 1060 are formed from the same storage device or memory. In some other examples, the block floating point memory 1060 can be formed by a memory structure and / or type different from the memory used to form the outlier memory 1070.

[0159] VIII. Example method for generating compressed activation values ​​including outliers

[0160] Fig.11 1100 is a flowchart outlining an example method for generating compressed activation values ​​including outliers, as may be performed in certain examples of the disclosed technology. As an example, the illustrated method may use a quantization-enabled system (such as described above with respect to Figure 1 As discussed above with regard to Figure 7 The quantization enabling system 110 ) described in further detail in the discussed quantization environment is executed. In some examples, the illustrated method is used as part of a method for training a neural network.

[0161] At process block 1110, an activation value in a first floating point format is generated. As further discussed above, the first floating point format can be a normal precision loading point format or a first block floating point format. The activation value can be generated by performing a forward propagation when training a layer of a neural network. In some examples, the value can be further compressed, for example, using entropy compression or another suitable compression scheme. In some examples, the first activation value is converted to a normal precision floating point format before being converted to a second block floating point format. In some other examples, the first activation value is converted directly from the first block floating point format to the second block floating point format.

[0162] At process block 1120, a second value for all activation values ​​is generated in a second block floating point format. For example, the second block floating point format may have a smaller number of mantissa bits than a normal precision floating point format or a first block floating point format. An example of a way to generate the second value is to use a shift controller and a shifter based on an exponent associated with each corresponding activation value. An exception selector may be used to determine a shared exponent. Based on the selected shared exponent, the mantissa bits of the uncompressed activation values ​​may be aligned by shifting the mantissa to the left or right.

[0163] At process block 1130, an exception value is generated for at least one but not all of the activation values. An exception selector can be used to determine a shared index. An activation value greater than a value that can be expressed using a shared index can be designated as an exception. In such a case, a portion of the mantissa that cannot be expressed using a shared index is generated as an exception value. In some examples, a predetermined number of exception values ​​are generated for a given set of activation values. In some examples, a stack or queue is used to store and any number of exception values. An exception index and / or an exception index can be associated with each exception mantissa. Therefore, by storing only exceptions for a subset of activation values, memory storage requirements can be reduced.

[0164] At process block 1140, a second activation value including a second value generated at process block 1120 and an outlier value generated at process block 1130 is stored in a storage device or memory. In some examples, the second value and the outlier value are stored in the same memory. In some other examples, the second value and the outlier value are stored in different memories. In some examples of the disclosed technology, a processor is used to perform the operations outlined in flowchart 1100. In some examples, the processor includes at least one of the following: a tensor processing unit, a neural network accelerator, a graphics processing unit, or a processor implemented in a reconfigurable logic array. In some examples, the outlier value is stored in a memory located on an integrated circuit different from the second value. In some examples, the second value is stored in a memory located on an integrated circuit different from the processor. The mass storage can be implemented in any suitable storage technology, including, for example, on-chip or off-chip DRAM, network accessible RAM, SSD drive, hard disk drive, or network accessible storage. In some examples, the mass storage is located on an integrated circuit different from the hardware accelerator used to process block floating point values. In some examples, the mass storage is located on the same integrated circuit as the hardware accelerator used to process the block floating point values. After the activation values ​​are stored in the mass storage, the computation of the neural network can proceed to the next layer. For example, once the activation values ​​of a layer have been stored in the mass storage, forward propagation can continue for multiple different layers in the neural network.

[0165] IX. Example Method for Anomaly Quantification

[0166] Fig.12 is a flowchart 1200 outlining an example method of generating quantified outliers, as may be performed in certain examples of the disclosed technology.

[0167] For example, how does hardware Fig.10 The discussed anomaly quantifier may be used to implement the illustrated method.

[0168] At process block 1210, an activation value in a first floating point format is received. The first floating point format may be a normal precision floating point format or a first block floating point format. For example, a normal precision floating point format number may be converted to a first block floating point format.

[0169] At process block 1220, the activation values ​​are analyzed to determine a sharing index. Any suitable method for determining a sharing index may be used. For example, the mean, median, or mode of the activation values ​​may be determined and used to determine the sharing index. In some examples, the most common index is selected as the sharing index. In some examples, a predetermined number of outliers to be generated are selected, and a common index is selected based on the accuracy of the remaining compressed activation values. For example, if there are eight outliers, then the remaining activation values ​​may be analyzed to determine a suitable sharing index.

[0170] At process block 1230, the activation value mantissa is shifted to generate a mantissa for a second value according to a shared exponent. A second value for all activation values ​​is generated. The second value may be stored in a storage device or memory. At process block 1240, the activation value may be further shifted to generate an abnormal value mantissa. In some examples, the mantissa is not further shifted, but the other bits of the mantissa are simply selected from the shifter to be stored as an abnormal value. The abnormal value and the abnormal index and the abnormal exponent in some examples may be stored in a storage device or memory. In some examples, the abnormal data and the second mantissa value are stored in the same memory. In some other examples, the abnormal data are stored in different memories or different parts of the same memory. In some examples, a stack or a queue is used to store the value of the abnormal data.

[0171] X. Sample computing environment

[0172] Fig.13 A generalized example of a suitable computing environment 1300 is illustrated in which the described embodiments, techniques, and technologies (including performing activation compression using outliers in a quantized environment) may be implemented.

[0173] The computing environment 1300 is not intended to imply any limitation on the scope of use or functionality of the technology, since the technology can be implemented in different general or special computing environments. For example, the disclosed technology can be implemented with other computer systems (including handheld devices, multiprocessor systems, programmable consumer electronics, network PCs, minicomputers, mainframe computers, etc.). The disclosed technology can also be practiced in a distributed computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

[0174] refer to Fig.13 , the computing environment 1300 includes at least one processing unit 1310 and a memory 1320. Fig.13, the most basic configuration 1330 is included within the dashed line. Processing unit 1310 executes computer executable instructions and can be a real or virtual processor. In a multi-processing system, multiple processing units execute computer executable instructions to increase processing power, and in this way, multiple processors can run simultaneously. Memory 1320 can be a volatile memory (e.g., registers, cache, RAM), a non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two. Memory 1320 stores software 1380, images, and videos that can implement the technology described herein, for example. The computing environment may have additional features. For example, computing environment 1300 includes storage 1340, one or more input devices 1350, one or more output devices 1360, and one or more communication connections 1370. An interconnection mechanism (not shown) (such as a bus, controller, or network) interconnects the components of computing environment 1300. Typically, operating system software (not shown) provides an operating environment for other software executed in computing environment 1300 and coordinates the activities of components of computing environment 1300.

[0175] Storage 1340 may be removable or non-removable and include disks, tapes or cartridges, CD-ROMs, CD-RWs, DVDs, or any other medium that can be used to store information and can be accessed within computing environment 1300. Storage 1340 stores instructions for software 1380, plug-in data, and messages that may be used to implement the techniques described herein.

[0176] Input device(s) 1350 may be a touch input device (such as a keyboard, keypad, mouse, touch screen display, pen, or trackball), a voice input device, a scanning device, or another device that provides input to the computing environment 1300. For audio, input device(s) 1350 may be a sound card or similar device that accepts audio input in analog or digital form, or a CD-ROM reader that provides audio samples to the computing environment 1300. Output device(s) 1360 may be a display, a printer, a speaker, a CD burner, or another device that provides output from the computing environment 1300.

[0177] Communication connection(s) 1370 enable communication with another computing entity over a communication medium (e.g., a connection network). The communication medium transmits information, such as computer executable instructions, compressed graphics information, video, or other data, in a modulated data signal. Communication connection(s) 1370 are not limited to wired connections (e.g., Megabit or Gigabit Ethernet, Infiniband, Fibre Channel via electrical or optical fiber connections) but also include wireless technologies (e.g., via Bluetooth, WiFi (IEEE 802.11a / b / n), WiMax, cellular, satellite, laser, infrared) and other suitable communication connections for providing network connectivity for the disclosed agents, bridges, and proxy data consumers. In a virtual host environment, the communication connection(s) may be a virtualized network connection provided by the virtual host.

[0178] Some embodiments of the disclosed methods may be performed using computer-executable instructions that implement all or a portion of the disclosed techniques in a computing cloud 1390. For example, the disclosed methods may be performed on a processing unit 1310 located in a computing environment 1330, or the disclosed methods may be performed on a server located in a computing cloud 1390.

[0179] Computer-readable media is any available media that can be accessed within the computing environment 1300. By way of example and not limitation, in the case of the computing environment 1300, computer-readable media include memory 1320 and / or storage 1340. As should be readily understood, the term computer-readable storage media includes media for data storage, such as memory 1320 and storage 1340, and does not include transmission media, such as modulated data signals.

[0180] XI. Experimental Results

[0181] Figures 14 to 17 is a graph plotting experimental data obtained when training an artificial neural network using activation compression with outliers.

[0182] Fig.141400 is a graph of a test case named ResNet-50. A plurality of graphs indicate the accuracy of a neural network when activation compression is performed using the indicated block floating point format. For example, "6w6a" indicates a block floating point format with a six-bit wide mantissa and a six-bit wide exponent. As another example, 4w6a indicates a block floating point format with a four-bit wide mantissa and a six-bit wide exponent. The accuracy obtained relative to a known set of test results is plotted on the y-axis. As shown, a first graph 1410 indicates that the most accurate result for this test case is obtained when an individual exponent is used for each block floating point value. A second graph 1420 indicates the accuracy obtained when two exceptions are used for every 128 activation values, where the exceptions are stored as normal precision floating points. A third graph 1425 indicates the accuracy obtained when two exceptions are used for every 128 activation values, where the exceptions are stored with "fracture precision", where the exception values ​​are stored together with the number of mantissa bits used for the block floating point format. In other words, the 6w6a case is stored with a 6-bit outlier and the 4w6a case is stored with a 4-bit outlier. The fourth graph 1430 and the fifth graph 1435 indicate the accuracy achieved when one outlier is used for every 128 activation values ​​in the case of normal precision and broken precision outliers. Thus, as shown, using broken format outliers (in this example, the outliers have the same number of mantissa bits as the BFP format values) can achieve similar accuracy to full precision outliers or even cases with all private exponents, while using lower cost hardware that can be easily integrated with block floating point hardware.

[0183] Fig.15 is with Fig.14 1500 for the same test case in FIG. 1501 . As shown, using an increasing number of outliers can improve the accuracy results obtained when training a neural network. As shown, the first graph 1510 indicates that the most accurate results for this test case are obtained when using individual exponents (private exponents) for each block floating point value. The second graph 1520 indicates the accuracy obtained when eight outliers are used per 128 activation values; the accuracy is comparable to the results obtained using all private exponents. The third graph 1530, the fourth graph 1540, and the fifth graph 1550 indicate the accuracy obtained when four, two, and one outlier are used per 128 activation values, respectively. The sixth graph 1560 indicates the accuracy obtained when a shared exponent across all 128 activation values ​​is used without outliers. Therefore, as shown, using more outliers improves accuracy and is comparable to the private exponent case for the eight outlier case.

[0184] Fig.16 is with Fig.13 and Fig.14 The same test case used in , but with every 16 activations instead of Fig.14 and Fig.15 1600 with multiple outliers for every 128 values ​​in . As shown, the accuracy of the test case using outliers is similar to the accuracy of using private exponents, which depends on the number of mantissa and exponent bits in the specific format. As shown, the first curve chart 1610 indicates that the most accurate results for this test case are obtained when using individual exponents (private exponents) for each block floating point value. The second curve chart 1620 indicates the accuracy obtained when using one normal precision floating point outlier for every 16 activation values. The third curve chart 1630 indicates the accuracy obtained when using one fractured precision outlier for every 16 activation values. The fourth curve chart 1640 indicates the accuracy obtained when using a shared exponent across every 16 activation values ​​without outliers.

[0185] Fig.17 1700 is a graph of different test cases named Inception-V3. As shown, the accuracy obtained using compressed outliers can approach the accuracy obtained using individual exponents for all test cases. The first graph 1710 indicates that the most accurate results for this test case are obtained when using individual exponents (private exponents) for each block floating point value. The second graph 1720 indicates the accuracy obtained when using one normal precision floating point outlier for every 16 activation values. The third graph 1730 indicates the accuracy obtained when using one fractured precision outlier for every 16 activation values. The fourth graph 1740 indicates the accuracy obtained when using a shared exponent across every 16 activation values ​​in the absence of outliers.

[0186] XII. Additional Examples of the Disclosed Technology

[0187] Additional examples of the disclosed subject matter are discussed herein in light of the examples discussed above.For example, the quantization enabling system 110 discussed above and the various techniques disclosed for generating and using outliers can be used in accordance with the specific examples and aspects of the disclosure discussed below.

[0188] A system of one or more computers may be configured to perform a particular disclosed neural network operation or action by having software, firmware, hardware, or a combination thereof installed on the system, the software, firmware, hardware, or a combination thereof causing the system to perform the operation in operation. One or more computer programs may be configured to perform a particular operation or action by including instructions that, when executed by a data processing device, cause the device to perform the action. In some examples of the disclosed technology, a computing system includes one or more processors, a memory (including a computer-readable storage device and / or a computer-readable memory), and an outlier quantizer formed from at least one processor and in communication with the memory. The computing system is configured to: generate, using at least one of the processors, first activation values ​​in a first floating point format, convert, using the outlier quantizer, one or more of the first activation values ​​to an outlier block floating point format, thereby generating a second activation value, the second activation value including: (1) a second value in a second block floating point format for all of the first activation values, and (2) an outlier value for at least one of the first activation values ​​but not all of the first activation values, and store, using at least one of the processors, the second activation value and the outlier value in the memory. In some examples of the computing system, the first floating point format is a block floating point format. In some examples, each of the second activation values ​​includes a mantissa having fewer bits than its corresponding mantissa in the first floating point format. In some examples, each outlier value includes a corresponding outlier exponent.

[0189] In some examples, the outlier quantizer includes: an outlier selector configured to select a shared index for the second value. In some examples, the outlier quantizer also includes: a shift controller coupled to the shifter, the shifter configured to: shift the first activation value based on the selected shared index to generate a mantissa for the second value and a mantissa for the outlier value. In some examples, the outlier quantizer also includes a comparator to identify whether a specific first activation value in the first activation values ​​is selected as an outlier value in the outlier values. In some examples, the outlier quantizer also includes an address register, and wherein the outlier quantizer is configured to store an index indicating an address for at least one of the outlier values ​​in the memory. In some examples, the outlier quantizer also includes a shifter, the shifter configured to shift the mantissa of the first activation value according to the shared index selected for the second value, a portion of each shifted first activation value forming a mantissa for a corresponding second value in the second value, and different portions of the shifted first activation values ​​forming an outlier mantissa for the second activation value having an outlier value. In some examples of computing systems, the processor includes at least one of: a tensor processing unit, a neural network accelerator, a graphics processing unit, or a processor implemented in a reconfigurable logic array; and the memory is located on an integrated circuit different from the processor, the memory includes dynamic random access memory (DRAM) or embedded DRAM, and the system also includes a hardware accelerator, the hardware accelerator includes a memory to temporarily store a first activation value for at least a portion of a layer of the neural network, the hardware accelerator memory includes static RAM (SRAM) or a register file.

[0190] In some examples of the disclosed technology, a method of operating a computing system that implements a neural network includes, with the computing system: generating, in a first block of floating point formats, first activation values; converting at least one but not all of the first activation values ​​to an exception block of floating point formats that is different from the first block of floating point formats, thereby generating exception activation values; and storing the exception activation values ​​in a computer-readable memory or a computer-readable storage device.

[0191] In some examples of the method, each abnormal activation value includes a first mantissa and a second mantissa, the first mantissa is associated with a shared index shared by all abnormal values, and the second mantissa is associated with an index different from the shared index. In some examples, the first mantissa and the second mantissa each include the same number of digits. In some examples, the method also includes identifying the shared index by determining at least one of the mean, median and / or mode of at least a portion of the first activation value. In some examples, the method also includes generating uncompressed activation values ​​by reading the stored abnormal activation values ​​from a computer-readable memory or a computer-readable storage device, wherein for each uncompressed activation value in the uncompressed activation value: when the uncompressed activation value is associated with the abnormal activation value, the corresponding uncompressed activation value in the uncompressed activation value is generated by combining a first value defined by a first mantissa for the uncompressed activation value and the shared index with a second value defined by a second mantissa, the second mantissa being associated with the abnormal index associated with the uncompressed activation value, and when the uncompressed activation value is not associated with the abnormal activation value, the corresponding uncompressed activation value in the uncompressed activation value is generated by generating a first value defined by a first mantissa for the uncompressed activation value and the shared index.

[0192] In some examples, the method further includes compressing the outlier activation values ​​stored in the computer-readable memory or computer-readable storage device prior to storage by one or more of the following techniques: entropy compression, zero compression, run-length encoding, compressed sparse row compression, or compressed sparse column compression.

[0193] In some examples, the method also includes: generating a first activation value by performing a forward propagation for at least one layer of the neural network; performing a backward propagation for at least one layer of the neural network by converting the stored second activation value into an activation value in a first block floating point format, thereby generating an uncompressed activation value; performing a gradient operation using the uncompressed activation value; and updating a weight for at least one node of the neural network based on the uncompressed activation value.

[0194] In some examples of the disclosed technology, one or more computer-readable storage devices or computer-readable storage media store computer-executable instructions that, when executed by a computer, cause the computer to perform a method of configuring a computer system to implement an artificial neural network, the instructions comprising: instructions causing the computer system to generate first activation values ​​in a first floating point format; instructions causing the computer system to generate a second activation value for each of the first activation values ​​by: determining that some of the first activation values ​​are to be stored as outliers having first mantissas associated with a shared exponent and corresponding second mantissas associated with the corresponding outlier exponents, and determining that remaining values ​​of the first activation values ​​are to be stored as the first mantissas associated with the shared exponent without the associated second mantissas; and storing the second activation values ​​in a computer-readable storage device or a computer-readable memory.

[0195] In some examples, the instructions also include instructions for causing the computer system to determine a shared index by: identifying a group of the largest outliers in the first activation values; and for the remaining groups of first activation values ​​not identified as being in the group of the largest outliers, determining the shared index by selecting the largest index in the remaining groups. In some examples, the instructions also include instructions for causing the computer system to perform a forward propagation for the artificial neural network to generate the first activation values; and instructions for causing the computer system to perform a backward propagation for the artificial neural network by converting at least some of the stored second activation values ​​into a first floating point format by: for the outliers, adding a first value described by a corresponding first mantissa of the outlier value and a shared index to a second value described by a corresponding second mantissa of the outlier value and a corresponding outlier index, and for the remaining values, generating a value described by a corresponding first mantissa of the remaining value and a shared index.

[0196] In some examples of computing systems, the processor includes at least one of: a tensor processing unit, a neural network accelerator, a graphics processing unit, or a processor implemented in a reconfigurable logic array; and the mass memory is located on an integrated circuit different from the processor. The computing system may also include a mass memory (including dynamic random access memory (DRAM) or embedded DRAM), and the system also includes a hardware accelerator, the hardware accelerator including a memory that temporarily stores a first activation value of at least a portion of only one layer of the neural network, the hardware accelerator memory including static RAM (SRAM). In some examples, the mass memory includes dynamic random access memory (DRAM) or embedded DRAM, and the system also includes a hardware accelerator, the hardware accelerator including a memory that temporarily stores a first activation value of at least a portion of only one layer of the neural network, the hardware accelerator memory including static RAM (SRAM) or a register file.

[0197] In some examples of the disclosed technology, a method of operating a computing system that implements a neural network includes using the computing system to: forward propagate a layer of the neural network to generate activation values ​​in a first block of floating point format; convert at least one activation value to a second block of floating point format different from the first block of floating point format to generate a compressed activation value, and store the compressed activation value in a computer-readable memory or a computer-readable storage device. Implementations of the described technology may include hardware, methods or processes, or computer software on a computer-accessible medium or storage device.

[0198] A general aspect of the method includes where the second block floating point format differs from the first block floating point format in at least one of the following ways: a different mantissa format, a different exponent format, an outlier format, or a different exponent sharing scheme. The method also includes a method where the second block floating point format has a lower numerical precision than the first block floating point format. Other examples of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0199] Implementations may include one or more of the following features. For example, the method also includes: further compressing the compressed activation values ​​stored in the computer-readable memory or computer-readable storage device before storage by further compressing the compressed activation values ​​stored in the computer-readable memory or computer-readable storage device by one or more of the following techniques before storage: entropy compression, zero compression, run-length encoding, compressed sparse row compression, or compressed sparse column compression. In some examples, the method also includes: using the computing system, performing back propagation for the layers of the neural network by converting the stored compressed activation values ​​to activation values ​​in a first block of floating point format to uncompressed activation values. The method may also include using the computing system to perform a gradient operation using the uncompressed activation values. The method may also include using the computing system to update the weights of at least one node of the neural network based on the uncompressed activation values, and performing back propagation for the layers of the neural network by converting the stored compressed activation values ​​to activation values ​​in a first block of floating point format to uncompressed activation values. The method may also include using the computing system to perform a gradient operation using the uncompressed activation values; and using the computing system to update a weight of a portion of at least one node of the neural network based on the uncompressed activation values, wherein the at least one node is one of: a long short-term memory node (LSTM) or a gated recurrent unit (GRU). The method also includes: selecting a second block of floating-point formats based on an attribute of the layer, the attribute being selected from the group consisting of: the layer is a convolutional layer, the layer includes a long short-term memory node (LSTM), or the layer includes a gated recurrent unit (GRU). The second block of floating-point formats may be selected based on an attribute of the layer, the attribute being selected from the group consisting of one or more of: the layer is a convolutional layer, the layer includes a long short-term memory node (LSTM), the layer includes a gated recurrent unit (GRU), the layer is fully connected to another layer, the layer is sparsely connected to another layer, the layer is an attention layer, and the layer is a normalization layer. One or more computer-readable storage devices or computer-readable storage media store computer-executable instructions that, when executed by a computer, cause the computer to perform a method of configuring a computer system to implement an artificial neural network, the instructions including instructions causing the computer system to implement a first layer of a neural network using first weights and / or first activation values ​​expressed in a first block floating point format. The one or more computer-readable storage devices also include instructions causing the computer system to forward propagate values ​​from the first layer of the neural network to a second layer of the neural network, thereby generating second activation values ​​expressed in the first block floating point format. The one or more computer-readable storage devices also include instructions causing the computer system to store the second activation values ​​in a second different block floating point format with an outlier value in a mass storage or storage device in communication with the computer system before performing a back propagation for the neural network.In some examples, one or more computer-readable storage devices further include instructions that cause the computer system to temporarily store the first weight and / or the first activation value in a memory different from the bulk memory or storage device. In some examples, one or more computer-readable storage devices further include instructions that cause the computer system to further compress the second activation value before storing the further compressed value in the bulk memory or storage device. In some examples, the instructions cause the computer system to select a second block floating point format based on aspects of the second layer. The computer-readable storage device may also include an apparatus and method for training a neural network accelerator using a quantized precision data format.

[0200] In some examples, the input tensor may be a two-dimensional matrix. Converting the input tensor from a normal precision floating point format to a block floating point format may include selecting a plurality of elements within a column of the two-dimensional matrix to share a common exponent in the block floating point format. Converting the input tensor from a normal precision floating point format to a block floating point format may include selecting a plurality of elements within a row of the two-dimensional matrix to share a common exponent in the block floating point format. A tensor may have any number of dimensions: one, two, three, four, five, or more, depending on the particular application.

[0201] In some examples of the disclosed technology, one or more computer-readable storage devices or computer-readable storage media store computer-executable instructions that, when executed by a computer, cause the computer to perform any one or more of the disclosed methods.

[0202] In view of the many possible embodiments to which the principles of the disclosed subject matter can be applied, it should be recognized that the illustrated embodiments are merely preferred examples and should not be considered to limit the scope of the claims to those preferred examples. Instead, the scope of the claimed subject matter is defined by the following claims. We therefore claim as our invention all that falls within the scope of these claims.

Claims

1. A computing system comprising: At least one hardware processor, at least one memory coupled to the at least one hardware processor, and computer readable instructions or hardware implemented logic configured to perform operations comprising: In a neural network, generating a plurality of first activation values ​​in a first floating point format for a set of input training data values; determining one or more outlier activation values ​​in at least a portion of the plurality of first activation values, wherein the outlier activation values ​​have additional bits of precision compared to non-outlier activation values ​​in the at least a portion of the plurality of first activation values; converting the at least a portion of the plurality of first activation values ​​in the first floating point format to a first block floating point format, wherein the first block floating point format uses a first shared common exponent, the converting providing at least one first non-exceptional activation value of the plurality of first activation values ​​in the first block floating point format and at least one first exceptional activation value of the one or more exceptional activation values ​​of the at least a portion of the plurality of first activation values ​​in the first block floating point format; converting, with an outlier quantizer, the additional bits of precision of the at least one first outlier activation value of the one or more outlier activation values ​​of the at least a portion of the plurality of first activation values ​​to a second block floating point format using a second shared common exponent selected independently of the first shared common exponent to produce a second activation value; storing the at least one first non-exception activation value in the first block floating point format, storing the at least one first exception activation value in the first block floating point format, and storing the additional bits of precision of the at least one first exception activation value in the second block floating point format to provide a stored compressed activation value; and During back-propagation, uncompressed activation values ​​corresponding to the at least one first non-anomalous activation value and the at least one first abnormal activation value are generated from the stored compressed activation values. 2 . The computing system of claim 1 , wherein the first floating point format is a third block floating point format. 3 . The computing system of claim 1 , wherein the activation value of the second activation value includes a mantissa having fewer bits than its corresponding mantissa in the first floating point format. 4 . The computing system of claim 1 , wherein the anomaly quantizer is configured to select the second shared common exponent.

5. The computing system of claim 1 , wherein the outlier quantizer comprises a shift controller coupled to a shifter, the shifter configured to shift the at least a portion of the plurality of first activation values ​​based on the selected first shared common exponent to produce a mantissa for the activation values ​​for the first block floating point format.

6. The computing system of claim 1 , wherein the anomaly quantifier comprises: A comparator is used to identify whether a first activation value of at least a portion of the plurality of first activation values ​​is selected as an abnormal activation value of the one or more abnormal activation values.

7. The computing system of claim 1 , wherein the anomaly quantizer comprises an address register, and wherein the anomaly quantizer is configured to store in the at least one memory an index indicating an address for at least one of the one or more anomaly activation values.

8. The computing system of claim 1, wherein: The outlier quantizer includes a shifter configured to shift mantissas of the at least a portion of the plurality of first activation values ​​according to the first shared common exponent.

9. The computing system of claim 1, wherein: The at least one hardware processor includes at least one of: a tensor processing unit, a neural network accelerator, a graphics processing unit, or a processor implemented in a reconfigurable logic array; and The at least one memory is located on an integrated circuit different from the at least one hardware processor, the memory comprising dynamic random access memory (DRAM) or embedded DRAM, and the computing system also includes a hardware accelerator, the hardware accelerator comprising a memory for temporarily storing at least a portion of the plurality of first activation values ​​for at least a portion of a layer of the neural network, the hardware accelerator memory comprising static RAM (SRAM) or a register file.

10. The computing system of claim 1, wherein the neural network is further configured to: Respective index values ​​of the one or more exception activation values ​​are stored, the index values ​​associating values ​​of the first block of floating point formats of the respective exception activation values ​​of the one or more exception activation values ​​with values ​​corresponding to the respective exception activation values ​​of the one or more exception activation values ​​in the second block of floating point formats.

11. The computing system of claim 1, wherein the first block floating point format is a common format as the second block floating point format except for using the first shared public exponent and the second shared public exponent.

12. The computing system of claim 1, wherein the second shared public exponent is selected such that the value in the second block floating point format has the same number of mantissa bits as the value in the first block floating point format.

13. The computing system of claim 1, wherein a corresponding outlier value of the one or more outlier activation values ​​has a first mantissa associated with the first shared public exponent and a second mantissa associated with the second shared public exponent.

14. A method for implementing a neural network, the method comprising: Using a computing system implementing the neural network: generating a plurality of first activation values ​​for a tensor of a set of input training data values ​​in a first block of floating point format, wherein the plurality of first activation values ​​share a first shared common exponent in the first block of floating point format and include at least one first non-outlier activation value in the plurality of first activation values ​​and at least a first outlier activation value in the plurality of first activation values, wherein the outlier activation value has additional bits of precision compared to the non-outlier values ​​in the plurality of first activation values; converting the additional bits of precision of the at least one first exception activation value to a second block floating point format using a second shared public exponent, wherein the exception activation value is less than all first activation values ​​in the plurality of first activation values, the exception activation values ​​in the second block floating point format having a second shared public exponent, the second shared public exponent being selected independently of the first shared public exponent, thereby producing a second activation value; storing the at least one first non-exception activation value in the first block floating point format, storing the at least one first exception activation value in the first block floating point format, and storing the additional bits of precision of the at least one first exception activation value in the second block floating point format to provide a stored compressed activation value; as well as During back-propagation, uncompressed activation values ​​corresponding to the at least one first non-anomalous activation value and the at least one first abnormal activation value are generated from the stored compressed activation values.

15. The method of claim 14, the at least one first exception activation value having a first mantissa associated with the first shared public exponent and a second mantissa associated with the second shared public exponent. The method of claim 15 , wherein the first mantissa and the second mantissa each comprise the same number of bits.

17. The method according to claim 15, further comprising: The first shared common index is identified by determining at least one of a median and / or a mode for at least a portion of the plurality of first activation values.

18. The method according to claim 14, further comprising: In generating uncompressed activation values, abnormal activation values ​​stored in the first block floating point format and the second block floating point format are read from the stored compressed activation values, wherein for each uncompressed activation value in the uncompressed activation values: When the uncompressed activation value is associated with an abnormal activation value, a corresponding uncompressed activation value in the uncompressed activation values ​​is generated by: combining a first value defined by a first mantissa for the uncompressed activation value and the first shared common exponent with a second value defined by a second mantissa associated with the second shared common exponent associated with the uncompressed activation value, and When the uncompressed activation value is not associated with an abnormal activation value, a corresponding uncompressed activation value among the uncompressed activation values ​​is generated by: A first value defined by a first mantissa for the uncompressed activation value and the first shared common exponent is generated.

19. The method according to claim 14, further comprising: With the computing system implementing the neural network, respective index values ​​are stored for respective ones of the plurality of exception activation values, the index values ​​associating values ​​of the second block of floating point format for respective ones of the plurality of exception activation values ​​with values ​​corresponding to the respective ones of the plurality of exception activation values ​​in the first block of floating point format.

20. One or more computer-readable storage devices or media storing computer-executable instructions that, when executed by a computer, cause the computer to perform operations comprising: generating a first activation value in floating point format for a first block of a set of input training data values ​​for the neural network; converting the first activation value to a second activation value in a first block floating point format having a first shared common exponent, the converting providing at least one first non-exception activation value among a plurality of the first activation values ​​in the first block floating point format and at least one first exception activation value among the plurality of the first activation values ​​in the first block floating point format, wherein the exception activation values ​​have additional bits of precision compared to the non-exception activation values ​​in the first block floating point format; converting the additional bits of precision of the at least one first exception activation value of the one or more exception values ​​to a second block floating point format using a second shared public exponent, wherein the second shared public exponent is selected independently of the first shared public exponent, thereby producing a second activation value; storing the at least one first non-exception activation value in the first block floating point format, storing the at least one first exception activation value in the first block floating point format, and storing the additional bits of precision of the at least one first exception activation value in the second block floating point format to provide a stored compressed activation value; as well as During back-propagation, uncompressed activation values ​​corresponding to the at least one first non-anomalous activation value and the at least one first abnormal activation value are generated from the stored compressed activation values.

21. One or more computer-readable storage devices or media according to claim 20, wherein the at least one first activation value of the second block floating point format includes a first mantissa associated with the second shared common exponent, and the value corresponding to the exception activation value of the second block floating point format includes a second mantissa associated with the first shared common exponent.

22. The one or more computer-readable storage devices or media of claim 21, wherein the first mantissa and the second mantissa each include the same number of bits.

Images