Neural network activation compression with non-uniform mantissa

Abstract

Apparatuses and methods for training a neural network accelerator using quantization precision data formats are disclosed, and in particular for storing activation values from a neural network in a compressed format having lossy or non-uniform mantissa for use during forward and backward propagation training of the neural network. In certain examples of the disclosed technology, a computing system includes a processor, a memory, and a compressor in communication with the memory. The computing system is configured to perform forward propagation for a layer of a neural network to produce first activation values in a first block floating point format. In some examples, the activation values generated by the forward propagation are converted by the compressor to a second block floating point format having a non-uniform and / or lossy mantissa. The compressed activation values are stored in the memory, where they can be retrieved for use during backward propagation.

Description

Background Technology

[0001] Machine learning (ML) and artificial intelligence (AI) technologies can be used to solve many complex computational problems, such as recognizing images and speech, analyzing and classifying information, and performing various classification tasks. Machine learning is a field of computer science that uses statistical techniques to enable computer systems to extract higher-level features from training datasets. Specifically, features can be extracted by training models such as artificial neural networks (NNs) or deep neural networks (DNNs). After the model is trained, new data can be applied to the model, and the trained model can be used to classify new data (e.g., extract higher-level features). Machine learning models are typically executed on general-purpose processors (also known as central processing units (CPUs)). However, training and / or using a model can be computationally expensive, and therefore feature extraction may not be possible to perform in real time using a general-purpose processor. Therefore, there are numerous opportunities to improve computer hardware and software to enable neural networks. Summary of the Invention

[0002] Apparatus and methods for storing activation values ​​from a neural network in a compressed format for use during forward and backward propagation training of the neural network are disclosed. Computational systems suitable for employing such neural networks include computers with general-purpose processors, neural network accelerators, or reconfigurable logic devices such as field-programmable gate arrays (FPGAs). Activation values ​​generated during forward propagation can be "hidden" (temporarily stored in a large-capacity memory) in a compressed format and retrieved for use during backward propagation. Activation values ​​used during training can be expressed in quantized or block floating-point (BFP) format. The stored activation values ​​can be in a format that is further compressed than the format used during training. In some examples, the compressed format includes lossy or non-uniform mantissas for compressing the values.

[0003] In some examples of the disclosed technology, the computer system includes a general-purpose and / or special-purpose neural network processor, a mass storage device including a computer-readable storage device or memory, and a block floating-point compressor communicating with the mass storage device. Due to forward propagation occurring during neural network training, activation values ​​are generated in a first block floating-point format. Block floating-point is used to convert the activation values ​​into a number format with a numerical precision less than that of the first block floating-point format. The compressed activation values ​​are stored in the mass storage device for use during backpropagation.

[0004] This summary is provided to introduce, in a simplified form, the selection of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. The foregoing and other objects, features, and advantages of the disclosed subject matter will become more apparent from the following detailed description with reference to the accompanying drawings. Attached Figure Description

[0005] Figure 1 It is a block diagram of a system for performing activation compression with enabled quantization, as may be implemented in some examples of the disclosed techniques.

[0006] Figure 2 It is a diagram depicting an example of a deep neural network, as can be modeled using some of the example methods and apparatuses disclosed in this paper.

[0007] Figure 3 It is a diagram depicting certain aspects of converting a normal floating-point format to a quantized floating-point format, as can be performed in some examples of the disclosed techniques.

[0008] Figure 4 Several example block floating-point formats that can be used to represent quantized neural network models are described, as can be used in some examples of the disclosed techniques.

[0009] Figure 5 Several example block floating-point formats that can be used to represent quantized neural network models are described, as can be used in some examples of the disclosed techniques.

[0010] Figure 6 It is a flowchart depicting an example method of training and quantizing a neural network for use, as can be implemented in some examples of the disclosed techniques.

[0011] Figure 7 It is a block diagram depicting an example environment for implementing active compression with a non-uniform mantissa floating-point format, as may be implemented in some examples of the disclosed techniques.

[0012] Figure 8 It is a block diagram depicting an example of active compression with a non-uniform mantissa floating-point format, as can be implemented in some examples of the disclosed techniques.

[0013] Figure 9 The figure illustrates the conversion of a uniform three-digit mantissa format to a non-uniform four-valued lossy mantissa format, as can be achieved in some examples of the disclosed techniques.

[0014] Figure 10The diagram illustrates the conversion of a uniform three-digit mantissa format to a non-uniform three-valued lossy mantissa format, as can be achieved in some examples of the disclosed techniques.

[0015] Figure 11 The diagram illustrates the conversion of a uniform three-digit mantissa format to a non-uniform binary lossy mantissa format, as can be achieved in some examples of the disclosed techniques.

[0016] Figure 12 The diagram illustrates the conversion of a sign value and a three-digit mantissa format into a non-uniform five-value lossy mantissa format, as can be achieved in some examples of the disclosed techniques.

[0017] Figure 13 This is a flowchart outlining an example method using compressed activation values ​​with non-uniform tails, as can be performed in some examples of the disclosed techniques.

[0018] Figure 14 This is a flowchart outlining an example method for training a neural network using activation compression with non-uniform mantissa floating-point values, as can be performed in some examples of the disclosed techniques.

[0019] Figure 15 This is a block diagram illustrating a suitable computing environment for implementing some examples of the disclosed techniques.

[0020] Figure 16 The graph shows the accuracy results obtained when training neural networks using different non-uniform tail schemes, as can be observed when implementing some examples of the disclosed techniques. Detailed Implementation

[0021] I. General Considerations

[0022] This disclosure is set forth in the context of representative embodiments, which are not intended to be limiting in any way.

[0023] As used herein, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” include the plural forms. Furthermore, the term “comprising” means “including.” Additionally, the term “coupled” covers mechanical, electrical, magnetic, optical, and other practices that couple or link items together, and does not exclude the presence of intermediate elements between the coupled items. Furthermore, as used herein, the term “and / or” means any one or more of the terms in the phrase.

[0024] The systems, methods, and apparatuses described herein should not be construed as limiting in any way. Rather, this disclosure is directed toward all novel and non-obvious features and aspects of the various disclosed embodiments, individually and in various combinations and sub-combinations with each other. The disclosed systems, methods, and apparatuses are not limited to any particular aspect or feature or combination thereof, nor are the disclosed things and methods required to have any one or more particular advantages or problems solved. Furthermore, any feature or aspect of the disclosed embodiments may be used in various combinations and sub-combinations with each other.

[0025] Although the operations of some disclosed methods are described in a specific, sequential order for ease of presentation, it should be understood that this descriptive approach involves rearrangement unless the specific language described below requires a particular order. For example, operations described sequentially may be rearranged or performed concurrently in some cases. Furthermore, for simplicity, the accompanying drawings may not show the various ways in which the disclosed things and methods can be used in conjunction with other things and methods. Additionally, the description sometimes uses terms such as “produce,” “generate,” “display,” “receive,” “verify,” “execute,” “perform,” “transform,” and “initiate” to describe the disclosed methods. These terms are high-level descriptions of the actual operations performed. The actual operations corresponding to these terms will vary depending on the specific implementation and will be readily discernible to those skilled in the art.

[0026] The operational theories, scientific principles, or other theoretical descriptions of the apparatuses or methods described herein are provided for the purpose of better understanding and are not intended to limit the scope. The apparatuses and methods in the appended claims are not limited to those that operate in a manner described by these operational theories.

[0027] Any disclosed method can be implemented as computer-executable instructions stored on one or more computer-readable media (e.g., computer-readable media such as one or more optical discs, volatile memory components such as DRAM or SRAM, or non-volatile memory components such as hard disk drives) and executed on a computer (e.g., any commercial computer, including smartphones or other mobile devices that include computing hardware). Any computer-executable instructions used to implement the disclosed technology, and any data created and used during the implementation of the disclosed embodiments, can be stored on one or more computer-readable media (e.g., computer-readable storage media). The computer-executable instructions can be, for example, part of a software application or a dedicated software application accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software can be executed, for example, on a single local computer or using one or more network computers in a networked environment (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network)) or other such networks.

[0028] For clarity, only selected aspects of the software-based implementation are described. Other details well-known in the art have been omitted. For example, it should be understood that the disclosed techniques are not limited to any particular computer language or program. For example, the disclosed techniques can be implemented by software written in C, C++, Java, or any other suitable programming language. Certain details of suitable computers and hardware are well-known and do not need to be elaborated in this disclosure.

[0029] Furthermore, any software-based implementation (including, for example, computer-executable instructions for causing a computer to perform any of the disclosed methods) can be uploaded, downloaded, or remotely accessed via suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, intranets, software applications, cables (including fiber optic cables), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communication means.

[0030] II. Overview of Quantitative Artificial Neural Networks

[0031] Artificial neural networks (ANNs, or "NNs" as used throughout this article) are applied to many applications in artificial intelligence and machine learning, including image recognition, speech recognition, search engines, and other suitable applications. Processing for these applications can occur on individual devices such as personal computers or mobile phones, but can also be performed in large data centers. Meanwhile, hardware accelerators that can be used with NNs include dedicated NN processing units, such as Tensor Processing Units (TPUs) and Field-Programmable Gate Arrays (FPGAs) programmed to accelerate neural network processing. Due to their flexible nature and low power consumption per unit of computation, such hardware devices are being deployed in consumer devices and data centers.

[0032] Traditionally, single-precision floating-point (32-bit floating-point or float32 format) has been used to train and deploy neural networks (NNs). However, it has been shown that lower-precision floating-point formats, such as 16-bit floating-point (float16) or fixed-point formats, can be used to perform inference operations with minimal loss of accuracy. On dedicated hardware such as FPGAs, lower-precision formats can significantly improve the latency and throughput of DNN processing.

[0033] Numbers represented in normal precision floating-point format (e.g., floating-point numbers expressed in 16-bit, 32-bit, 64-bit, or 80-bit floating-point format) can be converted to quantized precision format, which can provide performance benefits when performing operations. Specifically, NN weights and activation values ​​can be represented in lower precision quantization formats, introducing an acceptable level of error. Examples of lower precision quantization formats include formats with low bit widths (including by reducing the number of bits used to represent the mantissa or exponent) and block floating-point formats where two or more numbers share the same single exponent.

[0034] One characteristic of computation on FPGA devices is their typically lack of hardware floating-point support. While flexible logic can perform floating-point operations under penalty, the amount of logic required to support floating-point in FPGA implementations is often prohibitively large. Some newer FPGAs have been developed that support floating-point computation, but even on these FPGAs, the same device produces twice the computational output per unit time compared to using it in integer mode. Typically, neural networks (NNs) are created with floating-point computation in mind, but when an FPGA is targeted for NN processing, it would be beneficial if neural networks could be expressed using integer arithmetic. Examples of disclosed techniques include hardware implementations of block floating-point (BFP), including the use of BFP in NNs, FPGAs, and other hardware environments.

[0035] A typical floating-point representation in a computer system consists of three parts: the sign (s), the exponent (e), and the mantissa (m). The sign indicates whether the number is positive or negative. The exponent and mantissa are used in scientific notation:

[0036] Value = s × m × 2 e (Equation 1)

[0037] Within the precision limits of the mantissa, any number can be represented. Because the exponent is scaled by powers of 2 on the mantissa, just as the exponent is scaled by powers of 10 in scientific notation, very large numbers can be represented. The precision of the representation is determined by the precision of the mantissa. Typical floating-point representations use mantissas with a width of 10 (float16), 24 (float32), or 53 (float64) bits. (The last sentence appears to be incomplete and possibly refers to a different concept: "size greater than 2..."). 53 An integer can be approximated using float64 floating-point format, but it cannot be represented precisely because there aren't enough bits in the mantissa. A similar effect can occur for any fraction, where the fraction is represented by bits in the mantissa that are negative powers of 2. Many fractions cannot be represented precisely because they are irrational numbers in the binary number system. In both cases, there could be more precise representations, but they might require more bits in the mantissa. Ultimately, an infinite number of mantissa bits would be needed to precisely represent some numbers (e.g., ...). The 10-bit (half-precision floating-point), 24-bit (single-precision floating-point), and 53-bit (double-precision floating-point) mantissa limits are common trade-offs between mantissa storage requirements and representation precision in general-purpose computers.

[0038] For block floating-point format, a group of two or more numbers uses a single shared exponent, each number still having its own sign and mantissa. In some examples, the shared exponent is chosen to be the largest exponent of the original floating-point value. For the purposes of this disclosure, the term block floating-point (BFP) means a number system in which a single exponent is shared among two or more values, each value being represented by a sign and mantissa pair (regardless of whether there is an explicit sign bit or whether the mantissa itself is signed). In some examples, all values ​​of one or more rows or columns of a matrix or vector, or all values ​​of a matrix or vector, may share a common exponent. In other examples, the BFP representation may be unsigned. In some examples, some, but not all, elements in a matrix or vector BFP representation may include numbers represented as integers, floating-point numbers, fixed-point numbers, notated numbers, or other data formats mixed with numbers represented using a sign, mantissa, and exponent. In some examples, some or all elements in a matrix or vector BFP representation may include complex elements with two or more parts, such as: a complex number (a+bi) with an imaginary part, where ); a fraction in polar coordinates (r,θ) that includes both numerator and denominator, or other multi-component elements.

[0039] Specific parameters for the BFP format can be selected for a particular implementation to balance precision and storage requirements. For example, instead of storing an exponent for each floating-point number, a group of numbers can share the same exponent. To share the exponent while maintaining a high level of accuracy, the numbers should be approximately the same size, as differences in size are expressed in the mantissa. If the size differences are too large, the mantissa will overflow for large values, or the mantissa may be zero (“underflow”) for small values. Depending on the specific application, a certain amount of overflow and / or underflow may be acceptable.

[0040] The size of the mantissa can be adjusted to suit a specific application. This may affect the precision of the represented number, but the potential benefits are realized through the reduced representation size. For example, a normal single-precision floating-point number has a size of four bytes, but for some implementations of the disclosed techniques, only two bytes are used to represent the sign and mantissa of each value. In some implementations, the sign and mantissa of each value can be represented with one byte or fewer.

[0041] In some examples of the disclosed technology, the representation expressed above is used to derive the original numbers from that representation, but only a single exponent is stored for a set of numbers, each represented by a signed mantissa. Each signed mantissa can be represented using two bytes or fewer, thus saving approximately twice the memory storage compared to four-byte floating-point numbers. Furthermore, the memory bandwidth requirements for loading and storing these values ​​are also approximately half that of normal floating-point numbers.

[0042] Neural network operations are used in many artificial intelligence operations. Typically, much of the processing performed when implementing neural networks involves matrix-matrix or matrix-vector multiplication or convolution operations. These operations are computationally and memory bandwidth intensive, where the matrix size can be, for example, 1000×1000 elements (e.g., 1000×1000 numbers, each including a sign, mantissa, and exponent) or larger, and many matrices are used. As discussed in this paper, BFP techniques can be applied to such operations to reduce memory bandwidth and computational requirements in a given system, regardless of whether the system is an FPGA, CPU, or other hardware platform. As used herein, the term "element" refers to a member of such a matrix or vector.

[0043] As used herein, the term "tensor" refers to an attribute that can be used to represent a neural network and includes one-dimensional vectors as well as multidimensional array matrices of two, three, four, or higher dimensions. As used in this disclosure, unless otherwise stated, tensors do not require any other mathematical properties.

[0044] As used herein, the term "normal precision floating-point" refers to a floating-point format that has a mantissa, exponent, and optional sign and is natively supported by the native or virtual CPU. Examples of normal precision floating-point formats include, but are not limited to, IEEE 754 standard formats such as 16-bit, 32-bit, and 64-bit, or other processor-supported formats such as Intel AVX, AVX2, IA32, x86_64, or 80-bit floating-point formats.

[0045] As used herein, the term "lossy mantissa" refers to the mantissa of a discrete set of values ​​that represents a higher-precision mantissa as a lower-precision mantissa value. For example, a four-digit number with a sign bit and a mantissa bit, and a three-bit mantissa (which can represent eight values, such as the set {0,1,2,3,4,5,6,7}) can be converted into a lossy mantissa of a discrete set of mantissa values, such as any of the following sets: {0,1,3,7}, {0,1,7}, or {0,7}, depending on the lossy mantissa scheme chosen. The underlying representation of a lossy mantissa may vary. For the first three sets, example sets of their corresponding binary representations are {00,01,10,11}; {00,10,11}; and {0,1}, respectively.

[0046] As used herein, the term "non-uniform mantissa" refers to a property of certain lossy mantissas where at least one of the values ​​represented is not uniformly distributed. For the example in the preceding paragraph, a 3-digit mantissa is uniform because each value is spaced one unit apart. Two of the three sets of lossy mantissas are non-uniformly spaced, distributed at different intervals: for the first set, intervals of 1, 2, and 4 units; for the second set, intervals of 1 or 6 units; and for the third set, intervals of 7 units. The term non-uniform mantissa includes not only lossy mantissas with non-uniform intervals but also low-precision lossy mantissas that are non-uniformly mapped from higher-precision mantissas. For example, the higher-precision set {0,1,2,3,4,5,6,7} can be non-uniformly mapped as follows: {0} => {0}; {1,2} => {1}; {3,4,5,6} => {3}; {7} => {7}. As another example, the same higher-precision set can be non-uniformly mapped in another way: {0} => {0}; {1,2,3,4,5,6} => {1}; {7} => {7}. To give yet another example, the same higher-precision set can be non-uniformly mapped in another way: {0} => {0}; {1,2,3,4,5,6,7} => {7}. The third method has tail values ​​with uniform intervals (7 units) but a non-uniform mapping.

[0047] A given number can be represented using different precision formats (e.g., different quantization precisions). For example, a number can be represented using a higher precision format (e.g., float32) and a lower precision format (e.g., float16). Reducing the precision of a number can include decreasing the number of bits used to represent the mantissa or exponent. Furthermore, reducing the precision of a number can include narrowing the range of values ​​that can be used to represent the exponent, such as when multiple numbers share a common exponent. Similarly, increasing the precision of a number can include increasing the number of bits used to represent the mantissa or exponent. Furthermore, increasing the precision of a number can include increasing the range of values ​​that can be used to represent the exponent, such as when a number is separated from a set of numbers sharing a common exponent. As used herein, converting a number from a higher precision format to a lower precision format can be referred to as down-quantizing or quantizing the number. Converting a number from a lower precision format to a higher precision format can be referred to as up-quantizing or dequantizing the number.

[0048] As used herein, the term "quantized precision floating-point" refers to a floating-point format in which two or more values ​​of a tensor have been modified to have a lower precision than when the value is represented in normal precision floating-point. Specifically, many examples of quantized precision floating-point representations include block floating-point formats, where two or more values ​​of the tensor are represented by reference to a common exponent. Quantized precision floating-point numbers can be generated by selecting a common exponent for two, more, or all elements of a tensor and offsetting the mantissa of individual elements to match the shared common exponent. In some examples, groups of elements within a tensor may share a common exponent on, for example, per row, per column, per block, or other basis.

[0049] In one example of the disclosed technology, a neural network accelerator is configured to perform training operations on neural network layers, including forward and backward propagation. The values ​​of one or more neural network layers can be expressed in a quantized format with a precision lower than that of normal precision floating-point formats. For example, block floating-point formats can be used to accelerate computations performed in training and inference operations using a neural network accelerator. The use of quantization formats can improve neural network processing, for example, by allowing for faster hardware, reduced memory overhead, simpler hardware design, reduced energy consumption, reduced integrated circuit area, cost savings, and other technological improvements. It is often desirable to perform operations to mitigate noise or other inaccuracies introduced by using lower precision quantization formats. Furthermore, the temporary storage of parts of neural network training, such as activation values, can be improved by compressing a portion of these values ​​(e.g., for input, hidden, or output layers of the neural network) either from normal precision floating-point or from a first block floating-point number to a lower precision number format with lossy or non-uniform mantissas. Activation values ​​can be retrieved and dequantized later for use during backpropagation, for example, during the training phase.

[0050] The input tensor of a given layer can be converted from normal-precision floating-point format to quantized-precision floating-point format. Tensor operations can be performed using the converted input tensor with lossy or non-uniform mantissas. The result of the tensor operation can be converted from block floating-point format to normal-precision floating-point format. Tensor operations can be performed during the forward or backward propagation modes of the neural network. For example, during backward propagation mode, the input tensor can be the output error term from a layer adjacent to (e.g., behind) a given layer or the weights of a given layer. As another example, during forward propagation mode, the input tensor can be the output term from a layer adjacent to (e.g., before) a given layer or the weights of a given layer. The converted result can be used to generate the output tensor of the neural network layer, where the output tensor is in normal-precision floating-point format. In this way, the neural network accelerator can potentially become smaller and more efficient compared to a similar accelerator that only uses normal-precision floating-point format. Smaller, more efficient accelerators may have higher computational performance and / or higher energy efficiency. Furthermore, this neural network accelerator can potentially achieve higher accuracy compared to accelerators that only use quantized precision floating-point format. Improving the accelerator's accuracy can reduce the convergence time for training and allow for more accurate classification of neural network inputs. Reducing the computational complexity of the model can potentially reduce the time spent extracting features during inference, the setup time during training, and / or the energy consumption during training and / or inference.

[0051] three, Example architecture for implementing active compression with non-uniform mantissa blocks of floating-point.

[0052] Figure 1 This is a block diagram 100 outlining an example quantization-enabled system 110 for use in active compression with non-uniform mantissa floating-point, which can be implemented in some examples of the disclosed technology. (See diagram 100 for example quantization-enabled system 110 for use in active compression with non-uniform mantissa floating-point.) Figure 1As shown, the quantization-enabled system 110 may include multiple hardware resources, including a general-purpose processor 120 and dedicated processors such as a graphics processing unit 122 and a neural network accelerator 180. The processors are coupled to a memory 125 and a storage device 129, which may include volatile or non-volatile memory devices. Processors 120 and 122 execute instructions stored in the memory or storage device to provide a neural network module 130. The neural network module 130 includes a software interface that allows the system to be programmed to implement various types of neural networks. For example, software functions may be provided that allow applications to define neural networks, including weights, biases, activation functions, node values, and interconnections between neural network layers. Furthermore, software functions may be used to define state elements for recurrent neural networks. The neural network module 130 may also provide utilities to allow training and retraining of the neural network implemented with that module. Values ​​representing the neural network module are stored in the memory or storage device and manipulated by instructions executed by one of the processors. Values ​​stored in the memory or storage device can be represented using normal-precision floating-point and / or quantized floating-point values ​​(including floating-point values ​​with lossy or non-uniform mantissas).

[0053] In some examples, proprietary or open-source libraries or frameworks are provided to programmers to create, train, and evaluate neural networks. Examples of such libraries include TensorFlow, Microsoft Cognitive Toolkit (CNTK), Caffe, Theano, and Keras. In other examples, programming tools such as integrated development environments (IDEs) support programmers and users in defining, compiling, and evaluating neural networks.

[0054] The neural network accelerator 180 can be implemented as a custom or application-specific integrated circuit (e.g., including a system-on-a-chip (SoC) integrated circuit), as a field-programmable gate array (FPGA) or other reconfigurable logic, or as a software processor virtual machine hosted by a physical general-purpose processor. The neural network accelerator 180 may include a tensor processing unit 182, a reconfigurable logic device 184, and / or one or more neural processing cores (such as a quantization accelerator 186). The quantization accelerator 186 can be configured using hardware, software, or a combination of hardware and software. As an example, the quantization accelerator 186 can be configured and / or executed using instructions executable on the tensor processing unit 182. As another example, the quantization accelerator 186 can be configured by programming the reconfigurable logic device 184. As yet another example, the quantization accelerator 186 can be configured using hardwired logic gates of the neural network accelerator 180.

[0055] The quantization accelerator 186 can be programmed to perform operations on subgraphs, individual layers, or multiple layers of a neural network. For example, the quantization accelerator 186 can be programmed to perform operations on all or part of the layers of an NN. The quantization accelerator 186 can access local memory used to store weights, biases, input values, output values, forgotten values, state values, etc. The quantization accelerator 186 can have many inputs, where each input can be weighted by different weight values. For example, the quantization accelerator 186 can produce an input tensor and a dot product of the programmed input weights for the quantization accelerator 186. In some examples, the dot product can be adjusted by bias values ​​before being used as input to an activation function. For example, the output of the quantization accelerator 186 can be stored in local memory, where the output value can be accessed and sent to different NN processor cores and / or neural network modules 130 or memory 125. Intermediate values ​​in quantization accelerators can often be stored in smaller, more local memory, while values ​​that may not be needed until later in the training process can be stored in "mass memory," i.e., larger, less local memory (or storage devices, such as SSDs or hard disk drives). For example, during training forward propagation, once the activation values ​​for the next layer in the neural network have been computed, those values ​​may not be accessible until propagation through all layers is complete. Such activation values ​​can be stored in such mass memory.

[0056] The neural network accelerator 180 may include a plurality of 110 quantization accelerators 186 interconnected with each other via interconnects (not shown). The interconnects may carry data and control signals between individual quantization accelerators 186, memory interfaces (not shown), and input / output (I / O) interfaces (not shown). The interconnects may use electrical, optical, magnetic, or other suitable communication technologies to transmit and receive signals, and may provide communication connections arranged according to multiple different topologies depending on a specific desired configuration. For example, the interconnects may have crossbar switches, buses, point-to-point buses, or other suitable topologies. In some examples, any one of the plurality of quantization accelerators 186 may be connected to any other core, while in other examples, some cores may be connected to only a subset of other cores. For example, each core may be connected to only the nearest 4, 8, or 10 neighboring cores. The interconnects may be used to transmit input / output data to and from the quantization accelerators 186, as well as to transmit control signals and other information signals to and from the quantization accelerators 186. For example, each quantization accelerator 186 may receive and transmit semaphores indicating the execution status of an operation currently performed by each of the respective quantization accelerators 186. Furthermore, matrix and vector values ​​can be shared between quantization accelerators 186 via interconnects. In some examples, the interconnects are implemented as lines connecting the quantization accelerators 186 and the memory system, while in other examples, the core interconnects may include circuitry, switching and / or routing components (including active signal drivers and repeaters), or other suitable circuitry for multiplexing data signals on one or more interconnect lines. In some examples of the disclosed technology, the signals within and transmitted to / from the neural network accelerator 180 are not limited to full-swing electrical digital signals, but the neural network accelerator 180 may be configured to include differential signals, pulse signals, or other suitable signals for transmitting data and control signals.

[0057] In some examples, the quantization-enabled system 110 may include an optional quantization simulator that emulates the functionality of the neural network accelerator 180. The neural network accelerator 180 provides the functionality to convert data represented in full-precision floating-point format in the neural network module 130 into quantized format values. The neural network accelerator 180 can also perform operations using the quantized format values. This functionality will be discussed in more detail below.

[0058] The neural network module 130 can be used to specify, train, and evaluate neural network models using a toolflow that includes a hardware-independent modeling framework 131 (also referred to as a native framework or machine learning execution engine), a neural network compiler 132, and a neural network runtime environment 133. The memory includes computer-executable instructions for the toolflow, which includes the modeling framework 131, the neural network compiler 132, and the neural network runtime environment 133. The toolflow can be used to generate neural network data representing all or part of the neural network model, such as the following regarding… Figure 2 The neural network model discussed. It should be noted that although the toolflow is described as having three separate tools (131, 132, and 133), in various examples, the toolflow can have fewer or more tools. For example, the functionality of different tools (131, 132, and 133) can be combined into a single modeling and execution environment. In other examples of deploying neural network accelerators, such a modeling framework may not be included.

[0059] Neural network data can be stored in memory 125, which may include: local memory 126, typically implemented as static read-only memory (SRAM), embedded dynamic random access memory (eDRAM), latches or flip-flops implemented in a register file, block RAM, or other suitable structures; and mass storage 127, typically implemented in a memory structure that supports larger but often slower access than local memory 126. For example, mass storage may be off-chip DRAM, network-accessible RAM, SSD drives, hard disk drives, or network-accessible storage devices. Depending on the specific memory technology available, other memory structures (including those described for local memory) may be used to implement mass storage. Neural network data can be represented in one or more formats. For example, neural network data corresponding to a given neural network model may have different formats associated with each corresponding tool in the toolstream. Typically, neural network data may include descriptions of nodes, edges, groups, weights, biases, activation functions, and / or tensor values. As a specific example, neural network data may include source code, executable code, metadata, configuration data, data structures, and / or files representing the neural network model.

[0060] Modeling framework 131 can be used to define and use neural network models. As an example, modeling framework 131 may include predefined APIs and / or programming primitives that can be used to specify one or more aspects of a neural network model. Predefined APIs may include lower-level APIs (e.g., activation functions, cost or error functions, nodes, edges, and tensors) and higher-level APIs (e.g., layers, convolutional neural networks, recurrent neural networks, linear classifiers, etc.). "Source code" can be used as input to modeling framework 131 to define the topology of the graph for a given neural network model. Specifically, the APIs of modeling framework 131 can be instantiated and interconnected within the source code to specify complex neural network models. Data scientists can create different neural network models by using different APIs, different numbers of APIs, and APIs interconnected in different ways.

[0061] In addition to the source code, memory 125 can also store training data. The training data includes a set of input data to be applied to the neural network model and the expected output from the neural network model for each corresponding dataset of input data. Modeling framework 131 can be used to train the neural network model with the training data. The output of the training is the weights and biases associated with each node of the neural network model. After the neural network model is trained, modeling framework 131 can be used to classify new data applied to the trained neural network model. Specifically, the trained neural network model uses the weights and biases obtained from training to perform classification and recognition tasks on data not used to train the neural network model. Modeling framework 131 can use CPU 120 and dedicated processors (e.g., GPU 122 and / or neural network accelerator 180) to execute the neural network model with improved performance compared to using CPU 120 alone. In some examples, the performance can potentially achieve real-time performance for certain classification tasks.

[0062] Compiler 132 analyzes the source code and data provided to the neural network model (e.g., examples used to train the model) and transforms the model into a format that can be accelerated on the neural network accelerator 180, as will be described in more detail below. Specifically, compiler 132 transforms the source code into executable code, metadata, configuration data, and / or data structures for representing the neural network model and memory as neural network data. In some examples, compiler 132 may partition the neural network model into parts that use CPU 120 and / or GPU 122 (e.g., neural network 200) and other parts that can be executed on the neural network accelerator 180 (e.g., subgraphs, individual layers, or multiple layers of the neural network). Compiler 132 may generate executable code (e.g., runtime modules) for executing the NN assigned to CPU 120 and for communicating with the subgraphs, individual layers, or multiple layers of the neural network assigned to accelerator 180. Compiler 132 may generate configuration data for accelerator 180, which is used to configure accelerator resources to evaluate the subgraphs assigned to the optional accelerator 180. Compiler 132 can create data structures for storing values ​​generated by the neural network model during execution and / or training, as well as for communication between CPU 120 and accelerator 180. Compiler 132 can generate metadata that can be used to identify subgraphs, edge groups, training data, and various other information about the neural network model during runtime. For example, the metadata may include information for interfacing between different subgraphs or other parts of the neural network model.

[0063] The runtime environment 133 provides an executable environment or interpreter that can be used to train a neural network model during training mode and to evaluate the neural network model in training, inference, or classification modes. During inference mode, input data can be applied to the neural network model, and the input data can be classified based on the training of the neural network model. The input data can be archived data or real-time data.

[0064] The runtime environment 133 may include deployment tools that, during deployment mode, can be used to deploy or install all or part of the neural network onto the neural network accelerator 180. The runtime environment 133 may also include a scheduler that manages the execution of different runtime modules and communication between the runtime modules and the neural network accelerator 180. Therefore, the runtime environment 133 can be used to control the data flow between the nodes modeled on the neural network module 130 and the neural network accelerator 180.

[0065] In one example, neural network accelerator 180 receives and returns a normal precision value of 150 from neural network module 130. For example... Figure 1As illustrated, the quantization accelerator 186 can perform most of its operations using quantized floating-point values, and the interface between the quantization accelerator 186 and the neural network module 130 can use full-precision values ​​to transfer information between modules. Normal-precision values ​​can be represented in 16-bit, 32-bit, 64-bit, or other suitable floating-point formats. For example, values ​​representing a portion of the neural network, including edges, weights, activation values, or other suitable parameters for quantization, can be received. Normal-precision values ​​150 are provided to a normal-precision floating-point to quantization floating-point converter 152, which converts the normal-precision values ​​to quantized values. Quantization floating-point operations 154 can then be performed on the quantized values. The quantized values ​​can then be converted back to normal-precision floating-point format using a quantization floating-point to normal-precision floating-point converter 156 that produces the normal-precision floating-point values. As a specific example, the quantization accelerator 186 can be used to accelerate a given layer of a neural network, and can use quantized floating-point operations to perform vector-vector, matrix-vector, matrix-matrix, and convolution operations, and can use normal-precision floating-point operations to perform less computationally intensive operations (such as adding bias values ​​or calculating activation functions). Other examples of conversions that can be performed are converting quantized and normal-precision floating-point values ​​to normal floating-point or block floating-point values ​​with lossy or non-uniform mantissas. For example, a conversion to a block floating-point format with non-uniform mantissas to compress the activation value can be performed, and a conversion from a block floating-point format with non-uniform mantissas to a normal-precision floating-point or block floating-point number can be performed, for example, when retrieving the compressed activation value in a backpropagation for later use.

[0066] The conversions between normal floating-point and quantized floating-point values ​​performed by converters 152 and 156 are typically performed on a set of numbers represented as vectors or multidimensional matrices. In some examples, additional normal-precision operations 158 (including operations that may be expected in a particular neural network implementation) can be performed based on the normal-precision format, including adding bias to one or more nodes of the neural network, applying a hyperbolic tangent function or other such sigmoid function, or a rectification function (e.g., a ReLU operation) to the normal-precision value converted back from the quantized floating-point format.

[0067] In some examples, quantized values ​​are used only and stored in the logic gates and internal memory of neural network accelerator 180, and memory 125 and storage device 129 store only normal floating-point values. For example, neural network accelerator 180 can quantize the inputs, weights, and activations of a neural network module received from neural network module 130, and can dequantize the results of operations performed on neural network accelerator 180 before passing the values ​​back to neural network module 130. Values ​​can be passed between neural network module 130 and neural network accelerator 180 using memory 125, storage device 129, or an input / output interface (not shown). In other examples, the simulator provides a complete simulation of the quantization, including storing only a copy of the shared exponent and operating with a low mantissa width. Some results may differ on versions that perform the basic operations with normal floating-point numbers. For example, some examples can examine underflow or overflow conditions for finite quantization bit widths (e.g., 3-bit, 4-bit, or 5-bit wide mantissas).

[0068] The majority of the computational cost of a DNN is vector-vector, matrix-vector, and matrix-matrix multiplications and / or convolutions. These operations are quadratic in size with respect to the input, while operations such as bias addition and activation functions are linear in size with respect to the input. Therefore, in some examples, quantization is only applied to matrix-vector multiplication operations, which are implemented on the neural network accelerator 180. In these examples, all other operations are performed in normal precision formats, such as float16. Therefore, from the user's or programmer's perspective, the quantized-enabled system 110 accepts and outputs normal precision float16 values ​​to / from the neural network module 130 and outputs float16 formatted values. All conversions to and from block floating-point formats can be hidden from the programmer or user. In some examples, the programmer or user can specify certain parameters for the quantization operation. In other examples, the quantization operation can utilize the block floating-point format to reduce computational complexity, as described below. Figure 3 The subject of discussion.

[0069] Neural network accelerator 180 is used to accelerate the evaluation and / or training of neural network graphs or subgraphs, typically with increased speed and reduced latency, which is not achievable when evaluating subgraphs using only CPU 120 and / or GPU 122. In the illustrated example, the accelerator includes a tensor processing unit (TPU) 182, a reconfigurable logic device 184 (e.g., contained in one or more FPGAs or programmable circuit structures), and / or a quantization accelerator 186, but any suitable hardware accelerator for modeling neural networks can be used. Accelerator 180 may include configuration logic that provides a soft CPU. The soft CPU supervises the operation of the graph or subgraph accelerated on accelerator 180 and can manage communication with neural network module 130. The soft CPU can also be used to configure logic and control the loading and storage of data from RAM on the accelerator (e.g., in block RAM within the FPGA).

[0070] In some examples, the parameters of the neural network accelerator 180 can be programmable. The neural network accelerator 180 can be used to prototype all or part of a neural network model for training, inference, or classification. For example, quantization parameters can be selected based on accuracy or performance results obtained by prototyping the network within the neural network accelerator 180. After selecting the desired set of quantization parameters, the quantized model can be programmed into the accelerator 180 to perform further operations.

[0071] The compiler 132 and runtime environment 133 provide a fast interface between the neural network module 130 and the neural network accelerator 180. In practice, the user of the neural network model may not be aware that a portion of the model is being accelerated on the provided accelerator. For example, node values ​​are typically propagated in the model by writing tensor values ​​to a data structure that includes identifiers. The runtime environment 133 associates subgraph identifiers with the accelerator and provides the accelerator with logic for translating messages, transparently writing the values ​​of weights, biases, and / or tensors to the neural network accelerator 180 without program intervention. Similarly, the values ​​output by the neural network accelerator 180 can be transparently sent back to the neural network module 130 along with a message including the identifiers of the receiving nodes at the server and a payload, such as values ​​like weights, biases, and / or tensors that are sent back to the entire neural network model.

[0072] Figure 2The figure illustrates a simplified topology of a deep neural network (DNN) 200, which can be used to perform neural network applications, such as enhanced image processing, using the disclosed BFP implementation. One or more processing layers can be implemented using the disclosed techniques for quantization and BFP matrix / vector operations, including the use of one or more of the multiple neural network quantization accelerators 186 in the quantization-enabled system 110 described above. It should be noted that the applications of the neural network implementation disclosed herein are not limited to DNNs, but can also be used with other types of neural networks such as convolutional neural networks (CNNs), including implementations with long short-term memory (LSTM) or gated recurrent units (GRUs), or other suitable artificial neural networks that can be adapted to use the BFP methods and apparatus disclosed herein.

[0073] The DNN 200 can operate in at least two different modes. Initially, the DNN 200 can be trained in training mode and then used as a classifier in inference mode. During training mode, a training dataset can be applied to the input of the DNN 200, and various parameters of the DNN 200 can be tuned so that the DNN 200 can be used as a classifier when training is complete. Training includes: performing forward propagation on the training input data, calculating the loss (e.g., determining the difference between the DNN's output and its expected output), and performing backpropagation through the DNN to tune the parameters of the DNN 200 (e.g., weights and biases). When the architecture of the DNN 200 is suitable for classifying the training data, the parameters of the DNN 200 will converge and training can be completed. After training, the DNN 200 can be used in inference mode. Specifically, training or non-training data can be applied to the input of the DNN 200 and forward propagated through the DNN 200 so that the input data can be classified by the DNN 200.

[0074] like Figure 2 As shown, a first set of nodes 210 (including nodes 215 and 216) forms the input layer. Each node in set 210 is connected to each node in a first hidden layer formed by a second set of nodes 220 (including nodes 225 and 226). The second hidden layer is formed by a third set of nodes 230, including node 235. The output layer is formed by a fourth set of nodes 240 (including node 245). In this example, the nodes of a given layer are fully interconnected with the nodes of its one or more adjacent layers. In other words, a layer may include nodes that share a common input with other nodes in that layer and / or provide output to a common destination for other nodes in that layer. In other examples, a layer may include nodes that share a common input with other nodes in that layer and / or provide output to a subset of the common destination for other nodes in that layer.

[0075] During forward propagation, each node produces an output by applying weights to each input generated from the previous node and collecting the weights to produce an output value. In some examples, each individual node may apply an activation function (σ) and / or a bias (b). Typically, a properly programmed processor or FPGA can be configured to implement the nodes in the depicted neural network 200. In some example neural networks, the output function f(n) of the hidden combined node n can produce an output mathematically expressed as follows:

[0076]

[0077] Where w i It is applied (multiplied) by the input edge x i The weights are denoted by σ, where b is the bias value for node n, σ is the activation function for node n, and E is the number of input edges for node n. In some examples, the activation function produces continuous values ​​between 0 and 1 (represented as floating-point numbers). In other examples, the activation function produces binary 1 or 0 values, depending on whether the summation is above or below a critical point.

[0078] A given neural network can include thousands of individual nodes, and therefore performing all computations on each node in normal-precision floating-point can be computationally expensive. Implementations of more computationally expensive solutions can involve larger and more energy-intensive hardware than implementations of less computationally expensive solutions. However, performing operations using quantized floating-point can potentially reduce the computational complexity of the neural network. A simple implementation using only quantized floating-point can significantly reduce computational complexity, but due to errors introduced by quantization, this implementation may struggle to converge during training and / or correct classification of the input data. However, the quantized floating-point implementation disclosed in this paper can potentially improve some computational accuracy while also providing the benefit of reducing the complexity associated with quantized floating-point.

[0079] A DNN 200 can include nodes that perform operations in quantized floating-point. As a concrete example, the output function f(n) of the hidden combined node n can produce an output that can be mathematically expressed as follows:

[0080]

[0081] Where w i It is applied (multiplied) by the input edge x i The weights, Q(w) i Q(x) is the quantized floating-point value of the weight. i ) originates from the input edge x i The input quantized floating-point value, Q -1() is the dequantized representation of the dot product of vectors w and x, where b is the bias value for node n, σ is the activation function of node n, and E is the number of input edges of node n. By using quantized floating-point values ​​to perform the dot product, computational complexity can be potentially reduced (compared to using only normal-precision floating-point values), and by using other operations on the output function of normal-precision floating-point values, the accuracy of the output function can be potentially improved (compared to using only quantized floating-point values).

[0082] Neural networks can be trained and retrained by adjusting the constituent values ​​of the output function f(n). For example, by adjusting the weights w of the nodes. i Or the bias value b, which adjusts the behavior of the neural network by corresponding changes in the network's output tensor values. For example, during backpropagation, a cost function C(w,b) can be used to find suitable weights and biases for the network, where the cost function can be mathematically described as:

[0083]

[0084] Where w and b represent all weights and biases, n is the number of training inputs, and a is the vector of output values ​​from the network for the input vector of the training input x. By adjusting the network weights and biases, the cost function C can be driven to a target value (e.g., zero(0)) using various search techniques (e.g., stochastic gradient descent). When the cost function C is driven to the target value, the neural network is said to have converged. Similar to the output function f(n), the cost function can be implemented using quantized precision computer operations. For example, quantized floating-point values ​​and operations can be used to perform vector operations, and normal-precision floating-point values ​​can be used to perform non-vector operations.

[0085] Examples of suitable applications for this type of neural network BFP implementation include, but are not limited to: performing image recognition, performing speech recognition, image classification, translating speech into text and / or other languages, facial or other biometrics, natural language processing, automatic language translation, query processing in search engines, automatic content selection, analyzing emails and other electronic documents, relationship management, biomedical informatics, identifying candidate biomolecules, providing suggestions, or other classification and artificial intelligence tasks.

[0086] Network accelerators (such as Figure 1The network accelerator 180 in the diagram can be used to accelerate the computation of DNN 200. As an example, DNN 200 can be divided into different subgraphs or network layers, which can be accelerated individually. Specifically, each of layers 210, 220, 230, and 240 can be a subgraph or layer accelerated using the same or different accelerators. Quantized floating-point operations can be used to perform computationally expensive computations within a layer, while normal-precision floating-point operations can be used to perform computationally inexpensive computations. Values ​​can be passed from one layer to another using normal-precision floating-point operations. By accelerating a set of computations for all nodes within a layer, some computations can be reused, and the amount of computation performed by that layer can be reduced compared to accelerating individual nodes.

[0087] In some examples, computation can be accelerated using a set of parallel multiply-accumulate (MAC) units in each convolutional layer. Furthermore, parallel multiplier units can be used in fully connected and dense matrix multiplication stages. A set of parallel classifiers can also be used. This parallelization approach has the potential to further accelerate computation at the cost of increased control complexity.

[0088] Those skilled in the art who benefit from this disclosure will readily understand that the application of the neural network implementation can be used to utilize different aspects of the neural network, either individually or in various combinations and sub-combinations of each other. For example, the disclosed implementation can be used to train a neural network via gradient descent and / or backpropagation operations for the neural network. Furthermore, the disclosed implementation can be used for evaluating neural networks.

[0089] Four, Example quantization block floating-point format

[0090] Figure 3 Figure 300 illustrates an example of converting normal floating-point format to quantized block floating-point format, as may be used in some examples of the disclosed techniques. For example, the input tensor of a neural network represented as a normal floating-point number (e.g., in 32-bit or 16-bit floating-point format) can be converted to the block floating-point format illustrated.

[0091] As shown, many normal floating-point number 310 are represented such that each number, such as number 315 or number 316, includes a sign, exponent, and mantissa. For example, in IEEE 754 half-precision floating-point format, the sign is represented using 1 bit, the exponent using 5 bits, and the mantissa using 10 bits. When the floating-point number 310 in the neural network model is converted to a quantized precision, block floating-point number set, there exists a single exponent value shared by all numbers in the illustrated set. Thus, as shown, the block floating-point number set 320 is represented by a single exponent value 330, and each number in this set includes a sign and a mantissa. However, since the illustrated set of numbers has different exponent values ​​in floating-point format, the mantissa of each number can be shifted so that the same or similar numbers are represented in quantized format (e.g., shifted mantissas 345 and 346).

[0092] In addition, such as Figure 3 As shown, using block floating-point format can reduce the computational resources required for certain common operations. In the illustrated example, the dot product of two floating-point vectors is shown in both formal floating-point format (350) and block floating-point format (360). For the numbers represented in normal precision floating-point format operation 350, floating-point addition is required to perform the dot product operation. In the dot product of floating-point vectors, summation is performed in floating-point, where shifting may be necessary to align the values ​​with different exponents. On the other hand, for the block floating-point dot product operation 360, integer arithmetic can be used to compute the product to combine the mantissa elements shown. In other words, since the exponent part can be taken into account in the block floating-point representation, mantissa multiplication and addition can be performed entirely in fixed-point or integer representation. As a result, a large dynamic range for a set of numbers can be maintained using a shared exponent, while reducing computational costs by using more integer arithmetic instead of floating-point arithmetic. In some examples, operations performed by the quantized-enabled system 110 can be optimized to take advantage of the block floating-point format.

[0093] In some examples, the shared exponent of 330 is chosen as the largest exponent among the original normal precision numbers in the neural network model. In other examples, the shared exponent can be chosen in different ways, such as by selecting an exponent that is the mean or median of the normal floating-point exponent, or by selecting an exponent that maximizes the dynamic range of the value stored in the mantissa when their numbers are converted to a quantized number format. It is important to note that if the shared exponent and the original floating-point exponent of the value are different, some bits of the quantized mantissa may be lost. This occurs because the mantissa is shifted to correspond to the new shared exponent.

[0094] There are several possible choices regarding which values ​​in a block floating-point tensor will share the exponent. The simplest choice is to have the entire matrix or vector share the exponent. However, sharing the exponent at a finer granularity can reduce errors because it increases the likelihood that BFP numbers will use a shared exponent that is closer to their original normal floating-point format exponent. Therefore, the precision loss due to discarding mantissa bits (when the mantissa is shifted to correspond to the shared exponent) can be reduced.

[0095] For example, consider multiplying a row vector x by a matrix W: y = xW. If each column of W shares an exponent, then each dot product xW j (where W) j (This refers to the j-th column of W) and involves only a shared index for x and for W. j A shared index.

[0096] Figure 4 and Figure 5 The diagram illustrates an alternative block floating-point format that can be used for neural network computations. In the context of neural networks, the core operation is performing dot products. For example, the dot product is the core computation of matrix multiplication and convolution. Matrix multiplication involves the dot product of the rows / columns of a matrix with the input vector. Convolution involves the dot product of a filter with the input window. In the context of quantized floating-point, choosing a set of values ​​for the shared exponent can impact the complexity of the computer logic used to compute the dot product. The values ​​of the shared exponent can be referred to as the values ​​within the bounding box. The shape of the bounding box can potentially affect quantization error and computational cost. While clustering similar magnitude values ​​to create bounding boxes can reduce quantization error, tracking scaling factors used for arbitrary bounding box shapes can be expensive. Instead, matrices and filters can be divided into bounding boxes that may be more efficient for the operations performed by the neural network. Specifically, appropriate selection of bounding boxes can reduce the complexity of the computer circuitry used to implement neural network operations. Figure 4 The diagram illustrates a block floating-point format that may be well-suited for matrices, and Figure 5 The figure illustrates a block floating-point format that is well-suited for convolutional filters.

[0097] Figure 4 Figure 400 illustrates four alternative block floating-point formats that can be used in some examples of the disclosed technology. As shown, the first format 410 represents an entire value array 420 sharing a single exponent 425. In other words, the entire value array 420 is encapsulated within a single bounding box.

[0098] In the second format 430, a common exponent is shared on a column-by-column basis. In other words, the columns of the matrix are the bounding boxes of the values. Therefore, in this particular example, the block floating-point values ​​stored in the even-numbered columns 431 of the matrix each share a first single exponent 432. The block floating-point values ​​stored in the odd-numbered columns 435 each share a second single exponent 437. In other examples, each column of the array can be associated with a different shared exponent. For the 11-column block of the alternative format, there can be 11 corresponding shared exponents, one per column. In other examples, each row of the array can be associated with a different shared exponent, or odd and even rows can be associated with a shared common exponent.

[0099] The third format 450 is shown, where groups of elements in an array share a common exponent. For example, if a 15×15 value matrix shares an exponent according to the third format 450, then the first set of 5×5 element groups 455 and 456 share a single shared exponent 458. Similarly, the second 5×5 element groups in arrays 460 and 461 can each share a second single exponent 468. In other examples, each block can be associated with its corresponding shared exponent. In example format 450, a 15×15 matrix may have nine shared exponents.

[0100] The fourth format 470 is shown, where two shared indices are shared on a tiling plus per column basis. Therefore, the first set of numbers, including 480, 481, and 485, all share a single common index 488. Similarly, the second set of numbers, including sets 490 and 491, each shares a different second single index 495. In an alternative example, each group shown could have its own shared index.

[0101] In some examples, the computational cost of matrix-vector multiplication can be further reduced by decreasing the mantissa width. A wide range of values ​​with a shared common exponent can be expressed using only a few mantissa bits. For example, in a representation with a 4-bit mantissa and a 5-bit exponent, the value can be expressed in [2]. -14 0.0012,2 15 1.1112] or approximately [2] -17 ,2 16 The range is [00012, 11112]. In contrast, a 4-digit fixed-point number can only be represented in the range [00012, 11112] or approximately [2]. 0 ,2 4 Values ​​within the range.

[0102] Figure 5Figure 500 illustrates three alternative block floating-point formats that can be used in some examples of the disclosed techniques. These formats may be useful for two-dimensional convolutions, but they can also be generalized to higher-dimensional convolutions. As shown, the first format 510 represents an entire convolutional filter 512 that shares the value of a single exponent 514. Different convolutional filters 516 may share a single exponent 518. Thus, format 510 illustrates that the entire convolutional filter can be a bounding box of values.

[0103] In the second format 520, each spatial pixel can be a bounding box, thus sharing a common index on a per-spatial-pixel basis along the channel dimension. As shown, spatial pixel values ​​522 share a single index 524 and spatial pixel values ​​526 share a single index 528. For example, for a channel with dimensions [x,y,c...] i The input is [], where each spatial dimension x and y can be defined with c i The bounding box of the value. Similarly, for dimension [f] x ,f y ,c i ] of c o Convolutional filters, for each c o Each pixel of the filter (f x ,f y ) can be having c i The value is a separate bounding box. The bounding box size for this method is c. i .

[0104] In the third format 530, each spatial pixel can be subdivided along the channel dimension such that the bounding box includes the subdivision of the spatial pixels. As shown, subdivided spatial pixel values ​​532 share a single exponent 534 and subdivided spatial pixel values ​​536 share a single exponent 538. For small c i The cost of handling scaling factors can be significant. For example, the input image at the first layer of a deep convolutional neural network might have c i =3, corresponding to 3 color channels. Tracking the scaling factor used for each triple can be expensive. In this case, the convolution can be reshaped into matrix-matrix multiplication to increase the bounding box and reduce the overhead of tracking the bounding box. For example, each convolutional filter can be flattened to create a matrix with c o column and f x ,*f y ,*c i A matrix W of rows. An input matrix X can be created where each row is f. x ,*f y ,*c iThe vector corresponds to the input window swept by the convolution filter. The result Y = XW is a matrix that can be reshaped to match the output of the convolution operation. This is achieved by re-equating the convolution into matrix multiplication, as referenced above. Figure 4 The bounding box strategies discussed can be applied.

[0105] five, Example methods for training neural networks

[0106] Figure 6 This is a flowchart depicting a method 600 for training a neural network using a quantized model, as may be implemented in some examples of the disclosed technology. For example, training the neural network may include iterative processing with a training dataset, where method 600 is used to update the parameters of the neural network during a given iteration of the training data. As an example, method 600 may be provided by a quantization-enabled system (such as...) Figure 1 The system 110 with enabled quantization is used to execute this.

[0107] At process block 610, parameters of the neural network, such as weights and biases, can be initialized. As an example, weights and biases can be initialized to randomized, normal-precision floating-point values. As another example, weights and biases can be initialized to normal-precision floating-point values ​​computed from an earlier training set. The initial parameters can be stored in the memory or storage device of a quantized system. In one example, the parameters can be stored as quantized floating-point values, which reduces the amount of storage used to store the initial parameters.

[0108] At process block 620, the input values ​​of a given layer of the neural network can be forward-propagated. The input values ​​of a given layer of the neural network can be the outputs of another layer of the neural network. These values ​​can be passed between layers from the output of one layer to the input of the next layer using normal-precision floating-point numbers. The output function of the i-th layer can include terms mathematically described as follows:

[0109] y i =Q -1 (f(Q(y i-1 ),Q(W i Equation 5)

[0110] Where y i-1 W is the output from a layer that provides input to the i-th layer. i Here, f() is the weight tensor used for the i-th layer, f() is the forward function for that layer, Q() is the quantization function, and Q... -1() is the dequantization function. The output function of this layer can be a dequantized representation of f(), or alternatively, the output function can include additional terms such as activation functions or biases, which are performed using normal precision floating-point (after dequantization) or quantized floating-point (before dequantization). Typically, the inputs, outputs, and parameters of each layer are tensors. Typically, the inputs, outputs, and parameters of each layer will be vectors or matrices. The quantization function Q() converts normal precision floating-point values ​​to quantized floating-point values. The quantization function can be chosen to take into account the type of input data and the type of operation performed by the i-th layer. For example, when y i and W i It is a two-dimensional matrix and the output function includes a y-value. i-1 and W i When the terms of the cross product are used for y i-1 The quantization function can be used to include the bounding box y i-1 The bounding box of a row or a portion of a row, and used for W i Quantization functions can be used including W i The bounding box of a column or a portion of a column. When selecting the bounding box to follow the operator's flow, computation can be more efficient, resulting in smaller, faster, and more energy-efficient hardware. Dequantization function Q. -1 () converts a quantized floating-point value to a normal precision floating-point value.

[0111] At process block 630, a portion of the neural network, such as a layer that has just been forward-propagated to the next layer of the neural network, can be compressed and stored in memory. For example, activation values ​​computed as part of the forward propagation, as discussed in process block 620 above, can be compressed and stored in memory. This compression can be mathematically expressed as:

[0112] y ci =C(Q2(y) i (Equation 6a)

[0113] or

[0114] y ci =C(Q2(f(Q(y)) i-1 ),Q(W i Equation 6b)

[0115] Where y i The value generated for the layer via forward propagation at process block 620, C() is an optional additional compression function (which may include multiple compression operations), Q2() is the quantization function for a second floating-point format with lossy or non-uniform mantissa, and y ciThese are the compressed values ​​to be stored in memory. In some examples, the Q2() quantization function converts the value from a normal precision format to a smaller quantization format than that used in the quantization layer (as in Equation 6a). In other examples, the Q2() quantization function directly converts the value from the first block floating-point format used in the quantization layer to a second block floating-point format (as in Equation 6b). The compressed activation values ​​are expressed in a second block floating-point format that can be different from the first block floating-point format used to perform the forward propagation computation, and in at least one of the following ways: having a different mantissa format, having a different exponent format, or having a different exponent sharing scheme. For example, if forward propagation is performed using activation values ​​expressed in 8-bit format, these values ​​can be converted to a 4-bit format by converting the mantissa to a lossy or non-uniform mantissa. As another example, activation value exponents including a shared exponent in BBFP format can be converted from a 7-bit format to a 5-bit format. As another example, for a four-bit number with one sign bit and three mantissa bits, the 3-bit mantissa (which can represent 8 values, such as the set {0,1,2,3,4,5,6,7}) can be converted into a lossy mantissa with a discrete set of mantissa values, such as any of the following sets: {0,1,3,7}, {0,1,7}, or {0,7}, depending on the lossy mantissa scheme chosen. The underlying representation of the lossy mantissa can vary. For the three sets above, example sets of corresponding binary representations are {00,01,10,11}; {00,10,11}; and {0,1}. In some examples, multi-valued (non-binary) representations can be used as the underlying representation of the lossy mantissa. For example, multi-level EEPROM or flash memory devices can use multi-level storage cells to store the lossy mantissa.

[0116] Values ​​can be converted between two quantization formats using any suitable technique. For example, lookup tables, logic gates, arithmetic units, or other circuitry can be used to convert values ​​from normal-precision floating-point or block floating-point formats to floating-point formats with lossy or non-uniform mantissas. In some examples, the floating-point format with lossy or non-uniform mantissas is a block floating-point format, where the exponent is shared between two or more values. In other examples, a normal floating-point format is used, but with lossy or non-uniform mantissas.

[0117] In some examples, the compressed block floating-point format shares a common exponent in a manner different from the format used during forward propagation. Aspects of the shared format that can be modified include how the exponent is shared on a per-row, per-column, or per-block basis. In some examples, additional compression may be applied to the compressed block floating-point format before it is stored in memory. Examples of suitable techniques for further compressing the activation values ​​of the compressed quantized format include entropy compression (e.g., Huffman coding), zero compression, run-length compression, compressed sparse row compression, or compressed sparse column compression.

[0118] At process block 640, the loss of the neural network can be calculated. For example, the output y of the neural network can be compared with the expected output of the neural network. The difference between the output and the expected output can be used as input to the cost function that is used to update the parameters of the neural network.

[0119] At process block 650, the activation values ​​stored in memory are decompressed for backpropagation, and specifically used to calculate the output error term used in backpropagation for a particular layer. This method iterates at each layer, decompressing the activation values ​​of each layer, performing backpropagation on that layer, and then decompressing the activation values ​​of the previous layer. This decompression can be mathematically expressed as:

[0120] y i =Q2 -1 (C -1 (y ci (Equation 7a)

[0121] or

[0122] y i =C -1 (y ci (Equation 7b)

[0123] Where y ci It is a compressed value retrieved from memory, C -1 () is the decompression function (which can include multiple compression operations), and it is the inverse function of the compression function C(). Q2 -1 () is a function that translates quantized values ​​from a second-block floating-point format with lossy or non-uniform mantissas to a first-block floating-point format, and y i The values ​​are generated for the layers via forward propagation at process block 620. For example, after the forward propagation and loss calculation discussed above at process block 640 are completed for each layer and neural network as discussed above with respect to process blocks 620 and 630, the values ​​are typically backpropagated through the neural network, starting from the output layer. Therefore, depending on how the compressed quantization format differs from the format used for backpropagation, appropriate transformations of activation value mantissas, exponents, and / or exponent-sharing schemes can be performed. Furthermore, if additional compression such as entropy compression, zero compression, run-length encoding, compressed sparse row compression, or compressed sparse column compression is applied before storage in memory, these operations can be reversed before performing backpropagation on the layers at process block 660.

[0124] Different numbers of techniques can be used to dequantize lossy or non-uniform mantissas. For example, when the non-uniform mantissa set is {0,1,3,7} (e.g., stored in a two-bit mantissa format), those identical values ​​can be recovered as dequantized values ​​of the first normal precision (e.g., 8, 16, or 32 bits) or block floating-point (e.g., 3, 4, 5, or 6 bits) mantissa. In some examples, the dequantized mantissa can be an approximation. For example, for the set {0,1,3,7}, the dequantized mantissa could be {0,1,3,5}. In some examples, the dequantized value is chosen deterministically. For example, for the set {0,1,3,7}, the value 3 can be dequantized to 2 on the first encounter, 3 on the second, 4 on the third, and then back to 2 on the fourth encounter. In some examples, a randomly selected dequantized mantissa can be used. For example, for the set {0,1,3,7}, the non-uniform tail value 3 can be translated into a random value chosen from the set {2,3,5,6}, and the non-uniform tail value 7 can be translated into a random value chosen from the set {5,4,6,7}. In some examples, random values ​​are chosen based on a uniform distribution, while in other examples, a normal (Gaussian), Poisson, or other probability distribution is used to choose the dequantized tail.

[0125] At process block 660, the loss of the neural network can be backpropagated through the neural network. During backpropagation, the output error term can be calculated. and weighted error term The output error term can be mathematically described as:

[0126]

[0127] in W is the output error term from layers after the i-th layer. i Let g be the weight tensor of the i-th layer, g() be the backpropagation function of that layer, and Q() be the quantization function, where Q is the weight tensor of the i-th layer. -1 () is the dequantization function. The backward function g() can be the backward function of f(), with respect to y. i-1 The gradient or part of the gradient function. The output error term of this layer can be a dequantized representation of g(), or the output error term can include additional terms performed using normal precision floating-point (after dequantization) or quantized floating-point (before dequantization).

[0128] Weighted error term It can be described mathematically as:

[0129]

[0130] in It is the weight error term of the i-th layer. It is the output error term of the i-th layer, yi This is the output of the i-th layer, h() is the backpropagation function of that layer, Q() is the quantization function, and Q... -1 () is the inverse quantization function. The backward function h() can be the backward function of f(), with respect to W. i-1 The gradient or weight error is part of Equation 9. The weight error term for this layer can be a dequantized representation of h(), or the weight error term can include additional terms performed using normal precision floating-point (after dequantization) or quantized floating-point (before dequantization). The weight error term can include additional terms performed using normal precision floating-point.

[0131] At process block 670, the parameters of each layer can be updated. For example, the weights of each layer can be updated by calculating new weights based on training iterations. As an example, the weight update function can be mathematically described as:

[0132]

[0133] in W is the weight error term of the i-th layer, η is the learning rate of the i-th layer of the neural network, and W is the weight error term of the i-th layer. i This is the weight tensor of the i-th layer. In one example, the weight update function can be performed using normal precision floating-point.

[0134] six, Example of an environment for performing active compression using non-uniform mantissa blocks of floating-point.

[0135] Figure 7 This is a block diagram 700 illustrating an example of a suitable environment for performing activation compression and associated floating-point operations between a normal-precision floating-point domain, a quantized floating-point domain, and a floating-point domain with lossy or non-uniform mantissas. The floating-point domain with lossy or non-uniform mantissas can have an exponent for all values, or a block floating-point format with two or more shared exponents. As mentioned above, computationally expensive operations, such as vector-vector, vector-matrix, matrix-matrix, and convolution operations, can be performed by the quantization layer 710 in the quantization domain. Less computationally expensive operations, such as scalar addition and scalar multiplication, can be performed outside the quantization layer 710 in the normal-precision floating-point domain. Regarding neural networks, the neural network can be divided into layers (such as the quantization layer 710). For the backpropagation portion of training, the error output component 720 can receive activation values ​​and use additional functions, such as an error function or an objective function, to compute the output error term. In some examples, the output error term is calculated in the normal precision floating-point domain. As shown by the solid line in the output error component 720. In other examples, the output error component 720 calculates the error output item in block floating-point format.

[0136] Most computational work within a layer can be performed in the quantized floating-point domain, while less computationally expensive operations of the layer, such as adding bias values ​​or calculating activation functions, can be performed in the normal-precision floating-point domain. Values ​​used for interfacing between layers can be passed from one layer to another in the normal-precision floating-point domain. By quantizing the input specifically for a given layer, quantization can be tailored to the operations of that layer, making those operations more efficient. Specifically, the bounding box of the quantized floating-point format can be selected to reduce the complexity of the computer's arithmetic circuitry, potentially making the computer logic faster and / or more energy-efficient.

[0137] As an example, for the output value y of a given layer i Output error term Weight W i and weighted error term It can be stored in the normal-precision floating-point domain. During the forward propagation flow, the output value from an earlier layer can be transferred from the normal-precision floating-point domain to the quantized floating-point domain via quantizer 722, which converts from normal-precision floating-point to quantized floating-point. The output value from a given layer can be transferred from the quantized floating-point domain to the standard-precision floating-point domain via dequantizer 724, which converts from quantized floating-point to standard-precision floating-point. The weights for a given layer can be transferred from the normal-precision floating-point domain to the quantized floating-point domain via quantizer 742. Dequantized value y i Or quantized value Q -1 (y i The data can be sent to compressor 760, which compresses the value before storing it in mass storage 770.

[0138] Compressor 760 may be a block floating-point compressor, or provide other forms of compression to reduce the amount of data stored in mass storage 770. In such an example, the value expressed in a second block floating-point format has a lossy or non-uniform mantissa. Typically, the first block floating-point format used to represent the value during quantization layer 710 operation has a uniform mantissa (e.g., in normal precision floating-point or block floating-point format). For example, the first block floating-point format used in quantization layer 710 may have more bits of mantissa than the second block floating-point format, such as four, five, six, seven, or eight bits; and the second floating-point format may have fewer bits of non-uniform mantissa than the first floating-point format, such as three, four, four or five, five or six, or four to six bits, respectively. Those skilled in the art benefiting from this disclosure will readily understand that the aforementioned combinations of specific bit lengths of the first and second block floating-point formats are merely preferred examples, but other precision combinations of mantissa formats may be used in other examples. In some examples, the second floating-point format has a lower precision exponent than the first floating-point format. In some examples, the first block of floating-point format uses a different shared format than the shared format of the common exponent of the second block of floating-point format. For example, the shared format can differ based on the shared common exponent of the activation value used for compression, whether it is shared per row, per column, or per block. The precision of the second block of floating-point format can be adjusted in a variety of different ways, including rounding, truncation, and / or shifting of mantissa values.

[0139] In the illustrated example, the activation value is dequantized 724 to normal precision format before being converted to a second block floating-point format and stored in mass storage 770. In some examples, compressor 760 is configured to further compress the activation value of the second block floating-point format by performing at least one or more of the following compression operations: entropy compression, zero compression, run-length encoding, compressed sparse row compression, or compressed sparse column compression.

[0140] Massive memory 770 can be implemented using any suitable memory or storage technology. In some examples, memory storing temporary values ​​in quantization layer 710 is typically implemented as static RAM (SRAM), embedded dynamic RAM (eDRAM), implemented as register file, block RAM, or other suitable structures, while mass memory 770 is typically implemented in memory structures that support larger but often slower access, such as off-chip DRAM, network-accessible RAM, SSD drives, hard disk drives, or network-accessible storage devices. Those skilled in the art who benefit from this disclosure will readily understand that the type and arrangement of memory used to implement quantization layer 710 and mass memory 770 can be selected based on desired performance attributes such as cost, energy, and speed.

[0141] Decompressor 780 reads the compressed activation value from mass storage 770 and reverses the operation performed by compressor 760. In the example where additional compression is applied to quantized values, these values ​​are compressed. These values ​​can then be translated from the second block floating-point format to normal precision floating-point format. The second block floating-point format values ​​can be dequantized in various different ways, such as as described above at processing block 650. Figure 6 The methods discussed herein. For example, a dequantized non-uniform mantissa can be converted to the same value in a uniform mantissa format. Alternatively, a dequantized non-uniform mantissa can be converted to an approximation in a uniform mantissa format, an alternative value chosen deterministically, or a random value based on a distribution chosen for a particular non-uniform mantissa. In an alternative example, decompressor 780 outputs values ​​in quantized block floating-point format, as indicated by the dashed lines.

[0142] Output error component 720 can receive activation values ​​in normal precision floating-point format (as in Equation 6a). In an alternative example, output error component 720 can receive activation values ​​in quantized floating-point format (as in Equation 6b), such as a second-block floating-point format, as indicated by the dashed line. Output error component 720 calculates the derivatives of the activation values ​​used in backpropagation. Backpropagation is performed using quantization layer 710, as indicated in Figure 700.

[0143] The dashed arrows in Figure 700 depict alternative paths for compressing and decompressing activation values. As shown, with each successive layer of the feedforward neural network, a first block of activation values ​​in floating-point format is generated. The compressor can then directly convert the quantized activation values ​​from the first block of floating-point format to a second block of floating-point format without needing to convert them to a normal-precision format, thus producing compressed activation values.

[0144] During backpropagation flow 730, output error terms from the next layer can be transferred from the normal precision floating-point domain to the quantized floating-point domain via quantizer 732. Output error terms from a given layer can be transferred from the quantized floating-point domain to the normal precision floating-point domain via dequantizer 734. Weights for a given layer can be transferred from the normal precision floating-point domain to the quantized floating-point domain via quantizer 742. Weight error terms from a given layer can be transferred from the quantized floating-point domain to the normal precision floating-point domain via dequantizer 754. As backpropagation proceeds, the quantized activation values ​​in the second block of floating-point format are either converted to normal precision format by quantizer 732 and translated back to the first block of floating-point format, or alternatively, converted from the second block of floating-point format used to store activation values ​​in mass memory 770 to the first block of floating-point format used by quantization layer 710. In an example where additional compression (e.g., entropy coding, zero coding, or other additional compression schemes) is applied before storing data in mass memory 770, the data can be further decompressed before quantization.

[0145] Figure 8 This is a further elaboration on the above regarding... Figure 7 Figure 800 discusses the operation of the compressor 760. As shown in Figure 800, the activation value y is received from the dequantizer 724. i The dequantizer 724 can receive activation values ​​from the local memory used to implement the quantization layer 710.

[0146] like Figure 8 As shown, compressor 760 receives a set 810 of normal-precision floating-point values, for example, from dequantizer 724. This set 810 of normal-precision floating-point values ​​is provided to quantizer 820, which implements a quantization function Q2() that converts the normal-precision values ​​into a second block floating-point format with a lossy or non-uniform mantissa, different from the block floating-point format used by quantization layer 710. In the illustrated example, each of the normal-precision floating-point values ​​810 has (for each value) a 10-bit mantissa, a 5-bit digit, and a 1-bit sign bit. These normal-precision floating-point values ​​810 are converted into a set Q2(y) of values ​​in a second block floating-point format 830. i In the illustrated example, the values ​​expressed in the second block floating-point format 830 each have a non-uniform 3-bit mantissa, a 1-bit sign bit, and all share a 4-bit exponent. In some examples, the exponent is not shared by all values ​​in the set, but may be shared on a per-row, per-column, or per-block basis. Those skilled in the art who benefit from this disclosure will readily understand that any suitable floating-point format using lossy or non-uniform mantissas can be used for the second block floating-point format 830. For example, a four-bit number with a sign bit and a mantissa bit, and a three-bit mantissa (which can represent eight values, such as the set {0,1,2,3,4,5,6,7}), can be converted into a lossy mantissa with a discrete set of mantissa values, such as any of the following sets: {0,1,3,7}, {0,1,7}, or {0,7}, depending on the chosen lossy mantissa scheme. The underlying representation of a lossy mantissa can vary. For the three sets mentioned above, example sets of corresponding binary representations are {00,01,10,11}; {00,10,11}; and {0,1}. In some examples, multi-valued (non-binary) representations can be used as the underlying representation of a lossy mantissa. For example, multi-level EEPROM or flash memory devices can use multiple levels of storage cells to store lossy mantissas.

[0147] In some examples, the set of values ​​of the second block of floating-point format 830 can be provided to an additional compression unit 840 for further compression before being stored in mass storage 770. Examples of suitable techniques for further compression of the activation values ​​of the compressed quantized format include entropy compression, zero compression, run-length compression, compressed sparse row compression, or compressed sparse column compression.

[0148] Regardless of whether the quantized value in the second floating-point format 830 is subjected to additional compression, the compressed value y ci All of these are stored in the mass memory 770. The mass memory 770 is typically implemented in memory structures that support larger but often slower access, such as off-chip DRAM, network-accessible RAM, SSD drives, hard disk drives, or network-accessible storage devices. Those skilled in the art who benefit from this disclosure will readily understand that the type and arrangement of memory used to implement the mass memory 770 can be selected based on desired performance attributes such as cost, energy, and speed.

[0149] Figure 8 The figure also illustrates an alternative example apparatus 860 for storing values ​​in a second block of floating-point format, wherein instead of receiving values ​​in normal precision floating-point format, quantized values ​​are received directly from the quantization layer 710. In this alternative example apparatus 860, the quantized value f(Q(y)) is received directly from the quantization layer of the first block of floating-point format 870. i-1 Each value in the set has a 7-bit mantissa, a 1-bit sign bit, and a shared 6-bit exponent. In some examples, the shared exponent is not shared by all the values, but may be shared only with certain values ​​on a per-row, per-column, or per-block basis. These values ​​in the first block of floating-point format can be directly converted to the second block of floating-point format Q2(f(Q(y)). i-1 As shown. Before being stored in the mass storage 770, the set of values ​​of the second floating-point format 830 can also be provided to the additional compression unit 840.

[0150] Those skilled in the art who benefit from this disclosure will readily understand that the illustrated normal precision and block floating-point format are not limited to the specific choices of the mantissa, exponent, and sign illustrated, and the scheme for sharing the exponent among values ​​may also vary depending on the desired aspects of a particular implementation of the compressor 760. Furthermore, values ​​stored in the mass storage 770 can be decompressed by reversing the operations shown in Figure 800.

[0151] Different numbers of techniques can be used to dequantize lossy or non-uniform mantissas. For example, when the non-uniform mantissa set is {0,1,3,7} (e.g., stored in a two-bit mantissa format), those identical values ​​can be recovered as dequantized values ​​of the first normal precision (e.g., 8, 16, or 32 bits) or block floating-point (e.g., 3, 4, 5, or 6 bits) mantissa. In some examples, the dequantized mantissa can be an approximation. For example, for the set {0,1,3,7}, the dequantized mantissa could be {0,1,3,5}. In some examples, the dequantized value is chosen deterministically. For example, for the set {0,1,3,7}, the value 3 can be dequantized to 2 on the first encounter, 3 on the second, 4 on the third, and then back to 2 on the fourth encounter. In some examples, a randomly selected dequantized mantissa can be used. For example, for the set {0,1,3,7}, the non-uniform tail value 3 can be translated into a random value chosen from the set {2,3,5,6}, and the non-uniform tail value 7 can be translated into a random value chosen from the set {5,4,6,7}. In some examples, random values ​​are chosen based on a uniform distribution, while in other examples, a normal (Gaussian), Poisson, or other probability distribution is used to choose the dequantized tail.

[0152] seven, Non-uniform, lossy mantissa format example

[0153] Figures 9-12 The figures illustrate examples of non-uniform, lossy mantissa formats that can be used to store compressed activation values ​​in certain examples of the disclosed techniques. Those skilled in the art who benefit from this disclosure will readily understand that these examples are provided for illustrative purposes, but other lossy, non-uniform formats may be used in other examples.

[0154] Figure 9Figure 900 illustrates an example of converting a uniform 3-bit mantissa format to a non-uniform four-valued lossy mantissa format, which may be referred to as a "lightly lossy format". As shown in the illustrated example, the uniform set of values ​​910 of the uniform 3-bit mantissa format is mapped non-uniformly to the set of values ​​920 of the non-uniform four-valued lossy mantissa format. When the uniform mantissa value is 0, it is converted to a lossy mantissa value of 0. When the uniform mantissa value is 1 or 2, the value is converted to a lossy mantissa value of 1. When the uniform mantissa value is any one of 3, 4, 5, or 6, the value is converted to a lossy mantissa value of 3. When the uniform mantissa value is 7, the value is converted to a lossy mantissa value of 7. The base representation can be a four-bit representation; in other words, 0, 1, 3, and 7 can be represented as binary values ​​00, 01, 10, or 11, respectively. Other base representations can also be used. The starting source mantissa is not limited to three bits as in the illustrated example, but can have other numbers of bits, such as 4, 5, 6, ... 32 bits. More generally, a four-valued slightly lossy mantissa will map the higher precision mantissa values ​​in four range sets: {[0],[1,mid-point-1],[mid-point,max-1],[max]} to the lossy mantissa values ​​{0,1,mid-point,max}, where mid-point is a value in the set that is between 1 and max.

[0155] Figure 10 Figure 1000 illustrates an example of converting a uniform 3-bit mantissa format to a non-uniform 3-valued lossy mantissa format, which may be referred to as a "normal lossy format". As shown in the illustrated example, the uniform value set 1010 of the uniform 3-bit mantissa format is mapped non-uniformly to the value set 1020 of the non-uniform 3-valued lossy mantissa format. When the uniform mantissa value is 0, it is converted to a lossy mantissa value of 0. When the uniform mantissa value is any one of 1, 2, 3, 4, 5, or 6, the value is converted to a lossy mantissa value of 1. When the uniform mantissa value is 7, the value is converted to a lossy mantissa value of 7. The base representation can be a two-bit representation; in other words, 0, 1, and 7 can be represented as binary values ​​00, 01, 10, or 11, respectively. In other examples, other representations with different bit lengths can be used, for example, 0, 10, and 11 for 0, 1, and 7, respectively. Other base representations can also be used. The starting source mantissa is not limited to three bits as in the example shown, but can have other numbers of bits, such as 4, 5, 6, ... 32 bits. More generally, a normal lossy mantissa will map the higher precision mantissa in three range sets (minimum, median, and maximum): {[0], [1, max-1], [max]} to the lossy mantissa {0, 1, max}, where max (maximum value) is the maximum value in the set.

[0156] Figure 11Figure 1100 illustrates an example of converting a uniform 3-bit mantissa format to a non-uniform 3-valued lossy mantissa format, which may be referred to as a “heavily lossy format.” As shown in the illustrated example, the uniform set of values ​​1110 of the uniform 3-bit mantissa format is mapped non-uniformly to the set of values ​​1120 of the non-uniform binary lossy mantissa format. When the uniform mantissa value is 0, it is converted to a lossy mantissa value of 0. When the uniform mantissa value is any one of 1, 2, 3, 4, 5, 6, or 7, the value is converted to a lossy mantissa value of 7. The base representation can be a one-bit representation. Other base representations can also be used. The starting source mantissa is not limited to three bits as in the illustrated example, but can have other numbers of bits, such as 4, 5, 6, ... 32 bits. More generally, a heavily lossy mantissa maps the higher-precision mantissa value in two range sets: {[0], [1, max]} to the lossy mantissa value {0, max}, where max (maximum value) is the maximum value in the set.

[0157] Figure 12 Figure 1200 illustrates an example of converting a uniform value expressed as a sign / mantissa (as a sign bit and a 3-bit mantissa) to a non-uniform value, which may be referred to as a "full-range lossy format". As shown in the illustrated example, the set 1210 of uniform values ​​in the uniform format is mapped in a non-uniform manner to a set 1220 of values ​​in the non-uniform five-value lossy mantissa format. When the sign / mantissa is -7, it is converted to a lossy mantissa of -7. When the sign / mantissa is any one of -6, -5, -4, -3, or -2, it is converted to a lossy mantissa of -3. When the sign / mantissa is any one of -1, 0, or 1, it is converted to a lossy mantissa of 0. When the sign / mantissa is any one of 2, 3, 4, 5, or 6, it is converted to a lossy mantissa of 3. When the sign / mantissa is 7, it is converted to a lossy mantissa of 7. The underlying representation used for the lossy sign / mantissa can be, for example, a three-bit representation. Other underlying representation formats may also be used. The starting source mantissa is not limited to three bits as shown in the example illustrated, but can have other numbers of bits, such as 4, 5, 6, ... 32 bits.

[0158] eight, Example method using activation values ​​with non-uniform tails

[0159] Figure 13 This is a flowchart 1300 outlining an example method using activation values ​​expressed in a non-uniform mantissa, as may be performed in some examples of the disclosed techniques. As an example, the illustrated method can use a quantization-enabled system (such as those described above regarding...). Figure 1 The discussion and above about Figure 7 and Figure 8The quantization-enabled system 110, described in further detail in the quantization environment discussed, is used to perform this. In some examples, the illustrated method is used as part of a method for training a neural network.

[0160] At procedure block 1310, a forward propagation is performed on the layers of the neural network to generate first activation values ​​in a first floating-point format. The floating-point format can be either normal-precision floating-point or block floating-point format. The mantissa of the floating-point number in the first format is typically expressed as one of a set of uniform values. For example, a standard multi-bit binary value can be used to represent the mantissa.

[0161] At process block 1320, at least some of these first activation values ​​can be converted to a second-block floating-point format with a non-uniform mantissa to produce compressed activation values ​​in the second-block floating-point format. Any suitable format for expressing the non-uniform mantissa can be used. For example, for the above regarding... Figures 9-12 The examples discussed can use any slightly lossy, normally lossy, heavily lossy, or fully lossy mantissa format. In some examples, the second block of floating-point format has a lower precision exponent or a different exponent-sharing scheme compared to storing values ​​in the first block of floating-point format. In some examples, these values ​​can be further compressed, for example, using entropy compression or other suitable compression schemes. In some examples, the first activation value is converted to a normal precision floating-point format before being converted to the second block of floating-point format. In other examples, the first activation value is converted directly from the first block of floating-point format to the second block of floating-point format.

[0162] At process block 1330, the activation values ​​are stored in mass memory. Mass memory can be implemented using any suitable storage technology, including, for example, on-chip or off-chip DRAM, network-accessible RAM, SSD drives, hard disk drives, or network-accessible storage devices. In some examples, the mass memory resides on a different integrated circuit than the hardware accelerator used to process block floating-point values. In other examples, the mass memory resides on the same integrated circuit as the hardware accelerator used to process block floating-point values. After the activation values ​​are stored in mass memory, computation for the neural network can proceed to the next layer. For example, once the activation values ​​of one layer have been stored in mass memory, forward propagation can continue for many different layers in the neural network.

[0163] During backpropagation, the activation values ​​stored in the mass memory at process block 1330 are retrieved. For example, when performing backpropagation on a neural network, activation values ​​can be retrieved from the mass memory and used to compute gradients as part of the backpropagation process. The activation values ​​retrieved from the mass memory are converted from a second quantization format used to store the values ​​in memory to a first quantization format used when performing neural network computations. For example, a hardware accelerator can be configured to process the neural network using values ​​expressed in a first block floating-point format.

[0164] Nine, Example method for training neural networks using activation compression with non-uniform mantissa blocks of floating-point numbers.

[0165] Figure 14 This is a flowchart 1400 outlining an example method for training a neural network using activation compression with a non-uniform mantissa block floating-point format, as can be performed in some examples of the disclosed techniques. As an example, the illustrated method can use a quantization-enabled system (such as those described above regarding...). Figure 1 The discussion and above about Figure 7 and Figure 8 The quantization-enabled system 110, described in further detail in the quantization environment discussed, is used to perform this. In some examples, the illustrated method is used as part of a method for training a neural network.

[0166] At process block 1410, at least one layer of the neural network is implemented by providing first weights and / or first activation values ​​in a first block floating-point format. For example, a source neural network represented in normal-precision floating-point numbers can convert its weights and / or activation values ​​to the selected first block floating-point format. Other aspects of the neural network can also be implemented, such as providing parameters about network topology, sparsity, node types, and / or training parameters as part of implementing one or more layers of the neural network. After implementing the neural network, forward propagation is performed on the layers of the neural network to produce second activation values ​​in the first block floating-point format. The first block floating-point format can be selected based on the hardware accelerator used to process the activation values.

[0167] At process block 1420, at least one of the second activation values ​​generated during forward propagation is converted to a second block floating-point format with a non-uniform mantissa. For example, for the above regarding... Figures 9-12 The examples discussed can use any slightly lossy, normally lossy, heavily lossy, or fully lossy mantissa format. Other aspects of the second floating-point format can differ from the first. For example, the second floating-point format can have a smaller mantissa, a smaller exponent, or a different shared exponent scheme than the first floating-point format.

[0168] At process block 1430, some examples perform additional optional steps: applying further compression to the compressed activation values ​​before storing them in mass memory. Examples of suitable techniques for further compressing activation values ​​in compressed quantized formats include entropy compression (e.g., Huffman coding), zero compression, run-length compression, compressed sparse row compression, or compressed sparse column compression.

[0169] At process block 1440, the compressed activation value from process block 1420 (or the further compressed activation value generated at optional process block 1430) is stored in a storage device or memory (such as mass memory). Mass memory can be implemented using any suitable storage technology, including, for example, on-chip or off-chip DRAM, network-accessible RAM, SSD drives, hard disk drives, or network-accessible storage devices. In some examples, mass memory resides on a different integrated circuit than the hardware accelerator used to process block floating-point values. In some examples, mass memory resides on the same integrated circuit as the hardware accelerator used to process block floating-point values.

[0170] At process block 1450, backpropagation is performed on the layers of the neural network by converting the stored compressed activation values ​​to dequantized and (if compression was performed at process block 1430) uncompressed activation values. The uncompressed activation values ​​can be expressed in either first-quantization block floating-point format or normal-precision floating-point format. The format used for the uncompressed activation values ​​can be selected based on the hardware accelerator used to perform the neural network operation at process block 1460. A variety of techniques can be used to dequantize lossy or non-uniform mantissas. For example, when the set of non-uniform mantissas is {0,1,3,7} (e.g., stored in a two-bit mantissa format), those identical values ​​can be recovered as dequantized values ​​of first normal-precision (e.g., 8, 16, or 32-bit) or block floating-point (e.g., 3, 4, 5, or 6-bit) mantissas. In some examples, the dequantized mantissa can be an approximation. For example, for the set {0,1,3,7}, the dequantized mantissa could be {0,1,3,5}. In some examples, the dequantized value is selected deterministically. For example, for the set {0,1,3,7}, the value 3 can be dequantized to 2 the first time it is encountered, 3 the second time, 4 the third time, and then back to 2 the fourth time it is encountered. In some examples, a randomly selected dequantization mantissa can be used. For example, for the set {0,1,3,7}, the non-uniform mantissa 3 can be translated into a random value chosen from the set {2,3,5,6}, and the non-uniform mantissa 7 can be translated into a random value chosen from the set {5,4,6,7}. In some examples, a random value is chosen based on a uniform distribution, while in other examples, a normal (Gaussian), Poisson, or other probability distribution is used to choose the dequantization mantissa.

[0171] At process block 1460, a gradient operation is performed using the dequantized and (if compression was performed at process block 1430) uncompressed activation values. This gradient operation is used as part of the neural network training process. For example, the gradient operation can be used to determine the updated weights of the nodes in the neural network.

[0172] At process block 1470, the weights of each part of at least one node in the neural network are updated based on the compressed activation values. For example, the gradient computed at process block 1460 can be used to update the weights of the nodes in the neural network.

[0173] ten, Example computing environment

[0174] Figure 15 The figure illustrates a generalized example of a suitable computing environment 1500 in which the described embodiments, techniques and sciences can be implemented, including performing active compression with lossy or non-uniform mantissas in a quantization environment.

[0175] The Computing Environment 1500 is not intended to impose any limitations on the scope or functionality of the technology, as it can be implemented in various general-purpose or special-purpose computing environments. For example, the disclosed technology can be implemented using other computer system configurations, including handheld devices, multiprocessor systems, programmable consumer electronics, network PCs, minicomputers, mainframes, etc. The disclosed technology can also be practiced in distributed computing environments, where tasks are performed by remote processing devices linked via a communication network. In a distributed computing environment, program modules can reside in both local and remote memory storage devices.

[0176] refer to Figure 15 The computing environment 1500 includes at least one processing unit 1510 and a memory 1520. Figure 15In this diagram, the most basic configuration 1530 is included within the dashed lines. Processing unit 1510 executes computer-executable instructions and can be a real or virtual processor. In a multiprocessor system, multiple processing units execute computer-executable instructions to increase processing power, and therefore, multiple processors can operate simultaneously. Memory 1520 can be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of both. Memory 1520 stores, for example, software 1580 implementing the techniques described herein, images, and video. The computing environment can have additional features. For example, computing environment 1500 includes storage device 1540, one or more input devices 1550, one or more output devices 1560, and one or more communication connections 1570. Interconnection mechanisms (not shown), such as buses, controllers, or networks, interconnect the components of computing environment 1500. Typically, operating system software (not shown) provides an operating environment for other software executing in computing environment 1500 and coordinates the activities of the components of computing environment 1500.

[0177] Storage device 1540 may be removable or non-removable and includes a disk, magnetic tape or cassette, CD-ROM, CD-RW, DVD, or any other medium that can be used to store and access information within computing environment 1500. Storage device 1540 stores instructions, plug-in data, and messages for software 1580, which can be used to implement the techniques described herein.

[0178] One or more input devices 1550 may be touch input devices such as a keyboard, keypad, mouse, touchscreen display, pen, or trackball, voice input devices, scanning devices, or other devices that provide input to the computing environment 1500. For audio, one or more input devices 1550 may be a sound card or similar device that accepts audio input in analog or digital form, or a CD-ROM reader that provides audio samples to the computing environment 1500. One or more output devices 1560 may be a monitor, printer, speaker, CD burner, or other devices that provide output from the computing environment 1500.

[0179] One or more communication connections 1570 enable communication with another computing entity via a communication medium (e.g., a connected network). The communication medium conveys information such as computer-executable instructions, compressed graphics information, video, or other data in modulated data signals. One or more communication connections 1570 are not limited to wired connections (e.g., megabit or gigabit Ethernet, Infiniband, fiber optic channels connected electrically or optically) but also include wireless technologies (e.g., RF connections via Bluetooth, WiFi (IEEE 802.11a / b / n), WiMax, cellular, satellite, laser, infrared) and other suitable communication connections for providing network connectivity to exposed agents, bridges, and agent data consumers. In a virtual hosting environment, one or more communication connections may be virtualized network connections provided by the virtual host.

[0180] Some embodiments of the disclosed methods can be executed using computer-executable instructions that implement all or part of the disclosed technology in computing cloud 1590. For example, the disclosed methods can be executed on processing unit 1510 located in computing environment 1530, or the disclosed methods can be executed on a server located in computing cloud 1590.

[0181] Computer-readable media is any available medium that can be accessed within computing environment 1500. By way of example and not limitation, for computing environment 1500, computer-readable media includes memory 1520 and / or storage device 1540. It should be readily understood that the term computer-readable storage medium includes media used for data storage, such as memory 1520 and storage device 1540, but excludes transmission media such as modulated data signals.

[0182] XI. Experimental Results

[0183] Figure 16 Chart 1600 shows multiple graphs illustrating the accuracy of a neural network over more than 90 training epochs when it uses different schemes to compress activation values ​​with a lossy mantissa. The accuracy of the trained neural network typically improves as training progresses. The first graph 1610 represents the baseline case, where normal-precision floating-point values ​​are used to represent the neural network activation values ​​during training without compression. The second set of graphs 1620 represents various cases using different schemes to compress activation values ​​with a lossy mantissa. This second set includes seven-bit lossy mantissas, including those based on mild, normal, and severe non-uniform mantissa schemes, similar to those above regarding... Figures 9-11The discussion focuses on three-digit mantissas. The third graph, 1630, addresses the test case using a five-digit mantissa with normal and non-uniform mantissa schemes. The fourth set of graphs, 1640, addresses multiple additional test cases using different schemes for compressing activation values ​​with lossy mantissas, including a six-digit mantissa with a heavily non-uniform mantissa scheme, a five-digit mantissa with a heavily non-uniform mantissa scheme, and a four-digit mantissa with a slightly non-uniform scheme. The fifth set of graphs, 1650, addresses two test cases involving four-digit lossy mantissas based on either a normal or heavily non-uniform scheme.

[0184] Therefore, those skilled in the art who benefit from this disclosure will understand that the performance of a quantized-enabled system using lossy or non-uniform mantissas to implement a neural network can be improved compared to a baseline. The chosen approach can be selected to achieve the desired level of accuracy. For example... Figure 16 As shown, some of these schemes achieve accuracy close to the baseline normal precision floating-point case, while also achieving substantial improvements in memory usage, interconnect and bandwidth usage, network usage, and power usage, and reducing processor load, thereby improving the performance of the quantization system itself.

[0185] twelve, Additional examples of the disclosed technology

[0186] This article discusses additional examples of the disclosed topics based on the examples discussed above.

[0187] In some examples of the disclosed technology, a system of one or more computers can be configured to perform a specific operation or action by means of software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform actions during operation. One or more computer programs can be configured to perform a specific operation or action by means of instructions that cause the apparatus to perform actions when executed by a data processing device. One general aspect includes a computing system comprising: one or more processors, and a mass memory including computer-readable storage devices and / or memories. The computing system also includes a floating-point compressor formed by at least one of the processors, the block floating-point compressor communicating with the mass memory. The computing system is configured to: perform forward propagation against layers of a neural network to produce first activation values ​​in a first floating-point format having a normal mantissa format; convert at least one activation value among the activation values ​​to a second floating-point format to produce compressed activation values ​​by mapping the mantissa of the activation values ​​to a non-uniform mantissa format; and store the compressed activation values ​​in the mass memory. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0188] Some implementations may include one or more of the following features: A computation system in which the second floating-point format has a lower mantissa precision than the first floating-point format. A computation system in which the mapping includes: mapping a mantissa of two or more mantissa values ​​having a normal mantissa format to a single mantissa of a non-uniform mantissa format. In some examples, the mapping includes: mapping a first mantissa of one or more mantissa values ​​having a normal mantissa format to a single mantissa value of a non-uniform mantissa format; and mapping a second mantissa of at least one more mantissa value having a normal mantissa format to a single mantissa value of a non-uniform mantissa format. In some examples, the first and second floating-point formats are block floating-point formats, and the second floating-point format has a different common exponent sharing format from the first block floating-point format, the sharing format being different based on per-row, per-column, or per-block sharing of the common exponent for the compressed activation value. In some examples, the compressor is also configured to further compress the compressed activation value before storage by performing at least one or more of the following: entropy compression, zero compression, run-length encoding, compressed sparse row compression, or compressed sparse column compression. In some examples, the computational system is also configured to perform backpropagation on layers of a neural network by converting stored compressed activation values ​​into activation values ​​in a first floating-point format to produce uncompressed activation values. In some examples, the computational system may also utilize the uncompressed activation values ​​to perform gradient operations. In some examples, the layer is a first layer, the compressed activation values ​​are first compressed activation values, the non-uniform mantissa format is a first non-uniform mantissa format, and the computational system is further configured to: perform forward propagation on different second layers of the neural network using at least one processor to produce second activation values ​​in a first floating-point format; convert at least one of the second activation values ​​into a third floating-point format to produce second compressed activation values, the third floating-point format having activation value mantissas of a second non-uniform mantissa format different from the first non-uniform mantissa format, using a floating-point compressor; and store the second compressed activation values ​​in a mass memory using at least one processor.

[0189] In some examples of computing systems, the second floating-point format includes a lossy mantissa. In some examples, the second floating-point format includes a non-uniform mantissa. In some examples, the second floating-point format includes mantissa values ​​with non-uniform intervals. In some examples, the second floating-point format includes mantissa values ​​that are non-uniformly mapped from the first floating-point format. In some examples, slightly lossy mantissas, normally lossy mantissas, and / or heavily lossy mantissas are used. In some examples of computing systems, the third floating-point format includes a lossy mantissa. In some examples, the third floating-point format includes a non-uniform mantissa. In some examples, the third floating-point format includes mantissa values ​​with non-uniform intervals. In some examples, the third floating-point format includes mantissa values ​​that are non-uniformly mapped from either the first or second floating-point format. In some examples, slightly lossy mantissas, normally lossy mantissas, and / or heavily lossy mantissas are used.

[0190] In some examples, the processor includes at least one of the following: a tensor processing unit, a neural network accelerator, a graphics processing unit, or a processor implemented as a reconfigurable logic array; and the mass storage is located on an integrated circuit separate from the processor. The mass storage includes a computing system comprising dynamic random access memory (DRAM) or embedded DRAM, and the system further includes a hardware accelerator comprising memory temporarily storing at least a portion of the first activation values ​​of only one layer of a neural network, the hardware accelerator memory comprising static RAM (SRAM) or a register file. Implementations of the described techniques may include hardware, methods, or processes, or computer software on a computer-accessible medium.

[0191] One general aspect includes a method of operating a computing system implementing a neural network, the method comprising: using the computing system to forward propagate layers of the neural network to generate activation values ​​in a first floating-point format; converting at least one of the activation values ​​into a second block floating-point format having a non-uniform mantissa to generate compressed activation values; and storing the compressed activation values ​​in a computer-readable storage medium or storage device. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0192] Some implementations may include one or more of the following features. The method, wherein the second block of floating-point format has one of the following mantissa formats: a slightly lossy format, a normally lossy format, or a heavily lossy format. The method, wherein the second block of floating-point format has a slightly lossy mantissa format comprising a one-to-one mapping to a selected lowest-value mantissa of the first floating-point format. The method may also include a one-to-one mapping to a selected highest-value mantissa of the first floating-point format. The method may also include two-to-one or many-to-one mappings to at least two other mantissa values ​​of the first floating-point format. The method may also include non-uniform mappings to at least two mantissa values ​​of the first floating-point format. In some examples, the second block of floating-point format has a heavily lossy mantissa format comprising a one-to-one mapping to a selected lowest-value mantissa of the first floating-point format. The method may also include mappings to all other mantissa values ​​besides the selected lowest-value mantissa of the first floating-point format. The method further includes: prior to storage, further compressing the compressed activation values ​​stored in a computer-readable storage medium or storage device using at least one or more of the following techniques: entropy compression, zero compression, run-length encoding, compressed sparse row compression, or compressed sparse column compression. In some examples, the second block floating-point format has a two-to-one or many-to-one mapping for at least two mantissa values ​​of the first floating-point format, and the method further includes: using a computing system to dequantize the compressed activation values ​​by converting at least one mantissa of the compressed activation values ​​to an average value based on at least two mantissa values. In some examples, the second block floating-point format has a two-to-one or many-to-one mapping for at least two mantissa values ​​of the first floating-point format, and the method further includes: using a computing system to dequantize the compressed activation values ​​by converting at least one mantissa of the compressed activation values ​​to a randomly selected value from at least two mantissa values. In some examples, the method further includes: using a computing system to perform backpropagation on layers of a neural network by converting the stored compressed activation values ​​to activation values ​​in the first block floating-point format and then to uncompressed activation values. The method may further include: using a computational system to perform gradient operations using uncompressed activation values. The method may further include: using a computational system to update the weights of at least one node of the neural network based on the uncompressed activation values. The method may further include: using a computational system to perform backpropagation on a layer of the neural network by converting stored compressed activation values ​​into a first block of floating-point format activation values ​​and then into uncompressed activation values. The method may further include: using a computational system to perform gradient operations using uncompressed activation values; and using a computational system to update a portion of the weights of at least one node of the neural network based on the uncompressed activation values, wherein the at least one node is one of: a Long Short-Term Memory (LSTM) node, or a Gated Recurrent Unit (GRU).The method may further include selecting a second floating-point format based on layer attributes selected from the group consisting of: the layer is a convolutional layer; the layer includes long short-term memory (LSTM) nodes; the layer includes gated recurrent units (GRUs); the layer is fully connected to another layer; the layer is sparsely connected to another layer; the layer is an attention layer; and the layer is a normalization layer. Certain implementations of the described techniques may include hardware, methods, or processes, or computer software on a computer-accessible medium.

[0193] One general aspect includes one or more computer-readable storage devices or media storing computer-executable instructions that, when executed by a computer, cause the computer to perform a method of configuring a computer system to implement an artificial neural network, the instructions including: causing the computer system to implement a first layer of the neural network using first weights and / or first activation values ​​expressed in a first block floating-point format. The one or more computer-readable storage devices further include instructions causing the computer system to forward propagate values ​​from the first layer of the neural network to a second layer of the neural network, thereby generating second activation values ​​expressed in the first floating-point format. The one or more computer-readable storage devices further include instructions causing the computer system to store the second activation values ​​in a different second block floating-point format having a lossy mantissa format in a mass memory or storage device in communication with the computer system before performing a backpropagation for the neural network. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0194] In some examples of computer-readable storage media, the second floating-point format includes a lossy mantissa. In some examples, the second floating-point format includes a non-uniform mantissa. In some examples, the second floating-point format includes mantissa values ​​with non-uniform intervals. In some examples, the second floating-point format includes mantissa values ​​that are non-uniformly mapped from the first floating-point format. In some examples, slightly lossy mantissas, normally lossy mantissas, and / or heavily lossy mantissas are used. In some examples of computing systems, the third floating-point format includes a lossy mantissa. In some examples, the third floating-point format includes a non-uniform mantissa. In some examples, the third floating-point format includes mantissa values ​​with non-uniform intervals. In some examples, the third floating-point format includes mantissa values ​​that are non-uniformly mapped from the first or second floating-point format. In some examples, slightly lossy mantissas, normally lossy mantissas, and / or heavily lossy mantissas are used.

[0195] A general aspect also includes instructions that cause the computer system to temporarily store a first weight and / or a first activation value in a memory different from a mass storage or storage device. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0196] Another general aspect includes instructions that cause the computer system to further compress the second activation value before storing the further compressed value in a mass storage or storage device. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0197] Some of the disclosed methods may be implemented as computer-executable instructions stored on one or more computer-readable media (e.g., computer-readable media such as one or more optical discs, volatile memory components such as DRAM or SRAM, or non-volatile memory components such as hard disk drives) and executed on a computer (e.g., any commercial computer, including smartphones or other mobile devices that include computing hardware). Any computer-executable instructions used to implement the disclosed technology, and any data created and used during the implementation of the disclosed embodiments, may be stored on one or more computer-readable media (e.g., computer-readable storage media).

[0198] Given the many possible embodiments to which the principles of the disclosed subject matter can be applied, it should be understood that the illustrated embodiments are merely preferred examples and should not be construed as limiting the scope of the claims to those preferred examples. Rather, the scope of the claimed subject matter is defined by the following claims. Therefore, we claim all that falls within the scope of these claims as our invention.

Claims

1. A computing system for neural network activation compression using non-uniform mantissa, comprising: One or more processors; Mass storage devices, including computer-readable storage devices and / or memory; A block floating-point compressor formed by at least one of the processors, the block floating-point compressor communicating with the mass storage; and The computing system is configured as follows: Using at least one of the processors, forward propagation is performed on a layer of the neural network to produce a first activation value in a first floating-point format having a normal mantissa format. Using the block floating-point compressor, at least one of the activation values ​​is converted to a second floating-point format by mapping the mantissa of the activation value to a non-uniform mantissa format to produce a compressed activation value. The compressed activation value of the second floating-point format has a shared exponent value. If the activation value of the first floating-point format has an exponent value that is different from the shared exponent value, each mantissa value of the non-uniform mantissa format is shifted to ensure that the corresponding exponent value is the shared exponent value. The number of bits representing the mantissa of the compressed activation value of the second floating-point format is less than the number of bits representing the mantissa of the activation value of the first floating-point format. as well as The compressed activation value is stored in the mass memory using at least one of the processors.

2. The computing system according to claim 1, wherein the second floating-point format has a lower precision mantissa compared to the first floating-point format.

3. The computing system according to claim 1, wherein the mapping includes: Map the mantissas of two or more mantissas with the normal mantissa format to a single mantissa with the non-uniform mantissa format.

4. The computing system according to claim 1, wherein the mapping includes: Map a first mantissa having one or more mantissa values ​​in the normal mantissa format to a single mantissa value in the non-uniform mantissa format; as well as Map a second mantissa with at least one more mantissa value in the normal mantissa format to a single mantissa value in the non-uniform mantissa format.

5. The computing system of claim 1, wherein the compressor is further configured to further compress the compressed activation value prior to storage by performing at least one or more of the following: entropy compression, zero compression, run-length encoding, compressed sparse row compression, or compressed sparse column compression.

6. The computing system of claim 1, wherein the computing system is further configured to: Backpropagation is performed on layers of the neural network by converting the stored compressed activation values ​​into uncompressed activation values ​​in the first floating-point format; and Gradient operations are performed using the uncompressed activation values.

7. The computing system of claim 1, wherein the layer is a first layer, the compressed activation value is a first compressed activation value, the non-uniform mantissa format is a first non-uniform mantissa format, and wherein the computing system is further configured to: Using at least one of the processors, forward propagation is performed for different second layers of the neural network to generate second activation values ​​in the first floating-point format; Using the block floating-point compressor, for at least one of the second activation values, the at least one of the second activation values ​​is converted into a third floating-point format to generate a second compressed activation value, the third floating-point format having an activation value mantissa of a second non-uniform mantissa format, the second non-uniform mantissa format being different from the first non-uniform mantissa format; as well as The second compressed activation value is stored in the mass memory using at least one of the processors.

8. The computing system according to claim 1, wherein: The processor includes at least one of the following: a tensor processing unit, a neural network accelerator, a graphics processing unit, or a processor implemented as a reconfigurable logic array; and The large-capacity memory is located on a different integrated circuit than the processor.

9. The computing system of claim 1, wherein the mass storage includes dynamic random access memory or embedded dynamic random access memory, and the system further includes a hardware accelerator, the hardware accelerator including memory for temporarily storing at least a portion of the first activation values ​​for only one layer of the neural network, the hardware accelerator memory including static random access memory or register file.

10. A method for operating a computing system that utilizes non-uniform mantissa for neural network activation compression, the method comprising: Using at least one processor of the computing system, the layers of the neural network are forward-propagated to generate activation values ​​in a first floating-point format; Using the block floating-point compressor of the computing system, at least one activation value among the activation values ​​is converted to a second block floating-point format by mapping the mantissa of the activation value to a non-uniform mantissa format to produce a compressed activation value. The compressed activation value in the second block floating-point format has a shared exponent value, wherein if the activation value in the first floating-point format has an exponent value different from the shared exponent value, each mantissa value in the non-uniform mantissa format is shifted to ensure that the corresponding exponent value is the shared exponent value, and wherein the number of bits representing the mantissa of the compressed activation value in the second block floating-point format is less than the number of bits representing the mantissa of the activation value in the first floating-point format; and The compressed activation value is stored in a computer-readable storage device or memory using at least one processor of the computing system.

11. The method of claim 10, wherein the second block floating-point format has one of the following mantissa formats: a slightly lossy format, a normally lossy format, or a heavily lossy format.

12. The method of claim 10, wherein the second block floating-point format has a slightly lossy mantissa format, the slightly lossy mantissa format comprising: A one-to-one mapping of the lowest selected mantissa for the first floating-point format; A one-to-one mapping for the highest value mantissa selected for the first floating-point format; as well as A two-to-one or many-to-one mapping for at least two other mantissa values ​​in the first floating-point format.

13. The method of claim 10, wherein the second block floating-point format has a heavily lossy mantissa format, the heavily lossy mantissa format comprising: A one-to-one mapping of the lowest selected mantissa for the first floating-point format; as well as Mapping for all mantissa values ​​except for the lowest selected mantissa value in the first floating-point format.

14. The method of claim 10, further comprising: Prior to storage, the compressed activation value stored in the computer-readable storage memory or storage device is further compressed using one or more of the following techniques: Entropy compression, zero compression, run-length encoding, compressed sparse row compression, or compressed sparse column compression.

15. The method of claim 10, wherein the second block floating-point format has a two-to-one or many-to-one mapping for at least two mantissa values ​​of the first floating-point format, the method further comprising: Using the computing system, the compressed activation value is dequantized by converting at least one mantissa of the compressed activation value to the average of the at least two mantissas.

16. The method of claim 10, wherein the second block floating-point format has a two-to-one or many-to-one mapping for at least two mantissa values ​​of the first floating-point format, the method further comprising: Using the computing system, the compressed activation value is dequantized by converting at least one mantissa of the compressed activation value into a randomly selected value of the at least two mantissas.

17. The method of claim 10, further comprising: Using the computing system, backpropagation is performed on the layers of the neural network by converting the stored compressed activation values ​​into activation values ​​in the first floating-point format to generate uncompressed activation values; as well as Using the aforementioned computing system, gradient operations are performed using the uncompressed activation values; as well as Using the computing system, the weights of at least one node of the neural network are updated based on the uncompressed activation values.

18. One or more computer-readable storage devices or media storing computer-executable instructions, which, when executed by a computer, cause the computer to configure a computer system to implement a neural network using non-uniform mantissa activation compression, the instructions comprising: Instructions that enable the computer system to implement the first layer of the neural network using a first activation value expressed in a first block of floating-point format; Instructions that cause the computer system to propagate values ​​from the first layer of the neural network to the second layer of the neural network, thereby generating a second activation value expressed in a first block of floating-point format; The computer system generates compressed activation values ​​by mapping the mantissa of activation values ​​to a non-uniform mantissa format and converting at least one activation value to a second block floating-point format. The compressed activation values ​​in the second block floating-point format have a shared exponent value, wherein if the activation value in the first block floating-point format has an exponent value different from the shared exponent value, each mantissa value in the non-uniform mantissa format is shifted to ensure that the corresponding exponent value is the shared exponent value, and wherein the number of bits representing the mantissa of the compressed activation value in the second block floating-point format is less than the number of bits representing the mantissa of the activation value in the first block floating-point format; and Instructions that cause the computer system to store the second activation value in a different second block of floating-point format with a lossy mantissa format in a mass memory or storage device communicating with the computer system before performing backpropagation for the neural network.

19. The computer-readable storage device or medium of claim 18, further comprising: The instruction to cause the computer system to further compress the second activated value before storing the further compressed value in the mass storage or storage device.

20. The computer-readable storage device or medium of claim 18, further comprising: Instructions that enable the computer system to select the lossy mantissa format based on one aspect of the second layer.

Images