Training Neural Network Accelerators Using Mixed Precision Data Formats

Abstract

Technology related to training a neural network accelerator using a mixed precision data format is disclosed. In one example of the disclosed technology, a neural network accelerator is configured to accelerate a given layer of a multi-layer neural network. An input tensor of the given layer can be converted from a normal precision floating point format to a quantized precision floating point format. Tensor operations can be performed using the converted input tensors. The result of the tensor operation can be converted from a block floating point format to a normal precision floating point format. The converted result can be used to generate an output tensor of the layer of the neural network, where the output tensor is in a normal precision floating point format.

Description

Background Art

[0001] Machine learning (ML) and artificial intelligence (AI) technology can be used to solve many complex computing problems, such as recognizing images and voices, analyzing and classifying information, and performing various classification tasks. Machine learning is a field of computer science that uses statistical techniques to give computer systems the ability to extract higher-level features from a set of training data. Specifically, features can be extracted by training models such as artificial neural networks (NN) or deep neural networks (DNN). After the model is trained, new data can be applied to the model, and the trained model can be used to classify the new data (for example, higher-level features can be extracted). Machine learning models are usually executed on general-purpose processors (also referred to as central processing units (CPUs)). However, the amount of calculation using the model may be large, so it may not be possible to use a general-purpose processor to perform feature extraction in real time. Reducing the computational complexity of the model used can potentially reduce the time to extract features during reasoning, reduce the adjustment time during training, and / or reduce the energy consumption during training and / or reasoning. Therefore, there are a lot of opportunities to improve computer hardware and software to implement neural networks. Summary of the invention

[0002] Technology related to training a neural network accelerator using a mixed precision data format is disclosed. In one example of the disclosed technology, a neural network accelerator is configured to accelerate a given layer of a multi-layer neural network. An input tensor of the given layer can be converted from a normal precision floating point format to a quantized precision floating point format. Tensor operations can be performed using the converted input tensors. The result of the tensor operation can be converted from a block floating point format to a normal precision floating point format. The converted result can be used to generate an output tensor of the layer of the neural network, where the output tensor is in a normal precision floating point format. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 is a block diagram of a quantization-enabled system that may be implemented in certain examples of the disclosed technology.

[0004] Figure 2 is a diagram depicting a deep neural network that may be modeled using certain example methods and apparatus disclosed herein.

[0005] Figure 3 is a diagram depicting certain aspects of converting a normal floating point format to a quantized floating point format that may be performed in certain examples of the disclosed technology.

[0006] Figure 4 Depicted are a number of alternative block floating point formats that may be used to represent quantized neural network models that may be used in certain examples of the disclosed techniques.

[0007] Figure 5 Depicted are a number of alternative block floating point formats that may be used to represent quantized neural network models that may be used in certain examples of the disclosed techniques.

[0008] Figure 6 is a flow chart depicting a method of training a neural network for use with a quantized model that may be implemented in certain examples of the disclosed technology.

[0009] Figure 7 is a block diagram depicting a mixed-precision floating-point environment and a boundary between a normal-precision floating-point domain and a quantized floating-point domain that may be implemented in certain examples of the disclosed technology.

[0010] Figure 8 is a flowchart depicting a method for training a neural network accelerator using a mixed precision data format that may be implemented in certain examples of the disclosed technology.

[0011] Fig. 9 is a flowchart depicting a method for operating a neural network accelerator using a mixed precision data format that may be implemented in certain examples of the disclosed technology.

[0012] Fig.10 is a block diagram illustrating a suitable computing environment for implementing some embodiments of the disclosed technology. DETAILED DESCRIPTION

[0013] General considerations

[0014] The invention is described in the context of representative embodiments, which are not intended to be limiting in any way.

[0015] Unless the context clearly dictates otherwise, as used in this application, the singular forms "a", "an", and "the" include the plural forms. In addition, the term "includes" means "comprises". Furthermore, the term "coupled" includes mechanical, electrical, magnetic, optical, and other practical ways of coupling or linking items together, and does not exclude the presence of intermediate elements between the coupled items. Furthermore, as used herein, the term "and / or" means any one item or combination of items in the phrase.

[0016] The systems, methods, and devices described herein should not be construed as limiting in any way. Instead, the present disclosure relates to all novel and non-obvious features and aspects of the various disclosed embodiments, individually and in various combinations and sub-combinations with each other. The disclosed systems, methods, and devices are not limited to any particular aspect or feature or combination thereof, nor do the disclosed things and methods require the presence of any one or more specific advantages or problem-solving. In addition, any feature or aspect of the disclosed embodiments may be used in various combinations and sub-combinations with each other.

[0017] Although the operations of some disclosed methods are described in a specific sequential order for the convenience of presentation, it should be understood that this description includes rearrangement unless the specific language set forth below requires a specific order. For example, in some cases, the operations described in sequence can be rearranged or performed simultaneously. In addition, for the sake of simplicity, the accompanying drawings may not show the various ways in which the disclosed content and methods can be used in combination with other things and methods. In addition, this specification sometimes uses terms such as "produce", "generate", "display", "receive", "verify", "execute" and "initiate" to describe the disclosed methods. These terms are high-level descriptions of the actual operations performed. The actual operations corresponding to these terms will vary according to the specific order, and can be easily distinguished by those of ordinary skill in the art.

[0018] The theories of operation, scientific principles or other theoretical descriptions presented herein with reference to the devices or methods of the present disclosure have been provided to facilitate understanding and are not intended to limit the scope. The devices and methods in the appended claims are not limited to devices and methods that function in the manner described by such theories of operation.

[0019] Any of the methods disclosed may be implemented as computer-executable instructions stored on one or more computer-readable media (e.g., computer-readable media such as one or more optical media disks, volatile memory components (such as DRAM or SRAM), or non-volatile memory components (such as hard drives)) and executed on a computer (e.g., any commercial computer, including a smart phone or other mobile device that includes computing hardware). Any computer-executable instructions for implementing the disclosed techniques, as well as any data created and used during the implementation of the disclosed embodiments, may be stored on one or more computer-readable media (e.g., computer-readable storage media). The computer-executable instructions may be, for example, part of a dedicated software application or a software application accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software may be executed, for example, on a single local computer, or in a network environment using one or more network computers (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network), or other such network).

[0020] For clarity, only certain selected aspects of software-based implementations are described. Other details known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any particular computer language or program. For example, the disclosed technology can be implemented by software written in C, C++, Java, or any other suitable programming language. Similarly, the disclosed technology is not limited to any particular computer or hardware type. Certain details of suitable computers and hardware are known and need not be elaborated in this disclosure.

[0021] Furthermore, any software-based embodiment (including, for example, computer-executable instructions for causing a computer to perform any disclosed method) may be uploaded, downloaded, or remotely accessed via suitable communication means, including, for example, the Internet, the World Wide Web, an intranet, a software application, cable (including fiber optic cable), magnetic communication, electromagnetic communication (including RF, microwave, and infrared communication), electronic communication, or other such communication means.

[0022] Overview

[0023] Artificial neural networks (ANNs or "NNs" as used throughout this article) are used in many applications in artificial intelligence and machine learning, including image recognition, speech recognition, search engines, and other suitable applications. The processing of these applications can be performed on individual devices such as personal computers or cell phones, but can also be performed in large data centers. At the same time, hardware accelerators that can be used with NNs include dedicated NN processing units, such as tensor processing units (TPUs) and field programmable gate arrays (FPGAs) programmed to accelerate neural network processing. Due to their flexible nature and low power consumption per unit of computation, such hardware devices are being deployed in consumer devices as well as data centers.

[0024] Traditionally, NNs have been trained and deployed using single-precision floating point (32-bit floating point or float32 format). However, it has been shown that inference operations can be performed using lower-precision floating point formats such as 16-bit floating point (float16) or fixed point with minimal loss of accuracy. On dedicated hardware such as FPGAs, reduced-precision formats can greatly improve latency and throughput of DNN processing.

[0025] Numbers represented in a normal precision floating point format (e.g., floating point numbers represented in a 16-bit floating point format, a 32-bit floating point format, a 64-bit floating point format, or an 80-bit floating point format) can be converted to quantized precision format numbers, thereby allowing performance advantages to be obtained when performing operations. In particular, NN weights and activation values ​​can be represented in a lower precision quantized format that introduces an acceptable level of error. Examples of lower precision quantized formats include formats with reduced bit width (including by reducing the number of bits used to represent a number's mantissa or exponent) and block floating point formats in which two or more numbers share the same single exponent.

[0026] One of the characteristics of computation on FPGA devices is that it typically lacks hardware floating point support. Floating point operations can be performed with a penalty using flexible logic, but the amount of logic required to support floating point is often prohibitive in FPGA implementations. Some newer FPGAs have been developed that support floating point computations, but even on these FPGAs, the same device will produce twice as much computational output per unit time when used in integer mode. Typically, NNs are created with floating point computations in mind, but when FPGAs are used for NN processing, it would be beneficial if the neural network could be represented using integer arithmetic. Examples of the disclosed technology include hardware implementations of block floating point (BFP), including use of BFP in NN, FPGA, and other hardware environments.

[0027] A typical floating point representation in a computer system consists of three parts: the sign (s), the exponent (e), and the mantissa (m). The sign indicates whether the number is positive or negative. The exponent and mantissa are used in scientific notation:

[0028]

[0029] Within the limits of the precision of the mantissa, any number can be represented. Since the exponent scales the mantissa by powers of 2, just as the exponent scales by powers of 10 in scientific notation, very large numbers can be represented. The precision of the representation is determined by the precision of the mantissa. Typical floating point representations use a mantissa that is 10 (float16), 24 (float32), or 53 (float64) bits wide. 53 An integer of 2 can be approximated using the float64 floating point format, but cannot be represented exactly because there are not enough bits in the mantissa. A similar effect can occur with arbitrary fractions where the fraction is represented by the bits in the mantissa as a negative power of 2. There are many fractions that cannot be represented exactly because they are irrational numbers in the binary number system. In both cases more exact representations are possible, but they may require more bits in the mantissa. Ultimately, an infinite number of mantissa bits are required to represent some numbers exactly (e.g., ). The 10-bit (half-precision floating point), 24-bit (single-precision floating point), and 53-bit (double-precision floating point) mantissa limits are common tradeoffs between mantissa storage requirements and representation accuracy in general-purpose computers.

[0030] For block floating point format, a set of two or more numbers uses a single shared exponent, and each number still has its own sign and mantissa. In some examples, the shared exponent is selected to be the largest exponent of the original floating point value. For the purposes of this disclosure, the term block floating point (BFP) refers to a digital system in which a single exponent is shared between two or more values, each value represented by a sign and mantissa pair (whether or not there is an explicit sign bit, or whether the mantissa itself is signed). In some examples, all values ​​of one or more rows or columns of a matrix or vector, or all values ​​of a matrix or vector, can share a common exponent. In other examples, the BFP representation can be unsigned. In some examples, some but not all elements in a matrix or vector BFP representation may include numbers represented as integers, floating point numbers, fixed point numbers, symbols, or other data formats mixed with numbers represented with signs, mantissas, and exponents. In some examples, some or all elements in a matrix or vector BFP representation may include complex elements with two or more parts, for example: a complex number with an imaginary part (a+bi, where ); fractions in polar coordinates (r, θ) or other multi-component elements, including numerators and denominators.

[0031] The BFP format can be used to balance precision and storage requirements in a manner similar to ordinary floating points in some respects. First, a group of numbers can share the same exponent, rather than storing the exponent for each floating point number. In order to share the exponent while maintaining high precision, the numbers should have nearly the same size, because the difference in size is represented in the mantissa. If the difference in size is too large, the mantissa of the large value will overflow, or the mantissa of the small value can be zero ("underflow"). Depending on the specific application, a certain amount of overflow and / or underflow may be acceptable.

[0032] The size of the mantissa can be adjusted to suit a particular application. This can affect the precision of the number represented, but potential gains are achieved by reducing the size of the representation. For example, a normal single-precision floating point is four bytes in size, but for some implementations of the disclosed technology, only two bytes are used to represent the sign and mantissa of each value. In some implementations, the sign and mantissa of each value can be represented with one byte or less.

[0033] In some examples of the disclosed technology, the representation expressed above is used to derive the original number from the representation, but only a single exponent is stored for a set of numbers, where each number is represented by a signed mantissa. Each signed mantissa can be represented with two bytes or less, thus saving about 2 times the memory storage compared to a four-byte floating point. In addition, the memory bandwidth requirements for loading and storing these values ​​are also about half that of ordinary floating points.

[0034] Neural network operations are used in many artificial intelligence operations. Typically, most of the processing operations performed when implementing a neural network are to perform matrix × matrix or matrix × vector multiplication or convolution operations. Such operations are computationally and memory bandwidth intensive, where the size of the matrix can be, for example, 1000 × 1000 elements (e.g., 1000 × 1000 numbers, each number including a sign, a mantissa, and an exponent) or larger, and many matrices are used. As discussed herein, BFP techniques can be applied to such operations to reduce computational requirements and memory bandwidth in a given system, whether the system is an FPGA, a CPU, or other hardware platform. As used herein, the use of the term "element" herein refers to a member of such a matrix or vector.

[0035] As used herein, the term "tensor" refers to a multidimensional array that can be used to represent the characteristics of a NN and includes one-dimensional vectors and two-dimensional, three-dimensional, four-dimensional or higher-dimensional matrices. As used in this disclosure, unless otherwise specified, a tensor does not require any other mathematical properties.

[0036] As used herein, the term "normal precision floating point" refers to a floating point format having a mantissa, an exponent, and an optional sign and supported by a native or virtual CPU. Examples of normal precision floating point formats include, but are not limited to, IEEE 754 standard formats (such as 16 bits, 32 bits, 64 bits) or other processors supported by the processor (such as Intel AVX, AVX2, IA32, and x86_64 80-bit floating point formats).

[0037] A given number can be represented using different precision (e.g., mixed precision) formats. For example, a number can be represented by a higher precision format (e.g., float32) and a lower precision format (e.g., float16). Reducing the precision of a number can include reducing the number of bits used to represent the mantissa or exponent of a number. In addition, reducing the precision of a number can include narrowing the range of values ​​that can be used to represent a digital exponent, such as when multiple numbers share a common exponent. Similarly, increasing the precision of a number can include increasing the number of bits used to represent the mantissa or exponent of a number. In addition, increasing the precision of a number can include increasing the range of values ​​that can be used to represent the exponent of a number, such as when a number is separated from a group of numbers sharing a common exponent. As used herein, converting a number from a higher precision format to a lower precision format can be referred to as down-converting or quantizing the number. Converting a number from a lower precision format to a higher precision format can be referred to as up-converting or dequantizing the number.

[0038] As used herein, the term "quantized precision floating point" refers to a floating point format in which two or more values ​​of a tensor have been modified to have lower precision than when the values ​​are represented in normal precision floating point. Specifically, many examples of quantized precision floating point representations include block floating point formats, in which two or more values ​​of a tensor are represented with reference to a common index. Quantized precision floating point numbers can be generated by selecting a common index for two, more, or all elements of a tensor and shifting the mantissas of the individual elements to match the shared common index. In some examples, groups of elements within a tensor can share a common index, for example, on a per-row, per-column, per-block, or other basis.

[0039] In one example of the disclosed technology, a neural network accelerator is configured to accelerate a given layer of a multi-layer neural network using a mixed precision data format. For example, the mixed precision data format may include a normal precision floating point format and a quantized precision floating point format. The input tensor of a given layer may be converted from a normal precision floating point format to a quantized precision floating point format. Tensor operations may be performed using the converted input tensors. The result of the tensor operation may be converted from a block floating point format to a normal precision floating point format. The converted result may be used to generate an output tensor of the layer of the neural network, wherein the output tensor is in a normal precision floating point format. In this way, the neural network accelerator may become smaller and more efficient compared to similar accelerators that use only normal precision floating point formats. Smaller and more efficient accelerators may have higher computing performance and / or higher energy efficiency. In addition, the neural network accelerator may have higher accuracy compared to accelerators that use only quantized precision floating point formats. By improving the accuracy of the accelerator, the convergence time of the training may be reduced and the accelerator may be more accurate when classifying the input of the neural network.

[0040] Example architecture for training neural networks using mixed precision floating point

[0041] Figure 1 1 is a block diagram 100 outlining an exemplary quantization-enabled system 110 in which certain examples of the disclosed techniques may be implemented. Figure 1 As shown, the quantization-enabled system 110 may include multiple hardware resources, including a general-purpose processor 120 and a special-purpose processor, such as a graphics processing unit 122 and a neural network accelerator 180. The processor is coupled to a memory 125 and a storage device 127, which may include volatile or non-volatile memory devices. The processors 120 and 122 execute instructions stored in the memory or storage device to provide a neural network module 130. The neural network module 130 includes a software interface that allows the system to be programmed to implement various types of neural networks. For example, software functions that allow an application to define a neural network, including weights, biases, activation functions, node values, and interconnects between layers of the neural network, may be provided. In addition, the software functions may be used to define state elements of a recurrent neural network. The neural network module 130 may also provide utilities that allow training and retraining of a neural network implemented using the module. Values ​​representing the neural network module are stored in the memory or storage device and are operated on by instructions executed by one of the processors. The values ​​stored in the memory or storage device may be represented using normal precision floating point and / or quantized floating point values.

[0042] In some examples, a proprietary or open source library or framework is provided to the programmer to enable neural network creation, training, and evaluation. Examples of such libraries include TensorFlow, Microsoft Cognitive Toolkit (CNTK), Caffe, Theano, and Keras. In some examples, programming tools such as integrated development environments provide support for programmers and users to define, compile, and evaluate NNs.

[0043] The neural network accelerator 180 may be implemented as a custom or application specific integrated circuit (e.g., including a system on chip (SoC) integrated circuit), a field programmable gate array (FPGA), or other reconfigurable logic, or as a soft processor virtual machine hosted by a physical general purpose processor. The neural network accelerator 180 may include a tensor processing unit 182, a reconfigurable logic device 184, and / or one or more neural processing cores (such as a subgraph accelerator 186). The subgraph accelerator 186 may be configured using hardware, software, or a combination of hardware and software. As an example, the subgraph accelerator 186 may be configured and / or executed using instructions executable on the tensor processing unit 182. As another example, the subgraph accelerator 186 may be configured by programming the reconfigurable logic block 184. As another example, the subgraph accelerator 186 may be configured using hardwired logic gates of the neural network accelerator 180.

[0044] The subgraph accelerator 186 can be programmed to execute a subgraph or an individual node of a neural network. For example, the subgraph accelerator 186 can be programmed to execute a subgraph that includes a layer of the NN. The subgraph accelerator 186 can access local memory for storing weights, biases, input values, output values, etc. The subgraph accelerator 186 can have many inputs, each of which can be weighted by a different weight value. For example, the subgraph accelerator 186 can generate a dot product of an input tensor and the programmed input weights of the subgraph accelerator 186. In some examples, the dot product can be adjusted with a bias value before it is used as an input to an activation function. For example, the output of the subgraph accelerator 186 can be stored in a local memory, where the output value can be accessed and sent to different NN processor cores and / or neural network modules 130 or memory 125.

[0045] The neural network accelerator 180 may include a plurality 110 of subgraph accelerators 186 connected to each other via an interconnect (not shown). The interconnect may carry data and control signals between individual subgraph accelerators 186, memory interfaces (not shown), and input / output (I / O) interfaces (not shown). The interconnect may transmit and receive signals using electrical, optical, magnetic, or other suitable communication technologies, and may provide communication connections arranged according to a plurality of different topologies depending on a particular desired configuration. For example, the interconnect may have a crossbar switch, a bus, a point-to-point bus, or other suitable topology. In some examples, any of the plurality of subgraph accelerators 186 may be connected to any other core, while in other examples, some cores are only connected to a subset of the other cores. For example, each core may only be connected to the nearest 4, 8, or 10 neighboring cores. The interconnect may be used to transmit input to and / or output data from the subgraph accelerator 186, as well as to transmit control signals and other information signals to and from the subgraph accelerator 186. For example, each subgraph accelerator 186 may receive and transmit semaphores indicating the execution status of operations currently being performed by each of the corresponding subgraph accelerators 186. In addition, matrix and vector values ​​may be shared between subgraph accelerators 186 via interconnects. In some examples, the interconnects are implemented as lines connecting the subgraph accelerators 186 and a memory system, while in other examples, the core interconnect may include circuitry, switches, and / or routing components for multiplexing data signals on (one or more) interconnect lines, including active signal drivers and repeaters, or other suitable circuitry. In some examples of the disclosed technology, signals transmitted within and to / from the neural network accelerator 180 are not limited to full-swing electrical digital signals, but the neural network accelerator 180 may be configured to include differential signals, pulse signals, or other suitable signals for transmitting data and control signals.

[0046] In some examples, quantization-enabled system 110 may include an optional quantization emulator that emulates the functionality of neural network accelerator 180. Neural network accelerator 180 provides functionality that can be used to convert data represented in full precision floating point format in neural network module 130 to quantized format values. Neural network accelerator 180 can also perform operations using quantized format values. Such functionality will be discussed in more detail below.

[0047] The neural network module 130 can be used to specify, train, and evaluate a neural network model using a tool flow that includes a hardware-independent modeling framework 131 (also referred to as a native framework or machine learning execution engine), a neural network compiler 132, and a neural network runtime environment 133. The memory includes computer executable instructions for the tool flow, which includes the modeling framework 131, the neural network compiler 132, and the neural network runtime environment 133. The tool flow can be used to generate neural network data 200, which represents all or part of a neural network model, such as the following about Figure 2 Neural network model discussed. It should be noted that although the tool flow is described as having three separate tools (131, 132, and 133), in various examples, the tool flow can have fewer or more tools. For example, the functionality of different tools (131, 132, and 133) can be combined into a single modeling and execution environment.

[0048] Neural network data 200 may be stored in memory 125. Neural network data 200 may be represented in one or more formats. For example, neural network data 200 corresponding to a given neural network model may have a different format associated with each corresponding tool of a tool flow. Typically, neural network data 200 may include descriptions of nodes, edges, groups, weights, biases, activation functions, and / or tensor values. As a specific example, neural network data 200 may include source code, executable code, metadata, configuration data, data structures, and / or files for representing a neural network model.

[0049] The modeling framework 131 can be used to define and use neural network models. As an example, the modeling framework 131 may include predefined APIs and / or programming primitives that can be used to specify one or more aspects of a neural network model. The predefined APIs may include both lower-level APIs (e.g., activation functions, cost or error functions, nodes, edges, and tensors) and higher-level APIs (e.g., layers, convolutional neural networks, recurrent neural networks, linear classifiers, etc.). "Source code" can be used as input to the modeling framework 131 to define the topology of the graph of a given neural network model. In particular, the APIs of the modeling framework 131 can be instantiated and interconnected within the source code to specify complex neural network models. Data scientists can create different neural network models by using different APIs, different numbers of APIs, and interconnecting the APIs in different ways.

[0050] In addition to the source code, the memory 125 can also store training data. The training data includes a set of input data for application to the neural network model 200 and the expected output from the neural network model for each corresponding data set of the input data. The modeling framework 131 can be used to train the neural network model using the training data. The output of the training is the weights and biases associated with each node of the neural network model. After the neural network model is trained, the modeling framework 131 can be used to classify new data applied to the trained neural network model. Specifically, the trained neural network model uses the weights and biases obtained by the training to perform classification and recognition tasks on data that is not used to train the neural network model. The modeling framework 131 can use the CPU 120 and a dedicated processor (e.g., GPU 122 and / or neural network accelerator 180) to execute the neural network model with improved performance compared to using only the CPU 120. In some examples, the performance can potentially achieve real-time performance for certain classification tasks.

[0051] Compiler 132 analyzes source code and data provided for a neural network model (e.g., examples for training the model) and converts the model into a format that can be accelerated on neural network accelerator 180, as will be described in further detail below. Specifically, compiler 132 converts source code into executable code, metadata, configuration data, and / or data structures to represent the neural network model and memory as neural network data 200. In some examples, compiler 132 may divide the neural network model into a portion (e.g., neural network 200) that uses CPU 120 and / or GPU 122 and other portions (e.g., neural network subgraphs) that can be executed on neural network accelerator 180. Compiler 132 may generate executable code (e.g., a runtime module) to execute a subgraph assigned to CPU 120 and communicate with a subgraph assigned to accelerator 180. Compiler 132 may generate configuration data for accelerator 180 that is used to configure accelerator resources to evaluate a subgraph assigned to optional accelerator 180. Compiler 132 may create data structures for storing values ​​generated by the neural network model during execution and / or training, and for communication between CPU 120 and accelerator 180. Compiler 132 may generate metadata that may be used to identify subgraphs, edge groupings, training data, and various other information about the neural network model during runtime. For example, metadata may include information for interfacing between different subgraphs of the neural network model.

[0052] The runtime environment 133 provides an executable environment or interpreter that can be used to train the neural network model during training mode and can be used to evaluate the neural network model in training, reasoning or classification mode. During reasoning mode, input data can be applied to the neural network model input and the input data can be classified according to the training of the neural network model. The input data can be archived data or real-time data.

[0053] The runtime environment 133 may include a deployment tool that can be used to deploy or install all or part of a neural network to the neural network accelerator 180 during a deployment mode. The runtime environment 133 may also include a scheduler that manages the execution of different runtime modules and the communication between the runtime modules and the neural network accelerator 180. Thus, the runtime environment 133 may be used to control the data flow between the nodes modeled on the neural network module 130 and the neural network accelerator 180.

[0054] In one example, the neural network accelerator 180 receives and returns the normal precision value 150 from the neural network module 130. Figure 1 As shown, the subgraph accelerator 186 can use quantized floating point to perform most of its operations, and the interface between the subgraph accelerator 186 and the neural network module 130 can use full precision values ​​to communicate information between modules. Ordinary precision values ​​can be represented in 16-bit, 32-bit, 64-bit or other suitable floating point formats. For example, a value representing a portion of a neural network can be received, including edge weights, activation values ​​or other suitable quantization parameters. Ordinary precision values ​​150 are provided to ordinary precision floating point to quantized floating point converter 152, which converts the ordinary precision values ​​to quantized values. Then, quantized floating point operations 154 can be performed on the quantized values. Then, the quantized values ​​can be converted back to ordinary floating point format using a quantized floating point to ordinary floating point converter, which transmits ordinary precision floating point values. As a specific example, the subgraph accelerator 186 can be used to accelerate a given layer of a neural network, and vector-vector, matrix-vector, matrix-matrix and convolution operations can be performed using quantized floating point operations and less computationally intensive operations (such as adding bias values ​​or calculating activation functions) can be performed using ordinary floating point operations.

[0055] The conversion between normal floating point and quantized floating point performed by converters 152 and 156 is typically performed on sets of numbers represented as vectors or multi-dimensional matrices. In some examples, additional normal precision operations 158 (including operations that may be required in a particular neural network implementation) may be performed based on the normal precision format, including adding a bias to one or more nodes of the neural network, applying a hyperbolic tangent function or other such sigmoid function, or a rectification function (e.g., a ReLU operation) to the normal precision values ​​converted back from the quantized floating point format.

[0056] In some examples, quantized values ​​are only used and stored in the logic gates and internal memory of the neural network accelerator 180, and the memory 125 and the storage device 127 store only ordinary floating point values. For example, the neural network accelerator 180 can quantize the inputs, weights, and activations of the neural network model received from the neural network model 130, and can dequantize the results of the operations performed on the neural network accelerator 180 before passing the values ​​back to the neural network model 130. The memory 125, the storage device 127, or the input / output interface (not shown) can be used to pass values ​​between the neural network model 130 and the neural network accelerator 180. In other examples, the simulator provides a full simulation of quantization, including storing only one copy of the shared exponent and operating with a reduced mantissa width. In the version that performs basic operations in ordinary floating point, some results may be different. For example, the full simulation version can check for underflow or overflow conditions for limited quantization bit widths (e.g., 3, 4, or 5 bit wide mantissas).

[0057] Most of the computational cost of a DNN is vector-vector, matrix-vector, and matrix-matrix multiplications and / or convolutions. These operations are quadratic in the input size, while operations such as bias addition and activation functions are linear in the input size. Therefore, in some examples, quantization is only applied to matrix-vector multiplication operations, which are implemented on the neural network accelerator 180. In such examples, all other operations are performed in a normal precision format, such as float16. Therefore, from the user or programmer's perspective, the quantization-enabled system 110 accepts and outputs normal precision float16 values ​​from / to the neural network module 130 and outputs float16 format values. All conversions to and from block floating point formats can be hidden from the programmer or user. In some examples, the programmer or user can specify certain parameters for the quantization operation. In other examples, the quantization operation can utilize the block floating point format to reduce computational complexity, as described below with respect to Figure 3 discussed.

[0058] Neural network accelerator 180 is used to accelerate the evaluation and / or training of a neural network graph or subgraph, typically with increased speed and reduced latency that would not be achieved when evaluating the subgraph using only CPU 120 and / or GPU 122. In the example shown, the accelerator includes a tensor processing unit (TPU) 182, a reconfigurable logic device 184 (e.g., contained in one or more FPGAs or programmable circuit structures), and / or a subgraph accelerator 186, but any suitable hardware accelerator may be used to simulate a neural network. Accelerator 180 may include configuration logic that provides a soft CPU. The soft CPU supervises the operation of the acceleration graph or subgraph on accelerator 180 and may manage communications with neural network module 130. The soft CPU may also be used to configure logic and control the loading and storage of data from RAM on the accelerator, such as in block RAM in an FPGA.

[0059] In some examples, parameters of neural network accelerator 180 may be programmable. Neural network accelerator 180 may be used to prototype training, inference, or classification of all or part of neural network model 200. For example, quantization parameters may be selected based on accuracy or performance results obtained by prototyping the network within neural network accelerator 180. After a set of desired quantization parameters are selected, the quantization model may be programmed into accelerator 180 to perform other operations.

[0060] The compiler 132 and runtime 133 provide a fast interface between the neural network module 130 and the neural network accelerator 180. In practice, the user of the neural network model may not be aware that a portion of the model is being accelerated on the provided accelerator. For example, node values ​​are typically propagated in the model by writing tensor values ​​to a data structure that includes an identifier. The runtime 133 associates a subgraph identifier with an accelerator and provides the accelerator with logic for converting messages to write values ​​of weights, biases, and / or tensors transparently to the neural network accelerator 180 without program intervention. Similarly, values ​​output by the neural network accelerator 180 can be transparently sent back to the neural network module 130 along with a message that includes an identifier of the receiving node at the server and a payload that includes values ​​such as weights, biases, and / or tensors that are sent back to the entire neural network model.

[0061] Figure 2A simplified topology of a deep neural network (DNN) 200 that can be used to perform enhanced image processing using the disclosed BFP implementation is shown. One or more processing layers can be implemented using the disclosed techniques for quantization and BFP matrix / vector operations, including using one or more of the multiple neural network subgraph accelerators 186 in the quantization-enabled system 110 described above. It should be noted that the application of the neural network implementations disclosed herein is not limited to DNNs, but can also be used with other types of neural networks, such as convolutional neural networks (CNNs) (including implementations with long short-term memory (LSTM) or gated recurrent units (GRUs)) or other suitable artificial neural networks that can be adapted for use with the BFP methods and apparatus disclosed herein.

[0062] DNN 200 can operate in at least two different modes. Initially, DNN 200 can be trained in training mode and then used as a classifier in inference mode. During training mode, a set of training data can be applied to the input of DNN 200, and various parameters of DNN 200 can be adjusted so that when training is completed, DNN 200 can be used as a classifier. Training includes performing forward propagation of training input data, calculating losses (e.g., determining the difference between the output of DNN and the expected output of DNN), and performing back propagation through DNN to adjust the parameters of DNN 200 (e.g., weights and biases). When the architecture of DNN 200 is suitable for classifying training data, the parameters of DNN 200 will converge and training can be completed. After training, DNN 200 can be used in inference mode. Specifically, training or non-training data can be applied to the input of DNN 200 and forward propagated through DNN 200, so that the input data can be classified by DNN 200.

[0063] like Figure 2 As shown, a first node set 210 (including nodes 215 and 216) forms an input layer. Each node in set 210 is connected to each node in a first hidden layer formed by a second node set 220 (including nodes 225 and 226). The second hidden layer is formed by a third node set 230 including node 235. The output layer is formed by a fourth node set 240 (including node 245). In example 200, the nodes of a given layer are fully interconnected with the nodes of its (one or more) adjacent layers. In other words, a layer may include nodes that have common inputs with other nodes of the layer and / or provide outputs to a common destination of other nodes of the layer. In other examples, a layer may include a subset of nodes that have common inputs with other nodes of the layer and / or provide outputs to a subset of common destinations of other nodes of the layer.

[0064] During forward propagation, each of the nodes generates an output value by applying a weight to each input generated from the previous node and collecting the weights to generate an output value. In some examples, each individual node may apply an activation function (σ) and / or a bias (b). In general, a suitably programmed processor or FPGA may be configured to implement the nodes in the depicted neural network 200. In some example neural networks, the output function f(n) of the hidden combined node n may generate an output mathematically expressed as follows:

[0065]

[0066] where w i is applied (multiplied) to the input edge x i , b is the bias value of node n, σ is the activation function of node n, and E is the number of input edges of node n. In some examples, the activation function produces a continuous value (represented as a floating point number) between 0 and 1. In some examples, the activation function produces a binary 1 or 0 value, depending on whether the sum is above or below a threshold.

[0067] A given neural network may include thousands of individual nodes, so it may be computationally expensive to perform all of the calculations for the nodes in normal precision floating point. Implementations of computationally more expensive solutions may include hardware that is larger and consumes more energy than implementations of computationally less expensive solutions. However, performing operations using quantized floating point or mixed precision (using both normal precision floating point and quantized floating point) may reduce the computational complexity of the neural network. A simple implementation using only quantized floating point can significantly reduce computational complexity, but the implementation may have difficulty converging during training and / or correctly classifying input data due to errors introduced by quantization. However, a mixed precision implementation can potentially improve the accuracy of certain calculations while also providing the benefit of reducing the complexity associated with quantized floating point.

[0068] A mixed precision implementation of DNN 200 may include nodes that perform operations in both normal precision floating point and quantized floating point. As a specific example, the output function f(n) of a hidden combination node n may produce an output mathematically expressed as follows:

[0069]

[0070] where w i is applied (multiplied) to the input edge x i The weight of Q(w i ) is the quantized floating point value of the weight, Q(x i ) is derived from the input edge x i The quantized floating point value of the input, Q -1() is the dequantized representation of the quantized floating point value of the dot product of vectors w and x, b is the bias value of node n, σ is the activation function of node n, and E is the number of input edges of node n. Performing the dot product using quantized floating point values ​​can potentially reduce the computational complexity (compared to using only normal precision floating point values), and other operations on the output function using normal precision floating point values ​​can potentially increase the accuracy of the output function (compared to using only quantized floating point values).

[0071] A neural network can be trained and retrained by adjusting the values ​​of the components of the output function f(n). For example, by adjusting the weights w of the nodes i Or bias value b, the behavior of the neural network is adjusted by the corresponding change of the network output tensor value. For example, the cost function C(w, b) can be used during backpropagation to find suitable weights and biases for the network, where the cost function can be mathematically described as:

[0072]

[0073] Where w and b represent all weights and biases, n is the number of training inputs, and a is the vector of output values ​​from the network for the input vector of training inputs x. By adjusting the network weights and biases, the cost function C can be driven to a target value (e.g., zero (0)) using various search techniques (e.g., stochastic gradient descent). When the cost function C is driven to the target value, the neural network is said to converge. Similar to the output function f(n), the cost function can be implemented using mixed precision computer algorithms. For example, vector operations can be performed using quantized floating point values ​​and operations, and non-vector operations can be performed using normal precision floating point values.

[0074] Examples of suitable applications for such neural network BFP implementations include, but are not limited to: performing image recognition, performing speech recognition, classifying images, converting speech to text and / or other languages, facial or other biometric recognition, natural language processing, automatic language conversion, query processing in search engines, automatic content selection, analyzing emails and other electronic documents, relationship management, biomedical informatics, identifying candidate biomolecules, providing recommendations, or other classification and artificial intelligence tasks.

[0075] Network accelerators (such as Figure 1The network accelerator 180 in ( ) can be used to accelerate the computation of DNN200. As an example, DNN 200 can be divided into different subgraphs that can be individually accelerated. As a specific example, each of layers 210, 220, 230 and 240 can be a subgraph that is accelerated. The computations of the layer with higher computational cost can be performed using quantized floating point, and the computations of the layer with lower computational cost can be performed using normal precision floating point. Values ​​can be passed from one layer to another using normal precision floating point. By accelerating a set of computations for all nodes within a layer, certain computations can be reused and the computations performed by the layer can be reduced compared to accelerating individual nodes.

[0076] In some examples, a set of parallel multiply-accumulate (MAC) units in each convolutional layer can be used to speed up computation. Similarly, parallel multiplier units can be used in the fully connected and dense matrix multiplication stages. A set of parallel classifiers can also be used. This parallelization approach has the potential to further speed up computation at the expense of increased control complexity.

[0077] Those of ordinary skill in the art who have the benefit of this disclosure will readily appreciate that the application of the neural network implementation can be used for different aspects of using neural networks, whether alone or in combination or sub-combination with other neural networks. For example, the disclosed implementations can be used to implement neural network training via gradient descent and / or back-propagation operations of the neural network. In addition, the disclosed implementations can be used to evaluate neural networks.

[0078] Figure 3 300 is a diagram illustrating an example of converting a normal floating point format to a quantized block floating point format as may be used in certain examples of the disclosed technology. For example, an input tensor of a neural network represented as a normal floating point number (e.g., in a 32-bit or 16-bit floating point format) may be converted to the illustrated block floating point format.

[0079] As shown, many common floating point format numbers 310 are represented so that each number (e.g., number 315 or number 316) includes a sign, an exponent, and a mantissa. For example, for the IEEE 754 half-precision floating point format, the sign is represented by 1 bit, the exponent is represented by 5 bits, and the mantissa is represented by 10 bits. When the floating point format numbers 310 in the neural network model 200 are converted to a set of quantized precision block floating point format numbers, there is an exponent value shared by all the numbers in the set shown. Therefore, as shown, the set of block floating point numbers 320 is represented by a single exponent value 330, and each of the set of numbers includes a sign and a mantissa. However, since the set of numbers shown has different exponent values ​​in floating point format, the corresponding mantissa of each number can be shifted so that the same or similar numbers are represented in quantized format (e.g., shifted mantissas 345 and 346).

[0080] Furthermore, if Figure 3 As shown, using block floating point format can reduce the computing resources required for some common operations. In the example shown, the dot product of two floating point vectors is shown in formal floating point format (350) and block floating point format (360). For the numbers represented by ordinary precision floating point format operation 350, floating point addition is required to perform the dot product operation. In the dot product of floating point vectors, the summation is performed in floating point, which may require shifting to align the values ​​with different exponents. On the other hand, for block floating point dot product operation 360, integer arithmetic can be used to calculate the product to combine the mantissa elements shown. In other words, since the exponent part can be taken into account in block floating point representation, the multiplication and addition of the mantissa can be performed completely with fixed point or integer representation. Therefore, a shared exponent can be used to maintain a larger dynamic range of a set of digits while reducing the computational cost by using more integer arithmetic rather than floating point arithmetic. In some examples, the operations performed by the quantization simulator 140 can be optimized to utilize the block floating point format.

[0081] In some examples, shared exponent 330 is selected as the largest exponent among the original ordinary precision numbers in neural network model 200. In other examples, the shared exponent can be selected in a different manner, for example, by selecting an exponent that is the average or median of ordinary floating point exponents, or by selecting an exponent to maximize the dynamic range of the value stored in the mantissa when the digits of the value stored in the mantissa are converted to a quantized number format. It should be noted that if the shared exponent is different from the original floating point exponent of the value, some bits of the quantized mantissa may be lost. This occurs because the mantissa is shifted to correspond to the new shared exponent.

[0082] There are several possible choices about which values ​​in a block floating-point tensor will share an exponent. The simplest choice is to have the entire matrix or vector share an exponent. However, sharing the exponent at a finer granularity can reduce errors because it increases the likelihood that BFP numbers use a shared exponent that is closer to their original normal floating-point format exponent. Thus, the loss of precision due to discarding mantissa bits (when shifting the mantissa to correspond to the shared exponent) can be reduced.

[0083] For example, consider multiplying a row vector x by a matrix W: y = xW. If each column of W shares exponents, then each dot product xW j (W j is the jth column of W) involves only one shared index of x and W j A shared index.

[0084] Figure 4 and Figure 5An alternative block floating point format that can be used for neural network calculations is shown. In the context of neural networks, a core operation is to perform dot products. For example, dot products are the core calculations of matrix multiplication and convolution. Matrix multiplication involves the dot product of the rows / columns of a matrix with an input vector. Convolution involves the dot product of a filter with an input window. In the context of quantized floating point, the set of values ​​selected to share the exponent can have an impact on the complexity of the computer arithmetic logic used to calculate the dot product. The values ​​of the shared exponent can be referred to as the values ​​within the bounding box. The shape of the bounding box can affect quantization error and computational cost. While clustering similar amplitude values ​​to create a bounding box can reduce quantization error, tracking scaling factors for arbitrary bounding box shapes can be expensive. In contrast, matrices and filters can be divided into bounding boxes, which can be more efficient for operations performed by the neural network. In particular, the appropriate selection of bounding boxes can reduce the complexity of computer arithmetic circuits used to implement the operation of the neural network. Figure 4 shows that a block floating point format may be well suited for matrices, Figure 5 The block floating point format is shown to be well suited for convolution filters.

[0085] Figure 4 4 is a diagram 400 illustrating four alternative block floating point formats as may be used in certain examples of the disclosed technology. As shown, a first format 410 represents an entire array of values ​​420 sharing a single exponent 425. In other words, the entire array of values ​​420 is packed within a single bounding box.

[0086] In the second format 430, common indices are shared on a per-column basis. In other words, the columns of the matrix are bounding boxes of values. Thus, in this specific example, the block floating point values ​​stored in the even columns 431 of the matrix each share a first single index 432. The block floating point values ​​stored in the odd columns 435 each share a second single index 437. In other examples, each column of the array may be associated with a different shared index. For an 11-column tile of the alternative format, there may be 11 corresponding shared indices, one for each column. In other examples, each row of the array may be associated with a different shared index, or odd and even rows may be associated with a shared common index.

[0087] A third format 450 is shown in which groups of elements in an array share common indices. For example, if a 15×15 value matrix shares indices according to the third format 450, a first group of 5×5 element groups 455 and 456 share a single shared index 458. Similarly, a second group of 5×5 element groups in arrays 460 and 461 may each share a second single index 468. In other examples, each tile may be associated with its corresponding shared index. In the example format 450, a 15×15 matrix may have nine shared indices.

[0088] A fourth format 470 is shown in which two shared indices are shared on a tile plus per column basis. Thus, a first group of numbers including numbers 480, 481, and 485 all share a single common index 488. Similarly, a second group of numbers including groups 490 and 491 each share a different second single index 495. In an alternative example, each group shown may have its own shared index.

[0089] In some examples, the computational cost of matrix-vector multiplication can be further reduced by reducing the mantissa width. A large range of values ​​that share a common exponent can be represented with only a few bits of mantissa. For example, in a representation with a 4-bit mantissa and a 5-bit exponent, the value in [2 -14 0.0012,2 15 1.1112] or approximately [2 -17 , 2 16 ]. In contrast, 4-bit fixed-point numbers can only represent values ​​in the range [00012, 11112] or approximately [2 0 , 2 4 ] is a value in the range of ].

[0090] Figure 5 5 is a diagram 500 showing three alternative block floating point formats as may be used in certain examples of the disclosed technology. These formats may be useful for two-dimensional convolutions, but these formats may also be generalized to higher dimensional convolutions. As shown, a first format 510 represents an entire convolution filter 512 of values ​​sharing a single exponent 514. Different convolution filters 516 of values ​​may share a single exponent 518. Thus, the format 510 shows that the entire convolution filter may be a bounding box of values.

[0091] In the second format 520, each spatial pixel may be a bounding box such that a common index is shared on a per spatial pixel basis along the channel dimension. As shown, spatial pixel values ​​522 share a single index 524, and spatial pixel values ​​526 share a single index 528. For example, for a pixel with dimensions [Y, y, c i ], each spatial dimension x and y can be defined with c i Similarly, for a bounding box with dimensions [f x , f y , c i ]c o convolutional filters, c o Each pixel (f x , f y ) can be a i The bounding box size for this method is c i .

[0092] In the third format 530, each spatial pixel can be subdivided along the channel dimension so that the bounding box includes the subdivisions of the spatial pixel. As shown, the subdivided spatial pixel values ​​532 share a single index 534 and the subdivided spatial pixel values ​​536 share a single index 538. For smaller c i , the cost of handling scaling factors can be significant. For example, an input image at the first layer of a deep convolutional neural network may have c corresponding to 3 color channels. i =3. Tracking the scaling factor for each triplet may be expensive. In this case, the convolution can be reshaped into a matrix-matrix multiplication to increase the bounding box and reduce the overhead of tracking the bounding box. For example, each convolution filter can be flattened to create a matrix with co columns and f x , *f y , *c i The input matrix X can be created where each row is f corresponding to the input window swept by the convolution filter. x , *f y , c i vector. The result Y = XW is a matrix that can be reshaped to match the output of the convolution operation. By reformulating the convolution as a matrix multiplication, the above reference to matrix multiplication can be applied. Figure 4 Bounding box strategy for discussion.

[0093] Figure 6 6 is a flow chart depicting a method 600 for training a neural network using a quantized model as may be implemented in certain examples of the disclosed technology. For example, training the neural network may include iterating through a set of training data, where the method 600 is used to update parameters of the neural network during a given iteration of the training data. As an example, the method 600 may be performed by a quantization-enabled system, such as Figure 1 A quantization-enabled system 110 .

[0094] At process block 610, parameters of the neural network, such as weights and biases, may be initialized. As one example, the weights and biases may be initialized to random normal precision floating point values. As another example, the weights and biases may be initialized to normal precision floating point values ​​calculated from an earlier training set. The initial parameters may be stored in a memory or storage device of a quantization-enabled system. In one example, the parameters may be stored as quantized floating point values, which may reduce the amount of storage used to store the initial parameters.

[0095] At process block 620, the input values ​​of the neural network may be propagated forward through the neural network. The input values ​​of a given layer of the neural network may be the outputs of another layer of the neural network. These values ​​may be passed between layers from the outputs of one layer to the inputs of the next layer using normal precision floating points. The output function of the i-th layer may include terms that are mathematically described as follows:

[0096] y i Q -1 (f(Q(y i-l ), Q(W i )))

[0097] where y i-1 is the output from the layer that provides input to the i-th layer, W i is the weight tensor of the i-th layer, f() is the forward function of the layer, Q() is the quantization function, Q -1 () is the dequantization function. The output function of this layer can be a dequantized representation of f(), or the output function can include additions such as activation functions or the addition of biases, performed using normal precision floating point (after dequantization) or using quantized floating point (before dequantization). Typically, the inputs, outputs, and parameters of each layer are tensors. Typically, the inputs, outputs, and parameters of each layer will be vectors or matrices. The quantization function converts normal precision floating point values ​​to quantized floating point values. The quantization function can be chosen to take into account the type of input data and the type of operations performed by the i-th layer. For example, when y i and W i is a two-dimensional matrix and the output function consists of taking y i-I and W i When the cross product of i-I The quantization function can be used including y i-I A row or y i-I The bounding box of a part of a line, and W i The quantization function can be used including W i A column or W i The bounding box of a portion of a column. When the bounding box is selected to follow the operator's process, the computation can be more efficient, resulting in a smaller, faster, and more energy-efficient hardware implementation. The dequantization function converts a quantized floating-point value to a normal precision floating-point value.

[0098] At process block 630, the loss of the neural network may be calculated. For example, the output y of the neural network may be compared to the expected output y of the neural network. The difference between the output and the expected output can be an input to a cost function used to update the parameters of the neural network.

[0099] At process block 640, the loss of the neural network can be back-propagated through the neural network. During back-propagation, the output error term can be calculated and the weighted error term The output error term can be described mathematically as:

[0100]

[0101] in is the output error term of the layer after the i-th layer, W i is the weight tensor of the i-th layer, g() is the backward function of the layer, Q() is the quantization function, Q -1 () is the dequantization function. i-1 The backward function g() may be the backward function of f(). The output error term of the layer may be a dequantized representation of g(), or the output error term may include additional terms performed using normal precision floating point (after dequantization) or using quantized floating point (before dequantization).

[0102] Weight error term It can be described mathematically as:

[0103]

[0104] in is the weight error term of the i-th layer, is the output error term of the i-th layer, y i is the output of the i-th layer, h() is the backward function of the layer, Q() is the quantization function, Q -1 () is the dequantization function. The backward function h() can be the backward function of f(), which is used to calculate the value of W i-1 The weight error term for this layer can be a dequantized representation of h(), or it can include additional terms that are performed using normal precision floating point (after dequantization) or using quantized floating point (before dequantization). The weight error term can include additional terms that are performed using normal precision floating point.

[0105] At process block 650, the parameters of each layer may be updated. For example, the weights of each layer may be updated by calculating new weights based on the training iterations. As an example, the weight update function may be mathematically described as:

[0106]

[0107] in is the weight error term of the i-th layer, η is the learning rate of the i-th layer of the neural network, and Wi is the weight tensor of the i-th layer. In one example, the weight update function can be executed using normal precision floating point.

[0108] Figure 7700 is a block diagram depicting an example of a mixed precision floating point environment and a partition between a normal precision floating point domain and a quantized floating point domain. As described above, computationally expensive operations (such as vector-vector, vector-matrix, matrix-matrix, and convolution operations) can be performed by the quantized layer 710 in the quantized floating point domain. Operations with lower computational costs (such as scalar addition and scalar multiplication) can be performed outside the quantized layer 710 in the normal precision floating point domain. With respect to a neural network, the neural network can be divided into layers (such as layer 710). Most of the computational work within a layer can be performed in the quantized floating point domain, and computationally less expensive operations of the layer (such as adding bias values ​​or calculating activation functions) can be performed in the normal precision floating point domain. Values ​​of the interface between layers can be passed from one layer to another in the normal precision floating point domain. By quantizing the input specifically for a given layer, quantization can target the operation of the layer to make the operation of the layer more efficient. Specifically, the bounding box of the quantized floating point format can be selected to reduce the complexity of the computer arithmetic circuitry, thereby making the computer logic potentially faster and / or more energy efficient.

[0109] As an example, given the output value y of a layer i , output error term Weight W i and the weighted error term Can be stored in the normal precision floating point domain. During the forward propagation flow 720, the output values ​​from the earlier layers can be communicated from the normal precision floating point domain to the quantized floating point domain through the quantizer 722, which converts from normal precision floating point to quantized floating point. The output values ​​from a given layer can be communicated from the quantized floating point domain to the normal precision floating point domain through the dequantizer 724, which converts from quantized floating point to normal precision floating point. The weights of a given layer can be communicated from the normal precision floating point domain to the quantized floating point domain through the quantizer 742.

[0110] In the back propagation process 730, the output error term of the subsequent layer can be communicated from the normal precision floating point domain to the quantized floating point domain through the quantizer 732. The output error term from a given layer can be communicated from the quantized floating point domain to the normal precision floating point domain through the dequantizer 734. The weight of a given layer can be communicated from the normal precision floating point domain to the quantized floating point domain through the quantizer 742. The weight error term from a given layer can be communicated from the quantized floating point domain to the normal precision floating point domain through the dequantizer 754.

[0111] Example method for training a neural network using mixed-precision floating point

[0112] Figure 8800 is a flow chart depicting an example method 800 for training a neural network accelerator using a mixed precision data format. The mixed precision data format may include a normal precision floating point format and a quantized precision floating point format. As an example, the method 800 may be performed by a neural network accelerator of a quantization-enabled system, such as Figure 1 A neural network accelerator 180 of a quantization-enabled system 110.

[0113] At process block 810, an input tensor for a layer of a multi-layer neural network may be received. The input tensor for the layer may be an input during a forward propagation or a backward propagation mode of the neural network. For example, the input tensor may be an input value from an input edge of the layer (e.g., x i ) or the operational parameters of the layer, such as weights (e.g., W i ), convolution filters, or bias values. As another example, the input tensor may be an output error term back-propagated from an adjacent layer during back-propagation. The input tensor may be received from an input / output interface of a neural network accelerator, on-chip memory, off-chip memory, or other storage location. The input tensor may include a plurality of normal precision floating point values, each of which includes a sign, a mantissa value, and an exponent value. The input tensor may be a vector or a multidimensional array.

[0114] At process block 820, the input tensor of the layer can be converted from a normal precision floating point format to a quantized precision floating point format. Converting from a normal precision floating point format to a quantized precision floating point format may include selecting a bounding box around a set of normal precision floating point values, determining a shared exponent of a shared normal precision floating point value, and adjusting the bit width of the mantissa value of the normal precision floating point value (such as by reducing the number of bits of the mantissa in the quantized floating point value and shifting the mantissa value to account for the shared exponent). For example, the quantized precision floating point format may be a block floating point format that groups multiple values ​​together so that multiple mantissa values ​​can share a common exponent. The values ​​of the exponents grouped together and sharing the block floating point format within the bounding box can be selected based on the dimension of the input tensor and / or the operation to be performed within the layer. For example, the bounding box can be a row, a column, a portion of a row or column, a two-dimensional shape, or a three-dimensional shape. The bounding box used for quantization may be different during forward propagation than during reverse propagation.

[0115] As a specific example, the layer may receive a pair of input tensors A and B, which are two-dimensional matrices to be multiplied within the layer. In other words, the layer may perform a matrix multiplication AB. Performing the matrix multiplication AB includes taking the dot product of each row of A and each column of B. The bounding box may be selected to include each row of A (or at least a portion of the rows of A), and the bounding box may be selected around each column of B (or at least a portion of the rows of B). Thus, quantizing the input tensor A may include grouping mantissa values ​​of a row having a shared exponent, and quantizing the input tensor B may include grouping mantissa values ​​of a column having a shared exponent. For a three-dimensional convolutional filter, quantizing the input tensor may include grouping shared exponents of mantissa values ​​of spatial pixels along the channel dimension.

[0116] At process block 830, a tensor operation may be performed using the converted quantized precision floating point format of the input tensor as input. For example, the tensor operation may be a vector-vector, vector-matrix, matrix-matrix, or convolution operation. For example, the tensor operation may be an addition, multiplication, dot product, convolution, or other operation that combines tensor elements to produce a result. The output result of the tensor operation may be a scalar value or a tensor value.

[0117] The result of the tensor operation may be converted from the quantized precision floating point format to the normal precision floating point format at process block 840. The conversion from the quantized precision floating point format to the normal precision floating point format may include generating an exponent value for the normal precision floating point value and adjusting a mantissa value of the normal precision floating point value (such as increasing the number of bits of the mantissa value and / or shifting the mantissa value to account for the generated exponent).

[0118] At optional process block 850, an operation may be performed using the converted, normal precision floating point formatted result. For example, a scalar addition (such as adding a bias value) or an activation function may be calculated using the converted, normal precision floating point formatted result.

[0119] At process block 860, the converted, normal precision floating point formatted result may be used to update an operational parameter of the layer of the neural network, where the parameter is stored in normal precision floating point format. For example, the operational parameter may be a weight or bias value for the layer. By updating the operational parameter of the layer, the behavior of the neural network for a given input stimulus will be different from that of an untrained neural network. For example, by training the neural network and updating the operational parameter, the neural network may learn to classify input data (such as image data, audio data, or other sensory data) into categories.

[0120] Fig. 9900 is a flowchart depicting an example method 900 for operating a neural network accelerator using a mixed precision data format. The mixed precision data format may include a normal precision floating point format and a block floating point format. As an example, the method 900 may be performed by a quantization-enabled system, such as Figure 1 A neural network accelerator 180 of a quantization-enabled system 110.

[0121] At process block 910, a hardware accelerator may be configured to accelerate a layer of a multi-layer neural network. The hardware accelerator may include a tensor processing unit (TPU), a soft processor core, programmable logic (e.g., a programmable lookup table (LUT) and / or block RAM), and / or dedicated circuitry for accelerating neuron and / or neuron layer computations. Configuring the hardware accelerator may include loading instructions for the TPU, soft processor core, or other dedicated processor; and / or loading configuration data onto programmable logic resources such that the programmable logic performs the task of accelerating neural network operations. For example, the layer may be a convolutional layer, a fully connected layer, a partially connected layer, a long short-term memory (LSTM) network layer, or a recurrent neural network (RNN) layer.

[0122] At process block 920, the input tensor of the layer may be converted from a normal precision floating point format to a block floating point format. The input tensor of the layer may be an input during a forward propagation or reverse propagation mode of a neural network. For example, during forward propagation, the input tensor may be an input value from an input edge of the layer or an operating parameter of the layer, such as a weight or bias value. During reverse propagation, the input tensor may be an output error term back-propagated from an adjacent layer or an operating parameter of the layer. The input tensor may be received from an input / output interface, an on-chip memory, an off-chip memory, or other storage location of a neural network accelerator. The input tensor may include a plurality of normal precision floating point values, each of which includes a sign, a mantissa value, and an exponent value. Converting from a normal precision floating point format to a quantized precision floating point format may include selecting a bounding box around a set of normal precision floating point values, determining a shared exponent of shared normal precision floating point values, and adjusting the bit width of the mantissa value of the normal precision floating point value (such as by reducing the number of bits of the mantissa in the quantized floating point value and shifting the mantissa value to account for the shared exponent). For example, the quantized precision floating point format can be a block floating point format that groups multiple values ​​together so that multiple mantissa values ​​can share a common exponent. The value of the exponent that is grouped together and shares the block floating point format can be selected based on the dimensions of the input tensor and / or the operation to be performed within the layer.

[0123] At process block 930, a tensor operation may be performed using the converted block floating point format of the input tensor. For example, the tensor operation may be a vector-vector, vector-matrix, matrix-matrix, or convolution operation. For example, the tensor operation may be an addition, multiplication, dot product, convolution, or other operation that combines tensor elements to produce a result. The output result of the tensor operation may be a scalar value or a tensor value.

[0124] At process block 940, the result of the tensor operation can be converted from the block floating point format to the normal precision floating point format. For example, the result can be an intermediate value of a node of the layer, an output value of a node of the layer, or an operation parameter, such as a weight or bias value of the layer. The conversion from the quantized precision floating point format to the normal precision floating point format can include generating an exponent value for the normal precision floating point value and adjusting the mantissa value of the normal precision floating point value (such as increasing the number of bits of the mantissa value and / or shifting the mantissa value to account for the generated exponent).

[0125] At optional process block 950, an operation may be performed using the converted, normal precision floating point formatted result. For example, a scalar addition (such as adding a bias value) or an activation function may be calculated using the converted, normal precision floating point formatted result.

[0126] At process block 960, the converted, normal precision floating point formatted results may be used to generate an output tensor for the layer of the neural network, wherein the output tensor is in normal precision floating point format. Values ​​transmitted between layers of the neural network may be passed in normal precision floating point format, which may improve the accuracy of the neural network, thereby enabling faster convergence and more accurate reasoning during training. By updating the output tensors of each layer of the neural network, the neural network may potentially classify input data (such as image data, audio data, or other sensory data) into various categories.

[0127] Additional Examples of the Disclosed Technology

[0128] Additional examples of the disclosed subject matter are discussed herein based on the examples discussed above.

[0129] In one example of the disclosed technology, a computing system includes a computer-readable memory storing an operational parameter of a given layer of a neural network. The computing system also includes a hardware accelerator in communication with the computer-readable memory. The hardware accelerator is configured to receive an input tensor of a given layer of a multi-layer neural network. The input tensor is converted from a normal precision floating point format to a quantized precision floating point format. A tensor operation is performed using the input tensor converted to the quantized precision floating point format. A result of the tensor operation is converted from the quantized precision floating point format to a normal precision floating point format. The converted result in the normal precision floating point format is used to update an operational parameter stored in the computer-readable memory, wherein the parameter is stored in the normal precision floating point format. Updating the operational parameter using the converted result in the normal precision floating point format may include performing a scalar operation that generates the operational parameter using the converted result in the normal precision floating point format.

[0130] The quantized precision floating point format may be a block floating point format with multiple mantissa values ​​sharing a common exponent. For example, the input tensor may be a two-dimensional matrix, and the block floating point format may enable multiple mantissa values ​​within a given row to share a common exponent, and mantissa values ​​in different rows to have corresponding different exponents. The input tensor may be a convolution filter, and the block floating point format may enable multiple mantissa values ​​within a spatial pixel to share a common exponent.

[0131] The tensor operation may be a matrix-matrix multiplication. The tensor operation may be a dot product calculation. The tensor operation may be a convolution. The tensor operation may be a vector-vector or vector-matrix operation. The tensor operation may be performed during a forward propagation mode or a backward propagation mode of a neural network. For example, during the backward propagation mode, the input tensor may be an output error term from a layer adjacent to the given layer (e.g., after the given layer) or a weight of the given layer. As another example, during the forward propagation mode, the input tensor may be an output term from a layer adjacent to the given layer (e.g., before the given layer) or a weight of the given layer.

[0132] In one example, a method for a neural network accelerator includes configuring the neural network accelerator to accelerate a given layer of a multi-layer neural network. Converting an input tensor of the given layer from a normal precision floating point format to a block floating point format. Performing a tensor operation using the input tensor converted to the block floating point format. Converting a result of the tensor operation from the block floating point format to a normal precision floating point format. Using the converted result in the normal precision floating point format to generate an output tensor of the layer of the neural network, wherein the output tensor is in a normal precision floating point format.

[0133] Configuring the neural network accelerator to accelerate a given layer of the multi-layer neural network may include loading configuration data onto programmable hardware so that the programmable hardware performs operations of the given layer of the multi-layer neural network. Configuring the neural network accelerator to accelerate a given layer of the multi-layer neural network may include programming a tensor processing unit. Configuring the neural network accelerator to accelerate a given layer of the multi-layer neural network may include initializing weights of input edges of the given layer of the multi-layer neural network.

[0134] Converting an input tensor of a given layer from a normal precision floating point format to a block floating point format may include selecting a bounding box for a plurality of elements of the input tensor. The bounding box may be selected based on the tensor operation performed. The tensor operation performed may be a matrix-matrix multiplication, and the selected bounding box may be a column of the input tensor matrix. The tensor operation performed may be a matrix-matrix multiplication, and the selected bounding box may be a row of the matrix of the input tensor. Converting an input tensor of a given layer from a normal precision floating point format to a block floating point format may include: selecting a bounding box for a plurality of elements of the input tensor; identifying a shared exponent for a plurality of selected elements within the bounding box of the input tensor; scaling the mantissa values ​​of the elements of the input tensor so that the integer portion of the scaled mantissa has a selected number of bits for the block floating point format; removing fractional bits from the scaled integer portion of the mantissa; and rounding the mantissa to produce a block floating point value.

[0135] In one example, one or more computer-readable media store computer-executable instructions that, when executed by a neural network accelerator, cause the neural network accelerator to perform an operation. The operation includes converting an input tensor of a given layer of a multi-layer neural network from a normal precision floating point format to a block floating point format. The operation includes performing a tensor operation using an operation parameter of a given layer of the neural network and the input tensor converted to a block floating point format. For example, the tensor operation may be a convolution or a matrix-matrix multiplication. As another example, the tensor operation may be performed during a back-propagation mode or a forward propagation mode of the neural network. The operation includes converting a result of the tensor operation from a block floating point format to a normal precision floating point format. The operation includes updating an operation parameter stored in one or more computer-readable media using the converted result in a normal precision floating point format.

[0136] The input tensor may be a two-dimensional matrix. Converting the input tensor from a normal precision floating point format to a block floating point format may include selecting a plurality of elements within a column of the two-dimensional matrix to share a common index in the block floating point format. Converting the input tensor from a normal precision floating point format to a block floating point format may include selecting a plurality of elements within a row of the two-dimensional matrix to share a common index in the block floating point format.

[0137] Sample computing environment

[0138] Fig.10A generalized example of a suitable computing environment 1000 is shown in which the described embodiments, techniques, and technologies, including supporting a multi-language playback framework, may be implemented.

[0139] The computing environment 1000 is not intended to impose any limitations on the scope of use or functionality of the technology, as the technology can be implemented in different general-purpose or special-purpose computing environments. For example, the disclosed technology can be implemented with other computer system configurations, including handheld devices, multiprocessor systems, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, etc. The disclosed technology can also be implemented in a distributed computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed computing environment, program modules can be located in local and remote memory storage devices.

[0140] refer to Fig.10 , the computing environment 1000 includes at least one processing unit 1010, and a memory 1020. Fig.10 , this most basic configuration 1030 is included within the dashed line. Processing unit 1010 executes computer executable instructions and can be a real or virtual processor. In a multi-processing system, multiple processing units execute computer executable instructions to increase processing power, so multiple processors can run simultaneously. Memory 1020 can be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two. Memory 1020 stores software 1080, images, and videos that can, for example, implement the techniques described herein. The computing environment can have additional features. For example, computing environment 1000 includes storage 1040, one or more input devices 1050, one or more output devices 1060, and one or more communication connections 1070. An interconnect mechanism (not shown) (such as a bus, controller, or network) interconnects the components of computing environment 1000. Typically, operating system software (not shown) provides an operating environment for other software executed in computing environment 1000 and coordinates the activities of components of computing environment 1000.

[0141] Storage 1040 may be removable or non-removable and include disks, tapes or cartridges, CD-ROMs, CD-RWs, DVDs, or any other medium that can be used to store information and that can be accessed within computing environment 1000. Storage 1040 stores instructions, plug-in data, and messages that may be used to implement software 1080 of the techniques described herein.

[0142] The input device(s) 1050 may be a touch input device such as a keyboard, keypad, mouse, touch screen display, pen or trackball, voice input device, scanning device, or other device that provides input to the computing environment 1000. For audio, the input device(s) 1050 may be a sound card or similar device that accepts audio input in analog or digital form, or a CD-ROM reader that provides audio samples to the computing environment 1000. The output device(s) 1060 may be a display, printer, speaker, CD recorder, or another device that provides output from the computing environment 1000.

[0143] The communication connection(s) 1070 enable communication with another computing entity via a communication medium (e.g., a connection network). The communication medium communicates information such as computer executable instructions, compressed graphics information, video, or other data in a modulated data signal. The communication connection(s) 1070 are not limited to wired connections (e.g., Megabit or Gigabit Ethernet, Infiniband, Fibre Channel, or Fibre Channel connections), but also include wireless technologies (e.g., RF connections via Bluetooth, WiFi (IEEE 802.11a / b / n), WiMax, cellular, satellite, laser, infrared), and other suitable communication connections for providing network connectivity for the disclosed agents, bridges, and proxy data consumers. In a virtual host environment, the communication connection(s) may be a virtualized network connection provided by the virtual host.

[0144] Some embodiments of the disclosed methods may be performed using computer-executable instructions that implement all or a portion of the disclosed techniques in a computing cloud 1090. For example, the disclosed methods may be performed on a processing unit 1010 located in a computing environment 1030, or the disclosed methods may be performed on a server located in a computing cloud 1090.

[0145] Computer-readable media is any available media that can be accessed within computing environment 1000. By way of example and not limitation, with computing environment 1000, computer-readable media include memory 1020 and / or storage 1040. As should be readily understood, the term computer-readable storage media includes media used for data storage, such as memory 1020 and storage 1040, and does not include transmission media, such as modulated data signals.

[0146] In view of the many possible embodiments to which the principles of the disclosed subject matter may be applied, it should be recognized that the embodiments shown are merely preferred examples and should not be considered to limit the scope of the claims to these preferred examples. Rather, the scope of the claimed subject matter is defined by the appended claims. We therefore claim as our invention all that falls within the scope of these claims.

Claims

1. A computing system comprising: a computer readable memory storing operating parameters of a given layer of the neural network; as well as A hardware accelerator in communication with the computer-readable memory for accelerating tensor operations, the hardware accelerator being configured to: Receives an input tensor for a given layer of a multi-layer neural network; converting the input tensor from a normal precision floating point format to a quantized precision floating point format, the quantized precision floating point format being a block floating point format, wherein a first portion of the converted input tensor corresponds to a first portion of the input tensor, the first portion of the converted input tensor includes a first common exponent for the first portion of the input tensor and a first plurality of mantissa values, and a second portion of the converted input tensor corresponds to a second portion of the input tensor, the second portion of the converted input tensor includes a second common exponent for the second portion of the input tensor and a second plurality of mantissa values, wherein the first common exponent is different from the second common exponent; performing a tensor operation using the input tensor converted to the quantized precision floating point format; converting a result of the tensor operation from the quantized precision floating point format to the normal precision floating point format to provide a converted result in the normal precision floating point format; as well as Using the converted result in the normal precision floating point format to update the operational parameters of the given layer of the multi-layer neural network in a training iteration of a plurality of iterations of training of the multi-layer neural network, wherein the operational parameters of the given layer of the multi-layer neural network are stored in the normal precision floating point format.

2. The computing system of claim 1 , wherein the input tensor is a two-dimensional matrix and the quantized precision floating-point format is a block floating-point format in which multiple mantissa values ​​within a given row share a common exponent and mantissa values ​​in different rows have corresponding different exponents.

3. The computing system of claim 1, wherein the input tensor is a convolution filter and the quantized precision floating point format is a block floating point format in which multiple mantissa values ​​within a spatial pixel share a common exponent.

4. The computing system of claim 1 , wherein the tensor operation is performed during a back-propagation mode of the neural network, the input tensor being an output error term from an adjacent layer to the given layer.

5. The computing system of claim 1, wherein the tensor operation is a dot product calculation.

6. The computing system of claim 1, wherein the tensor operation is a convolution.

7. The computing system of claim 1, wherein the tensor operations are performed during a back-propagation phase of training the neural network.

8. The computing system of claim 1 , wherein updating the operating parameter using the converted result in the normal precision floating point format comprises: A scalar operation is performed that uses the converted result in the normal precision floating point format to generate the operation parameter.

9. The computing system of claim 8, wherein the scalar operation is performed on a single layer of the neural network.

10. The computing system of claim 8, wherein the scalar operation comprises adding a bias to the converted result.

11. The computing system of claim 8, wherein the scalar operation comprises applying an activation function to the transformed result.

12. A method implemented in a computing system, comprising: Converting an input tensor for a given layer of a multi-layer neural network from a normal precision floating point format to converted values ​​represented in a block floating point format by: (1) for a first portion of the input tensor, selecting a first bounding box that includes a first set of values ​​expressed in the normal precision floating point format and wherein the block floating point format uses a first common index for the converted values ​​of the first set of values; and (2) for a second portion of the input tensor, selecting a second bounding box that includes a second set of values ​​expressed in the normal precision floating point format and wherein the block floating point format uses a second common index for the converted values ​​of the second set of values, wherein the second set of values ​​is different from the first set of values ​​and the second common index is different from the first common index; performing a tensor operation using the converted values ​​in the input tensor converted to the block floating point format; Converting the result of the tensor operation from the block floating point format to the normal precision floating point format; generating an output tensor of the layer of the neural network using the converted result of the normal precision floating point format, wherein the output tensor is in a normal precision floating point format; as well as In a training iteration of a plurality of iterations of training of the multi-layer neural network, using the converted result in the normal precision floating point format to update an operational parameter of the multi-layer neural network, wherein the operational parameter of the given layer of the multi-layer neural network is maintained in the normal precision floating point format.

13. The method according to claim 12, further comprising: The configuration data is loaded onto programmable hardware of the neural network accelerator so that the programmable hardware performs the operation of the given layer of the multi-layer neural network.

14. The method according to claim 12, further comprising: Initializing weights of input edges of the given layer of the multi-layer neural network in a neural network accelerator.

15. The method of claim 12, wherein the first bounding box is a row of a matrix of the input tensor.

16. The method of claim 12, wherein the first bounding box is a column of a matrix of the input tensor.

17. The method of claim 12, wherein converting the input tensor for the given layer from the normal precision floating point format to the block floating point format comprises: selecting bounding boxes for a plurality of elements of the input tensor; identifying sharing indices for the first set of values ​​within the bounding box of the input tensor; scaling mantissa values ​​of the elements of the input tensor so that an integer portion of the scaled mantissa has a selected number of bits for the block floating point format; removing fractional bits from the scaled integer portion of the mantissa; as well as The mantissa is rounded to produce a block floating point value.

18. The method according to claim 12, wherein the multi-layer neural network is a recurrent neural network, further comprising: Configuring the neural network accelerator to accelerate the given layer of the multi-layer neural network includes programming hardware to perform the function of the layer of the recurrent neural network.

19. The method of claim 12, wherein the method is performed in a neural network accelerator.

20. One or more computer-readable media comprising: Computer executable instructions that, when executed by a computing device, cause the computing device to: convert an input tensor for a given layer of a multi-layer neural network from a normal precision floating point format to a block floating point format by: (1) for a first portion of the input tensor, selecting a first bounding box surrounding a first set of values ​​expressed in the normal precision floating point format, and wherein the block floating point format uses a first common exponent for the converted values ​​for the first set of values; and (2) for a second portion of the input tensor, selecting a second bounding box including a second set of values ​​expressed in the normal precision floating point format, and wherein the block floating point format uses a second common exponent for the converted values ​​for the second set of values, wherein the second set of values ​​is different from the first set of values, and the second common exponent is different from the first common exponent; computer-executable instructions that, when executed by the computing device, cause the computing device to perform a tensor operation using operational parameters of the given layer of the neural network and the input tensor converted to the block floating point format; computer executable instructions that, when executed by the computing device, cause the computing device to convert, using a neural network accelerator, a result of the tensor operation from the block floating point format to the common precision floating point format to provide a converted result in the common precision floating point format; as well as computer executable instructions that, when executed by the computing device, cause the computing device to update the operating parameters using the converted results in the normal precision floating point format in a training iteration of a plurality of iterations of training the multi-layer neural network.

21. The one or more computer-readable media of claim 20, wherein the input tensor is a two-dimensional matrix, and the computer-executable instructions, when executed by the computing device, cause the computing device to: convert the input tensor from the normal precision floating point format to the block floating point format comprises: In the block floating point format, a plurality of elements within a column of the two-dimensional matrix are selected to share a common exponent.

22. The one or more computer-readable media of claim 20, wherein the input tensor is a two-dimensional matrix, and the computer-executable instructions, when executed by the computing device, cause the computing device to: convert the input tensor from the normal precision floating point format to the block floating point format comprises: In the block floating point format, a plurality of elements within a row of the two-dimensional matrix are selected to share a common exponent.

23. The one or more computer-readable media of claim 20, wherein the tensor operations are performed during a back-propagation mode of the neural network.

Images