Double-Exponential Bounding Box Floating-Point Processor

Abstract

Apparatus and methods are disclosed for performing matrix operations using a double exponential format, including operations suitable for neural networks and other machine learning accelerators and applications. The disclosed matrix formats include a single exponential bounding box floating point (SE-BBFP) format and a double exponential bounding box floating point (DE-BBFP) format. A shared exponent for each element is determined for each element based on whether the element is used as a row of a matrix tile or a column of a matrix file, for example for a dot product operation. Computing systems suitable for using such neural networks include computers with general purpose processors, neural network accelerators, or reconfigurable logic devices such as field programmable gate arrays (FPGAs). Certain techniques disclosed herein can provide improved system performance while reducing memory and network bandwidth used.

Description

[Background technology]

[0001] Machine learning (ML) and artificial intelligence (AI) techniques can be useful for solving many complex computational problems, such as recognizing images and speech, analyzing and classifying information, and performing a variety of classification tasks. Machine learning is a field of computer science that uses statistical techniques to give computer systems the ability to extract higher level features from a set of training data. Specifically, features may be extracted by training a model, such as an artificial neural network (NN) or a deep neural network (DNN). After the model is trained, new data can be applied to the model, and new data can be classified (e.g., higher level features can be extracted) using the trained model. Machine learning models typically run on general-purpose processors (also called central processing units (CPUs)). However, training a model and / or using a model can be computationally expensive, and thus it may not be possible to perform feature extraction in real time using a general-purpose processor. Summary of the Invention [Problem to be solved by the invention]

[0002] Thus, there is ample opportunity for improvements in computer hardware and software for implementing neural networks. [Means for solving the problem]

[0003] Apparatus and methods are disclosed for performing matrix operations using a double exponent format, including operations suitable for neural networks and other machine learning accelerators and applications. The disclosed matrix formats include single exponent bounding box floating-point (SE-BBFP) format and dual exponent bounding box floating-point (DE-BBFP) format. The shared exponent for each element is determined for each element based on whether the element is used as a row of a matrix tile or a column of a matrix file, for example for a dot product operation. Computing systems suitable for using such neural networks include computers with general purpose processors, neural network accelerators, or reconfigurable logic devices, such as field programmable gate arrays (FPGAs). Certain techniques disclosed herein can provide improved system performance while reducing memory and network bandwidth used.

[0004] In some examples of the disclosed techniques, a computer system includes a general-purpose and / or special-purpose neural network processor and a memory. When forward propagation occurs during training of a neural network, activation values ​​are generated in a first shared exponential format, SE-BBFP or DE-BBFP. The compressed activation values ​​are stored in bulk memory for use during backward propagation. When matrix values ​​are stored in DE-BBFP format, the matrix may be used as a left or right operand for matrix operations that use a single set of mantissas.

[0005] In some examples of the disclosed techniques, a computer-implemented method includes using a processor to select a common exponent for bounding boxes of elements of an input matrix stored in double exponent format, where the common exponent is selected based on a smaller exponent for either a row or column of the bounding box of the elements; determining mantissas for the bounding boxes of the elements of the double exponent format matrix, where each of the determined mantissas is selected by comparing an exponent of a respective element to the common exponent; and storing the determined mantissas and common exponent in a computer-readable storage medium as a dual exponent format matrix.

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0007] The foregoing and other objects, features and advantages of the present invention will become more apparent from the following detailed description which proceeds with reference to the accompanying drawings. [Brief description of the drawings]

[0008] [Figure 1] FIG. 13 is a bounding box diagram of a dual exponent correspondence system for performing training and inference using DE-BBFP format values, as may be implemented in certain examples of the disclosed technology. [Diagram 2] FIG. 1 illustrates an example of a deep neural network, such as may be modeled using certain example methods and apparatus disclosed herein. [Diagram 3] FIG. 13 illustrates certain aspects of determining a row index when converting from a normal floating-point format to a DE-BBFP format, as may be performed in certain examples of the disclosed technology. [Figure 4]A diagram illustrating certain aspects of determining a column index when converting a normal floating-point format to a DE-BBFP format, as may be performed in certain examples of the disclosed technology. [Diagram 5] 13 shows an example of a double exponent and associated mantissa for a bounding box of a matrix element, as may be used in certain examples of the disclosed technology. [Figure 6] 13 shows an example of a double exponent and associated mantissa for a bounding box of a matrix element, as may be used in certain examples of the disclosed technology. [Figure 7] FIG. 13 is a diagram illustrating performing matrix operations using the DE-BBFP format, as may be implemented in certain examples of the disclosed technology. [Figure 8] FIG. 13 outlines a high-level architecture for performing tensor and matrix operations using DE-BBFP formatted data. [Figure 9] 1 is a flowchart outlining an example of performing neural network training using a DE-BBFP matrix, as may be implemented in certain examples of the disclosed technology. [Figure 10] 11 is a flowchart illustrating an example of using DE-BBFP format matrices to perform matrix operations, as may be implemented in certain examples of the disclosed technology. [Figure 11] 1 is a flowchart illustrating an example method for converting regular floating-point to DE-BBFP floating-point format, as may be implemented in certain examples of the disclosed technology. [Figure 12] 1 is a flowchart illustrating an example method for converting a DE-BBFP floating-point format matrix to a regular floating-point matrix, as may be implemented in certain examples of the disclosed technology. [Figure 13] FIG. 1 illustrates a suitable computing environment for implementing some embodiments of the disclosed technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] I. General considerations This disclosure has been described in the context of representative embodiments, which are not intended to be limiting in any way. As used in this application, the singular forms "a," "an," and "the" include the plural unless the context clearly dictates otherwise. Additionally, the term "include" means "comprise." Additionally, the term "coupled" encompasses mechanical, electrical, magnetic, optical, and other practical ways of coupling or linking items to one another and does not preclude the presence of intermediate elements between the coupled items. Additionally, as used herein, the term "and / or" refers to any one item or combination of items in the phrase.

[0010] The systems, methods, and apparatus described herein should not be construed as limiting in any way. Rather, the present disclosure is directed to all novel and non-obvious features and aspects of the various disclosed embodiments, both alone and in various combinations and subcombinations with each other. The disclosed systems, methods, and apparatus are not limited to any particular aspect or feature or combination thereof, and the disclosed and methods do not require that any one or more particular advantages exist or problems be solved. Furthermore, any features or aspects of the disclosed embodiments can be used in various combinations and subcombinations with each other.

[0011] Although operations of some of the disclosed methods are described in a particular sequential order for convenience of presentation, it should be understood that this style of description encompasses rearrangements unless a particular ordering is required by specific wording described below. For example, operations described sequentially may in some cases be rearranged or performed concurrently. Moreover, for simplicity, the accompanying drawings may not show the various ways in which the disclosed items and methods may be used with other items and methods. Furthermore, the description may use terms such as "generate," "produce," "display," "receive," "verify," "execute," "perform," "convert," and "initiate" to describe the disclosed methods. These terms are high-level descriptions of actual operations that are performed. The actual operations that correspond to these terms vary depending on the particular implementation and are readily discernible by those of ordinary skill in the art.

[0012] Any theories of operation, scientific principles, or other theoretical explanations presented herein for the devices or methods of the present disclosure are provided for purposes of better understanding and are not intended to limit the scope, and the devices and methods in the appended claims are not limited to devices and methods that function as described by such theories of operation.

[0013] Any of the disclosed methods may be implemented as computer-executable instructions stored on one or more computer-readable media (e.g., computer-readable media such as one or more optical media disks, volatile memory components (such as DRAM or SRAM), or non-volatile memory components (such as hard drives)) and executed on a computer (e.g., any commercially available computer, including a smart phone or other mobile device that includes computing hardware). Any of the computer-executable instructions for implementing the disclosed techniques, as well as any data created and used during the implementation of the disclosed embodiments, may be stored on one or more computer-readable media (e.g., computer-readable storage media). The computer-executable instructions may be, for example, part of a dedicated software application, or a software application accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software may be executed, for example, on a single local computer or in a networked environment using one or more networked computers (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network), or other such network).

[0014] For clarity, only certain selected aspects of the software-based implementation are described. Other details well known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any particular computer language or program. For example, the disclosed technology may be implemented by software written in C, C++, Java, or any other suitable programming language. Certain details of suitable computers and hardware are well known and need not be described in detail in this disclosure.

[0015] Additionally, any of the software-based embodiments (e.g., including computer-executable instructions for causing a computer to perform any of the disclosed methods) may be uploaded, downloaded, or remotely accessed via appropriate communications means, including, for example, the Internet, the World Wide Web, an intranet, a software application, cable (including fiber optic cable), magnetic communication, electromagnetic communication (including RF, microwave, and infrared communication), electronic communication, or other such communications means.

[0016] II. Overview of Artificial Neural Networks Using the Double Exponential Format Artificial neural networks (ANNs, or as used throughout this specification, "NNs") are applied to numerous applications in artificial intelligence and machine learning, including image recognition, speech recognition, search engines, and other suitable applications. Processing for these applications may occur on individual devices, such as personal computers or mobile phones, but may also be performed in large data centers. At the same time, hardware accelerators that may be used with NNs include specialized NN processing units, such as tensor processing units (TPUs) and field programmable gate arrays (FPGAs), that are programmed to accelerate neural network processing. Such hardware devices are being deployed in consumer devices as well as data centers due to their flexible nature and low power consumption per unit computation.

[0017] Traditionally, NNs have been trained and deployed using single-precision floating-point numbers (32-bit floating-point or float32 format). However, it has been shown that inference operations can be performed with minimal loss of accuracy using lower precision floating-point formats, such as 16-bit floating-point (float16) or fixed-point formats. On specialized hardware such as FPGAs, reduced precision formats can significantly improve the latency and throughput of DNN processing.

[0018] Numbers represented in normal precision floating-point format (e.g., floating-point numbers are represented in 16-bit, 32-bit, 64-bit, or 80-bit floating-point formats) can be converted to numbers in DE-BBFP format, which may allow performance benefits in performing operations. In particular, NN weights and activation values ​​can be represented in a low-precision DE-BBFP format with an acceptable level of error introduced.

[0019] One characteristic of computation on an FPGA device is that it typically lacks hardware floating-point support. Floating-point operations may be performed, with a penalty, using flexible logic, but often the amount of logic required to support floating point becomes prohibitive in an FPGA implementation. Some newer FPGAs have been developed that support floating-point computation, but even on these the same device can produce twice the computational output per unit time when used in integer mode. Typically, NNs are created with floating-point computation in mind, but when FPGAs are targeted for NN processing, it would be beneficial if the neural network could be expressed using integer arithmetic. Examples of the disclosed techniques include hardware implementations of double-exponential bounding box floating point (DE-BBFP), including the use of DE-BBFP in NNs, FPGAs, and other hardware environments. As used herein, double-exponential bounding box floating point refers to a matrix representation in which array elements can have two different exponents depending on whether the element is used as a row or column of the element in a matrix operation. In other words, each element uses a row index when the element is used on the left side of a matrix operation, and a column index that can be different from the row index when the element is used on the right side of a matrix operation. A "bounding box" or "bounding box" floating point refers to when some group of elements in a matrix (not necessarily all) share a common row or column index. For example, a matrix of 1024 x 1024 elements may be constructed from 64 x 64 tiles, with each tile having 16 x 16 elements. The rows and columns in each tile share a common index.

[0020] A typical floating-point representation in a computer system consists of three parts: the sign (s), the exponent (e), and the significand or mantissa (m). The sign indicates whether the number is positive or negative. The exponent and significand are used similarly to scientific notation: Value = s × m × 2 e (Formula 1)

[0021] As used herein, "mantissa" refers to the significant digit of a number as represented in scientific notation formats, including floating point and bounding box floating box formats. The mantissa is often referred to as the significand or modulus. Any number can be represented within the precision limits of the mantissa. The exponent scales the mantissa by powers of 2, just as the exponent does by powers of 10 in scientific notation, so the magnitude of very large numbers can be represented. The precision of the representation is determined by the precision of the mantissa. Typical floating-point representations use mantissas that are 10 (float16), 24 (float32), or 53 (float64) bits wide. 2 53 Integers with larger magnitudes can be approximated in the float64 floating-point format, but cannot be represented exactly because there are not enough bits in the mantissa. A similar effect can occur for any fraction when the fraction is represented by mantissa bits that take values ​​that are negative powers of two. There are many fractions that cannot be represented exactly in the binary number system because they are irrational. In both situations, more precise representations are possible, but they may require the mantissa to contain more bits. Ultimately, an infinite number of mantissa bits are needed to represent some numbers exactly.

number

[0022] In a bounding box floating-point format, a group of two or more numbers uses a single shared exponent, with each number still having its own sign and mantissa. In some examples, the shared exponent is selected to be the maximum exponent of the original floating-point value. For purposes of this disclosure, the term bounding box floating-point (DE-BBFP) refers to a number system in which a single exponent is shared across two or more values, each of which is represented by a sign and mantissa pair (there may be an explicit sign bit, or the mantissa itself may be signed). In some examples, all the values ​​of one or more rows or columns of a matrix or vector, or all the values ​​of a matrix or vector, may share a common exponent. In other examples, the DE-BBFP representation may be unsigned. In some examples, some, but not all, of the elements in a matrix or vector DE-BBFP representation may include numbers represented as integers, floating-point numbers, fixed-point numbers, symbols, or other data formats mixed with numbers represented with signs, mantissas, and exponents. In some examples, some or all of the elements in a matrix or vector DE-BBFP representation may include composite elements with two or more parts, such as complex numbers with an imaginary part (a+bi, where i=√(-1)); fractions with numerators and denominators, in polar coordinates (r,θ) or other multi-component elements.

[0023] The parameters of a particular DE-BBFP format may be chosen to trade off precision and storage requirements for a particular implementation. For example, rather than storing an exponent with every floating-point number, a group of numbers may share the same exponent. To share an exponent while maintaining a high level of accuracy, the numbers should have approximately the same absolute value, since the difference in absolute value is represented in the mantissa. If the difference in absolute value is too large, the mantissa may overflow for large values ​​or become zero ("underflow") for smaller values. Depending on the particular application, a certain amount of overflow and / or underflow may be acceptable.

[0024] The size of the mantissa can be adjusted to suit a particular application. This may affect the precision of the represented numbers, but potential gains are realized from the reduced representation size. For example, a typical single precision floating point has a size of 4 bytes, but for certain implementations of the disclosed technology, only 2 bytes are used to represent the sign and mantissa of each value. In some implementations, the sign and mantissa of each value can be represented in 1 byte or less.

[0025] In certain examples of the disclosed technology, the representations described above are used to derive original numbers from the representations, but only a single exponent is stored for a group of numbers, and each number is represented by a signed mantissa. Each signed mantissa can be represented in two bytes or less, thus providing approximately double the memory storage savings compared to 4-byte floating point. Furthermore, the memory bandwidth requirements for loading and storing these values ​​are also approximately half that of regular floating point.

[0026] Neural network operations are used in many artificial intelligence operations. Often, the majority of processing operations performed in implementing neural networks are in performing matrix-by-matrix or matrix-by-vector multiplication or convolution operations. Such operations are computationally and memory bandwidth intensive, with matrices of size, for example, 1000×1000 elements (e.g., 1000×1000 numbers, each including a sign, a mantissa, and an exponent) or more, and many matrices being used. As described herein, DE-BBFP techniques can be applied to such operations to reduce the computational and memory bandwidth requirements in a given system, whether an FPGA, a CPU, or another hardware platform. As used herein, the use of the term "element" herein refers to a member of such a matrix or vector.

[0027] As used herein, the term "tensor" refers to a multidimensional array that may be used to represent properties of a NN, including one-dimensional vectors, as well as matrices of two, three, four, or more dimensions. As used in this disclosure, tensors do not require any other mathematical properties unless specifically stated.

[0028] As used herein, the term "normal precision floating point" or "normal floating point" refers to a floating point number format in which each number has a mantissa, an exponent, and optionally a sign, and which is natively supported by a native or virtual CPU. Examples of normal precision floating point formats include, but are not limited to, IEEE754 standard formats such as 16-bit, 32-bit, 64-bit, or 80-bit floating point formats supported by processors such as Intel AVX, AVX2, IA32, x86_64, etc.

[0029] A given number may be represented using formats of different precision. For example, a number may be represented in a higher precision format (e.g., float32) and a lower precision format (e.g., float16). Reducing the precision of a number may include reducing the number of bits used to represent the mantissa or exponent of the number. Additionally, reducing the precision of a number may include reducing the range of values ​​that may be used to represent the exponent of a number, such as when multiple numbers share a common exponent. Similarly, increasing the precision of a number may include increasing the number of bits used to represent the mantissa or exponent of the number. Additionally, increasing the precision of a number may include increasing the range of values ​​that may be used to represent the exponent of a number, such as when the number is separated from a group of numbers that share a common exponent.

[0030] The term "quantized double exponential floating-point" or "quantized DE-BBFP" refers to a double exponential floating-point number format in which two or more values ​​of a tensor are modified to have less precision than when the values ​​are represented in normal precision floating point. For example, after converting a matrix value to double exponential format, a 16-bit mantissa may be quantized to any number of fewer bits, including 8, 7, 4, or 3 bits, to further reduce storage and processing hardware requirements. A double exponential 8-bit mantissa may be converted to any number of fewer bits, including 7, 4, or 3 bits. Quantization can produce similar results in deployed neural networks and is particularly useful in certain neural network processing applications during training and other operations where loss of precision is acceptable.

[0031] In one example of the disclosed technology, a neural network accelerator is configured to perform training operations for layers of a neural network, including forward propagation and backward propagation. Values ​​of one or more of the neural network layers may be represented in DE-BBFP or quantized DE-BBFP (QDE-BBFP) format. For example, the DE-BBFP format may be used to accelerate calculations performed in training and inference operations using the neural network accelerator. The use of the double exponential format may improve neural network processing, for example, by allowing faster hardware, reduced memory overhead, simpler hardware design, reduced energy usage, reduced integrated circuit area, cost savings, and other technological improvements. When it is not known whether a matrix is ​​used as a left operand, a right operand, or both a left and right operand for a matrix operation, the use of the DE-BBFP format allows matrix values ​​to be stored using a single set of mantissas, a set of column exponents, and a set of row exponents. In contrast, SE-BBFP format matrices where it is unknown whether a matrix is ​​used as a left operand, a right operand, or both a left and right operand, require that two sets of mantissas, each set shifted relative to its exponent, and two sets of exponents be stored. Thus, the use of dual format matrices as described herein provides substantial savings in memory, computation, and interconnect resources.

[0032] It is often desirable for operations to be performed to mitigate noise or other inaccuracies introduced by using a lower precision format. Additionally, parts of neural network training, such as temporary storage of activation values, can be improved by compressing some of these values ​​(e.g., for the input, hidden, or output layers of a neural network) from either normal precision floating point or DE-BBFP to the lower precision DE-BBFP format. The activation values ​​can then be retrieved later, for example for use during backpropagation during the training phase.

[0033] An input tensor for a given layer may be converted from normal precision floating-point format to DE-BBFP floating-point format. A tensor operation may be performed using the converted input tensor. A result of the tensor operation may be converted from DE-BBFP format to normal precision floating-point format. The tensor operation may be performed during a forward propagation mode or a backward propagation mode of the neural network. For example, during a backward propagation mode, the input tensor may be an output error term from a layer adjacent (e.g., subsequent) to the given layer, or a weight of the given layer. As another example, during a forward propagation mode, the input tensor may be an output term from a layer adjacent (e.g., preceding) to the given layer, or a weight of the given layer. The converted result may be used to generate an output tensor for that layer of the neural network, where the output tensor is in normal precision floating-point format. In this way, a neural network accelerator can potentially be made smaller and more efficient than a comparable accelerator that uses only normal precision floating point formats. A smaller, more efficient accelerator can improve computational performance and / or improve energy efficiency. By increasing the accuracy of the accelerator, convergence time for training can be reduced and the accelerator can be more accurate when classifying inputs to the neural network. Reducing the computational complexity of using the model can potentially reduce the time to extract features during inference, reduce the time for adjustments during training, and / or reduce energy consumption during training and / or inference.

[0034] On certain chips dedicated to machine learning / deep learning (ML / DL) processing, the three expensive hardware components are the arithmetic processing units, the on-chip memory, and the data fabric that moves weight and activation data between the processing units. A significant part of the computation is the matrix multiplication. To reduce the hardware complexity of the matrix multiplication unit, the DE-BBFP representation may be used. Typically, the data is stored in some floating-point / integer format and then converted to DE-BBFP at computation time to reduce the computation hardware.

[0035] The DE-BBFP representation of data provides an efficient way to store data in applications that use DE-BBFP arithmetic. DE-BBFP may be converted (on the fly) to single-exponential bounding box floating point (SE-BBFP) format prior to computation. Conversion from DE-BBFP to SE-BBFP is easy and nearly lossless. Data can be converted from floating point or integer format to DE-BBFP format and immediately stored. In some instances, the DE-BBFP representation requires 40-70% less storage compared to storing data in 16-bit or 32-bit regular floating point format. Data storage using the DE-BBFP format may also reduce the data fabric bandwidth required to move data both on-chip and off-chip by a similar percentage.

[0036] Example Architecture for Implementation III. Activation compression using the DE-BBFP format FIG. 1 is a bounding box diagram 100 outlining an exemplary double exponential correspondence system 110 as may be implemented in certain examples of the disclosed techniques, including for use in activation compression with DE-BBFP. As shown in FIG. 1, the double exponential correspondence system 110 may include several hardware resources, including a general-purpose processor 120 and special-purpose processors, such as a graphics processing unit 122 and a neural network accelerator 180. These processors are coupled to a memory 125 and a storage 129, which may include volatile or non-volatile memory devices. The processors 120 and 122 execute instructions stored in the memory or storage to provide a neural network module 130. The neural network module 130 includes software interfaces that allow the system to be programmed to implement various types of neural networks. For example, software functions may be provided that allow an application to define a neural network, including weights, biases, activation functions, node values, and interconnections between layers of the neural network. Additionally, software functions may be used to define state elements for a recurrent neural network. The neural network module 130 may further provide a utility that allows training and retraining of a neural network implemented with the module. Values ​​representing the neural network module are stored in memory or storage and manipulated by instructions executed by one of the processors. Values ​​stored in memory or storage may be represented using floating point values ​​in normal precision floating point, SE-BBFP, DE-BBFP, or QDE-BBFP format.

[0037] In some examples, proprietary or open source libraries or frameworks are provided to programmers to implement neural network creation, training, and evaluation. Examples of such libraries include TensorFlow, Microsoft Cognitive Toolkit (CNTK), Caffe, Theano, and Keras. In some examples, programming tools, such as integrated development environments, provide support for programmers and users to define, compile, and evaluate NNs.

[0038] Neural network accelerator 180 may be implemented as a custom or application specific integrated circuit (including, for example, a system-on-chip (SoC) integrated circuit), as a field programmable gate array (FPGA) or other reconfigurable logic, or as a soft processor virtual machine hosted by a physical general-purpose processor. Neural network accelerator 180 may include tensor processing unit 182, reconfigurable logic device 184, and / or one or more neural processing cores (such as DE-BBFP accelerator 186). DE-BBFP accelerator 186 may be configured in hardware, software, or a combination of hardware and software. As an example, DE-BBFP accelerator 186 may be configured and / or executed using instructions executable on tensor processing unit 182. As another example, DE-BBFP accelerator 186 may be configured by programming reconfigurable logic device 184. As another example, DE-BBFP accelerator 186 may be configured using hard-wired logic gates of neural network accelerator 180.

[0039] The DE-BBFP accelerator 186 may be programmed to execute a subgraph, an individual layer, or multiple layers of a neural network. For example, the DE-BBFP accelerator 186 may be programmed to execute operations for all or a portion of a layer of a NN. The DE-BBFP accelerator 186 may access local memory used to store weights, biases, input values, output values, forgetting values, state values, etc. The DE-BBFP accelerator 186 may have many inputs, and each input may be weighted by a different weight value. For example, the DE-BBFP accelerator 186 may generate a dot product of an input tensor with a programmed input weight for the DE-BBFP accelerator 186. In some examples, the dot product may be adjusted by a bias value before being used as an input to an activation function. The output of the DE-BBFP accelerator 186 may be stored in a local memory, where the output values ​​may be accessed and sent, for example, to a different NN processor core and / or neural network module 130 or memory 125. Intermediate values ​​in DE-BBFP may often be stored in smaller or more local memory, while values ​​that may not be needed until later in the training process may be stored in larger, less local memory (or storage devices such as on a solid-state drive (SSD) or hard drive) called "bulk memory". For example, during training forward propagation, once the activation values ​​for the next layer in a NN are calculated, those values ​​cannot be accessed until propagation through all layers is complete. Such activation values ​​may be stored in such bulk memory.

[0040] The neural network accelerator 180 may include a plurality 110 of DE-BBFPs 186 connected to each other via an interconnect (not shown). The interconnect may carry data and control signals between the individual DE-BBFP accelerators 186, a memory interface (not shown), and an input / output (I / O) interface (not shown). The interconnect may transmit and receive signals using electrical, optical, magnetic, or other suitable communication technologies and may provide communication connections arranged according to a number of different topologies, depending on the specific configuration desired. For example, the interconnect may have a crossbar, bus, point-to-point bus, or other suitable topology. In some examples, any one of the plurality of DE-BBFP accelerators 186 may be connected to any one of the other cores, while in other examples, some cores are connected to only a subset of the other cores. For example, each core may be connected to only its nearest 4, 8, or 10 neighboring cores. The interconnect may be used for transmitting input / output data to and from the DE-BBFP accelerators 186, and for transmitting control and other information signals to and from the DE-BBFP accelerators 186. For example, each of the DE-BBFP accelerators 186 may receive and transmit semaphores indicating the execution status of operations currently being executed by each respective DE-BBFP accelerator 186. Additionally, matrix and vector values ​​may be shared between the DE-BBFP accelerators 186 via the interconnect. In some examples, the interconnect is implemented as wires connecting the DE-BBFP accelerators 186 and a memory system, while in other examples, the core interconnect may include switches and / or routing components including circuits for multiplexing data signals onto the interconnect wires, active signal drivers and repeaters, or other suitable circuitry.In some examples of the disclosed technology, signals transmitted within and to / from neural network accelerator 180 are not limited to full-swing electrical digital signals, and neural network accelerator 180 may be configured to include differential signals, pulsed signals, or other suitable signals for transmitting data and control signals.

[0041] In some examples, the DE-BBFP enabled system 110 may include an optional DEBBFP emulator that emulates the functionality of the neural network accelerator 180. The neural network accelerator 180 provides functionality that may be used to convert data represented in full precision floating point format in the neural network module 130 to DE-BBFP format values. In some examples, the neural network accelerator 180 may perform operations using quantized DE-BBFP format values. Such functionality is described in further detail below.

[0042] The neural network module 130 may be used to specify, train, and evaluate neural network models using a tool flow that includes a hardware-agnostic modeling framework 131 (also referred to as a native framework or machine learning execution engine), a neural network compiler 132, and a neural network runtime environment 133. The memory includes computer-executable instructions for the tool flow that includes the modeling framework 131, the neural network compiler 132, and the neural network runtime environment 133. The tool flow may be used to generate neural network data 200 that represents all or a portion of a neural network model, such as the neural network model described below with respect to FIG. 2. It should be noted that although the tool flow is described as having three separate tools (131, 132, and 133), the tool flow may have fewer or more tools in various examples. For example, the functionality of the different tools (131, 132, and 133) may be combined into a single modeling and execution environment. In other instances where a neural network accelerator is deployed, such a modeling framework may not be included.

[0043] The neural network data 200 may be stored in memory 125, which may include local memory 126, typically implemented as static read only memory (SRAM), embedded dynamic random access memory (eDRAM), in latches or flip-flops in a register file, in a bounding box RAM, or other suitable structure, and bulk memory 127, typically implemented in a memory structure that is larger than the local memory 126 but often supports slower access. For example, the bulk memory may be off-chip DRAM, network accessible RAM, SSD drive, hard drive, or network accessible storage. Depending on the specific memory technology available, other memory structures may be used to implement the bulk memory, including the structures listed above for the local memory. The neural network data 200 may be represented in one or more formats. For example, the neural network data 200 corresponding to a given neural network model may have a different format associated with each respective tool of a tool flow. In general, neural network data 200 may include descriptions of nodes, edges, groupings, weights, biases, activation functions, and / or tensor values. As a specific example, neural network data 200 may include source code, executable code, metadata, configuration data, data structures, and / or files for representing a neural network model.

[0044] The modeling framework 131 may be used to define and use neural network models. As an example, the modeling framework 131 may include predefined APIs and / or programming primitives that may be used to specify one or more aspects of a neural network model. The predefined APIs may include both lower level APIs (e.g., activation functions, cost or error functions, nodes, edges, and tensors) and higher level APIs (e.g., layers, convolutional neural networks, recurrent neural networks, linear classifiers, etc.). "Source code" may be used as input to the modeling framework 131 to define the topology of the graph of a given neural network model. In particular, the APIs of the modeling framework 131 may be instantiated and interconnected within the source code to specify complex neural network models. Data scientists may create different neural network models by using different APIs, different numbers of APIs, and interconnecting those APIs in different ways.

[0045] In addition to the source code, the memory 125 may also store training data. The training data includes a set of input data to apply to the neural network model 200 and a desired output from the neural network model for each respective data set of input data. The modeling framework 131 may be used to train the neural network model with the training data. The output of the training is the weights and biases associated with each node of the neural network model. After the neural network model is trained, the modeling framework 131 may be used to classify new data that is applied to the trained neural network model. Specifically, the trained neural network model uses the weights and biases obtained from the training to perform classification and recognition tasks on data that was not used to train the neural network model. The modeling framework 131 may use the CPU 120 and a dedicated processor (e.g., GPU 122 and / or neural network accelerator 180) to execute the neural network model with improved performance compared to using only the CPU 120. In some examples, the performance may potentially achieve real-time performance for some classification tasks.

[0046] The compiler 132 analyzes the source code and data provided for the neural network model (e.g., examples used to train the model) and converts the model into a format that can be accelerated on the neural network accelerator 180, which is described in more detail below. Specifically, the compiler 132 converts the source code into executable code, metadata, configuration data, and / or data structures to represent the neural network model and memory as neural network data 200. In some examples, the compiler 132 can split the neural network model into portions that use the CPU 120 and / or GPU 122 (e.g., the neural network 200) and other portions that can be executed on the neural network accelerator 180 (e.g., subgraphs, individual layers, or multiple layers of the neural network). The compiler 132 can generate executable code (e.g., runtime modules) for executing the NN assigned to the CPU 120 and for communicating with the subgraphs, individual layers, or multiple layers of the neural network assigned to the accelerator 180. The compiler 132 can generate configuration data for the accelerator 180 that is used to configure accelerator resources to evaluate subgraphs assigned to any accelerator 180. The compiler 132 can create data structures for storing values ​​generated by the neural network model during execution and / or training, as well as for communication between the CPU 120 and the accelerator 180. The compiler 132 can generate metadata that can be used to identify subgraphs, edge groupings, training data, and various other information about the neural network model during runtime. For example, the metadata can include information for interfacing between different subgraphs or other portions of the neural network model.

[0047] The runtime environment 133 provides an executable environment or interpreter that can be used to train the neural network model during a training mode and can be used to evaluate the neural network model in a training, inference, or classification mode. During the inference mode, input data can be applied to the neural network model inputs and the input data can be classified according to the training of the neural network model. The input data can be archived data or real-time data.

[0048] The runtime environment 133 may include deployment tools that may be used to deploy or install all or part of the neural network on the neural network accelerator 180 during a deployment mode. The runtime environment 133 may further include a scheduler that manages the execution of different runtime modules and the communication between the runtime modules and the neural network accelerator 180. Thus, the runtime environment 133 may be used to control the flow of data between the nodes modeled on the neural network module 130 and the neural network accelerator 180.

[0049] In one example, the neural network accelerator 180 receives and returns normal precision values ​​150 from the neural network module 130. As shown in FIG. 1, the DE-BBFP accelerator 186 may perform most of its operations using DE-BBFP format floating point, and the interface between the DE-BBFP accelerator 186 and the neural network module 130 may use full precision values ​​to communicate information between the modules. The normal precision values ​​may be represented in 16-bit, 32-bit, 64-bit, or other suitable floating point format. For example, a portion of the values ​​representing the neural network may be received, including edge weights, activation values, or other suitable parameters for processing using the double exponential format. The normal precision values ​​150 are provided to a normal precision floating point to DE-BBFP converter 152, which converts the normal precision values ​​to double exponential format values. A double exponential floating point operation 154 may then be performed on the converted values. Suitable hardware for implementing the DE-BBFP processing unit 154 includes general-purpose processors, neural network accelerators, or reconfigurable logic devices such as field programmable gate arrays (FPGAs). In addition, for the SE-BBFP or DE-BBFP arrays generated by the accelerator 186, intermediate values ​​can be stored in memory (e.g., memory 135, storage 139, or other volatile or non-volatile memory in communication with the accelerator 186 or within the accelerator 210). Thus, successive matrix operations can be performed in the double exponential format. After one or more matrix operations, the double exponential format values ​​can then be converted back to normal floating-point format using the DE-BBFP to normal floating-point converter 156, which generates normal precision floating-point values. As a particular example, the DE-BBFP accelerator 186 may be used to accelerate a given layer of a neural network, where vector-vector, matrix-vector, matrix-matrix, and convolution operations may be performed using floating-point arithmetic in the SE-BBFP and DE-BBFP formats, and less computationally intensive operations (such as adding bias values ​​or calculating activation functions) may be performed using normal floating-point precision arithmetic. In some examples, the DE-BBFP format may be further quantized (e.g., by reducing the precision of the exponent and / or mantissa of a DE-BBFP element to fewer bits, by truncating or rounding values ​​to fewer bits).

[0050] The conversion between regular floating point and DE-BBFP performed by converters 152 and 156 is typically performed on a set of numbers represented as a vector or a multidimensional matrix. In some examples, additional regular precision operations 158, including operations that may be desirable in a particular neural network implementation, may be performed based on the regular precision format. This may include adding biases to one or more nodes of the neural network, applying a hyperbolic tangent function or other such sigmoid function, or a rectification function (e.g., a ReLU operation) to regular precision values ​​converted back from the DE-BBFP format.

[0051] In some examples, double exponential values ​​are used and stored only in the logic gates and internal memory of the neural network accelerator 180, and the memory 125 and storage 129 store only regular floating point values. Module Inputs, weights, and activations for the neural network model received from 130 may be converted to DE-BBFP, and results of operations performed on the neural network accelerator 180 may be converted back to normal floating point before passing the values ​​to the neural network model 130. Values ​​may be passed between the neural network model 130 and the neural network accelerator 180 using memory 125, storage device 129, or an input / output interface (not shown). In other examples, the emulator provides a full emulation of the DE-BBFP accelerator, including storing only one copy of the shared exponent and operating with a reduced mantissa width. Some results may differ with respect to a version in which the underlying operations are performed in normal floating point. For example, certain examples may check for underflow or overflow conditions for a limited quantized bit width (e.g., a 3-bit wide, 4-bit wide, or 5-bit wide mantissa).

[0052] The majority of the computational cost of a DNN is vector-vector, matrix-vector, and matrix-matrix multiplications and / or convolutions. These operations are quadratic in the input size, while operations such as bias addition and activation functions are linear in the input size. Thus, in some examples, the DE-BBFP format is only used for matrix-vector multiplication operations implemented on the neural network accelerator 180. In such examples, all other operations are performed in a normal precision format, such as float16. Thus, from a user or programmer's perspective, the DE-BBFP-enabled system 110 accepts and outputs normal precision float16 values ​​to / from the neural network module 130, and outputs float16 format values. All conversions to and from the bounding box floating point format can be hidden from the programmer or user. In some examples, the programmer or user may specify certain parameters for the DE-BBFP operations. In other examples, the DE-BBFP operations can utilize the bounding box floating point format to reduce computational complexity, as described below with respect to FIG. 3.

[0053] The neural network accelerator 180 is used to accelerate the evaluation and / or training of a neural network graph or subgraph, typically at increased speed and reduced latency not achieved when evaluating a subgraph using only the CPU 120 and / or GPU 122. In the illustrated example, the accelerator includes a tensor processing unit (TPU) 182, a reconfigurable logic device 184 (e.g., included in one or more FPGAs or programmable circuit fabrics), and / or a DE-BBFP accelerator 186, although any suitable hardware accelerator that models a neural network may be used. The accelerator 180 may include configuration logic that provides a soft CPU. The soft CPU may oversee the operation of the accelerated graph or subgraph on the accelerator 180 and manage communication with the neural network module 130. The soft CPU may also be used to configure logic and control the loading and storage of data from RAM on the accelerator, e.g., in a bounding box RAM in the FPGA.

[0054] In some examples, the parameters of the neural network accelerator 180 may be programmable. The neural network accelerator 180 may be used to prototype the training, inference, or classification of all or a portion of the neural network model 200. For example, the DE-BBFP parameters may be selected based on accuracy or performance results obtained by prototyping the network in the neural network accelerator 180. After a desired set of DE-BBFP parameters is selected, the model may be programmed into the accelerator 180 to perform further operations.

[0055] The compiler 132 and runtime 133 provide a high-speed interface between the neural network module 130 and the neural network accelerator 180. In effect, a user of the neural network model may be unaware that parts of the model are accelerated on a provided accelerator. For example, node values ​​are typically propagated in the model by writing tensor values ​​to a data structure that contains an identifier. The runtime 133 associates subgraph identifiers with accelerators, and the runtime 133 provides logic to translate messages to the accelerator and transparently write weight, bias, and / or tensor values ​​to the neural network accelerator 180 without programmatic intervention. Similarly, values ​​output by the neural network accelerator 180 are transparently sent back to the neural network module 130 using messages that contain an identifier of the receiving node at the server and a payload that contains values ​​such as weights, biases, and / or tensors that are sent back to the overall neural network model.

[0056] 2 illustrates a simplified topology of a deep neural network (DNN) 200 that may be used to perform enhanced image processing using the disclosed DE-BBFP implementation. One or more processing layers may be implemented using the disclosed techniques for SE-BBFP and DE-BBFP matrix / vector operations, including the use of one or more of the multiple neural networks DE-BBFP 186 in the DE-BBFP-enabled system 110 described above. It should be noted that the application of the neural network implementations disclosed herein is not limited to DNNs, but may also be used with other types of neural networks, such as convolutional neural networks (CNNs), including implementations with long short-term memory (LSTM) or gated recurrent units (GRUs), or other suitable artificial neural networks that may be adapted to use the DE-BBFP methods and apparatus disclosed herein.

[0057] The DNN 200 can operate in at least two different modes. Initially, the DNN 200 can be trained in a training mode and then used as a classifier in an inference mode. During the training mode, a set of training data can be applied to the input of the DNN 200, and various parameters of the DNN 200 can be adjusted such that upon completion of training, the DNN 200 can be used as a classifier. Training includes performing forward propagation of the training input data, calculating a loss (e.g., determining the difference between the output of the DNN and the expected output of the DNN), and performing backward propagation through the DNN to adjust the parameters of the DNN 200 (e.g., weights and biases). When the architecture of the DNN 200 is appropriate for classifying the training data, the parameters of the DNN 200 can converge and training can be completed. After training, the DNN 200 can be used in an inference mode. Specifically, training data or non-training data may be applied to the input of the DNN 200 and propagated forward through the DNN 200, such that the input data may be classified by the DNN 200.

[0058] As shown in FIG. 2, a first set of nodes 210 (including nodes 215 and 216) forms an input layer. Each node of the set 210 is connected to each node in a first hidden layer formed from a second set of nodes 220 (including nodes 225 and 226). The second hidden layer is formed from a third set of nodes 230 including node 235. The output layer is formed from a fourth set of nodes 240 (including node 245). In the example 200, the nodes of a given layer are fully interconnected to the nodes of its adjacent layer(s). In other words, a layer can include nodes that have inputs in common with other nodes of the layer and / or provide outputs to a common destination of the other nodes of the layer. In other examples, a layer can include nodes that have a subset of inputs in common with other nodes of the layer and / or provide outputs to a subset of common destinations of the other nodes of the layer.

[0059] During forward propagation, each of the nodes generates an output by applying a weight to each input generated from the preceding node and collecting the weights to generate an output value. In some examples, individual nodes may have an activation function (σ) and / or a bias (b) applied to them. In general, a suitably programmed processor or FPGA may be configured to implement the nodes in the illustrated neural network 200. In some example neural networks, the output function f(n) of a hidden combination node n may generate an output that is mathematically expressed as:

number

[0060] A given neural network may contain thousands of individual nodes, and therefore performing all of the calculations for the nodes in normal precision floating point may be computationally expensive. Implementations for more computationally expensive solutions may include larger and more energy-consuming hardware than implementations for less computationally expensive solutions. However, performing operations using DE-BBFP floating point may potentially reduce the computational complexity of the neural network. In some cases, a simple implementation using only DE-BBFP floating point may significantly reduce the computational complexity, but the implementation may have difficulty converging during training and / or correctly classifying input data due to errors introduced by DE-BBFP. However, the double exponential floating point implementation disclosed herein may potentially increase the accuracy of some calculations while also providing the benefit of reduced complexity associated with the double exponential floating point format.

[0061] DNN 200 may include nodes that perform operations in DE-BBFP floating point. As a particular example, the output function f(n) of hidden combinational node n may generate an output that is mathematically expressed as:

number

[0062] Neural networks can be trained and retrained by adjusting the component values ​​of the output function f(n). For example, the weights w for the nodes i By adjusting the weights and biases b, the behavior of the neural network is adjusted by a corresponding change in the network's output tensor values. For example, a cost function C(w,b) can be used during backpropagation to find appropriate weights and biases for the network, where the cost function can be mathematically written as:

number

[0063] Examples of suitable applications for such neural network DE-BBFP implementations include, but are not limited to, performing image recognition, performing speech recognition, classifying images, translating speech into text and / or other languages, facial recognition or other biometric recognition, natural language processing, automatic language translation, query processing in search engines, automatic content selection, analyzing emails and other electronic documents, relationship management, biomedical informatics, identifying candidate biomolecules, providing recommendations, or other classification and artificial intelligence tasks.

[0064] A network accelerator (such as network accelerator 180 of FIG. 1) may be used to accelerate the computations of DNN 200. As an example, DNN 200 may be divided into different subgraphs or network layers that may be accelerated individually. As a specific example, each of layers 210, 220, 230, and 240 may be a subgraph or layer that is accelerated using the same accelerator or a different accelerator. The computationally expensive computations of a layer may be performed using DE-BBFP format, and the cheaper computations of a layer may be performed using normal precision floating point. Values ​​may be passed from one layer to another using normal precision floating point. By accelerating a group of computations for all nodes in a layer, some of the computations may be reused and the computations performed by the layer may be reduced compared to accelerating individual nodes.

[0065] In some examples, a set of parallel multiply-accumulate (MAC) units in each convolutional layer can be used to speed up the computation. Parallel multiplier units can also be used in the fully connected dense matrix multiplication stages. A parallel set of classifiers can also be used. Such parallelization methods have the potential to further speed up the computation, at the cost of increased control complexity.

[0066] As will be readily appreciated by those of ordinary skill in the art having the benefit of this disclosure, the application of neural network implementations, whether alone or in combination or subcombination with one another, may be used for different aspects of using neural networks. For example, the disclosed implementations may be used to implement neural network training via gradient descent and / or backpropagation operations for neural networks. Additionally, the disclosed implementations may also be used for evaluation of neural networks.

[0067] IV. Exemplary Conversion and Use of SE-BBFP and DE-BBFP Format Matrices 3 is a diagram 300 illustrating an example of determining row indices when converting normal floating-point format to DE-BBFP format, as may be implemented in certain examples of the disclosed technology. For example, input tensors for a neural network represented as normal floating-point numbers (e.g., 32-bit or 16-bit floating-point format) may be converted to the bounding box floating-point format shown.

[0068] As shown, a matrix of numbers 310 in a normal floating-point format is represented such that each number, such as number 315 or number 316, includes a sign, an exponent, and a mantissa. For example, in the case of the IEEE 754 half-precision floating-point format, the sign is represented using one bit, the exponent is represented using five bits, and the mantissa is represented using ten bits. When the floating-point format numbers 310 in the neural network model 200 are converted to a set of numbers in the DE-BBFP format, there is one exponent value that is shared by all of the numbers in a particular row of the illustrated group. Thus, as shown, the set of SE-BBFP numbers 320 share a single exponent value (330, 331, 334, etc.) for each row, while each number in the set includes a sign and a mantissa. However, because the set of numbers shown have different exponent values ​​in the floating-point format, the respective mantissas of each number may be shifted (e.g., shifted mantissas 345 and 346) so that the same or similar numbers are represented in the SE-BBFP format. The illustrated matrix has 5 rows and 2 columns, however, typically matrices used for the disclosed applications will have the same number of rows and columns (eg, 8×8, 16×16, etc.).

[0069] The shared row exponent 330 (HEXP1) for the first row of matrix 310 is calculated as the maximum of the exponent values ​​in the first row: HEXP1=MAX(EXP(0,0)...EXP(0,1). The shared row exponent 331 (HEXP2) for the second row of matrix 310 is calculated as the maximum of the exponent values ​​in the second row: HEXP2=MAX(EXP(1,0)...EXP(1,1). The remaining three row exponents are calculated similarly.

[0070] In some examples, the shared row exponent 330-334 is selected to be the largest exponent from among the original normal precision numbers in the neural network model 200. In other examples, the shared row exponent may be selected differently, for example, by selecting an exponent that is the average or median of the normal floating-point exponents in a particular row, or by selecting an exponent that maximizes the dynamic range of values ​​stored in the mantissa when the number is converted to SE-BBFP / DE-BBFP number format. Note that if the shared exponent and the original floating-point exponent of the value are not the same, some bits of the shifted mantissa may be lost. This occurs because the mantissa is shifted to correspond to the new shared exponent.

[0071] There are several possible choices for which values ​​in a bounding box floating-point tensor share an exponent. The simplest choice is for all values ​​in a row or column of a matrix or matrix tile (e.g., a 16x16 tile) to share an exponent. However, sharing the exponent at a finer granularity can reduce errors, as it increases the likelihood that the DE-BBFP numbers use shared exponents that are closer to the exponents of the original regular floating-point format. Thus, the loss of precision due to dropping mantissa bits (when shifting the mantissa to correspond to the shared exponent) can be reduced.

[0072] In some examples, the computational cost of matrix-vector multiplication may be further reduced by reducing the width of the mantissa. A large range of values ​​with a shared common exponent may be represented with only a few bits of the mantissa. For example, in a representation with a 4-bit mantissa and a 5-bit exponent, values ​​in the range [2 -14 0.001 2 ,2 15 1.111 2 ], or about [2 -17 ,2 16 In contrast, 4-bit fixed-point numbers can be represented in the range [0001 2 , 1111 2], or about [2 0 , 2 4 ] can only represent values ​​within

[0073] 4 is a diagram 400 illustrating an example of determining column indices when converting normal floating-point format to DE-BBFP format, as may be implemented in certain examples of the disclosed technology. For example, input tensors for a neural network represented as normal floating-point numbers (e.g., 32-bit or 16-bit floating-point format) may be converted to the bounding box floating-point format shown.

[0074] As shown, the numbers 310 in the normal floating-point format described above with respect to FIG. 3 are represented such that each number, e.g., number 315 or number 316, includes a sign, an exponent, and a mantissa. When the floating-point formatted numbers 310 in the neural network model 200 are converted to a set of numbers in the DE-BBFP format, there is one exponent value shared by all of the numbers in a particular column of the illustrated group. Thus, as shown, the set of SE-BBFP numbers 320 share a single exponent value (430 and 431) for each column, while each number of the set of numbers includes a sign and a mantissa. However, because the set of numbers shown have different exponent values ​​in the floating-point format, the respective mantissas of each number may be shifted (e.g., shifted mantissas 445 and 446) so that the same or similar numbers are represented in the SE-BBFP format. As described in more detail below, once the conversion to DE-BBFP is complete, the shift of the mantissa is determined by the maximum value of the exponent for each element's row and column.

[0075] The shared row exponent 430 (VEXP1) for the first row of matrix 310 is calculated as the maximum of the exponent values ​​in the first row: VEXP1=MAX(EXP(0,0)...EXP(4,0)). The shared row exponent 331 (VEXP2) for the second row of matrix 310 is calculated as the maximum of the exponent values ​​in the second row: VEXP2=MAX(EXP(0,1)...EXP(4,1)). In this example, there are only two columns for simplicity of explanation, but additional column shared exponents would be calculated similarly.

[0076] In some examples, the shared column exponents 430 and 431 are selected to be the largest exponents from among the original normal precision numbers in the neural network model 200. In other examples, the shared row exponents may be selected differently, such as by selecting an exponent that is the average or median of the normal floating-point exponents in a particular row, or by selecting an exponent that maximizes the dynamic range of values ​​stored in the mantissa when the number is converted to SE-BBFP / DE-BBFP number format. Note that if the shared exponent and the original floating-point exponent of the value are not the same, some bits of the shifted mantissa may be lost. This occurs because the mantissa is shifted to correspond to the new shared exponent.

[0077] There are several possible choices for which values ​​in a bounding box floating-point tensor share exponents. The simplest choice is for all values ​​in a row or column of a matrix or matrix tile (e.g., a 16x16 tile) to share an exponent. However, sharing exponents at a finer granularity can reduce errors, as it increases the likelihood that DE-BBFP numbers use shared exponents that are closer to the exponents of the original regular floating-point format. Thus, the loss of precision due to dropping mantissa bits (when shifting the mantissa to correspond to the shared exponent) can be reduced.

[0078] In block-based floating-point matrix multiplication applications where matrices can be used as either the left or right operand, using the DE-BBFP representation allows the matrix data to be stored efficiently in a compact manner. In such cases, the mantissa for each matrix element in this representation is calculated as follows:

[0079] Consider the following simple 2x2 array (shown using decimal values). For ease of explanation, the signs of the elements are ignored:

number

number

number

[0080] To store the matrix in 8-bit SE-BBFP format along the rows, it is determined that the maximum exponent in each row is [135,133]. The single exponent representation of the matrix (along the horizontal) is the common exponent ordering of [135,133], and thus the mantissa matrix is

number

number

[0081] Similarly, to store a matrix using SE-BBFP along the columns, the vertical exponent is [130,135], and the single exponential representation of the matrix (along the columns) is

number

number

[0082] Note that the resulting matrices will have different precision, depending on the block-floating direction (along the rows or along the columns) and the correlation between elements in that direction.

[0083] To convert the SE-BBFP representation back to FP32, the normalized mantissa is calculated for each element using LS (left shift amount) as follows:

number

number

[0084] And the corresponding exponent is

number

[0085] For the double exponential format DE-BBFP, r Let (i,j) be the mantissa for the (i,j)th element of the matrix for single-exponent BBFP along a row. This is how the matrix is ​​stored when it is used as the left operand of a matrix multiplication operation. S cLet (i,j) be the mantissa for the (i,j)th element of a matrix for single-exponent BBFP along a column. This is how the matrix is ​​stored when it is used as the right operand of a matrix multiplication operation.

[0086] Then, the DE-BBFP mantissa S de (i,j) can be expressed as follows:

[0087] S de (i,j)=max{S r (i,j),S c (i,j)} The sign of each element in DE_BBFP is the same as the sign of the corresponding element in the SE_BBFP matrix. de (i,j) being the larger of the two mantissas preserves the precision of the more accurate mantissa.

[0088] Conceptually, the following C data structure can be used to store a DE-BBFP tile with 8-bit precision mantissa: struct DE_BBFP_1_8_TILE { UINT8_hexp

[16] ; UINT8_vexp

[16] ; BOOL_sign

[16]

[16] ; UINT8_significand

[16]

[16] ; };

[0089] A 16x16 tile with elements represented in the 16-bit floating-point format bfloat16 requires 512 bytes of storage. In one example, a single-exponent (SE-BBFP) tile uses 16 bytes for its shared exponent, 256 bytes for the mantissa, and 32 bytes (256 bits) for the sign, totaling 304 bytes. In one example, a double-exponent (DE-BBFP) tile uses 32 bytes for the shared exponent (vectors HEXP and VEXP), 256 bytes for the mantissa, and 32 bytes (256 bits) for the sign value. Thus, the total storage requirement for the DE-BBFP format in this example is 320 bytes, which is approximately 62.5% of the memory occupied by a tile in bfloat16 format.

[0090] In the above example, the DE-BBFP mantissa is obtained from the two SE-BBFP representations as follows:

number

[0091] V. DE-BBFP Tile Hardware Examples 5 is a block diagram 500 illustrating a high-level hardware architecture for storing and using DE-BBFP matrices as may be implemented in certain examples of the disclosed technology. For example, the DE-BBFP-enabled system 110 described above may be used to accelerate matrix operations for machine learning applications.

[0092] As shown, the memory includes a 16×16 element array, or tile. In this example, there are several columns, e.g., a first column 511 and a second column 512. The maximum value of the exponents for the 16 elements in each column 511, 512 is determined and the maximum exponent is stored in a common column exponent register 520. A first common exponent 521 corresponds to the column 511 of elements, and a second common exponent 522 corresponds to the second column of elements.

[0093] The same memory is shown at 530, in this case grouped by row for the same memory 510. As shown, a first row 531 of 16 values ​​is evaluated to find the maximum value and the maximum exponent 541 is stored in a common row exponent register 540. Similarly, a second row of 16 values ​​is evaluated to find the maximum value and the maximum exponent 541 is stored in a common row exponent register 540. line 532 is evaluated to find its maximum value, which is stored in the common row exponent register 540 at 542 .

[0094] When performing key operations using the DE-BBFP matrix, the maximum or minimum common exponent CEXP for each element can be found using a comparator normal 550. The output of the comparator 550 can be used to shift the mantissas for each of the elements stored in the memory 510.

[0095] 6 is a block diagram 500 illustrating an alternative example of a high-level hardware architecture for storing and using DE-BBFP matrices as may be implemented in certain examples of the disclosed technology. For example, the DE-BBFP-enabled system 110 described above may be used to accelerate matrix operations for machine learning applications.

[0096] Similar to the example of FIG. 6, the hardware includes memory that stores 16×16 element arrays, or tiles. However, in this example, each bounding box includes two columns. For example, the first two columns 611 are the first bounding box, and the second pair of columns 612 are assigned as the second bounding box. The maximum value of the exponents for the 32 elements in each pair of respective columns 611, 612 is determined, and the maximum exponent is stored in an exponent register 620. A first common exponent 621 corresponds to the first two columns 611 of elements, and a second common exponent 622 corresponds to the second column 612 of elements.

[0097] The same memory is shown at 630, this time showing grouping by row for the same memory 610. As shown, a first pair of row 631 of 32 values ​​is evaluated to find the maximum value and the maximum exponent 641 is stored in a common row exponent register 640. Similarly, a second pair of row 632 of 32 values ​​is evaluated to find its maximum value and stored in common row exponent register 640 at 642.

[0098] When performing key operations using the DE-BBFP matrix, the maximum or minimum common exponent CEXP for each element can be found using a comparator normal 650. The output of the comparator 650 can be used to shift the mantissas for each of the elements stored in the memory 610.

[0099] VI. Exemplary Methods of Matrix Operations Using SE-BBFP and DE-BBFP Matrices 7 is a block diagram 700 outlining an example of performing matrix operations using SE-BBFP and DE-BBFP matrices, as may be implemented in certain examples of the disclosed technology. For example, the DE-BBFP-enabled system 110 described above may be used to perform the operations described.

[0100] As shown, there are three tiles A, B, and C of 16×16 regular floating point values ​​stored in memory at 710, 720, and 730. The first matrix operation performed is an A×B matrix multiplication. Each floating point matrix is ​​converted to DE-BBFP format. The A matrix is ​​converted to DE-BBFP format matrix 740 and stored in memory or streamed; similarly, the B matrix is ​​converted to DE-BBFP format matrix 750.

[0101] One or more tensor operations are performed using the BBFP processing unit 760. The A matrix is ​​the left operand and is therefore converted from DE-BBFP to have a common row index stored in SE-BBFP format in memory 740 as shown. The B matrix is ​​the right operand and is therefore converted from DE-BBFP to have a common column index stored in SE-BBFP format in memory 750. The two matrices A and B are multiplied and the result is stored in memory 765 in normal floating-point format. The floating-point matrix stored in memory 765 may then be converted to DE-BBFP format for storage or subsequent tensor operations.

[0102] In other examples, since it is known that the A×B result will be the left operand for another matrix multiplication, the result is converted to SE-BBFP format with a common row index. If the use of the result is not known, the result is converted to DE-BBFP format and the appropriate row or column index can be used later. The input C matrix in memory 730 is converted to DE-BBFP format 780 with a common column index. One or more tensor operations are performed using BBFP processing unit 760. Here, the result of the second matrix multiplication (A×B)×C is used as a normal floating point format and stored in memory 795. In some examples, since the C matrix is ​​the right operand for the matrix multiplication, the C matrix is ​​converted directly to SE-BBFP instead of DE-BBFP. In some examples, one or more of tiles A, B, or C may be converted directly to SE-BBFP before being used for the matrix operation. In some examples, the BBFP processing unit 790 is the same unit as the BBFP processing unit 760, while in other examples, separate processing units are used.

[0103] Those skilled in the art having the benefit of this disclosure will readily appreciate that additional operations can be performed using matrices stored in SE-BBFP and DE-BBFP formats, with the results being subsequently converted to normal floating-point format. Additionally, any of the SE-BBFP matrices can also be stored as DE-BBFP format matrices.

[0104] VII. Exemplary Double Exponential Matrix Processing Architecture 8 is a block diagram outlining a double exponential matrix processing architecture as may be implemented in certain examples of the disclosed technology. For example, a DE-BBFP-enabled system 110 may implement a DE-BBFP processing unit 154 using the architecture shown in FIG. 8. Suitable hardware for implementing the DE-BBFP processing unit 154 may include a general-purpose processor, a neural network accelerator, or a combination of both. Ta, or field programmable gate arrays (FPGAs) Reconfigurable Logical Device include In some examples, the DE-BBFP processing unit 154 includes dedicated circuitry including exponent registers, memory, shifters, arithmetic units, logic units, etc., for performing the disclosed DE-BBFP operations.

[0105] As shown in FIG. 8, a first memory AMEM 810 stores data for DE-BBFP matrices received or converted from another format (e.g., converted from normal floating point). Before being used for matrix / tensor operations, a converter 815 converts the DE-BBFP data into an SE-BBFP format compatible with the operations to be performed. When an AMEM matrix is ​​used as a left matrix operand, the matrix is ​​converted to an SE-BBFP format with a common row index; conversely, when used as a right matrix operand, the matrix is ​​converted to an SE-BBFP format with a common column index. Depending on whether a given row index or column index is greater, the mantissa is shifted or maintained. For example, when a row index is used and the row index is greater than or equal to the column index, the associated mantissa is not shifted. On the other hand, when the row index is less than the column index, the mantissa is shifted accordingly. The SE-BBFP matrices generated by the converter 815 may be stored in a temporary memory or buffer, or may be directly streamed to be used in tensor operations performed using the BBFP processing unit 830.

[0106] A second memory BMEM 820 stores data for the DE-BBFP matrix in a manner similar to the AMEM matrix 810. These values ​​are converted to SE-BBFP values ​​before being consumed for the tensor operation 830 in a manner similar to the converter 815, depending on whether the BMEM value is used as a left or right operand.

[0107] The BBFP processing unit 830 performs one or more matrix operations using the SE-BBFP matrices generated by the converters 815, 825. Any suitable matrix operations may be performed using the (SE-BBFP converted) DE-BBFP matrices. For example, matrix multiplication, as well as addition, subtraction, inverse matrix or determinant calculations may be performed. Suitable hardware for implementing the BBFP processing unit 830 includes general purpose processors, neural network accelerators, or reconfigurable logic devices such as field programmable gate arrays (FPGAs). In some examples, the DE-BBFP processing unit 154 includes dedicated circuitry including exponent registers, memory, shifters, arithmetic units, logic units, etc. for performing the disclosed DE-BBFP operations.

[0108] The output of the DE-BBFP operation is in normal floating-point format since all output elements have new mantissas and exponents generated by the BBFP processing unit. The floating-point output can be converted to DE-BBFP, for example, using techniques similar to process blocks 1010, 1020, and 1030 described above with respect to FIG. 10. The converted DE-BBFP result (CMEM) is stored in memory 850. The CMEM can then be used for additional matrix operations by converting it to SE-BBFP and using it as a left or right operand for tensor operations.

[0109] At some point, the result data may be converted to a normal floating point format for use with hardware configured to operate in the normal floating point domain. For example, the floating point output may be stored in memory 860 in a normal floating point format. In some examples, the output is converted to a fixed point or integer format. The floating point output may be generated by a DE-BBFP to FP converter 865. For example, the operations described below with respect to FIG. 12 may be used to perform the conversion to a normal floating point format. Suitable hardware for implementing the DE-BBFP to FP converter 865 includes a general purpose processor, a neural network accelerator, or both reconfigurable logic devices such as a field programmable gate array (FPGA).

[0110] In other examples, normal floating-point output is not generated using converter 865, but rather uses the output of the tensor operation generated by BBFP processing unit 830. In some examples, further transformations are performed. For example, in some cases, the tensor operation output has an 8-bit mantissa and a 5-bit exponent and may be padded or shifted to be properly formatted in the desired floating-point output format. For example, bfloat16 has an 8-bit precision mantissa and exponent, while the float32 format has a 24-bit precision mantissa and an 8-bit exponent. Additionally, special cases (e.g., NaN, underflow / overflow, subnormal, or other special cases) may have different representations in the target floating-point output format.

[0111] VIII. Exemplary Methods of Neural Network Training 9 is a flow diagram illustrating a method 900 for training a neural network using a model having a DE-BBFP format, as may be implemented in certain examples of the disclosed technology. For example, training the neural network may include iterating through a set of training data, and method 900 may be used to update parameters of the neural network during a given iteration of the training data. As an example, method 900 may be performed by a DE-BBFP-enabled system, such as DE-BBFP-enabled system 110 of FIG. 1.

[0112] At process block 910, parameters such as weights and biases of the neural network may be initialized. As an example, the weights and biases may be initialized to random normal precision floating point values. As another example, the weights and biases may be initialized to normal precision floating point values ​​calculated from a previous training set. The initial parameters may be stored in a memory or storage device of the DE-BBFP enabled system. In one example, the parameters may be stored as DE-BBFP values, which may reduce the amount of storage used to store the initial parameters.

[0113] At process block 920, the input values ​​of the neural network may be forward propagated through the neural network. The input values ​​of a given layer of the neural network may be the output of another layer of the neural network. Values ​​may be passed between layers, from the output of one layer to the input of the next layer, using regular precision floating point numbers. The output function of layer i may include terms that can be mathematically described as: y i =f(DE(y i-1 ),DE(W i )) (Formula 5) where y i-1 is the output from the layer that provides the input to layer i, and W iis the weight tensor for layer i, f() is the layer's forward function, and DE() is the conversion function to DE-BBFP. The output function of a layer may be a floating-point value produced by the DE-BBFP operation represented as f(), or alternatively, the output function may include additional terms, such as the addition of an activation function or bias, performed using regular precision floating point (before conversion to DE-BBFP) or using DE-BBFP floating point (after conversion to DE-BBFP). In general, the inputs, outputs, and parameters of a layer are tensors. Typically, the inputs, outputs, and parameters of a layer are vectors or matrices. The DE-BBFP conversion function converts regular precision floating point values ​​to DE-BBFP values. A particular DE-BBFP format may be selected to take into account the type of input data and the type of operation performed by layer i. For example, y i and W i is a two-dimensional matrix and the output function is y i-1 and W i If it includes a term that takes the cross product with y i-1 The function for y i-1 You can use a bounding box that contains the line or part of the line, W i The function for W i A bounding box that contains the sequence or a part of the sequence of can be used. If the bounding boxes are chosen to follow the sequence of operators, the computation can be more efficient, thus making the hardware implementation smaller, faster and more energy efficient.

[0114] At process block 930, a portion of the neural network, such as a layer that has just been forward propagated to the next layer of the neural network, may be compressed and stored in memory. For example, the activation values ​​calculated as part of the forward propagation as described above in process block 920 may be compressed and stored in memory. This compression may be expressed mathematically as: y ci =C(y i ) (Equation 6) where y iare the values ​​produced by forward propagation through the layer in process block 920, C() is an optional additional compression function (which may include multiple compression operations), and y ci is the compressed value to be stored in memory. In other instances, a compression operation is not performed and thus the operations described in process block 930 are not performed. In such a case, a decompression operation as described below with respect to process block 950 is unnecessary and would not be performed.

[0115] In some examples, an additional quantization function may further compress the DE-BBFP values ​​into a smaller quantized format than that used in the DE-BBFP layer. In such cases, the compressed activation values ​​are represented in a second DE-BBFP format that may differ from the first DE-BBFP used to perform the forward propagation calculation (at least one of the following: having a different mantissa format or having a different exponent format). For example, if forward propagation was performed using activation mantissa values ​​represented in 9-bit format, these values ​​may be converted to 4-bit format by truncating or rounding the mantissa. As another example, activation value exponents, including shared exponents in the DE-BBFP format, may be converted from 7-bit format to 5-bit format. Values ​​may be converted between the formats used by any suitable technique. For example, truncation or rounding of the exponent may be performed, along with any mantissa shifts performed to compensate for the adjusted exponent. In some examples, the conversion may be performed using a table lookup or other technique.

[0116] In some examples, additional compression may be applied to the compressed bounding box floating point format before storing in memory. Examples of suitable techniques for further compressing the activation values ​​in the compressed format include entropy compression (e.g., Huffman coding), zero compression, run-length compression, compressed sparse row compression, or compressed sparse column compression.

[0117] At process block 940, the loss of the neural network may be calculated. For example, the output y of the neural network may be compared to the expected output ^y of the neural network. The difference between the output and the expected output may be an input to a cost function used to update the parameters of the neural network.

[0118] In process block 950, the activation values ​​stored in memory are decompressed for backpropagation, and in particular for the calculation of the output error terms used in backpropagation for a particular layer. The method may iterate through each layer, decompressing the activation values ​​for each layer, performing backpropagation for the layer, and then decompressing the activation values ​​for the preceding layer. This decompression can be expressed mathematically as: y i =C -1 (y ci ) (Formula 7) where y ci is the compressed value retrieved from memory, and C -1 () is an optional decompression function (which can include multiple compression operations) that is the inverse of the optional compression function C(), and y i are the values ​​generated by forward propagation through the layers in process block 920. For example, after forward propagation has been completed for all layers and the neural network as described above with respect to process blocks 920 and 930, and the loss has been calculated as described above in process block 940, values ​​are propagated backward through the neural network, typically starting at the output layer of the neural network. Additionally, if additional compression was applied prior to storage in memory, such as entropy compression, zero compression, run-length encoding, compressed sparse row compression, or compressed sparse column compression, these operations may be reversed before performing backward propagation through the layers in process block 960.

[0119] At process block 960, the neural network loss may be backpropagated through the neural network. During backpropagation, the output error term ∂y and the weight error term ∂W may be calculated. The output error term may be written mathematically as: ∂y i-1 =g(DE(∂y i ),DE(W i )) (Equation 9) Here, ∂y i-1 is the output error term from the layer after layer i, and W i is the weight tensor for layer i, g() is the inverse function of the layer, and DE() is the double exponential transformation function. The inverse function g() is i-1 or the inverse function of f() with respect to a portion of the gradient function.

[0120] The weighting error term ∂W can be written mathematically as: ∂W i =h(DE(y i ),DE(∂y i )) (Equation 9) Here, ∂W i is the weight error term for layer i, and ∂y i is the output error term for layer i, and y i is the output for layer i, h() is the inverse function of the layer, and DE() is the double exponential format function. The inverse function h() is i-1 , or the inverse function of f() for the portion of Equation 9 of the weight error. The weight error term may include additional terms that are implemented using regular precision floating point.

[0121] In process block 970, the parameters for each layer may be updated. For example, the weights for each layer may be updated by calculating new weights based on successive iterations of training. As an example, the weight update function may be written mathematically as follows: W i =W i +η×∂W i(Formula 10) Here, ∂W i is the weight error term for layer i, η is the learning rate for layer i of the neural network, and W i is the weight tensor for layer i. In one example, the weight update function may be implemented using normal precision floating point.

[0122] IX. High-Level Description of the Creation and Use of DE-BBFP Matrix Operations Figure 10 is a flow chart 1000 outlining an exemplary method for converting a regular floating-point matrix to a DE-BBFP matrix and using the matrix in a matrix operation. The DE-BBFP-enabled system 110 described above with respect to Figure 1 is one example of a system that may be used to perform the illustrated method. For example, the illustrated method may be implemented using the neural network operations described above in Figures 2 and 8, including the use of the number format described above with respect to Figure 3. As will be readily appreciated by those skilled in the art having the benefit of this disclosure, various variations of the DE-BBFP system and the double exponential format may be adapted to perform the illustrated method.

[0123] In process block 1010, the maximum row exponent (HEXP) and column exponent (VEXP) for each row and column in the normal floating-point format input matrix are determined. Thus, the mantissa m i,j , index e i,j , and the sign bit s i,j For an N×M input matrix with

[0124] In process block 1020, the mantissas are determined for the DE-BBFP matrix elements. For each element, a common exponent is selected that is the minimum of the row and column exponents for the respective element. The regular floating-point mantissa is scaled by the difference between the common exponent and the regular floating-point exponent. Optionally, the operation at 1020 may include, at 1025, selecting a common exponent by calculating the smaller exponent for either the row or column of the bounding box of an element minus a left shift amount to calculate a normalized mantissa for the respective row or column of the bounding box of an element.

[0125] At process block 1030, the common exponent and mantissa determined at process blocks 1010 and 1010 are stored in a memory or computer readable storage medium. As discussed above, the DE-BBFP format can provide substantial memory reduction over normal precision floating point values. In some examples, the DE-BBFP matrix values ​​may be further reduced by quantizing the exponent and / or mantissa or by lossless compression techniques.

[0126] At process block 1040, a matrix operation is performed using the DE-BBFP matrix. The DE-BBFP matrix is ​​converted to SE-BBFP before performing the matrix operation. For example, when the DE-BBFP is used as a left matrix, row exponents are used; conversely, when the DE-BBFP is used as a right matrix in an operation, column exponents are used. When used as a left matrix, the SE-BBFP exponents are zero-based. or When used as a right matrix, the SE-BBFP exponent is calculated as the maximum of either zero or the column index minus the row index. The SE-BBFP mantissa is calculated as the double exponent mantissa shifted right by the difference between the row / column indexes. Further details regarding converting DE-BBFP data to SE-BBFP data and performing the operations are described above with respect to FIG. 8.

[0127] Any suitable matrix operation may be performed using the DE-BBFP matrices (converted to SE-BBFP). For example, matrix multiplication, as well as addition, subtraction, inverse matrix or determinant calculations may be performed. Some operations may be performed with the matrix data stored in the DE-BBFP / SE-BBFP format before converting to normal floating-point values. For example, for machine learning / deep learning applications, a series of training or inference actions may be performed in the DE-BBFP domain before converting the values ​​to normal floating-point values.

[0128] In process block 1050, the DE-BBFP results from performing one or more matrix operations in process block 1040 are converted to regular floating point values, for example in float32 or bfloat16 format. In other examples, the floating point results generated by performing the DE-BBFP operations may be output directly as the floating point format. In some examples, additional conversions of the regular floating point values ​​are performed depending on the particular output format selected.

[0129] X. Conversion of normal floating-point matrix to DE-BBFP Figure 11 is a flow chart 1100 outlining an exemplary method for converting a regular floating-point matrix to a DE-BBFP matrix. The DE-BBFP-enabled system 110 described above with respect to Figure 1 is one example of a system that may be used to perform the illustrated method. For example, the illustrated method may be implemented using the neural network operations described above in Figures 2 and 9, including the use of the number format described above with respect to Figure 3. As will be readily appreciated by those of ordinary skill in the art having the benefit of this disclosure, various variations of the DE-BBFP system and the double exponential format may be adapted to perform the illustrated method.

[0130] In process block 1110, the maximum row exponent (HEXP) and column exponent (VEXP) for each row and column in the input matrix in normal floating point format are determined. Thus, the mantissa m i,j , index e i,j , and the sign bit s i,j For an N×M input matrix with V, HEXP will be a vector of length N and VEXP will be a vector of length M. These vectors can be expressed more formally as:

number

[0131] At processing block 1120, the implicit leading bit of the floating-point mantissa is restored; the leading bit is represented explicitly. If the floating-point mantissa is a normal floating-point value, then a 1 bit becomes the first bit of the double exponent mantissa. If the floating-point mantissa is a subnormal value, then a 0 (zero) bit becomes the first bit of the double exponent mantissa. In some examples, a subnormal mantissa is indicated when the corresponding exponent of a number has the smallest value representable in that format. In some examples, this subnormal condition may be indicated when all of the exponent bits for a number are 0.

[0132] Mantissa s i,j and its exponent e i,j The sign of is the same as the sign of its corresponding floating-point element. The mantissa with the leading bit restored is cast to an unsigned data type.

[0133] In process block 1130, the input matrix x i,j For each element in, the common exponent CEXP[i],[j] is determined as MIN(HEXP[i],[j]).

[0134] In process block 1140, the mantissa for a DE-BBFP format matrix is ​​calculated by dividing the common exponents CEXP[i],[j] of the elements by their ordinary exponents e i,j The difference between

[0135] At process block 1150, the mantissa is cast to the desired width in the DE-BBFP representation. In some examples, the mantissa is rounded up or down. In some examples, the mantissa is rounded by a rounding halfway from zero technique. In some examples, the mantissa is truncated.

[0136] There are several special cases that can be handled when converting regular floating-point to DE-BBFP. In some cases, when the input floating-point number is NaN (not a number), the output DE-BBFP number is also NaN. In some cases, a NaN in DE-BBFP can be represented with all 1's in its shared exponent bits, in which case the mantissa bits are don't care since the number is a NaN.

[0137] As another special case, when the input floating-point number is infinite, the output value in DE-BBFP is represented as NaN, with all of its shared exponent bits set to 1 and the mantissa bits being don't care.

[0138] As another special case, a DE-BBFP element is considered to be NaN when its row index is 255 and its column index is 256.

[0139] For denormalized values, the implied leading bit of the mantissa is 0 when the elements' common exponents are all zero, and 1 otherwise.

[0140] When intermediate rounding techniques away from zero are applied, it is possible for the mantissa to overflow. In such cases, the element exponents are not adjusted; rather, the largest possible mantissa is set to the element's value. To understand this, consider the same 2x2 matrix as described above for the SE-BBFP example.

number

number

number

number

[0141] XI. Converting DE-BBFP matrices to normal floating-point format FIG. 12 is a flow chart 1200 outlining an exemplary method of converting a DE-BBFP matrix to a normal floating-point matrix. The DE-BBFP-enabled system 110 described above with respect to FIG. 1 is an example of a system that may be used to perform the illustrated method. For example, the neural network operations described above in FIG. 2 and FIG. 8, including the use of the number format described above with respect to FIG. 3, may be used to implement the illustrated method. As will be readily appreciated by those skilled in the art having the benefit of this disclosure, various variations of the DE-BBFP system and double exponential format may be adapted to perform the illustrated method. For ease of explanation, conversion to bfloat16 floating-point numbers is discussed in this example, although any suitable normal floating-point format may be the target of the illustrated method.

[0142] In process block 1210, a common exponent CEXP for each element is selected as the minimum of that element's row exponent HEXP and column exponent VEXP. If both the HEXP and VEXP exponents for that element are maximum values, CEXP is set to that value. For example, for an 8-bit exponent used in bflot16, if both HEXP and VEXP are 255, then CEXP is 255.

[0143] In process block 1220, the normalized significand (NSIG) is determined by left shifting the DE-BBFP mantissa until its most significant bit is 1. The number of left shifts used, LS, is retained for subsequent operations.

[0144] At process block 1230, it is determined whether the element is denormal. If the value CEXP-LS is greater than zero, the element is determined to be a normal floating-point value and the method proceeds to process block 1240. If the value CEXP-LS is greater than zero, below If so, the element is non-normal and the method proceeds to process block 1260.

[0145] In process block 1240, the normal floating-point mantissa is determined. The normal floating-point mantissa has its leading bit dropped and the result is then shifted left by one bit. For example, to convert the DE-BBFP mantissa to a bfloat16, the most significant bit of the normalized mantissa NSIG is dropped and the result is shifted left by one bit. The bfloat16 mantissa can be calculated by (NSIG-0x80)<<1. The normal floating-point code is the same value as the DE-BBFP element code.

[0146] In process block 1250, the normal floating point exponent is determined by calculating CEXP-LS.

[0147] If the element is determined to be denormalized at process block 1230, the method proceeds to process block 1260. In this case, the floating-point mantissa is determined by right shifting the normalized mantissa NSIG determined at process block 1220 by LS-CEXP. For example, the mantissa for the denormalized bfloat16 format can be calculated as NSIG>>(LS-CEXP).

[0148] In processing block 1270, the normal floating point exponent is set to zero to indicate that the normal floating point element is denormal.

[0149] There are some special cases that can be handled when converting DE-BBFP to regular floating point. For example, if CEXP is 255 (when converting to bfloat16), the regular element is NaN (sign=0, exponent=255, mantissa=0x40). If the DE-BBFP mantissa is zero, the bfloat16 output is zero.

[0150] For example, row index HEXP=[135,133], column index VEXP=[130,135], and

number

number

number

number

number

[0151] XII. Exemplary Computing Environment FIG. 13 illustrates a generalized example of a suitable computing environment 1300 in which the described embodiments, techniques, and technologies may be implemented, including performing machine learning and deep learning using the double exponential matrix format as described above.

[0152] The computing environment 1300 is not intended to suggest any limitation as to the scope of use or functionality of the technology, and the technology may be implemented in a variety of general purpose or special purpose computing environments. For example, the disclosed technology may be implemented with other computer system configurations including handheld devices, multiprocessor systems, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. The disclosed technology may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0153] Referring to FIG. 13, a computing environment 1300 includes at least one processing unit 1310 and memory 1320. In FIG. 13, this most basic configuration 1330 is included within the dashed line. The processing unit 1310 executes computer-executable instructions and may be a real or virtual processor. In a multiprocessing system, multiple processing units execute computer-executable instructions to increase processing power, and thus multiple processors may run simultaneously. The memory 1320 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two. The memory 1320 stores, for example, software 1380, images, and videos that can implement the techniques described herein. The computing environment may have additional features. For example, the computing environment 1300 includes storage 1340, one or more input devices 1350, one or more output devices 1360, and one or more communication connections 1370. An interconnection mechanism (not shown), such as a bus, controller, or network, interconnects the components of computing environment 1300. Typically, operating system software (not shown) provides an operating environment for other software executing within computing environment 1300 and coordinates the activities of the components of computing environment 1300.

[0154] Storage 1340 may be removable or non-removable and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, CD-RWs, DVDs, or any other medium that can be used to store information and that can be accessed within computing environment 1300. Storage 1340 stores instructions, plug-in data, and messages for software 1380 that can be used to implement the techniques described herein.

[0155] The input devices 1350 may be a keyboard, keypad, mouse, touch screen display, touch input device such as a pen or trackball, a voice input device, a scanning device, or another device that provides input to the computing environment 1300. For audio, the input devices 1350 may be a sound card or similar device that accepts audio input in analog or digital form, or a CD-ROM reader that provides audio samples to the computing environment 1300. The output devices 1360 may be a display, printer, speakers, a CD writer, or another device that provides output from the computing environment 1300.

[0156] The communication connections 1370 enable communication over a communication medium (e.g., a connecting network) to another computing entity. The communication medium carries information such as computer-executable instructions, compressed graphics information, video, or other data in a modulated data signal. The communication connections 1370 are not limited to wired connections (e.g., megabit or gigabit Ethernet, InfiniBand, Fibre Channel over electrical or fiber optic connections), but also include wireless technologies (e.g., RF connections over Bluetooth, WiFi (IEEE 802.11a / b / n), WiMax, cellular, satellite, laser, infrared), and other suitable communication connections for providing network connectivity to the disclosed agents, bridges, and agent data consumers. In a virtual host environment, the communication connections may be virtualized network connections provided by the virtual host.

[0157] Some embodiments of the disclosed methods may be performed using computer-executable instructions implementing all or a portion of the disclosed techniques in a computing cloud 1390. For example, the disclosed methods may be performed on a processing unit 1310 located in the computing environment 1330, or the disclosed methods may be performed on a server located in the computing cloud 1390.

[0158] Computer readable media are any available media that can be accessed within computing environment 1300. By way of example, and not limitation, in computing environment 1300, computer readable media include memory 1320 and / or storage 1340. As will be readily understood, the term computer readable storage media includes media for data storage, such as memory 1320 and storage 1340, and does not include transmission media, such as modulated data signals.

[0159] XIII. Further Examples Additional examples are disclosed in the following clauses, which may be rearranged in combination and subcombinations with one another. [Clause 1] 1. A computer-implemented method comprising: using a processor to perform the steps of: selecting a common exponent for bounding boxes of elements of an input matrix to be stored in double exponent format, the common exponent being selected based on a smaller exponent for either a row or column of the bounding box of the elements; determining mantissas for the bounding boxes of elements of a double exponent format matrix, each of the determined mantissas being selected by comparing an exponent of a respective element to the common exponent; and storing the determined mantissas and the common exponent in a computer-readable storage medium as a double exponent format matrix. [Clause 2] 2. The method of claim 1, wherein selecting the common exponent comprises calculating a smaller exponent for either a row or column of the bounding box of an element minus a left shift amount to calculate a normalized mantissa for the respective row or column of the bounding box of an element. [Clause 3] 3. The method of claim 1 or 2, wherein determining the mantissa comprises left shifting mantissas in the input matrix by a difference between the common exponent and an input matrix exponent of the mantissa. [Article 4] 4. The method of any one of clauses 1 to 3, wherein the bounding box of elements in the input matrix includes normal floating-point elements, and determining the mantissa includes: restoring an implicit leading bit from a normal floating-point mantissa; and scaling the normal floating-point mantissa by the difference between the selected common exponent and a normal floating-point exponent. [Article 5] 5. The method of any one of clauses 1 to 4, further comprising: determining a left shift value of the mantissa indicating the number of shifts of the most significant "1" bit from the most significant bit position; and when the left shift value is greater than the common exponent, determining a normal mantissa by right shifting the mantissa by the difference between the left shift value and the common exponent. [Article 6] 6. The method of any one of clauses 1 to 5, further comprising: determining whether the double exponent format matrix is ​​to be used as a left-hand matrix or a right-hand matrix in a matrix operation; and based on the determination, converting the double exponent format matrix to a single exponent format matrix by selecting the common exponent based on a maximum exponent for a row of the bounding box of an element if the double exponent format matrix is ​​to be used as a left-hand matrix, or by selecting the common exponent based on a maximum exponent for a column of the bounding box of an element if the double exponent format matrix is ​​to be used as a right-hand matrix. [Article 7] 7. The method of any one of clauses 1 to 6, wherein the common exponent is a common row exponent selected based on a maximum exponent for a row of the bounding box of an element, the method further comprising: selecting a common column exponent for bounding boxes of elements of an input matrix to be stored in double exponent format, the common exponent selected based on a maximum exponent for a column of the bounding box of an element; and storing the common column exponent in the computer-readable storage medium, wherein each of the determined mantissas is selected by comparing an exponent of the respective element to the larger of the common row exponent or the common column exponent. [Article 8] 8. The method of any one of clauses 1-7, further comprising performing a matrix operation with the double exponential format matrix to generate a result matrix in double exponential format. [Article 9] 9. The method of claim 8, further comprising: converting the result matrix in double exponential format to a result matrix in normal floating point format; and storing the result matrix in normal floating point format in a computer-readable storage medium. [Article 10] 10. The method of any one of clauses 1 to 9, further comprising the step of quantizing the determined mantissa, the common exponent, or the determined mantissa and the common exponent. [Article 11] 11. The method of any one of clauses 1 to 10, wherein the bounding box of an element is a 16x16 element bounding box, and the input matrix includes a plurality of 16x16 element bounding boxes, each of the plurality of 16x16 element bounding boxes including a respective common exponent. [Article 12] 11. A method for training a neural network, comprising: performing a training operation for at least one layer of the neural network using the double exponential format matrix including the determined mantissa and the common exponent generated by the method of clause 1; and storing at least one of node weights, edge weights, bias values, or activation functions generated by performing the training operation in a computer-readable storage medium. [Article 13] A computer readable medium storing computer readable instructions which, when executed by a computer, cause the computer to perform the method of any one of clauses 1 to 12. [Article 14] 1. An apparatus having a memory, a common exponent register, and a processor, the processor performing the steps of: selecting a common exponent for bounding boxes of elements of an input matrix stored in the memory, the common exponent selected based on a maximum exponent for either a row or column of the bounding box of the element; determining mantissas for the bounding boxes of elements of a double exponent format matrix, each of the determined mantissas being selected by comparing an exponent of a respective element to the common exponent; and storing the common exponent in the common exponent register. [Article 15] 16. The apparatus of claim 15, further comprising a neural network accelerator formed from components including the memory, the common exponent register, and the processor, the apparatus configured to evaluate a neural network model by performing at least one training, inference, or classification operation using the double exponential format matrix. [Article 16] 16. The apparatus of claim 14 or 15, further comprising a floating-point to double exponential bounding box based floating-point (DE-BBFP) converter for receiving normal floating-point values ​​for the neural network model and generating the double exponential format matrix; and a DE-BBFP to floating-point converter for generating normal floating-point values ​​from a resultant double exponential format matrix by performing at least one matrix operation with the generated double exponential format matrix. [Article 17] 17. Apparatus according to any one of clauses 14 to 16, configured to perform at least one of the methods of clauses 1 to 12. [Article 18] A computer-readable storage medium storing a result matrix generated by performing a matrix operation with a double exponential format matrix. [Article 19] 19. The computer-readable storage medium of claim 18, wherein the result matrix is ​​a double exponent format matrix that includes a common exponent for each row or column of a bounding box of elements in the result matrix, the result matrix being generated by performing the matrix operation with the double exponent format matrix and another dual format matrix. [Article 20] 20. The computer-readable storage medium of claim 18 or 19, wherein the result matrix is ​​an array of ordinary floating-point numbers generated by converting a result of the matrix operation from a double exponential format matrix. [Article 21] 21. The computer-readable storage medium of any one of clauses 18-20, wherein each element within a bounding box of the double exponent format matrix has a mantissa, a row common exponent, and a column format exponent, the row common exponent being shared by each of the elements in a row of the bounding box and the column common exponent being shared by each of the elements in a column of the bounding box, and the result matrix is ​​generated by: for each element within the bounding box of the double exponent format matrix: selecting a minimum exponent of each of the element's row common exponents and column common exponents; calculating a normalized mantissa by shifting the element's mantissa to the left by a shift number until a most significant bit is 1; calculating a normalized exponent by subtracting the shift number from the minimum exponent; and storing the normalized mantissa and the normalized exponent in the result matrix in the computer-readable storage medium. [Article 22] 22. The computer-readable storage medium of claim 21, wherein calculating the normalized mantissa further includes dropping the most significant bit and shifting the mantissa to the left based on a number of bits in the double exponent format mantissa and a number of bits in the normal floating-point mantissa. [Article 23] 13. A computer-readable storage medium storing a result matrix generated by performing a matrix operation using a double exponential format matrix according to at least one of the methods of claims 1-12. In view of the many possible embodiments to which the principles of the disclosed subject matter may be applied, it should be recognized that the illustrated embodiments are merely preferred examples and should not be construed as limiting the scope of the claims to those preferred examples. Rather, the scope of the claimed subject matter is defined by the following claims. We therefore claim as our invention everything that comes within the scope of those claims.

Claims

1. A method performed by a neural network accelerator, the method comprising: selecting a common exponent for bounding boxes of elements of an input matrix to be stored in a double exponent format, the common exponent being selected based on the smaller exponent for either a row or column of the bounding box of an element, the double exponent format having a common row exponent selected for a given row of the input matrix and a common column exponent selected for a given column of the input matrix; determining mantissas for the bounding box of elements of a double exponent format matrix having the double exponent format, each determined mantissa being selected by comparing an exponent of a respective element to the common exponent; storing the determined mantissa and the common exponent in a computer readable storage medium as a dual exponent format matrix, the dual exponent format reducing storage requirements by storing shared exponents for rows and columns rather than individual exponents for each element while allowing reconstruction of the original matrix values. method.

2. 2. The method of claim 1 , wherein selecting the common exponent comprises calculating a smaller exponent for either a row or column of the bounding box of an element minus a left shift amount to calculate a normalized mantissa for each row or column of the bounding box of an element.

3. 2. The method of claim 1, wherein determining the mantissa comprises left-shifting the mantissas in the input matrix by a difference between the common exponent and an input matrix exponent of the mantissa.

4. determining a left shift value of the mantissa indicating the number of shifts of the most significant "1" bit from the most significant bit position; and when the left shift value is greater than the common exponent, determining a common mantissa by right shifting the mantissa by a difference between the left shift value and the common exponent. The method of claim 1.

5. determining whether the double exponential format matrix is ​​to be used as a left-hand matrix or a right-hand matrix in a matrix operation; and based on the determination, converting the double exponent format matrix to a single exponent format matrix by selecting the common exponent based on a maximum exponent for a row of the bounding box of an element if the double exponent format matrix is ​​used as a left matrix, or by selecting the common exponent based on a maximum exponent for a column of the bounding box of an element if the double exponent format matrix is ​​used as a right matrix. The method of claim 1.

6. The common row index is selected based on the maximum index for a row of the bounding boxes of elements, the method comprising: selecting the common row index based on a maximum index for a column of the bounding box of an element; and storing the common sequence index in the computer readable storage medium; each of the determined mantissas is selected by comparing the exponent of the respective element to the greater of the common row exponent or the common column exponent. The method of claim 1.

7. performing a matrix operation with the double exponential format matrix to generate a result matrix in double exponential format; and converting the result matrix in double exponential format to a result matrix in normal floating point format and storing the result matrix in normal floating point format in a computer readable storage medium. The method of claim 1.

8. 2. The method of claim 1 , wherein the bounding box of an element is a 16×16 element bounding box, and the input matrix includes a plurality of 16×16 element bounding boxes, each of the plurality of 16×16 element bounding boxes including a respective common exponent.

9. 1. A method of training a neural network comprising: performing a training operation for at least one layer of the neural network using the double exponential format matrix including the determined mantissa and the common exponent generated by the method of any one of claims 1 to 8; and storing at least one of the node weights, edge weights, bias values, or activation functions generated by performing the training operation in a computer-readable storage medium. method.

10. 1. An apparatus having a memory, a common exponent register, and a neural network accelerator configured to perform operations, the operations comprising: selecting a common exponent for a double exponent format for a bounding box of elements of an input matrix stored in the memory, the common exponent being selected based on a maximum exponent for either a row or a column of the bounding box of an element, the double exponent format having a common row exponent selected for a given row of the input matrix and a common column exponent selected for a given column of the input matrix; determining mantissas for the bounding box of elements of a double exponent format matrix having the double exponent format, each of the determined mantissas being selected by comparing an exponent of a respective element to the common exponent; storing the common exponent in the common exponent register, the dual exponent format reducing storage requirements by storing shared exponents for rows and columns rather than individual exponents for each element while allowing reconstruction of the original matrix values. Device.

11. a neural network accelerator formed from components including the memory, the common exponent register, and a processor; the apparatus is configured to evaluate a neural network model by performing at least one training or inference operation using the double exponential format matrix.

11. The apparatus of claim 10.

12. The neural network accelerator comprising: receiving normal floating point values ​​for the neural network model and generating the double exponential format matrix; Generate normal floating-point values ​​from the resulting double exponential format matrix by performing at least one matrix operation with the generated double exponential format matrix. and further configured to perform operations including:

11. The apparatus of claim 10.

Images