Histogram-based per-layer data format selection for hardware implementation of deep neural networks

Abstract

The present disclosure relates to histogram-based per-layer data format selection for deep neural network hardware implementations. A histogram-based method for selecting a fixed-point format for representing a set of values ​​input to and / or output from a layer of a deep neural network "DNN". The method includes obtaining a histogram representing the expected distribution of a set of values ​​for the layer, each bin of the histogram being associated with a frequency value and a representative value in a floating-point format; quantizing the representative value according to each of a plurality of possible fixed-point formats; for each of a plurality of possible fixed-point formats, estimating a total quantization error based on the frequency value of the histogram and a distance value for each bin, the distance value being based on the quantization of the representative value for the bin; and selecting the fixed-point format associated with the minimum estimated total quantization error as the best fixed-point format for representing a set of values ​​for the layer.

Description

Technical Field

[0001] The present invention relates to histogram-based per-layer data format selection for hardware implementation of deep neural networks. Background Art

[0002] A deep neural network (DNN) is a type of artificial neural network that can be used for machine learning applications. Specifically, DNN can be used for signal processing applications, including image processing and computer vision applications.

[0003] DNNs have been implemented in applications where power resources are not a significant factor. Nevertheless, DNNs are also applied in many different technology areas where the power consumption, processing power, or silicon area of ​​the hardware resources used to implement DNNs are limited.

[0004] Therefore, there is a need to implement hardware that is configured to implement DNNs in an efficient manner (i.e., in a manner that requires less silicon area or less processing power when operating). In addition, DNNs can be configured in a variety of different ways for a variety of different applications. Therefore, there is also a need for hardware for implementing DNNs that can flexibly support a variety of DNN configurations.

[0005] The embodiments described below are provided by way of example only and are not intended to limit implementations that address any or all shortcomings of known hardware implementations of deep neural networks. Summary of the invention

[0006] This Summary is provided to introduce some concepts that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0007] Described herein is a histogram-based method for selecting a fixed-point number format for representing a set of values ​​input to and / or output from a layer of a deep neural network (DNN). The method includes obtaining a histogram representing an expected distribution of a set of values ​​for the layer, wherein each bin of the histogram is associated with a frequency value; quantizing a representative value in a floating-point format for each bin according to each of a plurality of possible fixed-point number formats; for each of a plurality of possible fixed-point number formats, estimating a total quantization error resulting from quantization based on the frequency value of the histogram and a distance value for each bin, the distance value being based on the quantization of the representative value for the bin; and selecting a fixed-point number format for representing the set of values ​​based on the estimated total quantization error.

[0008] A first aspect provides a computer-implemented method for identifying a fixed-point number format for representing a set of values ​​input to and / or output from a layer of a deep neural network "DNN" for configuring a hardware implementation of the DNN, the method comprising: obtaining a histogram representing an expected distribution of a set of values ​​for the layer, wherein each bin of the histogram is associated with a frequency value; for each fixed-point number format in a plurality of fixed-point number formats: quantizing a representative value in a floating-point number format for each bin according to the fixed-point number format; and estimating a total quantization error associated with the fixed-point number format based on a frequency value and a distance value for each bin, the distance value being based on quantization of the representative value for the bin; and selecting a fixed-point number format in a plurality of fixed-point number formats that is associated with a minimum estimated total quantization error as the fixed-point number format for representing a set of values ​​for the layer.

[0009] A second aspect provides a computing-based device for identifying a fixed-point number format for representing a set of values ​​input to and / or output from a layer of a deep neural network "DNN" for configuring a hardware implementation of the DNN, the computing-based device comprising: at least one processor; and a memory coupled to the at least one processor, the memory comprising: a histogram representing an expected distribution of a set of values ​​for the layer, wherein each bin of the histogram is associated with a frequency value; and computer-readable code, when executed by the at least one processor, causes the at least one processor to perform the following operations: for each fixed-point number format in a plurality of fixed-point number formats: quantize a representative value in a floating-point format for each bin according to the fixed-point number format; and estimate a total quantization error associated with the fixed-point number format based on a frequency value and a distance value for each bin, the distance value being based on the quantization of the representative value for the bin; and select a fixed-point number format in the plurality of fixed-point number formats that is associated with a minimum estimated total quantization error as the fixed-point number format for representing a set of values ​​for the layer.

[0010] A third aspect provides a hardware implementation of a deep neural network "DNN", comprising: hardware logic configured to: receive input data values ​​for a layer of the DNN; receive information indicating a fixed-point format of the input data values ​​for the layer, the fixed-point format of the input data values ​​for the layer having been selected according to the method of the first aspect; interpret the input data values ​​based on the fixed-point format of the input data values ​​for the layer; and process the interpreted input data values ​​according to the layer to generate output data values ​​for the layer.

[0011] A hardware implementation of a DNN may be embodied in hardware on an integrated circuit. A method of manufacturing a hardware implementation in an integrated circuit manufacturing system may be provided. An integrated circuit definition data set may be provided that, when processed in an integrated circuit manufacturing system, configures the system to manufacture a hardware implementation of a DNN. A non-transitory computer-readable storage medium having a computer-readable description of a hardware implementation of a DNN stored thereon may be provided that, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the hardware implementation of the DNN.

[0012] An integrated circuit manufacturing system can be provided, comprising: a non-transitory computer-readable storage medium on which a computer-readable description of a hardware implementation of a DNN is stored; a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit embodying the hardware implementation of the DNN; and an integrated circuit generation system configured to manufacture the hardware implementation of the DNN based on the circuit layout description.

[0013] A computer program code for performing the method as described herein may be provided.A non-transitory computer readable storage medium having computer readable instructions stored thereon may be provided, which when executed at a computer system causes the computer system to perform the method as described herein.

[0014] As will be apparent to the skilled person, the above features may be combined as appropriate and may be combined with any aspects of the examples described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Embodiments will now be described in detail with reference to the accompanying drawings, in which:

[0016] Figure 1 is a schematic diagram of an example deep neural network (DNN);

[0017] Figure 2 is a schematic diagram of example data in DNN;

[0018] Figure 3 is a schematic diagram showing data input to and output from a layer of a DNN;

[0019] Figure 4 is a flow chart of an example histogram-based method for identifying an appropriate fixed-point number format for representing a set of values ​​input to or output from a layer of a DNN from a plurality of possible fixed-point number formats;

[0020] Figure 5 is a schematic diagram of an example histogram;

[0021] Figure 6is a schematic diagram of an example histogram in which a representative value of each bin of the histogram has been quantized;

[0022] Figure 7 is a schematic diagram of an example weighting function;

[0023] Figure 8 is a plot of the mantissa length versus the top-1 classification accuracy for the AlexNet DNN with the ImageNet validation dataset, where the exponent for the weights is chosen according to many different methods;

[0024] Fig. 9 is a plot of the mantissa length versus top-5 classification accuracy for the AlexNet DNN with the ImageNet validation dataset, where the exponent for the weights is chosen according to many different methods;

[0025] Fig.10 is a plot of the mantissa bit length versus top-1 classification accuracy for the AlexNet DNN with the ImageNet validation dataset, where the exponents for the weights and input data values ​​are chosen according to many different approaches;

[0026] Fig.11 is a plot of the mantissa bit length versus top-5 classification accuracy for the AlexNet DNN with the ImageNet validation dataset, where the exponents for the weights and input data values ​​are chosen according to many different methods;

[0027] Fig.12 is a plot of mantissa bit length versus top-1 classification accuracy for the GoogLeNet DNN with the ImageNet validation dataset, where the exponent for the weights is chosen according to a number of different approaches;

[0028] Fig.13 is a plot of the mantissa length versus top-5 classification accuracy for the GoogLeNet DNN with the ImageNet validation dataset, where the exponent for the weights is chosen according to many different methods;

[0029] Fig.14 is a plot of mantissa bit length versus top-1 classification accuracy for the GoogLeNet DNN with the ImageNet validation dataset, where the exponents for weights and input data values ​​are chosen according to many different approaches;

[0030] Fig.15 is a plot of the mantissa bit length versus top-5 classification accuracy for the GoogLeNet DNN with the ImageNet validation dataset, where the exponents for the weights and input data values ​​are chosen according to many different methods;

[0031] Fig.16 is a block diagram of an example hardware implementation of a DNN;

[0032] Fig.17 is a block diagram of an example computing-based device;

[0033] Fig.18 is a block diagram of an example computer system in which a hardware implementation of a DNN is implemented; and

[0034] Fig.19 is a block diagram of an example integrated circuit manufacturing system for generating an integrated circuit embodying a hardware implementation of a DNN as described herein.

[0035] The accompanying drawings show various examples. It will be appreciated by the skilled person that the element boundaries (e.g., boxes, box groups, or other shapes) shown in the accompanying drawings represent an example of boundaries. In some examples, an element may be designed as multiple elements, or multiple elements may be designed as one element. Common reference numerals are used throughout the accompanying drawings to appropriately represent similar features. DETAILED DESCRIPTION

[0036] The following description is presented by way of example to enable those skilled in the art to make and use the invention. The invention is not limited to the embodiments described herein, and various modifications of the disclosed embodiments will be apparent to those skilled in the art. The embodiments are described only by way of example.

[0037] A deep neural network (DNN) is an artificial neural network that includes multiple interconnected layers, enabling the DNN to perform signal processing tasks, including but not limited to computer vision tasks. Figure 1 An example DNN 100 is shown that includes a plurality of layers 102-1, 102-2, 102-3. Each layer 102-1, 102-2, 102-3 receives input data, processes the input data according to the layer to produce output data. The output data is provided as input data to another layer, or is output as the final output data of the DNN. For example, in Figure 1 In the DNN 100 of FIG. 1 , the first layer 102-1 receives raw input data 104 entering the DNN 100, and processes the input data according to the first layer 102-1 to generate output data. The output data of the first layer 102-1 becomes the input data of the second layer 102-2, and the second layer 102-2 processes the input data according to the second layer 102-2 to generate output data. The output data of the second layer 102-2 becomes the input data of the third layer 102-3, and the third layer 102-3 processes the input data according to the third layer 102-3 to generate output data. The output data of the third layer 102-3 is output as the output data 106 of the DNN.

[0038] The processing performed on the input data entering the layer depends on the type of layer. For example, each layer of a DNN can be one of a number of different types. Example DNN layer types include, but are not limited to: convolutional layers, activation layers, normalization layers, pooling layers, and fully connected layers. It will be apparent to one skilled in the art that these are example DNN layer types, and this is not an exhaustive list, and other DNN layer types may exist.

[0039] For a convolutional layer, the input data is processed by convolving it with the weights associated with the layer. Specifically, each convolutional layer is associated with a number of weights w0…w g The weights are grouped to form or define one or more filters (which may also be referred to as kernels), and each filter may be associated with an offset bias b.

[0040] refer to Figure 2 , Figure 2 An example overview of the data formats used in DNNs is shown. Figure 2 As shown, the data used in the DNN can be arranged as P planes of data, where each plane has dimensions x×y. The DNN may include one or more convolutional layers, each associated with a plurality of filters each including a plurality of weights. Each filter has dimensions m×n×P (i.e., each filter includes a set of m×n×P weights w), and is applied to the input data according to a convolution operation across a number of steps in directions s and t, such as Figure 2 As shown. The number of filters and the number of weights per filter can vary between convolutional layers. A convolutional neural network (CNN), which is a specific type of DNN effective for image recognition and classification, typically includes multiple convolutional layers.

[0041] An activation layer (which typically (but not necessarily) follows a convolutional layer) performs one or more activation functions on the input data going into the layer. An activation function takes a single number and performs some nonlinear mathematical operation on it. In some examples, an activation layer can be used as a rectified linear unit (ReLU) by implementing a ReLU function (i.e., f(x) = max(0, x)), or as a parametric rectified linear unit (PReLU) by implementing a PReLU function.

[0042] The normalization layer is configured to perform a normalization function (e.g., a local response normalization (LRN) function) on the input data. The pooling layer (which is usually (but not necessarily) inserted between consecutive convolutional layers) performs a pooling function (e.g., a maximum or average function) to summarize a subset of the input data. Therefore, the purpose of the pooling layer is to reduce the spatial size of the representation to reduce the number of parameters and calculations in the network, and thus also control overfitting.

[0043] A fully connected layer (which is usually (but not necessarily) followed by multiple convolutional and pooling layers) takes a set of three-dimensional input data values ​​and outputs an N-dimensional vector. In the case of DNNs for classification, N is the number of classes, and each value in the vector represents the probability of a certain class. The N-dimensional vector is generated by matrix multiplication of a set of weights (optionally followed by a bias offset). Therefore, a fully connected layer receives a set of weights and / or biases.

[0044] Therefore, if Figure 3 As shown, each layer 302 of the DNN receives input data values ​​and generates output data values; and some layers (e.g., convolutional layers and fully connected layers) also receive weights and / or biases.

[0045] The hardware implementation of the DNN includes hardware logic configured to process input data entering the DNN according to the layers of the DNN. Specifically, the hardware implementation of the DNN includes hardware logic configured to process the input data entering the layer according to each layer and generate output data for the layer (the output data becomes the input data entering another layer, or becomes the output of the DNN). For example, if the DNN includes a convolution layer and the convolution layer is followed by an activation layer, the hardware implementation of the DNN includes hardware logic configured to perform convolution on the input data entering the DNN using the weights and biases associated with the convolution layer to generate output data for the convolution layer, and hardware logic configured to apply the activation function to the input data entering the activation layer (i.e., the output data of the convolution layer) to generate output data for the DNN.

[0046] As known to those skilled in the art, for hardware used to process a set of values, each value is represented in a digital format. The two most appropriate digital formats are fixed-point format and floating-point format. As known to those skilled in the art, the fixed-point format has a fixed number of digits after the base point (e.g., a decimal point or a binary point). In contrast, the floating-point format has no fixed base point (i.e., the base point can "float"). In other words, the base point can be placed anywhere in the representation. Although representing the values ​​input to and output from the layers of the DNN in floating-point format can allow more accurate or precise output data to be generated, processing numbers in floating-point format in hardware is complex, which tends to increase the silicon area and complexity of the hardware compared to hardware that processes values ​​in fixed-point format. Therefore, the hardware implementation can be configured to represent the values ​​input to and / or output from the layers of the DNN in fixed-point format to reduce the area, power consumption, and memory bandwidth of the hardware implementation.

[0047] A common fixed-point number format is the Q format, which specifies a predetermined number of integer digits a and fractional digits b. Thus, a number can be represented as Qa.b, which requires a total of a+b+1 bits (including the sign bit). An example Q format is shown in Table 1 below.

[0048] Table 1

[0049] Q format describe Example Q4.4 4 integer digits and 4 decimal digits <![CDATA[0110.11102]]> Q0.8 0 integer digits and 8 decimal digits <![CDATA[.011011102]]>

[0050] However, a disadvantage of the Q format is that some of the bits used to represent the number may be considered redundant. In the example, the number range [-0.125, 0.125] will be represented as 3 bits of precision. The Q format required for this example range and precision is Q0.5. However, if we assume that the range of values ​​is known in advance, the first two bits of the number will never be used to determine the value represented in the Q format. For example, the first two bits represented do not contribute to the final number because they represent 0.5 and 0.25, respectively, which exceeds the required range. However, they are used to indicate the value of the third bit position (i.e., 0.125 and other values ​​attributed to the relative bit positions). Therefore, the above Q format is an inefficient fixed-point number format for use in hardware implementations of DNNs because some bits may not convey useful information.

[0051] Therefore, in some cases, instead of using the Q format, some hardware implementations may be configured to use the following fixed-point number format for values ​​input to or output from a layer of a DNN, where each value x is represented by a fixed integer exponent e and an n-bit mantissa m, in the format x=2 e m, which is defined by the exponent e and the number of mantissa bits n (ie, {e, n}). In some cases, the mantissa m may be represented in two's complement format, while in other cases other signed or unsigned integer formats may be used.

[0052] In order to reduce the size of the hardware implementation of the DNN and improve efficiency, the hardware implementation can be configured to represent the values ​​input to or output from the layer of the DNN using the following fixed-point number format, which uses the minimum number of bits that can represent the expected or desired range of each group of values. Since the range of different groups of values ​​(e.g., input data values, output data values, biases, and weights) may vary within and between layers, the hardware implementation can more efficiently process the DNN when it is possible to use a fixed-point number format that can vary within and between layers to represent the input data values, output data values, weights, and / or biases. For example, the hardware implementation can more efficiently implement the DNN by using a fixed-point number format including an exponent of 2 and a mantissa bit length of 6 to represent the input data values ​​for the first layer, using a fixed-point number format including an exponent of 3 and a mantissa bit length of 12 to represent the weights of the first layer, and using a fixed-point number format including an exponent of 4 and a mantissa bit length of 6 to represent the input data values ​​for the second layer. Therefore, it is desirable to identify, on a per-layer basis, a fixed-point number format (e.g., defined by exponent and mantissa bit lengths) used to represent sets of values ​​input to or output from a layer of a DNN to allow hardware implementations to efficiently process the layer.

[0053] A simple method (which may be referred to herein as the full-range method) for selecting a fixed-point number format for representing a set of values ​​input to or output from a layer of a DNN may include, for a given mantissa bit depth n (or a given exponent e), selecting the minimum exponent e (or minimum mantissa bit depth n) that covers the range of an expected set of values ​​x for a layer. For example, for a given mantissa bit depth n, the exponent e may be selected according to equation (1) such that the exponent e covers the entire range of x, where is the ceiling function:

[0054]

[0055] However, this approach is sensitive to outliers. Specifically, in cases where a set of values ​​x is expected to have outliers, precision is sacrificed to cover the outliers. This can result in large quantization errors (i.e., the error between a set of values ​​in a floating point format and a set of values ​​in a selected fixed point format). As a result, the error in the output data of the layer and / or DNN caused by quantization may be larger than the error in the case of using a fixed point format that covers a smaller range but has higher precision. As a result, the error in the output of the layer and / or DNN can be reduced by selecting a fixed point format that minimizes the total error introduced by quantizing a set of values ​​(rather than selecting a fixed point format that covers the entire expected range of a set of values ​​for a layer).

[0056] Thus, described herein is a histogram-based method for selecting a fixed-point format for representing a set of values ​​input to or output from a layer of a DNN based on an estimate of the total quantization error associated with possible fixed-point formats. Specifically, the method includes obtaining a histogram representing an expected distribution of a set of values ​​of a layer of the DNN, wherein each bin of the histogram is associated with a frequency value; quantizing a representative value of each bin according to each of a plurality of possible fixed-point formats; for each of a plurality of possible fixed-point formats, estimating a total quantization error based on the frequency values ​​of the histogram and a distance value for each bin, the distance value being based on the quantization of the representative value for the bin; and selecting the fixed-point format associated with the minimum estimated total quantization error as the optimal fixed-point format for representing a set of values ​​of the layer. Minimizing quantization error in this manner allows for resolving tradeoffs between accuracy and range coverage.

[0057] Once a fixed-point format is selected for representing a set of values ​​input to and / or output from a layer, the selected fixed-point format can be used to configure the hardware implementation of the DNN. For example, if the set of values ​​is input data values ​​for a particular layer, the selected fixed-point format can be used to configure the hardware implementation of the DNN to receive and process the input data values ​​of the particular layer using the selected fixed-point format. Further examples of how to use the selected fixed-point format to configure the hardware implementation of the DNN are described below.

[0058] As described in more detail below, the described method is shown to select a fixed point format for representing multiple sets of values ​​input to or output from multiple layers of a DNN, which method produces more accurate output of the DNN when used to process these layers compared to other methods (e.g., the full range method of selecting a fixed point format for representing multiple sets of values ​​for multiple layers of the DNN described above with respect to equation (1)). For example, in the case where the DNN is used as a classifier, the DNN is shown to produce more accurate classification when the fixed point format for representing multiple sets of values ​​input to or output from multiple layers of the DNN is selected according to the described method. This is particularly true when the method is used to select an appropriate exponent for a fixed point format having a low number of mantissa bits (e.g., less than 6 mantissa bits). The described method is also efficient in terms of computational power and time.

[0059] Reference now Figure 4 , Figure 4 An example histogram-based method 400 for determining an appropriate fixed-point number format for representing a set of values ​​input to or output from a layer of a DNN that can be used to configure a hardware implementation of a DNN is shown. The method 400 can be performed by a computing-based device (e.g., as described below with reference to Fig.17 For example, a computing-based device (e.g., computing-based device 1700) may include or have access to a computer-readable medium (e.g., memory) having computer-readable instructions stored thereon, which, when executed by a processor of the computing-based device, causes the computing-based device to perform Figure 4 Method 400.

[0060] The method begins at block 402, where a histogram representing the expected distribution of a set of values ​​(x) input to or output from a layer of a DNN is obtained. Figure 3 As described, each layer 302 of the DNN can have multiple sets of data input to or output from the layer. For example, each layer can receive input data values ​​and generate output data values; and some layers can also receive weights and / or biases. A set of values ​​input to or output from a layer can be all or part of the input data values ​​of the layer, all or part of the output data values ​​of the layer, all or part of the weights of the layer, or all or part of the biases of the layer.

[0061] A histogram representing the expected distribution of a set of values ​​is generated by dividing the range of expected values ​​in the set of values ​​into intervals or bins and counting how many expected values ​​fall into each bin. In the embodiments described herein, each bin (Bin i ) and indicates that the value in this group falls into this bin (Bin i ) values ​​of the expected number of frequency values ​​(F i ). For example, Figure 5 An example histogram H(x) is shown having five bins (Bin1, Bin2, Bin3, Bin4, Bin5), where each bin is associated with a frequency value (F1, F2, F3, F4, and F5).

[0062] Preferably, the density of bins is at least as high as the maximum density of quantization levels of a plurality of possible fixed-point number formats. For example, a fixed-point number format with a mantissa bit length of 4 has 16 quantization levels, so preferably there are at least 16 bins. Similarly, a fixed-point number format with a mantissa bit length of 8 has 256 quantization levels, so preferably there are at least 256 bins. The computation time for determining the total quantization error increases with the number of bins, so there is a tradeoff between computation time and accuracy. Testing has shown that good results can be obtained with approximately 1000 bins.

[0063] The actual set of input data values ​​for a layer of a DNN typically varies based on the inputs into the DNN, and typically cannot be precisely known in advance. However, the expected range of input data values ​​and the distribution within that range can be determined by, for example, running a set of training data or a known sample of a typical input data set through the DNN, recording the input data values ​​into each layer that can be used to generate a histogram of expected input data values ​​for that layer. The expected range and distribution of output data values ​​for a layer can be determined in a similar manner.

[0064] In contrast, the actual set of weights and biases for a layer is typically determined in advance during training of the DNN. Thus, a histogram of weights or biases for a layer can be generated directly from the actual set of weights or biases that will be used for the layer.

[0065] Once a histogram (H(x)) representing an expected distribution of a set of values ​​for a layer is obtained, method 400 proceeds to block 404 .

[0066] At block 404, a possible fixed point number format from a plurality of possible fixed point number formats is selected for analysis. The plurality of possible fixed point number formats includes a plurality of different fixed point number formats that may be used to represent the set of values ​​for the layer.

[0067] In some cases, each possible fixed-point number format is defined by an exponent e and a mantissa bit length n, or includes an exponent e and a mantissa bit length n. In some cases, method 400 can be used to identify the best exponent e for a particular mantissa bit length n. In these cases, each of the multiple possible fixed-point number formats can include the same mantissa bit length n, but include different exponents e. In other cases, method 400 can be used to identify the best mantissa bit length n for a particular exponent e. In these cases, each of the multiple possible fixed-point number formats can include the same exponent e, but include different mantissa bit lengths n.

[0068] In some cases, the plurality of possible fixed-point number formats may include all possible fixed-point number formats. For example, in the case of a fixed exponent, the plurality of possible fixed-point number formats may include possible fixed-point number formats combined with a fixed exponent e for each possible mantissa bit length n. In other cases, the plurality of possible fixed-point number formats may include a subset of all possible fixed-point number formats. The subset may include, for example, a possible fixed-point number format that is more likely to be an optimal fixed-point number format based on one or more criteria. For example, in the case of a fixed mantissa bit length, the plurality of possible fixed-point number formats may include possible fixed-point number formats combined with a specific mantissa bit length n and only a subset of possible exponents e. Once a possible fixed-point number format has been selected, the method proceeds to block 406.

[0069] At block 406, the representative value (x) of each bin of the histogram H(x) is quantized according to the possible fixed-point number format selected in block 404. i ). By quantizing only one value per bin (as opposed to quantizing all values ​​that fall within that bin), method 400 may be performed efficiently.

[0070] The representative value of the bin (x i ) is the value that falls into the bin. In some cases, such as Figure 5 As shown, the representative value of each bin (x i ) can be the center of the bin. Tests have shown that the center value of a bin is a good representative of the values ​​in the bin. However, in other cases, the representative value (x i ) can be another value that falls into the bin.

[0071] For each bin’s representative value (x i ) is a floating point number format. As mentioned above, where fixed point number formats have a fixed number of digits after the base point (e.g., decimal point or binary point), floating point representations have no fixed base point (i.e., the base point can "float"). In other words, the base point can be placed anywhere in the representation.

[0072] The most common floating point format is the Institute of Electrical and Electronics Engineers (IEEE) standard for floating point arithmetic (IEEE-754). IEEE-754 specifies that floating point numbers are represented by three numbers: sign, exponent, and mantissa (s, exp, mant). Typically, these three numbers (s, exp, mant) are interpreted for a fixed integer bias as shown in equation (2):

[0073] (-1) s 2 exp-bias 1.mant (2)

[0074] IEEE-754 defines four basic formats for floating-point numbers with different precisions as shown in Table 2. Specifically, they are encoded with 16, 32, 64, and 128 bits, respectively.

[0075] Table 2

[0076]

[0077] The representative value may be in one of the IEEE floating point number formats, or may be in another floating point number format.

[0078] As known to those skilled in the art, quantization is the process of converting a number from a higher precision format to a lower precision format. Quantizing a number in a higher precision format typically includes using the "closest" or "nearest" representable number in the lower precision format to represent the number in the higher precision format, where the "closest" or "nearest" representable number in the lower precision format is defined by a specific rounding mode (for example, but not limited to, round to nearest integer (RTN), round to zero (RTZ), round to nearest even (RTE), round to positive infinity (RTP), and round to negative infinity (RTN)). For example, quantizing the number 1.2 according to a 4-bit binary number format may include representing the number 1.2 with the binary number "0001" (the binary number "0001" is the decimal number "1" because 1 is the closest representable number in the 4-bit binary number format of 1.2). According to the representative value (x) of the bin of the histogram (H(x)) of a specific fixed-point number format f i The quantitative version of ) is denoted as Q(x i ,f). An example method of quantizing a floating point format value according to a fixed point format is described in detail below. However, any appropriate method may be used to quantize the representative value (x) of a bin of the histogram (H(x)) according to a possible fixed point format. i ).

[0079] Once the representative value (x) of each bin of the histogram has been quantized according to the possible fixed-point number format selected in block 404, i), method 400 proceeds to box 408.

[0080] At block 408, based on the frequency values ​​(F i ) and a distance value for each bin, based on the quantization of the representative value of the bin, to estimate the total quantization error (T(f)) caused by quantizing the set of values ​​according to a possible fixed-point number format (f).

[0081] Specifically, the frequency value (F i ) indicates how many values ​​in the set of values ​​fall into each bin. Therefore, if the quantization error of each value in a bin is approximated by a distance value that represents the difference between the representative value of the bin in floating point format and the representative value of the bin in possible fixed point format (in other words, the distance value represents the quantization error of the representative value)), then the total quantization error T(f) caused by quantizing the set of values ​​according to the possible fixed point format (f) can be calculated by calculating the frequency value (F) of each bin. i ) and the distance value E of each bin Q (x i ,f) is estimated by summing the products, as shown in formula (3):

[0082] T(f)=∑ i H(x i )E Q (x i ,f) (3)

[0083] Minimizing the quantization error caused by quantizing the set of values ​​according to the fixed point format will minimize the total quantization error in the output of the layer. For example, for a weight vector w of length k in floating point format and an input data vector d of length k and producing an output a=w T d convolutional layer or fully connected layer, where E q (w k ,f) is the weight w of a given fixed-point number format f (e.g., f = {e,n}) defined by the exponent e and the mantissa bit length n k The distance value, and E q (d k ,f) is an input data value d of a given fixed-point number format f′ (e.g., f′={e′,n′}) defined by exponent e′ and mantissa bit length n′ k The total error (E) caused by quantization is given by equation (4):

[0084] E=∑ k E q (w k ,f)E q (d k,f′) (4)

[0085] Treating the weight w and the input data value d as random variables (represented by capital letters W and D, respectively), assuming that W is independent of D, and considering the expectation, the error can be given by equation (5). From equation (5), it can be seen that the total quantization error is related to the sum of the weights E q The distance values ​​associated with (W, f) are proportional, as shown in equation (6).

[0086] ε{E}=len(W)ε{E q (W,f)}ε{E q (D,f′)} (5)

[0087] ε{E}∝ε{E q (W,f)} (6)

[0088] From equation (6), it can be seen that by minimizing the distance values ​​associated with the weights (thereby minimizing the quantization error), the total quantization error can be minimized. The same is true for the input data, that is, by minimizing the distance values ​​associated with the input data values ​​(thereby minimizing the quantization error), the total quantization error can be minimized. The total quantization error associated with the weights is given in equation (7), where P(W) is the probability distribution function:

[0089]

[0090] As we all know, the histogram is an estimate of the probability distribution of a continuous variable. We can substitute the histogram H into P and replace the integral with the sum of all bins in the histogram, which yields equation (8):

[0091] ε{E q (W,f)}=∑ i H(x i )E Q (x i ,f) (8)

[0092] Then, if the number of mantissa bits is fixed, the optimal exponent e of the format f = {e, n} * The index e can be chosen as the one that minimizes the error of equation (8) from a set of possible indices, which is expressed as equation (9):

[0093]

[0094] Similarly, if the exponent e is fixed, the optimal number of mantissa digits n for the format f = {e,n} * The number of mantissa digits n can be selected as the number of mantissa digits that minimizes the error of equation (8) from a set of possible mantissa digits, which is expressed as equation (10):

[0095]

[0096] The representative value of a bin in a specific fixed-point format f (x i ) q (x i ,f)) can be calculated in any suitable way (taking into account the distance that the values ​​in a set of values ​​are moved from their original floating point format when quantized). In some cases, the representative value (x) of a bin of the histogram (H(x)) of a particular fixed point format f i ) q (x i ,f)) can be simply calculated as a floating point number format (x i ) and the quantized version of the representative value (Q(x i ,f)) (i.e., the representative value of the fixed-point number format f), as shown in equation (11):

[0097] E q (x i ,f)=x i -Q(x i ,f) (11)

[0098] For example, Figure 6 As shown, if the representative value x1 of the first bin (Bin1) of the histogram (H(x)) is quantized to Q(x1,f), then according to equation (11), the distance value E for the representative value x1 is q (x1, f) is x1–Q(x1, f). Similarly, if the representative value x2 of the second bin (Bin2) of the histogram (H(x)) is quantized to Q(x2, f), then according to equation (11), the distance value E for the representative value x2 is q (x2,f) is x2–Q(x2,f).

[0099] Testing has shown that the output errors of some layers tend to have a Gaussian distribution. Since it is known that the sum of squares produces a more accurate estimate in the presence of Gaussian errors, a more accurate estimate of the total quantized error can be determined when the distance value of the representative value is calculated as the squared difference between the representative value in floating point format and the quantized version of the representative value, as shown in equation (12):

[0100] E q (x i ,f)=(x i -Q(x i ,f)) 2 (12)

[0101] In this case, for Figure 5 and Figure 6The total quantization error of the fixed-point format f of the example histogram H(x) can be estimated as (F1*(x1-Q(x1,f)) 2 )+(F2*(x2-Q(x2,f)) 2 )+(F3*(x3-Q(x3,f)) 2 )+(F4*(x4-Q(x4,f)) 2 )+(F5*(x5-Q(x5,f)) 2 Once the total quantization error associated with the possible fixed-point number formats has been estimated, the method 400 proceeds to block 410.

[0102] At block 410, it is determined whether there are any other possible fixed-point number formats for which an estimate of the total quantization error has not been determined. If it is determined that there is at least one possible fixed-point number format in the plurality of possible fixed-point number formats for which an estimate of the total quantization error has not been determined, the method 400 returns to block 404. However, if it is determined that an estimate of the total quantization error has been determined for each of the plurality of possible fixed-point number formats, the method 400 proceeds to block 412, where a possible fixed-point number format is selected as a desired or optimal fixed-point number format to represent the set of values ​​for the layer.

[0103] At block 412, one of the plurality of possible fixed-point number formats is selected as a desired or optimal fixed-point number format for representing the set of values ​​for the layer based on the estimated total quantization errors associated with the possible fixed-point number formats. In some cases, the possible fixed-point number format associated with the lowest estimated total quantization error is selected as the fixed-point number format for representing the set of values ​​for the layer. However, in other cases, other criteria may be used to select one of the fixed-point number formats.

[0104] Although in Figure 4 In the example method 400 of , a single representative value of each bin is quantized according to each of a plurality of possible fixed-point number formats, and the distance value of the quantized representative value is used to estimate the total quantization error of each possible fixed-point number format, but in other examples, multiple representative values ​​of one or more bins may be quantized. In these cases, a combination (e.g., an average or mean) of the distance values ​​of the multiple representative values ​​of each bin in the one or more bins may be used to estimate the total quantization error of each possible fixed-point number format.

[0105] Once it has been Figure 4Method 400 determines a fixed-point number format for representing a set of values ​​for a layer, and the identified fixed-point number format can be used to configure a hardware implementation of a DNN. For example, the identified fixed-point number format can be used to configure the DNN to expect to receive input data values ​​or weights in an identified format. This can allow the hardware implementation to more efficiently process the input data of the layer. In another example, the identified fixed-point number format for a layer can be used to configure the DNN to convert the received input data values ​​or weights to an identified fixed-point number format when the DNN does not receive the input data values ​​or weights for the layer in an identified fixed-point number format, so as to allow the hardware implementation to more efficiently process the input data values ​​or weights of the layer. In yet another example, the identified fixed-point number format can be used to configure the hardware implementation to convert the output data of another layer fed to the layer into an identified fixed-point number format so as to provide the output data of the other layer to the layer in the identified fixed-point number format. Refer to the following Fig.16 Describes an example hardware implementation of a DNN and how the hardware implementation may be configured using the identified format.

[0106] Figure 4 The method 400 can be used to determine an appropriate fixed-point number format for representing any set of values ​​input to or output from any layer of a DNN. For example, Figure 4 The method 400 can be used to determine an appropriate fixed-point number format for representing input data values, output data values, weights, or biases of any layer of a DNN. The method 400 can identify different fixed-point number formats for the same type of values ​​of different layers. For example, the method 400 can identify one fixed-point number format for representing input data values ​​of one layer, and identify a different fixed-point number format for representing input data values ​​of another layer.

[0107] In some cases, the hardware implementation of the DNN can support different fixed-point formats for different parts of the input data values, output data values, weights, and / or biases of a layer. For example, the weights of a layer are divided into filters, and the hardware implementation can use different fixed-point formats for different filters or different filter groups. In these cases, method 400 can be used to identify an appropriate fixed-point format for representing a portion of the input data values, output data values, weights, or biases of a layer.

[0108] Outlier Weighting

[0109] It has been shown that in some cases, some values ​​in a set of values ​​(e.g., some input data values ​​or some weights) may be more important than other values. For example, in some convolutional layers and some fully connected layers, larger weights tend to be associated with key features, and smaller weights tend to be associated with less important input data values. As a result, outliers may be more important than smaller weights or input data values, and thus quantization errors of outliers may have a greater impact on the overall error of a layer or DNN than quantization errors of smaller weights or input data values. Therefore, in these cases, a more appropriate trade-off between accuracy and range may give more weight to the quantization errors of representative values ​​that are outliers (e.g., falling outside the representable range of the fixed-point format) than to the quantization errors of representative values ​​that are not outliers (e.g., falling within the representable range of the fixed-point format). This trade-off can be achieved by applying weights to the quantization errors of representative values, where errors associated with representative values ​​that fall outside the range of values ​​that can be represented by a particular fixed-point format are given a higher weight (compared to quantization errors associated with representative values ​​that fall within the range of values ​​that can be represented by a particular fixed-point format). This encourages the optimization algorithm to increase the range to cover more outliers.

[0110] Thus, in some cases, the distance value of a representative value of a bin of a particular fixed-point number format f can be calculated as the product of the squared error (as described above with respect to equation (12)) and the weight α(x, f) associated with the representative value, as shown in equation (13), where lower weights are assigned to representative values ​​that fall within the representable range of the fixed-point number format and higher weights are assigned to representative values ​​that fall outside the representable range of the fixed-point number format, where x max is the largest representable number in fixed-point format, X min is the smallest representable number in fixed-point format, and a is a predetermined value:

[0111]

[0112] Any weighting function α(x,f) can be used, where higher weights are given to those that fall into fixed-point format (given by X max and X min A representative value is said to be outside the representable range of a fixed-point format (as defined by the fixed-point format) relative to the weight given to a representative value that is within the representable range of the fixed-point format. A representative value is said to be within the representable range of a fixed-point format if it is between the maximum representable number and the minimum representable number of the fixed-point format.

[0113] Testing has shown that in some cases, a weighting function that applies constant weights to representative values ​​that fall within the representable range of the fixed-point number format and linearly increasing weights to representative values ​​that fall outside the representable range of the fixed-point number format is effective. Examples of such weighting functions are shown in Figure 7 is shown in the diagram 700 of , and is represented by equation (14), where δ e is the distance between the quantization levels of the possible fixed-point number formats f, and c and γ are constants that can be determined empirically:

[0114]

[0115] Testing has shown that γ=20 works well in some cases. The constant c is selected to ensure continuity between quantized values ​​in a set of values ​​(i.e., values ​​that fall within the representable range of the fixed-point number format) and saturated values ​​in the set of values ​​(i.e., values ​​that fall outside the representable range of the fixed-point number format). Testing has shown that c=1-γ works well. It will be apparent to one skilled in the art that these are merely example values ​​for the constants and that other values ​​may be used.

[0116] Floating point format to fixed point format conversion

[0117] As mentioned above, in Figure 4 At block 406 of method 400, the representative input value (x) of the bin of the histogram (H(x)) i ) is quantized from floating point format to possible fixed point format. Equation (15) gives the value x for converting the floating point format i Quantized to a fixed-point value Q(x i ,f) where X max is the largest representable number in fixed-point format, X min is the smallest representable number in fixed-point format, and RND(x i ) is the rounding function:

[0118]

[0119] The formula shown in equation (15) quantizes an input data value or weight in a floating point format to the closest available representable number in a fixed point format, where the "nearest" available representable number is determined based on a rounding function RND. The rounding function RND can be, for example, one of round to zero (RTZ) and round to nearest even (RTE). Since the representative value in the fixed point format may have reduced precision relative to the original floating point format, quantization errors may occur. Test Results

[0120] Reference now Figures 8 to 15 , Figures 8 to 15 The classification accuracy of an example DNN is shown when the exponents for the weights of different layers are selected according to different methods, including the full-range method described above for equation (1), the full-range method described above for equation (2), and the classification accuracy of an example DNN when the exponents for the weights of different layers are selected according to different methods, including the full-range method described above for equation (1). Figure 4The method described above in which the distance value is calculated as the squared error, and Figure 4 Describes a method in which the distance value is calculated as the product of the weight and the squared error.

[0121] Figure 8 and 9 Graphs 800 and 900 are respectively shown of the top 1 and top 5 classification accuracy of the AlexNet DNN with the ImageNet validation dataset when the weights of each layer are represented in a fixed-point format (where the mantissa bit length is fixed and the exponent is selected according to various methods). Specifically, curves 802 and 902 show the top 1 / top 5 classification accuracy of the AlexNet DNN when the exponent for the weights of each layer is selected using the full range method described above for equation (1); curves 804 and 904 show the top 1 / top 5 classification accuracy of the AlexNet DNN when the exponent for the weights of each layer is selected using the full range method described above for equation (1) and reduced by 1; curves 806 and 906 show the top 1 / top 5 classification accuracy of the AlexNet DNN when the exponent for the weights of each layer is selected using the full range method described above for equation (1) and reduced by 1; curves 807 and 908 show the top 1 / top 5 classification accuracy of the AlexNet DNN when the exponent for the weights of each layer is selected using Figure 4 The top 1 / top 5 classification accuracy of AlexNet DNN when the method 400 is selected and the distance value is calculated as the squared error (e.g., the distance value is calculated according to equation (12)); and curves 808 and 908 show when the exponent for the weight of each layer is used Figure 4 The top 1 / top 5 classification accuracy of AlexNet DNN when the method 400 is selected and the distance value is calculated as the product of the weight and the squared error (e.g., the distance value is calculated according to equation (13)). As known to those skilled in the art, the top 1 classification accuracy is a measure of whether the highest output of the DNN is correctly classified, and the top 5 classification accuracy is a measure of whether the first five outputs of the DNN are correctly classified.

[0122] Fig.10 and 11Graphs 1000 and 1100 respectively show the top 1 and top 5 classification accuracy of AlexNet DNN when the weights and input data values ​​of each layer are represented in fixed-point format (where the mantissa bit length is fixed and the exponent is selected according to various methods). Specifically, curves 1002 and 1102 show the top 1 / top 5 classification accuracy of AlexNet DNN when the exponents for the weights and input data values ​​of each layer are selected using the full range method described above for equation (1); curves 1004 and 1104 show the top 1 / top 5 classification accuracy of AlexNet DNN when the exponents for the weights and input data values ​​of each layer are selected using the full range method described above for equation (1) and reduced by 1; curves 1006 and 1106 show the top 1 / top 5 classification accuracy of AlexNet DNN when the exponents for the weights and input data values ​​of each layer are selected using the full range method described above for equation (1) and reduced by 1. Figure 4 The top 1 / top 5 classification accuracy of AlexNet DNN when the method 400 is selected and the distance value is calculated as the squared error (e.g., the distance value is calculated according to equation (12)); and curves 1008 and 1108 show when the exponentiation of the weights and input data values ​​for each layer is used Figure 4 The top 1 / top 5 classification accuracy of AlexNet DNN when method 400 is selected and the distance value is calculated as the product of the weight and the squared error (e.g., the distance value is calculated according to equation (13)).

[0123] Fig.12 and 13 Graphs 1200 and 1300 are respectively shown of the top 1 and top 5 classification accuracy of the GoogLeNet DNN with the ImageNet validation dataset when the weights of each layer are represented in a fixed-point format (where the mantissa bit length is fixed and the exponent is selected according to various methods). Specifically, curves 1202 and 1302 show the top 1 / top 5 classification accuracy of the GoogLeNet DNN when the exponent for the weights of each layer is selected using the full range method described above for equation (1); curves 1204 and 1304 show the top 1 / top 5 classification accuracy of the GoogLeNet DNN when the exponent for the weights of each layer is selected using the full range method described above for equation (1) and reduced by 1; curves 1206 and 1306 show the top 1 / top 5 classification accuracy of the GoogLeNet DNN when the exponent for the weights of each layer is selected using the full range method described above for equation (1) and reduced by 1; curves 1207 and 1308 show the top 1 / top 5 classification accuracy of the GoogLeNet DNN when the exponent for the weights of each layer is selected using Figure 4 The top 1 / top 5 classification accuracy of the GoogLeNet DNN when the method 400 is selected and the distance value is calculated as the squared error (e.g., the distance value is calculated according to equation (12)); and curves 1208 and 1308 show that when the index for the weight of each layer is used Figure 4The top 1 / top 5 classification accuracy of the GoogLeNet DNN when method 400 is selected and the distance value is calculated as the product of the weight and the squared error (e.g., the distance value is calculated according to equation (13)).

[0124] Fig.14 and 15 Graphs 1400 and 1500 are respectively shown of the top 1 and top 5 classification accuracy of the GoogLeNet DNN when the weights and input data values ​​of each layer are represented in a fixed-point format (where the mantissa bit length is fixed and the exponent is selected according to various methods). Specifically, curves 1402 and 1502 show the top 1 / top 5 classification accuracy of the GoogLeNet DNN when the exponents for the weights and input data values ​​of each layer are selected using the full range method described above for equation (1); curves 1404 and 1504 show the top 1 / top 5 classification accuracy of the GoogLeNet DNN when the exponents for the weights and input data values ​​of each layer are selected using the full range method described above for equation (1) and reduced by 1; curves 1406 and 1506 show the top 1 / top 5 classification accuracy of the GoogLeNet DNN when the exponents for the weights and input data values ​​of each layer are selected using the full range method described above for equation (1) and reduced by 1. Figure 4 The top 1 / top 5 classification accuracy of the GoogLeNet DNN when the method 400 is selected and the distance value is calculated as the squared error (e.g., the distance value is calculated according to equation (12)); and curves 1408 and 1508 show when the exponentiation of the weights and input data values ​​for each layer is used Figure 4 The top 1 / top 5 classification accuracy of the GoogLeNet DNN when method 400 is selected and the distance value is calculated as the product of the weight and the squared error (e.g., the distance value is calculated according to equation (13)).

[0125] from Figure 8-15 It can be seen that the described histogram-based approach selects a fixed-point number format that improves classification accuracy, especially in the case of small numbers (ie, little-endian bit length), compared to selecting the fixed-point number format using the full-range approach.

[0126] Example Hardware Implementation of DNN

[0127] Reference now Fig.16 , Fig.16 shows that can be configured based on the use Figure 4 An example hardware implementation 1600 of a DNN of the format identified by method 400 .

[0128] Fig.16The hardware implementation 1600 is configured to compute the output of the DNN through a series of hardware passes (which may also be referred to as processing passes), wherein during each pass, the hardware implementation receives at least a portion of the input data of a layer of the DNN, and processes the received input data according to the layer (and optionally according to one or more subsequent layers) to produce processed data. The processed data is output to a memory to be used as input data for a subsequent hardware pass, or is output as the output of the DNN. The number of layers that the hardware implementation can process during a single hardware pass can be based on the size of the data, the hardware implementation, and the order of the layers. For example, where the hardware implementation includes hardware for executing each possible layer type, a DNN including a first convolutional layer, a first activation layer, a second convolutional layer, a second activation layer, and a pooling layer can receive initial DNN input data in a first hardware pass and process the input data according to the first convolutional layer and the first activation layer and then output the output of the activation layer to a memory, and then receive the data from the memory as input in a second hardware pass and process the data according to the second convolutional layer, the second activation layer, and the pooling layer to produce the output data of the DNN.

[0129] Fig.16 The example hardware implementation 1600 of includes: input module 1601, convolution engine 1602, accumulation buffer 1604, element-wise operation module 1606, activation module 1608, normalization module 1610, pooling module 1612, output interleaving module 1614, and output module 1615. Each module or engine implements or processes all or part of one or more types of layers. Specifically, convolution engine 1602 and accumulation buffer 1604 together implement or process convolutional layers or fully connected layers. Activation module 1608 processes or implements activation layers. Normalization module 1610 processes or implements normalization layers. Pooling module 1612 implements pooling layers, and output interleaving module 1614 processes or implements interleaving layers.

[0130] The input module 1601 is configured to receive input data to be processed and provide the input data to downstream modules for processing.

[0131] The convolution engine 1602 is configured to perform a convolution operation on the received input data using the weights associated with a particular convolution layer. The weights of each convolution layer of the DNN may be stored in a coefficient buffer 1616 (e.g., Fig.16 As shown), and when the convolution engine 1602 is processing a specific convolution layer, the weight of the specific convolution layer can be provided to the convolution engine 1602. In the case where the hardware implementation supports a variable weight format, the convolution engine 1602 can be configured to receive information indicating one or more formats of the weight of the current convolution layer being processed to allow the convolution engine to correctly interpret and process the received weights.

[0132] The convolution engine 1602 may include a plurality of multipliers (e.g., 128) and a plurality of adders (which sum the results of the multipliers to produce a single sum). Fig.16 A single convolution engine 1602 is shown in FIG. 1 , but in other examples, there may be multiple (eg, 8) convolution engines so that multiple windows can be processed simultaneously. The output of the convolution engine 1602 is fed to an accumulation buffer 1604 .

[0133] Accumulation buffer 1604 is configured to receive the output of the convolution engine and add the output of the convolution engine to the current contents of accumulation buffer 1604. In this manner, accumulation buffer 1604 accumulates the results of convolution engine 1602. Fig.16 A single accumulation buffer 1604 is shown in FIG, but in other examples, there may be multiple (e.g., 8, one accumulation buffer for each convolution engine) accumulation buffers. Accumulation buffer 1604 outputs the accumulation result to element-wise operation module 1606, which may operate or not operate on the accumulation result based on whether the element-wise layer is to be processed during the current hardware pass.

[0134] The element-wise operation module 1606 is configured to receive input data for the current hardware channel (e.g., when the convolution layer is not processed in the current hardware channel) or the accumulated results from the accumulation buffer 1604 (e.g., when the convolution layer is processed in the current hardware channel). The element-wise operation module 1606 can process the received input data, or pass the received input data to another module (e.g., the activation module 1608 and / or the normalization module 1610), depending on whether the element-wise layer is processed in the current hardware channel and / or whether the activation layer is to be processed before the element-wise layer. When the element-wise operation module 1606 is configured to process the received input data, the element-wise operation module 1606 performs an element-wise operation on the received data (optionally together with another data set (the other data set can be obtained from an external memory)). The element-wise operation module 1606 can be configured to perform any suitable element operation, such as, but not limited to, addition, multiplication, maximum, and minimum. The result of the element-wise operation is then provided to the activation module 1608 or the normalization module 1610, depending on whether the activation layer is to be processed after the element-wise layer.

[0135] The activation module 1608 is configured to receive one of the following as input data: the original input entering the hardware channel (through the element-wise operation module 1606) (for example, when the convolution layer is not processed in the current hardware channel); the accumulated data (through the element-wise operation module 1606) (for example, when the convolution layer is processed in the current hardware channel and the element-wise layer is not processed in the current hardware channel or the element-wise layer is processed in the current hardware channel but followed by the activation layer). The activation module 1608 is configured to apply the activation function to the input data and provide the output data back to the element-wise operation module 1606 (wherein the output data is forwarded directly to the normalization module 1610 or forwarded to the normalization module 1610 after being processed by the element-wise operation module 1606). In some cases, the activation function applied to the data received by the activation module 1608 may vary depending on the activation layer. In these cases, information specifying one or more properties of the activation function to be applied to each activation layer can be stored (e.g., in a memory), and relevant information for the activation layers processed in a particular hardware channel can be provided to activation module 1608 during the hardware channel.

[0136] In some cases, the activation module 1608 can be configured to store data representing the activation function in an entry of a lookup table. In these cases, the input data can be used to look up one or more entries in the lookup table and output a value representing the output of the activation function. For example, the activation module 1608 can be configured to calculate an output value by interpolating between two or more entries read from the lookup table.

[0137] In some examples, activation module 1608 can be configured to operate as a rectified linear unit (ReLU) by implementing a ReLU function. In the ReLU function, the output element y is calculated by identifying the maximum value i,j,k , as shown in equation (16), where y=0 for values ​​of x less than 0:

[0138] y i,j,k =f(x i,j,k )=max{0,x i,j,k} (16)

[0139] In other examples, the activation module 1608 can be configured to operate as a parameterized rectified linear unit (PReLU) function by implementing a PReLU function. The PReLU function performs similar operations to the ReLU function. Specifically, in w1, w2, b1, is a constant, PReLU is configured to generate output elements y i,j,k , as shown in equation (17):

[0140] y i,j,k =f(xi,j,k ;w1,w2,b1,b2)=max{(w1*x i,j,k +b1),(w2*x i,j,k +b2)} (17)

[0141] The normalization module 1610 is configured to receive one of the following as input data: the original input data for the hardware channel (through the element-by-element operation module 1606) (for example, when the convolution layer is not processed in the current hardware channel and neither the element-by-element layer nor the activation layer is processed in the current hardware channel); the accumulated output (through the element-by-element operation module 1606) (for example, when the convolution layer is processed in the current hardware channel and neither the element-by-element layer nor the activation layer is processed in the current hardware channel); and the output data of the element-by-element operation module and / or the activation module. The normalization module 1610 then performs a normalization function on the received input data to generate normalized data. In some cases, the normalization module 1610 can be configured to perform a local response normalization (LRN) function and / or a local contrast normalization (LCN) function. However, it is apparent to those skilled in the art that these are merely examples and the normalization module 1610 can be configured to implement any appropriate normalization function. Different normalization layers can be configured to apply different normalization functions.

[0142] The pooling module 1612 may receive normalized data from the normalization module 1610, or may receive input data entering the normalization module 1610 via the normalization module 1610. In some cases, data may be transferred between the normalization module 1610 and the pooling module 1612 via the XBar 1618. The term "XBar" as used herein refers to a simple hardware module that includes routing logic that connects multiple modules together in a dynamic manner. In this example, the XBar may dynamically connect the normalization module 1610, the pooling module 1612, and / or the output interleaving module 1614 based on which layers will be processed in the current hardware channel. Thus, the XBar may receive information for each channel indicating which modules 1610, 1612, 1614 are to be connected.

[0143] The pooling module 1612 is configured to perform a pooling function (such as, but not limited to, a maximum function or an average function) on the received data to generate pooled data. The purpose of the pooling layer is to reduce the spatial size of the representation to reduce the number of parameters and calculations in the network, and thus also control overfitting. In some examples, the pooling operation is performed on a sliding window defined by each pooling layer.

[0144] The output interleaving module 1614 may receive normalized data from the normalization module 1610, input data entering a normalization function (via the normalization module 1610), or pooled data from the pooling module 1612. In some cases, data may be transferred between the normalization module 1610, the pooling module 1612, and the output interleaving module 1614 via the XBar 1618. The output interleaving module 1614 is configured to perform a rearrangement operation to produce data in a predetermined order. This may include sorting and / or transposing the received data. The data produced by the last layer is provided to the output module 1615, where the data is converted into the desired output format for the current hardware channel.

[0145] Normalization module 1610, pooling module 1612 and output interleaving module 1614 can each access shared buffer 1620, and shared buffer 1620 can be used by these modules 1610, 1612 and 1614 to write data and retrieve data therefrom. For example, shared buffer 1620 can be used by these modules 1610, 1612, 1614 to rearrange the order of received data or generated data. For example, one or more of these modules 1610, 1612, 1614 can be configured to write data to shared buffer 1620 and read out the same data in a different order. In some cases, although each of normalization module 1610, pooling module 1612 and output interleaving module 1614 can access shared buffer 1620, each of normalization module 1610, pooling module 1612 and output interleaving module 1614 can be allocated a portion of shared buffer 1620 that only they can access. In these cases, each of the normalization module 1610 , the pooling module 1612 , and the output interleaving module 1614 may only be able to read from the shared buffer 1620 the data that they have written to the shared buffer 1620 .

[0146] As described above, the modules of the hardware implementation 1600 used or activated during any hardware pass are based on the layers processed during that hardware pass. Specifically, only modules or components related to the layers processed during the current hardware pass are used or activated. As described above, the layers processed during a particular hardware pass are determined based on the order of the layers in the DNN and, optionally, one or more other factors (e.g., the size of the data) (usually determined in advance by, for example, a software tool). For example, in some cases, the hardware implementation may be configured to perform processing of a single layer per hardware pass unless multiple layers can be processed without writing data to memory between layers. For example, if a first convolutional layer is immediately followed by a second convolutional layer, each convolutional layer will have to be executed in a separate hardware pass because the output data from the first convolution needs to be written to memory before it can be used as input to the second convolution. In each of these hardware passes, only modules, components, or engines related to the convolutional layers, such as the convolution engine 1602 and the accumulation buffer 1604, may be used or activated.

[0147] Although Fig.16 The hardware implementation 1600 of exemplifies a particular order in which modules, engines, etc. are arranged and thus how processing of data flows through the hardware implementation, but it should be understood that this is merely an example and that in other examples, modules, engines, etc. may be arranged in a different manner. Furthermore, other hardware implementations may implement additional or alternative types of DNN layers and thus may include different modules, engines, etc.

[0148] Fig.17 Various components of an exemplary general-purpose computing-based device 1700 are shown, which may be implemented as any form of computing and / or electronic device and in which embodiments of the above-described method 400 may be implemented.

[0149] The computing-based device 1700 includes one or more processors 1702, which may be microprocessors, controllers, or any other suitable type of processors for processing computer executable instructions to control the operation of the device in order to evaluate the performance of the integrated circuit defined by the hardware design in completing the task. In some examples, such as where a system-on-chip architecture is used, the processor 1702 may include one or more fixed function blocks (also referred to as accelerators) that implement in hardware (rather than software or firmware) a portion of a method for determining a fixed-point number format for representing a set of values ​​for input to and output from a layer of a DNN. Platform software including an operating system 1704 or any other suitable platform software may be provided at the computing-based device to enable application software, for example, to be executed on the device for implementing Figure 4Computer executable code 1705 of method 400 .

[0150] Computer executable instructions can be provided using any computer-readable medium accessible by computing-based device 1700. Computer-readable media may include, for example, computer storage media (e.g., memory 1706) and communication media. Computer storage media (i.e., non-transient machine-readable media), for example, memory 1706, include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, cassette, tape, disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information for access by a computing device. In contrast, communication media can embody computer-readable instructions, data structures, program modules or other data with modulated data signals (e.g., carrier waves) or other transmission mechanisms. As defined herein, computer storage media do not include communication media. Although computer storage media (i.e., non-transitory machine-readable media, such as memory 1706) are shown within computing-based device 1700, it should be understood that the storage devices may be distributed or located remotely and may be accessed via a network or other communication link (e.g., using communication interface 1708).

[0151] The computing-based device 1700 also includes an input / output controller 1710 that is arranged to output display information to a display device 1712, which may be separate from or integrated with the computing-based device 1700. The display information may provide a graphical user interface. The input / output controller 1710 is also arranged to receive and process input from one or more devices, such as a user input device 1714 (e.g., a mouse or keyboard). In one embodiment, if the display device 1712 is a touch-sensitive display device, the display device 1712 may also be used as a user input device 1714. The input / output controller 1710 may also output data to devices other than the display device, such as a locally connected printing device (e.g., a printer). Fig.17 not shown).

[0152] Fig.18 18 shows a computer system in which the hardware implementation of the DNN described herein can be implemented. The computer system includes a CPU 1802, a GPU 1804, a memory 1806, and other devices 1814, such as a display 1816, a speaker 1818, and a camera 1820. The hardware implementation 1810 of the DNN (corresponding to Fig.16 The hardware implementation 1600 of the DNN can be implemented on the GPU 1804, such as Fig.18 In other examples, the hardware implementation 1810 of the DNN may be implemented independently of the CPU or GPU and may have a separate connection to the communication bus 1822. In some examples, there may be no GPU, and the CPU may provide control information to the hardware implementation 1810 of the DNN. The components of the computer system may communicate with each other via the communication bus 1822.

[0153] Fig.16 The hardware implementation 1600 of the DNN is shown as including multiple functional blocks. This is merely illustrative and is not intended to define a strict division between different logical elements of such an entity. Each functional block may be provided in any appropriate manner. It should be understood that the intermediate values ​​formed by the hardware implementation or processing module of the DNN described herein do not need to be physically generated by the hardware implementation or processing module of the DNN at any point, and may simply represent a logical value between the input and output of the processing performed by the DNN hardware implementation or processing module.

[0154] The hardware implementation of the DNN described herein can be implemented in hardware on an integrated circuit. In general, any of the functions, methods, techniques, or components described above can be implemented in software, firmware, hardware (e.g., fixed logic circuits), or any combination thereof. The terms "module," "function," "component," "element," "unit," "block," and "logic" may be used herein to generally represent software, firmware, hardware, or any combination thereof. In the case of a software implementation, a module, function, component, element, unit, block, or logic represents a program code that performs a specified task when executed on a processor. The algorithms and methods described herein can be performed by one or more processors that execute code that causes (one or more) processors to perform the algorithm / method. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disks, flash memory, hard disk storage, and other memory devices that can use magnetic, optical, and other technologies to store instructions or other data and can be accessed by a machine.

[0155] The terms computer program code and computer readable instructions as used herein refer to any type of executable code for a processor, including code expressed in a machine language, an interpreted language, or a scripting language. Executable code includes binary code, machine code, byte code, code that defines an integrated circuit (e.g., a hardware description language or a netlist), and code expressed in a programming language code such as C, Java, or OpenCL. Executable code can be, for example, any type of software, firmware, script, module, or library that, when properly executed, processed, interpreted, compiled, run in a virtual machine or other software environment, enables a processor of a computer system to perform the tasks specified by the code in support of the executable code.

[0156] A processor, computer or computer system may be any type of device, machine or special purpose circuit, or a collection or portion thereof, that has processing capabilities to execute instructions. A processor may be any type of general or special purpose processor, such as a CPU, GPU, system on chip, state machine, media processor, application specific integrated circuit (ASIC), programmable logic array, field programmable gate array (FPGA), etc. A computer or computer system may include one or more processors.

[0157] It is also intended to cover software that defines the configuration of hardware as described herein, such as HDL (Hardware Description Language) software, e.g., for designing integrated circuits or for configuring a programmable chip to perform a desired function. That is, a computer-readable storage medium may be provided having encoded thereon computer-readable program code in the form of an integrated circuit definition data set that, when processed (i.e., run) in an integrated circuit manufacturing system, configures the system to manufacture a hardware implementation of the DNN described herein. The integrated circuit definition data set may be, for example, an integrated circuit description.

[0158] Thus, a method of manufacturing a hardware implementation of a DNN as described herein at an integrated circuit manufacturing system may be provided. In addition, an integrated circuit definition data set may be provided that, when processed in an integrated circuit manufacturing system, enables execution of a method of manufacturing a hardware implementation of a DNN.

[0159] An integrated circuit definition data set may be in the form of computer code, such as a netlist, code for configuring a programmable chip, a hardware description language (defining hardware suitable for fabrication in an integrated circuit at any level), including register transfer level (RTL) code, high-level circuit representations (e.g., Verilog, or VHDL), and low-level circuit representations (e.g., OASIS (RTM), and GDSII). A higher-level representation (e.g., RTL) that logically defines hardware suitable for fabrication in an integrated circuit may be executed at a computer system (configured to generate a manufacturing definition of an integrated circuit in the context of a software environment (including definitions of circuit elements and rules for combining such elements)) to generate a manufacturing definition of the integrated circuit defined by the representation. Similar to the case where software is typically executed at a computer system to define a machine, one or more intermediate user steps (e.g., providing commands, variables, etc.) may be required to cause the computer system (configured to generate a manufacturing definition of an integrated circuit) to execute the code defining the integrated circuit, thereby generating a manufacturing definition of the integrated circuit.

[0160] Now refer to Fig.19 An example of a hardware implementation of processing an integrated circuit definition data set at an integrated circuit manufacturing system to configure the system to manufacture a DNN is described.

[0161] Fig.19 An example of an integrated circuit (IC) manufacturing system 1902 (configured to manufacture a hardware implementation of a DNN described in any of the examples herein) is shown. Specifically, the IC manufacturing system 1902 includes a layout processing system 1904, and an integrated circuit generation system 1906. The IC manufacturing system 1902 is configured to receive an IC definition data set (e.g., defining a hardware implementation of a DNN described in any of the examples herein), process the IC definition data set, and generate an IC (e.g., the IC embodies the hardware implementation of the DNN described in any of the examples herein) based on the IC definition data set. The processing of the IC definition data set configures the IC manufacturing system 1902 to manufacture an integrated circuit embodying a hardware implementation of a DNN described in any of the examples herein.

[0162] The layout processing system 1904 is configured to receive and process an IC definition data set to determine a circuit layout. Methods for determining a circuit layout from an IC definition data set are known in the art, and may, for example, include synthesizing RTL code to determine a gate level representation of the circuit to be generated, for example, in terms of logic components (e.g., NAND, NOR, AND, OR, MUX, and FLIP-FLOP components). By determining the location information of the logic components, the circuit layout can be determined from the gate level representation of the circuit. This can be done automatically or with user participation in order to optimize the circuit layout. When the layout processing system 1904 has determined the circuit layout, it can output a circuit layout definition to the IC generation system 1906. The circuit layout definition can be, for example, a circuit layout description.

[0163] As is known in the art, IC generation system 1906 generates an IC according to a circuit layout definition. For example, IC generation system 1906 may implement a semiconductor device manufacturing process to generate an IC, which may include a multi-step sequence of photolithography and chemical processing steps during which electronic circuits are gradually formed on a wafer made of semiconductor material. The circuit layout definition may be in the form of a mask that may be used in a photolithography process for generating an IC according to the circuit definition. Alternatively, the circuit layout definition provided to IC generation system 1906 may be in the form of a computer-readable code that IC generation system 1906 may use to form an appropriate mask for use in generating an IC.

[0164] The different processes performed by IC manufacturing system 1902 may all be implemented at one location (e.g., by one party). Alternatively, IC manufacturing system 1902 may be a distributed system such that some processes may be performed at different locations and may be performed by different parties. For example, some stages in (i) synthesizing RTL code representing an IC definition data set to form a gate-level representation of a circuit to be generated, (ii) generating a circuit layout based on the gate-level representation, (iii) forming a mask based on the circuit layout, and (iv) using the mask to manufacture an integrated circuit may be performed at different locations and / or by different parties.

[0165] In other examples, processing an integrated circuit definition data set at an integrated circuit manufacturing system may configure the system to manufacture a hardware implementation of the DNN (without processing the IC definition data set to determine a circuit layout). For example, the integrated circuit definition data set may define a configuration of a reconfigurable processor (e.g., an FPGA), and processing of the data set may configure the IC manufacturing system to generate a reconfigurable processor having the defined configuration (e.g., by loading the configuration data into the FPGA).

[0166] In some embodiments, when the integrated circuit manufacturing definition data set is processed in the integrated circuit manufacturing system, it can cause the integrated circuit manufacturing system to generate the device as described herein. Fig.19 The described approach uses an integrated circuit manufacturing definition data set to configure an integrated circuit manufacturing system so that the device described herein can be manufactured.

[0167] In some examples, the integrated circuit definition data set may include software that runs on or in conjunction with hardware defined by the data set. Fig.19 In the example shown, the IC generation system can be further configured by the integrated circuit definition dataset to load firmware onto the integrated circuit according to the program code defined in the integrated circuit definition dataset when manufacturing the integrated circuit, or otherwise provide the integrated circuit with program code for use with the integrated circuit.

[0168] Compared with known implementations, the implementation of the concepts set forth in this application in devices, apparatuses, modules, and / or systems (and in the methods implemented herein) can produce performance improvements. Performance improvements can include one or more of increased computing performance, reduced delays, increased throughput, and / or reduced power consumption. During the manufacture of such devices, devices, modules, and systems (e.g., in integrated circuits), performance improvements can be weighed against physical implementations, thereby improving manufacturing methods. For example, performance improvements can be weighed against layout area, thereby matching the performance of known implementations, but using less silicon. This can be accomplished, for example, by reusing functional blocks in a serialized manner, or sharing functional blocks between elements of devices, devices, modules, and / or systems. On the contrary, the concepts proposed in this application that produce improvements (e.g., reduced silicon area) to the physical implementations of devices, devices, modules, and systems can be weighed against performance improvements. This can be accomplished, for example, by manufacturing multiple instances of modules within a predefined area budget.

[0169] The applicant hereby independently discloses each individual feature described herein and any combination of two or more such features, including such features or combinations that can be performed based on this specification as a whole according to the general common sense of those skilled in the art, regardless of whether such features or combinations of features solve any problems disclosed herein. In view of the foregoing description, it is obvious to those skilled in the art that various modifications can be made within the scope of the present invention.

Claims

1. A computer-implemented method (400) of identifying a fixed-point number format for representing a set of values ​​input to and / or output from a layer of a deep neural network (DNN) for use in configuring a hardware implementation of the DNN, the method (400) comprising: Obtaining a histogram representing an expected distribution of the set of values ​​for the layer, wherein each bin of the histogram is associated with a frequency value (402); For each fixed point number format (404, 410) of a plurality of fixed point number formats: quantizing a representative value of the floating point number format for each bin according to the fixed point number format (406); and estimating a total quantization error associated with the fixed point number format based on the frequency value and a distance value for each bin, the distance value being based on quantization of a representative value for the bin (408); and A fixed point number format associated with a minimum estimated total quantization error among a plurality of fixed point number formats is selected as the fixed point number format for representing the set of values ​​for the layer (412).

2. The method (400) of claim 1, wherein: Each of the plurality of fixed-point number formats includes an exponent and a mantissa bit length.

3. The method (400) of claim 2, wherein: Each of the plurality of fixed-point number formats includes a same mantissa bit length and a different exponent.

4. The method (400) of claim 2, wherein: Each of the plurality of fixed-point number formats includes a different mantissa bit length and a same exponent.

5. The method (400) of claim 1, wherein: Estimating the total quantization error associated with the fixed point number format includes determining the sum of the products of the frequency value and the distance value for each bin.

6. The method (400) of claim 1, wherein: The distance value for each bin is the difference between the representative value for the bin in the floating point number format and the representative value for the bin quantized according to the fixed point number format.

7. The method (400) of claim 1, wherein: The distance value for each bin is the square difference between the representative value for the bin in the floating point number format and the representative value for the bin quantized according to the fixed point number format.

8. The method (400) of claim 1, wherein: The distance value for each bin is the product of the square difference between the representative value for the bin in floating-point format and the representative value for the bin quantized according to the fixed-point format and the weight of the representative value of the bin, wherein the weight of the representative value outside the representable range of the fixed-point format is higher than the weight of the representative value within the representable range of the fixed-point format.

9. The method (400) of claim 8, wherein: When a representative value falls between a maximum representable number in the fixed-point number format and a minimum representable number in the fixed-point number format, the representative value is within a representable range of the fixed-point number format.

10. The method (400) of claim 8, wherein: The weights increase linearly outside the representable range of the fixed-point number format.

11. The method (400) according to any one of claims 1 to 10, further comprising storing a selected fixed-point number format for use in configuring a hardware implementation of the DNN.

12. The method (400) of any one of claims 1 to 10, further comprising configuring a hardware implementation of the DNN to represent the set of values ​​using a selected fixed-point number format.

13. The method (400) according to any one of claims 1 to 10, wherein: The representative value for a bin is the center value of that bin.

14. A computer-readable storage medium having computer-readable instructions encoded thereon, which, when executed on a computer system, cause the computer system to perform the method of any one of claims 1 to 10.

15. A computing-based device for identifying a fixed-point number format for representing a set of values ​​input to and / or output from a layer of a deep neural network (DNN) for use in configuring a hardware implementation of the DNN, the computing-based device comprising: at least one processor; as well as a memory coupled to the at least one processor, the memory comprising: a histogram representing the expected distribution of the set of values ​​for the layer, wherein each bin of the histogram is associated with a frequency value; and The computer readable code, when executed by the at least one processor, causes the at least one processor to perform the following operations: For each of several fixed-point number formats: quantizing the representative value of the floating point format for each bin according to the fixed point number format; and estimating a total quantization error associated with the fixed point number format based on a frequency value and a distance value for each bin, the distance value being based on quantization of a representative value for the bin; and A fixed-point number format associated with a minimum estimated total quantization error among the plurality of fixed-point number formats is selected as the fixed-point number format for representing the set of values ​​for the layer.

16. A hardware implementation device of a deep neural network (DNN), comprising: The hardware logic is configured as: Receiving an input data value into a layer of the DNN; receiving information indicating a fixed-point format of input data values ​​for the layer, the fixed-point format of input data values ​​for the layer having been selected according to the method of any one of claims 1 to 10; interpreting the input data value based on a fixed-point number format for the input data value for the layer; and The interpreted input data values ​​are processed according to the layer to generate output data values ​​for the layer.

17. The hardware implementation device of DNN according to claim 16, wherein: The hardware logic is also configured to: receiving information indicating a fixed-point format for output data values ​​for the layer, the fixed-point format for output data values ​​for the layer having been selected according to the method of any one of claims 1 to 10; and Convert the output data value for the layer to a fixed-point number format for the output data value for the layer.

18. The hardware implementation device of DNN according to claim 17, wherein: The fixed-point number format of the input data values ​​for the layer is different from the fixed-point number format of the output data values ​​for the layer.

19. The hardware implementation device according to claim 16, wherein: The hardware logic is also configured to: receiving a set of weights for the layer; Receiving information indicating a fixed-point number format for weights of the layer, the fixed-point number format for weights of the layer having been selected according to the method of any one of claims 1 to 10; Interpret the weights based on the fixed-point number format for the weights of this layer; and The interpreted input data values ​​are processed according to the interpreted weights to generate output data values ​​for the layer.

20. A computer-readable storage medium having stored thereon a computer-readable description of a hardware implementation according to claim 16, which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the hardware implementation.

Images