Hierarchical mantissa bit length selection for hardware implementation of deep neural networks
By hierarchically adjusting the fixed-point number format of the DNN layer, the problems of resource limitation and low efficiency in DNN hardware implementation are solved, and more efficient hardware resource utilization and multiple configuration support are achieved.
Patent Information
- Application Number
- CN201811300265.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2017-11-03
- Filing Date
- 2018-11-02
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2039-11-13
AI Technical Summary
Deep neural networks (DNNs) face power consumption, processing power and silicon area limitations in hardware implementation, and the existing fixed-point format is inefficient and cannot effectively support multiple DNN configurations.
The hierarchical method is used to select a fixed-point number format with reduced mantissa length, and the fixed-point number format of each layer is iteratively adjusted until the output error exceeds the threshold, thereby optimizing the hardware resource utilization.
It effectively reduces the silicon area and power consumption realized by hardware, improves processing efficiency, supports multiple DNN configurations, and reduces memory bandwidth requirements.
Smart Images

Figure CN110007959B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to neural networks and, more particularly, to hierarchical mantissa bit length selection for hardware implementation of deep neural networks. Background Art
[0002] A deep neural network (DNN) is a type of artificial neural network that can be used for machine learning applications. In particular, DNNs can be used for signal processing applications, including image processing and computer vision applications.
[0003] DNNs have been implemented in applications where power resources are not a significant factor. Nevertheless, DNNs are also applied in many different technology areas where the hardware resources used to implement DNNs are limited by power consumption, processing power, or silicon area.
[0004] Therefore, there is a need to implement hardware that is configured to implement DNNs in an efficient manner (i.e., in a manner that requires less silicon area or less processing power when operating). In addition, DNNs can be configured in a variety of different ways for a variety of different applications. Therefore, there is also a need for flexible hardware for implementing DNNs that can support a variety of DNN configurations.
[0005] The embodiments described below are provided by way of example only and are not intended to be limiting as to implementations that address any or all of the shortcomings of known hardware implementations of deep neural networks. Summary of the invention
[0006] This Summary is provided to introduce a selection of concepts that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.
[0007] Described herein is a hierarchical method for selecting a fixed-point number format with a reduced mantissa bit length for representing values input to and / or output from a layer of a DNN. The method begins with each layer having an initial fixed-point number format. The layers are divided into subsets of related layers, and the mantissa bit length of the fixed-point number format is iteratively reduced from the initial fixed-point number format on a per-subset basis. If the reduction causes the output error of the DNN to exceed an error threshold, the reduction is discarded and no further reduction is performed on the layers of the subset. Otherwise, further reduction is performed on the subset. Once no further reduction can be performed on any subset, the method is repeated for a continuously increasing number of subsets until each subset reaches a predetermined number of layers.
[0008] A first aspect provides a computer-implemented method for selecting a fixed-point format for representing values input to multiple layers of a deep neural network (DNN) and / or output from multiple layers of the DNN for configuring a hardware implementation of the DNN. The method includes: receiving a DNN instance, the instance being configured to represent the value of the layer using one or more initial fixed-point formats for each layer in a plurality of layers, each initial fixed-point format including exponent and mantissa bit lengths; forming multiple disjoint subsets from the plurality of layers; for each subset in the plurality of subsets, iteratively adjusting the fixed-point format of the layer in the subset to a fixed-point format with the next lowest mantissa bit length until the output error of the DNN instance exceeds an error threshold; in response to determining that the subset includes a number of layers greater than a lower limit threshold, forming a greater number of disjoint subsets from the plurality of layers and repeating the iterative adjustment; and in response to determining that the subset includes a number of layers less than or equal to the lower limit threshold, outputting the fixed-point format for the plurality of layers.
[0009] A second aspect provides a computing-based device for selecting a fixed-point format for representing a value input to a plurality of layers of a deep neural network (DNN) and / or output from a plurality of layers of the DNN for configuring a hardware implementation of the DNN. The computing-based device includes: at least one processor; and a memory coupled to the at least one processor, the memory including: a DNN instance configured to represent the value of the layer using one or more initial fixed-point formats for each layer in the plurality of layers, each initial fixed-point format including an exponent and an initial mantissa bit length; and a computer-readable code, when the computer-readable code is executed by the at least one processor, causing the at least one processor to: form a plurality of disjoint subsets from the plurality of layers; for each subset in the plurality of subsets, iteratively adjust the fixed-point format of the layer in the subset to a fixed-point format with the next lowest mantissa bit length until the output error of the DNN instance exceeds an error threshold; in response to determining that the subset includes a number of layers greater than a lower limit threshold, form a greater number of disjoint subsets from the plurality of layers and repeat the iteration; and in response to determining that the subset includes a number of layers less than or equal to the lower limit threshold, output the fixed-point format for the plurality of layers.
[0010] A third aspect provides a hardware implementation of a deep neural network (DNN), the hardware implementation comprising: hardware logic configured to: receive an input data value for a layer of the DNN; receive information indicating a fixed-point format of the input data value of the layer, wherein the fixed-point format of the input data value of the layer has been selected according to the method of the first aspect; parse the input data value based on the fixed-point format of the input data value of the layer; and process the parsed input data value according to the layer to generate an output data value of the layer.
[0011] A hardware implementation of a DNN may be embodied in hardware on an integrated circuit. A method of manufacturing a hardware implementation in an integrated circuit manufacturing system may be provided. An integrated circuit definition data set may be provided that, when processed in an integrated circuit manufacturing system, configures the system to manufacture a hardware implementation of a DNN. A non-transitory computer-readable storage medium may be provided on which a computer-readable description of a hardware implementation of a DNN is stored that, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the hardware implementation of the DNN.
[0012] An integrated circuit manufacturing system can be provided, comprising: a non-transitory computer-readable storage medium on which a computer-readable description of a hardware implementation of a DNN is stored; a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit embodying the hardware implementation of the DNN; and an integrated circuit generation system configured to manufacture the hardware implementation of the DNN based on the circuit layout description.
[0013] Computer program code for executing the methods described herein may be provided.A non-transitory computer readable storage medium having computer readable instructions stored thereon may be provided, which when executed at a computer system causes the computer system to execute the methods described herein.
[0014] As will be readily appreciated by the skilled person, the features described above may be combined as appropriate, and these features may be combined with any aspects of the examples described herein. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Examples will be described in detail with reference to the accompanying drawings, in which:
[0016] Figure 1 is a schematic diagram of an example deep neural network (DNN);
[0017] Figure 2 is a schematic diagram of example data in DNN;
[0018] Figure 3 is a flow chart of an example method for identifying a fixed-point number format having a shortened mantissa bit length for representing values input to and / or output from a layer of a DNN;
[0019] Figure 4 is a schematic diagram showing values input to and / or output from a layer of a DNN;
[0020] Figure 5 is a schematic diagram of an example DNN and an example sequence of the example DNN;
[0021] Figure 6is a schematic diagram showing partitioning a set of layers into subsets in multiple iterations;
[0022] Figure 7 is a flow chart of an example method for iteratively adjusting a fixed point number format of a layer in a subset to a fixed point number format having a next lowest mantissa bit length until an output error exceeds an error threshold;
[0023] Figure 8 is a schematic diagram showing the calculation of the output error of a tuned instance of a DNN relative to a baseline output;
[0024] Fig. 9 is a schematic diagram showing the calculation of the output error of the adjusted instance of the DNN relative to the known classification result;
[0025] Fig.10 When there is an initial fixed-point format for values of multiple types Figure 3 A flowchart of an example method of block 308;
[0026] Fig.11 It is shown that the use Figure 3 A graph of the weights of the layers of the AlexNet DNN and the length of the mantissa selected for the input data values by the method;
[0027] Fig.12 It is shown that the use Figure 3 A graph of the weights of the layers of the GoogLeNet DNN and the length of the mantissa selected for the input data values by the method;
[0028] Fig.13 is a block diagram of an example hardware implementation of a DNN;
[0029] Fig.14 is a block diagram of an exemplary computing-based device;
[0030] Fig.15 is a block diagram of an example computer system in which a hardware implementation of a DNN is implemented; and
[0031] Fig.16 is a block diagram of an example integrated circuit manufacturing system for generating integrated circuits embodying hardware implementations of the DNNs described herein.
[0032] The accompanying drawings show various examples. It will be appreciated by the skilled person that the element boundaries (e.g., boxes, box groups, or other shapes) shown in the accompanying drawings represent one example of boundaries. In some examples, one element may be designed as multiple elements, or multiple elements may be designed as one element. Common reference numerals may be used appropriately throughout the accompanying drawings to represent similar features. DETAILED DESCRIPTION
[0033] The following description is given by way of example so that those skilled in the art can make and use the invention. The invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be readily apparent to those skilled in the art. The embodiments are described only by way of example.
[0034] A deep neural network (DNN) is a form of artificial neural network that includes multiple interconnected layers that enable the DNN to perform signal processing tasks, including but not limited to computer vision tasks. Figure 1 An example DNN 100 is shown, which includes a plurality of layers 102-1, 102-2, 102-3. Each layer 102-1, 102-2, 102-3 receives input data, processes the input data according to the layer to produce output data. The output data is provided as input data to another layer, or is output as the final output data of the DNN. For example, in Figure 1 In the DNN 100 of FIG. 1 , the first layer 102-1 receives raw input data 104 for the DNN 100 and processes the input data according to the first layer 102-1 to generate output data. The output data of the first layer 102-1 becomes the input data of the second layer 102-2, and the second layer 102-2 processes the input data according to the second layer 102-2 to generate output data. The output data of the second layer 102-2 becomes the input data of the third layer 102-3, and the third layer 102-3 processes the input data according to the third layer 102-3 to generate output data. The output data of the third layer 102-3 is output as the output data 106 of the DNN.
[0035] The processing performed on the input data of a layer depends on the type of layer. For example, each layer of a DNN can be one of a variety of different types. Example DNN layer types include, but are not limited to: convolutional layers, activation layers, normalization layers, pooling layers, and fully connected layers. It is readily understood by those skilled in the art that these are exemplary DNN layer types, not an exhaustive list, and that other DNN layer types may exist.
[0036] For a convolutional layer, the input data is processed by convolving it with the weights associated with the layer. Specifically, each convolutional layer is associated with a number of weights w0…w g These weights may also be referred to as filter weights or coefficients. These weights may be grouped to form or define one or more filters, which may be referred to as kernels, and each filter may be associated with a compensation bias b.
[0037] refer to Figure 2 , which shows an example overview of the format of the data used in DNN. Figure 2As shown, the data used in the DNN can be arranged into P data planes, where each plane has dimensions x xy. The DNN may include one or more convolutional layers, each convolutional layer being associated with a plurality of filters formed by a plurality of weights. Each filter has dimensions m x n x P, and each filter is based on Figure 2 A convolution operation of several steps in directions s and t is shown being applied to the input data. The number of filters and the number of weights per filter can vary between convolutional layers. A convolutional neural network (CNN) is a specific type of DNN effective for image recognition and classification, which typically includes multiple convolutional layers.
[0038] The activation layer usually follows the convolutional layer, but this is not a requirement. The activation layer performs one or more activation functions on the input data of this layer. The activation function takes a single number and performs a nonlinear mathematical operation on it. In some examples, the activation layer can be used as a ReLU by implementing a Rectified Linear Unit (ReLU) function (i.e., f(x) = max(0, x)) or as a PReLU by implementing a Parametric Rectified Linear Unit (PReLU) function.
[0039] The normalization layer is configured to perform a normalization function, such as a local response normalization (LRN) function on the input data. Pooling layers are usually inserted between consecutive convolutional layers, but this is not a requirement. Pooling layers perform a pooling function (such as a maximum function or an average function) to aggregate subsets of the input data. Therefore, the purpose of the pooling layer is to reduce the spatial size of the representation and thus reduce the number of parameters and calculations in the network, thereby also controlling overfitting.
[0040] A fully connected layer is usually followed by multiple convolutional and pooling layers, but this is not a requirement. A fully connected layer takes a three-dimensional set of input data values and outputs an N-dimensional vector. In the case of DNNs for classification, N is the number of categories, and each value in the vector represents the probability of a certain category. This N-dimensional vector is produced by matrix multiplication of the input data values by a set of weights, followed by a bias compensation. Therefore, a fully connected layer receives a bias and a set of weights.
[0041] Thus, each layer of a DNN receives input data values and produces output data values; and some layers also receive weights and biases.
[0042] The hardware implementation of the DNN includes hardware logic that is configured to process the input data of the DNN according to the layers of the DNN. Specifically, the hardware implementation of the DNN includes hardware logic that is configured to process the input data of the layer according to each layer, and generate output data for the layer, which either becomes the input data of another layer or becomes the output of the DNN. For example, if the DNN includes a convolution layer followed by an activation layer, the hardware implementation of the DNN includes hardware logic that is configured to perform a convolution operation on the input data of the DNN using weights and offsets associated with the convolution layer to generate the output data of the convolution layer, and the hardware logic is configured to apply an activation function to the input data of the activation layer (i.e., the output data of the convolution layer) to generate the output data of the DNN.
[0043] As known to those skilled in the art, for hardware that processes a set of values, each value is represented in a digital format. The two most suitable digital formats are fixed-point format and floating-point format. As known to those skilled in the art, the fixed-point format has a fixed number of digits after the decimal point (e.g., a decimal point or a binary point). In contrast, the floating-point format does not have a fixed decimal point (i.e., it can "float"). In other words, the decimal point can be placed anywhere in the representation. Although representing the values input to the layers of the DNN and / or output from the layers of the DNN in floating-point format can allow more accurate or precise output data to be generated, processing numbers in floating-point format in hardware is complex, which increases silicon area and hardware complexity compared to hardware that processes values in fixed-point format. Therefore, the hardware implementation of the DNN can be configured to represent the values input to the layers of the DNN and / or output from the layers of the DNN in fixed-point format to reduce the area, power consumption, and memory bandwidth of the hardware implementation.
[0044] A common fixed-point number format is the Q format, which specifies a predetermined number of integer digits a and fractional digits b. Thus, a number can be represented as Qa.b, which requires a+b+1 digits in total (including the sign bit). An example Q format is shown in Table 1 below.
[0045] Table 1
[0046] Q format describe Example Q4.4 4 integer digits and 4 decimal digits <![CDATA[0110.11102]]> Q0.8 0 integer digits and 8 decimal digits <![CDATA[.011011102]]>
[0047] However, a disadvantage of the Q format is that some of the bits used to represent the number may be considered redundant. In the example, the number range [-0.125, 0.125] will be represented with 3 bits of precision. The Q format required for this example range and precision is Q0.5. However, if the range of values is assumed to be known in advance, the first two bits of the number will never be used to determine the value represented in the Q format. For example, the first two bits of the representation do not contribute to the final number because they represent 0.5 and 0.25, respectively, and therefore exceed the required range. However, they are used to indicate the value of the third bit position (i.e., 0.125 and greater due to the relative bit position). Therefore, the above Q format is an inefficient fixed-point number format for hardware implementations of DNNs because some bits may not convey useful information.
[0048] Therefore, in some cases, instead of using the Q format, some hardware implementations may be configured to use the following fixed-point format for the values of the DNN layer inputs and / or outputs, where each value x consists of a fixed integer exponent e and an n-bit mantissa m format x=2 e m, which is defined by the exponent e and the number of mantissa bits n ({e,n}). In some cases, the mantissa m can be represented in two's complement format, while in other cases other signed or unsigned integer formats can be used.
[0049] In order to reduce the size and improve the efficiency of the hardware implementation of the DNN, the hardware implementation can be configured to represent the values input to the layers of the DNN and / or output from the layers of the DNN in a fixed-point format that uses the minimum number of bits that can represent the expected or desired range of values. Since the range of different sets of values (e.g., input data values, output data values, biases, and weights) may vary within and between layers, the hardware implementation can process the DNN more efficiently when the fixed-point format used to represent the input data values, output data values, weights, and / or biases can vary within and between layers. For example, the hardware implementation can more efficiently implement the DNN by using the following fixed-point format: a fixed-point format including an exponent of 2 and a mantissa bit length of 6 to represent the input data values of the first layer; a fixed-point format including an exponent of 3 and a mantissa bit length of 12 to represent the weights of the first layer; and a fixed-point format including an exponent of 4 and a mantissa bit length of 6 to represent the input data values of the second layer.
[0050] Therefore, it is desirable to identify, on a per-layer basis, a fixed-point number format (e.g., including exponent and mantissa bit lengths) used to represent values input to or output from a layer of a DNN (e.g., input data values, output data values, weights, and / or biases). Some methods for identifying the fixed-point number format of a layer of a DNN have proven effective in identifying the exponent of each layer given the mantissa bit length, such as those described in the applicant's co-pending UK patent applications entitled "HISTOGRAM-BASED DATA FORMAT SELECTION FOR HARDWARE IMPLEMENTATION OF DEEP NEURAL NETWORK" and "END-TO-END DATA FORMAT SELECTION FOR HARDWARE IMPLEMENTATION OF DEEP NEURAL NETWORK," filed on the same day as the present application, which are incorporated herein by reference. However, reducing the mantissa bit length used to represent values in DNNs can significantly reduce the power consumption, memory size, and memory bandwidth of hardware implementations of DNNs.
[0051] Therefore, described herein is a hierarchical method for selecting a fixed-point number format with a reduced mantissa bit length for representing values input to and / or output from a layer of a DNN. The method described herein starts with one or more fixed-point number formats (including exponent and mantissa bit lengths) for each layer of the DNN, and reduces the mantissa bit lengths of those fixed-point number formats in a hierarchical manner given an acceptable output error. Specifically, in the method described herein, the layers are divided into disjoint subsets, and the mantissa bit lengths of the fixed-point number formats are iteratively reduced from the initial fixed-point number format on a per-subset basis. If the reduction causes the output error of the DNN to exceed an error threshold, the reduction is discarded and no further reduction is performed on the subset. However, if the reduction does not cause the output error of the DNN to exceed the error threshold, the layers in the subset are further reduced, and so on. Once no further reduction can be performed on any subset, the method is repeated for smaller subsets of increasing number until each subset reaches a predetermined number of layers (e.g., 1).
[0052] Since the described methods take advantage of the fact that related layers may have similar mantissa bit lengths (and therefore the mantissa bit lengths of the fixed-point formats of the related layers may be reduced in a similar manner), the methods are efficient in terms of time and computational resources. In particular, it has been demonstrated that the described methods produce results in a shorter time than methods that attempt to reduce the mantissa bit length in a naive layer-by-layer manner. Furthermore, since the evaluation of the impact of changes to one or more fixed-point formats is made based on the output of the DNN, the impact of the change(s) on the overall DNN behavior is taken into account, providing a realistic measure of the impact.
[0053] Reference now Figure 3 , which shows an example method 300 for selecting a fixed-point number format with a reduced mantissa bit length for representing values input to multiple layers of a DNN and / or output from multiple layers of a DNN, which can be used to configure a hardware implementation of a DNN. The multiple layers of the DNN may include all layers of the DNN or only a portion of the layers of the DNN. The method 300 can be implemented by a computing-based device, such as the following reference Fig.14 For example, a computing-based device (e.g., computing-based device 1400) may include or have access to a computer-readable medium (e.g., memory) having computer-readable instructions stored thereon, which, when executed by a processor of the computing-based device, causes the computing-based device to perform Figure 3 Method 300.
[0054] As mentioned above, each layer of a DNN can have multiple value sets, which can be represented in fixed-point format. Figure 4 As shown, each layer 402 receives input data values and generates output data values; and some layers (such as convolutional layers and fully connected layers) may also receive weights and biases. Each of the input data values, output data values, weights, and biases may be represented by a different fixed-point number format. Method 300 may be used to select a fixed-point number format for any combination of these value sets. For example, method 300 may be used to select a fixed-point number format for representing the weights of each layer; or, method 300 may be used to select a fixed-point number format for representing the weights of each layer and select a fixed-point number format for representing the input data values of each layer. Therefore, the values input to each layer and / or output from each layer may include one or more of the input data values, output data values, weights, and biases.
[0055] The method 300 begins at block 302, where a DNN instance is received, the instance being configured to represent values input to each of the multiple layers and / or output from each of the multiple layers using one or more initial or starting fixed-point number formats for each of the multiple layers. Such an instance of the DNN may be referred to as an initialized instance of the DNN. The DNN instance embodies the DNN in a form that can be used to test the response of the DNN to input data. The DNN instance includes, but is not limited to, a software model of the DNN or a hardware implementation of the DNN.
[0056] Each layer can have one or more initial fixed-point number formats. For example, in some cases, each layer can have one initial fixed-point number format for input data values and a different initial fixed-point number format for output data values; and other layers (such as convolutional layers and fully connected layers) can have separate initial fixed-point number formats for weights and biases. In other cases, some or all layers may have only one initial fixed-point number format - for example, a fixed-point number format for weights.
[0057] Each initial fixed-point number format includes, or is defined by, an exponent e and a mantissa bit length n. The initial or starting fixed-point number format of a layer may have been determined or selected by another method, such as those described in the applicant's co-pending UK patent applications entitled "HISTOGRAM-BASED DATA FORMAT SELECTION FOR HARDWARE IMPLEMENTATION OF DEEP NEURAL NETWORK" and "END-TO-END DATA FORMAT SELECTION FOR HARDWARE IMPLEMENTATION OF DEEP NEURAL NETWORK". In some cases, the initial fixed-point number formats of the layers include the same mantissa bit length but may have different exponents. For example, the initial fixed-point number format of one layer may include an exponent of 6 and a mantissa bit length of 12, and the initial fixed-point number format of another layer may include an exponent of 3 and a mantissa bit length of 12.
[0058] Initializing or configuring a DNN instance to use a specific fixed-point floating-point format to represent a set of values input to or output from a layer depends on the type of the value. For example, if a fixed-point format is used for values input to a layer, such as input data values, weights, or biases of a layer, initializing or configuring a DNN instance to use a specific fixed-point format to represent these values of the layer may include: converting these values to a specific fixed-point format and configuring the layer to parse the set of values according to the fixed-point format; or, configuring the layer to convert the set of values to a specific fixed-point format. However, if a fixed-point format is used for values output from a layer (such as output data values), initializing or configuring a DNN instance to use a specific fixed-point format to represent these values of the layer may include: configuring the layer to convert the output data values it generates to a fixed-point format.
[0059] Once the initialized instance of the DNN has been received, the method 300 proceeds to block 304 .
[0060] At block 304, multiple disjoint subsets are formed from multiple layers of the DNN. The term "subset" is used to mean a portion of something, but not all of it. A subset may include one element or multiple elements. Thus, a subset of layers of the DNN may include one or more layers of the DNN, but not all of the layers of the DNN. The term "disjoint" is used herein to mean non-overlapping, such that each layer cannot belong to more than one subset.
[0061] In some cases, forming multiple disjoint subsets from multiple layers may include determining a sequence of multiple layers and dividing the sequence of layers into multiple consecutive blocks of layers, where related layers in the multiple layers are very close to each other. For example, a sequence of multiple layers of a DNN may be determined, where in the sequence, each layer is preceded by the layer(s) on which it depends. As described above, the DNN includes multiple interconnected layers, where the output data of each layer (except the last layer) becomes the input data of the subsequent layer. A layer that receives input data from an earlier layer is considered to be dependent on the earlier layer.
[0062] For example, Figure 5An example DNN 500 including eight layers (L1, L2, L3, L4, L5, L6, L7, and L8) is shown. In this example, the output data of layer 1 (L1) is provided as input data to layers 2 and 3 (L2, L3), so layers 2 and 3 are said to be dependent on layer 1; and the output data of layer 3 (L3) is provided as input data to layer 6 (L6), so layer 6 (L6) is dependent on layer 3 (L3). Therefore, the following arbitrary sequence of layers will satisfy the sequence condition that the layer (or layers) on which it depends is preceded by each layer in the sequence: L1 precedes L2 and L3; L2 precedes L4; L4 precedes L5; L3 precedes L6; L5 and L6 precede L7; L7 precedes L8. Examples of such sequences include the sequence {L1, L2, L4, L5, L3, L6, L7, L8} and the sequence {L1, L3, L6, L2, L4, L5, L7, L8}.
[0063] Tests show that related layers are likely to have similar optimal mantissa bit lengths. By ordering the layers as described in this article, when the layers are divided into subsets based on the sequence, the subsets are more likely to contain related layers. Once the layer sequence of the DNN is determined, the layer sequence can be divided into subsets of consecutive layers.
[0064] If the layers in a subset are consecutive layers in a sequence, the subset is said to include a set of consecutive layers. Figure 5 In the sequence {L1, L2, L4, L5, L3, L6, L7, L8}, the subset including L1, L2 and L4 will be a subset of consecutive layers because there are no other layers between L1, L2 and L4 in the sequence. In contrast, the subset including L2, L5 and L7 is not a subset of consecutive layers because there are layers between L2 and L5 and between L5 and L7 in the sequence that are not in the subset.
[0065] In some cases, a sequence of DNN layers is divided into blocks or subsets of layers with an initial subset size G, where G is an integer greater than or equal to 2. The initial subset size G can be an input parameter of method 300 and can therefore vary between implementations of the method. This allows the initial subset size G to vary between different DNNs and for different types of values. For example, different initial subset sizes G can be used when method 300 is used to select a fixed-point format for the weights of a layer and when selecting a fixed-point format for the input data values of a layer. The initial subset size G can be selected experimentally (e.g., selected from a power of 2 starting from 2, depending on the size of the DNN). For example, Figure 6 As shown, when the initial subset size G is four, Figure 5The example DNN 500 of can be divided into two subsets 602, 604, each having four layers. The first subset 602 includes consecutive layers L1, L2, L4, L5, and the second subset 604 includes consecutive layers L3, L6, L7, L8.
[0066] Although the above paragraphs describe how to form multiple disjoint subsets from multiple layers by sorting the layers and dividing the layer sequence into multiple continuous subsets, it is easy for a person skilled in the art to understand that this is only an example, and multiple subsets can be formed in any suitable manner. For example, in other cases, layers with similar properties (e.g., convolutional layers with the same filter or kernel size) can be placed in the same subset; layers from the same local structure (e.g., the initial module in GoogLeNet or a single branch in a branch network) can be placed in the same subset; or the layers can be randomly divided into subsets.
[0067] In some cases, the first layer and / or the last layer of the DNN are not included in the multiple layers and therefore are not included in any subset. This is because the mantissa bit length of the first and last layers generally has a large impact on the overall accuracy of the DNN. Therefore, in some cases, the initialization instance of the DNN is configured to use a fixed-point number format with a high mantissa bit length (e.g., 12 bits or longer) for the first and / or last layers of the DNN, and the method 300 does not adjust the mantissa bit length of the fixed-point number format associated with the first and / or last layers.
[0068] Once a plurality of disjoint subsets are formed from the plurality of layers, the method 300 proceeds to block 306 .
[0069] At block 306, one of the subsets of block 304 is identified as the current subset for processing. In some cases, if a sequence is determined, the subsets are processed in an order corresponding to the order of the layers in the sequence. Thus, in these cases, the subset comprising the initial layer in the sequence is processed first, then the subset comprising the next layer in the sequence is processed, and so on, until the end of the sequence is reached. For example, referring to Figure 6 , the first subset 602 may be processed first, and then the second subset 604 may be processed. In other cases, the subset with the largest total number of mantissa bits in the fixed-point number format is selected to be processed first, and the subset with the second largest total number of mantissa bits in the fixed-point number format is selected to be processed next, and so on. However, it will be readily appreciated by those skilled in the art that these are merely examples, and the subsets may be processed in another order. Once one of the subsets is identified as the current subset for processing, the method 300 proceeds to block 308.
[0070] At block 308, an attempt is made to reduce the mantissa bit length of the fixed point number format for the layers in the current subset.
[0071] Specifically, the fixed-point number format used by the DNN instance for one or more value sets of the layers in the current subset is iteratively adjusted to a fixed-point number format with the next lowest mantissa bit length until the output error of the DNN instance exceeds an error threshold (ETh).
[0072] During each iteration, a fixed-point number format having a next lowest mantissa bit length compared to the current fixed-point number format is determined for each layer in the subset. For example, if the subset includes four layers, a fixed-point number format having a next lowest mantissa bit length is determined for each of the four layers. Determining a fixed-point number format having a next lowest mantissa bit length relative to a particular fixed-point number format may include (i) selecting a next lowest mantissa bit length (compared to the mantissa bit length of the current fixed-point number format) supported by a hardware implementation of the DNN to be configured; and (ii) adjusting an exponent of the current fixed-point number format to account for a reduction in the number of mantissa bits. Adjusting the exponent of the current fixed-point number format to account for a reduction in the number of mantissa bits may include increasing the exponent by the same number of bits removed from the mantissa so that the fixed-point number format covers the same range of values. For example, if the mantissa bits are reduced by 4 bits, the exponent may be increased by 4.
[0073] In particular, some hardware implementations may only support certain sets of mantissa bit lengths. For example, some hardware implementations of DNNs may only support mantissa bit lengths of 4, 5, 6, 7, 8, 10, 12, and 16. It will be readily appreciated by those skilled in the art that this is merely an example, and other hardware implementations of DNNs may support different sets of mantissa bit lengths and / or different numbers of mantissa bit lengths.
[0074] Table 2 shows fixed-point number formats with the next lowest mantissa bit length, for example, fixed-point number formats with mantissa bit lengths of 4, 5, 6, 7, 8, 10, 12, and 16 supported by the hardware implementation. The first example fixed-point number format includes an exponent of 2 and a mantissa bit length of 8, and the next lowest mantissa bit length supported is 7. This reduces the mantissa bit length by 1 bit, thereby increasing the exponent by 1 to 3. The second example fixed-point number format includes an exponent of 5 and a mantissa bit length of 12, and the next lowest mantissa bit length supported is 10. This reduces the mantissa bit length by 2 bits, thereby increasing the exponent by 2 to 7. The third example fixed-point number format includes an exponent of 1 and a mantissa bit length of 16, and the next lowest mantissa bit length supported is 12. This reduces the mantissa bit length by 4 bits, thereby increasing the exponent by 4 to 5.
[0075] Table 2
[0076]
[0077] Once the fixed-point format with the next lowest mantissa bit length is determined for each layer in the subset, the DNN instance is adjusted to use the determined fixed-point format to represent the values of the layers of the subset. The test input data is then applied to the adjusted DNN instance, and the output of the adjusted DNN instance in response to the test input data is recorded. The output error of the adjusted DNN instance is then determined based on the recorded output data. If the output error does not exceed the error threshold (ETh), the adjustment is maintained and another iteration is performed. However, if the output error exceeds the error threshold (ETh), the adjustment can be discarded (for example, the DNN instance can be adjusted to use the previous fixed-point format for the layers of the subset). By measuring the output of the DNN, the impact of the fixed-point format adjustment on the behavior of the entire DNN is considered.
[0078] Reference below Figure 7 An example method for iteratively adjusting the fixed-point format for the layers in the subset to a fixed-point format with the next lowest mantissa bit length until the output error of the DNN instance exceeds an error threshold is described in detail. Figure 7 Describes an example of how the output error of a DNN instance can be calculated.
[0079] Once the error threshold is reached, the method 300 proceeds to block 310 .
[0080] At block 310, it is determined whether there are more subsets for which an attempt has not been made to reduce the mantissa bit length of the fixed-point number format of the layers in the subset. If there is at least one subset for which an attempt has not been made to reduce the mantissa bit length of the fixed-point number format of the layers in the subset, the method 300 proceeds to block 312. However, if there are no subsets for which an attempt has not been made to reduce the mantissa bit length of the fixed-point number format of the layers in the subset, indicating that an attempt has been made to reduce the mantissa bit length of the fixed-point number format of each layer, the method 300 proceeds to block 314.
[0081] At block 312, the next subset for processing is identified as the current subset. The same method used to select the first subset for processing in block 306 may be used. For example, as described above, in some cases, if a sequence of layers is determined, the subsets may be processed in an order corresponding to the order of the layers in the sequence. In other cases, the subset with the largest total number of mantissa bits in the fixed point format is selected for processing first, followed by the subset with the second largest total number of mantissa bits in the fixed point format, and so on. Once another subset is selected for processing, the method 300 returns to block 308, where an attempt is made to reduce the length of the mantissa bits in the fixed point format of the layers of the new current subset.
[0082] At box 314, it is determined whether the subset includes a number of layers less than or equal to a lower threshold (LTh). The number of layers in the subset may also be referred to herein as the size of the subset. In some cases, the lower threshold (LTh) is set to 1 to ultimately attempt to reduce the mantissa bit length of the fixed-point format of each layer individually. However, in order to reduce the time to complete the method (i.e., reduce the time to identify the fixed-point format of the input value set of the layer of the DNN), the lower threshold (LTh) may be set to a value greater than 1. For example, in some cases, the lower threshold (LTh) may be set to 2. If the subset includes a number of layers less than or equal to the lower threshold (LTh), the method proceeds to box 318. However, if the subset includes a number of layers greater than the lower threshold (LTh), the method 300 proceeds to box 316.
[0083] At block 316, a greater number of disjoint subsets are formed from the plurality of layers. Figure 6 In the example of , in the first iteration, the layers are divided into two subsets 602, 604, each of which has four layers. In this example, in the second iteration, the layers are divided into four subsets 606, 608, 610, 612, each of which includes two layers. The first subset 606 of the second iteration includes L1 and L2, the second subset 608 of the second iteration includes L4 and L5, the third subset 610 of the second iteration includes L3 and L6, and the fourth subset 612 of the second iteration includes L7 and L8. In some cases, each iteration can increase the number of subsets by a predetermined factor (e.g., 2). Similarly, in the third iteration, the layers are divided into eight subsets 614, 616, 618, 620, 622, 624, 626, 628, each of which includes only a single layer.
[0084] In some cases, a greater number of disjoint subsets may be formed by partitioning the layers of each current subset into a plurality of subsets. In other cases, a greater number of disjoint subsets may be formed by generating new subsets from the original set of all layers. In the case of determining a sequence of layers, the layers may be partitioned into a greater number of multiple consecutive subsets (based on the sequence). Once a greater number of subsets are formed from the plurality of layers, the method 300 returns to block 306, where, on a per-subset basis, an attempt is made to reduce the number of mantissa bits of the fixed-point number format of the layers of each subset.
[0085] At block 318, a fixed-point format of the plurality of layers in the DNN instance is output. In some cases, the fixed-point format may be output to a memory.
[0086] Despite Figure 3In the method 300, the fixed-point format of the layers for one subset is iteratively adjusted until the error threshold is met, and then the fixed-point format of the layers for the next subset is iteratively adjusted until the error threshold is met, but in other examples, the adjustment of the fixed-point format of different subsets can be performed in an interleaved manner. Specifically, in other examples, the fixed-point format of the layers of all subsets can be adjusted once, and then the fixed-point format of the layers of any subset can be adjusted a second time. For example, if there are three subsets, the fixed-point format of the layers of the first subset can be adjusted once, and then the fixed-point format of the layers of the second subset can be adjusted once, and then the fixed-point format of the layers of the third subset can be adjusted once, and then the fixed-point format of the layers of the second subset can be adjusted a second time, and so on. If at any time the adjustment of the fixed-point format of the layers of the subset causes the output error of the DNN instance to exceed the error threshold (ETh), the adjustment is discarded (i.e., the previous fixed-point format of the layers of the subset is adjusted to be used by the DNN instance), and no further adjustment is made to the fixed-point format of the layers of the subset. However, the fixed-point number formats of the layers of the other subsets may be further adjusted until such adjustment causes the output error of the DNN instance to exceed an error threshold (ETh).
[0087] Therefore, in the example of three subsets, if the first adjustment of the fixed-point format of the layers of the first subset causes the output error of the DNN instance to exceed the error threshold (ETh), but the first adjustment of the fixed-point format of the layers of the second and third subsets does not cause the output error to exceed the error threshold, the fixed-point format of the layers of the second and third subsets can be adjusted a second time. Compared with the method of making all adjustments to the fixed-point format of one subset before making any adjustments to the fixed-point format of other subsets, using this interleaved adjustment method can distribute errors more evenly among subsets.
[0088] Once based on Figure 3If the method 300 of determines a fixed-point number format with a reduced mantissa bit length for representing a value input to a layer of a DNN and / or output from a layer of the DNN, one or more identified fixed-point number formats may be used to configure the hardware implementation of the DNN. For example, one or more of the identified fixed-point number formats may be used to configure the DNN to parse input data values, weights, and / or biases in the identified (one or more) formats. This may allow the hardware implementation to more efficiently process the input data, weights, and / or biases of the layer. In another example, the fixed-point number format of the identified layer may be used to configure the DNN to convert the received input data values, weights, and / or biases to the identified fixed-point number format when it does not receive the input data values, weights, and / or biases in the identified (one or more) fixed-point number formats for the layer, so as to allow the hardware implementation to more efficiently process the input data values, weights, or biases of the layer. In yet another example, the identified fixed-point number format can be used to configure a hardware implementation to convert output data of one layer fed to another layer into the identified fixed-point number format so that the output data will be provided to the other layer in the identified fixed-point number format. Fig.13 Describes example hardware implementations of DNNs and how the hardware implementations can be configured using the identified formats.
[0089] Reference now Figure 7 , which shows an example method 700 for iteratively adjusting the fixed-point format of the layers in the subset to the fixed-point format with the next lowest mantissa bit length until the output error exceeds the error threshold. In the method 700, the fixed-point format with the next lowest mantissa bit length is determined for each layer in the subset; the DNN instance is adjusted to use the determined fixed-point format for the layers of the subset; the test input data is applied to the adjusted DNN instance, and the output data of the DNN instance in response to the test input data is recorded; and the output error of the adjusted DNN instance is determined. If the output error does not exceed the error threshold (ETh), the method is repeated; however, if the output error exceeds the error threshold (ETh), the adjustment is discarded and the method ends. Figure 7 Method 700 is an example implementation of block 308 of method 300 .
[0090] Method 700 begins at box 702, where (one or more) fixed-point number formats having a next lowest mantissa bit length compared to the current (one or more) fixed-point number formats are determined for each layer in the subset, and the DNN instance is adjusted to use the fixed-point number format with the next lowest mantissa bit length to represent one or more sets of values for each layer.
[0091] As described above, determining a fixed-point format with a next lowest mantissa bit length relative to a particular fixed-point format may include: (i) selecting a next lowest mantissa bit length (compared to the mantissa bit length of the current fixed-point format) supported by the hardware implementation of the DNN to be configured; and (ii) adjusting the exponent of the current fixed-point format to account for the reduction in the number of mantissa bits. Adjusting the exponent of the current fixed-point format to account for the reduction in the number of mantissa bits may include: increasing the exponent by the same number of bits removed from the mantissa so that the new fixed-point format covers the same range of values. For example, if the number of mantissa bits is reduced by 4, the exponent may be increased by 4.
[0092] Once the fixed-point number format(s) with the next lowest mantissa bit length is determined for each layer of the subset, the DNN instance is adjusted to represent the associated value sets of the layers in the subset according to the fixed-point number format with the next lowest mantissa bit length. Once the DNN instance is adjusted to represent the associated value sets of the layers of the subset according to the fixed-point number format with the next lowest mantissa bit length, the method 700 proceeds to block 704.
[0093] At box 704, the test input data is provided to the adjusted DNN instance, and the output of the adjusted DNN instance in response to the test input data is recorded. In the case where the DNN is a classification network, the output of the DNN instance can be a set of logits. As known to those skilled in the art, the classification network determines the probability that the input data falls into each of a plurality of categories. The network typically determines a data vector having one element corresponding to each category, and each of these elements is called a logit. For example, a classification network with 1425 potential class labels can output a vector of 1425 logits. In some cases, instead, the output of the DNN instance can be the output of a SoftMax function applied to the logits. As known to those skilled in the art, the SoftMax function is a transformation of the logits output by the DNN so that the values associated with each classification add up to 1. This allows the output of the SoftMax function to represent a probability distribution over each category. The output of the SoftMax function can be called the SoftMax normalized logit. The SoftMax function can be expressed as shown in equation (1), where s i is the softmax output of category i, z i is the number of pairs of class i, i and j are the vector indices corresponding to the classes:
[0094]
[0095] In some cases, multiple sets of test input data may be provided to the adjusted DNN instance. For example, in some cases, the input data to the expected DNN is an image, and 10-50 test images may be provided to the adjusted DNN instance. In these cases, the output data may include output data (e.g., logit counts) generated in response to each test input (e.g., test image). Once the output data of the adjusted DNN instance in response to the test input data has been recorded, the method 700 proceeds to block 706.
[0096] At block 706, an error in the output of the adjusted DNN instance is determined, which may be referred to herein as an output error associated with the adjusted DNN instance. The error quantitatively represents the quality of the output.
[0097] In some cases, such as Figure 8 As shown, the error of the output of the adjusted DNN instance can be calculated as the error of the output relative to the baseline output (800). In some cases, the baseline output can be the output of a DNN instance configured to represent the values input to each layer and output from it in floating point format. Such an instance may be referred to herein as a floating point instance of the DNN. As described above, since values can generally be more accurately or more precisely represented in floating point format, a DNN instance configured to represent the values input to each layer and output from it using a floating point format represents a DNN instance that will produce the best or most accurate output. Therefore, the output data generated by the floating point instance of the DNN can be used as a benchmark or baseline output, wherein the accuracy of the output data generated by the adjusted DNN instance is measured based on the benchmark or baseline output.
[0098] In other cases, the baseline output may be the output of a DNN instance configured to represent the values input to and output from each layer according to the initial fixed-point number format of each layer. This may be referred to as the starting instance of the DNN. Any change to the mantissa bit length according to the described method will result in a change in the accuracy of the output relative to the starting instance of the DNN. Thus, the output data generated by the starting instance of the DNN may be used as a benchmark or baseline output, wherein the accuracy of the output data generated by the adjusted DNN instance is measured based on the benchmark or baseline output.
[0099] The error between the baseline output of the DNN and the output of the adjusted DNN instance can be determined in any suitable manner. For example, where the output is a set of logits, the error between the baseline output and the output of the adjusted DNN instance can be calculated as the L1 distance between the corresponding logits. This is illustrated in formula (2), where z is the set of logits in the baseline output and z' is the set of logits in the output of the adjusted DNN instance:
[0100] ∑i |z i -z′ i | (2)
[0101] In other examples, the error between the baseline output and the output of the tuned DNN instance can be calculated as the L1 distance between the outputs of the SoftMax function, with or without an additional temperature parameter T as shown in Equation (3). Increasing the temperature makes the SoftMax values "softer" (i.e., less saturated to 0 and 1), making them easier to train.
[0102]
[0103] In other cases, such as Fig. 9 As shown, the error of the output can be calculated based on the ground truth accuracy of the output (900). In other words, the error can indicate the accuracy of the output of the adjusted DNN instance relative to the known results for the test input data. For example, based on the known correct classification or label for the test input data, the output error can be the top 1 (Top-1) classification accuracy or the top 5 (Top-5) classification accuracy. As known to those skilled in the art, the top 1 classification accuracy is a measure of the frequency that the highest classification of the DNN output is the correct classification, and the top 5 classification accuracy is a measure of the frequency that the correct classification is the top five classifications of the DNN output.
[0104] In other cases, the output error can be calculated as Figure 8 and Fig. 9 For example, the output error can be calculated as a weighted sum of the error for the correct classification label and the error for the baseline output (which can be a floating point instance of the DNN (i.e., a DNN instance configured to represent the input and output of each layer in floating point format) or a starting instance of the DNN (i.e., a DNN instance configured to use the initial fixed point format of the layer)).
[0105] Once the error of the output of the adjusted DNN instance is determined, method 700 proceeds to box 708.
[0106] At block 708, it is determined whether the output error exceeds an error threshold (ETh). The error threshold may represent an acceptable level of output error for the DNN. Determining whether the error threshold (ETh) has been exceeded may be based on how the output error is determined. For example, where the output error is calculated as the difference between the output of the adjusted DNN instance and the baseline output, the output error may exceed the error threshold when the output error is greater than the error threshold. Conversely, where the output error is a measure of the accuracy of the DNN, the output error may exceed the error threshold when the output error is below the error threshold (e.g., the top 1 or top 5 accuracy is below the acceptable accuracy).
[0107] If it is determined that the output error determined in block 706 does not exceed the error threshold (ETh) (i.e., it is within an acceptable output error level), the adjustment is accepted and the method 700 may proceed to block 710 or may return directly to block 702, where further adjustments to the fixed-point format of the layers of the subset are attempted. Specifically, in some cases, the fixed-point format of the layers of the subset is adjusted until the output error of the adjusted DNN instance exceeds the error threshold. In these cases, after determining that the output error does not exceed the error threshold, the method 700 may proceed directly to block 702. However, in other cases, in order to reduce the time required to complete the method, there may be a maximum number of iterations / adjustments for each subset, and if the maximum number of adjustments / iterations is reached before the error threshold is exceeded, no further adjustments to the fixed-point format of the layers of the subset are made. In these cases, after determining that the output error does not exceed the error threshold, the method 700 may proceed to block 710.
[0108] At block 710, it is determined whether the maximum number of iterations of method 700 has been executed. If it is determined that the maximum number of iterations has been executed, then method 700 ends. However, if it is determined that the maximum number of iterations of method 700 has not been executed, then method 700 returns to block 702, where further adjustment of the fixed-point number format of the layers of the subset is attempted.
[0109] At block 712, after determining that the output error exceeds the error threshold, the adjustment made in block 702 is rejected. Rejecting the most recent adjustment may include re-adjusting the DNN instance to use a fixed-point format for a subset of layers used by the DNN instance before the adjustment. Once the adjustment is rejected, the method 700 ends.
[0110] As mentioned above, each layer can have multiple value sets, each of which can be represented in a different fixed-point format. Figure 3 In the case where the method 300 adjusts the fixed-point format of at least two different types of values (e.g., the fixed-point format of the input data value and the fixed-point format of the weight), the block 308 may be executed once each time it occurs. Figure 7Method 700 of , wherein the fixed-point format for all different types of values is adjusted simultaneously (e.g., the fixed-point format of the input data values and the fixed-point format of the weights of the subset of layers can be adjusted simultaneously); or the fixed-point format for different values can be performed separately based on each occurrence of block 308 Figure 7 For example, method 700 may be performed once to adjust the fixed-point format of the input data value, and method 700 may be performed a second time to adjust the fixed-point format of the weight.
[0111] In the case where method 700 is performed separately for fixed-point formats of different types of values, the order in which the fixed-point formats of different types of values are reduced may affect the accuracy with which the effect of adjusting the fixed-point format on the output error can be determined. Specifically, input data values, weights, and biases are inputs to the layer, so the fixed-point formats used for these types of values affect the output data values. Therefore, if the fixed-point formats of the input data values, weights, and biases are reduced before reducing the fixed-point formats of the output data values, a more accurate estimate of the effect of adjusting the fixed-point formats of the output data values on the output error of the DNN can be determined. Specifically, this allows the reduced fixed-point formats of the input values to be considered when reducing the fixed-point formats of the output data values. Testing also shows that the output error can be reduced if the fixed-point formats of the input data values are reduced before reducing the fixed-point formats of the weights and biases.
[0112] Reference now Fig.10 , which shows the method for implementing Figure 3 The method 1000 of block 308 of method 300 of claim 1000, wherein the fixed-point number formats to be reduced include fixed-point number formats associated with values of different types. The method 1000 begins at block 1002, where a determination is made as to whether there are fixed-point number formats of input data values to be reduced. If there are fixed-point number formats of input data values to be reduced, those fixed-point number formats are first reduced, and the method 1000 proceeds to block 1004. If there are no fixed-point number formats of input data values to be reduced, the method 1000 proceeds to block 1006. At block 1004, the mantissa bit lengths of the fixed-point number formats of the input data values for the layers in the subset are iteratively reduced until an error threshold is exceeded (e.g., based on, for example, Figure 7 Method 700). Method 1000 then proceeds to block 1006.
[0113] At block 1006, it is determined whether there are fixed-point number formats of weights to be reduced. If there are fixed-point number formats of weights to be reduced, then those fixed-point number formats are next reduced and the method 1000 proceeds to block 1008. If there are no fixed-point number formats of weights to be reduced, then the method 1000 proceeds directly to block 1010. At block 1008, the mantissa bit length of the fixed-point number formats of the weights of the layers in the subset is iteratively reduced until an error threshold is exceeded (e.g., based on, for example, Figure 7 Method 700). The method 1000 then proceeds to block 1010.
[0114] At block 1010, it is determined whether there are fixed-point number formats of the bias to be reduced. If there are fixed-point number formats of the bias to be reduced, then those fixed-point number formats are next reduced and the method 1000 proceeds to block 1012. If there are no fixed-point number formats of the bias to be reduced, then the method 1000 proceeds directly to block 1014. At block 1012, the mantissa bit lengths of the fixed-point number formats of the biases of the layers in the subset are iteratively reduced until an error threshold is exceeded (e.g., based on, for example, Figure 7 Method 700). Method 1000 then proceeds to block 1014.
[0115] At block 1014, a determination is made as to whether there are fixed-point formats of output data values to be reduced. If there are fixed-point formats of output data values to be reduced, then those fixed-point formats are reduced next and method 1000 proceeds to block 1016. If there are no fixed-point formats of output data values to be reduced, then method 1000 ends. At block 1016, the mantissa bit lengths of the fixed-point formats of the output data values of the layers in the subset are iteratively reduced until an error threshold (e.g., based on the error threshold) is exceeded. Figure 7 Method 700). Then method 1000 ends.
[0116] Example Results
[0117] Reference now Fig.11 and Fig.12 And Tables 3 and 4, which show the selected mantissa bit lengths for the layers of AlexNet DNN and GoogLeNet DNN. Specifically, Table 3 shows the average mantissa bit lengths of the layers of AlexNet DNN for different error thresholds (expressed as a fraction of the L1 length of the baseline output logit) when 50 test images are used, the initial subset size G is set to 2, and the fixed-point format of the first and last layers is set to 12 bits (except for the content described below).
[0118] Table 3
[0119]
[0120] Similarly, Table 4 shows the average mantissa bit length of the layers of the GoogLeNet DNN for different error thresholds (expressed as a fraction of the L1 length of the baseline output logit) when 50 test images are used, the initial subset size G is set to 4, and the fixed-point number format of the first and last layers is set to 12 bits.
[0121] Table 4
[0122]
[0123] These tables show that the method described in this paper can reduce the number of bits non-uniformly across the network, and that higher thresholds require more bits.
[0124] Fig.11 The selected mantissa bit lengths for the weights 1102 and input data values 1104 of each convolutional layer of the GoogLeNet DNN are graphically shown with an error threshold of 0.9.
[0125] Fig.12 The selected mantissa bit lengths for the weights 1202 and input data values 1204 of each convolutional layer of the GoogLeNet DNN are graphically shown with an error threshold of 0.995.
[0126] Example Hardware Implementation of DNN
[0127] Reference now Fig.13 , which shows an example hardware implementation 1300 of a DNN, which can be based on the use of Figure 3 , Figure 7 and Fig.10 The method 300, 700 and 1000 are configured according to the fixed-point number format recognized by the method 300, 700 and 1000.
[0128] Fig.13The hardware implementation 1300 is configured to compute the output of the DNN through a series of hardware channels (which may also be referred to as processing channels), wherein during each channel, the hardware implementation receives at least a portion of the input data of a layer of the DNN and processes the received input data according to the layer (and optionally according to one or more subsequent layers) to produce processed data. The processed data is either output to a memory for use as input data for a subsequent hardware channel or output as an output of the DNN. The number of layers that the hardware implementation can process during a single hardware channel can be based on the size of the data, the hardware implementation, and the order of the layers. For example, where the hardware implementation includes hardware for executing each possible layer type, a DNN including a first convolutional layer, a first activation layer, a second convolutional layer, a second activation layer, and a pooling layer can receive initial DNN input data in a first hardware channel and process the input data according to the first convolutional layer and the first activation layer and output the output of the activation layer to a memory, and then in a second hardware channel, receive the data from the memory as input and process the data according to the second convolutional layer, the second activation layer, and the pooling layer to produce output data of the DNN.
[0129] Fig.13 An example hardware implementation 1300 of a DNN includes: an input module 1301, a convolution engine 1302, an accumulation buffer 1304, an element-wise operation module 1306, an activation module 1308, a normalization module 1310, a pooling module 1312, an output interleaving module 1314, and an output module 1315. Each module or engine implements or processes all or part of one or more types of layers. Specifically, the convolution engine 1302 and the accumulation buffer 1304 together implement or process a convolution layer or a fully connected layer. The activation module 1308 processes or implements an activation layer. The normalization module 1310 processes or implements a normalization layer. The pooling module 1312 implements a pooling layer, and the output interleaving module 1314 processes or implements an interleaving layer.
[0130] The input module 1301 is configured to receive input data of the current hardware channel and provide it to downstream modules for processing. The downstream module receiving the input data depends on the layer to be processed in the current hardware channel.
[0131] The convolution engine 1302 is configured to perform a convolution operation on the received input data using the weights associated with a particular convolution layer. The weights of each convolution layer of the DNN may be stored in a coefficient buffer 1316, as shown in FIG. Fig.13As shown, and when the convolution engine 1302 is processing a particular convolution layer, the weight of the particular convolution layer can be provided to the convolution engine 1302. In the case where the hardware implementation supports a variable weight format, the convolution engine 1302 can be configured to receive information indicating one or more formats of the weights of the current convolution layer being processed to allow the convolution engine to correctly parse and process the received weights. The format of the weights of each convolution layer can be determined by the method described herein.
[0132] The convolution engine 1302 may include a plurality of multipliers (eg, 128) and a plurality of adders that add the results of the multipliers to produce a single sum. Fig.13 A single convolution engine 1302 is shown in FIG, but in other examples, there may be multiple (eg, 8) convolution engines so that multiple windows can be processed simultaneously. The output of the convolution engine 1302 is fed to an accumulation buffer 1304.
[0133] Accumulation buffer 1304 is configured to receive the output of the convolution engine and add it to the current contents of accumulation buffer 1304. In this manner, accumulation buffer 1304 accumulates the results of convolution engine 1302. Fig.13 A single accumulation buffer 1304 is shown in FIG, but in other examples, there may be multiple (e.g., 8, 1 for each convolution engine) accumulation buffers. Accumulation buffer 1304 outputs the accumulation result to element-level operation module 1306, which may or may not operate on the accumulation result depending on whether an element-level layer is to be processed during the current hardware pass.
[0134] The element-level operation module 1306 is configured to either receive input data of the current hardware channel (e.g., when the convolution layer is not processed in the current hardware channel) or receive the accumulation result from the accumulation buffer 1304 (e.g., when the convolution layer is processed in the current hardware channel). The element-level operation module 1306 can either process the received input data or pass the received input data to another module (e.g., the activation module 1308 and / or the normalization module 1310), depending on whether the element-level layer is processed in the current hardware channel and / or whether the activation layer is to be processed before the element layer. When the element-level operation module 1306 is configured to process the received input data, the element-level operation module 1306 performs an element-level operation on the received data (optionally another data set (which can be obtained from an external memory)). The element-level operation module 1306 can be configured to perform any suitable element-level operation, such as, but not limited to, addition, multiplication, maximum, and minimum. The results of the feature-level operations are then provided to the activation module 1308 or the normalization module 1310, depending on whether the activation layer is to be processed after the feature-level layer.
[0135] The activation module 1308 is configured to receive as input data one of: the raw input of the hardware channel (via the element-level operation module 1306) (e.g., in the case where the convolution layer is not processed in the current hardware channel); the accumulated data (via the element-level operation module 1306) (e.g., in the case where the convolution layer is processed in the current hardware channel and the element-level layer is either not processed in the current hardware channel or is processed in the current hardware channel but after the activation layer). The activation module 1308 is configured to apply the activation function to the input data and provide the output data back to the element-level operation module 1306, which forwards it to the normalization module 1310 directly or after it is processed by the element-level operation module 1306. In some cases, the activation function applied to the data received by the activation module 1308 may vary depending on the activation layer. In these cases, information specifying one or more attributes of the activation function to be applied to each activation layer may be stored (e.g., in a memory), and the relevant information for the activation layer processed in the hardware channel may be provided to the activation module 1308 during a particular hardware channel.
[0136] In some cases, the activation module 1308 can be configured to store data representing the activation function in an entry of a lookup table. In these cases, the input data can be used to look up one or more entries in the lookup table and output a value representing the output of the activation function. For example, the activation module 1308 can be configured to calculate the output value by interpolating between two or more entries read from the lookup table.
[0137] In some examples, activation module 1308 can be configured to operate as a rectified linear unit (Rectified Linear Unit, ReLU) function. In the ReLU function, the output element y is calculated by identifying the maximum value as shown in formula (4). i,j,k , where y=0 for values of x less than 0:
[0138] yi ,j,k =f(x i,j,k )=max{0,x i,j,k} (4)
[0139] In other examples, the activation module 1308 can be configured to operate as a parametric rectified linear unit (PReLU) by implementing a PReLU function. The PReLU function performs similar operations to the ReLU function. Specifically, When is a constant, PReLU is configured to generate the output feature y as shown in formula (5) i,j,k :
[0140] y i,j,k =f(x i,j,k ;w1,w2,b1,b2)=max{(w1*x i,j,k +b1),(w2*x i,j,k +b2)} (5)
[0141] The normalization module 1310 is configured to receive one of the following as input data: the original input data of the hardware channel (via the element-level operation module 1306) (for example, when the convolution layer is not processed in the current hardware channel and neither the element-level layer nor the activation layer is processed in the current hardware channel); the accumulated data (via the element-level operation module 1306) (for example, when the convolution layer is processed in the current hardware channel and neither the element-level layer nor the activation layer is processed in the current hardware channel); and the output data of the element-level operation module and / or the activation module. Then, the normalization module 1310 performs a normalization function on the received input data to generate normalized data. In some cases, the normalization module 1310 can be configured to perform a local response normalization (LRN) function and / or a local contrast normalization (LCN) function. However, it is easy for a person skilled in the art to understand that these are only examples, and the normalization module 1310 can be configured to implement any suitable one or more normalization functions. Different normalization layers can be configured to apply different normalization functions.
[0142] The pooling module 1312 may receive normalized data from the normalization module 1310, or may receive input data for the normalization module 1310 via the normalization module 1310. In some cases, data may be transferred between the normalization module 1310 and the pooling module 1312 via the XBar 1318. The term "XBar" as used herein refers to a simple hardware module that includes routing logic that connects multiple modules together in a dynamic manner. In this example, the XBar may dynamically connect the normalization module 1310, the pooling module 1312, and / or the output interleaving module 1314, depending on which layers are to be processed in the current hardware channel. Thus, the XBar may receive information in each hardware channel indicating which modules 1310, 1312, 1314 are to be connected.
[0143] The pooling module 1312 is configured to perform a pooling function on the received data to generate pooled data, wherein the pooling function is, for example, but not limited to, a maximization function or an average function. The purpose of the pooling layer is to reduce the spatial size of the representation, thereby reducing the number of parameters and calculations in the network, and thus also controlling overfitting. In some examples, the pooling operation is performed on a sliding window defined based on each pooling layer.
[0144] The output interleaving module 1314 may receive normalized data from the normalization module 1310, input data of a normalization function (via the normalization module 1310), or pooled data from the pooling module 1312. In some cases, these data may be transmitted between the normalization module 1310, the pooling module 1312, and the output interleaving module 1314 via the XBar 1318. The output interleaving module 1314 is configured to perform a rearrangement operation to generate data in a predetermined order. This may include sorting and / or transposing the received data. The data generated by the last layer is provided to the output module 1315, where the data is converted into the desired output format for the current hardware channel. The desired output format may be determined according to the method described herein. For example, the desired output format may be determined as a fixed-point format for representing the output data values of the last layer of the hardware channel or a fixed-point format for representing the input data values of the first layer of the next hardware channel.
[0145] Normalization module 1310, pooling module 1312 and output interleaving module 1314 can all access shared buffer 1320, and shared buffer 1320 can be used by these modules 1310, 1312 and 1314 to write data therein and obtain data therefrom. For example, shared buffer 1320 can be used by these modules 1310, 1312, 1314 to rearrange the order of received data or generated data. For example, one or more of these modules 1310, 1312, 1314 can be configured to write data to shared buffer 1320 and read the same data in another order. In some cases, although each of normalization module 1310, pooling module 1312 and output interleaving module 1314 can access shared buffer 1320, each of normalization module 1310, pooling module 1312 and output interleaving module 1314 can be allocated a portion of shared buffer 1320 that only they can access. In these cases, each of the normalization module 1310 , the pooling module 1312 , and the output interleaving module 1314 may only be able to read from the shared buffer 1320 the data that they have written to the shared buffer 1320 .
[0146] As described above, the modules used or activated by the hardware implementation 1300 during any hardware pass are based on the layers to be processed during that hardware pass. In particular, only modules or components related to the layers to be processed during the current hardware pass are used or activated. As described above, the layers to be processed during a particular hardware pass are determined (typically determined in advance, for example, by a software tool) based on the order of the layers in the DNN and optionally one or more other factors (e.g., the size of the data). For example, in some cases, the hardware implementation may be configured to perform processing of a single layer on a per hardware pass basis unless multiple layers can be processed without writing data to memory between layers. For example, if a second convolutional layer follows a first convolutional layer, each convolutional layer must be executed in a separate hardware pass because the output data from the first hardware convolution needs to be written to memory before it can be used as input to the second hardware convolution. In each of these hardware passes, only modules, components, or engines related to the convolutional layers, such as the convolution engine 1302 and the accumulation buffer 1304, may be used or activated.
[0147] although Fig.13 The hardware implementation 1300 shows a specific order in which modules, engines, etc. are arranged and thus shows how processing of data flows through the hardware implementation, but it should be understood that this is merely an example and that in other examples, modules and engines, etc. may be arranged in other ways. Furthermore, other hardware implementations may implement additional or alternative DNN layer types and therefore may include different modules, engines, etc.
[0148] Fig.14 Various components of an exemplary general-purpose computing-based device 1400 are shown, which can be implemented as any form of computing and / or electronic device and in which embodiments of the above-mentioned methods 300, 700, 1000 can be implemented.
[0149] The computing-based device 1400 includes one or more processors 1402, which may be microprocessors, controllers, or any other suitable type of processors, for processing computer-executable instructions to control the operation of the device, thereby selecting a fixed-point number format for a set of values input to and output from a layer of the DNN. In some examples, such as where a system-on-chip architecture is used, the processor 1402 may include one or more fixed function blocks (also referred to as accelerators) that implement a portion of a method for processing computer-executable instructions to control the operation of the device in hardware (rather than software or firmware). Platform software including an operating system 1404 or any other suitable platform software may be provided at the computing-based device to enable application software, such as for implementing Figure 3 The computer executable code 1405 of the method 300 is executed on the device.
[0150] Computer executable instructions may be provided using any computer-readable medium accessible by the computing-based device 1400. Computer-readable media may include, for example, computer storage media, such as memory 1406 and communication media. Computer storage media (i.e., non-transient machine-readable media), such as memory 1406, include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, etc. Computer storage media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other non-transmission medium that can be used to store information for access by a computing device. In contrast, communication media may embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transmission mechanism. As defined herein, computer storage media do not include communication media. Although computer storage media (i.e., non-static machine-readable media, such as memory 1406) are shown within computing-based device 1400, it should be appreciated that the storage devices may be distributed or remotely distributed and accessed via a network or other communications link (e.g., using communications interface 1408).
[0151] The computing-based device 1400 also includes an input / output controller 1410, which is arranged to output display information to a display device 1412, which may be separate from or integrated with the computing-based device 1400. The display information may provide a graphical user interface. The input / output controller 1410 is also arranged to receive and process input from one or more devices, such as a user input device 1414 (e.g., a mouse or keyboard). In one embodiment, if the display device 1412 is a touch-sensitive display device, the display device 1412 may also be used as a user input device 1414. The input / output controller 1410 may also output data to devices other than the display device, such as a locally connected printing device (e.g., a printer). Fig.14 not shown).
[0152] Fig.15 15 shows a computer system that can implement the hardware implementation of the DNN described herein. The computer system includes a CPU 1502, a GPU 1504, a memory 1506, and other devices 1514, such as a display 1516, a speaker 1518, and a camera 1520. The hardware implementation 1510 of the DNN (corresponding to Fig.13 The hardware implementation 1300 of the DNN can be implemented on the GPU 1504, such as Fig.15In some examples, there may be no GPU and the CPU may provide control information for the hardware implementation 1510 of the DNN. The components of the computer system may communicate with each other via a communication bus 1522. In other examples, the hardware implementation 1510 of the DNN may be implemented independently of the CPU or GPU and may have a separate connection to the communication bus 1522.
[0153] Fig.13 The hardware implementation 1300 of the DNN is shown as including multiple functional blocks. This is merely illustrative and does not intend to define a strict division between the different logical elements of these entities. Each functional block may be provided in any suitable manner. It will be understood that the intermediate values formed by the hardware implementation or processing module of the DNN described herein do not need to be physically generated at any point by the hardware implementation of the DNN, but may simply represent logical values that are convenient for describing the processing performed by the hardware implementation of the DNN between its input and output.
[0154] The hardware implementation of the DNN described herein can be embodied in hardware on an integrated circuit. In general, any of the functions, methods, techniques, or components described above can be implemented in software, firmware, hardware (e.g., fixed logic circuits), or any combination thereof. The terms "module", "function", "component", "element", "unit", "block", and "logic" may be used herein to generally represent software, firmware, hardware, or any combination thereof. In the case of software implementation, a module, function, component, element, unit, block, or logic represents a program code that performs a specified task when executed on a processor. The algorithms and methods described herein can be performed by one or more processors that execute code, which causes the one or more processors to perform the algorithm / method. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disks, flash memory, hard disk storage, and other memory devices that can use magnetic, optical, or other technologies to store instructions or other data that can be accessed by a machine.
[0155] The terms "computer program code" and "computer-readable instructions" are used herein to refer to any kind of executable code for a processor, including code expressed in a machine language, a parsed language, or a scripting language. Executable code includes binary code, machine code, byte code, code that defines an integrated circuit (e.g., a hardware description language or a netlist), and code expressed in a programming language code (e.g., C, Java, or OpenCL). Executable code can be, for example, any kind of software, hardware, script, module, or library that, when properly executed, processed, parsed, assembled, executed in a virtual machine or other software environment, causes a processor of a computer system that supports the executable code to perform the tasks specified by the code.
[0156] A processor, computer, or computer system may be any kind of device, machine, or special circuit, or a collection or part thereof, that has processing capabilities so that it can execute instructions. A processor may be any kind of general or special processor, such as a CPU, GPU, system on a chip, state machine, media processor, application specific integrated circuit (ASIC), programmable logic array, field programmable gate array (FPGA), etc. A computer or computer system may include one or more processors.
[0157] It is also intended to cover software that defines the hardware configuration described herein, for example, HDL (Hardware Description Language) software for designing integrated circuits or for configuring programmable chips to achieve desired functionality. That is, a computer-readable storage medium may be provided having encoded thereon computer-readable program code in the form of an integrated circuit definition data set that, when processed (i.e., run) in an integrated circuit manufacturing system, configures the system to manufacture a hardware implementation of the DNN described herein. The integrated circuit definition data set may be, for example, an integrated circuit description.
[0158] Thus, a method of manufacturing a hardware implementation of the DNN described herein at an integrated circuit manufacturing system is provided. Furthermore, an integrated circuit definition data set is provided, which, when processed by the integrated circuit manufacturing system, causes the method of manufacturing a hardware implementation of the DNN to be performed.
[0159] The integrated circuit definition data set may be in the form of computer code, for example, as a netlist, code for configuring a programmable chip, as a hardware description language (including register transfer level (RTL) code) that defines hardware suitable for the manufacture of an integrated circuit at any level, as a high-level circuit representation (e.g., Verilog or VHDL), and as a low-level circuit representation (e.g., OASIS (RTM) and GDSII). The higher-level representation (e.g., RTL) that logically defines hardware suitable for the manufacture of an integrated circuit may be processed at a computer system configured to generate a manufacturing definition of an integrated circuit in the context of a software environment, wherein the manufacturing definition includes definitions of circuit elements and rules for combining these elements to generate a manufacturing definition representing the defined integrated circuit. In the general case of executing software at a computer system to define a machine, one or more intermediate user steps (e.g., providing commands, variables, etc.) may be required to cause the computer system configured to generate a manufacturing definition of an integrated circuit to execute the code that defines the integrated circuit to thereby generate the manufacturing definition of the integrated circuit.
[0160] Reference now Fig.16, describing an example of processing an integrated circuit definition dataset at an integrated circuit manufacturing system to configure the system to manufacture a hardware implementation of a DNN.
[0161] Fig.16 An example of an integrated circuit (IC) manufacturing system 1602 configured to manufacture a hardware implementation of a DNN described in any of the examples herein is shown. Specifically, the IC manufacturing system 1602 includes a layout processing system 1604 and an integrated circuit generation system 1606. The IC manufacturing system 1602 is configured to receive an IC definition data set (e.g., defining a hardware implementation of a DNN described in any of the examples herein), process the IC definition data set, and generate an IC according to the IC definition data set (e.g., an IC definition data set embodying a hardware implementation of a DNN described in any of the examples herein). The processing of the IC definition data set configures the IC manufacturing system 1602 to manufacture an integrated circuit embodying a hardware implementation of a DNN described in any of the examples herein.
[0162] The layout processing system 1604 is configured to receive and process an IC definition data set to determine a circuit layout. Methods for determining a circuit layout based on an IC definition data set are known in the art, and may include, for example, synthesizing RTL code to determine a gate-level representation of the circuit to be generated (e.g., in terms of logic components (e.g., NAND, NOR, AND, OR, MUX, and FLIP-FLOP components)). The circuit layout may be determined from the gate-level representation of the circuit by determining location information of the logic components. This may be implemented automatically or with user participation to optimize the circuit layout. When the layout processing system 1604 has determined the circuit layout, it may output a circuit layout definition to the IC generation system 1606. The circuit layout definition may be, for example, a circuit layout description.
[0163] IC generation system 1606 generates an IC according to the circuit layout definition, as is known in the art. For example, IC generation system 1606 may perform a semiconductor device fabrication process to generate the IC, which may include a multi-step sequence of photolithography processes and chemical processing steps during which electronic circuits are gradually formed on a wafer made of semiconductor material. The circuit layout definition may be in the form of a mask that may be used in the photolithography process to generate the IC according to the circuit definition. Alternatively, the circuit layout definition provided to IC generation system 1606 may be in the form of a computer readable code, which IC generation system 1606 may use to form an appropriate mask for generating the IC.
[0164] The different processes performed by IC manufacturing system 1602 may all be implemented at one location, e.g., by one party. Alternatively, IC manufacturing system 1602 may be a distributed system so that some of the processes may be performed at different locations and by different parties. For example, some of the following stages may be performed at different locations and / or by different parties: (i) synthesizing RTL code representing an IC definition data set to form a gate-level representation of a circuit to be generated, (ii) generating a circuit layout based on the gate-level representation, (iii) forming a mask based on the circuit layout, and (iv) fabricating the integrated circuit using the mask.
[0165] In other examples, processing of an integrated circuit definition data set at an integrated circuit manufacturing system may configure the system to manufacture a hardware implementation of a DNN without processing the IC definition data set to determine a circuit layout. For example, an integrated circuit definition data set may define a configuration of a reconfigurable processor (e.g., an FPGA), and processing of the data set may configure the IC manufacturing system to generate a reconfigurable processor having the defined configuration (e.g., by loading the configuration data into the FPGA).
[0166] In some embodiments, when the integrated circuit manufacturing definition data set is processed in the integrated circuit manufacturing system, it can cause the integrated circuit manufacturing system to generate the device described herein. Fig.16 The configuration of an integrated circuit manufacturing system in the described manner may enable the apparatus described herein to be manufactured.
[0167] In some examples, the integrated circuit definition data set may include software running on or in combination with hardware defined at the data set. Fig.16 In the example shown, the IC generation system can be further configured by the integrated circuit definition data set to load firmware into the integrated circuit or provide program code to the integrated circuit for use by the integrated circuit according to the program code defined in the integrated circuit definition data set when manufacturing the integrated circuit.
[0168] The concepts set forth in this application can produce performance improvements compared to known implementations in devices, apparatuses, modules, and / or systems (and methods implemented herein).Performance improvements can include one or more of increased computing performance, reduced delays, increased throughput, and / or reduced power consumption.During (e.g., in integrated circuits) manufacturing these devices, devices, modules, and systems, a balance can be made between performance improvements and physical implementations, thereby improving manufacturing methods.For example, a balance can be made between performance improvements and layout areas, thereby matching the performance of known implementations but using less silicon.This can be achieved, for example, by reusing functional blocks in a continuous manner or sharing functional blocks between elements of devices, devices, modules, and / or systems.On the contrary, the concepts of this application that lead to improvements in the physical implementation of devices, devices, modules, and systems (e.g., silicon area reduction) can be exchanged for improved performance.This can be achieved, for example, by manufacturing multiple instances of modules in a predetermined area budget.
[0169] The applicant discloses each individual feature described herein separately, and any combination of two or more such features, so that these features or combinations can be implemented based on the specification according to the common knowledge of those skilled in the art, regardless of whether these features or feature combinations solve any problems disclosed herein. In view of the above description, it is easy for those skilled in the art to understand that various modifications can be made within the scope of the present invention.
Claims
1. A computer-implemented method for selecting a fixed-point number format for representing values input to and / or output from a plurality of layers of a deep neural network (DNN) for configuring a hardware implementation of the DNN, the method comprising: receiving an instance of the DNN, the instance configured to represent values of each layer of the plurality of layers using one or more initial fixed-point number formats for the layer, each initial fixed-point number format comprising exponent and mantissa bit lengths; identifying a sequence of the plurality of layers, wherein each layer in the sequence is preceded by any layer of the plurality of layers on which it depends; forming a plurality of disjoint subsets from the plurality of layers, wherein each of the subsets comprises a set of consecutive layers in the sequence; for each subset in a plurality of subsets, iteratively adjusting a fixed-point number format for a layer in the subset to a fixed-point number format having a next lowest mantissa bit length until an output error of the instance of the DNN exceeds an error threshold; In response to determining that the subset includes more than a lower threshold number of layers, forming a greater number of disjoint subsets from the plurality of layers and repeating the iterative adjustment; and In response to determining that the subset includes a number of layers less than or equal to the lower threshold, a fixed point number format for the plurality of layers is output.
2. The method of claim 1, wherein: Iteratively adjusting the fixed-point number format for the layers in the subset to a fixed-point number format having a next lowest mantissa bit length comprises: For each layer of the subset of fixed-point number formats, determining a fixed-point number format having a next lowest mantissa bit length; adjusting a fixed-point number format for each layer in the subset used by an instance of the DNN to the determined fixed-point number format having a next lowest mantissa bit length; determining an output of the adjusted instance of the DNN in response to test input data; determining an output error of the adjusted instance of the DNN; In response to determining that the output error exceeds the error threshold, reversing the adjustment to the instance of the DNN; and In response to determining that the output error does not exceed the error threshold, determining a fixed-point number format, adjusting the fixed-point number format, determining an output, and determining an output error are repeated.
3. The method according to claim 1 or 2, wherein: The multiple layers do not include a first layer of the DNN and / or a last layer of the DNN.
4. The method according to claim 1 or 2, wherein: The first adjustment to the fixed-point format is performed on all subsets before the second adjustment to the fixed-point format is performed on any subset.
5. The method according to claim 1 or 2, wherein: Before performing the first adjustment on the fixed-point number format of the layers in the second subset, all iterative adjustments on the fixed-point number format of the layers in the first subset are completed.
6. The method according to claim 1 or 2, wherein: There is an initial fixed-point number format for input data values of at least one layer of the plurality of layers, and there is an initial fixed-point number format for weights of at least one layer of the plurality of layers, and iteratively adjusting the fixed-point number format for the layers in the subset to a fixed-point number format having a next lowest mantissa bit length until an output error of the instance of the DNN exceeds an error threshold comprises: iteratively adjusting the fixed-point number format of input data values for the layers in the subset to a fixed-point number format having a next lowest mantissa bit length until an output error of the instance of the DNN exceeds the error threshold; and After iteratively adjusting the fixed-point number format for the input data value, iteratively adjusting the fixed-point number format of the weights of the layers in the subset to a fixed-point number format with a next lowest mantissa bit length until an output error of the instance of the DNN exceeds the error threshold.
7. The method of claim 6, wherein: There is an initial fixed-point number format for output data values of at least one layer in the plurality of layers, and iteratively adjusting the fixed-point number format for the layers in the subset to a fixed-point number format having a next lowest mantissa bit length until an output error of the instance of the DNN exceeds an error threshold comprises: After iteratively adjusting the fixed-point number format for the input data values, iteratively adjusting the fixed-point number format for the output data values of the layers in the subset to a fixed-point number format having a next lowest mantissa bit length until an output error of the instance of the DNN exceeds an error threshold.
8. The method according to claim 1 or 2, wherein: The DNN is a classification network, and the output error is a top-1 classification accuracy or a top-5 classification accuracy of an output of an instance of the DNN in response to test input data.
9. The method according to claim 1 or 2, wherein: The DNN is a classification network, and the output error is the sum of the L1 differences between the logarithm of the output of the instance of the DNN in response to the test input data and the logarithm of the baseline output, or the output error is the sum of the L1 differences between the SoftMax normalized logarithm of the output of the instance of the DNN in response to the test input data and the SoftMax normalized logarithm of the baseline output.
10. The method of claim 9, further comprising: The baseline output is generated by applying the test input data to an instance of the DNN configured to represent values input to and output from each layer of the DNN using a floating point number format.
11. The method according to claim 1 or 2, wherein: The lower threshold number of layers is 1.
12. The method according to claim 1 or 2, wherein: The lower threshold number of layers is greater than one.
13. The method according to claim 1 or 2, wherein: Forming a greater number of disjoint subsets from the plurality of layers includes dividing the layers in each subset into a plurality of disjoint subsets and / or forming twice as many disjoint subsets from the plurality of layers.
14. The method according to claim 1 or 2, wherein: Values input to and / or output from the multiple layers include one or more of the following: input data values, output data values, weights, and biases.
15. The method according to claim 1 or 2, wherein: The DNN is a convolutional neural network.
16. The method of claim 1 or 2, further comprising: The hardware implementation of the DNN is configured to represent a value of at least one of the plurality of layers using a fixed-point number format output for the at least one layer.
17. A computer-readable storage medium having computer-readable instructions stored thereon, which, when executed on a computer system, causes the computer system to execute the method of claim 1 or 2.
18. A hardware implementation device of a deep neural network DNN, comprising: Hardware logic, which is configured as: Receiving an input data value, a set of weights, or a set of biases for a layer of the DNN; Receiving information indicating a fixed-point number format of an input data value, a weight set, or a bias set for the layer, wherein the fixed-point number format of the input data value, the weight set, or the bias set for the layer is selected according to the method of claim 1 or 2; Parsing an input data value, a weight set, or a bias set for the layer based on a fixed-point number format of the input data value, a weight set, or a bias set; and The parsed input data values, sets of weights, or sets of biases are processed according to the layer to generate output data values for the layer.
19. The hardware implementation device of DNN according to claim 18, wherein: The hardware logic is also configured to: receiving information indicating a fixed-point format of output data values for the layer, the fixed-point format of output data values for the layer being selected according to the method of claim 1 or 2; and The output data value of the layer is converted to a fixed point number format for the output data value of the layer.
