Low power hardware architecture for convolutional neural networks

Abstract

Dynamic data quantization can be applied to minimize power consumption of a system implementing a convolutional neural network (CNN). Under such a quantization scheme, a quantized representation of a 3x3 m-bit activation value array can include 9 n-bit mantissa values and one exponent shared among the n-bit mantissa values (n < m); and a quantized representation of a 3x3 kernel having p-bit parameter values can include 9 q-bit mantissa values and one exponent shared among the q-bit mantissa values (q < p). Convolution of the kernel with activation data can include computing a dot product of the 9 n-bit mantissa values with the 9 q-bit mantissa values and summing the shared exponent. In a CNN having multiple kernels, multiple compute units (each corresponding to one of the kernels) can receive the quantized representation of the 3x3 m-bit activation value array from a same quantization-calibration module.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 16 / 948,164, filed on September 4, 2020. Technical Field

[0003] This invention relates to a hardware architecture for reducing the power consumption of convolutional neural networks (CNNs), and more specifically to a hardware architecture for applying dynamic data quantization schemes to CNNs. Background Technology

[0004] Today, Convolutional Neural Networks (CNNs) are widely used to perform tasks such as image recognition, object recognition, and image segmentation. CNNs require intensive computational processing for many applications, which can lead to high power consumption. This paper describes a hardware architecture for reducing the power consumption of CNNs.

[0005] The paper “End-to-End DNN Training with Block Floating Point Arithmetic” by Drumond et al., ARXIV.ORG, Cornell University Library, 201 Olin Library, Cornell University Ithaca, NY 14853, April 4, 2018 (2018-04-04), XP080867642, describes a hybrid approach that employs a dense block floating-point (BFP) algorithm for both dot product computation and floating-point (FP) algorithms. Summary of the Invention

[0006] According to one embodiment of the invention, a dynamic data quantization scheme is used to minimize the power consumption of a convolutional neural network (CNN). Data quantization reduces the bit width of the data signal and thus reduces power consumption. However, the trade-off of data quantization is a reduction in the numerical precision of the quantized values. A specific data quantization method (i.e., dynamic data quantization) is employed to minimize the precision loss caused by data quantization. The dynamic data quantization scheme utilizes the characteristic that the dynamic range of local blocks of the activation data (i.e., the dimension corresponding to the convolution kernel) is typically smaller than the dynamic range of the entire array of activation data. Therefore, each local block of the activation data is quantized independently of its neighboring blocks, such that the bits of the quantized output are used only to represent values ​​within a more constrained local dynamic range. The dynamic data quantization scheme is similarly applied to quantize each element in the convolution kernel.

[0007] A quantized representation of a 3x3 m-bit activation value array includes 9 n-bit mantissa values and one exponent shared between the n-bit mantissa values, where n is less than m. A quantized representation of a 3x3 kernel with p-bit parameter values includes 9 q-bit mantissa values and one exponent shared between the q-bit mantissa values, where q is less than p. Convolution of the kernel with the activation data includes computing the dot product of the 9 n-bit mantissa values with the 9 q-bit mantissa values and summing the two shared exponents. In scenarios with multiple convolution kernels, multiple compute units (each corresponding to one of the convolution kernels) receive the quantized representation of the 3x3 m-bit activation value array from the same quantizer-calibration module.

[0008] In one embodiment of the invention, the quantized representation of the kernel is pre-computed and stored in a storage element, so that when the convolution is performed during the application (inference) phase of the model of the CNN, the quantized representation of the kernel can be read directly from the memory element. In another embodiment, the quantization of the kernel can be performed "on-the-fly", which can be necessary during the training phase of the CNN.

[0009] These and other embodiments of the invention are more fully described in connection with the following drawings. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1A A hardware architecture of a system configured to convolve multiple kernels with activation data is shown, according to one embodiment of the invention, wherein the convolution involves computing the dot product of a quantized representation of an activation value array (also referred to as "local patch") with a quantized representation of a convolution kernel.

[0011] Figure 1B A hardware architecture of a system configured to convolve multiple kernels with activation data is shown, according to one embodiment of the invention, wherein the convolution involves computing the dot product of a quantized representation of an activation value array (also referred to as "local patch") with a quantized representation of a convolution kernel. Figure 1A Variations of the hardware architecture shown in

[0012] Figure 2 A circuit diagram of a separator is shown, according to one embodiment of the invention.

[0013] Figure 3 A block diagram with additional details regarding the inputs and outputs of the quantizer-calibration module used to quantize an activation value array is shown, according to one embodiment of the invention.

[0014] Figure 4 A block diagram with additional details of the quantizer-calibration module of Figure 3 is shown, according to one embodiment of the invention.

[0015] Figure 5A and Figure 5B Mathematical examples of the quantizer-calibration operation, according to various embodiments of the invention, are shown.

[0016] Figure 6A and Figure 6BA bit-level example of a quantizer-calibration operation is shown in accordance with various embodiments of the application.

[0017] Figure 7 A block diagram showing additional details of an input and output with respect to a quantizer-calibration module for quantizing a convolution kernel in accordance with one embodiment of the application.

[0018] Figure 8A A block diagram showing additional details of "vertical" electrical interconnections associated with a compute unit in accordance with one embodiment of the application.

[0019] Figure 8B A block diagram showing additional details of "horizontal" electrical interconnections associated with a compute unit in accordance with one embodiment of the application.

[0020] Figure 9 A block diagram showing additional details of an input and output with respect to a compute unit in accordance with one embodiment of the application.

[0021] Figure 10A and Figure 10B A mathematical example of computing a dot product of a quantized representation of an array of activation values and a quantized representation of a convolution kernel in accordance with various embodiments of the application is shown. DETAILED DESCRIPTION

[0022] In the following detailed description of preferred embodiments, reference is made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration specific embodiments in which the application can be practiced. It is to be understood that other embodiments can be utilized and structural changes can be made without departing from the scope of the present application. Descriptions associated with any one drawing can be applied to different drawings that contain the same or similar components / steps.

[0023] Before discussing the details of a system for performing convolution operations, a technical overview is presented to provide a more general context in which the concepts of the present application can be practiced and understood. As known in the art, a convolution is an integral mathematical operation in a convolutional neural network (CNN). The central task in a convolution is to compute a dot product between a kernel and a "local patch" of activation data, which has dimensions matching the size of the kernel. Activation data refers to either raw input data (e.g., sensor data) or output from a previous stage of a CNN.

[0024] For simplicity of explanation, a two-dimensional kernel will be discussed, and specifically a 3x3 kernel will be discussed, but it should be understood that the concepts described herein can apply to kernels having other dimensions (e.g., 4x4, 5x5) or higher dimensional kernels (e.g., 3x3x3). In a convolution operation, the kernel is conceptually “shifted” relative to the activation data in a horizontal and / or vertical manner, and the dot product operation is repeated for each shifted position of the kernel. While the “shifting” of the kernel is a useful way to visualize the convolution operation, another way to visualize the convolution operation is to place repeated instances of the kernel at different shifted positions relative to the activation data. The latter visualization is more computationally efficient because it allows multiple instances of the dot product operation to be computed in parallel (i.e., simultaneously). It is additionally noted that the application of the concepts described herein are not limited to convolution operations, and can be applied more generally to dot products or multiplications of two matrices having small matrix sizes (e.g., no more than 5x5 = 25 values).

[0025] One important goal of the present disclosure is to reduce the power consumption of CNNs, as low power designs provide many advantages. First, low power designs reduce the need for heat dissipation components (e.g., heat sinks, fans, etc.) to cool the integrated circuits in which the CNNs are instantiated. Second, low power designs allow integrated circuits implementing convolutional networks to be placed at power-limited edge devices (e.g., cameras or other image sensing devices), rather than at core devices (e.g., servers, multi-core processors) where power is more readily available. One reason for performing data analysis operations at edge devices is that the bandwidth of sensor data received at edge devices can be very large. Thus, it can be more efficient to extract relevant information (e.g., recognized stop signs, stop lights, pedestrians, etc.) at the edge device, and only transmit the relevant information to the core device, rather than transmitting the entire sensor data stream to the core device. Efficiency can include reducing the wires needed to transmit sensor data from the edge device to the core device. In the context of a car, the edge devices can be embodied as various sensors in the car (e.g., image sensors in the front bumper, image sensors in the back bumper, LiDAR sensors mounted on the roof of the car, etc.), and the core device can be a multi-core processor located in the trunk of the car that draws power directly from the car’s battery.

[0026] To reduce the power consumption of a CNN, it is very important to reduce the power consumption of the circuit that performs the convolution operation, because the convolution operation is repeatedly performed by a CNN. The strategy used herein to reduce power consumption is to reduce the bit-width of the inputs to the mathematical operators (e.g., multipliers and adders) of the convolution circuit. Bit-width reduction results in quantized mathematical operations, with rounding and truncation being common examples of quantization. While the use of quantized inputs reduces power consumption (because there are fewer signal transitions between logical 0 and logical 1), quantization comes with the trade-off of loss of numerical precision. Thus, the specific design goal of reducing power consumption is to quantize the inputs to the mathematical operators, and at the same time, to minimize the loss of numerical precision.

[0027] The quantization scheme that meets this design goal in the context of convolution is dynamic data quantization. Consider the activation data at a certain time instant, which can be viewed as an X-by-Y array of values (or more generally, an X-by-Y-by-Z volume of values of three-dimensional activation data). While the dynamic range of the activation data across the entire X-by-Y array can be large, the dynamic range of a “local patch” of activation data that matches the kernel dimension is typically much smaller. It can be appreciated that the intensity levels and colors of an image are typically fairly constant locally (e.g., the color of the sky or the color of a house is uniform), except where there can be object edges. Thus, in the dynamic data quantization scheme, each “local patch” of activation values is quantized with a set number of bits to maximize the numerical precision of the (more limited) local dynamic range. It is apparent that the numerical precision of this scheme is improved over a quantization scheme that uses the same number of bits to represent numbers in the (larger) entire dynamic range of the X-by-Y array of values. The concept of dynamic data quantization will be better understood based on the examples provided in Figure 5A Figure 5B and Figure 6A Figure 6B .

[0028] One focus of the present invention is to apply the concept of dynamic data quantization to a specific hardware architecture that convolves multiple kernels with activation data. One example of a convolver array 118 and additional components that surround the convolver array 118 is shown as system 100 in Figure 1A At an abstract level, the activation data (measured from a single time instant) can be understood to flow “down” in the vertical direction, and the kernel data from the multiple kernels can be understood to flow “right” in the horizontal direction. It should be understood that in Figure 1A ​​In the context of this disclosure, references such as down, right, vertical, and horizontal are used for ease of explanation and can be changed depending on the physical location and orientation of the components in the hardware architecture. At each "intersection" of activation data and kernel data (i.e., the intersection of the data paths), there is a compute unit (e.g., a convolver) configured to compute the dot product of a "local block" of activation data and a point of the kernel. This is now explained in more detail below Figure 1A .

[0029] Figure 1A A hardware architecture of a system 100 configured to convolve a plurality of kernels (e.g., kernel A and kernel B) with activation data is shown, where the central task of the convolution involves computing the dot product of a quantized representation of an array of activation values (previously referred to as a "local block") and a quantized representation of one of the kernels. While two kernels are shown in Figure 1A for simplicity of explanation and description, it is understood that more kernels are typically used.

[0030] Activation data is read from memory element 102 and separated into columns of activation values by separator 106. For simplicity of explanation, in the example of Figure 1A , the activation data consists of only 3 columns of values, while more typically, the activation data can include hundreds of columns of values. The reader is expected to generalize the architecture of the simplified example of Figure 1A to the more typical column width of the activation data (i.e., by replicating the "middle" columns of system 100 - i.e., segments 108b, quantizer-calibration module 110b, compute unit 114b, compute unit 116b).

[0031] Suppose the activation data includes the following array of data A,

[0032]

[0033] Activation data can be provided to separator 106 row by row, and separator 106 can separate the elements of each row into individual activation values that are emitted to one of the segment components (108a, 108b, and 108c). More specifically, at one clock cycle (of a processor (not shown)), separator 106 can receive the last row of array A and emit activation value a 4,1 to segment 108a, activation value a 4,2 to segment 108b, and activation value a 4,3 to segment 108c. At the next clock cycle, separator 106 can receive the second to last row of array A and emit activation value a 3,1 to segment 108a, activation value a 3,2 to segment 108b, and activation value a 3,3 to segment 108c, and so on.Figure 1A In the context of this, the activation data can be interpreted as “flowing downwards” due to this staggered delivery of activation data to the convolutional array 118.

[0034] Each of the segmentation elements 108a, 108b, and 108c can be configured to output the three most recent activation values ​​(i.e., the activation values ​​most recently received by the segmentation element) during each clock cycle. For example, continuing the discussion of the same activation data array A, segment 108a can output activation value a during one clock cycle. 4,1 0, 0; Output the activation value a during the next clock cycle. 3,1 a 4,1 0; output the activation value a during the next clock cycle. 2,1 a 3,1 a 4,1 And so on (assuming the memory elements of the segmented elements are initialized to 0). In Figure 2 The diagram shows a possible implementation of one of the segmented elements. Segmented element 108 (which may represent any one of segmented elements 108a, 108b, 108c) may include three memory elements 202a, 202b, 202c connected in series (e.g., three D flip-flops). The output of each of the memory elements may be provided as the output of segmented element 108.

[0035] The outputs of segmented elements 108a, 108b, and 108c can be provided to one or more of quantizer-calibration modules 110a, 110b, and 110c. The input of quantizer-calibration module 110b is the more general case and will be discussed first, followed by the corresponding inputs of quantizer-calibration modules 110a and 110c (which are related to boundary cases). During a clock cycle, quantizer-calibration module 110b can receive a vector of three activation values ​​from segmented element 108a, a vector of three activation values ​​from segmented element 108b, and a vector of three activation values ​​from segmented element 106c. When considered in an aggregated manner, these three vectors can form a 3x3 array of activation values ​​(i.e., corresponding to the previously discussed “local blocks” of activation data with dimensions corresponding to the kernel dimension).

[0036] For example, continuing the discussion of the same active data array A, the quantizer-calibration module 110b can receive data during one clock cycle.

[0037]

[0038] Received during the next clock cycle

[0039]

[0040] Received during the next clock cycle

[0041]

[0042] And so on.

[0043] In contrast to quantizer-calibration module 110b, quantizer-calibration module 110a can only receive a vector of activation values from two segment elements (i.e., 108a and 108b), and the “left” three inputs of quantizer-calibration module 110a can be hardwired to zero in a zero padding scheme (or to another value in another padding scheme). Likewise, quantizer-calibration module 110c can only receive a vector of activation values from two segment elements (i.e., 108b and 108c), and the “right” three inputs can be hardwired to zero in a zero padding scheme (or to another value in another padding scheme).

[0044] The inputs and outputs of one of the quantizer-calibration modules are shown in more detail in block diagram 300. As previously mentioned, quantizer-calibration module 110 can receive a 3 by 3 array of activation values. As further specified in Figure 3 Figure 3 As further specified in, each of the activation values can consist of m bits. The output of quantizer-calibration module 110 can include nine n-bit mantissas and one exponent shared by the nine n-bit mantissas, where n is less than m. If it is not already apparent, n being less than m reflects the bit-width reduction scheme discussed previously for saving power. While there can be some overhead due to the introduction of the shared exponent, such overhead can be negligible when considering the greater power savings provided by the reduced bit-width of each of the nine activation values.

[0045] In later discussions, the quantized representation of a kernel will have a similar representation, so for the sake of clarity, the nine n-bit mantissas from quantizer-calibration module 110 will occasionally be referred to as nine n-bit (activation) mantissas, and the shared exponent will occasionally be referred to as the shared (activation) exponent. The meaning of the “shared” exponent and nine n-bit mantissas will be more clearly understood in the context of the examples of FIGS. 5 and 6 below. Together, the shared exponent and nine n-bit mantissas form a quantized representation of the 3 by 3 array of activation values (or “local block”).

[0046] Figure 4 A block diagram 400 showing additional details of quantizer-calibration module 110, which can include a bit range determination module 402 and a calibration module 404. Bit range determination module 402 can determine an appropriate quantization bit range based on the nine activation values. More specifically, the quantization bit range can be based on the maximum value in the 3 by 3 array of activation values, the median value in the 3 by 3 array of activation values, and / or the spatial center of the 3 by 3 array of activation values (i.e., the activation value from the following 3 by 3 array of activation values​ a 2,2 The value of quantization bits can be based on the maximum non-zero bit of the maximum value in the 3x3 activation array, the maximum non-zero bit of the median value in the 3x3 activation array, and / or the maximum non-zero bit of the spatial center of the 3x3 activation array.

[0047] In most implementations, the bit width of the quantization scheme is set, so the bit range determination module 402 can be used to fix the end bit position of the quantization bit range, where the start bit position in the quantization bit range is determined as the end bit position - preset bit width + 1. Alternatively, the bit range determination module 402 can be used to fix the start bit position of the quantization bit range, where the end bit position in the quantization bit range is determined as the start bit position + preset bit width - 1. The start bit position can be encoded with the "shared exponent," as shown below. Figure 6A The example below will make this clearer. Alternatively, the end bit position can encode the "shared exponent," as shown below. Figure 6B The examples in the text will make it clearer.

[0048] A bit range can be provided to a calibration module 404, which extracts bits from each of the m-bit values ​​according to the determined bit range. Figure 4 In the example, assuming the bit range includes n bits, the output of calibration module 404 includes 9 n-bit values ​​(although only one of the n-bit values ​​has been shown for ease of description). Intuitively, if the "local block" of activation values ​​includes one or more large values, then the n bits are the n most significant bits of the corresponding m-bit value; if the "local block" of activation values ​​includes only medium and small values, then the n bits are the n middle bits of the corresponding m-bit value; and if the "local block" of activation values ​​includes only small values, then the n bits are the n least significant bits of the corresponding m-bit value.

[0049] Figure 5A A mathematical example 500 of a quantizer-calibration operation according to one embodiment of the present invention is shown. Figure 5A In the example, the end position of the quantization bit range is determined based on the largest non-zero bit of the maximum value in the 3x3 activation array. The maximum value is 2. 12 +2 8 +2 5 Therefore, the end bit position is set to 12 (corresponding to 2). 12 The largest non-zero bit). In other words, in Figure 5A In the example, the end position of the quantization bit range is determined as the position of the largest non-zero bit among all nine activation values.

[0050] The example uses a fixed 8-bit width, so the starting bit position is 5 (i.e., 12 - 8 + 1), where bit position 5 is mathematically represented as 2. 5Further, the present quantization scheme employs truncation (as opposed to rounding), thus omitting any bits from bit positions 0-4 (without affecting the value of the bit at bit position 5). Based on such quantization bit range (i.e., retaining bits 5 through 12), the activation value 2 11 +2 8 +2 2 may be quantized to 2 11 +2 8 ; the activation value 2 12 +2 8 +2 5 may remain unchanged, as all of its bits are within the quantization bit range; and so on.

[0051] Following the quantization operation in Figure 5A , the exponent represented by the starting bit position of the quantization bit range can be factored out of each of the quantized values (in a circuit-level implementation, this can correspond to a "bit align" or "bit shift" operation). In the example of Figure 5A , the quantized value 2 11 +2 8 may be factored as 2 5 (2 6 +2 3 ); the quantized value 2 12 +2 8 +2 5 may be factored as 2 5 (2 7 +2 3 +2 0 ); and so on. Since 2 5 is common to all of the factored expressions, 2 5 (or just "5") can be mathematically referred to as the shared exponent.

[0052] To summarize the quantization-align operation using the example provided in Figure 5A , the following nine 13-bit activation values,

[0053] 2 11 +2 8 +2 2

[0054] 2 12 +2 8 +2 5

[0055] 2 10 +2 5 +2 1

[0056] 2 9 +2 7 +2 6

[0057] 2 6 +2 5

[0058] 2 3 +2 0

[0059] 2 12 +2 3

[0060] 2 7 +2 5 +2 0

[0061] 2 10 +2 8

[0062] may be converted by the quantization-calibration module 110 into a shared (active) exponent of 2 5 and the following nine 8-bit (active) mantissa values,

[0063] 2 6 +2 3

[0064] 2 7 +2 3 +2 0

[0065] 2 5 +2 0

[0066] 2 4 +2 2 +2 1

[0067] 2 1 +2 0 0

[0069] 2 7

[0070] 2 2 +2 0

[0071] 2 5 +2 3

[0072] Figure 5B The mathematical example 550 presented in FIG. 6A shows that there is some flexibility in the choice of the shared exponent. Whereas in Figure 5A the shared exponent is chosen to be the exponent that represents the starting position of the quantization range (i.e., 2 5 ), Figure 5B the shared exponent in FIG. 6B is chosen to be the exponent that represents the ending position of the quantization range (i.e., 212 As shown below Figure 6A and Figure 6B It will be even more apparent that the choice of the shared exponent will affect the mathematical value represented by each bit in the mantissa. At a high level, it is understandable that any choice of the shared exponent is satisfactory as long as the internal "statistics" keep track of the choice of the shared exponent (whether it is the start bit position or the end bit position).

[0073] Figure 6A Example 600 at the bit level (or circuit level) shows that a quantization step can be performed by eliminating existing bits 0-4, and a calibration step can be performed by shifting existing bits 5-12 to new bit positions 0-7, respectively. If not immediately obvious, the bit values ​​in the table correspond numerically to... Figure 5A The leftmost column shows nine 13-bit activation values, and the table below corresponds numerically to the values. Figure 5A The rightmost column shows nine 8-digit last digits. Although in... Figure 6A It is not labeled, but "internal statistics" track each of the 8-digit last digits (see [link]). Figure 6A (See the table below), bit position 0 corresponds to exponent 2. 0 Position 1 corresponds to exponent 2. 1 ... and position 7 corresponds to exponent 2. 7 (i.e., comparison) Figure 5A "After the calibration column" and Figure 6A (See the table below). In other words, for this particular choice of the shared exponent, when each of the 13-bit values ​​is converted to its corresponding 8-bit mantissa, the mathematical value represented by each of the bits remains unchanged. (See table below). Figure 6B As the lieutenant general described, this mapping of the position of the 8-bit mantissa may differ depending on the choice of the shared exponent.

[0074] Additionally, it should be noted that Figure 6A The number of bits to eliminate (i.e., the number of bits to shift) shown depends on the specific activation values. Due to the dynamic data quantization scheme, the number of bits to shift is expected to be different for another set of nine activation values.

[0075] Figure 6B Bit-level example 650 and Figure 6A The examples in [the document] are essentially similar, except that the shared exponent is chosen as the end position of the quantization range. For example, from [the document]... Figure 6B Obviously, Figure 6A and Figure 6B The only difference between them is the choice of the shared index. Figure 6B The meaning of the shared exponent selection is that, for each of the 8 tail digits (comparison) Figure 5B "After the calibration column" and Figure 6Bbit position 0 corresponds to the exponent 2 -7 bit position 1 corresponds to the exponent 2 -6 and bit position 7 corresponds to the exponent 2 0 Again, this mapping is taken into account in the "internal statistics." In the example of Figure 6B it is apparent that the bit-to-value mapping for 13-bit values is different from the bit-to-value mapping for 8-bit mantissa values.

[0076] Figure 7 A block diagram 700 of a quantizer-calibration module 112, which can represent any quantizer-calibration module (e.g., 112a, 112b) used to convert a 3x3 kernel into its quantized representation, is shown. The input to the quantizer-calibration module 112 can include nine p-bit values, and the output of the quantizer-calibration module 112 can include nine q-bit (kernel) mantissa values (where q is less than p) and one (kernel) exponent shared by the mantissa values, which combination forms the quantized representation of the 3x3 kernel. The internal details and operations of the quantizer-calibration module 112 can be similar to the quantizer-calibration module 110, and therefore additional details are not provided in connection with the quantizer-calibration module 112 shown in Figure 7 for the sake of simplicity of explanation.

[0077] Figure 8A The shorthand notation used in Figure 1A is expanded in Figure 1A (and also replicated in the circuit diagram 802 in Figure 8A ). The computation unit 114a receives, as input, ten "vertical" input signals (corresponding to the shared exponent and the nine n-bit mantissa values from the quantizer-calibration module 110a). Figure 1A The notation in may initially give the impression that the computation unit 114a generates ten output signals, which are then provided to the computation unit 116a, which outputs another ten output signals, but this is not the case.

[0078] Figure 8A The full representation (i.e., the full electrical interconnection) of the shorthand notation is shown in the circuit diagram 804 of Figure 1A . In fact, the ten output signals from the quantizer-calibration module 110a are provided to each of the computation units (i.e., 114a and 116a), which are spatially located in the same "column" as the quantizer-calibration module 110a. In the context of Figure 1A , the outputs of the quantizer-calibration module 110a are provided to the computation unit 114a, the computation unit 116a, and other computation units in the "first column" (not shown).(Not shown in the image); the output of the quantizer-calibration module 110b is provided to the calculation unit 114b, the calculation unit 116b, and other calculation units in the "second column" (not shown in the image); Figure 1A (not shown in the image); and the output of the quantizer-calibration module 110c is provided to the calculation unit 114c, the calculation unit 116c, and other calculation units in the "third column" (not shown in the image); Figure 1A (Not shown in the image).

[0079] It should be noted that propagating the output of a quantizer-calibration module to multiple computational units located in the same column is a circuit-level optimization employed in one embodiment of the present invention. This design is far more efficient (in terms of reducing power consumption and chip footprint) than having the quantizer-calibration module already replicated for each of the computational units in the same column (i.e., one instance of the quantizer-calibration module 110a for computational unit 114a, another instance of the quantizer-calibration module 110a for computational unit 116b, and so on).

[0080] Figure 8B Further expansion Figure 1A Similar shorthand symbols are used. Figure 1A In China (and also in) Figure 8B (Copied from circuit diagram 806) The calculation unit 114a also receives ten “level” input signals (corresponding to the shared exponent and nine q-bit tail values ​​from the quantizer-calibration module 112a) as inputs. Figure 1A The symbols in the diagram may initially give the impression that computing unit 114a generates ten output signals and then provides the output signals to computing unit 114b; and computing unit 114b generates ten output signals and then provides the output signals to computing unit 114c, but this is not the case.

[0081] The complete representation of shorthand symbols (i.e., the complete electrical interconnection) is shown in Figure 8B In the circuit diagram 808, ten output signals from the quantizer-calibration module 112a are actually provided to each of the computing units (i.e., 114a, 114b, and 114c), which are spatially located in the same "row" as the quantizer-calibration module 112a. Figure 1A In this context, the output of quantizer-calibration module 112a is provided to computation units 114a, 114b, and 114c; the output of quantizer-calibration module 112b is provided to computation units 116a, 116b, and 116c; and for Figure 1A The other rows not shown are similar.

[0082] Note that the propagation of the output of one quantizer-calibration to multiple compute units located in the same row is also a circuit-level optimization employed in one embodiment of the invention. Such a design is much more efficient (in terms of reduced power consumption and chip real estate) than a case where the quantizer-calibration module has been replicated for each of the compute units in the same row (i.e., one instance of quantizer-calibration module 112a for compute unit 114a, another instance of quantizer-calibration module 112a for compute unit 114b, another instance of quantizer-calibration module 112a for compute unit 114c, and so on).

[0083] Figure 9 a block diagram 900 showing additional details of the inputs and outputs of the compute units shown in Figure 1A and Figure 1B a block diagram 900 showing additional details of the inputs and outputs of the compute units shown in Figure 9 As shown, compute unit 114 can receive as inputs a quantized representation of a 3x3 array of activation values (including a shared activation exponent and 9 n-bit activation mantissa values) and a quantized representation of a 3x3 kernel (including a shared kernel exponent and 9 q-bit kernel mantissa values). An adder of compute unit 114 can compute the sum of the shared activation exponent and the shared kernel exponent. In addition, a multiplier and an adder of compute unit 114 can compute the dot product of the 9 n-bit activation mantissa values and the 9 q-bit kernel mantissa values.

[0084] Figure 10A a mathematical example 1000 showing the computation of the dot product of a quantized representation of an array of activation values and a quantized representation of a convolution kernel (where the shared exponent is chosen to be the starting bit position of the respective quantization ranges of the kernel and the activation values). In the mathematical example, the quantized representation of the array of activation values includes

[0085] 2 6 +2 3

[0086] 2 7 +2 3 +2 0

[0087] 2 5 +2 0

[0088] 2 4 +2 2 +2 1

[0089] 2 1 +2 0 0

[0091] 2 7

[0092] 22 +2 0

[0093] 2 5 +2 3

[0094] where the shared exponent is 2 5 . The quantized representation of the 3x3 kernel includes

[0095] 2 3 +2 1 +2 0 0

[0097] 2 1

[0098] 2 5 +2 2

[0099] 2 3 +2 0

[0100] 2 3 0

[0102] 2 1

[0103] 2 3 +2 2

[0104] where the shared exponent is 2 2 .

[0105] The computing unit 114 computes the dot product of the two quantized representations as follows:

[0106] (2 3 +2 1 +2 0 )(2 6 +2 3 )+

[0107] (0)(2 7 +2 3 +2 0 )+

[0108] (2 1 )(2 5 +2 0 )+

[0109] (2 5 +2 2 )(2 4 +2 2 +2 1 )+

[0110] (2 3 +2 0 )(2 1 +2 0 )+

[0111] (2 3 )(0)+

[0112] (0)(2 7 )+

[0113] (2 1 )(2 2 +2 0 )+

[0114] (2 3 +2 2 )(2 5 +2 3 )

[0115] The sum of the shared exponent and the shared mantissa exponent is calculated as follows: 2 + 5 (indicating exponent 2 2+5 ).

[0116] Figure 10B A mathematical example 1050 of calculating the dot product of a quantized representation of an activation value array and a quantized representation of a convolution kernel is shown (where the shared exponents are chosen to be the end bit positions of the respective quantization ranges of the kernel and the activation values). In the mathematical example, the quantized representation of the activation value array includes

[0117] 2 -1 +2 -4

[0118] 2 0 +2 -4 +2 -7

[0119] 2 -2 +2 -7

[0120] 2 -3 +2 -5 +2 -6

[0121] 2 -6 +2 -7 0

[0123] 2 0

[0124] 2 -5 +2 -7

[0125] 2 -2 +2 -4

[0126] where the shared exponent is 2 12 A quantized representation of the 3x3 kernel includes

[0127] 2 -2 +2 -4 +2 -5 0

[0129] 2 -4

[0130] 2 0 +2 -3

[0131] 2 -2 +2 -5

[0132] 2 -2 0

[0134] 2 -4

[0135] 2 -2 +2 -3

[0136] where the shared exponent is 2 7 .

[0137] The computing unit 114 computes the dot product of the two quantized representations as follows:

[0138] (2 -2 +2 -4 +2 -5 )(2 -1 +2 -4 )+

[0139] (0)(2 0 +2 -4 +2 -7 )+

[0140] (2 -4 )(2 -2 +2 -7 )+

[0141] (2 0 +2 -3 )(2 -3 +2 -5 +2 -6 )+

[0142] (2 -2 +2 -5 )(2 -6 +2 -7 )+

[0143] (2 -2 )(0)+

[0144] (0)(2 0 )+

[0145] (2 -4 )(2 -5 +2 -7 )+

[0146] (2 -2 +2 -3 )(2 -2 +2 -4 )

[0147] The sum of the shared activation exponent and the shared mantissa exponent is computed as follows: 7 + 12 (representing exponent 2 7+12 ).

[0148] In the system 100 shown in Figure 1A , the quantized representation of the kernel can be pre-computed before the model application (inference) phase and stored in the memory element 104. During the convolution computation of the model application phase, the pre-computed quantized representation of the kernel can be received by the computation unit directly from the memory element 104. Using the pre-computed quantized representation of the kernel during the model application phase allows the circuit chip to concentrate its resources on other computations (e.g., convolution operations).

[0149] Conversely, when the parameters of the kernel are adjusted and the pre-computation of the quantized representation of the kernel is not possible, Figure 1B The hardware architecture of the system 150 shown in

[0150] While not discussed above, note that the dot product computed by each compute unit can be temporarily stored in the respective compute unit and accumulated with the dot products from other "channels" of activation data. When the activation data is three-dimensional, there can be activation data with multiple channels (e.g., one channel corresponding to data sensed by a red light sensor, one channel corresponding to data sensed by a green light sensor, and one channel corresponding to data sensed by a blue light sensor). In such scenarios, the kernel can have dimensions 3x3x3, including a 3x3 red kernel, a 3x3 green kernel, and a 3x3 blue kernel. In a convolution operation, the dot product of the 3x3 red kernel with a block of the corresponding activation data can be added to the dot product of the 3x3 green kernel with the same block of activation data, and further added to the dot product of the 3x3 blue kernel with the same block.

[0151] While in the above examples, dynamic data quantization is applied to both the activation data and the kernel data, this need not be true in all implementations. In alternative implementations, dynamic data quantization can be applied only to the activation data, and the kernel data can be unquantized or statically quantized (i.e., in a data-independent manner).

[0152] Accordingly, a low-power hardware architecture for convolutional neural networks has been described. It is to be understood that the above description is intended to be illustrative, and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reviewing the above description. The scope of the application should, therefore, be determined not with reference to the above description, but instead with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. A system (100, 150) comprising: a first computing unit (114a) configured to: receive a first plurality of quantized activation values, the first plurality of quantized activation values represented by a first plurality of activation mantissa values and by a first activation exponent shared by the first plurality of activation mantissa values, wherein the first plurality of quantized activation values is a quantized representation of a first matrix having values receive a first quantized convolution kernel represented by a first plurality of kernel mantissa values and by a first kernel exponent shared by the first plurality of kernel mantissa values; compute a first dot product of the first plurality of activation mantissa values and the first plurality of kernel mantissa values; and compute a first sum of the shared first activation exponent and the shared first kernel exponent; a second computing unit (116a) configured to: receive the first plurality of quantized activation values; receive a second quantized convolution kernel represented by a second plurality of kernel mantissa values and by a second kernel exponent shared by the second plurality of kernel mantissa values; compute a second dot product of the first plurality of activation mantissa values and the second plurality of kernel mantissa values; and compute a second sum of the shared first activation exponent and the shared second kernel exponent; a third computing unit (114b) configured to: receive a second plurality of quantized activation values, the second plurality of quantized activation values represented by a second plurality of activation mantissa values and by a second activation exponent shared by the second plurality of activation mantissa values, wherein the second plurality of quantized activation values is a quantized representation of a second matrix having values wherein six values in the first matrix are the same as six values in the second matrix; receive the first quantized convolution kernel; compute a third dot product of the second plurality of activation mantissa values and the first plurality of kernel mantissa values; and compute a third sum of the shared second activation exponent and the shared first kernel exponent; and a fourth computing unit (116b) configured to: receive the second plurality of quantized activation values; receive the second quantized convolution kernel; compute a fourth dot product of the second plurality of activation mantissa values and the second plurality of kernel mantissa values; and compute a fourth sum of the shared second activation exponent and the shared second kernel exponent.

2. The system (100) of claim 1, wherein the first quantized convolution kernel is received from a memory element (104) during a model application phase.

3. The system (150) of claim 1, wherein the first quantized convolution kernel is received from a quantizer-calibration module (112a) during a model application phase.

4. The system (100, 150) of claim 1, wherein the first quantized convolution kernel is a 3 by 3 kernel. ​ ​

Images