Energy-efficient computational near-memory binary neural network circuits

Abstract

To provide a compute-near-memory binary neural network accelerator with digital circuits that achieves energy efficiency surpassing that with analog circuits.SOLUTION: In an image classifier 100, a neural network accelerator comprises a binary convolutional neural network comprising: a plurality of execution units; and a near memory latch array comprising a plurality of sets of latches finely interleaved with the plurality of execution units to reduce energy consumption and enable a high-bandwidth memory access to the plurality of execution units. A set of latches is communicatively coupled to each of the execution units to store a result of a matrix multiplication in the execution unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to neural networks, and more particularly to optimizing the energy efficiency of binary neural network circuits. [Background technology]

[0002] Deep learning is an approach to artificial intelligence that uses neural networks. Neural networks have multiple interconnected processing nodes that provide a means to perform machine learning by analyzing multiple examples. For example, an object recognition system may analyze thousands of images of objects (e.g., trucks) so that it can learn to find visual patterns to identify the objects in the images.

[0003] A neural network may be organized into layers of nodes, and data may travel through the neural network, with each node being coupled to several nodes below it in the layer and several nodes above it in the layer, and the node receiving data from several nodes below it in the layer and sending data to several nodes above it in the layer.

[0004] A node assigns numerical values ​​known as "weights" to connections in a neural network. When a neural network is active, a node receives data items (numbers) from the node's connections and multiplies the data items by their associated weights. The node then adds the resulting products together to produce a single result (number). The node passes the result (the sum of multiple products) to an activation function (a nonlinear function) and forwards the result to the node's output connections.

[0005] In deep learning systems, neural network models are stored in memory, and computational logic in the processor performs multiply-accumulate (MAC) calculations on parameters (e.g., weights) stored in memory. Data transfer between memory and the processor is an energy-intensive process that can consume up to 90% of the power of a machine learning workload.

[0006] Software in a conventional computing architecture, known as the von Neumann architecture, requires data transfer between separate computational logic in a processor (e.g., a central processing unit (CPU)) and memory. The von Neumann architecture, which requires large amounts of data transfer between the processor and memory, can be a bottleneck for machine learning applications. In addition, data transfer between memory and the processor is an energy-intensive process. [Brief description of the drawings]

[0007] Features of embodiments of the claimed invention will become apparent upon reading the following detailed description and upon reference to the accompanying drawings, in which like reference numerals refer to like elements and in which: [Figure 1] FIG. 2 is a block diagram of an image classifier. [Diagram 2] FIG. 2 is a block diagram of a binary neural network accelerator within the image classifier shown in FIG. 1, which leverages ComputeNear Memory (CNM) technology to maximize energy efficiency. [Diagram 3] FIG. 3 is a block diagram illustrating a portion of one wide vector dot product execution unit shown in FIG. [Figure 4A] 2 illustrates an example of stride operation in the image classifier shown in FIG. 1 . [Figure 4B] 2 illustrates an example of stride operation in the image classifier shown in FIG. 1 . [Figure 4C] 2 illustrates an example of stride operation in the image classifier shown in FIG. 1 . [Figure 4D] 2 illustrates an example of stride operation in the image classifier shown in FIG. 1 . [Figure 4E] 2 illustrates an example of stride operation in the image classifier shown in FIG. 1 . [Diagram 5] 1 shows a memory map of memory banks to allow single cycle access to a 2×2 window. [Figure 6A] 2 illustrates an example of a stride operation using a snake pattern in the image classifier shown in FIG. 1 . [Figure 6B] 2 illustrates an example of a stride operation using a snake pattern in the image classifier shown in FIG. 1 . [Figure 6C] 2 illustrates an example of a stride operation using a snake pattern in the image classifier shown in FIG. 1 . [Figure 6D] 2 illustrates an example of a stride operation using a snake pattern in the image classifier shown in FIG. 1 . [Figure 6E] 2 illustrates an example of a stride operation using a snake pattern in the image classifier shown in FIG. 1 . [Figure 7] FIG. 1 is a block diagram of an embodiment of a binary neural network accelerator with high parallelism. [Figure 8] 1 is a flow graph illustrating an embodiment of a method for providing an energy-efficient binary neural network circuit. [Figure 9] FIG. 1 is a block diagram of an embodiment of a computer system having an image classifier. The following detailed description proceeds with reference to exemplary embodiments of the claimed invention, to which many alternatives, modifications and variations will be apparent to those skilled in the art. It is therefore intended that the claimed invention be broadly construed and defined as set forth in the appended claims. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Image classification is the process of assigning a class to an image, for example, assigning a class of car to an image or the probability that an image is a car. Convolutional Neural Networks (CNNs) are fundamental visual processing building blocks for image and video processing, which are essential for applications ranging from self-driving autonomous vehicles to security and surveillance.

[0009] A convolutional neural network is a deep learning model that can be used in image classification. A convolutional neural network has an input layer, an output layer, and a hidden layer. The hidden layer may include a convolutional layer, a regularized linear activation layer, a pooling layer, and a fully connected layer.

[0010] Convolutional neural networks use convolutional and pooling layers for feature abstraction from the input image. These layers of convolutional neural networks have neurons organized in three dimensions: width, height, and channels. Processing convolutional layers is usually based on performing convolution operations with N×N×C weight filters (also called kernels). The weight filters in a convolutional neural network layer have three dimensions: width (N), height (N), and channels (C). For example, a color input image has three channels (red, green, and blue). A particular layer may also contain multiple weight filters. The number of channels at the output of a convolutional neural network layer is equal to the number of filters.

[0011] The weighting filter is shifted by a stride (one or more pixels) over the input image (also called a layer) to provide an output image: at each position, the value of the filter is multiplied by the value in a subregion in the input image.

[0012] One type of pooling layer used by convolutional neural networks is max pooling. Max pooling is a technique used to reduce the dimensionality of an input image by selecting the maximum pixel value in a subregion of the input image. A filter of size m x m pixels is shifted by a stride. At each position, the maximum pixel value in each subregion where the filter is convolved in the input image is output.

[0013] The values ​​in the sub-regions in the input image (also called activations) and the values ​​of the filters of the convolutional neural network (also called weights) can be floating-point values, fixed-point values, or multi-bit integers. The activations and weights are binarized in a binary convolutional neural network (also called a binary neural network or binarized neural network). The activations and weights may be one of two binary values: +1, -1, where -1 is encoded with 0 and +1 with 1. Performing an exclusive-negated-or (XNOR) operation on the binary encoding (0,1) is equivalent to performing a multiplication on the binary value (-1,1).

[0014] Binary neural networks reduce multipliers to a single gate operation for efficient computation, greatly lowering memory bandwidth requirements. These memory and computation advantages allow for much greater energy efficiency when compared to floating-point, fixed-point and multi-bit integer based neural networks.

[0015] With binary neural networks, the cost of memory access compared to computation is greatly increased due to the lower computational cost. Based on static random access memory (SRAM) or register file access without data reuse, for a target energy efficiency of 1000 Tera Operations per Second (TOP) per Watt, memory access consumes 10 to 100 times more energy than multiplication. Similarly, the cost of data movement is also large when compared to the cost of computation. For example, the energy cost of a multiplication operation is roughly equal to the cost of moving one bit 25 micrometers (μm). The cost of memory access and data movement is as important as optimizing the energy efficiency of the execution unit.

[0016] A binary neural network accelerator may use analog or digital circuits for computation. The analog circuit may include switched capacitors to perform analog additions and accumulate the results of the binary multiplications, which are passed to a comparator to digitize the analog values ​​into 1-bit digital values.

[0017] However, analog circuits are more susceptible to process, voltage, and temperature variations when compared to digital circuits. Also, analog circuits are usually difficult to scale with process technology. Analog circuits are more vulnerable to circuit noise and can produce nondeterministic outputs due to noise sources and low noise margins. Also, analog circuits cannot tolerate arbitrarily large dot product sizes due to precision limitations and device mismatches caused by circuit noise. A dot product is a scalar value that is the result of an operation between two equal-length sequences of numbers, called vectors.

[0018] In one embodiment, a compute near memory binary neural network accelerator with digital circuits achieves energy efficiency comparable to or better than a compute near memory binary neural network accelerator with analog circuits. Also, the compute near memory binary neural network accelerator with digital circuits is more process scalable, robust to process, voltage, and temperature (PVT) variations, and immune to circuit noise. The compute near memory binary neural network accelerator with digital circuits uses multiple circuit design techniques for energy-efficient computation, such as compute near memory design principles, multiple voltage domains with clock skew tolerant pipelines, near threshold voltage operation, lightweight pipelines for maximum near threshold voltage energy efficiency, and energy-balanced memory / interconnect / compute with highly parallel execution units.

[0019] Various embodiments and aspects of the present invention will now be described with reference to the following detailed description. The accompanying drawings illustrate various embodiments. The following description and drawings are illustrative of the present invention and should not be construed as limiting the present invention. Numerous specific details have been set forth in order to provide a thorough understanding of various embodiments of the present invention. However, in certain instances, well-known or conventional details are not described in order to briefly describe the embodiments of the present invention.

[0020] References herein to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described with respect to that embodiment may be included in at least one embodiment of the invention. The appearances of the phrase "in one embodiment" in various places in this specification do not necessarily all refer to the same embodiment.

[0021] 1 is a block diagram of an image classifier 100 that includes a binary neural network accelerator 102. The binary neural network accelerator 102 includes a static random access memory (SRAM) 110 that stores an input image 106, and a controller 118 that moves a portion of the input image 106 from the static random access memory 110 to an interleaved memory compute 112 for processing. The binary neural network accelerator 102 performs feature abstraction from the input image 106 using a filter 104 to provide a classification 108 of the input image 106. An image classification application may be implemented in the controller 118 as programmed instructions.

[0022] ComputeNeer memory reduces memory access and data movement costs by tightly interleaving memory with computation units. The interleaved memory compute structure increases the effective bandwidth between the computation units and memory and reduces data movement. Typically, ComputeNeer memory technology involves decomposing a large static random access memory (SRAM) array into smaller memory banks with uniformly distributed computation blocks. However, in binary neural networks, the multipliers of binary neural networks are implemented as exclusive-negated-or (XNOR), so the energy consumed for data movement is large compared to the computation cost.

[0023] FIG. 2 is a block diagram of a binary neural network accelerator 102 in the image classifier 100 of FIG. 1, which leverages ComputeNear Memory (CNM) technology to maximize energy efficiency.

[0024] The binary neural network accelerator 102 is implemented as a two-level hierarchy of static random access memory 110 and controller 118 surrounded by an interleaved memory compute 112, which includes a wide vector dot product execution unit 202 and an interleaved latch-based compute near memory array 204. The latch-based compute near memory array 204 may also be referred to as a near memory latch array. A set of weights is stored in vector latches 208, 210 located in the dot product execution unit 202. The set of weights may be reused multiple times over the course of a convolution operation. Storing two sets of weights in the vector latches 208, 210 allows for twice as much reuse of input data compared to a single set of weights, and also reduces input energy / switch activity by twice as much.

[0025] To reduce energy consumption for data movement and to enable high bandwidth memory access to multiple execution units, the memory is implemented as a latch-based ComputeNear memory array 204 rather than a static random access memory. The latch-based ComputeNear memory array 204, which contains a set of bit latches 214, 216, is interleaved with the computational units, including the wide vector dot product execution unit 202. A vector dot product is the dot product of the coordinates of two vectors.

[0026] In one embodiment, the SRAM 110 has 129 kilobytes (KB), each set of vector latches 208, 210 and bit latches 214, 216 has 1024 bits, there are 128 dot product execution units 202, and each wide vector dot product execution unit 202 has 1024 components. The number of operations (additions or multiplications) performed per cycle with N dot product execution units of M width with L sets of latches is 2*N*M. The number of bits read from the static random access memory 110 (without stride data reuse) per cycle is M / L. The number of operations performed per bit is 2*N*L. In this embodiment (N=128, M=1024, L=2), 512 bits are read from the static random access memory 110 per cycle, 256K operations per cycle, and the number of operations per bit is 512. In general, the choices of N, M, and L for a particular system should be chosen such that for an energy-efficient design, the number of operations per bit is at least 128 (2*N*L>128). To optimize energy efficiency by amortizing the costs of memory access and data movement over many binary neural network operations, the number of operations performed per bit read from memory is 128 or greater.

[0027] In one embodiment, two sets of 1024-bit latches 214, 216 are assigned to each wide vector dot product execution unit 202. Two sets of 1024-bit latches 214, 216 are communicatively coupled to each of the wide vector dot product execution units 202 to store the results of the computations in the wide vector dot product execution units 202. The output of each set of 1024-bit latches 214, 216 is input to a 2:1 multiplexer 218. The multiplexer selects the output of one 1024-bit latch in each set of 1024-bit latches to input to the next computation unit, including the wide vector dot product execution unit 202. The two sets of 1024-bit latches 214, 216 balance the tradeoff between reduced activation memory energy / bandwidth and the energy cost of adding the multiplexer 218 to the data path. The energy consumed by the multiplexer 218 is equivalent to a binary multiplication operation. In other embodiments, there may be more than two sets of 1024-bit latches 214, 216 and more than one multiplexer 218.

[0028] Area-efficient design is also essential for energy efficiency. Reducing the area reduces the distance of data movement between the latch set in the latch-based ComputeNear memory array 204 and the wide vector dot product execution unit 202, which has a higher computation density (Tera Operations Per Second (TOPS) / mm2) than the outer product execution unit. To further improve the area efficiency of the local weight memory storage, vector latches are used instead of standard latches. Memory footprint is also reduced by reusing local activation memory to store two intermediate inputs and output activations of a single layer. After the computation of a layer is completed, the previous input activations may be released and reused to store the incoming output activations of a new layer. This recycling of activation memory results in a reduction of activation memory proportional to the number of layers in a binary neural network, greatly reducing the size and cost of activation memory accesses.

[0029] Data movement energy is reduced by reducing the average data bandwidth between memory and execution units. A highly parallel design with wide vector dot product execution units and two sets of local weights per execution unit has a high reuse of inputs and weights, which reduces the required bandwidth between memory and execution units. To reduce the required output bandwidth by using accumulation, very wide execution units are incorporated, reducing the total number of outputs.

[0030] Figure 3 is a block diagram illustrating an embodiment of one wide vector dot product execution unit 202 shown in Figure 2. A dot product is a method of multiplying two vectors together, where the result of the multiplication is a scalar. The scalar result is the dot product of the coordinates of the two vectors.

[0031] In the example shown in Figure 3, a 1024-bit data vector X and a 1024-bit weight vector W are multiplied by performing an XOR operation in NEXOR gate 302 and reducing the output result of the XOR operation in reduction circuit 304. The reduced result of the XOR operation from reduction circuit 304 is stored in accumulation circuit 306. In one embodiment, the scalar result is 10 bits and an additional bit (the sign bit) is used to store the sign of the 10-bit scalar result. The wide vector dot product execution unit 202 shown in Figure 3 amortizes the cost of the 16-bit accumulation over many operations while also minimizing data movement per operation to provide the scalar result.

[0032] The cross product of two coordinate vectors is a matrix. If the two vectors have dimensions n and m, the cross product is an n by m matrix. Compared to the cross product execution unit, the wide vector dot product execution unit 202 is about four times more energy efficient and about five times more area efficient.

[0033] Data reuse and low data movement are essential for energy-efficient binary neural network accelerators. High parallelism is needed to create a system with a balanced energy between computation and memory access. Data reuse is a way of describing how memory access is shared among multiple operations to create a balanced system. With binary neural networks, the cost of memory access compared to computation is greatly increased due to the lower computation cost.

[0034] Figures 4A-E show an example of stride operation in the image classifier 100 shown in Figure 1. In the illustrated example, the stride is 1 and a 2x2 convolution kernel is traversed with the stride from left to right across the 3x3 input image using a raster scan memory access pattern. Intersection of the weight kernel with the image represents a single vector dot product with a scalar output.

[0035] Referring to Figure 4A, a 2x2 convolution kernel is placed on input data 402, 404, 408, and 410 in a 3x3 input image. The result of the dot product of the 2x2 kernel with the input data 402, 404, 408, and 410 is represented as a scalar output 420 in Figure 4E.

[0036] Referring to Figure 4B, a 2x2 convolution kernel is placed on input data 404, 406, 410 and 412 in a 3x3 input image. The result of the dot product of the 2x2 kernel with the input data 404, 406, 410 and 412 is represented as a scalar output 422 in Figure 4E. When the stride is 1, two successive moving window operations share 50% of the same input data. Referring to Figures 4A and 4B, both include input data 404 and input data 410.

[0037] Referring to Figure 4C, a 2x2 convolution kernel is placed on the input data 408, 410, 414, and 416 in the 3x3 input image. The result of the dot product of the 2x2 kernel with the input data 408, 410, 414, and 416 is represented as a scalar output 424 in Figure 4E.

[0038] Referring to Figure 4D, a 2x2 convolution kernel is placed on input data 410, 412, 416, and 418 in a 3x3 input image. The result of the dot product of the 2x2 kernel with input data 410, 412, 416, and 418 is represented as a scalar output 426 in Figure 4E. When the stride is 1, two consecutive moving window operations share 50% of the same input data. Referring to Figures 4C and 4D, both include input data 410 and input data 416. To reduce input memory accesses, the binary neural network accelerator exploits data reuse in strided operations.

[0039] Figure 5 shows a memory map of memory banks to allow single cycle access to a 2x2 window. As explained with respect to the example of Figure 4, when the stride is 1, two consecutive moving window operations share 50% of the same input data.

[0040] To maximize data reuse of the input data, the input data may be stored in multiple memory banks, with each of the four inputs in a 2×2 window being mapped to a different memory bank. In the example of width (6)×height (4) input data, which may also be referred to as a checkerboard memory map, shown in FIG. 5, input data 502, 506, 510, 526, 530, and 534 are mapped to a first memory bank, input data 504, 508, 512, 528, 532, and 536 are mapped to a second memory bank, input data 514, 518, 522, 538, 542, and 546 are mapped to a third memory bank, and input data 516, 520, 524, 540, 544, and 548 are mapped to a fourth memory bank.

[0041] With this input storage pattern, any 2×2 window of the 6×4 input data shown in FIG. 5 can be accessed in a single memory read cycle, with each of the four input data being accessed from its respective memory bank. However, data can be reused from the previous cycle, so each successive cycle only needs to read from two memory banks. For example, the 2×2 window of input data 502, 504, 514, and 516 may be accessed in a single memory cycle. The checkerboard memory map does not require additional registers to achieve strided data reuse.

[0042] 6A-E show an example of a snake pattern stride operation in the image classifier 100 shown in FIG. 1. To reduce input memory accesses, the binary neural network accelerator exploits data reuse in the stride operation. In the illustrated example, the stride is 1 and a 2×2 convolution kernel is moved over the input image with strides using a snake pattern of memory accesses. A portion of the input image with width 6 and height 4 is shown in FIG. 6A-6D. The intersection of the weight kernel with the input image represents a single vector dot product with a scalar output.

[0043] Referring to Figure 6A, a 2x2 convolution kernel is placed on input data 602, 604, 614, and 616 in an input image. The result of the dot product of the 2x2 kernel with input data 602, 604, 614, and 616 is represented as a scalar output 650 in Figure 6E.

[0044] Referring to Figure 6B, a 2x2 convolution kernel is moved down by one stride and placed on top of input data 614, 616, 626 and 628 in the input image. The result of the dot product of the 2x2 kernel with input data 614, 616, 626 and 628 is represented as a scalar output 652 in Figure 6E. When the stride is 1, two successive moving window operations share 50% of the same input data. Referring to Figures 6A and 6B, both include input data 614 and input data 616.

[0045] Referring to Figure 6C, a 2x2 convolution kernel is placed on the input data 616, 618, 628, and 630 in the input image. The result of the dot product of the 2x2 kernel with the input data 616, 618, 628, and 630 is represented as a scalar output 654 in Figure 6E.

[0046] Referring to Figure 6D, a 2x2 convolution kernel is placed over input data 604, 606, 616, and 618 in the input image. The result of the dot product of the 2x2 kernel with input data 604, 606, 616, and 618 is represented as a scalar output 656 in Figure 6E. When the stride is 1, two successive moving window operations share 50% of the same input data. Referring to Figures 6C and 6D, both include input data 616 and input data 618.

[0047] By streaming the input data in a snake-like pattern versus the raster scan shown in Figure 4, the data processing order is rearranged and max pooling operations may be performed between convolution operations without additional latency or memory storage. The max pooling operations to select the maximum pixel value in a sub-region of the input image are merged into the convolution stage, eliminating the memory access energy (access to static random access memory 110) of the max pooling stage. For example, in addition to computing the dot product of a 2x2 kernel with the input data at each location, a max pooling operation is performed to output the maximum value in each 2x2 sub-region where the filter is convolved in the image.

[0048] A standard binary neural network activation function is a sign operation. The operation performed by a max pooling layer is sign(max(a,b)), where a and b are the inputs. The sign(max(a,b)) operation may be rewritten equivalently as max(sign(a),sign(b)). Instead of performing an operation that uses all the bits of a and b to determine which one is larger, and then returns the sign of a or b, the operation is reduced to comparing the sign bits (most significant bit) of one bit of a and b.

[0049] A non-linear code activation function may be used to further reduce the output bandwidth by quantizing the 16-bit accumulator values ​​to a single bit. An embodiment with 128 parallel execution units and a dot product width of 1024 bits balances the input and output memory bandwidth. The peak memory to execution unit bandwidth is 1024 bits per cycle. The peak execution unit to memory bandwidth is 256 bits per cycle. "Circuit Optimization"

[0050] The binary neural network accelerator 102 described with respect to Figure 2 is an all-digital circuit. Near-threshold voltage (NTV) operation is the key to the success of all-digital circuits with high energy efficiency comparable to the analog circuits on which they are designed.

[0051] In general, the use of near-threshold voltage operation in analog circuits is more difficult because the voltage headroom required for functional analog circuits is relatively high. Near-threshold voltage operation also increases the effects of threshold voltage device mismatch in analog circuits. The dynamic energy of digital circuits scales approximately as the square of the supply voltage. In one embodiment of an image processor using a fully digital binary neural network accelerator 102, voltage scaling to near-threshold voltage results in a 2-3 times improvement in energy efficiency.

[0052] 1, to obtain maximum benefit from voltage scaling, a separate power rail 116 from the power supply 114 provides a constant nominal voltage to the static random access memory 110 to maintain correct function, while the dot product execution units 202 in the binary neural network accelerator 102 are scaled close to their true energy optimum. Highly parallel circuits also maximize the energy efficiency of near-threshold voltage operation.

[0053] The energy optimum occurs when the dynamic energy is approximately equal to the leakage energy. For digital arithmetic circuits, this point is usually near the subthreshold region. However, when a relatively large static random access memory 110 is added to the system, the leakage power increases, the energy optimum operating voltage increases, and the energy efficiency decreases. For near-threshold voltage operation, the static random access memory 110 does not scale well to low voltage operation, so the leakage cost of the static random access memory becomes more prominent. Thus, the leakage associated with the static random access memory 110 is approximately constant.

[0054] Highly parallel circuits allow the relative cost of added memory leakage to be amortized over the dynamic energy of many dot-product execution units 202 (FIG. 2), reducing the energy-optimal voltage. Also, because there are two voltage domains 120, 122 (FIG. 1), voltage scaling between these voltage domains may create skew between the clock trees in the two voltage domains 120, 122, introducing potential timing variations. A negative-edge flip-flop 124 at the boundary of the two voltage domains 120, 122 prevents minimum delay hold variations from voltage scaling.

[0055] A lightweight pipeline design is used to achieve maximum energy efficiency at near-threshold voltages. At typical operating voltages well above the threshold voltage, the pipeline runs at a lower voltage at the same operating frequency, improving performance at nominal operating voltages and improving energy efficiency. However, since the operating frequency decreases exponentially with NTV voltage scaling, the energy optimum is not significantly affected by the pipeline. For this reason, a very lightweight pipeline is used in binary neural network accelerators, so that the energy cost of the pipeline registers is only a small fraction of that of a binary neural network accelerator.

[0056] To estimate the energy improvement from lightweight pipelining, let us assume that a conventional design that consumes 40% of its power at nominal operating voltage in the pipeline register is re-pipelined with fewer stages, so that its power in the pipeline register is only 10%. In this scenario, the energy improvement at the optimum energy point is about 30%. This estimate takes advantage of the fact that the optimum energy point voltage of digital logic circuits in a particular process is nearly the same. This optimum energy voltage is between Vdd and Vmin (the minimum functional operating voltage close to the threshold voltage).

[0057] To amortize the costs of memory access and data movement, the parallelism required for a binary neural network is higher than for a multi-bit accelerator. For example, in a typical 8-bit integer neural network accelerator with four 8-bit multiply-accumulate (MAC) units, the computational cost in the MAC unit is comparable to, or ideally much greater than, the combined cost of accessing a static random access memory (SRAM) and moving data from the SRAM to the four 8-bit MAC units in parallel over a bus or interconnect.

[0058] Figure 7 is a block diagram of an embodiment of a binary neural network accelerator 102 with high parallelism. In the binary neural network accelerator 102 described with respect to Figures 1-3, two bits are multiplied by performing an exclusive NOT OR (XNOR) operation on the binary encoding (0,1). The XNOR operation consumes much less energy than accessing the static random access memory 110, data movement, and accumulation operations. The parallelism of the binary neural network accelerator 102 is increased, amortizing the cost of memory and data movement over many binary operations.

[0059] 7, data is moved in parallel from the static random access memory 110 over a bus or interconnect to multiple 1-bit MACs 702, each of which performs an exclusive NOT OR (XNOR) operation. In one embodiment, there are 256 1-bit MACs 702. The 256 1-bit MACs 702 performing 256 MAC operations in parallel is equivalent in terms of energy and area to a typical 8-bit integer neural network accelerator with four 8-bit multiply-accumulate (MAC) units.

[0060] 8 is a flow graph illustrating an embodiment of a method for providing an energy-efficient binary neural network circuit. At block 800, weights are stored near the compute in the near memory latch array 204 shown in FIG.

[0061] At block 802, a matrix multiplication is performed using the stored weights and a binary neural network NOR multiplier.

[0062] At block 804, a convolution or full join operation is performed using the dot product method.

[0063] At block 806, if a max pooling operation is to be performed, processing continues at block 808. Otherwise, processing continues at block 810. At block 808, a max pooling operation is performed between the convolution operations.

[0064] At block 810, if there is another layer to be processed, processing continues at block 800 to process the next layer, otherwise processing ends.

[0065] Figure 9 is a block diagram of an embodiment 900 of a computer system including the image classifier 100 shown in Figure 1. The computer system 900 may correspond to a computing device including, but not limited to, a server, a workstation computer, a desktop computer, a laptop computer, and / or a tablet computer.

[0066] The computer system 900 includes a system-on-chip (SOC or SoC) 904 that combines a processor, graphics, memory, and input / output (I / O) control logic into one SoC package. The SoC 904 includes at least one central processing unit (CPU) module 908, a volatile memory controller 914, a graphics processor unit (GPU) 910, and an image classifier 100. In other embodiments, the volatile memory controller 914 may be external to the SoC 904. The CPU module 908 includes at least one processor core 902 and a level 2 (L2) cache 906. In one embodiment, the image classifier 100 may be included within the CPU module 908.

[0067] Although not shown, each of the processor cores 902 may include one or more instruction / data caches, execution units, prefetch buffers, instruction queues, branch address calculation units, instruction decoders, floating point units, retirement units, etc. According to one embodiment, the CPU module 908 may correspond to a single-core or multi-core general-purpose processor, such as those offered by Intel® Corporation.

[0068] The graphics processor unit (GPU) 910 may include one or more GPU cores and a GPU cache that may store graphics-related data for the GPU cores. The GPU cores may include one or more execution units and one or more instruction and data caches therein. The graphics processor unit (GPU) 910 may also include other graphics logic units, such as one or more vertex processing units, rasterization units, media processing units, and codecs, which are not shown in FIG. 9.

[0069] Within the I / O subsystem 912 are one or more I / O adapters 916 for converting host communications protocols used within the processor core 902 into protocols compatible with particular I / O devices. Some of the protocols for which adapters may be used for conversion include Peripheral Component Interconnect (PCI) Express (PCIe), Universal Serial Bus (USB), Serial Advanced Technology Attachment (SATA), and Institute of Electrical and Electronics Engineers (IEEE) 1594 "Firewire."

[0070] The I / O adapters 916 may communicate with external I / O devices 924 which may include, for example, a display and / or touch screen display 940, a printer, a keypad, a keyboard, user interface devices including wired and / or wireless communication logic, storage devices 918 including hard disk drives ("HDD"), solid state drives ("SSD"), removable storage media, digital video disk (DVD) drives, compact disk (CD) drives, redundant array of independent disks (RAID), tape drives or other storage devices. Multiple storage devices may be communicatively and / or physically coupled together through one or more buses using one or more of a variety of protocols, including, but not limited to, Serial Attached SCSI (Small Computer System Interface) (SAS), Peripheral Component Interconnect Express (PCIe), NVM Express (NVMe) over PCIe (Peripheral Component Interconnect Express), Serial ATA (Advanced Technology Attachment) (SATA).

[0071] There may also be one or more I / O adapters for wireless protocols, such as IEEE 802.15 and Bluetooth 4.0 used in personal area networks, IEEE 802.11-based wireless protocols used in wireless local area networks, and cellular protocols, among others.

[0072] A non-volatile memory (NVM) device is a memory whose state is deterministic even when power is interrupted to the device. In one embodiment, an NVM device may include a block-addressable memory device, such as NAND technology, or more specifically, multi-threshold level NAND flash memory (e.g., single-level cell ("SLC"), multi-level cell ("MLC"), quad-level cell ("QLC"), tri-level cell ("TLC"), or some other NAND). NVM devices may also include byte-addressable, write-in-place, three-dimensional cross-point memory devices, or other byte-addressable, write-in-place NVM devices (also referred to as persistent memories), such as single-level or multi-level phase change memory (PCM) or switched phase change memory (PCMS), NVM devices using chalcogenide phase change materials (e.g., chalcogenide glasses), metal oxide-based, oxygen vacancy-based and conductive bridge random access memory (CB RAM), nanowire memory, ferroelectric random access memory (FeRAM, FRAM®), magnetoresistive random access memory (MRAM) incorporating memristor technology, spin-transfer torque (STT) MRAM, spintronic magnetic junction memory-based devices, magnetic tunnel junction (MTJ)-based devices, DW (domain wall) and SOT (spin orbit transfer)-based devices, resistive memories including thyristor-based memory devices, or any combination of the above, or other memories.

[0073] The I / O adapter 916 may include a Peripheral Component Interconnect Express (PCIe) adapter communicatively coupled to a host interface 928 of the storage device 918 through a bus 944 using the NVM EXPRESS (NVMe) over PCIe (Peripheral Component Interconnect Express) protocol. The Non-Volatile Memory Express (NVMe) standard defines a register-level interface for host software to communicate with non-volatile memory subsystems (e.g., solid-state drives (SSDs)) over the Peripheral Component Interconnect Express (PCIe) high-speed serial computer expansion bus. The NVM EXPRESS standard is available at www.nvmexpress.org. The PCIe standard is available at www.pcisig.com.

[0074] Volatile memory is memory whose state (and therefore the data stored therein) is indeterminate when power is removed from the device. Dynamic volatile memory requires the data stored within the device to be updated in order to maintain its state. An example of dynamic volatile memory includes some variants of DRAM (Dynamic Random Access Memory) or Synchronous DRAM (SDRAM). Memory subsystems as described herein may include DDR3 (Double Data Rate Version 3, originally released by JEDEC (Joint Electronic Device Engineering Council) on June 27, 2007), DDR4 (DDR Version 4, initial specification published by JEDEC in September 2012), DDR4E (DDR Version 4), LPDDR3 (Low Power DDR Version 3, JESD209 3B, August 2013 by JEDEC), LPDDR4 (LPDDR Version 4, JESD209-4, originally published by JEDEC in August 2014), WIO2 (Wide The standard may be compatible with multiple memory technologies, such as High Bandwidth Memory (HBM) (High Bandwidth Memory, JESD325, originally published by JEDEC in October 2013), DDR5 (DDR version 5, currently under review by JEDEC), LPDDR5 (LPDDR5, currently under review by JEDEC), HBM2 (HBM version 2), those currently under review by JEDEC, or other or combinations of multiple memory technologies, and technologies based on derivatives or extensions of such specifications. JEDEC standards are available at www.jedec.org.

[0075] Operating system 142 is software that manages computer hardware and software, including memory allocation and access to I / O devices. Examples of operating systems include Microsoft® Windows®, Linux®, iOS®, and Android®.

[0076] Fully digital Compute Near Memory binary neural network accelerators offer much greater energy efficiency compared to floating point and multi-bit integer based neural networks. The high energy efficiency offered by binary neural network optimized hardware accelerators makes them well suited for edge and Internet of Things (IoT) applications where energy efficiency is essential to operation for battery powered devices such as drones or wearables. Thermally limited applications such as wearable devices or thermal throttled server accelerators are also ideal use cases.

[0077] Fully digital Compute Near Memory Binary Neural Network Accelerators achieve energy efficiency comparable to or better than analog-based binary neural network accelerators. Fully digital designs are more process scalable, robust to PVT (process, voltage, temperature) variations, insensitive to circuit noise, and require lower design costs. Compute Near Memory Binary Neural Network Accelerators operate across a wide range of voltage operating points to trade off energy and performance. Fully digital Compute Near Memory Binary Neural Network Accelerators also achieve high area efficiency (TOPS / mm) due primarily to the high area cost of passive analog components such as capacitors and resistors. 2 ), as well as the requirement to improve device integrity for larger devices.

[0078] The flow diagrams shown herein provide examples of various process operation sequences. The flow diagrams may show processes to be performed by software or firmware routines and physical operations. In one embodiment, the flow diagrams may show the states of a finite state machine (FSM), which may be implemented in hardware and / or software. Although the order of operations is shown in a particular sequence or order, it may be changed unless otherwise stated. Thus, the illustrated embodiment should be understood as an example, and the process may be performed in a different order, and some operations may be performed in parallel. Also, one or more operations may be omitted in various embodiments, and thus not all operations are required in all embodiments. Other process flows are possible.

[0079] To the extent that various operations or functions are described herein, they may be described or defined as software code, instructions, configurations, and / or data. The content may be directly executable ("object" or "executable" form), source code, or difference code ("delta" or "patch" code). The software content of the embodiments described herein may be provided via an article of manufacture with content stored thereon, or by a method of operating a communications interface and transmitting data over the communications interface. Machine-readable storage may cause a machine to perform the described functions or operations, and includes any mechanism for storing information in a form accessible by a machine (e.g., a computing device, an electronic system, etc.), including recordable and non-recordable media (e.g., read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.). A communications interface includes any mechanism for interfacing with any medium, such as hardwired, wireless, optical, etc., for communicating with another device, such as a memory bus interface, a processor bus interface, an Internet connection, a disk controller, etc. A communications interface may be configured to provide configuration parameters and / or send signals to prepare the communications interface to provide data signals describing software content. A communications interface may be accessed via one or more commands or signals sent to the communications interface.

[0080] Various components described herein may be means for performing the described operations or functions. Each component described herein includes software, hardware, or a combination thereof. A component may be implemented as a software module, a hardware module, special purpose hardware (e.g., application specific hardware, application specific integrated circuits (ASICs), digital signal processors (DSPs), etc.), an embedded controller, hardwired circuitry, etc.

[0081] Besides what is described herein, various modifications may be made to the disclosed embodiments without departing from the scope of the embodiments and implementations of the present invention.

[0082] Therefore, the descriptions and examples herein should be construed as illustrative rather than limiting, and the scope of the present invention should be defined only by reference to the claims that follow. [Other possible claims] [Item 1] 1. A neural network accelerator comprising a binary convolutional neural network, comprising: The above binary convolutional neural network is A plurality of execution units; a near memory latch array having a set of latches that are finely interleaved with the execution units to reduce energy consumption and enable high bandwidth memory access to the execution units, the set of latches being communicatively coupled to each of the execution units for storing results of matrix multiplications in the execution units. [Item 2] 2. The neural network accelerator of claim 1, wherein each of the plurality of execution units performs the matrix multiplication using an inner product. [Item 3] 2. The neural network accelerator of claim 1, wherein the convolution operation shifts weight filters over the input image with strides in a snake-like pattern. [Item 4] 4. The neural network accelerator of item 3, wherein the max pooling operation is performed during the convolution operation. [Item 5] 5. The neural network accelerator of claim 4, wherein the max pooling operation uses a sign bit to select the maximum value of the subregion. [Item 6] The binary convolutional neural network further comprises: 2. The neural network accelerator of claim 1, comprising: a memory for storing an input image; and a first power rail for providing a first power from a power source to the memory, the first power rail being separate from a second power rail for providing a second power to the binary convolutional neural network. [Item 7] 7. The neural network accelerator of claim 6, wherein the memory has multiple banks for storing the input images, and each input in each window is mapped to a different memory bank, allowing single cycle access to the window. [Item 8] 2. The neural network accelerator of claim 1, wherein the binary convolutional neural network uses a lightweight pipeline to optimize energy efficiency. [Item 9] The binary convolutional neural network further comprises: 2. The neural network accelerator of item 1, having a memory for storing an input image, wherein the number of operations performed per bit read from the memory is 128 or more to optimize energy efficiency by amortizing memory access and data movement costs over many binary neural network operations. [Item 10] interleaving a set of latches with the execution units to reduce energy consumption and enable high bandwidth memory access to the execution units; communicatively coupling a set of latches to each of the plurality of execution units for storing results of the matrix multiplication in the execution units. [Item 11] 11. The method of claim 10, further comprising: performing, by each of the plurality of execution units, the matrix multiplication using an inner product. [Item 12] Item 11. The method of item 10, further comprising shifting the weighting filters on the input image with a stride of a snake-like pattern by a convolution operation. [Item 13] Item 13. The method of item 12, further comprising performing a max pooling operation during the convolution operation. [Item 14] 1. A system comprising a neural network accelerator having a binary convolutional neural network and a display communicatively coupled to a processor for displaying an input image, the system comprising: The above binary convolutional neural network is A plurality of execution units; a near memory latch array having a set of latches that are finely interleaved with the execution units to reduce energy consumption and enable high bandwidth memory access to the execution units, the set of latches being communicatively coupled to each of the execution units for storing results of matrix multiplications in the execution units. [Item 15] Item 15. The system of item 14, wherein each of the plurality of execution units performs the matrix multiplication using an inner product. [Item 16] Item 15. The system of item 14, wherein the convolution operation shifts a weight filter on the input image with strides in a snake-like pattern. [Item 17] Item 17. The system of item 16, wherein the max pooling operation is performed during the convolution operation. [Item 18] The binary convolutional neural network further comprises: 15. The system of claim 14, further comprising a memory for storing the input image and a first power rail providing a first power from a power source to the memory, the first power rail being separate from a second power rail providing a second power to the binary convolutional neural network. [Item 19] Item 15. The system of item 14, wherein the binary convolutional neural network uses a lightweight pipeline to optimize energy efficiency. [Item 20] The binary convolutional neural network further comprises: Item 15. The system of item 14, having a memory for storing the input image, wherein the number of operations performed per bit read from the memory is 128 or more to optimize energy efficiency by amortizing memory access and data movement costs over many binary neural network operations.

Claims

1. A neural network accelerator comprising a convolutional neural network, The convolutional neural network includes: A plurality of execution units; a near memory latch array having a set of latches that are finely interleaved with the execution units to reduce energy consumption and enable high bandwidth memory access to the execution units, the set of latches being communicatively coupled to each of the execution units for storing results of matrix multiplications in the execution units; A neural network accelerator in which the convolution operation shifts weight filters over the input image with strides in a snake-like pattern.

2. 2. The neural network accelerator of claim 1, wherein each of the plurality of execution units performs the matrix multiplication using an inner product.

3. The neural network accelerator of claim 1 or 2, wherein a max pooling operation is performed during the convolution operation.

4. The neural network accelerator of claim 3 , wherein the max pooling operation uses a sign bit to select a maximum value of a subregion.

5. The convolutional neural network further comprises:

5. The neural network accelerator of claim 1, further comprising: a memory for storing an input image; and a first power rail for providing a first power from a power supply to the memory, the first power rail being separate from a second power rail for providing a second power to the convolutional neural network.

6. A neural network accelerator comprising a convolutional neural network, The convolutional neural network includes: A plurality of execution units; a near memory latch array having a set of latches that are finely interleaved with the execution units to reduce energy consumption and enable high bandwidth memory access to the execution units, the set of latches being communicatively coupled to each of the execution units for storing results of matrix multiplications in the execution units; a memory for storing an input image; a first power rail that provides a first power from a power supply to the memory, the first power rail being separate from a second power rail that provides a second power to the convolutional neural network.

7. 7. The neural network accelerator of claim 5 or 6, wherein the memory has multiple banks for storing the input images, and each input within a window is mapped to a different memory bank, allowing single cycle access to the window.

8. 8. The neural network accelerator of claim 1, wherein the convolutional neural network uses a lightweight pipeline to optimize energy efficiency.

9. The convolutional neural network further comprises:

5. The neural network accelerator of claim 1, further comprising a memory for storing an input image, the number of operations performed per bit read from the memory being greater than or equal to 128 to optimize energy efficiency by amortizing memory access and data movement costs over many neural network operations.

10. interleaving a set of latches with a plurality of execution units to reduce energy consumption and enable high bandwidth memory access to the plurality of execution units; communicatively coupling a set of latches to each of the plurality of execution units for storing results of the matrix multiplication in the execution units; and shifting the weighting filter over the input image with strides in a snake-like pattern by a convolution operation.

11. 11. The method of claim 10, further comprising: performing, by each of the plurality of execution units, the matrix multiplication using an inner product.

12. The method of claim 10 or 11, further comprising a step of performing a max pooling operation during the convolution operation.

13. Apparatus comprising means for carrying out the method according to any one of claims 10 to 12.

14. A system comprising a neural network accelerator having a convolutional neural network and a display communicatively coupled to the processor for displaying an input image, comprising: The convolutional neural network includes: A plurality of execution units; a near memory latch array having a set of latches that are finely interleaved with the execution units to reduce energy consumption and enable high bandwidth memory access to the execution units, the set of latches being communicatively coupled to each of the execution units for storing results of matrix multiplications in the execution units; A system in which a convolution operation shifts a weight filter over the input image with strides in a snake-like pattern.

15. 15. The system of claim 14, wherein each of the plurality of execution units performs the matrix multiplication using an inner product.

16. The system of claim 14 or 15, wherein a max pooling operation is performed during the convolution operation.

17. The convolutional neural network further comprises:

17. The system of claim 14, further comprising: a memory for storing an input image; and a first power rail for providing a first power from a power source to the memory, the first power rail being separate from a second power rail for providing a second power to the convolutional neural network.

18. A system comprising a neural network accelerator having a convolutional neural network and a display communicatively coupled to the processor for displaying an input image, comprising: The convolutional neural network includes: A plurality of execution units; a near memory latch array having a set of latches that are finely interleaved with the execution units to reduce energy consumption and enable high bandwidth memory access to the execution units, the set of latches being communicatively coupled to each of the execution units for storing results of matrix multiplications in the execution units; a memory for storing an input image; a first power rail that provides a first power from a power supply to the memory, the first power rail being separate from a second power rail that provides a second power to the convolutional neural network.

19. 19. The system of claim 14, wherein the convolutional neural network uses a lightweight pipeline to optimize energy efficiency.

20. The convolutional neural network further comprises:

17. The system of claim 14, further comprising a memory for storing the input image, wherein the number of operations performed per bit read from the memory is 128 or more to optimize energy efficiency by amortizing memory access and data movement costs over many neural network operations.

Images