Exploiting input data sparsity in neural network computation units

Abstract

To provide a method for executing machine learning calculation using a calculation unit of special purpose, a storage device and an electronic system.SOLUTION: In a method by a computing device, a controller at least partially receives input activations provided from a memory address position associated with an index, determines whether or each of the input activations has either a zero value or a non-zero value, stores at least the one input activation in a memory bank and generates an index including the one or more memory address positions having the input activation value that is the non-zero value, broadcasts at least the one input activation on a data bus accessible by the one or more units among calculation arrays from the memory bank, detects the broadcast, and provides a control signal for preventing multiplication operation associated with zero input to at least the one unit among the calculation arrays.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] background This specification relates to performing machine learning calculations using special-purpose computing units. [Background technology]

[0002] A neural network is a machine learning model that uses one or more layers of the model to generate an output (e.g., a classification) for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer in the network). Each layer of the network generates an output from the received input according to the current values ​​of a corresponding set of parameters.

[0003] Some neural networks include one or more convolutional neural network layers. Each convolutional neural network layer has an associated set of kernels. Each kernel contains values ​​set by a user-created neural network model. In some implementations, the kernels identify specific image shapes, shapes, or colors. The kernels can be represented as a matrix structure of weight inputs. Each convolutional layer can also process a set of activation inputs. The set of activation inputs can also be represented as a matrix structure. Summary of the Invention [Problem to be solved by the invention]

[0004] overview One method of performing convolution calculations involves a large number of matrix multiplications in a large dimensional space. A processor or controller device of a computation unit may compute the matrix multiplications by brute force methods. For example, Although time intensive, a processor can repeatedly calculate the individual sums and products for a convolution calculation. The degree to which a processor can parallelize calculations is limited by its architecture. [Means for solving the problem]

[0005] Innovative aspects of the subject matter described herein can be embodied in a computer-implemented method. The method includes: a computing device receiving a plurality of input activations provided, at least in part, from a source external to the computing device; and a controller of the computing device determining whether each of the plurality of input activations is a zero value or a non-zero value. The method further includes storing at least one input activation in a memory bank of the computing device. Storing the at least one input activation includes the controller generating an index including one or more memory address locations having input activation values ​​that are non-zero values. The method further includes the controller providing, from the memory bank, the at least one input activation to a data bus accessible by one or more units of a computational array. The activation is, at least in part, associated with the index. The address is provided from the specified memory address location.

[0006] In some implementations, the index is created based on a bitmap including a plurality of bits, each bit of the bitmap indicating at least one of a non-zero input activation value or a zero input activation value. In some implementations, the method further includes providing a first input activation having a non-zero value to cause at least one unit to perform a calculation using the non-zero value, and thereafter providing a second input activation having a zero value to prevent at least one unit from performing a calculation that may be performed using the zero value.

[0007] In some implementations, the preventing step occurs in response to the controller determining that the input activation is provided from a memory address location not associated with the index. In some implementations, the method further includes the steps of: the controller detecting that the input activation is provided from a memory address location not associated with the index; and, in response to the detecting step, providing a control signal to at least one unit of the computational array to prevent a multiplication operation associated with the zero input activation value.

[0008] In some implementations, the method further includes the controller mapping a first portion of the tensor computation that uses a first input activation to a first unit and mapping a second portion of the tensor computation that also uses the first input activation to a second unit different from the first unit. In some implementations, the method further includes sequentially providing a single input activation to the data bus. The single input activation is accessed and selected from a memory address location associated with the index. In some implementations, the providing step further includes not providing input activations having a zero value.

[0009] Another innovative aspect of the subject matter described herein can be embodied in one or more machine-readable storage devices executable by one or more processing devices and storing instructions for performing the following operations: a computing device receiving a plurality of input activations provided, at least in part, from a source external to the computing device; and a controller of the computing device determining, by the controller, whether each of the plurality of input activations is a zero value or a non-zero value. The following operations further include storing at least one of the input activations in a memory bank of the computing device. Storing the at least one input activation includes the controller generating an index including one or more memory address locations having input activation values ​​that are non-zero values. The following operations further include the controller providing, from the memory bank, at least one input activation to a data bus accessible by one or more units of a computational array, the activation being provided, at least in part, from a memory address location associated with the index.

[0010] Another innovative aspect of the subject matter described herein can be embodied in an electronic system that includes a controller disposed in a computing device and including one or more processing devices, and one or more machine-readable storage devices for storing instructions executable by the one or more processing devices to perform the following operations: The method includes an operation of a device receiving a plurality of input activations provided at least in part from a source external to the computing device, and an operation of the controller determining whether each of the plurality of input activations is a zero value or a non-zero value. The following operations further include an operation of storing at least one of the input activations in a memory bank of the computing device. The operation of storing the at least one input activation includes an operation of generating an index including one or more memory address locations having an input activation value that is a non-zero value. The following operations further include an operation of the controller providing, from the memory bank, at least one input activation to a data bus accessible by one or more units of a computational array. The activation is provided, at least in part, from a memory address location associated with the index.

[0011] The subject matter described herein can be implemented in particular embodiments to achieve one or more of the following advantages: Activations accessible from a first memory and weights accessible from a second memory can be traversed in a single computing system based on memory address values ​​retrieved from a register; a controller of the computing system can compress activation data by storing only non-zero values ​​in the first memory, thereby saving memory storage space and corresponding bandwidth; matrix multiplication is performed in the computing system based in part on providing primarily non-zero input activations; and, if the computing system uses a communication scheme involving primarily non-zero activation values, eliminating powers of zero can improve or facilitate computational efficiency.

[0012] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs encoded on computer storage devices configured to perform the operations of the methods. One or more computer systems may be so configured by software, firmware, hardware, or combinations thereof installed on the systems that, when operated, cause the systems to perform the operations. One or more computer programs may be so configured by having instructions that, when executed by a data processing device, cause the devices to perform the operations.

[0013] The subject matter described herein also relates to image recognition and / or classification methods / systems that can be implemented using the disclosed techniques to exploit sparsity in input data when a computing unit of a hardware computing system processes inputs for neural network layers to perform inference calculations.

[0014] The details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from this description, the accompanying drawings, and the appended claims. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 illustrates an exemplary computational structure including activations and parameters. [Figure 2] FIG. 1 illustrates an exemplary computational structure including activation and multiple parameter structures for two or more output depths. [Figure 3] FIG. 1 illustrates an exemplary computing system for supplying input activations to one or more parameters. [Figure 4]FIG. 1 illustrates an example architecture that includes a memory unit that provides input activations to one or more multiply accumulate (MAC) operators. [Figure 5] 1 is an exemplary flowchart of a process for reducing parameter calculations to exploit sparsity in input data. DETAILED DESCRIPTION OF THE INVENTION

[0016] Like reference numbers and designations in the various drawings indicate like elements. Detailed Description The subject matter described herein relates to reducing the computations performed within a computational unit or tile of an exemplary neural network hardware computing system. Generally, as part of computing neural network inference, input activations are multiplied by parameters or weight values ​​to generate output activations. Here, inputs and input activations may refer to data elements contained in multidimensional data structures, such as tensors, matrices, and / or data arrays, commonly used in neural networks. Due to the algorithmic nature of computational inference for deep neural networks, the majority of input activations are zero. In other words, current computational units perform numerous unnecessary computations, including multiplying zero (input activation values) by a number (e.g., weights).

[0017] This specification describes, in part, a more efficient activation storage and communication scheme and a custom architecture design for deep neural network processing, particularly for processing convolutional layers of neural networks. Unlike traditional hardware accelerators that perform time-consuming dense matrix multiplications, this specification describes an architecture that 1) can skip or bypass calculations when it recognizes zero input values, and 2) can reduce memory usage in the compute unit by storing compressed input activations that contain only non-zero values. Overall, the teachings of this specification improve compute unit performance for neural network inference calculations while saving energy by skipping unnecessary calculations.

[0018] 1 shows an exemplary computational structure 100 including an activation structure 102 and a parameter structure 104. The activation structure 102 may include a first data structure 102a including multiple data elements corresponding to a first input depth (indicated by the subscript 0). Similarly, the activation structure 102 may also include a second data structure 102b including multiple data elements corresponding to a second input depth (indicated by the subscript 1). The multiple data elements shown in the data structure 102a are denoted as a0, b0, c0, d0, and the multiple data elements shown in the data structure 102b are denoted as a1, b1, c1, d1. Each data element (a0, a1, b0, b1, c0, d0, etc.) of the data structures 102a / 102b is an input activation value, and each input depth corresponds to the depth of the input to a neural network layer. In some implementations, a neural network layer may have an input depth of one, and in other implementations, a neural network layer may have an input depth of two or more.

[0019] The parameter structure 104 can be described in a manner similar to the activation structure 102. The parameter structure 104 includes a first data structure 104a and a second data structure 104b. Each data structure 104a / 104b may include multiple data elements, each including a kernel value. As shown in FIG. 1, the multiple data elements corresponding to the data structure 104a are denoted as x0, y0, z0, and the multiple data elements corresponding to the data structure 104b are denoted as x1, y1, z1.

[0020] As mentioned above, each layer of the neural network generates an output from the inputs it receives according to the values ​​of its corresponding set of operands. Similarly, each convolutional layer can process a set of activation inputs, which can be represented as a matrix structure. A convolutional neural network layer will also have an associated set of kernels containing values, which can also be represented as a matrix structure of weights. In Figure 1, activation structure 102 can correspond to a matrix structure with one or more activation inputs, and parameter structure 104 can correspond to a matrix structure with one or more kernel or weight parameters.

[0021] As described in more detail below, various layers of a neural network process machine learning inference by performing a large amount of calculations, including matrix multiplication. The calculation process performed within a neural network layer (e.g., a convolutional layer) may include multiplying an input activation (i.e., a first operand) by a weight (i.e., a second operand) in one or more cycles and performing an accumulation of products over many cycles. An output activation is generated based on the multiplication and accumulation operations performed on the two operands.

[0022] As shown, equation 106 provides an example of a serial sequence-based mathematical operation that may be performed when multiplying an input activation associated with a data element of activation structure 102 by a kernel value or weight / parameter associated with a data element of parameter structure 104. For example, in equation 106, if index "i" equals 0, then the input activation associated with data element a0 of activation structure 102 is multiplied by the weight / parameter associated with data element x0 of parameter structure 104. Furthermore, because equation 106 is partially a serial-based equation, additional multiplication operations may occur between sets of operands corresponding to other data elements of activation structure 102 and parameter structure 104. In some implementations, multiplication of one set of operands may generate a partial sum 106a / 106b for a particular output feature or activation. Thus, the partial sums may be added to generate an output feature, as shown in equation 106.

[0023] The neural network can be embodied in one or more hardware computing systems including multiple computing units configured to facilitate the machine learning inference workload of the network layers. Each computing unit can process a subset of the computations for a given layer. In some implementations, the structure 100 can be embodied in one or more computing units, each including at least two memory banks and one or more multiply-accumulate (MAC) cells that can collectively form a MAC array (described below).

[0024] In one example, a first memory bank 108 of an exemplary computing unit may be configured to store data associated with the activation structure 102 and to receive and write input activation values ​​to memory address locations within the memory bank 108. Similarly, a second memory bank 110 of an exemplary computing unit may be configured to store data associated with the parameter structure 104 and to receive and write weight values ​​to memory address locations within the memory bank 110. In this example, each element (e.g., a0, b0, c0, d0) of the data elements 102a may be stored at a corresponding respective memory address in the first memory bank 108. Similarly, each element (e.g., x0, y0, z0) of the data elements 104a may be stored at a corresponding respective memory address in the second memory 110.

[0025] In some implementations, the first memory bank 108 and the second memory bank 110 are each a single volatile memory unit or multiple volatile memory units. In some other implementations, memory bank 108 and memory bank 110 each are a single non-volatile memory unit or multiple non-volatile memory units. Memory banks 108 and 110 may also be another form of computer-readable storage medium, for example, a floppy disk device, a hard disk device, an optical disk device, or a tape device, or an array of devices including flash memory or other similar solid-state memory devices, or devices in a storage area network or other configuration.

[0026] Generally, a computing unit of a hardware computing system may include one or more registers to track memory address values. Data elements of the matrix structure corresponding to the activation structure 102 may be accessed from a first memory bank 108, and data elements of the matrix structure corresponding to the parameter structure 104 may be accessed from a second memory bank 110. An exemplary control device of a computing tile / computation unit may access and / or traverse the data elements of the matrix structure based on address values ​​accessible from one or more registers. An exemplary computation unit / tile, including an exemplary control device, activation structure 102, parameter structure 104, first memory bank 108, and second memory bank 110, is described in more detail below in connection with FIG. 3 .

[0027] Additionally, additional details and descriptions regarding a hardware computing system for facilitating neural network tensor calculations and matrix multiplication for neural network inference workloads are provided in U.S. Patent Application Publication No. 2016 / 0102094, filed on October 27, 2016, entitled "Neural Network Compute Tile." No. 15 / 335,769, the entire disclosure of which is incorporated herein by reference in its entirety.

[0028] FIG. 2 illustrates an exemplary computational structure 200 including an activation structure and multiple parameter structures for two or more output feature depths. In some implementations, a neural network may have multiple layers that generate outputs with multiple output feature depths. In some implementations, each parameter structure may be responsible for a corresponding one of the output depths. Thus, computational structure 200 illustrates a scalable computing structure in which additional parameter structures 104a / 104b / 104c are added to facilitate calculations associated with N values ​​of output depths. N is variable and may have an integer value ranging from 1 to 5, for example, or alternatively, an integer value ranging from 1 to N, depending on the preference or need of a computing system designer.

[0029] As shown by data path 105, individual input activation values ​​for elements associated with data structure 102a can be supplied to each parameter structure 104a / 104b / 104c for use in the calculations performed by the composite operators associated with the respective parameter structures 104. Each parameter structure 104 can then pass the activation values ​​received from its left neighbor to its right neighbor in a pipelined manner. Alternatively, activations can be provided and consumed by each parameter structure 104 simultaneously.

[0030] Matrix 202 may represent an exemplary matrix structure corresponding to activations 102. More specifically, element row 202a may correspond to data structure 102a, and element row 202b may correspond to data structure 102b. Generally, as an example, first parameter structure 104(1) may represent a matrix structure corresponding to activations 102 to perform calculations associated with space 206. While the first parameter structure 104(1) is accessed, the second parameter structure 104(2) is accessed to perform calculations associated with the z-dimension. Although not shown, additional calculations can also be performed corresponding to the z-dimension. As an example, element row 202a can be in the R-plane of an RGB image, and element row 202b can be in the G-plane of the same RGB image. An exemplary convolutional layer of a neural network typically generates multiple output features. Exemplary output features can include an output feature for classifying apples and another output feature for classifying bananas. With respect to data structure 204, spaces 206 and 208 can represent different planes for different classifications.

[0031] 3 shows an exemplary computing system 300 for providing input activations to one or more parameter structures. The computing system 300 generally includes a controller 302. The controller 302 provides one or more control signals 310 to cause input activations for the activation structure 102 to be stored in or retrieved from memory addresses in the memory bank 108. Similarly, the controller 302 also provides one or more control signals 310 to cause weights for the parameter structures 104a / 104b / 104c to be stored in or retrieved from memory addresses in the memory bank 110. The computing system 300 further includes one or more multiply-accumulate (MAC) cells / units 304, an input activation bus 306, and an output activation bus 308. The control signal 310 can, for example, cause the memory bank 108 to provide one or more input activations to the input activation bus 306, cause the memory bank 110 to provide one or more weights to the parameter structure 104a / 104b / 104c, and / or cause the MAC unit 304 to perform a calculation that generates an output activation that is provided to the output activation bus 308.

[0032] The controller 302 may include one or more processing units and memory. In some embodiments, the processing unit of the controller 302 may include one or more processors (e.g., microprocessors or central processing units (CPUs)). , graphics processing unit (GPU), application specific integrated circuit (ASIC), or various processors In alternative embodiments, controller 302 may include other storage or computing resources / devices (e.g., buffers, registers, control circuitry, etc.) that provide additional processing options for performing one or more of the decisions and calculations described herein.

[0033] In some implementations, the processing unit of the controller 302 executes programmed instructions stored in memory to cause the controller 302 and the computing system 300 to perform one or more functions described herein. The memory of the controller 302 may include one or more non-transitory machine-readable storage media. The non-transitory machine-readable storage media may include solid-state memory, magnetic and optical disks, portable computer diskettes, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (e.g., EPROM, EEPROM, or flash memory), or any other tangible medium capable of storing information.

[0034] Generally, the computing system 300 is an exemplary computing unit or tile, and may include additional hardware structures for performing computations associated with multi-dimensional data structures, such as tensors, matrices, and / or data arrays. In some implementations, input activation values ​​may be preloaded into memory bank 108 for activation structure 102, and weight values ​​may be generated by computing system 300. The second memory bank 110 may be preloaded with received data values ​​that arrive at the computing system 300 from an external or higher level control device associated with the neural network hardware computing system.

[0035] Instructions, inputs or input activations, and weights can be provided to system 300 from an external source, such as an external input / output (I / O) device or an upper control device associated with the neural network hardware computing system. In some implementations, one or more data buses provide data communication between the external source (e.g., a control device) and system 300. The data buses are used to provide instructions, inputs or activations and weights from an exemplary I / O device to each of multiple systems 300 or between multiple compute tiles (e.g., multiple systems 300) included in the hardware computing system for neural networks.

[0036] The system 300 can receive instructions that specify specific computational operations to be performed by the system 300. Additionally, the controller 302 can execute programmed instructions, for example, to analyze a data stream associated with received input activations. Analyzing the input activation data stream may enable the controller 302 to detect or determine whether the value associated with each of the input activations is a zero value or a non-zero value. In some implementations, the controller 302 analyzes the exemplary input activation data stream and maps each detected zero activation value and each detected non-zero activation value to a bit vector or bit map 303.

[0037] 3, bitmap 303 may map detected zero-value input activations and detected non-zero-value input activations using binary values. For example, a binary value of "0" may correspond to a detected zero-value input activation value, and a binary value of "1" may correspond to a detected non-zero-value input activation value. For example, bitmap 303 may be an 8-bit bitmap, where odd-numbered bit positions containing a binary "1" correspond to non-zero activation values, and even-numbered bit positions containing a binary "0" correspond to zero activation values.

[0038] The controller 302 can cause the input activations to be stored in the memory banks 108. In general, the data values ​​stored in the memory banks 108 are typically written to their respective corresponding memory address locations. Then, when a data value, such as an input activation, is needed to perform a particular computational operation, the address location in the memory banks 108 can be accessed by an exemplary control device (e.g., the controller 302). The controller 302 can use the bitmap 303 to index memory address locations that contain non-zero input activation values.

[0039] In some implementations, controller 302 uses bitmap 303 to determine which input activations should be written to memory bank 108. For example, analysis of bitmap 303 may indicate that only activation values ​​corresponding to bitmap locations 1, 3, 5, and 7 (non-zero values) should be written to address locations in memory bank 108. Furthermore, data values ​​associated with bitmap locations 2, 4, 6, and 8 (zero values) are written to the controller when activation values ​​are provided on input bus 306. Activation values ​​can be written to or discarded from memory address locations that may or may not be accessed by the controller 302. This allows the bitmap 303 to be used as a basis for compressing zero activation values. In this case, compression occurs if a zero-value input activation has not been written to a memory address location, thereby reducing overall memory usage and freeing up address locations for storing other data values.

[0040] Controller 302 may provide one or more control signals 310 to memory banks 108 to load input activations from memory banks 108 onto input activation bus 306 and provide these values ​​to an array of computational units including MAC 304. In some implementations, bitmap 303, or a non-zero memory address index corresponding to bitmap 303, may be referenced by controller 302 to determine which memory address value must be accessed to provide the non-zero activation value. The activation value is provided by controller 302 from memory bank 108 to data bus 306.

[0041] In some implementations, the input activation is provided, at least in part, from a memory address location associated with the index or bitmap 303. In other implementations, the controller 302 can detect or determine, based on one of the bitmaps 303 or the index, whether the provided input activation has a zero value. In response to making this determination, the controller 302 can further provide control signals to units in the computational array or MACs 304 to prevent, stop, or inhibit unnecessary multiplication operations (e.g., powers of zero) from being performed. Energy savings can be realized within the computing system 300 by providing a zero activation value and subsequently or simultaneously disabling the computational operation associated with that activation.

[0042] As described above, the index includes all memory address locations having input activations with non-zero values. Data bus 306 is accessible by one or more units of the computational array. These units of the computational array can receive one or more non-zero activation values ​​from data bus 306 and perform computations related to matrix multiplication based on the received activation values. In some implementations, computing system 300 will simply provide input activations from memory address locations corresponding to the indexed addresses. Thus, zero activations will not be provided on input bus 306, thereby disabling or preventing the computational operation from being performed. When computing system 300 uses this communication scheme, computational efficiency can be improved or facilitated by eliminating powers of zero.

[0043] Over a given computation cycle, the computation system 300 may need to access elements of the activation structure 102 and the parameter structure 104 to perform multiplication operations associated with inference computations for neural network layers. As described above, particular memory address values ​​for the memory banks 108 and 110 may correspond to elements of the activation structure 102 and the parameter structure 104, respectively.

[0044] Over the cycle in which a computation is performed, the controller 302 will provide one input activation value at a time, and the array of computation units, including the MAC cells 304, will multiply the activations by weights to generate various output activations for a given input activation. Each element or MAC cell 304 (described above as a data structure) may be responsible for various output depths of the neural network layer. Generally, whenever the controller 302 detects a zero activation value, the controller 302 may 1) not store the activation value in the memory bank 108, 2) not provide an activation value, or 3) provide a control signal to a particular computation unit to provide the value and prevent the particular computation unit from performing the multiplication operation corresponding to the zero activation value.

[0045] The array of computational units is fully controlled by the controller 302, which can determine when it is necessary to skip or prevent certain computations based on the detection of zero activation values. Thus, there is no need to add complex hardware structures within the array of computational units to skip certain computations. Furthermore, input activation values ​​may be analyzed as they arrive at the computing system 300 for storage in the memory bank 108. In response to the analysis of the input activations, the controller 302 may execute instructions to efficiently compress the activation data by storing only non-zero values ​​in the memory 108, thereby saving memory storage space and corresponding bandwidth.

[0046] When the computing system 300 receives the input activations and weights, the controller 302 may, for example, perform one or more direct memory access operations. Performing these memory access operations includes storing the input activations corresponding to the dimension elements of the activation structure 102 at address locations in the memory bank 108. Similarly, the controller 302 may also store parameters corresponding to the dimension elements of the parameter structure 104 at address locations in the memory bank 110. In addition to the bitmap 303, the controller 302 may further include one or more address registers. The one or more address registers maintain memory addresses from which particular input activations (e.g., having zero or non-zero values) are to be fetched. Furthermore, the one or more registers may also store memory addresses from which corresponding weights by which particular input activations are multiplied are to be fetched.

[0047] As described above, controller 302 identifies memory addresses for non-zero activation values ​​based in part on bitmap 303. In some implementations, controller 302 reads bitmap 303 and, for example, determines at least two memory addresses having non-zero activation values. If controller 302 is configured to provide a zero activation value and then skip or disable calculations for the zero activation value, controller 302 may also determine at least one memory address having a zero activation value. In this implementation, controller 302 can reference the above-mentioned registers to determine the corresponding weight (and memory address) for the first input activation and the corresponding weight (and memory address) for the second input activation.

[0048] As mentioned above, the controller 302 maintains one or more address registers in memory so that, to mitigate or prevent any potential mismatch of operands (input activations and weights), when a zero-valued input activation is detected, the controller 302 can disable the corresponding computation unit, skip loading the particular weight, retrieve the appropriate corresponding weight (and memory address) for the next non-zero input activation, and resume computing the output activations for the given neural network layer.

[0049] In some implementations, the output activations calculated in a first neural network layer are used as input activations to a next second layer in the network (e.g., the next hidden layer or output layer of the network). Generally, each layer of a neural network generates outputs from received inputs according to the current values ​​of a corresponding set of respective parameters. In some cases, the controller 302 can execute programmed instructions (i.e., output logic) to analyze a data stream associated with the output activations provided to the output activation bus 308. Analyzing the output activation data stream may enable the controller 302 to detect or determine whether the value associated with each of the output activations is a zero value or a non-zero value. The controller 302 can analyze the exemplary output activation data stream and map each detected non-zero activation value to a bitmap 305. By using the mapped non-zero activation values ​​in the bitmap 305, only non-zero values ​​can be provided as input activations to subsequent computing systems 300 involved in computations associated with the next second layer in the network.

[0050] In an alternative implementation, there may be several computational operations in which a single non-zero input activation is used as an operand for several multiplication operations involving various weights for a given dimension element of parameter structure 104 (i.e., iterating through the “x” or “y” dimension). For example, when controller 302 causes memory bank 108 to provide a first input activation (e.g., a non-zero value), parameter structure 104a receives the activation, and the corresponding weight at a given address is also loaded into parameter structure 104a. Parameter structure 104a will begin updating the particular numerical value of the partial sum (e.g., indicated by variable “K”) that the first input activation affects over K computational cycles. As a result, during these K cycles, parameter structure 104a will not receive any additional input activations. Controller 302 can then provide a control signal to memory bank 108 to cause it to provide the next input activation to input activation bus 306.

[0051] 4 illustrates an exemplary architecture including a memory bank 108 that provides activations 404 to one or more multiply-accumulate (MAC) operators via an input bus 306. A shift register 404 can provide a shifting function such that the activations 404 are routed one at a time to the input bus 306 for reception by one or more MAC operators in the MAC cells 304. As shown, in one implementation, the activations 406 may have an activation value of zero and therefore may not be consumed by the MAC cells 304.

[0052] Generally, a MAC cell 304 containing a MAC operator is defined as a computation unit configured to compute partial sums and, in some implementations, write the partial sum data to an output bus 308. As shown, a cell 304 may be comprised of one or more MAC operators. In one implementation, the number of MAC operators in a MAC cell 304 is referred to as the issue width of the cell. As an example, a dual issue cell may have: It refers to a cell with two MAC operators that can calculate the multiplication of two activation values ​​(from memory bank 108) with two parameters (from memory 110) and perform the addition of the results of the two multipliers with the current partial sum.

[0053] As mentioned above, the input bus 306 is connected to the linear unit (i.e., MAC array 304). A communication bus that provides input activations to the MAC operators. In some implementations, the same input is shared among all MAC operators. The width of the input bus 306 must be wide enough to supply the input activations to the corresponding number of cells for a given MAC array 304. Consider the following example to illustrate the structure of the input bus 306. If the number of cells in a linear unit is equal to 4 and the activation width is equal to 8 bits, the input bus 306 can be configured to provide up to four input activations per cycle. In this example, all cells in the MAC array 304 will only access one of the four provided activations.

[0054] In some examples, the instruction data 312 may indicate that cells of the MAC array 304 will need to perform a calculation using the same input activations. This may be referred to as Zout partitioning within the cells of the MAC array 304. Similarly, if cells of the MAC array 304 require different activations to perform a calculation, Zin partitioning occurs within the cells. In the former case, a single input activation is replicated four times, with four activations read from the memory bank 108 provided over four cycles. In the latter case, a read of the memory bank 108 is required every cycle.

[0055] 5 is an example flowchart of a process for reducing parameter calculations to exploit sparsity in input data. At block 502, computing system 300 receives input activations having either zero or non-zero activation values. As mentioned above, in some implementations, computing system 300 can receive the input activations from a host interface device or higher-level controller of an example neural network hardware system.

[0056] In block 504, the controller 302 determines whether each of the input activations is a zero value or a non-zero value. In some implementations, the controller 302 analyzes the input activation data stream and maps each detected zero and non-zero value to a bitmap 303 that contains binary values ​​corresponding to zero input activation values ​​("0") and non-zero input activation values ​​("1").

[0057] In block 506, the controller 302 stores the received input activations in the memory bank 108. Storing the input activations may include the controller 302 generating indices of one or more memory address locations having input activations that include non-zero values. In some implementations, the indices are created based on the bitmap 303. For example, because each bit in the bitmap 303 indicates either a non-zero activation value or a zero activation value, the controller 302 may reference the bitmap 303 to create indices of memory address locations having non-zero values ​​when writing the input activations to the memory bank 108.

[0058] At block 508, the controller 302 provides at least one input activation from the memory bank 108 to the data bus 306. In some implementations, the input activation is provided, at least in part, from a memory address location identified in an index. As described above, the index identifies all memory address locations that store input activations with non-zero values. The data bus 306 is accessible by one or more units of the computational array. The units of the computational array receive one or more non-zero activation values ​​from the data bus 306 to perform calculations related to matrix multiplication. In some implementations, the computational system 300 will simply provide input activations from memory addresses corresponding to the indexed addresses. When the computational system 300 uses this communication scheme, it can improve computational efficiency by eliminating zero multiplications.

[0059] In block 510, in implementations where all activation values, not just input activations, are provided from indexed addresses, controller 302 detects that input activations are provided from a memory address that is not associated with any indexed address that contains a non-zero activation value. In response to this detecting step, controller 302 can provide a control signal to at least one unit of the computational array to prevent multiplication operations associated with zero inputs. When computing system 300 employs this communication scheme, energy savings can be realized by preventing unnecessary or wasteful computations that do not yield useful results (e.g., useful results including computation of partial sums or output activations).

[0060] Embodiments of the subject matter and functional operations described herein may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware containing the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or to control the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded in an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information to be transmitted to a receiver device suitable for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these.

[0061] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output, or may be implemented as special purpose logic circuitry, such as a field programmable gate array (FPGA) or application specific integrated circuit (ASIC), a general purpose graphics processing unit (GPGPU), or some other processing unit.

[0062] Computers suitable for running computer programs include, and by way of example may be based on, general purpose microprocessors, special purpose microprocessors, or both, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for executing or carrying out instructions and one or more memory devices for storing instructions and data. Typically, a computer may also include one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data. It will be operatively coupled to receive data from, transfer data to, or both receive and transfer data to, the one or more mass storage devices, although a computer need not have such devices.

[0063] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, and magnetic disks, such as internal or removable hard disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0064] While this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features that are described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described as working in a particular combination, and even initially claimed as such, one or more features of a claimed combination may, in some cases, be deleted from that combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0065] Similarly, although operations are shown in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown to achieve desirable results, or that all of the shown operations be performed to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated into a single software product or packaged in multiple software products.

[0066] Further implementations are summarized in the following examples. Example 1: A computer-implemented method including: a computing device receiving a plurality of input activations provided, at least in part, from a source external to the computing device; a controller of the computing device determining whether each of the plurality of input activations has a zero value or a non-zero value; storing at least one of the input activations in a memory bank of the computing device; the controller generating an index including one or more memory address locations having input activation values ​​that are non-zero values; and the controller providing, from the memory bank, at least one input activation to a data bus accessible by one or more units of a computational array, the activation provided, at least in part, from the memory address location associated with the index.

[0067] Example 2: The method of example 1, wherein the index is created based on a bitmap including a plurality of bits, each bit of the bitmap having a non-zero input activation. Indicates at least one of the following activation values:

[0068] Example 3: The method of Example 1 or Example 2, further comprising providing a first input activation having a non-zero value, causing at least one unit to perform a calculation using the non-zero value, and thereafter providing a second input activation having a zero value, preventing at least one unit from potentially performing a calculation using the zero value.

[0069] Example 4: The method of example 3, wherein the preventing step is performed in response to the controller determining that the input activation is provided from a memory address location not associated with the index.

[0070] Example 5: The method of Example 4, further including: detecting, by the controller, that the input activation is provided from a memory address location not associated with the index; and, in response to the detecting, providing a control signal to at least one unit of the computational array to prevent a multiplication operation associated with the zero input activation value.

[0071] Example 6: The method of any one of Examples 1 to 5, further including the controller mapping a first portion of a tensor computation that uses a first input activation to a first unit and mapping a second portion of the tensor computation that also uses the first input activation to a second unit different from the first unit.

[0072] Example 7: The method of any one of Examples 1 to 6, further comprising: sequentially providing a single input activation to the data bus, the single input activation being accessed and selected from a memory address location associated with the index.

[0073] Example 8: The method of any one of Examples 1 to 7, wherein the providing step further comprises not providing the input activation with a zero value.

[0074] Example 9: One or more machine-readable storage devices storing instructions executable by one or more processing devices to perform the following operations: a computing device receiving a plurality of input activations provided, at least in part, from a source external to the computing device; a controller of the computing device determining whether each of the plurality of input activations has a zero value or a non-zero value; storing at least one of the input activations in a memory bank of the computing device; the controller generating an index including one or more memory address locations having input activation values ​​that are non-zero values; and the controller providing, from the memory bank, at least one input activation to a data bus accessible by one or more units of a computational array, the activation being provided, at least in part, from a memory address location associated with the index.

[0075] Example 10: The machine-readable storage device of Example 9, wherein the index is created based on a bitmap including a plurality of bits, each bit of the bitmap indicating at least one of a non-zero-input activation value or a zero-input activation value.

[0076] Example 11: The machine-readable storage device of Example 9 or Example 10, further including an operation of providing a first input activation having a non-zero value, causing at least one unit to perform a calculation using the non-zero value, and thereafter providing a second input activation having a zero value, preventing at least one unit from potentially performing a calculation using the zero value.

[0077] Example 12: The machine-readable storage device of Example 11, wherein the preventing action is performed in response to the controller determining that the input activation is provided from a memory address location not associated with the index.

[0078] Example 13: The machine-readable storage device of Example 12, further including: an operation in which the controller detects that the input activation is provided from a memory address location not associated with the index; and an operation in response to the detecting operation, providing a control signal to at least one unit of the computational array to prevent a multiplication operation associated with the zero input activation value.

[0079] Example 14: The machine-readable storage device of any one of Examples 9 to 13, wherein the following operations further include: the controller mapping a first portion of a tensor computation that uses a first input activation to a first unit, and mapping a second portion of the tensor computation that also uses the first input activation to a second unit different from the first unit.

[0080] Example 15: An electronic system includes a controller disposed in a computing device and including one or more processing devices, and one or more machine-readable storage devices executable by the one or more processing devices and for storing instructions for performing the following operations: the computing device receiving a plurality of input activations provided, at least in part, from a source external to the computing device; the controller determining whether each of the plurality of input activations has a zero value or a non-zero value; storing at least one of the input activations in a memory bank of the computing device; the controller generating an index including one or more memory address locations having input activation values ​​that are non-zero values; and the controller providing, from the memory bank, at least one input activation to a data bus accessible by one or more units of a computational array, the activation being provided, at least in part, from a memory address location associated with the index.

[0081] Example 16: The electronic system of Example 15, wherein the index is created based on a bitmap including a plurality of bits, each bit of the bitmap indicating at least one of a non-zero input activation value or a zero input activation value.

[0082] Example 17: The electronic system of Example 15 or Example 16, further including an operation of providing a first input activation having a non-zero value, causing at least one unit to perform a calculation using the non-zero value, and thereafter providing a second input activation having a zero value, preventing at least one unit from potentially performing a calculation using the zero value.

[0083] Example 18: The electronic system of example 17, wherein the preventing action is This occurs in response to the controller determining that the operation is provided from a memory address location not associated with the index.

[0084] Example 19: The electronic system of Example 17 or Example 18, further including: detecting, by the controller, that the input activation is provided from a memory address location not associated with the index; and, in response to the detecting, providing a control signal to at least one unit of the computational array to prevent a multiplication operation associated with the zero input activation value.

[0085] Example 20: The electronic system of any one of Examples 15 to 19, wherein the following operations further include the controller mapping a first portion of a tensor calculation that uses a first input activation to a first unit and mapping a second portion of the tensor calculation that also uses the first input activation to a second unit different from the first unit.

[0086] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. 1. A hardware integrated circuit configured to implement a neural network including a plurality of neural network layers, the hardware integrated circuit comprising: a first computation unit configured to generate an output of a first neural network layer, the first computation unit comprising: a first plurality of multiply-accumulate cells configured to multiply non-zero inputs in a first batch of inputs by corresponding weights to generate the outputs; a first input bus coupled to each of the first plurality of multiply-accumulate cells; a first input memory coupled to the first input bus, the first input memory configured to store only non-zero inputs in the first batch of inputs, the first computing unit further comprising: a first controller configured to generate control signals that cause the first input bus to provide a respective non-zero input in the first batch of inputs to each of the first plurality of multiply-accumulate cells;

2. 10. The hardware integrated circuit of claim 1, further comprising: a second computation unit configured to generate outputs of a second, different neural network layer in response to processing non-zero inputs in a different batch of inputs compared to the first batch of inputs.

3. The method further includes a second calculation unit, the second calculation unit comprising: a second plurality of multiply-accumulate cells; a second input bus coupled to each of the second plurality of multiply-accumulate cells; and a second controller configured to generate control signals that cause the second input bus to provide a respective non-zero input in a second batch of inputs to each of the second plurality of multiply-accumulate cells.

4. 4. The hardware integrated circuit of claim 3, wherein the second computational unit is configured to generate outputs of a second neural network layer in response to processing non-zero inputs in the second batch of inputs with the second plurality of multiply-accumulate cells.

5. The first computing unit: receiving a first batch of the inputs to be processed through the first neural network layer and a corresponding first set of weights for the first neural network layer; 5. The hardware integrated circuit of claim 4, configured to process each non-zero input in the first batch of inputs with i) a corresponding weight in the first set of weights and ii) a corresponding multiply-accumulate cell that receives the non-zero input from the first input bus.

6. 6. The hardware integrated circuit of claim 5, wherein the first computation unit further includes a first weight memory coupled to each multiply-accumulate cell of the first plurality of multiply-accumulate cells, the first weight memory configured to store the first set of weights.

7. Each of the first plurality of multiply-accumulate operation cells comprises: receiving respective non-zero inputs from the first input memory via the first input bus; receiving corresponding weights from the first weight memory for the first neural network layer; 7. The hardware integrated circuit of claim 6, configured to perform multiplication of each non-zero input with the corresponding weight to generate the output for the first neural network layer.

8. The second computing unit further comprises: a second input memory coupled to the second input bus, the second input memory configured to store only non-zero inputs in the second batch of inputs, the second computing unit further comprising:

8. The hardware integrated circuit of claim 7, further comprising: a second weight memory coupled to each multiply-accumulate cell of the second plurality of multiply-accumulate cells, the second weight memory configured to store the second set of weights.

9. Each of the second plurality of multiply-accumulate operation cells comprises: receiving respective non-zero inputs from the second input memory via the second input bus; receiving corresponding weights from the second weight memory for the second neural network layer; 9. The hardware integrated circuit of claim 8, configured to perform multiplication of each non-zero input with the corresponding weight to generate the output for the second neural network layer.

10. 1. A method implemented using an integrated circuit that implements a neural network including multiple neural network layers, comprising: identifying each non-zero activation value from among the plurality of activations; sequentially providing each non-zero activation value to an input data bus of said integrated circuit; providing each of a plurality of multiply accumulate cells (MACs) of the integrated circuit with a respective non-zero activation value from the input data bus; generating an output of a neural network layer based on a matrix multiplication performed by each of the plurality of MACs using the respective non-zero activation values.

11. Identifying each non-zero activation value from among the plurality of activations comprises: receiving a data stream of input activations at a computational tile of the integrated circuit; and identifying a respective non-zero activation value based on an analysis of each input activation in the data stream of input activations.

Images