Exploiting Input Data Sparsity in Neural Network Computational Units

Abstract

The present disclosure relates to utilizing input data sparsity in a neural network computational unit. A computer-implemented method includes receiving input activations by a computational device and determining, by a controller of the computational device, whether each input activation has a zero value or a non-zero value. The method further includes storing at least one input activation in a memory bank of the computational device. Storing the at least one input activation includes generating an index, the index including one or more memory address locations having non-zero input activation values. The method still further includes providing, by the controller, the at least one input activation from the memory bank onto a data bus accessible by one or more units of a computational array. The activation is provided at least in part from a memory address location associated with the index.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201710908258.5, filed on September 29, 2017. Technical Field

[0003] This application relates to exploiting input data sparsity in neural network computational units. Background Art

[0004] This specification relates to performing machine learning calculations using specialized compute units.

[0005] A neural network is a machine learning model that uses one or more layers to generate an output, such as a classification, for a received input. Some neural networks also include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or output layer of the network. Each layer of the network generates an output from the received input based on the current values of its corresponding set of parameters.

[0006] Some neural networks include one or more convolutional neural network layers. Each convolutional neural network layer has an associated set of kernels. Each kernel includes values established by a user-created neural network model. In some implementations, a kernel recognizes specific image contours, shapes, or colors. A kernel can be represented as a matrix structure of weight inputs. Each convolutional layer can also process a set of activation inputs. The activation input set can also be represented as a matrix structure. Summary of the Invention

[0007] One way to perform convolution calculations requires many matrix multiplications in a large dimensional space. A processor or controller device in a computational unit can perform matrix multiplications using brute force. For example, although this is computationally and time-intensive, a processor can repeatedly compute the sums and products of the convolution operations. However, the extent to which a processor can parallelize calculations is limited by its architecture.

[0008] Innovative aspects of the subject matter described herein can be embodied in a computer-implemented method. The method includes receiving, by a computing device, a plurality of input activations, at least a portion of the plurality of input activations being provided from a source external to the computing device, and determining, by a controller of the computing device, whether each of the plurality of input activations has one of a zero value and a non-zero value. The method further includes storing at least one of the input activations in a memory bank of the computing device, wherein storing the at least one input activation includes generating, by the controller, an index, the index including one or more memory address locations having non-zero input activation values. The method further includes providing, by the controller, at least one input activation from the memory bank onto a data bus accessible by one or more units of a computing array, wherein the activation is provided at least in part from a memory address location associated with the index.

[0009] In some implementations, the index is created based on a bitmap comprising a plurality of bits, wherein each bit of the bitmap indicates at least one of a non-zero input activation value or a zero input activation value. In some implementations, the method further includes providing a first input activation having a non-zero value to at least one unit for performing a computation using the non-zero value, and subsequently providing a second input activation having a zero value, and preventing, in the at least one unit, the computation that would otherwise be performed using the zero value.

[0010] In some implementations, preventing occurs in response to the controller determining that the input activation is provided from a memory address location not associated with the index. In some implementations, the method further includes detecting, by the controller, that the input activation is provided from a memory address location not associated with the index, and in response to detecting, providing a control signal to at least one element of the computation array to prevent a multiplication operation associated with the zero input activation value.

[0011] In some implementations, the method further includes mapping, by the controller, a first portion of a tensor computation using a first input activation to a first unit and mapping a second portion of the tensor computation also using the first input activation to a second unit different from the first unit. In some implementations, the method further includes sequentially providing a single input activation onto the data bus, the single input activation being accessed and selected from a memory address location associated with the index. In some implementations, providing further includes not providing an input activation having a value of zero.

[0012] Another innovative aspect of the subject matter described herein can be embodied as one or more machine-readable storage devices storing instructions executable by one or more processing devices to perform operations, the operations comprising: receiving, by a computing device, a plurality of input activations, at least a portion of the plurality of input activations being provided from a source external to the computing device, and determining, by a controller of the computing device, whether each of the plurality of input activations has one of a zero value and a non-zero value. The operations further comprise storing at least one of the input activations in a memory bank of the computing device, wherein storing the at least one input activation comprises generating, by the controller, an index comprising one or more memory address locations having non-zero input activation values. The operations further comprise providing, by the controller, at least one input activation from the memory bank onto a data bus accessible by one or more units of a computing array, wherein the activation is provided at least in part from a memory address location associated with the index.

[0013] Another innovative aspect of the subject matter described herein can be embodied as an electronic system comprising: a controller disposed within a computing device, the controller comprising one or more processing devices; and one or more machine-readable storage devices for storing instructions executable by the one or more processing devices to perform operations, the operations comprising: receiving, by the computing device, a plurality of input activations, the input activations provided at least in part from a source external to the computing device; and determining, by the controller, whether each of the plurality of input activations has one of a zero value and a non-zero value. The operations further comprise storing at least one of the input activations in a memory bank of the computing device, wherein storing the at least one input activation comprises generating, by the controller, an index, the index comprising one or more memory address locations having non-zero input activation values. The operations further comprise providing, by the controller, at least one input activation from the memory bank onto a data bus accessible by one or more units of a computing array, wherein the activation is provided at least in part from a memory address location associated with the index.

[0014] The subject matter described in this specification can be implemented in certain embodiments to achieve one or more of the following advantages. In a single computing system, activations accessible from a first memory and weights accessible from a second memory can be traversed based on memory address values retrieved from registers. A controller of the computing system can compress activation data by storing only non-zero values in the first memory, thereby saving storage space and corresponding bandwidth. Matrix multiplication occurs in the computing system based at least in part on providing primarily non-zero input activations. Moreover, when the computing system uses a communication scheme that primarily includes non-zero activation values, computational efficiency can be improved or accelerated by eliminating multiplications with zeros.

[0015] Other implementations of this aspect and other aspects include corresponding systems, devices, and computer programs configured to perform the actions of the method, the computer programs being encoded on a computer storage device. The system of one or more computers can be configured using software, firmware, hardware, or a combination thereof that is installed on the system and causes the system to perform the actions during operation. The one or more computer programs can be configured using instructions that, when executed by a data processing device, cause the device to perform the actions.

[0016] The subject matter described in this specification also relates to image recognition and / or classification methods / systems. When a computing unit of a hardware computing system processes inputs to a neural network layer to perform inference computations, the disclosed techniques for exploiting input data sparsity can be used to implement the system.

[0017] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Illustration of an example computation structure including activations and parameters.

[0019] Figure 2 Illustration of example computational structures including activations with output depth greater than one and multiple parameter structures.

[0020] Figure 3 An example computing system for feeding input activations to one or more parameters is illustrated.

[0021] Figure 4 The diagram illustrates an example architecture including a memory cell that provides input activations to one or more multiply-accumulate (MAC) operators.

[0022] Figure 5 is an example flow chart of a process for reducing parameter computation and exploiting input data sparsity.

[0023] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0024] The subject matter described in this specification relates to reducing the computations that occur within a computational unit or tile of an example neural network hardware computing system. Typically, as part of computing a neural network inference, input activations are multiplied by parameter or weight values to produce output activations. Here, input and input activations can refer to data elements included in multidimensional data structures commonly used in neural networks, such as tensors, matrices, and / or data arrays. Due to the algorithmic properties of computational inference for deep neural networks, most of the input activations are zero. In other words, current computational units perform a large number of unnecessary computations, including multiplying a number (e.g., a weight) by zero (the input activation value).

[0025] This specification describes, in part, a more efficient activation storage and communication scheme and a custom architecture design for deep neural network processing, particularly for processing convolutional layers of neural networks. Unlike conventional hardware accelerators that perform dense matrix multiplication over time, this specification describes an architecture that can: 1) skip or bypass computations when zero input values are seen; and 2) reduce memory usage in compute units by storing compressed input activations that only include non-zero values. Overall, through the teachings of this specification, compute unit performance for neural network inference computations is improved and energy savings are achieved by skipping unnecessary computations.

[0026] Figure 1 The diagram illustrates an example computation structure 100 including an activation structure 102 and a parameter structure 104. Activation structure 102 may include a first data structure 102a comprising a plurality of data elements corresponding to a first input depth (labeled with subscript 0). Similarly, activation structure 102 may also include a second data structure 102b comprising a plurality of data elements corresponding to a second input depth (labeled with subscript 1). The plurality of data elements shown in data structures 102a and 102b are denoted as a0, b0, c0, d0 and a1, b1, c1, d1, respectively. Each data element (a0, a1, b0, b1, c0, d0, etc.) of data structures 102a / b is an input activation value, and each input depth corresponds to a depth of input to a neural network layer. In some implementations, a neural network layer may have an input depth of one, while in other implementations, a neural network layer may have an input depth greater than one.

[0027] The parameter structure 104 can be described in a similar manner to the activation structure 102. The parameter structure 104 includes a first data structure 104a and a second data structure 104b. Each data structure 104a / b can include multiple data elements, each of which contains a kernel value. Figure 1 As shown in FIG, a plurality of data elements corresponding to data structures 104a and 104b are indicated as x0, y0, z0 and x1, y1, z1, respectively.

[0028] As discussed above, each layer of a neural network generates an output from the input it receives based on the values of its corresponding set of operands. Like other neural network layers, each convolutional layer can process a set of activation inputs that can be represented as a matrix structure. A convolutional neural network layer will also have an associated kernel set of values, which can also be represented as a matrix structure of weights. Figure 1 , the activation structure 102 may correspond to a matrix structure having one or more activation inputs and the parameter structure 104 may correspond to a matrix structure having one or more kernel or weight parameters.

[0029] As described in more detail below, the various layers of a neural network process machine learning inference by performing a large number of calculations including matrix multiplications. The calculation process performed within a neural network layer (e.g., a convolutional layer) can include multiplying an input activation (i.e., a first operand) by a weight (i.e., a second operand) over one or more cycles and performing accumulation of the product over many cycles. An output activation is generated based on the multiplication and accumulation operations performed on these two operands.

[0030] As shown, equation 106 provides an example series-based mathematical operation that may be performed when an input activation associated with a particular data element of activation structure 102 is multiplied by a kernel value or weight / parameter associated with a particular data element of parameter structure 104. For example, in equation 106, when index "i" is equal to 0, the input activation associated with data element a0 of activation structure 102 is multiplied by the weight / parameter associated with data element x0 of parameter structure 104. Moreover, because equation 106 is partially a series-based equation, additional multiplication operations may occur between operand sets corresponding to other data elements of activation structure 102 and parameter structure 104. In some implementations, the multiplication of the operand sets may produce partial sums 106a / b for a particular input feature or activation. Thus, as shown in equation 106, the partial sums may be added to produce an output feature.

[0031] A neural network can be embodied as one or more hardware computing systems that include multiple computing units configured to accelerate the amount of machine learning inference work at a network layer. Each computing unit can handle a subset of the computations for a given layer. In some implementations, structure 100 can be embodied as one or more computing units, each including at least two memories and one or more multiply-accumulate (MAC) cells, which can be collectively formed into a MAC array (described below).

[0032] In one example, a first memory bank 108 of the example computational unit stores data associated with the activation structure 102 and can be configured to receive and write input activation values to memory address locations within the memory bank 108. Similarly, a second memory bank 110 of the example computational unit stores data associated with the parameter structure 104 and can be configured to receive and write weight values to memory address locations within the memory bank 110. In this example, each element of the data elements 102a (e.g., a0, b0, c0, d0) can be stored at a corresponding memory address of the first memory bank 108. Similarly, each element of the data elements 104a (e.g., x0, y0, z0) can be stored at a corresponding memory address of the second memory 110.

[0033] In some implementations, the first memory bank 108 and the second memory bank 110 are each volatile memory cells. In some other implementations, the memory bank 108 and the memory bank 110 are each non-volatile memory cells. The memory banks 108, 110 may also be another form of computer-readable storage medium, such as a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration.

[0034] Typically, a computing unit of a hardware computing system may include one or more registers to keep track of memory address values. Data elements of the matrix structure corresponding to the activation structure 102 may be accessed from the first memory group 108 and data elements of the matrix structure corresponding to the parameter structure 104 may be accessed from the second memory group 110. An example control device of a computing module / computing unit may access and / or traverse the data elements of the matrix structure based on the address values obtained from the one or more registers. An example computing unit / module including an example control device, an activation structure 102, a parameter structure 104, a first memory group 108, and a second memory group 110 is described below in conjunction with Figure 3 is described in more detail in .

[0035] Furthermore, additional details and description regarding a hardware computing system for accelerating neural network tensor computations and matrix multiplication for neural network inference workloads are described in U.S. patent application Ser. No. 15 / 335,769, entitled “Neural Network Compute Tile,” filed on Oct. 27, 2016. The entire disclosure of U.S. patent application Ser. No. 15 / 335,769 is incorporated herein by reference.

[0036] Figure 2 The diagram illustrates an example computation structure 200 including an activation structure and multiple parameter structures for an output feature depth greater than one. In some implementations, a neural network may have multiple layers that generate outputs with multiple output feature depths. In some implementations, each parameter structure may be responsible for a corresponding one of the output depths. Thus, computation structure 200 depicts a scalable computation structure in which additional parameter structures 104a / b / c are added to facilitate computations associated with N output depths. N is variable and may have an integer value ranging from, for example, 1 to 5, or alternatively, 1 to N depending on the preferences or needs of the computation system designer.

[0037] As shown in data path 105, the respective input activation values for the elements associated with data structure 102a can be fed into each parameter structure 104a / b / c for calculation performed by the multiplier associated with the corresponding parameter structure 104. Each parameter structure 104 can then pass the activation values received from its left neighbor to its right neighbor in a pipelined manner. Alternatively, activations can be provided and consumed by each parameter structure 104 simultaneously.

[0038] Matrix 202 can represent an example matrix structure corresponding to activations 102. More specifically, element row 202a can correspond to data structure 102a and element row 202b can correspond to data structure 102b. Typically, and by way of example, first parameter structure 104(1) is accessed to perform computations associated with space 206 and second parameter structure 104(2) is accessed to perform computations associated with space 208. Although not shown, additional computations corresponding to the z dimension can also be performed. As an example, element row 202a can be in the R plane of an RGB image and element row 202b can be in the G plane of the same RGB image. An example convolutional layer of a neural network typically produces multiple output features. Example output features can include an output feature for classifying an apple and another output feature for classifying a banana. With respect to data structure 204, spaces 206 and 208 can represent different planes for different classifications.

[0039] Figure 3The diagram illustrates an example computing system 300 for feeding input activations to one or more parameter structures. The computing system 300 generally includes a controller 302 that provides one or more control signals 310 to cause input activations for the activation structure 102 to be stored in or retrieved from a memory address of the memory bank 108. Similarly, the controller 302 also provides one or more control signals 310 to cause weights for the parameter structures 104a / b / c to be stored in or retrieved from a memory address of the memory bank 110. The computing system 300 further includes one or more multiply-accumulate (MAC) cells / units 304, an input activation bus 306, and an output activation bus 308. The control signals 310 can, for example, cause the memory bank 108 to provide one or more input activations to the input activation bus 306, cause the memory bank 110 to provide one or more weights to the parameter structures 104a / b / c, and / or cause the MAC cell 304 to perform a computation that produces output activations that are provided to the output activation bus 308.

[0040] The controller 302 may include one or more processing units and memory. In some embodiments, the processing unit of the controller 302 may include one or more processors (e.g., a microprocessor or a central processing unit (CPU)), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a combination of different processors. In alternative embodiments, the controller 302 may include other storage or computing resources / devices (e.g., buffers, registers, control circuits, etc.) to provide additional processing options for performing one or more of the determinations and calculations described in this specification.

[0041] In some implementations, the processing unit of the controller 302 executes programmed instructions stored in the memory to cause the controller 302 and the computing system 300 to perform one or more functions described herein. The memory of the controller 302 may include one or more non-transitory machine-readable storage media. Non-transitory machine-readable storage media may include solid-state memory, magnetic and optical disks, portable computer disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (e.g., EPROM, EEPROM, or flash memory), or any other tangible medium capable of storing information.

[0042] In general, computing system 300 is an example computing unit or module and can include additional hardware structures to perform computations associated with multi-dimensional data structures such as tensors, matrices, and / or data arrays. In some implementations, input activation values can be preloaded into memory bank 108 for activation structure 102 and weight values can be preloaded into second memory bank 110 using data values received by computing system 300 arriving at computing system 300 from an external or higher-level control device associated with the neural network hardware computing system.

[0043] Instructions, inputs or input activations, and weights can be provided to the system 300 from external sources, such as external input / output (I / O) devices or high-level control devices associated with the neural network hardware computing system. In some implementations, one or more data buses provide data communication between the external sources (e.g., control devices) and the system 300. The data buses are used to provide instructions, inputs or activations, and weights from example I / O devices to each of the multiple systems 300 or between multiple computing modules (e.g., multiple systems 300) included in the hardware computing system for the neural network.

[0044] The system 300 is capable of receiving instructions that define specific computing operations to be performed by the system 300. And

[0045] , the controller 302 can execute programmed instructions to, for example, analyze a data stream associated with received input activations. Analyzing the input activation data stream can enable the controller 302 to detect or determine whether the value associated with each input activation is a zero value or a non-zero value. In some implementations, the controller 302 analyzes the example input activation data stream and maps each detected zero activation value and each detected non-zero activation value to a bit vector or bitmap 303.

[0046] like Figure 3 As shown in , bitmap 303 can use binary values to map detected zero-valued input activations and detected non-zero-valued input activations. For example, a binary value of "0" can correspond to a detected zero input activation value and a binary value of "1" can correspond to a detected non-zero input activation value. For example, bitmap 303 can be an 8-bit bitmap, where odd bit positions including binary "1" correspond to non-zero activation values and even bit positions including binary "0" correspond to zero activation values.

[0047] Controller 302 may cause the input activations to be stored in memory bank 108. Typically, the data values stored in memory bank 108 are each written to a corresponding memory address location. The address locations in memory bank 108 may be subsequently accessed by an example control device (e.g., controller 302) when a data value, such as an input activation, is needed to perform a particular computational operation.

[0048] In some implementations, the controller 302 uses the bitmap 303 to determine which input activations to write to the memory bank 108. For example, analysis of the bitmap 303 may indicate that only activation values corresponding to bitmap positions 1, 3, 5, and 7 (non-zero values) should be written to address locations in the memory bank 108. Furthermore, the data values associated with bitmap positions 2, 4, 6, and 8 (zero values) may be either discarded or written to memory address locations that may or may not be accessed by the controller 302 when the activation values are provided to the input bus 306. Thus, the bitmap 303 may be used as a basis for compressing zero activation values, where compression occurs when zero-valued input activations are not written to memory address locations, thereby reducing overall memory usage and freeing up address locations for storing other data values.

[0049] The controller 302 may provide one or more control signals 310 to the memory bank 108 to load input activations from the memory bank 108 onto the input activation bus 306 and provide values to the array of computational units including the MAC 304. In some implementations, the bitmap 303, or a non-zero memory address index corresponding to the bitmap 303, may be referenced by the controller 302 to determine which memory address values should be accessed to provide non-zero activation values. The activation values are provided by the controller 302 from the memory bank 108 to the data bus 306.

[0050] In some implementations, the input activation is provided at least in part from a memory address location associated with an index or bitmap 303. In other implementations, the controller 302 can detect or determine whether the provided input activation has a zero value based on one of the bitmap 303 or the index. In response to making such a determination, the controller 302 can then provide a control signal to a unit in the computation array or MAC 304 to prevent, suspend, or disable unnecessary multiplication operations (e.g., multiplication by zero) from occurring. By providing a zero activation value and subsequently or simultaneously disabling the computational operations associated with the activation within the computing system 300, energy savings can be achieved.

[0051] As discussed above, the index includes all memory address locations with input activations having non-zero values. Data bus 306 is accessible by one or more units of the computation array. The units of the computation array can receive one or more non-zero activation values from data bus 306 to perform computations related to matrix multiplication based on the received activation values. In some implementations, computing system 300 will only provide input activations from memory address locations corresponding to the index address. Therefore, no zero activations will be provided to input bus 306 and, therefore, computation operations will not be disabled or prevented from occurring. When computing system 300 uses this computation scheme, computation efficiency can be improved or accelerated by eliminating zero multiplications.

[0052] For a given computation cycle, computing system 300 may require access to elements of activation structure 102 and parameter structure 104 to perform multiplication operations associated with inference computations for a neural network layer. As noted above, specific memory address values for memory bank 108 and memory bank 110 may correspond to elements of activation structure 102 and parameter structure 104, respectively.

[0053] For a cycle in which a computation is performed, controller 302 will provide one input activation value at a time and the array of computational cells, including MAC cell 304, will multiply the activation by the weights to produce a different output activation for a given input activation. Each element of the array of computational cells (described above as a parameter structure) or MAC cell 304 can be responsible for a different output depth of the neural network layer. In general, whenever controller 302 detects a zero activation value, controller 302 may: 1) not store the activation value in memory bank 108; 2) not provide the activation value, or 3) provide a value and provide a control signal to a particular computational cell so that the cell does not perform a multiplication operation corresponding to the zero activation value.

[0054] The array of computational units is fully controlled by controller 302, which can determine when to skip or prevent specific computations based on the detection of zero activation values. Thus, no additional complex hardware structure within the array of computational units is required to skip specific computations. Furthermore, input activation values can be analyzed upon arrival at computing system 300 for storage in memory bank 108. In response to analyzing the input activations, controller 302 can execute instructions to effectively compress the activation data by storing only non-zero values in memory bank 108, thereby saving memory storage space and corresponding bandwidth.

[0055] When the computing system 300 receives the input activations and weights, the controller 302 may, for example, perform one or more direct memory access operations. The execution of these memory access operations includes storing the input activations corresponding to the dimension elements of the activation structure 102 in the address locations of the memory group 108. Similarly, the controller 302 may also store the parameters corresponding to the dimension elements of the parameter structure 104 in the address locations of the memory group 110. In addition to the bitmap 303, the controller 302 may further include one or more address registers to hold the memory addresses of the specific input activations (e.g., having a zero value or a non-zero value) to be pre-fetched. Moreover, the one or more registers will also store the memory addresses of the corresponding weights to be multiplied by the specific input activations.

[0056] As discussed above, the controller 302 identifies memory addresses for non-zero activation values based in part on the bitmap 303. In some implementations, the controller 302 reads the bitmap 303 and, for example, determines at least two memory addresses with non-zero activation values. If the controller 302 is configured to provide a zero activation value and then skip or disable calculations for zero activation values, the controller 302 may also determine at least one memory address with a zero activation value. In this implementation, the controller 302 may reference the registers mentioned above to determine the corresponding weight (and memory address) for the first input activation and to determine the corresponding weight (and memory address) for the second input activation.

[0057] As noted above, the controller 302 maintains one or more address registers in memory. Thus, to mitigate or prevent any potential inconsistencies in operands (input activations and weights), upon detecting a zero-valued input activation, the controller 302 can disable the corresponding computational unit, skip loading the particular weight, and retrieve the appropriate corresponding weight (and memory address) for the next non-zero input activation to resume computing the output activation for the given neural network layer.

[0058] In some implementations, the output activations computed at a first neural network layer are used as input activations for a subsequent second layer in the network, such as a subsequent hidden layer or output layer of the network. Typically, each layer of the neural network generates an output from a received input according to the current values of a corresponding set of parameters. In some cases, the controller 302 can execute programmed instructions (i.e., output logic) to analyze a data stream associated with output activations provided to the output activation bus 308. Analyzing the output activation data stream can enable the controller 302 to detect or determine whether the value associated with each output activation is a zero value or a non-zero value. The controller 302 can analyze the example output activation data stream and map each detected non-zero activation value to a bitmap 305. The mapped non-zero activation values in the bitmap 305 can be used to supply only non-zero values as input activations to a subsequent computing system 300 responsible for computations associated with the subsequent second layer in the network.

[0059] In an alternative implementation, there may be some computation operations in which a single non-zero input activation is used as an operand for several multiplication operations that cover various weights for elements of a given dimension of parameter structure 104 (i.e., iterate the "x" or "y" dimension). For example, when controller 302 causes memory bank 108 to provide a first input activation (e.g., a non-zero value), parameter structure 104a receives the activation and the corresponding weight at a given address is also loaded into parameter structure 104a. Parameter structure 104a will be updated by a specific number of partial sums (e.g., represented by variable "K") that the first input activation affects over K computation cycles. As a result, for these K cycles, parameter structure 104a will not receive any additional input activations. Controller 302 can then provide a control signal to memory bank 108 to cause the next input activation to be provided to input activation bus 306.

[0060] Figure 4 The diagram illustrates an example architecture including a memory bank 108 that provides activations 404 to one or more multiply-accumulate (MAC) operators via an input bus 306. A shift register 404 may provide a shifting function whereby activations 404 are sent one at a time onto the input bus 306 to be received by one or more MAC operators in a MAC cell 304. As shown, in one implementation, activations 406 may have an activation value of zero and, therefore, may not be consumed by a MAC cell 304.

[0061] Generally, a MAC cell 304 including a MAC operator is defined as a computational unit that computes a partial sum and, in some implementations, is configured to write the partial sum data to an output bus 308. As shown, a cell 304 can be composed of one or more MAC operators. In one implementation, the number of MAC operators in a MAC cell 304 is referred to as the issue width of the cell. As an example, a dual-issue cell refers to a cell having two MAC operators that can compute the multiplication of two activation values (from memory bank 108) and two parameters (from memory 110) and perform an addition between the results of the two multipliers and the current partial sum.

[0062] As described above, the input bus 306 is a communication bus that provides input activations to the MAC operators of the linear unit (i.e., the MAC array 304). In some implementations, the same input is shared among all MAC operators. The width of the input bus 306 must be wide enough to supply input activations to the corresponding number of cells for a given MAC array 304. Consider the following example to illustrate the structure of the input bus 306. When the number of cells in the linear unit is equal to four and the activation width is equal to eight bits, the input bus 306 can be configured to provide up to four input activations per cycle. In this example, each cell in the MAC array 304 will not access data beyond the four activations provided.

[0063] In some examples, the instruction data 312 may indicate that cells of the MAC array 304 will need to use the same input activations to perform computations. This may be referred to as Zout partitioning within a cell of the MAC array 304. Similarly, Zin partitioning within a cell occurs when the MAC array 304 requires different activations to perform computations. In the former example, a single input activation is repeated four times and four activations are provided over four cycles to read from the memory bank 108. In the latter example, a read from the memory bank 108 is required every cycle.

[0064] Figure 5 is an example flow chart of a process for reducing parameter computation and exploiting input data sparsity. At block 502, computing system 300 receives an input activation having a zero activation value or a non-zero activation value. As discussed above, in some implementations, computing system 300 may receive the input activation from a host interface device or a higher-level controller of an example neural network hardware system.

[0065] At block 504, the controller 302 determines whether each input activation is a zero value or a non-zero value. In some implementations, the controller 302 analyzes the input activation data stream and maps each detected zero value and non-zero value to a bitmap 303 that includes binary values corresponding to a zero input activation value ("0") and a non-zero input activation value ("1").

[0066] At block 506, the controller 302 stores the received input activation in the memory bank 108. Storing the input activation may include the controller 302 generating an index of one or more memory address locations having input activations that include non-zero values. In some implementations, the index is created based on the bitmap 303. For example, because each bit of the bitmap 303 indicates a non-zero activation value or a zero activation value, the bitmap 303 may be referenced by the controller 302 to create an index of the memory address locations that have non-zero values when writing the input activations to the memory bank 108.

[0067] At block 508, the controller 302 provides at least one input activation from the memory bank 108 onto the data bus 306. In some implementations, the input activation is provided at least in part from the memory bank location identified in the index. As discussed above, the index identifies all memory address locations storing input activations having non-zero values. The data bus 306 is accessible by one or more units of the computation array. The units of the computation array receive one or more non-zero activation values from the data bus 306 to perform computations related to matrix multiplication. In some implementations, the computing system 300 will only provide input activations from the memory address corresponding to the index address. When the computing system 300 uses this communication scheme, computational efficiency can be improved by eliminating zero multiplications.

[0068] At block 510, in an implementation where all activation values are provided rather than only input activations from indexed addresses, the controller 302 detects that an input activation is provided from a memory address that is not associated with any indexed address that includes a non-zero activation value. In response to this detection step, the controller 302 can then provide a control signal to at least one element of the computation array to prevent multiplication operations associated with zero inputs. When the computing system 300 uses this communication scheme, energy savings can be achieved by preventing unnecessary or wasteful computations that do not generate useful results (e.g., computations that include partial sums or output activations).

[0069] Embodiments of the subject matter and functional operations described in this specification may be implemented using digital electronic circuits, tangibly embodied computer software or firmware, computer hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of the above. The embodiments of the subject matter described in this specification may be implemented as encoded on a tangible, non-transitory program carrier so as to be executed by a data processing device or to control the operation of the data processing device. Alternatively or in addition, program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of the above.

[0070] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, and the apparatus can also be implemented as special purpose logic circuitry, such as an FPGA (field programmable gate array), an ASIC (application-specific integrated circuit), a GPGPU (general purpose graphics processing unit), or some other processing unit.

[0071] Computers suitable for the execution of computer programs include, for example, a central processing unit (CPU) that can be based on a general or special microprocessor or both, or any other type. Typically, the CPU will receive instructions and data from a read-only memory (ROM) or a random access memory (RAM) or both. The essential elements of a computer are a CPU for carrying out or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, or will be operatively coupled to receive data therefrom or to transmit data thereto. However, a computer need not have such a device.

[0072] Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; and magnetic disks such as internal hard disks or removable disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0073] Although this specification contains many specific implementation details, these should not be interpreted as limiting the scope of any invention that may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Specific features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented independently in multiple embodiments or in any suitable subcombination. Moreover, although features may be described above as acting in a particular combination, or even initially claimed as such, one or more features from a claimed combination may in some cases be removed from the combination, and a claimed combination may be directed to a subcombination or variations of the subcombination.

[0074] Similarly, although operations are described in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed, to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments. It should be understood that the program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0075] Further implementation is summarized in the following example:

[0076] Example 1: A computer-implemented method comprising: receiving, by a computing device, a plurality of input activations, the input activations being provided at least in part from a source external to the computing device; determining, by a controller of the computing device, whether each of the plurality of input activations has one of a zero value or a non-zero value; storing at least one of the input activations in a memory bank of the computing device; generating, by the controller, an index comprising one or more memory address locations having input activation values having a non-zero value; and providing, by the controller, at least one input activation from the memory bank onto a data bus accessible by one or more units of a computing array, wherein the activation is provided at least in part from a memory address location associated with the index.

[0077] Example 2: The method of Example 1, wherein the index is created based on a bitmap comprising a plurality of bits, wherein each bit of the bitmap indicates at least one of a non-zero input activation value or a zero input activation value.

[0078] Example 3: The method of Example 1 or 2, further comprising providing a first input activation having a non-zero value to enable at least one unit to perform a computation using the non-zero value, and subsequently providing a second input activation having a zero value, and preventing computations that would otherwise be performed using the zero value in at least one unit.

[0079] Example 4: The method of Example 3, wherein preventing occurs in response to the controller determining that the input activation is provided from a memory address location not associated with the index.

[0080] Example 5: The method of Example 4 further includes detecting, by the controller, that the input activation is provided from a memory address location not associated with the index, and in response to the detection, providing a control signal to at least one unit of the computational array to prevent multiplication operations associated with the zero input activation value.

[0081] Example 6: A method as described in one of Examples 1 to 5, wherein the method further includes mapping, by the controller, a first part of a tensor calculation activated using a first input to a first unit and mapping a second part of the tensor calculation also activated using the first input to a second unit different from the first unit.

[0082] Example 7: The method of any of Examples 1 to 6, further comprising sequentially providing a single input activation onto the data bus, the single input activation accessing and selecting from a memory address location associated with the index.

[0083] Example 8: The method of any of Examples 1 to 7, wherein providing further comprises not providing input activations having a value of zero.

[0084] Example 9: One or more machine-readable storage devices storing instructions, the instructions being executable by one or more processing devices to perform operations, the operations comprising: receiving, by a computing device, a plurality of input activations, the input activations being provided at least in part from a source external to the computing device; determining, by a controller of the computing device, whether each of the plurality of input activations has one of a zero value or a non-zero value; storing at least one of the input activations in a memory bank of the computing device; generating, by the controller, an index comprising one or more memory address locations having input activation values that are non-zero; and providing, by the controller, at least one input activation from the memory bank onto a data bus accessible by one or more units of a computing array, wherein the activation is provided at least in part from a memory address location associated with the index.

[0085] Example 10: The machine-readable storage device of Example 9, wherein the index is created based on a bitmap comprising a plurality of bits, wherein each bit of the bitmap indicates at least one of a non-zero input activation value or a zero input activation value.

[0086] Example 11: The machine-readable storage device of Example 9 or 10, further comprising providing a first input activation having a non-zero value to cause at least one unit to perform a computation using the non-zero value, and subsequently providing a second input activation having a zero value, and preventing, in at least one unit, computations that would otherwise be performed using the zero value.

[0087] Example 12: The machine-readable storage device of example 11, wherein preventing occurs in response to the controller determining that the input activation is provided from a memory address location not associated with the index.

[0088] Example 13: The machine-readable storage device of Example 12 further includes detecting, by the controller, that the input activation is provided from a memory address location not associated with the index, and in response to detecting, providing a control signal to at least one unit of the computational array to prevent multiplication operations associated with the zero input activation value.

[0089] Example 14: A machine-readable storage device as described in one of Examples 9 to 13, wherein the operation further includes mapping, by the controller, a first portion of a tensor computation activated using a first input to a first unit and mapping a second portion of the tensor computation also activated using the first input to a second unit different from the first unit.

[0090] Example 15: An electronic system comprising: a controller disposed in a computing device, the controller comprising one or more processing devices; and one or more machine-readable storage devices for storing instructions, the instructions being executable by the one or more processing devices to perform operations, the operations comprising: receiving, by the computing device, a plurality of input activations, the input activations being provided at least in part from a source external to the computing device; determining, by the controller, whether each of the plurality of input activations has one of a zero value or a non-zero value; storing at least one of the input activations in a memory bank of the computing device; generating, by the controller, an index comprising one or more memory address locations having input activation values having a non-zero value; and providing, by the controller, at least one input activation from the memory bank onto a data bus accessible by one or more units of a computing array, wherein the activation is provided at least in part from a memory address location associated with the index.

[0091] Example 16: The electronic system of Example 15, wherein the index is created based on a bitmap comprising a plurality of bits, wherein each bit of the bitmap indicates at least one of a non-zero input activation value or a zero input activation value.

[0092] Example 17: The electronic system of Example 15 or 16, further comprising providing a first input activation having a non-zero value to enable at least one unit to perform a calculation using the non-zero value, and subsequently providing a second input activation having a zero value, and preventing calculations that would otherwise be performed using the zero value in at least one unit.

[0093] Example 18: The electronic system of Example 17, wherein preventing occurs in response to the controller determining that the input activation is provided from a memory address location not associated with the index.

[0094] Example 19: The electronic system of Example 17 or 18 further includes detecting, by the controller, that the input activation is provided from a memory address location not associated with the index, and in response to the detection, providing a control signal to at least one unit of the computational array to prevent multiplication operations associated with the zero input activation value.

[0095] Example 20: An electronic system as described in one of Examples 15 to 19, wherein the operation further includes mapping, by the controller, a first portion of a tensor calculation activated using a first input to a first unit and mapping a second portion of the tensor calculation also activated using the first input to a second unit different from the first unit.

[0096] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method for performing neural network computations for a neural network comprising a plurality of layers, the method comprising: receiving, by a computing device, a plurality of inputs for processing at a first layer of the neural network, the plurality of inputs being provided at least in part from a source external to the computing device; determining, by a controller of the computing device, whether each of the plurality of inputs has one of a zero value or a non-zero value; determining, by the controller, a mapping of a subset of inputs to address locations of a memory bank at the computing device based on the determined value of each input, wherein inputs having a value of zero are excluded from the subset; storing the subset of inputs in the memory bank using the determined mapping of the subset of inputs to address locations of the memory bank; and Inputs from a subset of the inputs are provided by the controller from the memory bank to a data bus coupled to one or more multiply-accumulate units of a computation array, wherein the inputs are provided from memory address locations to which the subset of inputs are mapped.

2. The method according to claim 1, wherein Determining the mapping of a subset of the inputs comprises: generating a bitmap comprising a plurality of bits, wherein each of the plurality of bits of the bitmap indicates at least one of a non-zero input value or a zero input value; and Wherein storing the subset of the input comprises storing the subset of the input based on the bitmap.

3. The method according to claim 1, further comprising: providing the input based on the determined value of the input, performing a calculation by a first multiply-accumulate unit using the determined value of the input, Then providing a second input having a value of zero, and Calculations that would otherwise be performed using the zero value are prevented in the second multiply-accumulate unit.

4. The method according to claim 3, wherein: Preventing occurs in response to the controller determining that the second input is provided from a memory address location that does not store an input having a value greater than zero.

5. The method according to claim 4, further comprising: detecting, by the controller, that the second input is provided from a memory address location that is not associated with the memory address locations to which the subset of inputs is mapped, and In response to detecting, a control signal is provided to the second multiply-accumulate unit to prevent a multiplication operation using the second input.

6. The method according to claim 1, wherein The method further comprises: mapping, by the controller, a first portion of a tensor computation to a first multiply-accumulate unit, the first portion using a first input from the subset of inputs, and A second portion of the tensor computation is mapped to a second multiply-accumulate unit different from the first multiply-accumulate unit, the second portion also using the first input in the subset of the inputs.

7. The method according to claim 1, further comprising: A single input is sequentially provided onto the data bus, the single input being obtained from the memory address locations to which the subset of the inputs is mapped.

8. The method according to claim 1, wherein Providing the inputs from the subset of the inputs to the data bus includes providing no inputs having a value of zero.

9. One or more non-transitory machine-readable storage devices for storing instructions executable by one or more processing devices of a controller of a computing device to cause performance of operations comprising: receiving, by the computing device, a plurality of inputs for processing at a first layer of a neural network, the plurality of inputs being provided at least in part from a source external to the computing device; determining, by the controller, whether each of the plurality of inputs has one of a zero value or a non-zero value; determining, by the controller, a mapping of a subset of inputs to address locations of a memory bank at the computing device based on the determined value of each input, wherein inputs having a value of zero are excluded from the subset; storing the subset of inputs in the memory bank using the determined mapping of the subset of inputs to address locations of the memory bank; and Inputs from a subset of the inputs are provided by the controller from the memory bank to a data bus coupled to one or more multiply-accumulate units of a computation array, wherein the inputs are provided from memory address locations to which the subset of inputs are mapped.

10. The machine-readable storage device of claim 9, wherein: Determining the mapping of a subset of the inputs comprises: generating a bitmap comprising a plurality of bits, wherein each of the plurality of bits of the bitmap indicates at least one of a non-zero input value or a zero input value; and Wherein storing the subset of the input comprises storing the subset of the input based on the bitmap.

11. The machine-readable storage device according to claim 9, wherein: The operations further include: providing the input based on the determined value of the input, performing a calculation by a first multiply-accumulate unit using the determined value of the input, Then providing a second input having a value of zero, and Calculations that would otherwise be performed using the zero value are prevented in the second multiply-accumulate unit.

12. The machine-readable storage device of claim 11, wherein: Preventing occurs in response to the controller determining that the second input is provided from a memory address location that does not store an input having a value greater than zero.

13. The machine-readable storage device of claim 12, wherein: The operations further include: detecting, by the controller, that the second input is provided from a memory address location that is not associated with the memory address locations to which the subset of inputs is mapped, and In response to detecting, a control signal is provided to the second multiply-accumulate unit to prevent a multiplication operation using the second input.

14. The machine-readable storage device of claim 9, wherein: The operations further include: mapping, by the controller, a first portion of a tensor computation to a first multiply-accumulate unit, the first portion using a first input from the subset of inputs, and A second portion of the tensor computation is mapped to a second multiply-accumulate unit different from the first multiply-accumulate unit, the second portion also using the first input in the subset of the inputs.

15. An electronic system for executing a neural network computation comprising a plurality of layers, the electronic system comprising: a controller located in the computing device, the controller including one or more processing devices; and one or more non-transitory machine-readable storage devices for storing instructions executable by the one or more processing devices to cause performance of operations comprising: receiving, by the computing device, a plurality of inputs for processing at a first layer of the neural network, the plurality of inputs being provided at least in part from a source external to the computing device; determining, by the controller, whether each of the plurality of inputs has one of a zero value or a non-zero value; determining, by the controller, a mapping of a subset of inputs to address locations of a memory bank at the computing device based on the determined value of each input, wherein inputs having a value of zero are excluded from the subset; storing the subset of inputs in the memory bank using the determined mapping of the subset of inputs to address locations of the memory bank; and Inputs from a subset of the inputs are provided by the controller from the memory bank to a data bus coupled to one or more multiply-accumulate units of a computation array, wherein the inputs are provided from memory address locations to which the subset of inputs are mapped.

16. The electronic system according to claim 15, wherein: Determining the mapping of a subset of the inputs comprises: generating a bitmap comprising a plurality of bits, wherein each of the plurality of bits of the bitmap indicates at least one of a non-zero input value or a zero input value; and Wherein storing the subset of the input comprises storing the subset of the input based on the bitmap.

17. The electronic system according to claim 15, wherein: The operations further include: providing the input based on the determined value of the input, performing a calculation by a first multiply-accumulate unit using the determined value of the input, Then providing a second input having a value of zero, and Calculations that would otherwise be performed using the zero value are prevented in the second multiply-accumulate unit.

18. The electronic system according to claim 17, wherein: Preventing occurs in response to the controller determining that the second input is provided from a memory address location that does not store an input having a value greater than zero.

19. The electronic system according to claim 17, wherein: The operations further include: detecting, by the controller, that the second input is provided from a memory address location that is not associated with the memory address locations to which the subset of inputs is mapped, and In response to detecting, a control signal is provided to the second multiply-accumulate unit to prevent a multiplication operation using the second input.

20. The electronic system according to claim 15, wherein The operations further include: mapping, by the controller, a first portion of a tensor computation to a first multiply-accumulate unit, the first portion using a first input from the subset of inputs, and A second portion of the tensor computation is mapped to a second multiply-accumulate unit different from the first multiply-accumulate unit, the second portion also using the first input in the subset of the inputs.

Images