Resequencing feature map channels using sparsity information

Abstract

Methods and systems, including computer-readable media, are described for resequencing feature map channels of a neural network layer using channel sparsity information for the layer. A computing system obtains one or more reference images and generates a histogram that includes multiple bins based on pixel values of the one or more reference images. For each layer of a multi-layer neural network the system: i) computes a sparsity value for each feature map channel of the layer based on the histogram; ii) determines a sparsity correlation among feature map channels of the layer based on each sparsity value for each feature map channel of the layer; and iii) re-sequences the feature map channels of the layer based on the sparsity correlation.

Description

RESEQUENCING FEATURE MAP CHANNELS USING SPARSITY INFORMATIONBACKGROUND

[0001] This specification generally relates to feature maps for neural networks.

[0002] In the context of machine-learning, neural networks employ one or more layers of nodes to generate an output, e.g., a classification, for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to one or more other layers in the network, e.g., other hidden layers or the output layer of the network. A layer of the neural network can generate an output from a received input in accordance with current values of a respective set of parameters. Some neural networks can be convolutional neural networks configured for image processing or recurrent neural networks (RNNs) configured for speech and language processing. Different ty pes of neural network architectures can be used to perform a variety' of tasks related to classification or pattern recognition, predictions that involve data modeling, and information clustering.

[0003] A neural network layer of a convolutional neural network can have an associated set of kernels, which may correspond to parameters or weights. The associated set of kernels are used to process inputs (e.g., a batch of inputs) through the neural network layer to generate a corresponding output of the layer for computing a neural network inference. A batch of inputs and a set of kernels can be represented as a tensor, i.e., a multidimensional array, of inputs and weights. A hardware circuit that implements a neural network includes memory' with locations that are identified by an address value. The memory' locations can correspond to elements of a tensor and the tensor elements may be traversed or accessed using control logic of the circuit. For example, the control logic can determine or compute a memory address value of an element to load or store the corresponding data value of the element.SUMMARY

[0004] In general, a convolutional neural network (CNN) can generate outputs for a given inference task (e.g., objection detection) by processing images through layers of the CNN in accordance with a respective set of weights for each of the layers that receive pixels or activated values of the image. Convolutional neural networks are trained to detect certain “features” such as straight lines, edges, curves, or even objects associated with items in an image. Dunng an example training phase, a CNN can be trained to perform an imageprocessing task(s), such as classifying or detecting certain items in an image, e.g., detecting a fox or an airplane depicted in the image.

[0005] The process of training a CNN model and compiling the resulting program code for executing that model yields an arbitrary ordering of feature map channels for each layer of the trained CNN. For a given layer, this arbitrary ordering of feature map channels causes randomized sequences of sparse feature map channels to be adjacent to non-sparse feature map channels. More specifically, the random ordering of sparse feature map channels that are interspersed among non-sparse feature map channels degrades computational efficiency of integrated circuits that perform the linear algebra computations for generating outputs for a given image processing task, such as item classification.

[0006] For a CNN or other artificial neural network that includes multiple neural network layers, this specification describes techniques for determining a re-sequencing of feature map channels of those neural network layers based on a determined sparsity correlation among the respective feature map channels of each neural network layer. The sparsity correlation among the feature map channels of the neural network layer is determined based on the respective sparsity value for each feature map channel of the layer. The feature map channels of each layer are re-sequenced during a compile time that precedes a run-time inference task. More specifically, a system uses a special-purpose compiler to re-sequence the feature map channels in response to compiling program code for executing the CNN model.

[0007] One aspect of the subject matter described in this specification can be embodied in a computer-implemented method for processing input images through multiple layers of a neural network. The method includes obtaining one or more reference images and generating a histogram including multiple bins based on pixel values of the one or more reference images. For each layer of the multiple neural network layers, the method includes: i) computing a sparsity value for each feature map channel of the layer based on the histogram; ii) determining a sparsity correlation among feature map channels of the layer based on each sparsity' value for each feature map channel of the layer; and iii) re-sequencing the feature map channels of the layer based on the sparsity correlation.

[0008] These and other implementations can each optionally include one or more of the following features. For example, in some implementations, computing a sparsity value for each feature map channel of the layer includes computing a sparsity' value for each feature map channel based on a percentage of zero value activations of the feature map channel. The activations correspond to pixel values of features that are present in one or more of the reference images. Re-sequencing the feature map channels of a layer includes: i) configuringsubsets of sequential zeros in a channel dimension of the layer; and ii) causing a corresponding increase in activation locality in the channel dimension of the layer.

[0009] In some implementations, for each layer of the multiple neural network layers, the method further includes re-sequencing filters for the layer based on the re-sequenced feature map channels of the layer and the sparsity correlation among feature map channels of the layer. For each layer of the multiple neural network layers, the method also includes resequencing a set of kernels of each filter for the layer based on the re-sequenced feature map channels of the layer and the sparsity correlation among feature map channels of the layer.

[0010] In some implementations, determining a sparsity correlation among feature map channels includes: i) determining a measure of similarity between sparsity of a first feature map channel and sparsity of a second, different feature map channel; and ii) determining the sparsity' correlation based on the measure of similarity'. Determining a measure of similarity can includes determining a measure of similarity based on a comparative sparsity operation performed across multiple feature map channels of a neural network layer and across each neural network layer of the multiple neural network layers. Performing a comparative sparsity' operation can includes performing a comparative sparsity operation based on a sparsity' value of the first feature map channel and a sparsity value of the second, different feature map channel.

[0011] The method can further includes generating an output binary by a compiler in response to compiling program code for a neural network model. The output binary encodes the re-sequenced feature map channels of each layer of the multiple layers. Re-sequencing the feature map channels of the layer includes re-sequencing the feature map channels of the layer using the compiler in response to compiling the program code for the neural network model. The feature map channels of each layer can be re-sequenced during a compile time that precedes a run-time inference task. The method includes processing an input image using re-sequenced feature map channels for tw o or more layers of the neural netw ork.

[0012] The method further includes generating an output representing a result of the runtime inference task based on neural network computations that are performed when the input image is processed through the two or more layers of the neural network using the resequenced feature map channels for the two or more layers. In some implementations, obtaining the one or more reference images and generating the histogram includes obtaining the one or more reference images and generating the histogram during a compile time when program code is compiled for execution on a computing device. The program code is forexecuting a machine-learning model that is based on the neural network including the multiple layers.

[0013] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices. A system of one or more computers can be so configured by virtue of software, firmware, hardware, or a combination of them installed on the system that in operation causes the system to perform the actions. One or more computer programs can be so configured by virtue of having instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0014] The subject matter described in this specification can be implemented in particular embodiments to realize one or more of the following advantages. Techniques are described for implementing a special-purpose compiler that integrates re-sequencing capabilities for resequencing feature map channels of a neural network layer. The compiler is configured to generate executable code for a model binary that enables a special-purpose hardware integrated circuit to process inputs through a neural network layer with improved efficiency relative to prior approaches. For example, a system’s compiler uses sparsity attributes of feature map channels of a neural network layer to re-sequence the feature maps for that layer.

[0015] The system then generates a model binary' that encodes the re-sequenced feature map channels for each layer across a multi-layer neural network. This unique encoding of the model binary enables an integrated circuit that executes the model binary to efficiently group several instances of non-sparse operations (e g., multiplication by non-zeros) separately from instances of sparse operations (e.g., multiplication by zeros). This enables the integrated circuit to dynamically disable or clock-gate certain compute units or portions of its circuitry upon detecting that operations for a neural network inference will involve negligible computations that use sparse values, such as multiplication by zero.

[0016] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Fig. 1 is a block diagram of an example computing system for performing operations relating to a neural network.

[0018] Fig. 2 shows example feature maps of a neural network layer.

[0019] Fig. 3 shows an example feature map re-sequencing operation.

[0020] Fig. 4 shows example aspects of re-sequencing feature maps of adjacent channels of a neural network.

[0021] Figs. 5A and 5B show example aspects of re-sequenced feature map channels with reference to a compute cell of an integrated circuit.

[0022] Fig. 6 shows an example re-sequencing of feature map channels of a neural network layer with reference to an associated set of kernels for the layer.

[0023] Fig. 7 is an example process for re-sequencing feature map channels of a neural network layer.

[0024] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0025] Fig. 1 is a block diagram of an example computing system 100 for performing operations relating to re-sequencing feature map channels of an artificial neural network.

[0026] The system 100 includes a compiler 102 configured to generate executable code, which can include assembly code, machine code, object code, or other code corresponding to an example low-level programing language. Compiler 102 can be an example program that translates computer code written in one programming language (the source language) into another language (the target language). In general, compiler!02 translates source code from a high-level programming language to a low-level programming language, such as assembly or machine code, to create an executable program.

[0027] The system 100 uses compiler 102 to generate executable code based on compiler operations performed on input data 104. As indicated above, the input data 104 can include program / source code 106 (“source code 106”) written using a high-level programming language. The compiler 102 generates the executable code in response to compiling the source code 106 based in part on a set of compiler operations 108. In some implementations, the compiler operations 108 are a set of core compiler operations generally (or normally) performed by some, most, or any example compiler(s).

[0028] For instance, the compiler operations 108 can include multiple compiler phases or stages, such as lexical analysis, syntactic analysis and / or parsing, semantic analysis, and code generation, including intermediate code generation and / or code optimization. In the example of Fig. 1, compiler 102 can be a special-purpose program that translates source code 106 into machine code, bytecode, or another machine-readable programming language. The sourcecode 106 can be written in an example high-level, human-readable programming language such as Python, Java, or C++.

[0029] In the example of Fig. 1, system 100 and the compiler 102 are integrated in, or accessible by, an example computing device 107, such as a consumer electronic device or mobile / client device. In some implementations, computing device 107 is represented byexample items such as tablets, laptops, Chromebooks. eNotebooks, Netbooks, or other related mobile computers. In some implementations, the system 100, compiler 102, and input data 104, including source code 106, are accessed using a desktop computer, network server, or related cloud-based asset.

[0030] In some implementations, the source code 106 is example code generated using open-source platforms such as TensorFlow® or OpenAI®. In the context of machine learning, the compiler 102 is configured to generate executable code in the form of a model binary 110 that is used to execute an example machine-learning (ML) model. For example, the ML model is based on a multi-layer neural network. The multi-layer neural network is a software construct. Thus, the neural network can be any type of artificial (e.g.. non-organic) neural network, such as a CNN or a recunent neural network (RNN). The compiler 102 includes a re-sequencing engine 112 that performs operations specific to re-sequencing feature map channels of an artificial neural network. The compiler 102 can include the resequencing engine 112 as a local resource, be configured to access the re-sequencing engine 112, or both.

[0031] For a neural network that includes multiple neural network layers, the system 1 0 uses the re-sequencing engine 112 to determine a re-sequencing of a respective set of feature map channels of each neural network layer. As explained below with reference to Fig. 2 and Fig. 3, the system 100 can determine the re-sequencing based at least on a computed sparsitycorrelation among the respective feature map channels of the neural network layer. For clarity, feature map and feature map channel are used interchangeably in this document to refer to feature maps of a channel of a neural network layer.

[0032] The system 100 leverages the re-sequencing engine 112 to implement a specialpurpose compiler 102 that integrates sparsity-based re-sequencing capabilities for resequencing feature map channels of the neural network layer. The re-sequencing engine 1 12 includes an image analysis module 120 and feature map sparsity7analyzer 124, each of which are described in detail below. The re-sequencing engine 112 leverages each of these features to re-sequence a set of feature map channels for a neural network layer based on the determined sparsity attributes of the feature map channels of the neural network layer.

[0033] For example, the image analysis module 120 processes one or more reference images 116 and generates corresponding outputs that are used to determine sparsity attributes of neural network layers involved in processing the reference images 116. The image analysis module 120 can process the reference images 116 as an initial step for determining a corresponding sparsity value of each feature map channel of a neural network layer. In some implementations, the image analysis module 120 processes the reference images 116 by performing an inference pass on each reference image (122). Performing the inference pass can involve computing a corresponding layer output for generating an example inference output for a reference image 116.

[0034] For example, the image analysis module 120 can process a reference image 116 by generating an image recognition output based on inference computations performed for a machine-learning task. As a precursor to performing the inference computation, the system 100 can generate one or more histograms based on pixel values of the reference image(s) 116. The histogram can include multiple bins derived from pixel values of the reference image(s) 116. In some implementations, the image analysis module 120 processes the reference image(s) 116 and generates the inference output using, for example, the histogram bins and an intermediate version 121 of the model binary 110. The intermediate version 121 of model binary 110 can be a portion of executable code generated by the compiler 102 based on the core compiler operations 108 performed on source code 106.

[0035] In more detail, a re-sequencing flag 114 is passed to the re-sequencing engine 112 to trigger re-sequencing operations to generate a final target binary. In some implementations, the re-sequencing engine 112 can use computational outputs of each neural network layer to set a re-sequencing flag 114. For example, the re-sequencing flag 114 can be set based on layer outputs computed using the image analysis module 120. The resequencing flag 114 is used to trigger sparsity analysis and / or sparsity calculations performed by sparsity & correlation logic for the feature map sparsity analyzer 124 of the re-sequencing engine 112. In some implementations, the re-sequencing flag 114 is set based on a threshold measure of sparsity. The threshold measure of sparsity is defined based on a sparsity value(s) that is user specified, dynamically determined, or both.

[0036] In some implementations, the measure of sparsity is observed or detected from a kernel filter of weights (or parameters) that is applied to process pixel bins of a reference image through a corresponding neural network layer. Relatedly, the threshold measure of sparsity can be observed or detected using a quantity of zero (sparse) values that are computed as outputs of the neural network layer. For example, the outputs can be computedin response to convolving the kernel filter of weights over inputs derived from pixel values of a reference image 116.

[0037] The feature map sparsity analyzer 124 includes a feature map sparsity module 126 and a sparsity correlation module 128. Modules 126 and 128 cooperate to determine overall sparsity attributes of feature map channels of a neural network layer. Additionally, the modules 126 and 128 are used to generate sparsity & correlation feature maps 130 that indicate the overall sparsity attributes of the feature map channels. An example sparsity & correlation feature map 130 is described with reference to Fig. 3. The re-sequencing engine 112 re-sequences the ordering of feature map channels of each respective neural network layer, including associated filters / kemels (132). For example, the re-sequencing engine 112 re-sequences the ordering based on sparsity computations and analysis performed by the feature map sparsity analyzer 124, which uses modules 126 and 128 to compute the sparsity attributes for that layer. Modules 126 and 128 are described in more detail below at least with reference to Fig. 2 and Fig. 3.

[0038] The system 100 then generates executable code (“model binary 110”) that encodes the re-sequenced feature map channels for each layer across a multi-layer neural network. For example, an output binary such as model binary' 110 is generated by compiler 102 in response to compiling program code 106 for a neural network model, e.g., a CNN model. The model binary 110 generated by compiler 102 is ultimately provided to and / or loaded on an example hardware integrated circuit 134 (“integrated circuit 134”). The model binary 110 is encoded or otherwise configured to reflect the re-sequenced, optimized feature map channels for one or more layers of the neural network.

[0039] As indicated at Fig. 1, the integrated circuit 134 can be installed on a system-on- chip (SoC) represented by a hardware circuit (or integrated circuit) of an example target user / client device, consumer electronic device, or mobile device. Each of these devices can include example items such as a smartphone 136a, tablet 136b, or laptop 136c. The devices may also include other items such as a Chromebook, eNotebook, Netbook, smart speaker, or smartwatch or related wearable device. In some implementations, the circuit 134 (and the SoC) is an integrated circuit of a desktop computer, network server, or related cloud-based asset.

[0040] Fig. 2 shows examples of a set of feature maps or feature map channels 200 for a layer of a neural network. In the example of Fig. 2, the neural network is a CNN, however other types of feature map channels and neural network architectures are also within the scope of this specification.

[0041] For context, a neural network layer can include one or more channels and each channel includes a corresponding feature map. For a given image processing application, each channel can be tuned or configured (and later used) to detect a particular feature(s) of an image or input sample during a phase of training a ML model. For example, a neural network underlying the ML model can be trained to perform one or more image processing tasks. For convolutional neural networks, the training involves convolving a kernel filter of weights (or parameters), where each filter can define a specific feature (e.g., a horizontal edge line) that is detected in, or extracted from, the input sample (e.g., an image).

[0042] In some implementations, feature map channels of convolutional neural networks are tuned or encoded, e.g., during training, to detect specific types of features such as straight lines, edges, contours, colors, or even objects. As a result of this training process, each channel can develop a certain sensitivity to particular features that are detectable in an image in accordance with a given set of filters (or kernels). Accordingly, a feature map channel is the output of applying a filter to a given neural network layer and can represent a mapping of where specific types of features are found in an image.

[0043] More specifically, a collection of feature maps can be a representation of an input sample (or image) in a space, such as an embedding space, spanned by a filter bank that includes the set of filters. An example feature map for a given channel can be represented by an encoding, e.g., a numerical vector, that the neural network uses to process and detect specific features in a subsequent input sample. The encoding for a feature map channel can have a measure of sparsity. For example, the measure of sparsity can be characterized by a number of zeros in a vector of numeric values that represents or corresponds to a given feature map channel.

[0044] As shown at Fig. 2. a neural network ML model can be trained to detect a fox using images 202 or to detect an airplane using images 204. Existing compilers that compile the source code for the ML model arrange or sequence feature map channels as-is, e.g., based on a configuration of the feature maps after model training. In some implementations, a training phase of an ML model causes default, erratic sparsity arrangements among the feature map channels (e.g., color channels) of a given layer. Notably, whether detecting a fox or an airplane, sparsity patterns can be generally consistent irrespective of the item (fox or airplane) being detected in an image input. Thus, irrespective of the type of image or contents of the image, existing compilers determine an arbitrary' ordering for a sequencing of channels for feature maps of a given neural network layer.

[0045] In the example of Fig. 2, for each of images 202, 204, an existing compiler generates a first (default) channel sequence 208, 212, respectively. For example, a neural network model trained on images 202 yields a default or original channel order 208. Likewise, a neural network model trained on images 204 yields a default or original channel order / sequence 212. For the reasons discussed above, these default channel sequences 208, 212 have no (or minimal) sparsity -based locality in the channel dimension for a given neural network layer. For example, the lack of sparsity locality across the channel dimension can be based on erratic and / or inconsistent sparsity arrangements that result from model training where weights / parameters and features values converge to reflect certain features or feature maps.

[0046] As described above, the re-sequencing techniques disclosed in this specification can be used to implement a special-purpose compiler 102 configured to determine a resequenced channel ordering for a given neural network layer to provide sparsity-based locality in the channel dimension. For example, after performing compiler operations 108, the system 100 can generate first channel sequences 208, 212 that have no (or negligible) locality in the channel dimension. The compiler 102 can then use the re-sequencing engine 112 to determine and / or generate second channel sequences 210, 214 that are re-sequenced to provide sparsity -based locality in the channel dimension. Thus, relative to the first channel sequences 208. 212, the second channel sequences 210, 214 represents a new, re-sequenced channel order sorted by sparsity attributes of the feature maps of the neural network layer.

[0047] In some implementations, the compiler 102 is configured to generate an intermediate instance 121 of an executable code or binary' for a neural network model based on the compiler operations 108. The first channel sequences 208, 212 can be encoded in the intermediate instance 121 of the executable code or binary and then passed to the resequencing engine 112. The re-sequencing engine 112 uses the intermediate instance 121 of the model binary' and its feature map sparsity analyzer 124 to determine sparsity attributes of feature map channels for some (or all) of the layers of the neural network model.

[0048] The re-sequencing engine 112 determines the sparsity attributes by processing a set of reference images 116 using the image analysis module 120, as described above. For example, using one or more histogram bins, the image analysis module 120 can compute a sparsity' value for each feature map channel of a given neural network layer. The resequencing engine 112 computes a first sparsity attribute for a first feature map channel 218 and a second, different sparsity attribute for a second, different feature map channel 220. Inthe example of Fig. 2, the first feature map channel 218 is sparse, whereas the second, different feature map channel 220 is non-sparse.

[0049] The feature map sparsity analyzer 124 computes a respective sparsity value for each feature map channel of a neural network layer. In some implementations, feature map sparsity analyzer 124 computes a respective sparsity' value for each feature map channel based on a percentage of zero value activations of the feature map channel. The activations can correspond to, and be generally derived from, pixel values of features that are present in one or more of the reference images 1 16, 202, 204. For example, the feature map sparsity module 126 can analyze intermediate outputs and activation values that are computed when processing input pixels of the reference images 116 through the neural network layer. The feature map sparsity module 126 can then compute a respective sparsity value for each feature map channel based on the intermediate outputs and activation values for the layer.

[0050] In some implementations, the feature map sparsity' module 126 determines one or more groupings of activated values and determines a respective sparsity' value for multiple feature map channels based on those groupings. For example, the feature map sparsity module 126 can determine the one or more groupings of activated values based on the result of applying non-linear (or linear) activation functions to accumulated values. The accumulated values result from convolution computations that are performed to process the input pixels through the neural network layer. In general, the convolution computations involve linear algebra concepts such as matrix multiplication, vector math, and dot product summations.

[0051] Fig. 3 shows an example feature map re-sequencing operation. The feature map sparsity' module 126 is configured to generate a set of sparsity' values 302, where the set includes a respective sparsity value for each feature map channel of a neural network layer. In some implementations, the feature map sparsity module 126 generates a sparsity value for a feature map channel by mapping an observed measure of sparsity to a range from 0 to 1 , where “0” indicates zero / no (or minimal) sparsity, “0.5” indicates some measure of partial (or intermediate) sparsity, and “1” indicates full sparsity (or substantial) sparsity. In the example of Fig. 2. the feature map sparsity module 126 can: i) generate a sparsity value of 0.98 for feature map channel 218 to indicate substantial sparsity and ii) generate a sparsity value of 0.52 for feature map channel 220 to indicate partial or intermediate sparsity'.

[0052] The sparsity correlation module 128 is configured to: i) receive the sparsity values; ii) process each respective sparsity value; iii) determine a re-sequencing of the feature map channels based on the processing at step ii); and iii) generate re-sequenced feature mapchannels 214 in response to determinations at step iii). For example, sparsity correlation module 128 is configured to determine sparsity correlation among feature map channels of the neural network layer based on each sparsity value for each feature map channel of the layer. In some implementations, feature map sparsity analyzer 124 generates a final sorting and re-sequencing of feature map channels for a given layer based on a sorting function applied by the sparsity correlation module 128.

[0053] For example, the sparsity correlation module 128 can sort a set of feature map channels by arranging the channels to have a sparsity ordering 304 that coincides with the respective sparsity values for each channel. In some implementations, the set of sparsity' values 302 and the sparsity' ordering 304 combine to form an example sparsity & correlation feature map 130. The sparsity correlation module 128 can receive a set of sparsity values and a corresponding indicator 306 that links a respective sparsity value in the set of sparsity values to a particular feature map channel. For example, the feature map sparsity' analyzer 124 can determine and / or generate a respective identifier (e g., FMC1, FMC2, etc.) for each feature map channel of a neural network layer. In some implementations, the feature map sparsity' module 126 uses the respective identifiers to generate the indicator 306.

[0054] Fig. 4 shows example aspects of re-sequencing feature maps of adjacent channels of a neural netw ork layer w ith reference to a compute cell of an integrated circuit. In the example of Fig. 4, block 402 show s a default, erratic sparsity arrangement among the feature maps of four adjacent channels 408-1. 408-2, 408-3, 408-4 of a given neural network layer. As discussed above, an example neural network model trained on reference images, such as images 202, 204 yields a default / original channel ordering or sequence as shown at block 402, where there is no sparsity -based locality in the channel dimension.

[0055] Much like the default channel order 208. 212, this default sequencing / ordering of channels 408-1, 408-2, 408-3, 408-4 has no (or negligible) sparsity-based locality in the channel dimension for its corresponding neural network layer. For example, a default ordering that alternates between sparse (e.g., "zero") and non-sparse (e.g., “non-zero”) feature maps lacks sparsity-based locality in the channel dimension. Similarly, default orderings that arbitrarily arrange sparse (e.g., ‘"zero’7) and non-sparse (e.g., "non-zero”) feature maps will lack meaningful sparsity -based locality in the channel dimension.

[0056] In some implementations, sparsity-based locality' in the channel dimension is characterized by a meaningful and / or logical arrangement of feature map channels that allow s for exploiting the sparsity locality to achieve reduced power consumption and compute overhead at the integrated circuit 134. The logical arrangement is represented by feature maporderings generated based on re-sequencing operations executed by the re-sequencing engine 112, as described above with reference to Fig. 1.

[0057] The logical arrangement or re-sequencing can include determining a first grouping of sparse (e.g., “zero”) feature map channels and determining a second, different grouping of non-sparse (e.g., “non-zero”) feature map channels. For example, block 404 illustrates the result of re-sequencing the feature maps of block 402 such that sparse (e.g., “zero”) feature map channels 408-2. 408-4 are grouped together and non-sparse (e.g., “non-zero”) feature map channels 408-3, 408-1 are grouped together.

[0058] In the example of Fig. 4, the re-sequenced feature map channels 408-2, 408-4, 408-3, 408-1 provide sparsity-based locality in the channel dimension, much like the resequenced channel orderings 210, 214 described with reference to Fig. 2. This locality can be leveraged to realize power savings at the integrated circuit 134 and to streamline computations for a layer of a neural network implemented at the integrated circuit 134. This is described in more detail below, at least with reference to Figs. 5 A and 5B.

[0059] Block 402 also includes a cell 410. which is an example compute cell of integrated circuit 134. In some implementations, the integrated circuit 134 can include N number of cells, where N is an integer greater than 1. This definition for N also applies to other examples and elements of system 100. Cell 410 includes one or more multiply accumulators (MACs) 412-1, 412-2. 412-3, 412-4 that perform arithmetic operations such as multiplication and addition. For example, each of MACs 412-1, 412-2, 412-3, 412-4 can be used to compute dot products and related matrix multiplications that are performed to convolve an input tensor of image pixel values with a weight tensor (e.g., a kernel filter of weights). In some implementations, the computations are convolution operations performed to generate an output for a convolutional neural network layer.

[0060] In some implementations, a lack of sparsity -based locality in the channel dimension leads to reduced efficiencies and / or degrades performance of the integrated circuit 134. For example, a default channel ordering that alternates between sparse (e.g., “zero”) and non-sparse (e.g., “non-zero”) feature maps or arbitrarily arranges sparse feature map channels adjacent to non-sparse feature map channels does not allow for exploiting the power savings and computational benefits of sparsity -based locality. An indication of this is shown at block 402.

[0061] Figs. 5A and 5B show additional example aspects of re-sequenced feature map channels with reference to a compute cell of an integrated circuit.

[0062] In the example of Fig. 5A, block 502 shows a result of re-sequencing feature maps to form a first grouping of adjacent non-sparse (e.g., "non-zero”) feature map channels, whereas the example of Fig. 5B, block 504 shows a result of re-sequencing feature maps to form a second, different grouping of adjacent sparse (e.g., “zero”) feature map channels. The non-zero feature map channels are characterized based on non-zero weight values in a corresponding filter for the layer, whereas the zero feature map channels are characterized based on zero weight values in a corresponding filter for the layer.

[0063] The sparse / zero feature map channels of block 504 allow for increased activation locality7in the channel dimension, based on the large chunks of consecutive zeros in the channel dimension. In some implementations, the re-sequencing engine 112 can detect or determine a certain grouping of sparse feature map channels, which allows for a certain measure of predictability7that can be exploited to realize efficiencies in computations required for accelerating a neural network model. For example, sparsity -based locality in the feature map channels allows for predicting, inferring, or otherwise detecting an upcoming sparse sequence of weight values (e.g., a sparsity attribute) of zero (or non-zero) weight values.

[0064] As indicated above, cell 410 can be one of multiple cells in a computational unit of integrated circuit 134. The computational unit can include an array of cells 410 and the integrated circuit 134 can be a special-purpose processor, such as a tensor processing unit (TPU) or neural network processor that executes a machine-learning model for processing an inference workload. This inference processing can involve computing convolutions (e.g., group convolutions) where data values and / or operands, such as input pixels and w eight values, associated with a given feature map channel are routed to cell 410 to perform the convolution based on computations performed at MACs 412-1, 412-2, 412-3, 412-4.

[0065] The integrated circuit 134 can include multiple compute tiles and each compute tile can include a corresponding controller 510 and a memory 512 that stores input pixels, activations, and a set of weights for a neural netw ork layer. The controller 510 can generate control signals 514 for controlling or triggering computations at cell 410 and for passing or broadcasting operands from the memory 512. In some implementations, each compute tile includes an input activation bus 516 coupled to a computational unit that includes multiple cells 410 and MACs 412. The controller 510 can generate control signals 514 to obtain operands stored at the memory' 512 of the compute tile.

[0066] For example, the controller 510 can generate control signals 514 to obtain an example input vector and corresponding weights stored at memory 512. Each input obtained from memory 512 is provided to input activation bus 516 for routing (e.g., direct routing) to acompute cell 410. Similarly, a set of weights can be obtained from memoir 512 and routed to a cell 410. For example, the set of weights can be routed to a cell 410 or MAC 412 using a signal path that differs from the activation bus 516. Each cell 410 can include example registers for storing activation and weight values that are used for a computation associated with a neural network layer.

[0067] In some implementations, a particular compute tile may be tasked with executing a portion of a convolution computation for a neural network layer. For example, based on its tasking, the compute tile can perform multiplication operations that use: i) an input or activation (e.g., a first operand) stored in a memory location of the integrated circuit 134 and ii) a weight value (e.g., second operand) stored in a different memory location. The multiplications are performed at each MACs 412-1. 412-2, 412-3, 412-4 of a given cell 410, using respective multiplication circuitry of each MAC.

[0068] In the example of Fig. 5B, the “zero” corresponds to operands that have a weight value of zero. The controller 510 can determine or detect that a grouping of adjacent feature map channels are sparse channels that each have a concentration of “zero” values associated with a filter for the layer. For example, the control logic of controller 510 can make this determination by analyzing instructions, opcodes, and / or data values that are provided to the tile from a higher-level controller of the integrated circuit 134. In some implementations, the instructions and opcodes are generated from the model binary 110.

[0069] To conserve resources and reduce power consumption, a multiplication between an input and weight operands can be skipped or disabled. For example, because the product of multiplication by zero yields a zero value, the control logic of controller 510 can detect prospective zero value multiplications and generate gating control signals 518 to preclude occurrence of those multiplications at the cell 410. For example, the controller 510 can use the gating control signals 518 to disable (or skip) the computation at the cell 410 or at a MAC 412. In some implementations, the controller 510 can use control signals 514 to manage read operations at memory 512 to preclude / prevent certain weight value operands from being routed to, or consumed by. the cell 410 or MACs 412-1, 412-2, 412-3, 412-4.

[0070] As shown in the example of Fig. 5B. the system 100 can leverage sparsity locality in the channel dimension to create a grouping of zeros for at least four consecutive activations. For clarity, a weight or activation value of zero for a given feature map channel indicates that a corresponding filter for that channel is not tuned to detect, from pixels of an input image, any features that are relevant to generating an inference output for that inputimage. The system 100 can then use the controller 510 to skip computations involving these zero value weights and / or activations.

[0071] In some implementations, the controller 510 is configured to dynamically disable or clock-gate certain compute units or portions of its circuitry upon detecting that operations for a neural network inference workload will involve negligible computations that use sparse / zero values, such as multiplication by zero. For example, in response to determining to skip one or more of the computations, the controller 510 can generate and / or pass a gating signal(s) 518 to the computational unit to clock gate one or more cells 410 (or MACs 412) that would otherwise be used to perform one or more of these sparse multiplications.

[0072] This determination to skip a subset of multiplication operations and clock gate at least a portion of the circuitry in the computational unit translates to significant power savings over time. For example, clock gating circuitry in the computational unit prevents circuit toggles that would otherwise consume power and reduce an amount of charge in a rechargeable battery of device 136 that uses the integrated circuit 134 to compute inferences locally at the device.

[0073] As discussed above, an array of cells 410 forms a computational unit of a tile in an application-specific / special-purpose integrated circuit (ASIC / SPIC) 134, such as a TPU, a special-purpose graphics processing unit (GPU), a hardware accelerator, or a combination of these. Irrespective of its processor implementation, the re-sequencing engine 112 can be used, with a corresponding compiler 102 for that processor, to realize power savings and reductions in compute overhead at integrated circuit 134.

[0074] Fig. 6 shows an example re-sequencing of feature map channels of a neural network layer with reference to an associated set of kernels for a neural network layer.

[0075] In the example of Fig. 6, prior approaches to performing convolution computations at a first neural network layer (e.g., layer N) were performed using the inefficient erratic sparsity arrangement for the feature map channels 602 of layer N. The activation values generated at layer A were then fed as inputs to layer N + 1, for processing using the erratic sparsity arrangement for the feature map channels 604 of layer N + 1. As discussed above, the inefficiencies of this prior approach wastes battery power and processor cycles of a device 136 that includes the integrated circuit 134.

[0076] In some implementations, the re-sequencing engine 112 identifies or detects the erratic sparsity arrangement of feature map channels 602 in layer A and generates resequenced feature map channels 612 for in layer A. Likewise, the re-sequencing engine 112 identifies or detects the erratic sparsity arrangement of feature map channels 604 of layer A +1 and generates re-sequenced feature map channels 614 for in layer TV + 1. The re-sequencing engine 112 can use at least the sparsity correlation module 128 to perform the steps of identifying or detecting the inconsistent sparsity of channels 602, 604 and to generate the resequenced feature map channels 612, 614 that have sparsity-based locality.

[0077] The re-sequencing engine 112 can also re-sequence respective kernels in a set of kernel filters 620 for a neural network layer based on the re-sequenced feature map channels of the neural network layer and the sparsity correlation among feature map channels of the layer. For example, for each layer among multiple neural network layers, the re-sequencing engine 112 can re-sequence kernel filters for the neural network layer based on the resequenced feature map channels for the layer.

[0078] In some implementations, the kernel filters are convolved over regions of an input image using pixel values that are represented as a multi-dimensional input tensor. The convolution computations yield accumulated values and an activation function can be applied to the accumulated values to generate a set of activation values. For example, activation values can be computed and generated as an output of layer TV using the re-sequenced feature map channels 612 of layer N. In some implementations, a rectified linear activation function (ReLU) is applied to compute the activation values that are generated as the output of layer N. The activation values that are generated as the output of layer TV are passed as inputs to layer TV + 1 and are processed using the re-sequenced feature map channels 614 of layer N + 1.

[0079] Fig. 7 is an example process 700 for re-sequencing feature map channels of a neural network layer included among multiple layers of an example convolutional neural network. Process 700 is also implemented or executed at system 100 using at least the compiler 102 and re-sequencing engine 112 described above with reference to Fig. 1. Hence, descriptions of process 700 will reference the above-mentioned computing resources of system 100. In some examples, the steps or actions of process 700 are enabled by programmed software instructions, firmware instructions, or both. Each type of instruction may be stored in a non-transi lory machine-readable storage device and is executable by one or more of the processors or other resources described in this specification.

[0080] Referring again to process 700, the system 100 obtains one or more reference images (702). The system 100 generates a histogram that includes multiple bins based on pixel values of the one or more reference images (704). For each of one or more layers of the multi-layer neural network, system 100 computes a sparsity value for each feature map channel of the layer based on the histogram (706). In some implementations, feature mapsparsity analyzer 124 computes a respective sparsity value for each feature map channel based on a percentage of zero value activations of the feature map channel. The activations can correspond to pixel values of features that are present in one or more of the reference images 202, 204.

[0081] For each of the one or more layers, system 100 also determines a sparsitycorrelation among feature map channels of the layer based on each sparsity value for each feature map channel of the layer (708). In some implementations, the re-sequencing engine 1 12 determines a sparsity correlation among feature map channels at least by determining a measure of similarity- between: i) sparsity of a first feature map channel (e.g., FMC2) and ii) sparsity- of a second, different feature map channel (e.g., FMC10). The re-sequencing engine 112 can then determine the sparsity correlation based on the measure of similarity. For example, the sparsity- correlation module 128 determines or computes a measure of similarity based on the respective sparsity value for the first and second feature map channels.

[0082] The measure of similarity- also provides a more granular measure of sparsity -based locality and can be determined by computing a numerical delta between the respective sparsity- values for the first and second feature map channels. For example, a sparsity value of .87 can be computed for FMC9 of CNN-lay erl and a sparsity value of .91 can be computed for FMC5 of CNN-layerl. The sparsity- correlation module 128 can use both the sparsity values (.87, .91) and the .04 delta between the sparsity values as an indicator that FMC5 and FMC9 of CNN-layerl have a sufficiently high measure of similarity and a sufficiently high measure of sparsity -based locality.

[0083] The sparsity correlation module 128 can then determine that FMC5 and FMC9 should be sequenced adjacently when the re-sequencing engine 112 re-sequences all feature map channels for CNN-layerl. In some implementations, the re-sequencing engine 112 determines sparsity- correlation and sorting among all feature map channels of a neural network layer using the same (or substantially similar) comparative sparsity operations described in the preceding example involving FMC5 and FMC9.

[0084] For each of the one or more layers, system 100 also re-sequences the feature map channels of the layer based on the sparsity correlation (710). For example, the re-sequencing engine 112 can re-sequence the feature map channels by configuring or coalescing subsets of sequential zeros in a channel dimension of a neural network layer. The re-sequencing engine 112 coalesces groups of sequential zeros to cause a corresponding increase in sparsity-based activation locality in the channel dimension of the neural network layer. The re-sequencing engine 112 can be integrated in a compiler 102 and automatically triggered to initiate re-sequencing feature map channels of multiple neural network layers in response to compiling the program code 106 for the neural network model.

[0085] As described above, the feature map channels can be re-sequenced during a compile time operation that precedes a run-time inference task. The re-sequenced feature map channels are encoded in a model binary generated by the compiler 102 as a result of the compile-time operations. The model binary can be downloaded or otherwise installed onto an integrated circuit 134 of a device 136, such as an edge device or smartphone. The integrated circuit 134 can be a TPU or hardware accelerator installed on the smartphone. The model binary' 110 is executed by the TPU to implement a neural network model.

[0086] The device 136 processes input images using the re-sequenced feature map channels for two or more layers of the neural network model implemented by the TPU. The TPU is configured to generate an output representing a result of the run-time inference task. For example, the TPU generates the output based on neural network computations that are performed when the input image is processed through the two or more layers of the neural network using the re-sequenced feature map channels for the two or more layers. The neural network computations can be computations associated with a CNN or related deep learning ML model.

[0087] In some implementations, the respective steps of process 700 are performed at a hardware integrated circuit as part of a larger compute operation to generate an ML output, including an output for a layer of a neural network that is the basis for one or more ML models. For example, the output can be a portion of a computation for a ML task or inference workload to generate an image processing, speech processing, or image recognition output. Workload tasks other than image and speech processing are also supported and within the scope of this disclosure. As indicated above, a portion of the integrated circuit can include special-purpose processors or hardw are ML accelerators configured to accelerate computations for generating a broad range of data processing outputs.

[0088] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus.

[0089] Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0090] The term "computing system” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0091] A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0092] A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication netw ork.

[0093] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (general purpose graphics processing unit).

[0094] Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memon or a random access memory or both. Some elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0095] Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory7, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry7.

[0096] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., LCD (liquid cry stal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

[0097] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matterdescribed in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.

[0098] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0099] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0100] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0101] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

What is claimed is:

1. A computer-implemented method for processing input images through a plurality of layers of a neural network, the method comprising: obtaining one or more reference images; generating a histogram comprising a plurality of bins based on pixel values of the one or more reference images; for each layer of the plurality of layers of the neural network: computing, based on the histogram, a sparsity value for each feature map channel of the layer; determining a sparsity correlation among feature map channels of the layer based on each sparsity value for each feature map channel of the layer; and re-sequencing the feature map channels of the layer based on the sparsity correlation.

2. The method of claim 1, wherein computing a sparsity value for each feature map channel of the layer comprises: computing a sparsity value for each feature map channel based on a percentage of zero value activations of the feature map channel, wherein the activations correspond to pixel values of features that are present in one or more of the reference images.

3. The method of claim 2, wherein re-sequencing the feature map channels of a layer comprises: configuring subsets of sequential zeros in a channel dimension of the layer; and causing a corresponding increase in activation locality in the channel dimension of the layer.

4. The method of claim 1, further comprising: for each layer of the plurality of layers of the neural network: re-sequencing filters for the layer based on the re-sequenced feature map channels of the layer and the sparsity correlation among feature map channels of the layer.

5. The method of claim 4, further comprising: for each layer of the plurality of layers of the neural network: re-sequencing a set of kernels of each filter for the layer based on the resequenced feature map channels of the layer and the sparsity correlation among feature map channels of the layer.

6. The method of claim 1, wherein determining a sparsity correlation among feature map channels comprises: determining a measure of similarity between sparsity of a first feature map channel and sparsity of a second, different feature map channel; and determining the sparsity correlation based on the measure of similarity.

7. The method of claim 6, wherein determining a measure of similarity comprises: determining a measure of similarity based on a comparative sparsity' operation performed across a plurality of feature map channels of a neural network layer and across each neural network layer of the plurality of neural network layers.

8. The method of claim 7, wherein performing a comparative sparsity operation comprises: performing a comparative sparsity operation based on a sparsity value of the first feature map channel and a sparsity’ value of the second, different feature map channel.

9. The method of claim 1 , further comprising: generating an output binary’ by a compiler in response to compiling program code for a neural network model, wherein the output binary encodes the re-sequenced feature map channels of each layer of the plurality of layers.

10. The method of claim 9, wherein re-sequencing the feature map channels of the layer comprises: re-sequencing the feature map channels of the layer using the compiler in response to compiling the program code for the neural network model.

11. The method of claim 9, wherein feature map channels of each layer are re-sequenced during a compile time that precedes a run-time inference task, and the method further comprises: processing an input image using re-sequenced feature map channels for two or more layers of the neural network.

12. The method of claim 11. further comprising: generating an output representing a result of the run-time inference task based on neural network computations that are performed when the input image is processed through the two or more layers of the neural network using the re-sequenced feature map channels for the two or more layers.

13. The method of claim 1, wherein obtaining the one or more reference images and generating the histogram comprises: obtaining the one or more reference images and generating the histogram during a compile time when program code is compiled for execution on a computing device, where the program code is for executing a machine-learning model that is based on the neural network comprising the plurality of layers.

14. A system for processing input images through a plurality of layers of a neural network, the system comprising: a processing device; and a non-transitory machine-readable storage device storing instructions that are executable by the processing device to cause performance of operations comprising: obtaining one or more reference images; generating a histogram comprising a plurality of bins based on pixel values of the one or more reference images; for each layer of the plurality of layers of the neural network: computing, based on the histogram, a sparsity value for each feature map channel of the layer; determining a sparsity correlation among feature map channels of the layer based on each sparsity value for each feature map channel of the layer; andre-sequencing the feature map channels of the layer based on the sparsity correlation.

15. The system of claim 14, wherein computing a sparsity value for each feature map channel of the layer comprises: computing a sparsity value for each feature map channel based on a percentage of zero value activations of the feature map channel. wherein the activations correspond to pixel values of features that are present in one or more of the reference images.

16. The system of claim 15, wherein re-sequencing the feature map channels of a layer comprises: configuring subsets of sequential zeros in a channel dimension of the layer; and causing a corresponding increase in activation locality in the channel dimension of the layer.

17. The system of claim 14, wherein the operations further comprise: for each layer of the plurality of layers of the neural network: re-sequencing filters for the layer based on the re-sequenced feature map channels of the layer and the sparsity correlation among feature map channels of the layer.

18. The system of claim 17, wherein the operations further comprise: for each layer of the plurality of layers of the neural network: re-sequencing a set of kernels of each filter for the layer based on the resequenced feature map channels of the layer and the sparsity correlation among feature map channels of the layer.

19. The system of claim 14, wherein determining a sparsity correlation among feature map channels comprises: determining a measure of similarity between sparsity of a first feature map channel and sparsity of a second, different feature map channel; and determining the sparsity correlation based on the measure of similarity.

20. The system of claim 19, wherein determining a measure of similarity comprises: determining a measure of similarity based on a comparative sparsity operation performed across a plurality of feature map channels of a neural network layer and across each neural network layer of the plurality of neural network layers.

21. The system of claim 20, wherein performing a comparative sparsity operation comprises: performing a comparative sparsity operation based on a sparsity value of the first feature map channel and a sparsity value of the second, different feature map channel.

22. The system of claim 14, wherein the operations further comprise: generating an output binary by a compiler in response to compiling program code for a neural network model, wherein the output binary' encodes the re-sequenced feature map channels of each layer of the plurality of layers.

23. The system of claim 22, wherein re-sequencing the feature map channels of the layer comprises: re-sequencing the feature map channels of the layer using the compiler in response to compiling the program code for the neural network model.

24. The system of claim 22, wherein feature map channels of each layer are re-sequenced during a compile time that precedes a run-time inference task, and the operations further comprise: processing an input image using re-sequenced feature map channels for two or more layers of the neural network.

25. The system of claim 24, wherein the operations further comprise: generating an output representing a result of the run-time inference task based on neural network computations that are performed when the input image is processed through the two or more layers of the neural network using the re-sequenced feature map channels for the two or more layers.

26. The system of claim 14, wherein obtaining the one or more reference images and generating the histogram comprises: obtaining the one or more reference images and generating the histogram during a compile time when program code is compiled for execution on a computing device, where the program code is for executing a machine-learning model that is based on the neural network comprising the plurality of layers.

27. A non-transitory machine-readable storage device storing instructions for processing input images through a plurality of layers of a neural network, the instructions being executable by a processing device to cause performance of operations comprising: obtaining one or more reference images; generating a histogram comprising a plurality of bins based on pixel values of the one or more reference images; for each layer of the plurality of layers of the neural network: computing, based on the histogram, a sparsity value for each feature map channel of the layer; determining a sparsity correlation among feature map channels of the layer based on each sparsity7value for each feature map channel of the layer; and re-sequencing the feature map channels of the layer based on the sparsity correlation.

Images