Superpixel methods for convolutional neural networks

Abstract

The present application relates to a superpixel method for a convolutional neural network. The method includes transforming an X-by-Y-by-Z input tensor into an X'-by-Y'-by-Z' input tensor, wherein X' is less than or equal to X, Y' is less than or equal to Y, and Z' is greater than or equal to Z; obtaining one or more modified weight matrices, wherein the modified weight matrices operate on the X'-by-Y'-by-Z' input tensor to generate a U'-by-V'-by-W' output tensor, and the U'-by-V'-by-W' output tensor includes a transformed U'-by-V'-by-W output tensor, wherein U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W; and processing the X'-by-Y'-by-Z' input tensor using the modified weight matrices to generate the U'-by-V'-by-W' output tensor, wherein the U'-by-V'-by-W' output tensor includes the U-by-V-by-W output tensor.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201710570292.6, filed on July 13, 2017. Technical Field

[0003] This specification is about processing inputs through layers of a neural network to generate outputs. Background Art

[0004] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output based on a given input. Some neural networks also include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to another layer in the network, such as the next hidden layer or the output layer. Each layer of the network generates an output based on the input it receives, depending on the current values ​​of its corresponding set of parameters. Summary of the Invention

[0005] Generally, one innovative aspect of the subject matter described in this specification can be embodied in a method for more efficiently performing computations for a convolutional neural network layer configured to receive an X by Y by Z input tensor and process the received input tensor using one or more convolutional neural network layer weight matrices to generate a U by V by W output tensor. In some aspects, the method includes: transforming the X by Y by Z input tensor into an X' by Y' by Z' input tensor, where X' is less than or equal to X, Y' is less than or equal to Y, and Z' is greater than or equal to Z; obtaining one or more modified convolutional neural network layer weight matrices, where (i) the modified weight matrices operate on the X' by Y' by Z' input tensor to generate a U' by V' by W' output tensor, and (ii) the U' by V' by W' output tensor includes the transformed U by V by W output tensor, where U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W; and processing the X' by Y' by Z' input tensor using the modified weight matrices to generate the U' by V' by W' output tensor, where the U' by V' by W' output tensor includes the U by V by W output tensor.

[0006] Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method. The system of one or more computers can be configured to perform specific operations or actions by relying on software, firmware, hardware, or any combination thereof installed on the system, which can cause the system to perform the actions when in operation. One or more computer programs can be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause the device to perform the actions.

[0007] The foregoing and other embodiments can optionally include one or more of the following features, alone or in combination.

[0008] In some embodiments, X' is equal to X divided by a natural number N rounded up, Y' is equal to Y divided by a natural number M rounded up, and Z' is equal to Z multiplied by NM.

[0009] In some implementations, the X times Y times Z input tensor represents XY first inputs and the X' times Y' times Z' input tensor represents X'Y' super inputs, where each super input includes a plurality of first inputs.

[0010] In some implementations, the number of the plurality of first inputs is equal to NM inputs.

[0011] In some cases, the operation of the modified weight matrix on the X' by Y' by Z' input tensor is equivalent to the operation of the convolutional neural network layer weight matrix on the X by Y by Z input tensor.

[0012] In some implementations, an entry of the modified weight matrix includes one or more convolutional neural network layer weight matrices.

[0013] In some cases, the modified weight matrix includes a matrix having a band structure.

[0014] In some embodiments, U' is equal to U divided by a natural number P rounded up, V' is equal to V divided by a natural number Q rounded up, and W' is equal to W multiplied by PQ.

[0015] In some embodiments, the U times V times W output tensor represents UV first outputs and the U' times V' times W' output tensor represents U'V' super outputs, where each super output includes a plurality of first outputs.

[0016] In some cases, the plurality of first outputs is equal to PQ outputs.

[0017] In some embodiments, the convolutional neural network layer includes a pooling sublayer.

[0018] In some cases, the method further includes replacing the aggregation sublayer with strides on the transformed input tensor.

[0019] In some embodiments, the convolutional neural network layer comprises a stride S in the X dimension and a stride T in the Y dimension, and wherein the relationship among the stride S, the stride T, the output tensor U by V by W, and the transformed output tensor U' by V' by W' is: U'=ceiling(U / NS) and V'=ceiling(V / MT).

[0020] In some cases, the U times V times W output tensor represents UV first outputs and the U' times V' times W' output tensor represents U'V' super outputs, where each super output includes multiple first outputs, and the number of first outputs in the multiple first outputs depends on the dimension of the convolutional neural network layer weight matrix.

[0021] In some embodiments, the U times V times W output tensor represents UV first outputs and the U' times V' times W' output tensor represents U'V' super outputs, where each super output includes multiple first outputs, and the number of outputs in the multiple first outputs depends on one or more of the following: (i) the architecture of the convolutional neural network layer, (ii) the architecture of the neural network model including the convolutional neural network layer, or (iii) the device implementing the convolutional neural network layer.

[0022] In some cases, the modification to the one or more convolutional neural network layer weight matrices is a software-only modification.

[0023] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages.

[0024] Convolutional neural network layers can be computed using matrix multiplication operations. However, when the input or output depth of a convolutional neural network layer is shallow, both the software library and the hardware matrix multiplication unit may perform inefficiently because these input and output depths map to the dimensions of one of the matrices being multiplied. As an example, an input image may have RGB data, resulting in an input depth of 3, and the convolutional neural network layer may produce an output depth of 16, where 3 and 16 are shallow in the above sense. This creates inefficiency because a typical modern computer architecture can handle layers with high depth up to a certain depth with the same or similar efficiency as layers with shallow depth due to the architectural design of its caches, memory accesses, and computational units.

[0025] Standard operations in a convolutional neural network layer map a single input, such as a single pixel, to a single row or column of a matrix multiplication operation. The superpixel convolutional neural network layer described in this specification groups data equivalent to multiple inputs, such as multiple input pixels, in exchange for a spatial extent or indexing (e.g., X and Y dimensions) for a depth extent or indexing (e.g., Z dimension). The grouped inputs are more efficiently implemented and processed using a matrix multiplication unit, thereby improving the computational efficiency of the system implementing the superpixel convolutional neural network layer technology. For example, the in-memory representation of the neural network layer configured to process the grouped inputs in the superpixel CNN can be more compact, thereby enabling the layer to store and process the received inputs in a low-latency access memory of limited size. Therefore, a system implementing such a technology can benefit from higher computational throughput, lower computational latency, shorter I / O transmission time, and better energy usage of convolutional neural networks on any matrix machine, such as a central processing unit, a graphics processor unit, or a dedicated neural network processing unit. Furthermore, the system can implement the techniques without modifying the system hardware, ie, the techniques can be implemented entirely using system software and are therefore applicable to all machines used to perform matrix multiplication.

[0026] The superpixel convolutional neural network layer described in this specification can include a reduced number of kernel elements than a standard convolutional neural network layer, which in turn can reduce the number of matrix multiplication operations that need to be performed when using the superpixel convolutional neural network layer.

[0027] In addition, the system implementing a superpixel convolutional neural network layer described in this specification can transform a convolutional neural network layer with a stride greater than 1 into a superpixel convolutional neural network layer with a stride equal to 1. Some computer architectures that implement neural network systems may require a superpixel convolutional neural network layer with a stride equal to 1, such as those architectures that can only directly perform convolution calculations with a stride equal to 1, such as architectures that include matrix multiplication units such as systolic arrays. In addition, in some cases, a superpixel convolutional neural network layer with a stride equal to 1 can operate more efficiently on the architecture, for example because off-chip processing or other alternatives required to perform convolution calculations with a stride greater than 1 may not be required. Therefore, the computational efficiency and usability of the system implementing a superpixel convolutional neural network layer are improved.

[0028] The subject matter described in this specification also relates to image recognition methods / systems using superpixel convolutional neural network layers.

[0029] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1A is an example of a superpixel convolutional neural network system.

[0031] Figure 1B is a diagram showing the relationship between the convolutional neural network layer and the superpixel convolutional neural network layer.

[0032] Figure 2 is a flowchart of an example process for efficiently performing computations for a convolutional neural network layer.

[0033] Figure 3 is an example illustration of superpixel input tensor transformation.

[0034] Figure 4 is an example illustration of the modification of the weight matrix of a convolutional neural network layer.

[0035] Figure 5 is an example illustration of convolutional neural network layer weight matrix modifications including superpixel kernel striding.

[0036] Throughout the drawings, like reference numbers and designations indicate like elements. DETAILED DESCRIPTION

[0037] Figure 1A An example superpixel convolutional neural network system 100 is shown. The superpixel convolutional neural network system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below can be implemented.

[0038] The superpixel convolutional neural network system 100 is a machine learning system that receives a system input 102 and generates a system output 104 based on the system input 102.

[0039] The superpixel convolutional neural network system 100 can be configured to receive digital data input and generate a score or classification output based on the input. For example, if the input to the superpixel convolutional neural network system 100 is an image or features extracted from an image, then for a given image, the output generated by the superpixel convolutional neural network system 100 can be a score for each of a set of object categories, for example using one or more fully connected or convolutional neural network layers included in the neural network system 100. Each score can represent an estimated likelihood that the image contains an object belonging to that category.

[0040] In particular, the superpixel convolutional neural network system 100 implements a convolutional neural network that includes multiple convolutional neural network layers, including convolutional neural network layer A 106, convolutional neural network layer B 108, and superpixel convolutional neural network layer 110. A convolutional neural network layer is a neural network layer whose neurons or nodes are logically arranged into a three-dimensional array: width, height, and depth. Neurons within a convolutional neural network layer are connected to a small area of ​​the layer before the convolutional neural network layer, which is called a receptive field or kernel patch, and each neuron within the convolutional neural network layer detects the same feature, constituting the property of translation invariance. A convolutional neural network layer can be associated with each weight matrix or kernel that stores the current values ​​of the parameters of the layer. The layer generates an output based on the input according to the current values ​​of the parameters of the neural network layer, for example by performing multiple matrix multiplication operations. An example convolutional neural network layer is described in more detail in “Large-scale Video Classication with Convolutional Neural Networks,” Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei, Proceedings of International Computer Vision and Pattern Recognition (CVPR 2014), IEEE.

[0041] Each layer of the superpixel convolutional neural network is configured to receive a corresponding layer input and process the layer input to generate a corresponding layer output based on the input. For example, a convolutional neural network layer included in the superpixel convolutional neural network system 100 can be configured to receive an X by Y by Z input tensor and process the received input tensor using one or more corresponding convolutional neural network layer weight matrices or kernels to generate a U by V by W output tensor. A tensor is a multidimensional array of numbers or other values ​​such as strings, with a specific order corresponding to the dimensions of the array. For example, a scalar value is an order 0 tensor, a numeric vector is an order 1 tensor, and a matrix is ​​a order 2 tensor.

[0042] For example, a convolutional neural network layer can be configured to receive a 64×64×3 input tensor representing an image comprising 64×64 pixels with a feature depth of 3, where the feature depth is represented by red, green, and blue feature channels. The convolutional neural network layer can process the received 64×64×3 input tensor using one or more corresponding convolutional neural network layer matrices to generate a corresponding output tensor.

[0043] In some embodiments, a convolutional neural network layer may include one or more hyperparameters that control the size and output of the convolutional neural network layer. For example, a convolutional neural network layer may include a kernel stride that controls how much a convolution filter or weight matrix is ​​shifted in X and Y between evaluations of the filter relative to the input data. In some embodiments, a convolutional neural network layer may include or be connected to a pooling layer—a form of nonlinear downsampling. For example, a convolutional neural network layer may include a pooling stride that performs a depthwise slice operation on an input tensor and spatially resizes the input tensor. Continuing with the above example, the convolutional neural network layer may therefore generate a corresponding U by V by W output tensor, where U is less than or equal to X, V is less than or equal to Y, and W is greater than or equal to Z, for example, a 32 by 32 by 32 output tensor (e.g., corresponding to a 7 by 7 kernel stride and a 2 by 2 pooling stride). In general, kernel striding can be thought of as omitting the original computation (e.g., "skip by 2 on X and skip by 2 on Y between evaluations"), while pooling is a post-pass after the stride-1 evaluation (e.g., in pooling, the computation is done with a stride of 1, and the maximum value in each group within the pooling window is chosen, e.g., a 2×2 portion of the input data.)

[0044] At least one convolutional neural network layer included in the superpixel neural network system 100 is a superpixel convolutional neural network layer, for example, the superpixel convolutional neural network layer 110. A superpixel convolutional neural network layer is a convolutional neural network layer that applies a superpixel transformation (alternatively referred to as a modification) to a received convolutional layer input and a convolutional neural network layer weight matrix to generate a superpixel convolutional neural network layer of a different shape but mathematically equivalent.

[0045] The superpixel convolutional neural network layer 110 may receive input, such as input 112, from a convolutional neural network layer preceding the superpixel convolutional neural network layer 110. For example, the received input may include output generated by the convolutional neural network layer preceding the superpixel convolutional neural network layer 110.

[0046] The superpixel convolutional neural network layer 110 may include a superpixel generator 114. The superpixel generator 114 is configured to receive a convolutional neural network input and transform the received input into a superpixel layer input. The superpixel generator 114 groups together multiple components of the received input, trading spatial extents or indices, such as the X and Y dimensions, for depth extents or indices, such as the Z dimension. This has the effect that the depth of the convolutional neural network layers in the Z layer is no longer shallow, so that the inefficiencies caused by the shallow Z layer can be avoided.

[0047] For example, the superpixel convolutional neural network layer 110 may receive as input an X by Y by Z input tensor, which represents, for example, XY (X times Y) pixels at a depth of Z. The superpixel generator 114 is configured to transform the X by Y by Z input tensor into an X' by Y' by Z' superpixel input tensor, where X' is less than or equal to X, Y' is less than or equal to Y, and Z' is greater than or equal to Z. Figure 3 Illustrated are example superpixel input tensor transformations performed by the superpixel generator.

[0048] The superpixel convolutional neural network layer 110 includes a superpixel weight matrix 116. The superpixel weight matrix 116 is a modified (alternatively referred to as a transformed) convolutional neural network layer weight matrix corresponding to the convolutional neural network layer. For example, in response to selecting a corresponding convolutional neural network layer to be transformed into a superpixel convolutional neural network layer, the convolutional neural network layer weight matrix can be modified accordingly. The superpixel weight matrix 116 operates on the transformed convolutional neural network layer input.

[0049] For example, a convolutional neural network layer corresponding to the superpixel neural network layer 110 can be configured to receive an X by Y by Z input tensor. The superpixel neural network layer 110 can transform the received input into an X' by Y' by Z' superpixel input tensor. In this case, the superpixel weight matrix can include a modified weight matrix that operates on the X' by Y' by Z' input tensor to generate a U' by V' by W' output tensor, where X' is not necessarily equal to U', Y' is not necessarily equal to V', and Z' is not necessarily equal to W'. The U' by V' by W' output tensor is a transformed U by V by W output tensor, where U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W. For example, the U' by V' by W' output tensor can represent a superpixel output tensor, such as a clustered component of the U by V by W output tensor. In some embodiments, the clusters used to create a superpixel input tensor or represent a superpixel output tensor can be different, for example, the number of components and the way in which the components are grouped together can be different. Reference below Figure 4An example convolutional neural network layer weight matrix modification is illustrated. In addition, an example algorithm (including comments) for generating superpixel weight matrices (with or without stride, e.g., stride_x = stride_y = 1) is given below:

[0050]

[0051]

[0052]

[0053] The example SuperIndex routine used above can be defined by, but is not limited to, the following:

[0054]

[0055]

[0056] The above is for illustrative purposes only. For example, in some embodiments, the super-indexing routine may not rely on the packing order within the superpixel.

[0057] The superpixel convolutional neural network layer 110 processes the transformed input tensor using the modified weight matrix to generate a transformed convolutional neural network layer output, such as a superpixel layer output 118. The generated transformed convolutional neural network layer output can be mathematically equivalent to the neural network output generated by processing the untransformed convolutional neural network input using the unmodified convolutional neural network layer weight matrix. All output values ​​of the unmodified computation can be present in the generated transformed convolutional neural network layer output in such a form that they can be computed by performing an anti-homomorphic transformation, as described below with reference to Figure 1B As described. For example, the generated transformed convolutional neural network layer outputs may include a superset of outputs including neural network outputs corresponding to the neural network outputs generated by processing the (untransformed) convolutional neural network inputs using the (unmodified) convolutional neural network layer. In some embodiments, the additional outputs contained in the superset may be provided to subsequent layers of the superpixel neural network system 100. For example, the additional outputs may correspond to "partial sums" of the outputs "off the edge" of the initially requested untransformed outputs. If the superpixel computation is processing a "tile" of a larger computation with X and Y that is much larger than what can be efficiently processed in a single superpixel computation, then the partial sums may be saved and reused for greater efficiency. Reference below Figure 2 Describe in more detail the use of superpixel convolutional neural network layers to process input.

[0058] In some embodiments, the superpixel convolutional neural network layer 110 may include a superpixel deformatter that is configured to receive superpixel outputs and transform the received superpixel outputs into normally ordered outputs. For example, if the current superpixel convolutional neural network layer produces 4×4 superpixel outputs and the next superpixel convolutional neural network layer is configured to process 8×8 superpixels, the system can use the superpixel deformatter to transform the 4×4 superpixels into 8×8 superpixels. In some embodiments, the superpixel deformatter of a superpixel convolutional neural network layer, such as layer 110, can follow the superpixel weight matrix, such as weight matrix 116, and ungroup the outputs generated by the weight matrix. In other embodiments, the superpixel deformatter of a superpixel convolutional neural network layer can be included with or as part of the superpixel generator of the next superpixel convolutional neural network layer in the sequence of superpixel convolutional neural network layers.

[0059] The superpixel convolutional neural network layer 110 can be included at various positions in the sequence of neural network layers, and in some embodiments, multiple superpixel convolutional neural network layers can be included in the sequence. For example, in some embodiments, one or more of convolutional neural network layers A and B can be superpixel convolutional neural network layers. In embodiments where multiple superpixel convolutional neural network layers are included in the sequence of neural network layers, each superpixel convolutional neural network layer can apply a corresponding transformation to the received input to generate a corresponding super input, where the generated corresponding super input can include super inputs of different sizes.

[0060] The following table lists an example layout for a system with multiple superpixel convolutional neural network layers:

[0061] layer Superpixels at each layer (XxY) Pixels per superpixel Original input depth Z Transformed input depth U Convolution 0 / stride 2 8x8 8x8 3 192 Convolution 1 / stride 2 8x8 4x4 16 256 Convolution 2 8x8 2x2 32 128 Convolution 3 8x8 2x2 64 256 Convolution 4 / stride 2 8x8 2x2 64 256 Output 4x4 2x2 64 256

[0062] In the example layout above, most neural network layers include 8×8 sets of superpixels per layer. The number of pixels per superpixel is adjusted in response to kernel stride. The "Original Input Depth (Z)" column specifies the input depth in the original neural network model, and the "Transformed Input Depth (U)" column specifies the input depth after the input and weights are transformed using the specified number of pixels per superpixel.

[0063] A second example layout for a system of multiple superpixel convolutional neural network layers is shown in the table below:

[0064]

[0065] The superpixel convolutional neural network system 100 can be trained on multiple batches of training examples in order to determine the training values ​​of the parameters of the neural network layers, i.e., to adjust the values ​​of the parameters from the initial values ​​to the training values. For example, during training, the superpixel convolutional neural network system 100 can process a batch of training examples and generate a corresponding neural network output for each training example in the batch. The neural network output can then be used - for example, by conventional gradient descent and backpropagation neural network training techniques - to adjust the values ​​of the parameters of the components of the superpixel convolutional neural network system 100. Since the modified weight matrix makes NM copies of each original weight, where N and M represent the superpixel size, as described below with reference to Figure 2 As described in step 202 of , during the back-propagation training process, the system can add weight gradients across all NM replicas to determine the net change in the value of each original weight. Superpixel neural network layer training is a reverse process, where the depth range or index can be exchanged for the spatial range or index.

[0066] Once the neural network is trained, the superpixel convolutional neural network system 100 can receive new neural network inputs for processing and process the neural network inputs through the neural network layers to generate new neural network outputs for the inputs based on the trained values ​​of the parameters of the components of the superpixel convolutional neural network system 100.

[0067] In some embodiments, the superpixel neural network system 100 may include one or more deconvolutional neural network layers. In these cases, the above components and techniques may be reversed to achieve the same effect, i.e., the superpixel deconvolutional neural network layer may group multiple inputs together, trading depth ranges or indices for spatial ranges or indices, and the corresponding deconvolutional neural network layer weight matrices may be modified accordingly.

[0068] Figure 1B is a diagram 160 showing the relationship between a convolutional neural network layer and a superpixel convolutional neural network layer.

[0069] As shown in diagram 160 , a convolutional neural network layer may be configured to perform a convolution operation “convolve” 162 on an input activation I and a weight matrix W to generate a layer output activation O 164 .

[0070] Similarly, for example, referring to Figure 1A The superpixel convolutional neural network layer of the described modified or transformed convolutional neural network layer can be configured to perform a convolution operation “convolve” 168 on the transformed input activations Transform(I) and the transformed weight matrix Transform(W) to generate transformed layer output activations 170.

[0071] Diagram 160 shows two homomorphic arrows connecting the input I to the transformed input Transform(I) and the weight matrix W to the transformed weight matrix Transform(W), respectively. The homomorphic arrows represent the homomorphic relationship between the original convolutional neural network layer and the modified convolutional neural network layer. For example, the arrows map the original model activations and weights to the modified or transformed model activations and weights. Similarly, diagram 160 shows an anti-homomorphic arrow connecting the transformed output Transform(O) to the output O. The anti-homomorphism represents the anti-homomorphic relationship between the original convolutional neural network layer and the modified convolutional neural network layer. For example, the arrows map the transformed output back to the original output.

[0072] Here is an example algorithm for transforming the original model activations:

[0073]

[0074]

[0075] Here is an example algorithm for anti-homomorphism:

[0076]

[0077]

[0078] As shown in diagram 160, because the input I and weight matrix W may need to be transformed before applying the convolution, the number of operations required to generate the convolutional neural network layer output using the superpixel convolutional neural network layer may be greater than the number of operations required by the corresponding original neural network layer. However, due to the superpixel convolutional neural network layer architecture, performing the larger number of operations can be computationally faster at runtime compared to the smaller number of operations performed by the corresponding original neural network layer.

[0079] Figure 2 is a flow chart of an example process 200 for efficiently performing computations for a convolutional neural network layer configured to receive an X by Y by Z input tensor and process the received input using one or more convolutional neural network layer weight matrices to generate a U by V by W output tensor. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a computer program, such as a computer program program, suitably programmed according to this specification, Figure 1A The superpixel convolutional neural network system 100 is capable of performing process 200.

[0080] The system transforms the X by Y by Z input tensor into an X' by Y' by Z' superpixel input tensor (step 202). The system transforms the input tensor by grouping multiple inputs together and exchanging depth ranges or indices for spatial ranges or indices. For example, the X' by Y' by Z' input tensor can have different dimensions than the X by Y by Z input tensor. More specifically, X' can be less than or equal to X, Y' can be less than or equal to Y, and Z' can be greater than or equal to Z. In some implementations, X' is equal to X divided by a natural number N rounded up (ceiling) - for example, the smallest integer greater than or equal to X divided by the natural number N, Y' is equal to Y divided by a natural number M rounded up, and Z' is equal to Z multiplied by NM.

[0081] An X by Y by Z input tensor may represent XY first inputs. For example, an X by Y by Z input tensor may represent an image of X by Y pixels with a feature depth of Z. Similarly, an X' by Y' by Z' input tensor may represent X'Y' super inputs, where each super input includes multiple first inputs. For example, an X' by Y' by Z' input tensor may represent an image of X' by Y' super pixels with a depth of Z'. In some implementations, the number of multiple first inputs is equal to NM inputs, for example, each super input may be composed of NM first inputs. Figure 3 An example transformation of the input tensor is shown in Figure .

[0082] The system obtains one or more modified convolutional neural network layer weight matrices (step 204). The modified weight matrices are operated on the X' times Y' times Z' input tensor to generate a U' times V' times W' output tensor. In some embodiments, the operation of the modified weight matrix on the X' times Y' times Z' input tensor is equivalent to the operation of the convolutional neural network layer weight matrix on the X' times Y' times Z input tensor, for example, the operation of the modified weight matrix on the X' times Y' times Z' input tensor achieves the same effect. The U' times V' times W' output tensor may include a transformed U' times V' times W output tensor, where U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W. For example, the U' times V' times W' output tensor may represent a grouped plurality of outputs from the U' times V' times W output tensor, where the spatial range or index has been traded for the depth range or index as described above.

[0083] The elements of the modified weight matrix include one or more convolutional neural network layer weight matrices, for example, the modified weight matrix may include a block matrix that can be interpreted as being decomposed into segments called blocks or sub-matrices. In some embodiments, the elements of the modified weight matrix can exhibit a banded structure. For example, a convolutional neural network layer can be represented by a fully connected matrix in which each input is connected to each output. The fully connected matrix exhibits a banded structure in which the majority of the matrix's multiple elements are zero and a small number of diagonal bands in the matrix have non-zero elements.

[0084] Typically, for one dimension, the number of modified kernel elements can be equal to ceiling((superinput_size+original convolutional neural network layer kernel width-1) / superinput_size). Figure 4 Illustration of convolutional neural network layer weight matrix modification.

[0085] Optionally, the system can implement an efficient kernel stride for the convolutional neural network layer (step 206). For example, in some embodiments, the convolutional neural network layer may include a pooling sublayer. In these cases, the system can replace the pooling sublayer with a stride on the transformed input tensor X' by Y' by Z'. For example, as described below with reference to step 208, in some embodiments, the U by V by W output tensor can represent UV first outputs, and the U' by V' by W' output tensor can represent U'V' super outputs, where each super output includes multiple first outputs. In such a case, the effective kernel stride size implemented may not be equal to the number of outputs in the multiple first outputs. The effect of the kernel stride can be combined into a modified weight matrix. For example, an (untransformed) convolutional neural network layer can generate a layer output O by applying a convolution to the corresponding input I and the corresponding weight matrix W and applying a stride to the convolution input I and the weight matrix W - i.e., by performing two operations. A superconvolutional neural network layer with a strided transformation can generate a transformed layer output transform(O) by applying a convolution to the transformed corresponding input transform(I) and the strided, transformed weight matrix strided_transform(2,W).

[0086] In some embodiments, a convolutional neural network layer may already include a kernel stride. In these cases, the system may replace the kernel stride with an efficient kernel stride. The system may implement efficient kernel stride by using different input and output superpixel sizes, where the ratio of the superpixel sizes is equal to the stride. For example, if a convolutional neural network layer includes a stride S in the X dimension and a stride T in the Y dimension, and the convolutional neural network layer is transformed into a super convolutional neural network layer with a superpixel input of size NM, then by selecting an output superpixel size of (N / S)(M / T), the system implements the kernel stride of the superpixel modified weight matrix. In some embodiments, the relationship between the stride S, the output tensor U times V times W, and the transformed output tensor U' times V' times W' can be U'=ceiling(U / NS) and V'=ceiling(V / MT). Referring below Figure 5 Figure 1 illustrates achieving efficient kernel strides.

[0087] The system processes the X' by Y' by Z' input tensor using the modified weight matrix to generate a transformed convolutional neural network layer output—e.g., a U' by V' by W' output tensor (step 208). The U' by V' by W' output tensor includes the U by V by W output tensor, e.g., the U' by V' by W' output tensor can include a superset of the outputs of the U by V by W output tensor.

[0088] The U' times V' times W' output tensor can have different dimensions than the U times V times W output tensor. More specifically, U' can be equal to U divided by the rounded-up natural number P, V' can be equal to V divided by the rounded-up natural number Q, and W' can be equal to W times PQ. In some embodiments, the natural numbers P and Q can be equal to the natural numbers N and M described above with reference to step 202, respectively. Furthermore, in some embodiments, the P and Q of a current superpixel CNN layer can be equal to the N and M of a subsequent superpixel CNN layer, e.g., where the system does not include a superpixel generator and a superpixel deformatter between the current superpixel CNN layer and the subsequent superpixel CNN layer.

[0089] The U times V times W output tensor may represent UV first outputs of depth W. For example, the U times V times W output tensor may represent a feature map of U times V outputs of depth W. Similarly, the U' times V' times W' output tensor may represent U'V' super outputs, where each super output includes multiple first outputs. For example, the U' times V' times W' output tensor may represent an image of U' times V' pixels of depth W'. In some embodiments, the number of multiple first outputs is equal to P×Q output pixels, for example, each super output may be composed of PQ first outputs.

[0090] The natural numbers P and Q may be different from the natural numbers N and M described above with reference to step 202. For example, the size of the super input described above may not be equal to the size of the super output. In some embodiments, the number of first outputs in the plurality of first outputs—e.g., the size of the super output—depends on the dimensions of the convolutional neural network layer weight matrix. In some embodiments, the number of outputs in the plurality of first outputs depends on one or more of: (i) the architecture of the convolutional neural network layer, (ii) the architecture of the neural network model including the convolutional neural network layer, or (iii) the device implementing the convolutional neural network layer.

[0091] As an example, assume that the original convolutional layer has an input size of X times Y times Z and produces an output size of U times V times W, and that the superpixel transformed convolutional layer has an input size of X' times Y' times Z' and produces an output size of U' times V' times W'. Let KL denote the number of convolution kernel elements in the weight matrix, and K'L' denote the number of convolution kernel elements in the modified weight matrix. Further assume that the input and output pixels / superpixels are the same and are specified by N and M. Then, U' = ceiling(U / N), V' = ceiling(V / M), and W' = W×N×M. Furthermore, K' = ceiling((N+K-1) / N) and L' == ceiling((M+L-1) / N). Cases where PQ may not be equal to NM include: when the system may implement kernel striding or when the number of pixels per transformed superpixel changes from input to output.

[0092] Figure 3 is an example diagram 300 of a superpixel input tensor transformation. For example, the example diagram shows the superpixel input tensor transformation as described above with reference to FIG. Figure 2 The superpixel input tensor transformation described in step 202 can be performed by a system of one or more computers located in one or more locations. For example, a computer program such as a computer program program according to the present specification can be used to perform the transformation. Figure 1A The superpixel generator 114 is capable of performing the illustrated transformation.

[0093] The example diagram shows an example X by Y by Z input tensor 302. Figure 3 As shown above, the input tensor includes XY inputs, each with a depth of Z. Figure 2 As described above, an X by Y by Z input tensor can be transformed into an X' by Y' by Z' superpixel input tensor by grouping multiple inputs together. During grouping, indices or ranges on the spatial dimensions (X and Y dimensions) are exchanged for indices or ranges on the depth dimension (Z dimension). In the example diagram 300, the X by Y by Z input tensor has been transformed into an X / 2 by Y / 2 by 4Z superpixel input tensor 304. Figure 3As shown, the superpixel input tensor includes (X / 2)(Y / 2) inputs, each with a depth of 4Z. Each superpixel input included in the input tensor 304 represents 4 original inputs, and therefore represents 4 times the amount of data represented by the original inputs.

[0094] Figure 4 is an example diagram 400 of modification of the weight matrix of a convolutional neural network layer. For example, the example diagram shows the above reference Figure 2 The convolutional neural network layer weights are modified as described in step 204. The modifications shown can be performed by a system of one or more computers located in one or more locations. For example, a superpixel convolutional neural network layer appropriately programmed according to the present specification, such as Figure 1A The superpixel convolutional neural network layer 110 can perform the modifications shown.

[0095] In step (a), the example diagram 400 shows a 3 by 3 kernel block 402 for a convolutional neural network layer, where each element of the kernel block comprises a two-dimensional weight matrix, where the dimension is equal to the original input depth times the original output depth, e.g., Z times W, which is suitable for matrix multiplication. The example diagram further shows four 2 by 2 grids 404-410 representing the input. The inputs are grouped into 2 by 2 super inputs, each super input comprising 2 by 2 original inputs. The example diagram shows a super output 412 comprising four original outputs.

[0096] At step (b), example diagram 400 shows the same two-dimensional convolution as shown in step (a), expanded to compute dot products for each of the four original outputs. The shading behind the 2x2 grid, such as shading 414, shows which 3x3 subset of the input data affects each output. As shown by the shading of each grid, each super input affects each output, but the useful fraction of each super input is 9 / 16.

[0097] In step (c), example diagram 400 shows example modified two-dimensional weight matrices 416-422 that can be used to calculate a super output. Each of matrices 416-422 is a kernel element that together constitutes a 2x2 modified kernel block. The dimensions of matrices 416-422 are 4Z by 4W, making them 16 times the original two-dimensional matrices of kernel block 402. The vertical axis represents input depth, inputs within the super input, and kernel elements. The horizontal axis represents output depth and outputs within the super output. The example diagram in step (c) depicts matrix multiplication. Column 424 represents inputs, and when an input is shown at the same height as an element of the modified weight matrix 416-422, it is multiplied by the weight. All weight-input products in the column can be added to produce the total value of the output within the super output. A total of 4 super inputs, e.g., 16 original inputs, are shown for calculating one super output, e.g., 4 original outputs.

[0098] Each of the 4 super inputs has a different X and Y coordinate relationship to the super output. For illustration purposes, Figure 4 2x2 pixels per superpixel are shown, and a 2x2 set of superpixels is shown. Furthermore, 2x2=4 weight matrices are drawn. The number of pixels per superpixel and superpixel set can vary, for example, based on factors such as the system architecture. For example, in some cases, there may be 4x4 pixels per superpixel and the large image size is 100x100 superpixels (thus 400x400 original input pixels). In this case, as long as the ceiling ((pixels_per_superpixel+kernel_size-1) / pixels_per_superpixel) is less than or equal to 2 in each dimension, there may still be 2x2=4 weight matrices.

[0099] As described above, the modified weight matrices are kernel elements. In this regard, the modification process can be viewed as having reduced the number of kernel elements in a block by changing a 3x3 convolution on the input to a 2x2 convolution on the super input, where each kernel element is a larger two-dimensional matrix, for example, 4 times larger in both rows and columns (corresponding to the input depth Z and the output depth W). Each column in the stacked modified weight matrices 416-422 contains a copy of each element of the 3x3 kernel 402. In some embodiments, the modified weight matrices can exhibit a banded structure, as shown, for example, by the kernel elements labeled in a similar diagonal manner, such as diagonal line 426.

[0100] Figure 5 is an example diagram 500 of a convolutional neural network layer weight matrix modification including superpixel kernel strides. For example, the example diagram shows the weight matrix modification of the convolutional neural network layer weight matrix modified as described above with reference to FIG. Figure 2 The illustrated embodiments may be performed by a system of one or more computers located in one or more locations. For example, a superpixel convolutional neural network layer appropriately programmed according to the present specification, such as Figure 1A The superpixel convolutional neural network layer 110 can perform the illustrated embodiment.

[0101] Example diagram 500 shows a weight graph with a 5x5 kernel block 506, whose elements are two-dimensional weight matrices, where the dimensions are equal to the original input depth by the original output depth, e.g., Z by W; 4x4 super inputs 502, each comprising 4x4 original inputs; and 2x2 super outputs 504, each comprising 2x2 original outputs. The example kernel stride is 2x2. Example diagram 500 includes four super pixel kernel elements 508-514. Each super pixel kernel element is a 16 Z by 4 W matrix. The input / super input versus output / super output changes due to the 2x2 kernel stride, so there are only 4 outputs for every 16 inputs.

[0102] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., modules of one or more computer program instructions, encoded on a tangible, non-transitory program carrier, for execution by a data processing device or for controlling the operation of the data processing device. Alternatively or additionally, program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof. However, a computer storage medium is not a propagation signal.

[0103] The term "data processing apparatus" includes all types of apparatus, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The apparatus may include dedicated logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0104] A computer program (which may also be referred to as a program, software, software application, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing portions of one or more modules, subroutines, or code).

[0105] As used in this specification, "engine" or "software engine" refers to a software-implemented input / output system that provides an output that is different from the input. An engine can be a coded functional block, such as a library, platform, software development kit ("SDK") or object. Each engine can be implemented on any appropriate type of computing device, such as a server, mobile phone, tablet computer, notebook computer, music player, e-book reader, laptop or desktop computer, PDA, smart phone or other fixed or portable device including one or more processors and computer-readable media. In addition, two or more of the engines can be implemented on the same computing device or on different computing devices.

[0106] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0107] For example, a computer suitable for executing a computer program can be based on a general or special microprocessor or both or any other type of central processing unit. Typically, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or implementing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operably coupled to one or more large-capacity storage devices for storing data to receive data from it or to transmit data to it or both, such as magnetic, magneto-optical disks or optical disks. However, a computer does not need to have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0108] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0109] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide for interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on a user's client device in response to a request received from the web browser.

[0110] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an embodiment of the subject matter described in this specification), or a combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0111] A computing system may include a client and a server. The client and server are typically remote from each other and typically interact via a communication network. The relationship between the client and the server is generated by computer programs running on their respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a user device, for example, in order to display data to a user interacting with the user device and receive user input from the user, the user device acting as a client. Data generated at the user device, for example, as a result of a user interaction, may be received from the user device at the server.

[0112] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope of what is claimed, but rather as descriptions of features that may be peculiar to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable sub-combination. Furthermore, although features may be described above as acting in certain combinations, and even initially claimed as such, one or more features of the claimed combination may, in some cases, be removed from the combination.

[0113] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown, or in sequential order, or that all illustrated operations be performed, to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated into a single software product or packaged into multiple software products.

[0114] Further implementations of the disclosed technology are described in the following examples:

[0115] Example 1: A computer-implemented method for more efficiently performing computations of a convolutional neural network layer, the convolutional neural network layer being configured to receive an X by Y by Z input tensor and process the received input tensor using one or more convolutional neural network layer weight matrices to generate a U by V by W output tensor, the method comprising: transforming the X by Y by Z input tensor into an X' by Y' by Z' input tensor, where X' is less than or equal to X, Y' is less than or equal to Y, and Z' is greater than or equal to Z; obtaining one or more modified convolutional neural networks A layer weight matrix, wherein (i) the modified weight matrix operates on the X' by Y' by Z' input tensor to generate a U' by V' by W' output tensor, and (ii) the U' by V' by W' output tensor includes the transformed U by V by W output tensor, where U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W; and processes the X' by Y' by Z' input tensor using the modified weight matrix to generate a U' by V' by W' output tensor, where the U' by V' by W' output tensor includes the U by V by W output tensor.

[0116] Example 2: The method of Example 1, wherein X' is equal to X divided by a natural number N rounded up, Y' is equal to Y divided by a natural number M rounded up, and Z' is equal to Z multiplied by NM.

[0117] Example 3: The method of Example 2, wherein the X times Y times Z input tensor represents XY first inputs and the X' times Y' times Z' input tensor represents X'Y' super inputs, wherein each super input includes multiple first inputs.

[0118] Example 4: The method of Example 3, wherein the number of the plurality of first inputs is equal to NM inputs.

[0119] Example 5: The method of any one of Examples 1 to 4, wherein the operation of the modified weight matrix on the X' by Y' by Z' input tensor is equivalent to the operation of the convolutional neural network layer weight matrix on the X by Y by Z input tensor.

[0120] Example 6: The method of any of Examples 1 to 5, wherein the elements of the modified weight matrix include one or more convolutional neural network layer weight matrices.

[0121] Example 7: The method of any of Examples 1 to 6, wherein the modified weight matrix comprises a matrix having a band structure.

[0122] Example 8: The method of any one of Examples 1 to 7, wherein U' is equal to U divided by a natural number P rounded up, V' is equal to V divided by a natural number Q rounded up, and W' is equal to W multiplied by PQ.

[0123] Example 9: The method of Example 8, wherein the convolutional neural network layer includes a stride S in the X dimension and a stride T in the Y dimension, and wherein the relationship between the stride S, the stride T, the output tensor U by V by W, and the transformed output tensor U' by V' by W' is: U'=ceiling(U / NS) and V'=ceiling(V / MT).

[0124] Example 10: The method of Example 8 or 9, wherein the U times V times W output tensor represents UV first outputs and the U' times V' times W' output tensor represents U'V' super outputs, wherein each super output includes a plurality of first outputs.

[0125] Example 11: The method of Example 10, wherein the plurality of first outputs is equal to PQ outputs.

[0126] Example 12: The method of any one of Examples 1 to 11, wherein the convolutional neural network layer includes a pooling sublayer.

[0127] Example 13: The method of Example 12, wherein the method further comprises replacing the aggregation sublayer with strides on the transformed input tensor.

[0128] Example 14: The method of Example 12 or 13, wherein the convolutional neural network layer includes kernel striding, and wherein the method further includes: implementing efficient kernel striding based on the transformed input tensor and the modified weight matrix.

[0129] Example 15: The method of any of Examples 1 to 14, wherein the U by V by W output tensor represents UV first outputs and the U' by V' by W' output tensor represents U'V' super outputs, wherein each super output includes multiple first outputs, and wherein the number of first outputs in the multiple first outputs depends on the dimension of the convolutional neural network layer weight matrix.

[0130] Example 16: The method of any of Examples 1 to 15, wherein the U times V times W output tensor represents UV first outputs and the U' times V' times W' output tensor represents U'V' super outputs, wherein each super output includes multiple first outputs, and wherein the number of outputs in the multiple first outputs depends on one or more of the following: (i) the architecture of the convolutional neural network layer, (ii) the architecture of the neural network model including the convolutional neural network layer, or (iii) the device that implements the convolutional neural network layer.

[0131] Example 17: The method of any of Examples 1 to 16, wherein the modification to the one or more convolutional neural network layer weight matrices is a software-only modification.

[0132] Example 18: A system comprising one or more computers and one or more storage devices storing instructions operable, when executed by the one or more computers, to cause the one or more computers to perform operations for more efficiently performing computations for a convolutional neural network layer, the convolutional neural network layer being configured to receive an X by Y by Z input tensor and process the received input tensor using one or more convolutional neural network layer weight matrices to generate a U by V by W output tensor, the operations comprising: transforming the X by Y by Z input tensor into an X' by Y' by Z' input tensor, where X' is less than or equal to X, Y' is less than or equal to Y, and Z' greater than or equal to Z; obtaining one or more modified convolutional neural network layer weight matrices, wherein (i) the modified weight matrices operate on the X' by Y' by Z' input tensor to generate a U' by V' by W' output tensor, and (ii) the U' by V' by W' output tensor comprises a transformed U by V by W output tensor, wherein U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W; and processing the X' by Y' by Z' input tensor using the modified weight matrices to generate the U' by V' by W' output tensor, wherein the U' by V' by W' output tensor comprises the U by V by W output tensor.

[0133] Example 19: The system of Example 18, wherein X' is equal to X divided by a natural number N rounded up, Y' is equal to Y divided by a natural number M rounded up, and Z' is equal to Z multiplied by NM.

[0134] Example 20: The system of Example 19, wherein the X times Y times Z input tensor represents XY first inputs and the X' times Y' times Z' input tensor represents X'Y' super inputs, wherein each super input includes a plurality of first inputs.

[0135] Example 21: The system of Example 20, wherein the number of the plurality of first inputs is equal to NM inputs.

[0136] Example 22: The system of any of Examples 18 to 21, wherein the operation of the modified weight matrix on the X' by Y' by Z' input tensor is equivalent to the operation of the convolutional neural network layer weight matrix on the X by Y by Z input tensor.

[0137] Example 23: The system of any of Examples 18 to 22, wherein the elements of the modified weight matrix comprise one or more convolutional neural network layer weight matrices.

[0138] Example 24: The system of any of Examples 18 to 23, wherein the modified weight matrix comprises a matrix having a band structure.

[0139] Example 25: The system of any of Examples 18 to 24, wherein U' is equal to U divided by a natural number P rounded up, V' is equal to V divided by a natural number Q rounded up, and W' is equal to W multiplied by PQ.

[0140] Example 26: The system of Example 25, wherein the convolutional neural network layer comprises a stride S in the X dimension and a stride T in the Y dimension, and wherein the relationship between the stride S, the stride T, the output tensor U by V by W, and the transformed output tensor U' by V' by W' is: U' = ceiling(U / NS) and V' = ceiling(V / MT).

[0141] Example 27: The system of Example 25 or 26, wherein the U times V times W output tensor represents UV first outputs and the U' times V' times W' output tensor represents U'V' super outputs, wherein each super output includes a plurality of first outputs.

[0142] Example 28: The system of Example 27, wherein the plurality of first outputs is equal to PQ outputs.

[0143] Example 29: The system of Example 18, wherein the convolutional neural network layer includes a pooling sublayer.

[0144] Example 30: The system of Example 29, wherein the operation further comprises replacing the aggregation sublayer with strides on the transformed input tensor.

[0145] Example 31: The system of any of Examples 18 to 30, wherein the convolutional neural network layer includes kernel strides, and wherein the method further comprises: implementing efficient kernel strides based on the transformed input tensor and the modified weight matrix.

[0146] Example 32: The system of any of Examples 18 to 31, wherein the U by V by W output tensor represents UV first outputs and the U' by V' by W' output tensor represents U'V' super outputs, wherein each super output includes multiple first outputs, and wherein the number of first outputs in the multiple first outputs depends on the dimension of the convolutional neural network layer weight matrix.

[0147] Example 33: The system of any of Examples 18 to 32, wherein the U by V by W output tensor represents UV first outputs and the U' by V' by W' output tensor represents U'V' super outputs, wherein each super output includes multiple first outputs, and wherein the number of first outputs in the multiple first outputs depends on one or more of: (i) the architecture of the convolutional neural network layer, (ii) the architecture of the neural network model including the convolutional neural network layer, or (iii) the device that implements the convolutional neural network layer.

[0148] Example 34: The system of any of Examples 18 to 33, wherein the modification to the one or more convolutional neural network layer weight matrices is a software-only modification.

[0149] Example 35: A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for more efficiently performing computations for a convolutional neural network layer, the convolutional neural network layer being configured to receive an X by Y by Z input tensor and process the received input tensor using one or more convolutional neural network layer weight matrices to generate a U by V by W output tensor, the operations comprising: transforming the X by Y by Z input tensor into an X' by Y' by Z' input tensor, where X' is less than or equal to X, Y' is less than or equal to Y, and Z' is greater than or equal to Z; obtaining a one or more modified convolutional neural network layer weight matrices, wherein (i) the modified weight matrix operates on the X' by Y' by Z' input tensor to generate a U' by V' by W' output tensor, and (ii) the U' by V' by W' output tensor comprises a transformed U by V by W output tensor, wherein U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W; and processing the X' by Y' by Z' input tensor using the modified weight matrix to generate the U' by V' by W' output tensor, wherein the U' by V' by W' output tensor comprises the U by V by W output tensor.

[0150] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order or sequential sequence shown to achieve the desired results. In certain embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A method performed using a convolutional neural network implemented on a hardware integrated circuit and configured to generate a classification output based on an input being an image or features extracted from an image, the method comprising: receiving an input tensor for a layer of the convolutional neural network having a stride greater than one, the input tensor having a plurality of dimensions and a corresponding plurality of inputs; generating a transformed input tensor from the input tensor by grouping the plurality of inputs of the input tensor by trading spatial extents or indices with respect to X and Y dimensions of the input tensor for depth extents or indices with respect to a Z dimension of the input tensor; transforming, based on the grouping of the plurality of inputs of the input tensor, the layer of the convolutional neural network having the stride greater than one into a superpixel layer of the convolutional neural network having a stride equal to one, the superpixel layer comprising a reduced number of kernel elements relative to an untransformed layer of the convolutional neural network; Processing the transformed input tensor using a modified weight matrix representing a weight matrix of the superpixel layer of the convolutional neural network with the stride equal to one, wherein the modified weight matrix is ​​a modified version of a weight matrix of the layer of the convolutional neural network, comprising: applying a convolution to an input of the transformed input tensor using the modified weight matrix; and In response to processing the transformed input tensor, generating a transformed layer output of the superpixel layer, the transformed layer output comprising an output mathematically corresponding to a neural network output generated by processing the input tensor using an unmodified version of the weight matrix of the layer of the convolutional neural network.

2. The method according to claim 1, wherein Applying the convolution involves: When the convolution filter of the modified weight matrix processes the input of the transformed input tensor, the input of the transformed input tensor is convolved with the corresponding weights of the modified weight matrix based on a kernel stride, and the kernel stride controls the shift of the convolution filter.

3. The method according to claim 1, wherein Transforming the layer of the convolutional neural network into the superpixel layer comprises: Applying a superpixel transformation to the convolutional neural network layer input of the input tensor and an unmodified convolutional neural network layer weight matrix to generate a superpixel convolutional neural network layer of a different shape but mathematically equivalent shape.

4. The method according to claim 1, wherein: Processing the transformed input tensor using the modified weight matrix of the superpixel layer requires fewer matrix multiplications than processing the input tensor using the unmodified weight matrix of the untransformed layer of the convolutional neural network.

5. The method according to any one of claims 1 to 4, wherein: A corresponding depth dimension of an input tensor to the transform is larger than a corresponding depth dimension of the input tensor.

6. A system for performing neural network computations using a convolutional neural network implemented on a hardware integrated circuit and configured to generate a classification output based on input that is an image or features extracted from an image, the system comprising: A processor and a non-transitory machine-readable storage device storing instructions executable by the processor to perform operations comprising: receiving an input tensor for a layer of the convolutional neural network having a stride greater than one, the input tensor having a plurality of dimensions and a corresponding plurality of inputs; generating a transformed input tensor from the input tensor by grouping the plurality of inputs of the input tensor by trading spatial extents or indices with respect to X and Y dimensions of the input tensor for depth extents or indices with respect to a Z dimension of the input tensor; transforming, based on the grouping of the plurality of inputs of the input tensor, the layer of the convolutional neural network having the stride greater than one into a superpixel layer of the convolutional neural network having a stride equal to one, the superpixel layer comprising a reduced number of kernel elements relative to an untransformed layer of the convolutional neural network; Processing the transformed input tensor using a modified weight matrix representing a weight matrix of the superpixel layer of the convolutional neural network with the stride equal to one, wherein the modified weight matrix is ​​a modified version of a weight matrix of the layer of the convolutional neural network, comprising: applying a convolution to an input of the transformed input tensor using the modified weight matrix; and In response to processing the transformed input tensor, generating a transformed layer output of the superpixel layer, the transformed layer output comprising an output mathematically corresponding to a neural network output generated by processing the input tensor using an unmodified version of the weight matrix of the layer of the convolutional neural network.

7. The system according to claim 6, wherein: Applying the convolution involves: When the convolution filter of the modified weight matrix processes the input of the transformed input tensor, the input of the transformed input tensor is convolved with the corresponding weights of the modified weight matrix based on a kernel stride, and the kernel stride controls the shift of the convolution filter.

8. The system according to claim 6, wherein: Transforming the layer of the convolutional neural network into the superpixel layer comprises: Applying a superpixel transformation to the convolutional neural network layer input of the input tensor and an unmodified convolutional neural network layer weight matrix to generate a superpixel convolutional neural network layer of a different shape but mathematically equivalent shape.

9. The system of claim 6, wherein: Processing the transformed input tensor using the modified weight matrix of the superpixel layer requires fewer matrix multiplications than processing the input tensor using the unmodified weight matrix of the untransformed layer of the convolutional neural network.

10. The system according to any one of claims 6 to 9, wherein: A corresponding depth dimension of an input tensor to the transform is larger than a corresponding depth dimension of the input tensor.

11. A non-transitory machine-readable storage device storing instructions for performing neural network computations using a convolutional neural network implemented on a hardware integrated circuit and configured to generate a classification output based on input that is an image or features extracted from an image, the instructions being executable by a processor to perform operations comprising: receiving an input tensor for a layer of the convolutional neural network having a stride greater than one, the input tensor having a plurality of dimensions and a corresponding plurality of inputs; generating a transformed input tensor from the input tensor by grouping the plurality of inputs of the input tensor by trading spatial extents or indices with respect to X and Y dimensions of the input tensor for depth extents or indices with respect to a Z dimension of the input tensor; transforming, based on the grouping of the plurality of inputs of the input tensor, the layer of the convolutional neural network having the stride greater than one into a superpixel layer of the convolutional neural network having a stride equal to one, the superpixel layer comprising a reduced number of kernel elements relative to an untransformed layer of the convolutional neural network; Processing the transformed input tensor using a modified weight matrix representing a weight matrix of the superpixel layer of the convolutional neural network with the stride equal to one, wherein the modified weight matrix is ​​a modified version of a weight matrix of the layer of the convolutional neural network, comprising: applying a convolution to an input of the transformed input tensor using the modified weight matrix; and In response to processing the transformed input tensor, generating a transformed layer output of the superpixel layer, the transformed layer output comprising an output mathematically corresponding to a neural network output generated by processing the input tensor using an unmodified version of the weight matrix of the layer of the convolutional neural network.

12. The non-transitory machine-readable storage device of claim 11, wherein: Applying the convolution involves: When the convolution filter of the modified weight matrix processes the input of the transformed input tensor, the input of the transformed input tensor is convolved with the corresponding weights of the modified weight matrix based on a kernel stride, and the kernel stride controls the shift of the convolution filter.

13. The non-transitory machine-readable storage device of claim 11, wherein: Transforming the layer of the convolutional neural network into the superpixel layer comprises: Applying a superpixel transformation to the convolutional neural network layer input of the input tensor and an unmodified convolutional neural network layer weight matrix to generate a superpixel convolutional neural network layer of a different shape but mathematically equivalent shape.

14. The non-transitory machine-readable storage device of claim 11 , wherein: Processing the transformed input tensor using the modified weight matrix of the superpixel layer requires fewer matrix multiplications than processing the input tensor using the unmodified weight matrix of the untransformed layer of the convolutional neural network.

15. The non-transitory machine-readable storage device according to any one of claims 11 to 14, wherein: A corresponding depth dimension of an input tensor to the transform is larger than a corresponding depth dimension of the input tensor.

Images