Super pixel method for convolutional neural network

Abstract

To provide a method, system and device for efficiently executing a calculation of convolutional neural network layer.SOLUTION: A method includes a step of converting an X×Y×Z input tensor into an X'×Y'×Z' input tensor, X' is X or less, Y' is Y or less, and Z' is Z or more. Also, the method includes a step of acquiring a corrected weight matrix, the corrected weight matrix generates a U'×V'×W' output tensor by acting on the X'×Y'×Z' input tensor, the U'×V'×W' output tensor includes a converted U×V×W output tensor, U' is U or less, V' is V or less, and W' is W or more. Further, the method includes a step of generating a U'×V'×W' output tensor by processing the X'×Y'×Z' input tensor by using the corrected weight matrix.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] background This specification relates to generating outputs by processing inputs through layers of neural networks. [Background technology]

[0002] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a given input. Some neural networks contain one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to another layer in the network, such as the next hidden layer or the output layer. Each layer of the network generates an output from the inputs it receives according to the current values ​​of each set of parameters. Summary of the Invention [Means for solving the problem]

[0003] overview In general, one innovative aspect of the subject matter described herein can be embodied in a method for more efficiently performing a convolutional neural network layer computation, the convolutional neural network layer being configured to receive an X×Y×Z input tensor and generate a U×V×W output tensor by processing the received input tensor with one or more convolutional neural network layer weight matrices. In some aspects, the method includes converting an X×Y×Z input tensor to an X′×Y′×Z′ input tensor, where X′ is less than or equal to X, Y′ is less than or equal to Y, and Z′ is greater than or equal to Z, and the method further includes obtaining one or more modified convolutional neural network layer weight matrices, where (i) the modified weight matrices operate on the X′×Y′×Z′ input tensor to generate a U′×V′×W′ output tensor, and (ii) the U′×V′×W′ output tensor comprises a transformed U×V×W output tensor, where U′ is less than or equal to U, V′ is less than or equal to V, and W′ is greater than or equal to W, and the method further includes generating a U′×V'×W′ output tensor by processing the X′×Y'×Z′ input tensor with the modified weight matrices, where the U′×V'×W′ output tensor comprises a U×V×W output tensor.

[0004] Other embodiments of this aspect include corresponding computer systems, devices, and computer programs stored on one or more computer storage devices, each configured to perform the actions of the methods described above. A system of one or more computers can be configured to perform specific operations or actions through software, firmware, hardware, or any combination thereof, which is installed on the system and, when operated, causes the system to perform the actions. One or more computer programs can be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause the device to perform the actions.

[0005] Each of the above and other embodiments may optionally include one or more of the following features, either alone or in combination.

[0006] In some implementations, X' is equal to the ceiling of X divided by a natural number N. So, Y' is equal to the ceiling of Y divided by a natural number M, and Z' is equal to Z multiplied by NM. equal.

[0007] In some implementations, an XxYxZ input tensor represents an XY first input, and an X'xY'xZ' input tensor represents an X'Y' super input, each super input including multiple first inputs.

[0008] In some implementations, the number of the plurality of first inputs is equal to NM inputs. In some implementations, the operation of the modified weight matrix on an X'×Y'×Z' input tensor is equivalent to the operation of a convolutional neural network layer weight matrix on an X×Y×Z input tensor.

[0009] In some implementations, the entries of the modified weight matrix include one or more convolutional neural network layer weight matrices.

[0010] In some cases, the modified weight matrix includes a matrix having a banded structure.

[0011] In some implementations, U' is equal to the ceiling of U divided by the natural number P. Thus, V' is equal to the ceiling of V divided by a natural number Q, and W' is equal to W multiplied by PQ.

[0012] In some implementations, a UxVxW output tensor represents a UV primary output, and a U'xV'xW' output tensor represents a U'V' super output, each super output including multiple primary outputs.

[0013] In some cases, the plurality of first outputs is equal to PQ outputs. In some implementations, a convolutional neural network layer includes a pooling sublayer.

[0014] In some cases, the method further includes replacing the pooling sublayer with striding on the transformed input tensor.

[0015] In some implementations, a convolutional neural network layer includes a stride S in the X dimension and a stride T in the Y dimension, and the relationship between the strides S, T, the output tensor U×V×W, and the transformed output tensor U'×V'×W' is given by U'=ceiling(U / NS) and V'=ceiling(V / MT).

[0016] In some cases, the U×V×W output tensor represents the U-V first outputs, and the U'×V'×W' output tensor represents the U'V' super-output, each super-output including multiple first outputs, the number of first outputs among the multiple first outputs depending on the dimensions of the convolutional neural network layer weight matrix.

[0017] In some implementations, the U×V×W output tensor represents the U-V first outputs, the U'×V'×W' output tensor represents the U'V' super-output, each super-output includes a plurality of first outputs, and the number of outputs among the plurality of first outputs depends on one or more of: (i) an architecture of the convolutional neural network layer; (ii) an architecture of the neural network model including the convolutional neural network layer; or (iii) a device that realizes the convolutional neural network layer.

[0018] In some cases, one or more convolutional neural network layer weight matrices This fix is ​​a software only fix.

[0019] The subject matter described herein can be implemented in particular embodiments to realize one or more of the following advantages.

[0020] Convolutional neural network layers can be computed using matrix multiplication operations. However, both software libraries and hardware matrix multiplication units may work less efficiently when the input or output depth of a convolutional neural network layer is shallow, because the input depth and output depth are mapped to the dimensions of one of the matrices being multiplied. As an example, an input image may have RGB data that results in an input depth of 3, and a convolutional neural network layer may generate an output of depth 16, where 3 and 16 are shallow in the sense described above. The reason for the inefficiency is that a typical modern computer architecture can process very deep layers up to a certain depth with the same or similar efficiency as shallow layers, due to the architectural design of its caches, memory accesses, and computational units.

[0021] A typical operation in a convolutional neural network layer maps one input, e.g., one pixel, to one row or column of a matrix multiplication operation. A superpixel convolutional neural network layer, as described herein, groups multiple inputs, e.g., multiple input pixels, that represent data in a depth dimension or index (e.g., Z dimension) over a spatial dimension or index (e.g., X and Y dimensions). The grouped inputs are more efficiently realized and processed using a matrix multiplication unit, thereby increasing the computational efficiency of a system that implements the superpixel convolutional neural network layer technique. For example, the in-memory representation of a neural network layer in a superpixel CNN configured to process grouped inputs is more compact, so that the neural network layer can store and process the inputs it receives in a low-latency access memory that has a limited size. A system that implements such a technique can thus benefit from the higher computational throughput, lower computational latency, shorter input / output transfer time, and better energy utilization of convolutional neural networks on any matrix machine, e.g., a central processing unit, a graphics processing unit, or a dedicated neural network processing unit. In addition, the system may implement the techniques without modifying system hardware; i.e., the techniques may be realized entirely using system software, and therefore may be applicable to all machines used to perform matrix multiplication.

[0022] A superpixel convolutional neural network layer, as described herein, may include a smaller number of kernel elements compared to a standard convolutional neural network layer, thus reducing the number of matrix multiplication operations that need to be performed when using a superpixel convolutional neural network layer.

[0023] In addition, a system implementing a superpixel convolutional neural network layer may convert a convolutional neural network layer having a stride greater than one to a superpixel convolutional neural network layer having a stride equal to one, as described herein. A superpixel convolutional neural network layer having a stride equal to one may be necessary for some computer architectures implementing a neural network system, for example, computer architectures that can only directly perform convolutional calculations having a stride equal to one, such as architectures that include matrix multiplication units, such as systolic arrays. In addition, in some cases, a superpixel convolutional neural network layer having a stride equal to one may be computed more efficiently on the above architectures. This may be necessary, for example, for a convolutional neural network layer having a stride greater than one. This may eliminate the need for off-chip processing or other solutions that require performing large convolutional calculations, thereby improving the computational efficiency and convenience of systems that implement superpixel convolutional neural network layers.

[0024] The subject matter described herein also relates to image recognition methods / systems that use superpixel convolutional neural network layers.

[0025] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0026] [Figure 1A] This is an example of a superpixel convolutional neural network system. [Figure 1B] FIG. 1 illustrates the relationship between a convolutional neural network layer and a superpixel convolutional neural network layer. [Diagram 2]FIG. 1 is a flow diagram of an example process for efficiently performing computations of a convolutional neural network layer. [Diagram 3] 1 is an illustrative example of superpixel input tensor transformation. [Figure 4] 1 is an illustrative example of convolutional neural network layer weight matrix modification. [Diagram 5] 1 is an example of an illustration of weight matrix modification of a convolutional neural network layer including superpixel kernel striding processing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] Like reference numbers and designations in the various drawings indicate like elements. Detailed Description 1A illustrates an example superpixel convolutional neural network system 100. Superpixel convolutional neural network system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations that can implement the systems, components, and techniques described below.

[0028] The superpixel convolutional neural network system 100 is a machine learning system that receives a system input 102 and generates a system output 104 from the system input 102.

[0029] The superpixel convolutional neural network system 100 can be configured to receive digital data input and generate a score or classification output based on the input. For example, if the input to the superpixel convolutional neural network system 100 is an image or features extracted from an image, the output that the superpixel convolutional neural network system 100 generates, for example, for a given image using one or more fully connected or convolutional neural network layers included in the neural network system 100, can be a score for each of a set of object categories. Each score can represent an estimated likelihood that the image contains an image of an object belonging to that category.

[0030] In particular, the superpixel convolutional neural network system 100 implements a convolutional neural network that includes multiple convolutional neural network layers, including convolutional neural network layer A 106, convolutional neural network layer B 108, and superpixel convolutional neural network layer 110. A convolutional neural network layer is a layer whose neurons or nodes are arranged in a three-dimensional array, i.e., a pixel array with a width of 100 . A convolutional neural network layer is a neural network layer logically arranged in size, height, and depth. Neurons in a convolutional neural network layer are connected to a small region of the layer preceding it, called a receptive field or kernel patch, and each neuron in a convolutional neural network layer detects the same features, which constitutes the property of translation invariance. A convolutional neural network layer can be associated with a respective weight matrix or kernel that stores the current values ​​of the parameters of the layer. The layer generates outputs from inputs according to the current values ​​of the parameters of the neural network layer, for example by performing multiple matrix multiplication operations. An example of a convolutional neural network layer is described in more detail in "Large-scale Video Classification with Convolutional Neural Networks" by Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei, Proceedings of International Computer Vision and Pattern Recognition (CVPR 2014), IEEE.

[0031] Each layer of the superpixel convolutional neural network is configured to receive a respective layer input and process the layer input to generate a respective layer output from the input. For example, a convolutional neural network layer included in the superpixel convolutional neural network system 100 may be configured to receive an X×Y×Z input tensor and process the received input tensor with one or more convolutional neural network layer weight matrices or kernels to generate a U×V×W output tensor. A tensor is a multidimensional array of numbers or other values, e.g., strings, with a particular order corresponding to the dimensionality of the array. For example, a scalar value is a zero-order tensor, a vector of numbers is a first-order tensor, and a matrix is ​​a second-order tensor.

[0032] For example, a convolutional neural network layer may be configured to receive a 64×64×3 input tensor representing an image including 64×64 pixels with a feature depth of three, where the feature depth is represented by red, green, and blue feature channels. The convolutional neural network layer may generate respective output tensors by processing the received 64×64×3 input tensor with one or more respective convolutional neural network layer matrices.

[0033] In some implementations, the convolutional neural network layer may include one or more hyperparameters that control the size and output volume of the convolutional neural network layer. For example, the convolutional neural network layer may include a kernel stride that controls the amount that the convolutional filter or weight matrix shifts in X and Y between evaluations of the filter on the input data. In some implementations, the convolutional neural network layer may include or be connected to a pooling layer that is a form of nonlinear downsampling. For example, the convolutional neural network layer may include a pooling stride that operates on a depth slice of the input tensor to spatially resize the input tensor. Continuing with the example described above, the convolutional neural network layer may thus generate respective U×V×W outputs, where U is less than or equal to X, V is less than or equal to Y, and W is greater than or equal to Z, and the output tensor is, for example, a 32×32×32 output tensor (e.g., corresponding to a 7×7 kernel stride and a 2×2 pooling stride). In general, kernel striding can be thought of as omitting the initial computation (e.g., "skip X by 2 and Y by 2 between evaluations"), while pooling is a post-pass after stride 1 evaluations (e.g., in pooling, the computations are generated with a stride of 1). Then, the maximum value of each group in a pooling window is selected, e.g. a 2x2 portion of the input data.

[0034] At least one of the superpixel neural network systems 100 The convolutional neural network layer is a superpixel convolutional neural network layer, such as superpixel convolutional neural network layer 110. A superpixel convolutional neural network layer is a convolutional neural network layer that applies a superpixel transformation (alternatively, a correction) to received convolutional layer inputs and convolutional neural network layer weight matrices to generate a geometrically different but mathematically equivalent superpixel convolutional neural network layer.

[0035] The superpixel convolutional neural network layer 110 can receive inputs, such as input 112, from a convolutional neural layer prior to the superpixel convolutional neural network layer 110. For example, the received inputs can include outputs generated by convolutional neural network layers prior to the superpixel convolutional neural network layer 110.

[0036] The superpixel convolutional neural network layer 110 may include a superpixel generator 114. The superpixel generator 114 is configured to receive convolutional neural network inputs and convert the received inputs into superpixel layer inputs. The superpixel generator 114 groups components of the received inputs in a depth dimension or index, e.g., Z dimension, over a spatial dimension or index, e.g., X and Y dimensions. As a result, the depth of the convolutional neural network layers in the Z layer is no longer shallow, thereby avoiding inefficiencies due to the shallowness of the Z layer.

[0037] For example, the superpixel convolutional neural network layer 110 may receive as input an X×Y×Z input tensor, e.g., representing XY (X by Y) with depth Z. The superpixel generator 114 is configured to convert this X×Y×Z input tensor into an X′×Y′×Z′ superpixel input tensor, where X′ is less than or equal to X, Y′ is less than or equal to Y, and Z′ is greater than or equal to Z. An example of the conversion of the superpixel input tensor performed by the superpixel generator is shown in FIG. 3 below.

[0038] The superpixel convolutional neural network layer 110 includes a superpixel weight matrix 116. The superpixel weight matrix 116 is a modified (alternatively, transformed) convolutional neural network layer weight matrix corresponding to the convolutional neural network layer. For example, the convolutional neural network layer weight matrix may be modified according to the selection of each convolutional neural network layer to be transformed into a superpixel convolutional neural network layer. The superpixel weight matrix 116 operates on the transformed convolutional neural network layer input.

[0039] For example, a convolutional neural network layer corresponding to the superpixel neural network layer 110 may be configured to receive an X×Y×Z input tensor. The superpixel neural network layer 110 may convert the received input into an X′×Y′×Z′ superpixel input tensor. In this case, the superpixel weight matrix may include a modified weight matrix that operates on the X′×Y′×Z′ input tensor to generate a U′×V′×W′ output tensor. X′ is not necessarily equal to U′, Y′ is not necessarily equal to V′, and Z′ is not necessarily equal to W′. The U′×V′×W′ output tensor is a transformed U×V×W output tensor, where U′ is less than or equal to U, V′ is less than or equal to V, and W′ is greater than or equal to W. For example, the U′×V′×W′ output tensor may represent a superpixel output tensor, e.g., a grouping of components of a U×V×W output tensor. In some implementations, using grouping to create a superpixel input tensor or represent a superpixel output tensor may not be the same, e.g., the number of components and the number of groups of components may vary. The manner in which the weights are grouped can vary. An example convolutional neural network layer weight matrix modification is described below with reference to FIG. 4. In addition, an example algorithm (with comments) for generating superpixel weight matrices (with or without striding, e.g., stride_x=stride_y=1) is as follows: do.

[0040]

number

number

[0041] An example of a SuperIndex routine used in the above description can be defined, but is not limited to, by the following:

[0042]

number

[0043] The above is for illustrative purposes only: for example, in some implementations, the SuperIndex routine may not depend on the packing order within superpixels.

[0044] The superpixel convolutional neural network layer 110 processes the transformed input tensors with the modified weight matrix to generate a transformed convolutional neural network layer output, e.g., a superpixel output 118. This generated transformed convolutional neural network layer output may be mathematically equivalent to a neural network output generated by processing the untransformed convolutional neural network input with the unmodified convolutional neural network layer weight matrix. All output values ​​of the unmodified computation may be present in the generated transformed convolutional neural network layer output in a form that can be calculated by performing an inverse homomorphic transformation, as described below with reference to FIG. 1B. For example, the generated transformed convolutional neural network layer output may include a superset of outputs that includes neural network outputs corresponding to neural network outputs generated by processing the (untransformed) convolutional neural network with the (unmodified) convolutional neural network layer. In some implementations, additional outputs included in this superset may be provided to subsequent layers of the superpixel neural network system 100. For example, the additional outputs may correspond to "partial sums" of outputs "off the edges" of the originally requested untransformed output. If the superpixel computation processes a larger "tile" of computations with larger X and Y than is efficient for processing a single superpixel computation, the partial sums can be stored and reused for greater efficiency. Processing inputs with superpixel convolutional neural network layers is described in more detail below with reference to FIG. 2.

[0045] In some implementations, the superpixel convolutional neural network layer 110 may include a superpixel unformatter configured to receive the superpixel outputs and convert the received superpixel outputs into a regular order output. For example, if a current superpixel convolutional neural network layer generates 4×4 superpixel outputs, while a next superpixel convolutional neural network layer is configured to process 8×8 superpixels, the system may convert the 4×4 superpixels into 8×8 superpixels by using the superpixel unformatter. In some implementations, the superpixel unformatter for a superpixel convolutional neural network layer, e.g., layer 110, may perform ungrouping of the outputs generated by the weight matrix according to a superpixel weight matrix, e.g., weight matrix 116. In other implementations, the superpixel unformatter for a superpixel convolutional neural network layer may include a superpixel unformatter configured to ungroup the outputs generated by the weight matrix according to a superpixel weight matrix, e.g., weight matrix 116. The superpixel generator can be included as part of or together with the superpixel generator for the next superpixel convolutional neural network layer in the network layer.

[0046] The superpixel convolutional neural network layer 110 may be included at various locations in the series of neural network layers. In some implementations, multiple superpixel convolutional neural network layers may be included in the series of neural network layers. For example, in some implementations, one or more of the convolutional neural network layers A and B may be a superpixel convolutional neural network layer. In implementations in which multiple superpixel convolutional neural network layers are included in the series of neural network layers, each superpixel convolutional neural network layer may generate a respective super-input by applying a respective transformation to the received input, and each generated super-input may include a super-input with a different size.

[0047] An example layout for a system of multiple superpixel convolutional neural network layers is shown in the table below.

[0048] [Table 1]

[0049] In the above example layout, most neural network layers contain 8x8 sets of superpixels per layer. The number of pixels per superpixel is adjusted according to the kernel stride processing. The "Original Input Depth (Z)" column shows the input depth of the original neural network model, and the "Transformed Input Depth (U)" column shows the input depth after transforming the input and weights with the specified pixels per superpixel.

[0050] A second example of the layout of a system consisting of multiple superpixel convolutional neural network layers is shown in the table below.

[0051] [Table 2]

[0052] The superpixel convolutional neural network system 100 can be trained on multiple batches of training examples to determine post-trained values ​​of the parameters of the neural network layers, i.e., to adjust the values ​​of the parameters from their initial values ​​to their post-trained values. For example, during training, the superpixel convolutional neural network system 100 can process a batch of training examples to generate a respective neural network for each training example in the batch. The neural network output can then be used to adjust the values ​​of the parameters of the components of the superpixel convolutional neural network system 100, for example, through conventional gradient descent and backpropagation neural network training techniques. Since the modified weight matrix makes NM copies of each original weight during the backpropagation training process (N and M represent the superpixel size, as described below with reference to step 202 of FIG. 2), the system can determine the net value change of each original weight by adding the weight gradient across all NM copies. Training the superpixel neural network layers is an inverse process that may prioritize spatial dimensions or indices over depth dimensions or indices.

[0053] Once the neural network training is complete, the superpixel convolutional neural network system 100 can receive a new neural network input to be processed and process the neural network input through the neural network layers according to the trained values ​​of the parameters of the components of the superpixel convolutional neural network system 100 to generate a new neural network output for the input.

[0054] In some implementations, the superpixel neural network system 100 may include one or more deconvolutional neural network layers. In these cases, the same descent can be obtained by reversing the components and techniques described above, i.e., the superpixel deconvolutional neural network layers can group multiple inputs in favor of spatial magnitude or index over depth magnitude or index and modify the respective deconvolutional neural network layer weight matrices accordingly.

[0055] Figure 1B shows the convolutional neural network layers and the superpixel convolutional neural network. FIG. 160 shows the relationship with the network layer.

[0056] As shown in FIG. 160, a convolutional neural network layer can be configured to generate activations O 164 as the layer output by performing a convolution operation “convolve” 162 on input activations I and a weight matrix W.

[0057] Similarly, a superpixel convolutional neural network layer, such as the modified or transformed convolutional neural network layer described above with reference to FIG. 1A, can be configured to generate activations 170 as the transformed layer output by performing a convolution operation “convolve” 168 on the transformed input activations Transform(I) and the transformed weight matrix Transform(W).

[0058] FIG. 160 shows two homomorphism arrows connecting the input I to the transformed input Transform(I) and the weight matrix W to the transformed weight matrix Transform(W). These homomorphism arrows represent homomorphic relationships between the original convolutional neural network layer and the modified convolutional neural network layer. These arrows, for example, map the original model activations and weights to the modified or transformed model activations and weights. Similarly, FIG. 160 shows an inverse homomorphism arrow connecting the transformed output Transform(O) to the output O. The inverse homomorphism represents an inverse homomorphism relationship between the original convolutional neural network layer and the modified convolutional neural network layer. This arrow, for example, maps the transformed output back to the original output.

[0059] An example of an algorithm for transforming the original model activations is as follows.

[0060]

number

[0061] An example of an inverse homomorphism algorithm is as follows.

[0062]

number

[0063] As shown in FIG. 160, the number of operations required to generate a convolutional neural network layer output using a superpixel convolutional neural network layer may be greater than the number of operations required by the corresponding original neural network layer because the input I and the weight matrix W may need to be transformed before applying the convolution. However, a superpixel convolutional neural network layer architecture that performs this greater number of operations may be computationally faster to run than the fewer operations performed by the corresponding original neural network layer.

[0064] 2 is a flow diagram of an example process 200 for efficiently performing computations of a convolutional neural network layer configured to receive an XxYxZZ input tensor and generate a UxVxW output tensor by processing the received input tensor with one or more convolutional neural network layer weight matrices. For convenience, the process 200 is described as being performed by a system of one or more computers located at one or more locations. By way of example, a superpixel convolutional neural network system suitably programmed in accordance with this specification, such as the superpixel convolutional neural network system 100 of FIG. 1A, may perform the process 200.

[0065] The system converts an X×Y×Z input tensor into an X'×Y'×Z' superpixel input tensor (step 202). The system converts the input tensor by grouping multiple inputs in favor of depth dimension or index over spatial dimension or index. For example, the X'×Y'×Z' input tensor may have different dimensions than the X×Y×Z input tensor. More specifically, X' may be less than or equal to X, Y' may be less than or equal to Y, and Z' may be greater than or equal to Z. In some implementations, X' may be a ceiling of X divided by a natural number N, e.g., the smallest integer greater than or equal to X divided by a natural number N. Y' is equal to the ceiling of Y divided by the natural number M, and Z' is equal to Z multiplied by NM.

[0066] An X×Y×Z input tensor can represent XY first inputs. For example, an X×Y×Z input tensor can represent an image of X×Y pixels at a feature depth Z. Similarly, an X'×Y'×Z' input tensor can represent X'Y' super-inputs, each super-input including multiple first inputs. For example, an X'×Y'×Z' input tensor can represent an image of X'×Y' super-pixels at a depth Z'. In some implementations, the number of multiple first inputs is equal to NM inputs. For example, each super-input may be composed of NM first inputs. An example of a transformation of an input tensor is shown in FIG. 3 below.

[0067] The system obtains one or more modified convolutional neural network layer weight matrices (step 204). The modified weight matrices operate on the X'×Y'×Z' input tensors to generate a U'×V'×W' output tensor. In some implementations, the operation of the modified weight matrices on the X'×Y'×Z' input tensors is equivalent to the operation of the convolutional neural network layer weight matrices on the X×Y×Z input tensors, e.g., the operation of the modified weight matrices on the X'×Y'×Z' input tensors produces the same effect. The U'×V'×W' output tensor may include a transformed U×V×W output tensor, where U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W. For example, the U'×V'×W' output tensor may represent a grouping of multiple outputs from the U×V×W output tensor, where a depth dimension or index is prioritized over a spatial dimension or index as described above.

[0068] The modified weight matrix entries may include one or more convolutional neural network layer weight matrices, e.g., the modified weight matrix may include a block matrix, which may be interpreted as being divided into sections called blocks or sub-matrices. In some implementations, the modified weight matrix entries may exhibit a banded structure. For example, a convolutional neural network layer may be represented by a fully connected matrix, where each input is connected to each output. The fully connected matrix exhibits a banded structure. With a banded structure, most entries of the matrix are zero and a small number of diagonal bands in the matrix have non-zero entries.

[0069] In general, for one dimension, the number of modified kernel elements is ceiling((superinput size + original convolutional neural network layer kernel width - 1) / super The size of the input will be equal to (ceiling((superinput_size + original convolutional neutral network layer kernel width - 1) / superinput_size)). The modification of the network layer weight matrices is described below with reference to FIG.

[0070] Optionally, the system can implement efficient kernel stride processing for the convolutional neural network layer (step 206). For example, in some implementations, the convolutional neural network layer can include a pooling sublayer. In these cases, the system can implement the pooling sublayer as a pooling sublayer for the transformed input tensor X'×Y'×Z. ' can be replaced by striding on '. For example, as described below with reference to step 208, in some implementations, a U×V×W output tensor can represent a UV first output, and a U'×V'×W' output tensor can represent a U'V' super output, each super output including multiple first outputs. In such cases, the effective kernel striding size achieved may not be equal to the number of outputs of the multiple first outputs. The effect of kernel striding can be incorporated into the modified weight matrix. For example, a (non-transformed) convolutional neural network layer can generate a layer output O by applying a convolution to each input I and a respective weight matrix W and applying striding to the convolved input I and weight matrix W, i.e., by performing two operations. A transformed superpixel convolutional neural network layer with striding can generate a transformed layer output transform(O) by applying a convolution to each transformed input transform(I) and a strided transformed weight matrix strided_transform(2,W).

[0071] In some implementations, the convolutional neural network layer may already include kernel striding. In these cases, the system may replace this kernel striding with efficient kernel striding. The system may achieve efficient kernel striding by using different input and output superpixel sizes, where the ratio of the superpixel sizes is equal to the stride. For example, if the convolutional neural network layer includes a stride S in the X dimension and a stride T in the Y dimension, and the convolutional neural network layer is converted to a superpixel convolutional neural network layer with a size of the superpixel input being NM by selecting an output superpixel size (N / S)(M / T), the system achieves kernel striding of the superpixel modified weight matrix. In some implementations, the relationship between the stride S, the output tensor U×V×W, and the converted output tensor U′×V′×W′ can be shown by U′=ceiling(U / NM) and V′=ceiling(V / MT). The realization of efficient kernel striding is described below with reference to FIG. 5.

[0072] The system processes the X'xY'xZ' input tensor with the modified weight matrix to generate a transformed convolutional neural network layer output, e.g., a U'xV'xW' output tensor (step 208). The U'xV'xW' output tensor includes the UxVxW output tensor, e.g., the U'xV'xW' output tensor may include a superset of outputs including the UxVxW output tensor.

[0073] The U'xV'xW' output tensor may have different dimensions than the UxVxW output tensor. More specifically, U' may be equal to the ceiling of U divided by the natural number P, V' may be equal to the ceiling of V divided by the natural number Q, and W' may be equal to W multiplied by PQ. In some implementations, the natural numbers P and Q may be equal to the natural numbers N and M, respectively, described above with reference to step 202. Additionally, in some implementations, P and Q of the current superpixel convolutional neural network layer may be equal to N and M of the next superpixel convolutional neural network layer, for example, if the system does not include a superpixel generator and a superpixel unformatter between the current superpixel convolutional neural network layer and the next superpixel convolutional neural network layer.

[0074] The U×V×W output tensor can represent the UV first output at depth W. For example, the U×V×W output tensor can represent the feature map of the U×V output at depth W. Similarly, the U'×V'×W' output tensor can represent the U'V' super output, with each super The super-output includes a number of first outputs. For example, a U'xV'xW' output tensor can represent an image of U'xV' pixels at depth W'. In some implementations, the number of the multiple first outputs may be equal to PxQ output pixels, e.g., each super-output may be composed of PQ first outputs.

[0075] The natural numbers P and Q may be different from the natural numbers N and M described above with reference to step 202. For example, the size of the super-input may not be the same as the size of the super-output. In some implementations, the number of first outputs among the plurality of first outputs, e.g., the size of the super-output, depends on a dimension of the convolutional neural network layer weight matrix. In some implementations, the number of outputs among the plurality of first outputs depends on one or more of: (i) an architecture of the convolutional neural network layer, (ii) an architecture of a neural network model that includes the convolutional neural network layer, or (iii) an apparatus that realizes the convolutional neural network layer.

[0076] As an example, assume that the input size of the first convolution layer is X×Y×Z and produces an output size of U×V×W, and that the input size of the superpixel transformation convolution layer is X'×Y'×Z' and produces an output size of U'×V'×W'. Let KL denote the number of convolution kernel elements in the weight matrix, and K'L' denote the number of convolution kernel elements in the modified weight matrix. Assume further that the input and output pixels / superpixels are identical and specified by N and M. Then, U'=ceiling(U / N), V'=ceiling(V / M), and W'=W×N×M. Additionally, K'=ceiling((N+K-1) / N) and L'=ceiling((M+L-1) / N). Cases where PQ may not be equal to NM include cases where the system may be performing kernel stride processing or the number of pixels per transformed superpixel has changed from input to output.

[0077] 3 is an example diagram 300 of a superpixel input tensor transformation. For example, this example diagram illustrates the superpixel input tensor transformation described above with reference to step 202 of FIG. 2. The illustrated transformation may be performed by a system of one or more computers located in one or more locations. By way of example, a superpixel generator appropriately programmed in accordance with this specification, such as superpixel generator 114 of FIG. 1A, may perform the illustrated transformation.

[0078] The illustrative example shows an example XxYxZ input tensor 302. As shown in FIG. 3, the input tensor includes XY inputs, each of depth Z. As previously described with reference to FIG. 2, the XxYxZ input tensor can be converted to an X'xY'xZ' superpixel input tensor by grouping the inputs. During this grouping, index or size in the depth dimension (Z dimension) takes precedence over index or layout in the spatial dimensions (X and Y dimensions). In the illustrative illustration 300, the XxYxZ input tensor has been converted to an X / 2xY / 2x4Z superpixel input tensor 304. As shown in FIG. 3, the superpixel input tensor includes (X / 2)(Y / 2) inputs, each of depth 4Z. Each superpixel input included in the input tensor 304 represents four of the original inputs, and thus represents four times the amount of data represented by the original inputs.

[0079] FIG. 4 is an example diagram 400 of convolutional neural network layer weight matrix modification. For example, this example diagram illustrates the convolutional neural network layer weight matrix modification described above with reference to step 204 of FIG. 2. The modifications shown may be performed by a system of one or more computers located at one or more locations. By way of example, a superpixel convolutional neural network layer weight matrix may be appropriately programmed in accordance with the present disclosure. A neural network layer, such as the superpixel convolutional neural network layer 110 of FIG. 1A, can perform the illustrated correction.

[0080] In step (a), the illustrative example 400 shows a 3×3 kernel patch 402 of a convolutional neural network layer, where each element of the kernel patch includes a 2-dimensional weight matrix, with dimensions equal to the original input depth×original output depth suitable for matrix multiplication, e.g., Z×W. The illustrative example further shows 2×2 lattices 404-410 representing the inputs. The inputs are grouped into 2×2 super-inputs, where each super-input includes the original 2×2 inputs. The illustrative example shows a super-output 412 that includes four original outputs.

[0081] In step (b), an illustrative example 400 shows the same two-dimensional convolution shown in step (a) extended to a dot product calculation for each of the four original outputs. The shading behind the 2x2 grid, such as shading 414, shows which 3x3 subsets of the input data affect each output. As the shading for each grid shows, all super-inputs affect all outputs, but the useful fraction of each super-input is 9 / 16.

[0082] In step (c), the illustrated example 400 shows an example of modified two-dimensional weight matrices 416-422. These weight matrices can be used to calculate the super-output. Each of the matrices 416-422 is a kernel element that constitutes a 2×2 modified kernel patch. The dimensions of the matrices 416-422 are 4Z×4W, and therefore 16 times larger than the original two-dimensional matrix of the kernel patch 402. The vertical axis represents the input depth, the inputs in the super-input, and the kernel elements. The horizontal axis represents the output depth, and the outputs in the super-output. This illustrated example shows the matrix multiplication in step (c). The column 424 represents an input, and if an input is shown at the same height as an entry of the modified weight matrices 416-422, it is multiplied by this weight. All the products of the weights and inputs in one column are added together to obtain the sum of the outputs in the super-output. Shown are 16 original inputs that contribute to the computation of a total of 4 super-inputs, eg 1 super-output, eg 4 original outputs.

[0083] Each of the four super-inputs has a different X and Y coordinate relationship to the super-output. For illustration purposes, FIG. 4 shows 2×2 pixels per superpixel and 2×2 sets of superpixels. Thus, 2×2=4 weight matrices are depicted. The number of pixels per superpixel and per set of superpixels may vary based on, for example, system architecture, among other factors. For example, in some cases there may be 4×4 pixels per superpixel, or a larger image size of 100×100 superpixels (hence 400×400 original input pixels). In this case, there will still be 2×2=4 weight matrices as long as ceiling((pixels per superpixel+kernel size−1) / pixels per superpixel) (ceiling((pixels_per_superpixel + kernel_size−1) / pixels_per_superpixel)) is less than or equal to 2 in each dimension.

[0084] As noted above, the modified weight matrices are kernel elements. In this regard, the modification process can be viewed as reducing the number of kernel elements in the patch by changing a 3×3 convolution on the input to a 2×2 convolution on the super-input, with each kernel element being a larger two-dimensional matrix, e.g., four times larger in both rows and columns (corresponding to the input depth Z and output depth W). The stacked modified weight matrices 416-422 include one copy of each of the elements of the 3×3 kernel 402. In some implementations, the modified weight matrices are, for example, similarly labeled kernel elements in the diagonal direction, e.g., as shown by diagonal 426. , may exhibit a band structure.

[0085] 5 is an example diagram 500 of a convolutional neural network layer weight matrix modification including superpixel kernel striding. For example, this diagram shows an efficient realization of the kernel striding described above with reference to step 206 of FIG. 2. The illustrated implementation may be performed by a system of one or more computers located at one or more locations. By way of example, a superpixel convolutional neural network layer appropriately programmed in accordance with this specification, such as the superpixel convolutional neural network layer 110 of FIG. 1A, may perform the illustrated implementation.

[0086] The illustrated example 500 shows a weight diagram with a 5×5 kernel patch 506 whose elements are two-dimensional weight matrices whose dimensions are equal to the original input depth×original output depth, e.g., Z×W, 4×4 super-inputs 502 each containing a 4×4 original input, and 2×2 super-outputs 504 each containing a 2×2 original output. An example kernel stride is 2×2. The illustrated example 500 includes four superpixel kernel elements 508-514. Each superpixel kernel element is a 16Z×4W matrix. The input / super-input to output / super-output is scaled by the 2×2 kernel stride, so there are only 4 outputs for each 16 inputs.

[0087] The embodiments and functional operations of the subject matter described herein can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or in a combination of one or more of these. The embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or control of the operation of a data processing apparatus. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information to be transmitted to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage medium, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. However, the computer storage medium is not a propagating signal.

[0088] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. The apparatus may include special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program, such as code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0089] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) may be written in any programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be implemented in any form, including as a stand-alone program or as a module, or as a component, subroutine, or other unit suitable for use in a computing environment. A computer program may or may not correspond to a file in a file system. A program may be stored in part of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, or in a file dedicated to the program, or in multiple associated files, e.g., multiple files storing one or more modules, subprograms, or portions of code.

[0090] As used herein, "engine" or "software engine" refers to a software-implemented input / output system that provides an output that is distinct from an input. An engine may be a coding block of functionality, such as a library, a platform, a software development kit ("SDK"), or an object. Each engine may be implemented on any suitable type of computing device, such as a server, a mobile phone, a tablet computer, a notebook computer, a music player, an e-reader, a laptop or desktop computer, a PDA, a smartphone, or other stationary or mobile device that includes one or more processors and a computer-readable medium. In addition, two or more of the engines may be implemented on the same computing device or on different computing devices.

[0091] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may be performed by, and an apparatus may be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0092] A computer suitable for executing a computer program may be based, by way of example, on a general purpose or special purpose microprocessor or both, or on any other type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions, and one or more memory devices for storing instructions and data. Typically, a computer includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to said mass storage devices for receiving data from, transferring data to, or both. However, a computer need not have such devices. Additionally, a computer may be embedded in another device, such as, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash device, to name a few.

[0093] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, dedicated logic circuitry.

[0094] To interact with a user, embodiments of the subject matter described herein may provide information to a user. The computer may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying the image, and a keyboard and pointing device, such as a mouse or trackball, for allowing the user to provide input to the computer. Other types of devices may be used to interact with the user. For example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Additionally, the computer may interact with the user by sending documents to or receiving documents from a device used by the user, for example by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0095] Embodiments of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., as a data server, or a middleware component, e.g., an application server, or a front-end component, e.g., a client computer having a graphical user interface or web browser to enable user interaction with an implementation of the subject matter described herein, or any combination of one or more of such back-end, middleware, or front-end components. The components of the system may be interconnected by a form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks ("LANs") and wide area networks ("WANs"), e.g., the Internet.

[0096] A computing system may include clients and servers. Clients and servers are typically remote from each other and typically interact through a communications network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having the client-server relationship. In some embodiments, a server sends data, e.g., HTML pages, to a user device acting as a client, e.g., for displaying the data to a user interacting with the user device and for receiving user input from the user. Data generated at the user device, e.g., as a result of the user's interaction, can be received from the user device at the server.

[0097] Although the present specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in the present specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. In addition, although the features may be described as functioning in a particular combination and initially claimed as such, one or more features included in the claimed combination may, in some cases, be omitted from the combination.

[0098] Similarly, although operations are shown in a particular order in the figures, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed to achieve desired results. Multitasking and parallel processing may be advantageous in certain situations. Additionally, the separation of various system modules and components in the above embodiments should not be understood to require such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0099] Further implementations of the disclosed techniques are described in the following examples. Example 1: A computer-implemented method for more efficiently performing a convolutional neural network layer computation, the convolutional neural network layer configured to receive an X×Y×Z input tensor and generate a U×V×W output tensor by processing the received input tensor with one or more convolutional neural network layer weight matrices, the method comprising converting the X×Y×Z input tensor into an X′×Y′×Z′ input tensor, where X′ is less than or equal to X, Y′ is less than or equal to Y, and Z′ is greater than or equal to Z, and converting the one or more modified convolutional neural network layer weight matrices into a U×V×W output tensor. (i) the modified weight matrix is ​​operated on the X'×Y'×Z' input tensor to generate a U'×V'×W' output tensor; and (ii) the U'×V'×W' output tensor comprises a transformed U×V×W output tensor, where U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W; and (ii) the U'×V'×W' output tensor comprises a transformed U×V×W output tensor, where U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W, and

[0100] Example 2: X' is equal to the ceiling of X divided by a natural number N, and Y' is equal to a natural number M. and Z' is equal to Z multiplied by NM, as described in Example 1.

[0101] Example 3: The method of example 2, wherein the X×Y×Z input tensor represents an XY first input, and the X′×Y′×Z′ input tensor represents an X′Y′ super input, each super input including a plurality of first inputs.

[0102] Example 4: The method of example 3, wherein the number of the plurality of first inputs is equal to NM inputs. Example 5: The method of any one of Examples 1 to 4, wherein the action of the modified weight matrix on the X'×Y'×Z' input tensor is equivalent to the action of the convolutional neural network layer weight matrix on the X×Y×Z input tensor.

[0103] Example 6: The method of any one of Examples 1 to 5, wherein the modified weight matrix entries include one or more convolutional neural network layer weight matrices.

[0104] Example 7: The method of any one of Examples 1 to 6, wherein the modified weight matrix includes a matrix having a band structure.

[0105] Example 8: U' is equal to the ceiling of U divided by the natural number P, and V' is equal to the natural number Q. The method of any one of Examples 1 to 7, wherein W′ is equal to the ceiling of V divided by PQ, and W′ is equal to W multiplied by PQ.

[0106] Example 9: The method of Example 8, wherein the convolutional neural network layer includes a stride S in the X dimension and a stride T in the Y dimension, and the relationship between the strides S, T, the output tensor U×V×W, and the transformed output tensor U′×V′×W′ is given by U′=ceiling(U / NS) and V′=ceiling(V / MT).

[0107] Example 10: The method of example 8 or example 9, wherein the U×V×W output tensor represents a UV first output, and the U′×V′×W′ output tensor represents a U′V′ super output, each super output including a plurality of first outputs.

[0108] Example 11: The method of example 10, wherein the plurality of first outputs is equal to PQ outputs. Example 12: The method of any one of Examples 1 to 11, wherein the convolutional neural network layer includes a pooling sublayer.

[0109] Example 13: The method of Example 12, further comprising replacing the pooling sublayer with strided processing on the transformed input tensor.

[0110] Example 14: The method of Example 12 or Example 13, wherein the convolutional neural network layer includes kernel stride processing, and the method further includes implementing efficient kernel stride processing based on the transformed input tensor and a modified weight matrix.

[0111] Example 15: The method according to any one of Examples 1 to 14, wherein the U×V×W output tensor represents a U-V first output, and the U'×V'×W' output tensor represents a U'V' super-output, each super-output including a plurality of first outputs, and the number of first outputs among the plurality of first outputs depends on a dimension of the convolutional neural network layer weight matrix.

[0112] Example 16: The method of any one of Examples 1 to 15, wherein the U×V×W output tensor represents a U-V first output, and the U'×V'×W' output tensor represents a U'V' super-output, each super-output including a plurality of first outputs, and a number of outputs among the plurality of first outputs depends on one or more of: (i) an architecture of the convolutional neural network layer; (ii) an architecture of a neural network model including the convolutional neural network layer; or (iii) an apparatus for implementing the convolutional neural network layer.

[0113] Example 17: The method of any one of Examples 1 to 16, wherein the modification of the one or more convolutional neural network layer weight matrices is a software-only modification.

[0114] Example 18: A system comprising one or more computers and one or more storage devices, the one or more storage devices storing operable instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for more efficiently performing computations of a convolutional neural network layer configured to receive an X×Y×Z input tensor and generate a U×V×W output tensor by processing the received input tensor with one or more convolutional neural network layer weight matrices, the operations including converting the X×Y×Z input tensor to an X′×Y′×Z′ input tensor, where X′ is less than or equal to X, and (ii) the U'xV'xW' output tensor comprises a transformed U'xV'xW output tensor, wherein U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W, and the method comprises: processing the X'xY'xZ' input tensor with the modified weight matrix to generate the U'xV'xW' output tensor, wherein the U'xV'xW' output tensor comprises the U'xVxW output tensor.

[0115] Example 19: X' is equal to the ceiling of X divided by a natural number N, and Y' is a natural number is equal to the ceiling of Y divided by M, and Z' is equal to Z multiplied by NM, as shown in Example 18. The system is as follows:

[0116] Example 20: The system of Example 19, wherein the X×Y×Z input tensor represents an XY first input, and the X′×Y′×Z′ input tensor represents an X′Y′ super input, each super input including a plurality of first inputs.

[0117] Example 21: The system of example 20, wherein the number of the plurality of first inputs is equal to NM inputs. Example 22: The system of any one of Examples 18 to 21, wherein the action of the modified weight matrix on the X'×Y'×Z' input tensor is equivalent to the action of the convolutional neural network layer weight matrix on the X×Y×Z input tensor.

[0118] Example 23: The system of any one of Examples 18 to 22, wherein the modified weight matrix entries include one or more convolutional neural network layer weight matrices.

[0119] Example 24: A system described in any one of Examples 18 to 23, wherein the modified weight matrix includes a matrix having a band structure.

[0120] Example 25: U' is equal to the ceiling of U divided by the natural number P, and V' is a natural number The system of any one of Examples 18-24, wherein W' is equal to a ceiling of V divided by Q, and W' is equal to W multiplied by PQ.

[0121] Example 26: The system described in Example 25, wherein the convolutional neural network layer includes a stride S in the X dimension and a stride T in the Y dimension, and the relationship between the strides S, T, the output tensor U×V×W, and the transformed output tensor U'×V'×W' is given by U'=ceiling(U / NS) and V'=ceiling(V / MT).

[0122] Example 27: The system described in Example 25 or Example 26, wherein the U×V×W output tensor represents a UV first output, and the U'×V'×W' output tensor represents a U'V' super output, each super output including a plurality of first outputs.

[0123] Example 28: The system of example 27, wherein the plurality of first outputs is equal to PQ outputs. Example 29: The system of example 18, wherein the convolutional neural network layer includes a pooling sublayer.

[0124] Example 30: The system of Example 29, wherein the operations further include replacing the pooling sublayer with strided processing on the transformed input tensor.

[0125] Example 31: The system of any one of Examples 18 to 30, wherein the convolutional neural network layer includes kernel stride processing, and the method further includes implementing efficient kernel stride processing based on the transformed input tensor and a modified weight matrix.

[0126] Example 32: The system of any one of Examples 18 to 31, wherein the U×V×W output tensor represents U-V first outputs, and the U'×V'×W' output tensor represents a U'V' super output, each super output including a plurality of first outputs, and the number of first outputs among the plurality of first outputs depends on a dimension of the convolutional neural network layer weight matrix.

[0127] Example 33: The U×V×W output tensor represents a U-V first output, and the U'×V'×W' output tensor represents a U'V' super-output, each super-output including a plurality of first outputs, and the number of outputs among the plurality of first outputs is determined by (i) the number of the first outputs of the convolutional neural network layer. The system of any one of Examples 18 to 32, wherein the system relies on one or more of: (i) an architecture of a neural network model including the convolutional neural network layer; (ii) an architecture of a neural network model including the convolutional neural network layer; or (iii) an apparatus for implementing the convolutional neural network layer.

[0128] Example 34: The system of any one of Examples 18 to 33, wherein the modification of the one or more convolutional neural network layer weight matrices is a software-only modification.

[0129] Example 35: A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for more efficiently performing computations of a convolutional neural network layer configured to receive an X×Y×Z input tensor and generate a U×V×W output tensor by processing the received input tensor with one or more convolutional neural network layer weight matrices, the operations including converting the X×Y×Z input tensor to an X′×Y′×Z′ input tensor, where X′ is less than or equal to X, Y′ is less than or equal to Y, and Z′ is greater than or equal to Z, and obtaining a modified convolutional neural network layer weight matrix, where (i) the modified weight matrix operates on the X'×Y'×Z' input tensor to generate a U'×V'×W' output tensor, and (ii) the U'×V'×W' output tensor comprises a transformed U×V×W output tensor, where U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W; and generating the U'×V'×W' output tensor by processing the X'×Y'×Z' input tensor with the modified weight matrix, where the U'×V'×W' output tensor comprises the U×V×W output tensor.

[0130] Specific embodiments of the subject matter are described above. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order or sequential order depicted to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. 1. A computer-implemented method for more efficiently performing a convolutional neural network layer computation, the convolutional neural network layer configured to receive an X×Y×Z input tensor and generate a U×V×W output tensor by processing the received input tensor with one or more convolutional neural network layer weight matrices, the method comprising: converting the X×Y×Z input tensor to an X′×Y′×Z′ input tensor, where X′ is less than or equal to X, Y′ is less than or equal to Y, and Z′ is greater than or equal to Z; obtaining one or more modified convolutional neural network layer weight matrices, (i) the modified weight matrices operating on the X'xY'xZ' input tensor to generate a U'xV'xW' output tensor, (ii) the U'xV'xW' output tensor comprises a transformed UxVxW output tensor, where U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W; generating the U'xV'xW' output tensor by processing the X'xY'xZ' input tensor with the modified weight matrix, wherein the U'xV'xW' output tensor comprises the UxVxW output tensor.

2. X' is equal to the ceiling of X divided by a natural number N, and Y' is equal to the ceiling of X divided by a natural number M.

2. The method of claim 1, wherein Z′ is equal to a ceiling of Y, and Z′ is equal to Z multiplied by NM.

3. 3. The method of claim 2, wherein the XxYxZ input tensor represents an XY first input and the X'xY'xZ' input tensor represents an X'Y' super input, each super input including multiple first inputs.

4. The method of claim 3 , wherein a number of the plurality of first inputs is equal to NM inputs.

5. 2. The method of claim 1 , wherein the operation of the modified weight matrix on the X′×Y′×Z′ input tensor is equivalent to the operation of the convolutional neural network layer weight matrix on the X×Y×Z input tensor.

6. 2. The method of claim 1 , wherein the modified weight matrix entries include one or more convolutional neural network layer weight matrices.

7. The method of claim 1 , wherein the modified weight matrix comprises a matrix having a banded structure.

8. U' is equal to the ceiling of U divided by the natural number P, and V' is equal to the ceiling of U divided by the natural number Q.

2. The method of claim 1, wherein W′ is equal to a ceiling of V obtained by multiplying PQ by W′.

9. 9. The method of claim 8, wherein the convolutional neural network layer includes a stride S in the X dimension and a stride T in the Y dimension, and the relationship between the strides S, T, the output tensor U×V×W, and the transformed output tensor U′×V′×W′ is given by U′=ceiling(U / NS) and V′=ceiling(V / MT).

10. 9. The method of claim 8, wherein the UxVxW output tensor represents a UV primary output, and the U'xV'xW' output tensor represents a U'V' super-output, each super-output including a plurality of primary outputs.

11. The method of claim 10 , wherein the plurality of first outputs is equal to PQ outputs.

12. The method of claim 1 , wherein the convolutional neural network layer includes a pooling sublayer.

13. The method of claim 12 , further comprising replacing the pooling sublayer with strided operations on the transformed input tensors.

14. 2. The method of claim 1 , wherein the convolutional neural network layer includes kernel striding, the method further comprising: implementing efficient kernel striding based on the transformed input tensor and a modified weight matrix.

15. the U×V×W output tensor represents a U-V primary output, and the U′×V′×W′ output tensor represents a U′V′ super-output, each super-output including a plurality of primary outputs; 2. The method of claim 1 , wherein a number of first outputs in the plurality of first outputs depends on a dimension of the convolutional neural network layer weight matrix.

16. the U×V×W output tensor represents a U-V primary output, and the U′×V′×W′ output tensor represents a U′V′ super-output, each super-output including a plurality of primary outputs; 2. The method of claim 1 , wherein a number of outputs in the first plurality of outputs depends on one or more of: (i) an architecture of the convolutional neural network layer; (ii) an architecture of a neural network model that includes the convolutional neural network layer; or (iii) an apparatus that implements the convolutional neural network layer.

17. 2. The method of claim 1, wherein the modification of the one or more convolutional neural network layer weight matrices is a software-only modification.

18. 1. A system comprising one or more computers and one or more storage devices, the one or more storage devices storing operable instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for more efficiently performing computations of a convolutional neural network layer configured to receive an X×Y×Z input tensor and generate a U×V×W output tensor by processing the received input tensor with one or more convolutional neural network layer weight matrices, the operations comprising: converting the X×Y×Z input tensor to an X′×Y′×Z′ input tensor, where X′ is less than or equal to X, Y′ is less than or equal to Y, and Z′ is greater than or equal to Z; obtaining one or more modified convolutional neural network layer weight matrices, (i) the modified weight matrices operating on the X'xY'xZ' input tensor to generate a U'xV'xW' output tensor, (ii) the U'xV'xW' output tensor comprises a transformed UxVxW output tensor, where U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W; generating the U'xV'xW' output tensor by processing the X'xY'xZ' input tensor with the modified weight matrix, wherein the U'xV'xW' output tensor comprises the UxVxW output tensor.

19. X' is equal to the ceiling of X divided by a natural number N, and Y' is equal to the ceiling of X divided by a natural number M.

20. The system of claim 18, wherein Z′ is equal to a ceiling of Y, and Z′ is equal to Z multiplied by NM.

20. 20. The system of claim 19, wherein the XxYxZ input tensor represents an XY first input and the X'xY'xZ' input tensor represents an X'Y' super input, each super input including multiple first inputs.

21. 21. The system of claim 20, wherein a number of the plurality of first inputs is equal to NM inputs.

22. 20. The system of claim 18, wherein an action of the modified weight matrix on the X'xY'xZ' input tensor is equivalent to an action of the convolutional neural network layer weight matrix on the XxYxZ input tensor.

23. 20. The system of claim 18, wherein the modified weight matrix entries include one or more convolutional neural network layer weight matrices.

24. The system of claim 18 , wherein the modified weight matrix comprises a matrix having a banded structure.

25. U' is equal to the ceiling of U divided by the natural number P, and V' is equal to the ceiling of U divided by the natural number Q.

20. The system of claim 18, wherein W′ is equal to a ceiling of V obtained by multiplying PQ by W.

26. 26. The system of claim 25, wherein the convolutional neural network layer includes a stride S in the X dimension and a stride T in the Y dimension, and the relationship between the strides S, T, the output tensor U×V×W, and the transformed output tensor U′×V′×W′ is given by U′=ceiling(U / NS) and V′=ceiling(V / MT).

27. 26. The system of claim 25, wherein the UxVxW output tensor represents a UV primary output and the U'xV'xW' output tensor represents a U'V' super output, each super output including a plurality of primary outputs.

28. 28. The system of claim 27, wherein the plurality of first outputs is equal to PQ outputs.

29. 20. The system of claim 18, wherein the convolutional neural network layer includes a pooling sublayer.

30. 30. The system of claim 29, wherein the operations further comprise replacing the pooling sublayer with strided operations on the transformed input tensor.

31. 20. The system of claim 18, wherein the convolutional neural network layer includes kernel striding, the method further comprising: implementing efficient kernel striding based on the transformed input tensor and a modified weight matrix.

32. the U×V×W output tensor represents a U-V primary output, and the U′×V′×W′ output tensor represents a U′V′ super-output, each super-output including a plurality of primary outputs; 20. The system of claim 18, wherein a number of first outputs in the plurality of first outputs depends on a dimension of the convolutional neural network layer weight matrix.

33. the U×V×W output tensor represents a U-V primary output, and the U′×V′×W′ output tensor represents a U′V′ super-output, each super-output including a plurality of primary outputs; The number of outputs in the first plurality of outputs is determined based on (i) the architecture of the convolutional neural network layer, and (ii) the architecture of the neural network including the convolutional neural network layer.

20. The system of claim 18, wherein the method further comprises: (i) determining whether the convolutional neural network layers are convoluted; (ii) determining whether the convolutional neural network layers are convoluted; or (iii) determining whether the convolutional neural network layers are convoluted.

34. 20. The system of claim 18, wherein the modification of the one or more convolutional neural network layer weight matrices is a software-only modification.

35. 1. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for more efficiently performing computations of a convolutional neural network layer configured to receive an XxYxZ input tensor and generate a UxVxW output tensor by processing the received input tensor with one or more convolutional neural network layer weight matrices, the operations comprising: converting the X×Y×Z input tensor to an X′×Y′×Z′ input tensor, where X′ is less than or equal to X, Y′ is less than or equal to Y, and Z′ is greater than or equal to Z; obtaining one or more modified convolutional neural network layer weight matrices, (i) the modified weight matrices operating on the X'xY'xZ' input tensor to generate a U'xV'xW' output tensor, (ii) the U'xV'xW' output tensor comprises a transformed UxVxW output tensor, where U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W; generating the U'xV'xW' output tensor by processing the X'xY'xZ' input tensor with the modified weight matrix, wherein the U'xV'xW' output tensor comprises the UxVxW output tensor.

Images