Superpixel method for convolutional neural networks

Abstract

To provide a method, system and device for efficiently executing a calculation of convolutional neural network layer.SOLUTION: A method includes a step of converting an X×Y×Z input tensor into an X'×Y'×Z' input tensor, X' is X or less, Y' is Y or less, and Z' is Z or more. Also, the method includes a step of acquiring a corrected weight matrix, the corrected weight matrix generates a U'×V'×W' output tensor by acting on the X'×Y'×Z' input tensor, the U'×V'×W' output tensor includes a converted U×V×W output tensor, U' is U or less, V' is V or less, and W' is W or more. Further, the method includes a step of generating a U'×V'×W' output tensor by processing the X'×Y'×Z' input tensor by using the corrected weight matrix.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] Background This specification relates to generating an output by processing an input through layers of a neural network.

Background Art

[0002] A neural network is a machine learning model that uses one or more layers of non-linear units to predict an output for a given input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to another layer in the network, such as the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values of each set of parameters.

Summary of the Invention

Means for Solving the Problems

[0003] Summary In general, one innovative aspect of the subject matter described herein can be realized in a method for more efficiently performing computations on convolutional neural network layers. This convolutional neural network layer is configured to receive an X×Y×Z input tensor and generate a U×V×W output tensor by processing the received input tensor using one or more convolutional neural network layer weight matrices. In some respects, this method comprises the step of transforming an X×Y×Z input tensor into an X'×Y'×Z' input tensor, where X' is less than or equal to X, Y' is less than or equal to Y, and Z' is greater than or equal to Z. This method further comprises the step of obtaining one or more modified convolutional neural network layer weight matrices, where (i) the modified weight matrices act on the X'×Y'×Z' input tensor to produce a U'×V'×W' output tensor, and (ii) the U'×V'×W' output tensor comprises the transformed U×V×W output tensor, where U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W. This method further comprises the step of generating a U'×V'×W' output tensor by processing the X'×Y'×Z' input tensor with the modified weight matrices, where the U'×V'×W' output tensor comprises the U×V×W output tensor.

[0004] Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded in one or more computer storage devices, each configured to perform the actions of the method described above. A system consisting of one or more computers can be configured to perform a specific operation or action by software, firmware, hardware, or any combination thereof, and such software, firmware, hardware, or any combination thereof can be installed in the system and cause the system to perform the action when it is running. One or more computer programs can be configured to perform a specific operation or action by including instructions that, when executed by a data processing device, cause the device to perform the action.

[0005] Each of the above and other embodiments may optionally include one or more of the following features, either individually or in combination:

[0006] In some implementations, X' is equivalent to the ceiling of X divided by a natural number N. Furthermore, Y' is equal to the ceiling of Y divided by a natural number M, and Z' is Z multiplied by NM. equal.

[0007] In some implementations, the X×Y×Z input tensor represents the XY first input, the X'×Y'×Z' input tensor represents the X'Y' superinput, and each superinput contains multiple first inputs.

[0008] In some implementations, the number of first inputs is equal to NM inputs. In some implementations, the effect of a modified weight matrix on an X'×Y'×Z' input tensor is equivalent to the effect of a convolutional neural network layer weight matrix on an X×Y×Z input tensor.

[0009] In some implementations, the entries in the modified weight matrix include one or more convolutional neural network layer weight matrices.

[0010] In some cases, the modified weight matrix includes a matrix with a band structure.

[0011] In some implementations, U' is equivalent to the ceiling of U divided by a natural number P. Therefore, V' is equal to the ceiling of V divided by a natural number Q, and W' is equal to W multiplied by PQ.

[0012] In some implementations, the U×V×W output tensor represents the UV first output, the U'×V'×W' output tensor represents the U'V' superoutput, and each superoutput contains multiple first outputs.

[0013] In some cases, multiple first outputs are equal to PQ outputs. In some implementations, the convolutional neural network layers include pooling sublayers.

[0014] In some cases, the method further includes the step of replacing the pooling sublayer with striding on the transformed input tensor.

[0015] In some implementations, a convolutional neural network layer includes an X-dimensional stride S and a Y-dimensional stride T, and the relationship between the strides S and T, the output tensor U×V×W, and the transformed output tensor U'×V'×W' is denoted by U'=ceiling(U / NS) and V'=ceiling(V / MT).

[0016] In some cases, the U×V×W output tensor represents the UV first output, and the U'×V'×W' output tensor represents the U'V' superoutput, where each superoutput contains multiple first outputs, and the number of first outputs among them depends on the dimension of the convolutional neural network layer weight matrix.

[0017] In some implementations, the U×V×W output tensor represents the UV first output, the U'×V'×W' output tensor represents the U'V' superoutput, and each superoutput contains multiple first outputs, the number of outputs among the multiple first outputs depending on one or more of the following: (i) the architecture of the convolutional neural network layer, (ii) the architecture of the neural network model containing the convolutional neural network layer, or (iii) the device that implements the convolutional neural network layer.

[0018] In some cases, one or more convolutional neural network layer weight matrices The fix is ​​a software-only modification.

[0019] The subject matter described herein can be implemented in particular embodiments to achieve one or more of the following advantages:

[0020] Convolutional neural network layers can be computed using matrix multiplication. However, both software libraries and hardware matrix multiplication units may function inefficiently when the input or output depth of the convolutional neural network layer is shallow. This is because the input and output depths are mapped to the dimensions of one of the matrices being multiplied. For example, an input image may have RGB data with an input depth of 3, and the convolutional neural network layer may produce an output with a depth of 16. Here, 3 and 16 are shallow in the sense described above. The reason for this inefficiency is that typical modern computer architectures, due to their cache, memory access, and computing unit architecture design, can process extremely deep layers up to a certain depth limit with the same or similar efficiency as shallow layers.

[0021] The standard operation in a convolutional neural network layer is to map a single input, such as a single pixel, to a single row or column of a matrix multiplication operation. A superpixel convolutional neural network layer, as described herein, groups multiple inputs corresponding to data, such as multiple input pixels, prioritizing depth magnitude or metric (e.g., Z dimension) over spatial magnitude or metric (e.g., X and Y dimensions). By more efficiently realizing and processing the grouped inputs using matrix multiplication units, the computational efficiency of the system implementing the superpixel convolutional neural network layer technique is increased. For example, because the in-memory representation of the neural network layer in a superpixel CNN configured to process grouped inputs is more compact, the neural network layer can store and process the received input in a limited-size, low-latency access memory. Systems implementing such techniques can therefore benefit from the higher computational throughput, lower computational latency, shorter input / output transfer times, and better energy utilization of convolutional neural networks on any matrix machine, such as a central processing unit, graphics processing unit, or dedicated neural network processing unit. In addition, the system can implement the above techniques without modifying the system hardware; that is, the techniques may be implemented entirely using system software and can therefore be applied to all machines used to perform matrix multiplication.

[0022] As described herein, a superpixel convolutional neural network layer can have fewer kernel elements compared to a standard convolutional neural network layer, and therefore the number of matrix multiplication operations that need to be performed when using a superpixel convolutional neural network layer can be reduced.

[0023] In addition, a system for implementing a superpixel convolutional neural network layer can convert a convolutional neural network layer with a stride greater than 1 into a superpixel convolutional neural network layer with a stride equal to 1, as described herein. A superpixel convolutional neural network layer with a stride equal to 1 may be necessary for computer architectures that can only directly perform convolutional calculations with a stride equal to 1, such as some computer architectures for implementing a neural network system, for example, an architecture including a matrix multiplication unit such as a systolic array. In addition, in some cases, a superpixel convolutional neural network layer with a stride equal to 1 can be more efficiently computed on the above architecture. This is because, for example, off-chip processing or other measures required to perform convolutional calculations with a stride greater than 1 may become unnecessary. Therefore, the computational efficiency and convenience of a system for implementing a superpixel convolutional neural network layer are improved. This is because off-chip processing or other measures required to perform convolutional calculations with a stride greater than 1 may become unnecessary. Therefore, the computational efficiency and convenience of a system for implementing a superpixel convolutional neural network layer are improved.

[0024] The subject matter described herein also relates to an image recognition method / system using a superpixel convolutional neural network layer.

[0025] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from this description, the drawings, and the claims.

Brief Description of the Drawings

[0026] [Figure 1A] An example of a superpixel convolutional neural network system. [Figure 1B] A diagram showing the relationship between a convolutional neural network layer and a superpixel convolutional neural network layer. [Figure 2]This is a flowchart illustrating an example of a process for efficiently performing computations on convolutional neural network layers. [Figure 3] This is an example of a diagram illustrating the superpixel input tensor transform. [Figure 4] This is an example of a diagram illustrating the modification of the layer weight matrix in a convolutional neural network. [Figure 5] This is an example of a diagram illustrating the weight matrix modification of a convolutional neural network layer, including superpixel kernel stride processing. [Modes for carrying out the invention]

[0027] Similar reference numbers and names in various drawings indicate the same components. Detailed explanation Figure 1A shows an example of a superpixel convolutional neural network system 100. The superpixel convolutional neural network system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, capable of realizing the following systems, components, and technologies.

[0028] The superpixel convolutional neural network system 100 is a machine learning system that receives a system input 102 and generates a system output 104 from this system input 102.

[0029] The superpixel convolutional neural network system 100 can be configured to receive digital data input and generate a score or classification output based on this input. For example, if the input to the superpixel convolutional neural network system 100 is an image or features extracted from an image, the output that the superpixel convolutional neural network system 100 generates for a given image using one or more fully connected or convolutional neural network layers included in the neural network system 100 may be a score for each of a set of object categories. Each score may represent the estimated likelihood that the image contains images of objects belonging to that category.

[0030] In particular, the superpixel convolutional neural network system 100 realizes a convolutional neural network that includes multiple convolutional neural network layers, including convolutional neural network layer A 106, convolutional neural network layer B 108, and superpixel convolutional neural network layer 110. Each convolutional neural network layer has neurons or nodes that are arranged in a three-dimensional array, i.e., width A convolutional neural network is a neural network layer logically arranged in terms of height and depth. Neurons within a convolutional neural network layer are connected to small regions of layers preceding it, called receptive fields or kernel patches, and each neuron within the convolutional neural network layer detects identical features that constitute the property of translational invariance. A convolutional neural network layer can be associated with a weight matrix or kernel that stores the current values ​​of the layer's parameters. The layer generates an output from its input according to the current values ​​of the neural network layer's parameters, for example, by performing multiple matrix multiplication operations. An example of a convolutional neural network layer is described in more detail in "Large-scale Video Classification with Convolutional Neural Networks" (Proceedings of International Computer Vision and Pattern Recognition (CVPR 2014), IEEE) by Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei.

[0031] Each layer of a superpixel convolutional neural network is configured to receive its own layer input and process this input to generate its own layer output. For example, a convolutional neural network layer in the superpixel convolutional neural network system 100 can be configured to receive an X×Y×Z input tensor and process this input tensor using one or more convolutional neural network layer weight matrices or kernels to generate a U×V×W output tensor. A tensor is a multidimensional array consisting of numbers or other values, such as strings, having a specific order corresponding to the dimensionality of the array. For example, scalar values ​​are zero-order tensors, numerical vectors are first-order tensors, and matrices are second-order tensors.

[0032] For example, a convolutional neural network layer can be configured to receive a 64x64x3 input tensor representing an image containing 64x64 pixels with a feature depth of 3, where the feature depth is represented by red, green, and blue feature channels. The convolutional neural network layer can then generate its respective output tensor by processing the received 64x64x3 input tensor using one or more respective convolutional neural network layer matrices.

[0033] In some implementations, a convolutional neural network layer may include one or more hyperparameters that control the size and output volume of the convolutional neural network layer. For example, a convolutional neural network layer may include a kernel stride that controls the amount by which the convolutional filter or weight matrix shifts in X and Y between evaluations of the filter on the input data. In some implementations, a convolutional neural network layer may include or be connected to a pooling layer, which is a form of nonlinear downsampling. For example, a convolutional neural network layer may include a pooling stride that acts on the depth slice of the input tensor to spatially resize the input tensor. Continuing with the example above, the convolutional neural network layer can thus produce a U×V×W output, where U is less than or equal to X, V is less than or equal to Y, and W is greater than or equal to Z, and the output tensor is, for example, a 32×32×32 output tensor (corresponding to, for example, a 7×7 kernel stride and a 2×2 pooling stride). Generally, kernel stride processing can be considered as skipping the initial computation (for example, "skipping X twice and Y twice between evaluations"), while pooling is a post-pass after evaluation with a stride of 1 (for example, in pooling, the computed value is generated with a stride of 1). The maximum value for each group in the pooling window is selected (for example, a 2x2 portion of the input data).

[0034] At least one of the superpixel neural network systems 100 The convolutional neural network layer is a superpixel convolutional neural network layer, for example, superpixel convolutional neural network layer 110. A superpixel convolutional neural network layer is a convolutional neural network layer that applies a superpixel transformation (also called a modification) to the received convolutional layer input and convolutional neural network layer weight matrix in order to generate a superpixel convolutional neural network layer that is mathematically equivalent but has a different shape.

[0035] The superpixel convolutional neural network layer 110 can receive inputs, such as input 112, from convolutional neural layers preceding it. For example, the received input may include outputs generated by convolutional neural network layers preceding the superpixel convolutional neural network layer 110.

[0036] The superpixel convolutional neural network layer 110 may include a superpixel generator 114. The superpixel generator 114 is configured to receive a convolutional neural network input and convert this input into a superpixel layer input. The superpixel generator 114 groups the components of the received input, prioritizing the depth or index, such as the Z dimension, over the spatial size or index, such as the X and Y dimensions. As a result, the depth of the convolutional neural network layer in the Z layer is no longer shallow, thus avoiding the inefficiency caused by a shallow Z layer.

[0037] For example, a superpixel convolutional neural network layer 110 can receive an X×Y×Z input tensor as input, for example, representing XY (X multiplied by Y) with depth Z. The superpixel generator 114 is configured to transform this X×Y×Z input tensor into an X'×Y'×Z' superpixel input tensor, where X' is less than or equal to X, Y' is less than or equal to Y, and Z' is greater than or equal to Z. An example of the transformation of the superpixel input tensor performed by the superpixel generator is shown in Figure 3 below.

[0038] The superpixel convolutional neural network layer 110 includes a superpixel weight matrix 116. The superpixel weight matrix 116 is the modified (or, alternatively, transformed) convolutional neural network layer weight matrix corresponding to the convolutional neural network layer. For example, the convolutional neural network layer weight matrix may be modified depending on the selection of each convolutional neural network layer that transforms into the superpixel convolutional neural network layer. The superpixel weight matrix 116 acts on the transformed convolutional neural network layer input.

[0039] For example, a convolutional neural network layer corresponding to a superpixel neural network layer 110 can be configured to receive an X×Y×Z input tensor. The superpixel neural network layer 110 can transform the received input into an X'×Y'×Z' superpixel input tensor. In this case, the superpixel weight matrix may include a modified weight matrix that, by acting on the X'×Y'×Z' input tensor, generates a U'×V'×W' output tensor. X' is not necessarily equal to U', Y' is not necessarily equal to V', and Z' is not necessarily equal to W'. The U'×V'×W' output tensor is the transformed U×V×W output tensor, where U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W. For example, the U'×V'×W' output tensor can represent a group of components of a superpixel output tensor, such as a U×V×W output tensor. In some implementations, using grouping to create or represent a superpixel input tensor may not be equivalent, for example, the number of components and the grouping of components. The method of modification can vary. As an example, the modification of the layer weight matrix of a convolutional neural network is explained below with reference to Figure 4. In addition, an example of an algorithm (including comments) for generating a superpixel weight matrix (with or without stride processing, e.g., stride_x=stride_y=1) is as follows: ru.

[0040]

number

number

[0041] The SuperIndex routines used in the above description can be defined as follows, but are not limited to these examples.

[0042]

number

[0043] The examples shown above are for illustrative purposes only. For instance, in some implementations, the SuperIndex routine may not depend on the packing order within the superpixel.

[0044] The superpixel convolutional neural network layer 110 processes the transformed input tensor using a modified weight matrix to produce a transformed convolutional neural network layer output, such as a superpixel output 118. This produced, transformed convolutional neural network layer output may be mathematically equivalent to the neural network output produced by processing the untransformed convolutional neural network input using the untransformed convolutional neural network layer weight matrix. All output values ​​of the untransformed computation may exist in the produced transformed convolutional neural network layer output in a form that can be computed by performing an inverse homomorphic transformation, as described below with reference to Figure 1B. For example, the produced transformed convolutional neural network layer output may include a superset of outputs that includes a neural network output corresponding to the neural network output produced by processing the (untransformed) convolutional neural network using the (untransformed) convolutional neural network layer. In some implementations, the additional outputs included in this superset can be provided to subsequent layers of the superpixel neural network system 100. For example, the additional output may correspond to a "partial sum" of the "off-edge" outputs of the initially requested unconverted output. If a superpixel computation processes one "tile" of larger computations with larger X and Y dimensions than is efficient for processing one superpixel computation, the partial sum can be saved and reused for higher efficiency. Processing inputs using superpixel convolutional neural network layers will be explained in more detail below with reference to Figure 2.

[0045] In some implementations, a superpixel convolutional neural network layer 110 may include a superpixel unformatter configured to receive superpixel outputs and convert these received superpixel outputs into outputs in normal order. For example, if the current superpixel convolutional neural network layer generates 4x4 superpixel outputs, and the next superpixel convolutional neural network layer is configured to process 8x8 superpixels, the system can convert the 4x4 superpixels to 8x8 superpixels by using the superpixel unformatter. In some implementations, the superpixel unformatter for a superpixel convolutional neural network layer, e.g., layer 110, can perform ungrouping of the outputs generated by a superpixel weight matrix, e.g., weight matrix 116. In other implementations, the superpixel unformatter for a superpixel convolutional neural network layer may include a superpixel unformatter for a series of superpixel convolutional neural networks It can be included as part of, or together with, a superpixel generator for the next superpixel convolutional neural network layer in the twerk layer.

[0046] The superpixel convolutional neural network layer 110 may be included at various locations within a series of neural network layers. In some implementations, multiple superpixel convolutional neural network layers may be included within a series of neural network layers. For example, in some implementations, one or more of the convolutional neural network layers A and B may be superpixel convolutional neural network layers. In implementations where multiple superpixel convolutional neural network layers are included within a series of neural network layers, each superpixel convolutional neural network layer can generate its own superinput by applying its respective transformation to the input it receives, and each generated superinput may contain superinputs of different sizes.

[0047] The following table shows an example of a system layout consisting of multiple superpixel convolutional neural network layers.

[0048] [Table 1]

[0049] In the example layout above, most neural network layers contain 8x8 sets of superpixels per layer. The number of pixels per superpixel is adjusted according to the kernel stride processing. The "Original Input Depth (Z)" column shows the input depth of the original neural network model, and the "Transformed Input Depth (U)" column shows the input depth after transforming the input and weights using the specified number of pixels per superpixel.

[0050] The following table shows a second example of a system layout consisting of multiple superpixel convolutional neural network layers.

[0051] [Table 2]

[0052] The superpixel convolutional neural network system 100 can be trained in multiple batches of training examples to obtain the post-training values ​​of the neural network layer parameters, that is, to adjust the parameter values ​​from their initial values ​​to their post-training values. For example, during training, the superpixel convolutional neural network system 100 can process batches of training examples and generate a separate neural network for each training example in this batch. The neural network output can then be used to adjust the parameter values ​​of the components of the superpixel convolutional neural network system 100, for example, through conventional gradient descent and backpropagation neural network training techniques. The modified weight matrix makes NM copies of each of the original weights during the backpropagation training process (where N and M represent the superpixel size, which will be explained below with reference to step 202 in Figure 2), so the system can obtain the net change in value of each of the original weights by adding the weight gradient across all NM copies. Training a superpixel neural network layer is an inverse process in which spatial size or metric may be preferred over depth size or metric.

[0053] Once the neural network training is complete, the superpixel convolutional neural network system 100 can receive a new neural network input to be processed and generate a new neural network output for input by processing this neural network input through the neural network layer according to the trained values ​​of the parameters of the components of the superpixel convolutional neural network system 100.

[0054] In some implementations, the superpixel neural network system 100 may include one or more deconvolutional neural network layers. In these cases, the same descent can be obtained by reversing the above components and techniques; that is, the superpixel deconvolutional neural network layer can group multiple inputs, prioritizing spatial size or metric over depth size or metric, and modify the respective deconvolutional neural network layer weight matrix accordingly.

[0055] Figure 1B shows the convolutional neural network layer and the superpixel convolutional neural Figure 160 shows the relationship with the network layer.

[0056] As shown in Figure 160, a convolutional neural network layer can be configured to generate an activation O 164 as a layer output by performing a convolution operation 162 on the input activation I and weight matrix W.

[0057] Similarly, a superpixel convolutional neural network layer, such as the modified or transformed convolutional neural network layer described earlier with reference to Figure 1A, can be configured to produce an activation 170 as a transformed layer output by performing a convolution operation "convolve" 168 on the transformed input activation Transform(I) and the transformed weight matrix Transform(W).

[0058] Figure 160 shows two homomorphic arrows connecting input I to the transformed input Transform(I) and weight matrix W to the transformed weight matrix Transform(W). These homomorphic arrows represent the homomorphism between the original convolutional neural network layer and the modified convolutional neural network layer. These arrows map, for example, the original model activations and weights to the modified or transformed model activations and weights. Similarly, Figure 160 shows an inverse homomorphic arrow connecting the transformed output Transform(O) to output O. The inverse homomorphism represents the inverse homomorphism between the original convolutional neural network layer and the modified convolutional neural network layer. This arrow, for example, maps the transformed output to the original output.

[0059] An example of an algorithm for transforming the original model activation is as follows:

[0060]

number

[0061] An example of an inverse homomorphism algorithm is as follows:

[0062]

number

[0063] As shown in Figure 160, the number of operations required to generate the output of a convolutional neural network layer using a superpixel convolutional neural network layer may be greater than the number of operations required by the corresponding original neural network layer. This is because the input I and weight matrix W may need to be transformed before applying the convolution. However, this superpixel convolutional neural network layer architecture, which performs more operations, will likely be faster in runtime than the corresponding original neural network layer, which performs fewer operations.

[0064] Figure 2 is a flowchart of an example process 200, which is a process for efficiently performing computations on a convolutional neural network layer configured to receive an X×Y×ZZ input tensor and generate a U×V×W output tensor by processing this input tensor using one or more convolutional neural network layer weight matrices. For convenience, process 200 is described as being performed by a system consisting of one or more computers located in one or more locations. As an example, a superpixel convolutional neural network system appropriately programmed according to this specification, such as the superpixel convolutional neural network system 100 in Figure 1A, can perform process 200.

[0065] This system transforms an X×Y×Z input tensor into an X'×Y'×Z' superpixel input tensor (step 202). This system transforms the input tensor by grouping multiple inputs, prioritizing depth or index over spatial size or index. For example, the X'×Y'×Z' input tensor may have different dimensions than the X×Y×Z input tensor. More specifically, X' may be less than or equal to X, Y' may be less than or equal to Y, and Z' may be greater than or equal to Z. In some implementations, X' is the ceiling of X divided by a natural number N, for example, the minimum integer greater than or equal to X divided by a natural number N. In this system, Y' is equal to the ceiling of Y divided by a natural number M, and Z' is equal to Z multiplied by NM.

[0066] An X×Y×Z input tensor can represent an XY first input. For example, an X×Y×Z input tensor can represent an X×Y pixel image with feature depth Z. Similarly, an X'×Y'×Z' input tensor can represent an X'Y' superinput, each superinput containing multiple first inputs. For example, an X'×Y'×Z' input tensor can represent an X'×Y' superpixel image with depth Z'. In some implementations, the number of multiple first inputs is equal to NM inputs. For example, each superinput may consist of NM first inputs. An example of input tensor transformations is shown in Figure 3 below.

[0067] This system obtains one or more modified convolutional neural network layer weight matrices (step 204). These modified weight matrices act on the X'×Y'×Z' input tensor to generate a U'×V'×W' output tensor. In some implementations, the action of the modified weight matrix on the X'×Y'×Z' input tensor is equivalent to the action of the convolutional neural network layer weight matrix on the X×Y×Z input tensor; for example, the action of the modified weight matrix on the X'×Y'×Z' input tensor produces the same effect. The U'×V'×W' output tensor may contain the transformed U×V×W output tensor, where U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W. For example, the U'×V'×W' output tensor can represent a group of multiple outputs from the U×V×W output tensor, in which case the depth magnitude or index takes precedence over the spatial magnitude or index, as mentioned earlier.

[0068] The entries in the modified weight matrix include one or more convolutional neural network layer weight matrices, and may include block matrices that can be interpreted as being divided into sections called blocks or submatrices. In some implementations, the entries in the modified weight matrix may represent a band structure. For example, a convolutional neural network layer can be represented by a fully connected matrix where each input is connected to each output. A fully connected matrix represents a band structure. In the case of a band structure, most of the entries in the matrix are zero, and a fraction of diagonal bands in the matrix have non-zero entries.

[0069] Generally, for one dimension, the number of modified kernel elements is: ceiling((superinput size + original convolutional neural network layer kernel width - 1) / super - Input size) (ceiling((superinput_size + original convolutional neutral network layer kernel width - 1) / superinput_size)) should be equal to this. Convolutional Neural The modification of the network layer weight matrix will be explained below with reference to Figure 4.

[0070] Optionally, this system can implement efficient kernel stride processing for convolutional neural network layers (step 206). For example, in some implementations, the convolutional neural network layers may include a pooling sublayer. In these cases, the system processes the pooling sublayer using the transformed input tensor X'×Y'×Z This can be replaced with a stride operation on '. For example, in some implementations, as will be explained below with reference to step 208, the U×V×W output tensor can represent the UV first output, and the U'×V'×W' output tensor can represent the U'V' super output, with each super output containing multiple first outputs. In such cases, the efficient kernel stride operation size achieved does not have to be equal to the number of outputs of the multiple first outputs. The effect of the kernel stride operation can be incorporated into the modified weight matrix. For example, an (untransformed) convolutional neural network layer can produce a layer output O by performing two operations: applying the convolution to each input I and each weight matrix W, and applying the stride operation to the convolved input I and weight matrix W. A transformed superpixel convolutional neural network layer with stride operation can produce a transformed layer output transform(O) by applying the convolution to each transformed input transform(I) and the strided and transformed weight matrix strided_transform(2,W).

[0071] In some implementations, the convolutional neural network layer may already include kernel stride processing. In these cases, the system may replace this kernel stride processing with a more efficient kernel stride processing method. The system can achieve efficient kernel stride processing by using different input and output superpixel sizes, where the ratio of superpixel sizes is equal to the stride. For example, if the convolutional neural network layer includes an X-dimensional stride S and a Y-dimensional stride T, and the system transforms the convolutional neural network layer into a superpixel convolutional neural network layer with a superpixel input size of NM by selecting output superpixel sizes (N / S)(M / T), then the system achieves kernel stride processing of the superpixel modification weight matrix. In some implementations, the relationship between the stride S, the output tensor U×V×W, and the transformed output tensor U'×V'×W' can be shown by U'=ceiling(U / NM) and V'=ceiling(V / MT). The implementation of efficient kernel stride processing will be explained below with reference to Figure 5.

[0072] This system processes the X'×Y'×Z' input tensor using a modified weight matrix to generate a transformed convolutional neural network layer output, for example, a U'×V'×W' output tensor (step 208). The U'×V'×W' output tensor contains the U×V×W output tensor, and for example, the U'×V'×W' output tensor may contain a superset of outputs that contain the U×V×W output tensor.

[0073] The U'×V'×W' output tensor may have different dimensions than the U×V×W output tensor. More specifically, U' may be equal to the ceiling of U divided by a natural number P, V' may be equal to the ceiling of V divided by a natural number Q, and W' may be equal to W multiplied by PQ. In some implementations, the natural numbers P and Q may be equal to the natural numbers N and M described earlier with reference to step 202, respectively. In addition, in some implementations, P and Q of the current superpixel convolutional neural network layer may be equal to N and M of the next superpixel convolutional neural network layer, for example, if the system does not include a superpixel generator and a superpixel unformatter between the current superpixel convolutional neural network layer and the next superpixel convolutional neural network layer.

[0074] The U×V×W output tensor can represent the first UV output of depth W. For example, the U×V×W output tensor can represent the feature map of the U×V output of depth W. Similarly, the U'×V'×W' output tensor can represent the U'V' super output, and each super —The output contains multiple first outputs. For example, a U'×V'×W' output tensor can represent an image of U'×V' pixels with a depth of W'. In some implementations, the number of first outputs is equal to P×Q output pixels; for example, each superoutput may consist of PQ first outputs.

[0075] The natural numbers P and Q may be different from the natural numbers N and M described earlier with reference to step 202. For example, the size of the superinput may not be the same as the size of the superoutput. In some implementations, the number of first outputs among multiple first outputs, for example, the size of the superoutput, depends on the dimension of the convolutional neural network layer weight matrix. In some implementations, the number of outputs among multiple first outputs depends on one or more of the following: (i) the architecture of the convolutional neural network layer, (ii) the architecture of the neural network model containing the convolutional neural network layer, or (iii) the device that implements the convolutional neural network layer.

[0076] As an example, assume that the input size of the first convolutional layer is X×Y×Z and the resulting output size is U×V×W, and that the input size of the superpixel transformation convolutional layer is X'×Y'×Z' and the resulting output size is U'×V'×W'. Let KL represent the number of convolution kernel elements in the weight matrix, and K'L' represent the number of convolution kernel elements in the modified weight matrix. Furthermore, assume that the input and output pixels / superpixels are identical and specified by N and M. Then U'=ceiling(U / N), V'=ceiling(V / M), and W'=W×N×M. In addition, K'=ceiling((N+K-1) / N) and L'=ceiling((M+L-1) / N). Cases in which PQ may not be equal to NM include cases where the system may be performing kernel stride processing, or where the number of pixels per transformed superpixel changes from input to output.

[0077] Figure 3 is an example illustration of a superpixel input tensor transform. For example, this example illustration shows the superpixel input tensor transform described earlier with reference to step 202 of Figure 2. The transform shown can be performed by a system consisting of one or more computers located in one or more locations. For example, a superpixel generator appropriately programmed according to this specification, such as the superpixel generator 114 in Figure 1A, can perform the transform illustrated.

[0078] This diagram illustrates an example X×Y×Z input tensor 302. As shown in Figure 3, the input tensor contains XY inputs, each with a depth of Z. As explained earlier with reference to Figure 2, the X×Y×Z input tensor can be transformed into an X'×Y'×Z' superpixel input tensor by grouping multiple inputs. During this grouping, the index or size in the depth dimension (Z dimension) takes precedence over the index or layout in the spatial dimensions (X and Y dimensions). In example diagram 300, the X×Y×Z input tensor is transformed into an X / 2×Y / 2×4Z superpixel input tensor 304. As shown in Figure 3, the superpixel input tensor contains (X / 2)(Y / 2) inputs, each with a depth of 4Z. Each superpixel input in input tensor 304 represents four of the original inputs, thus representing four times the amount of data represented by the original inputs.

[0079] Figure 4 is an example illustration of a convolutional neural network layer weight matrix modification. For example, this example illustration shows the convolutional neural network layer weight matrix modification described earlier with reference to step 204 in Figure 2. The modification shown can be performed by a system consisting of one or more computers located in one or more locations. As an example, a superpixel convolutional neural network appropriately programmed according to this specification... A neural network layer, such as the superpixel convolutional neural network layer 110 in Figure 1A, can perform the illustrated modifications.

[0080] In step (a), this example diagram 400 shows a 3x3 kernel patch 402 of a convolutional neural network layer, where each element of this kernel patch contains a 2D weight matrix, the dimension of which is equal to the original input depth × original output depth suitable for matrix multiplication, e.g., Z × W. This example diagram further shows 2x2 grids 404-410 representing the inputs. The inputs are grouped to form 2x2 superinputs, each superinput containing the original 2x2 inputs. This example diagram shows a superoutput 412 containing four original outputs.

[0081] In step (b), example diagram 400 shows the same two-dimensional convolution as shown in step (a), extended to the calculation of the dot product of each of the four original outputs. Shading behind the 2x2 grid, e.g., shading 414, shows which 3x3 subsets of the input data affect each output. As the shading for each grid shows, all superinputs affect all outputs, but the useful fraction for each superinput is 9 / 16.

[0082] In step (c), diagram example 400 shows an example of modified 2D weight matrices 416-422. These weight matrices can be used to calculate the super-output. Matrices 416-422 are kernel elements that make up a 2x2 modified kernel patch. The dimensions of matrices 416-422 are 4Z×4W, and therefore 16 times the size of the original 2D matrix of kernel patch 402. The vertical axis represents the input depth, the input in the super-input, and the kernel elements. The horizontal axis represents the output depth and the output in the super-output. This diagram example shows matrix multiplication in step (c). Column 424 represents the input, and if the input is shown at the same height as the entries in the modified weight matrices 416-422, it is multiplied by this weight. The sum of the outputs in the super-output is obtained by adding up all the products of the weights and inputs in one column. The 16 original inputs that contribute to the calculation of a total of four superinputs, for example, one superoutput, for example, four original outputs, are shown.

[0083] Each of the four superinputs has a different X and Y coordinate relationship with respect to the superoutput. For illustrative purposes, Figure 4 shows 2x2 pixels per superpixel and 2x2 sets of superpixels. Thus, 2x2=4 weight matrices are drawn. The number of pixels per superpixel and per set of superpixels can vary, for example, based on the system architecture, among other factors. For example, in some cases there may be 4x4 pixels per superpixel, and the image size may be large, consisting of 100x100 superpixels (and thus 400x400 original input pixels). In this case, as long as ceiling((pixels per superpixel + kernel size - 1) / pixels per superpixel)(ceiling((pixels_per_superpixel + kernel_size - 1) / pixels_per_superpixel)) is 2 or less in each dimension, there will still be 2x2=4 weight matrices.

[0084] As mentioned earlier, the modified weight matrix consists of kernel elements. In this regard, the modification process can be seen as reducing the number of kernel elements in the patch by changing the 3x3 convolution for the input to a 2x2 convolution for the superinput, with each kernel element being a larger two-dimensional matrix, e.g., four times larger in both rows and columns (corresponding to the input depth Z and output depth W). The stacked modified weight matrices 416-422 contain one copy of each element of the 3x3 kernel 402. In some implementation examples, the modified weight matrix is ​​shown, for example, by the diagonal, similarly labeled kernel elements, e.g., diagonal 426. , it may exhibit a band structure.

[0085] Figure 5 is an example illustration of a convolutional neural network layer weight matrix modification including superpixel kernel stride processing. For example, this example illustration shows an efficient implementation of the kernel stride processing described above, with reference to step 206 in Figure 2. The shown implementation example can be run by a system consisting of one or more computers located in one or more locations. As an example, a superpixel convolutional neural network layer appropriately programmed according to this specification, such as the superpixel convolutional neural network layer 110 in Figure 1A, can implement the illustrated implementation.

[0086] Example diagram 500 shows a weight diagram, along with a 5x5 kernel patch 506 whose elements are 2D weight matrices with dimensions equal to the original input depth × original output depth, for example, Z × W, a 4x4 superinput 502 each containing a 4x4 original input, and a 2x2 superoutput 504 each containing a 2x2 original output. An example kernel stride is 2x2. Example diagram 500 contains four superpixel kernel elements 508-514. Each superpixel kernel element is a 16Z × 4W matrix. Since the input / superinput vs. output / superoutput is varied by a 2x2 kernel stride, there are only 4 outputs for each 16 inputs.

[0087] The embodiments and functional operations of the subject matter described herein can be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or in combination of one or more of these. The embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-temporary program carrier for execution by or control of the operation of a data processing device. Alternatively or in addition thereto, the program instructions can be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information to be transmitted to a suitable receiving device for execution by the data processing device. Computer storage media may be machine-readable storage media, machine-readable storage substrates, random or serial access memory devices, or a combination of one or more of these. However, computer storage media are not propagating signals.

[0088] The term “data processing device” encompasses all types of devices, machines, and equipment used for data processing, including, for example, programmable processors, computers, or multiple processors or computers. Such devices may include dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, such devices may also include code that creates an execution environment for the computer program, such as processor firmware, protocol stacks, database management systems, operating systems, or code comprising one or more of these.

[0089] Computer programs (which may be called, or described as, programs, software, software applications, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may also be written as standalone programs or modules, including components, subroutines, or other units suitable for use in a computing environment. It can be deployed in a specific state. A computer program may or may not correspond to files in the file system. A program can be stored in part of a file that holds one or more scripts stored in a markup language document that holds other programs or data, or in a single file dedicated to the program, or in multiple files that store a part of a module, subprogram, or code that is linked to it.

[0090] As used herein, "engine" or "software engine" refers to a software-implemented input / output system that provides different inputs and outputs. An engine may be a library, platform, software development kit ("SDK"), or a functional coding block such as an object. Each engine can be implemented on any suitable type of computing device, such as a server, mobile phone, tablet computer, notebook computer, music player, e-reader, laptop or desktop computer, PDA, smartphone, or other stationary or mobile device including one or more processors and computer-readable media. In addition, two or more engines may be implemented on the same computing device or on different computing devices.

[0091] The processes and logic flows described herein can be executed by one or more programmable computers that execute one or more computer programs to perform their functions by acting on input data and producing outputs. The processes and logic flows may also be executed by dedicated logic circuits, such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the devices may be implemented as dedicated logic circuits, such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits).

[0092] A computer suitable for running computer programs may, for example, be based on a general-purpose or dedicated microprocessor, or both, or on any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory or random-access memory, or both. Essential elements of a computer are a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. Generally, a computer includes one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, or is operationally coupled to such mass storage devices for receiving data from, transferring data to, or both. However, a computer does not have to have such devices. In addition, a computer may be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a Universal Serial Bus (USB) flash device.

[0093] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, which include, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. Dedicated logic circuits may be added to the processor and memory, or the processor and memory may be incorporated into the dedicated logic circuits.

[0094] For interaction with the user, embodiments of the subject matter described herein provide information to the user. This can be implemented on a computer having a display device for displaying information, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, for allowing the user to provide input to the computer. Interaction with the user may also be carried out using other types of devices. For example, the feedback given to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, the computer may implement interaction with the user by sending documents to or receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0095] Embodiments of the subject matter described herein can be implemented in a computing system including, for example, a backend component as a data server, or a computing system including a middleware component such as an application server, or a computing system including a client computer having a graphical user interface or web browser to enable interaction between a user and a frontend component, such as an implementation of the subject matter described herein, or a computing system including any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by a form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0096] A computing system may include a client and a server. The client and server are typically geographically separated and communicate through a communication network. The relationship between the client and the server is established by computer programs running on each computer that have a client-server relationship. In some embodiments, the server transmits data, such as an HTML page, to a user device acting as a client, for example, to display data to a user interacting with the user device and to receive user input from the user. Data generated on the user device, for example, as a result of interactions with the user, can be received by the server from the user device.

[0097] This specification contains many specific details of implementation, but these should not be construed as limitations on the scope of any invention or claimable scope, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in terms of separate embodiments may also be realized in combination in a single embodiment. Conversely, various features described in terms of a single embodiment may also be realized separately or as any appropriate subordinate combination in multiple embodiments. In addition, while some features may be described and initially claimed as functioning in a particular combination, one or more features included in the claimed combination may, in some cases, be omitted from that combination.

[0098] Similarly, although operations are shown in a specific order in the drawings, this should not be interpreted as requiring such operations to be performed in a specific or sequential order, or that all shown operations must be performed to obtain the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be understood as requiring such separation in all embodiments. It should be understood that the described program components and systems can generally be integrated into one software product or packaged into multiple software products.

[0099] Further implementation examples of the disclosed technology are illustrated in the following examples. Example 1: A computer-based method for more efficiently performing calculations of a convolutional neural network layer, wherein the convolutional neural network layer is configured to receive an X×Y×Z input tensor and process the received input tensor using one or more convolutional neural network layer weight matrices to generate a U×V×W output tensor, the method comprising the step of transforming the X×Y×Z input tensor into an X'×Y'×Z' input tensor, where X' is less than or equal to X, Y' is less than or equal to Y, and Z' is greater than or equal to Z, and one or more modified convolutional neural network The process includes the step of obtaining a twerk layer weight matrix, (i) the modified weight matrix acts on the X'×Y'×Z' input tensor to generate a U'×V'×W' output tensor, (ii) the U'×V'×W' output tensor includes the transformed U×V×W output tensor, where U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W, and the process includes the step of generating the U'×V'×W' output tensor by processing the X'×Y'×Z' input tensor using the modified weight matrix, where the U'×V'×W' output tensor includes the U×V×W output tensor.

[0100] Example 2: X' is equal to the ceiling of X divided by a natural number N, and Y' is a natural number M. The method described in Example 1, where Y divided by is equal to the ceiling, and Z' is equal to Z multiplied by NM.

[0101] Example 3: The method described in Example 2, wherein the above X×Y×Z input tensor represents the XY first input, and the above X'×Y'×Z' input tensor represents the X'Y' superinput, and each superinput contains multiple first inputs.

[0102] Example 4: The method described in Example 3, wherein the number of the multiple first inputs is equal to NM inputs. Example 5: The method according to any one of Examples 1 to 4, wherein the action of the modified weight matrix on the above X'×Y'×Z' input tensor is equivalent to the action of the convolutional neural network layer weight matrix on the above X×Y×Z input tensor.

[0103] Example 6: The modified weight matrix entries described above include one or more convolutional neural network layer weight matrices, as described in any one of Examples 1 to 5.

[0104] Example 7: The modified weight matrix described above is the method according to any one of Examples 1 to 6, which includes a matrix having a band structure.

[0105] Example 8: U' is equal to the ceiling of U divided by a natural number P, and V' is a natural number Q. The method described in any one of Examples 1 to 7, where V divided by is equal to the ceiling, and W' is equal to W multiplied by PQ.

[0106] Example 9: The convolutional neural network layer described above includes an X-dimensional stride S and a Y-dimensional stride T, and the relationship between the strides S and T, the output tensor U×V×W, and the transformed output tensor U'×V'×W' is denoted by U'=ceiling(U / NS) and V'=ceiling(V / MT), as described in Example 8.

[0107] Example 10: The method according to Example 8 or Example 9, wherein the above U×V×W output tensor represents the UV first output, and the above U'×V'×W' output tensor represents the U'V' super output, and each super output contains multiple first outputs.

[0108] Example 11: The method in Example 10, where the above multiple first outputs are equal to PQ outputs. Example 12: The method according to any one of Examples 1 to 11, wherein the convolutional neural network layer described above includes a pooling sublayer.

[0109] Example 13: The method according to Example 12, further comprising the step of replacing the pooling sublayer with a stride operation on the transformed input tensor.

[0110] Example 14: The method according to Example 12 or Example 13, wherein the convolutional neural network layer includes kernel stride processing, and the method further includes the step of performing efficient kernel stride processing based on the transformed input tensor and the modified weight matrix.

[0111] Example 15: The method according to any one of Examples 1 to 14, wherein the above U×V×W output tensor represents the UV first output, the above U'×V'×W' output tensor represents the U'V' super output, each super output contains multiple first outputs, and the number of first outputs among the above multiple first outputs depends on the dimension of the above convolutional neural network layer weight matrix.

[0112] Example 16: The method according to any one of Examples 1 to 15, wherein the above U×V×W output tensor represents the UV first output, the above U'×V'×W' output tensor represents the U'V' super output, each super output contains a plurality of first outputs, and the number of outputs among the plurality of first outputs depends on one or more of the following: (i) the architecture of the convolutional neural network layer, (ii) the architecture of the neural network model containing the convolutional neural network layer, or (iii) the device that implements the convolutional neural network layer.

[0113] Example 17: The modification of the weight matrix of one or more convolutional neural network layers described above is a software-only modification, as described in any one of Examples 1 through 16.

[0114] Example 18: A system comprising one or more computers and one or more storage devices, wherein the one or more storage devices store arithmetic instructions that, when executed by the one or more computers, cause the one or more computers to perform an operation, the operation being an operation for more efficiently performing calculations of a convolutional neural network layer configured to receive an X×Y×Z input tensor and generate a U×V×W output tensor by processing the received input tensor using one or more convolutional neural network layer weight matrices, the operation including converting the X×Y×Z input tensor to an X'×Y'×Z' input tensor, where X' is less than or equal to X, Y' is less than or equal to Y, Z' is greater than or equal to Z, and the process involves obtaining one or more modified convolutional neural network layer weight matrices, (i) the modified weight matrices act on the X'×Y'×Z' input tensor to generate a U'×V'×W' output tensor, and (ii) the U'×V'×W' output tensor includes a transformed U×V×W output tensor, where U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W, and the process involves processing the X'×Y'×Z' input tensor using the modified weight matrices to generate the U'×V'×W' output tensor, where the U'×V'×W' output tensor includes the U×V×W output tensor.

[0115] Example 19: X' is equal to the ceiling of X divided by a natural number N, and Y' is a natural number Z' is equal to the ceiling of Y divided by M, and Z' is equal to Z multiplied by NM, as shown in Example 18. The system.

[0116] Example 20: The system described in Example 19, wherein the above X×Y×Z input tensor represents the XY first input, and the above X'×Y'×Z' input tensor represents the X'Y' superinput, and each superinput contains multiple first inputs.

[0117] Example 21: The system described in Example 20, where the number of the above multiple first inputs is equal to NM inputs. Example 22: The system described in any one of Examples 18 to 21, wherein the action of the modified weight matrix on the above X'×Y'×Z' input tensor is equivalent to the action of the convolutional neural network layer weight matrix on the above X×Y×Z input tensor.

[0118] Example 23: The entries in the modified weight matrix above are from any one of the systems described in Examples 18 to 22, which include one or more convolutional neural network layer weight matrices.

[0119] Example 24: The above modified weight matrix is ​​a system described in any one of Examples 18 to 23, including a matrix having a band structure.

[0120] Example 25: U' is equal to the ceiling of U divided by a natural number P, and V' is a natural number The system described in any one of Examples 18-24, where V divided by Q is equal to the ceiling, and W' is equal to W multiplied by PQ.

[0121] Example 26: The above convolutional neural network layer includes an X-dimensional stride S and a Y-dimensional stride T, and the relationship between the strides S, T, the output tensor U×V×W, and the transformed output tensor U'×V'×W' is denoted by U'=ceiling(U / NS) and V'=ceiling(V / MT), as in the system described in Example 25.

[0122] Example 27: The system described in Example 25 or Example 26, wherein the above U×V×W output tensor represents the UV first output, and the above U'×V'×W' output tensor represents the U'V' super output, and each super output contains multiple first outputs.

[0123] Example 28: The system described in Example 27, where the above multiple first outputs are equal to PQ outputs. Example 29: The system described in Example 18, wherein the convolutional neural network layer includes a pooling sublayer.

[0124] Example 30: The system according to Example 29, further comprising replacing the pooling sublayer with a stride operation on the transformed input tensor.

[0125] Example 31: The system according to any one of Examples 18 to 30, wherein the convolutional neural network layer includes kernel stride processing, and the method further includes performing efficient kernel stride processing based on the transformed input tensor and the modified weight matrix.

[0126] Example 32: The system described in any one of Examples 18 to 31, wherein the above U×V×W output tensor represents the UV first output, the above U'×V'×W' output tensor represents the U'V' super output, each super output contains multiple first outputs, and the number of first outputs among the above multiple first outputs depends on the dimension of the convolutional neural network layer weight matrix.

[0127] Example 33: The above U×V×W output tensor represents the UV first output, the above U'×V'×W' output tensor represents the U'V' super output, each super output contains multiple first outputs, and the number of outputs among the above multiple first outputs is (i) the number of outputs in the above convolutional neural network layer A system according to any one of Examples 18 to 32, which depends on one or more of the following: (ii) an architecture of a neural network model including the convolutional neural network layer described above, or (iii) a device for implementing the convolutional neural network layer described above.

[0128] Example 34: The above modification of one or more convolutional neural network layer weight matrices is a software-only modification, as described in any one of Examples 18-33.

[0129] Example 35: A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform an operation, wherein the operation is for more efficiently performing calculations of a convolutional neural network layer configured to receive an X×Y×Z input tensor and generate a U×V×W output tensor by processing the received input tensor using one or more convolutional neural network layer weight matrices, wherein the operation includes converting the X×Y×Z input tensor to an X'×Y'×Z' input tensor, where X' is less than or equal to X, Y' is less than or equal to Y, and Z' is greater than or equal to Z. The process includes obtaining the above modified convolutional neural network layer weight matrix, (i) the above modified weight matrix acts on the above X'×Y'×Z' input tensor to generate a U'×V'×W' output tensor, and (ii) the above U'×V'×W' output tensor includes the transformed U×V×W output tensor, where U' is less than or equal to U, V' is less than or equal to V, and W' is greater than or equal to W, and the process includes generating the above U'×V'×W' output tensor by processing the above X'×Y'×Z' input tensor using the above modified weight matrix, where the above U'×V'×W' output tensor includes the above U×V×W output tensor.

[0130] Specific embodiments of the subject matter are as described above. Other embodiments are included in the scope of the following claims. For example, the operations described in the claims may be performed in a different order and still yield the desired results. As an example, the processes described in the accompanying drawings do not necessarily require that the desired results be obtained in the specific order or sequence shown. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. A method performed using a convolutional neural network implemented on a hardware integrated circuit, wherein the method is The method includes receiving an input tensor for a layer of a convolutional neural network having a stride greater than 1, wherein the input tensor has multiple dimensions and multiple inputs, and the method further includes The method further includes generating a modified input tensor from the input tensor based on a stride greater than 1, wherein the modified input tensor has multiple dimensions and corresponding multiple inputs, and the method further includes The method includes processing the modified input tensor using a modified weight matrix for the superpixel layer of the convolutional neural network having a stride equal to 1, wherein processing includes applying a convolution to the input of the modified input tensor using the modified weight matrix, and the method further includes A method comprising generating a transformed layer output of the superpixel layer in response to processing the modified input tensor, wherein the transformed layer output includes an output that is mathematically equivalent to an output generated by a neural network layer by processing the input tensor with an unmodified version of the modified weight matrix.

2. The method according to claim 1, wherein applying the convolution involves convolving the input of the modified input tensor with the corresponding weights of the modified weight matrix as the convolution filter is processed on the input of the modified input tensor, based on a kernel stride that controls the shift of the convolution filter of the modified weight matrix.

3. The method according to claim 1, comprising grouping a plurality of inputs of the input tensor by exchanging at least the size of the space or an index relating to the X and Y dimensions of the input tensor with the size of the depth or an index relating to the Z dimension of the input tensor.

4. The method further includes, based on grouping the aforementioned multiple inputs, transforming the layers of the convolutional neural network having a stride greater than 1 into the superpixel layers having a stride equal to 1. The method according to claim 3, wherein the superpixel layer includes a reduced number of kernel elements compared to the unconverted layer of the convolutional neural network.

5. Converting the layers of the aforementioned convolutional neural network into the superpixel layers is, The method according to claim 4, comprising applying a superpixel transform to the input of the convolutional neural network layer of the input tensor and the unmodified convolutional neural network layer weight matrix to generate a superpixel convolutional neural network layer that is different in shape but mathematically equivalent.

6. The method according to claim 4, wherein processing the modified input tensor using the modified weight matrix for the superpixel layer requires fewer matrix multiplications than processing the input tensor using the unmodified weight matrix for the untransformed layer of the convolutional neural network.

7. The method according to any one of claims 1 to 6, wherein generating the transformed layer output of the superpixel layer includes generating a score or classification output corresponding to the input of the input tensor, which is an extracted feature of an image.

8. Each of the input tensor and the modified input tensor is a multidimensional tensor having a corresponding depth dimension. The method according to claim 1, wherein the corresponding depth dimension of the modified input tensor is greater than the corresponding depth dimension of the input tensor.

9. A system for performing neural network computations using a convolutional neural network implemented on a hardware integrated circuit, wherein the system is The system comprises a processor and a non-temporary, machine-readable storage device for storing instructions, wherein the instructions are instructions that the processor can execute to perform an operation, and the operation is, The operation includes receiving an input tensor for a layer of a convolutional neural network having a stride greater than 1, wherein the input tensor has multiple dimensions and multiple inputs, and the operation further includes The operation includes generating a modified input tensor from the input tensor based on a stride greater than 1, wherein the modified input tensor has multiple dimensions and corresponding multiple inputs, and the operation further includes The operation includes processing the modified input tensor using a modified weight matrix for the superpixel layer of the convolutional neural network having a stride equal to 1, the processing includes applying a convolution to the input of the modified input tensor using the modified weight matrix, and the operation further includes A system comprising generating a transformed layer output of the superpixel layer in response to processing the modified input tensor, wherein the transformed layer output includes an output that is mathematically equivalent to the output generated by the neural network layer by processing the input tensor with an unmodified version of the modified weight matrix.

10. The system according to claim 9, wherein applying the convolution includes convolving the input of the modified input tensor with the corresponding weights of the modified weight matrix as the convolution filter is processed on the input of the modified input tensor, based on a kernel stride that controls the shift of the convolution filter of the modified weight matrix.

11. The system according to claim 9, wherein the operation includes grouping a plurality of inputs of the input tensor by exchanging at least the size of the space or an index relating to the X and Y dimensions of the input tensor with the size of the depth or an index relating to the Z dimension of the input tensor.

12. The operation includes, based on grouping the plurality of inputs, converting the layers of the convolutional neural network having a stride greater than 1 to the superpixel layers having a stride equal to 1, The system according to claim 11, wherein the superpixel layer includes a reduced number of kernel elements compared to the unconverted layer of the convolutional neural network.

13. Converting the layers of the aforementioned convolutional neural network into the superpixel layers is, The system according to claim 12, comprising applying a superpixel transform to the input of the convolutional neural network layer of the input tensor and to the unmodified convolutional neural network layer weight matrix to generate a superpixel convolutional neural network layer that is different in shape but mathematically equivalent.

14. The system according to claim 12, wherein processing the modified input tensor using the modified weight matrix for the superpixel layer requires fewer matrix multiplications than processing the input tensor using the unmodified weight matrix for the untransformed layer of the convolutional neural network.

15. The system according to any one of claims 10 to 14, wherein generating the transformed layer output of the superpixel layer includes generating a score or classification output corresponding to the input of the input tensor, which is an extracted feature of an image.

16. Each of the input tensor and the modified input tensor is a multidimensional tensor having a corresponding depth dimension. The system according to any one of claims 10 to 15, wherein the corresponding depth dimension of the modified input tensor is greater than the corresponding depth dimension of the input tensor.

17. A computer program that stores instructions for performing neural network computations using a convolutional neural network implemented on a hardware integrated circuit, wherein the instructions are executable instructions for a processor to perform an operation, and the operation is The operation includes receiving an input tensor for a layer of a convolutional neural network having a stride greater than 1, wherein the input tensor has multiple dimensions and multiple inputs, and the operation further includes The operation includes generating a modified input tensor from the input tensor based on a stride greater than 1, wherein the modified input tensor has multiple dimensions and corresponding multiple inputs, and the operation further includes The operation includes processing the modified input tensor using a modified weight matrix for the superpixel layer of the convolutional neural network having a stride equal to 1, the processing includes applying a convolution to the input of the modified input tensor using the modified weight matrix, and the operation further includes A computer program comprising generating a transformed layer output of the superpixel layer in response to processing the modified input tensor, wherein the transformed layer output includes an output that is mathematically equivalent to the output produced by a neural network layer by processing the input tensor with an unmodified version of the modified weight matrix.

18. The computer program according to claim 17, wherein the operation includes grouping a plurality of inputs of the input tensor by exchanging at least the size of the space or an index relating to the X and Y dimensions of the input tensor with the size of the depth or an index relating to the Z dimension of the input tensor.

19. The operation includes, based on grouping the plurality of inputs, converting the layers of the convolutional neural network having a stride greater than 1 to the superpixel layers having a stride equal to 1, The computer program according to claim 18, wherein the superpixel layer includes a reduced number of kernel elements compared to the unconverted layer of the convolutional neural network.

20. The computer program according to claim 19, wherein converting the layers of the convolutional neural network to the superpixel layers comprises applying the superpixel transform to the convolutional neural network layer input of the input tensor and the unmodified convolutional neural network layer weight matrix to generate a superpixel convolutional neural network layer that is different in shape but mathematically equivalent.

Images