Spatial local transformation of matrices

Abstract

The invention discloses spatial local transformation of a matrix. A method includes receiving a respective size of each of a plurality of dimensions of a kernel convolved with an input tensor to generate an output value; and generating a control pattern for calculating the output value, the control pattern being a two-dimensional square matrix in which the size of each dimension is equal to the product of the width and height of the kernel, the length of the two-dimensional square matrix being equal to the product of the sizes of the dimensions of the kernel, the control pattern is generated by generating a value for each position of the control pattern based on a location of the position in the control pattern and a respective size of each of the dimensions of the core, the value indicates a location for a flattened version access value from the input tensor for convolution with the kernel.

Description

[0001] This application is a divisional application of the Chinese patent application with application number 201980074328.9 and invention name “Space-local transformation of matrix”. The international application date of the parent application is November 19, 2019, and the international application number is PCT / US2019 / 062303. Technical Field

[0002] The present disclosure relates generally to matrix computations, and in particular to spatially local transformations of matrices. Background Art

[0003] Modern neural networks include multiple layers. Each layer can include a large number of input values, which are then transformed to generate outputs (i.e., activations), which are used as inputs for subsequent layers. Typically, these input and output values ​​are represented as matrices (e.g., arrays of values ​​with one to multiple dimensions). A common transformation performed on these input values ​​is convolution. Convolution applies a kernel to adjacent values ​​in the input to generate output values, and the kernel includes weight values ​​and can also be represented as a matrix. This operation is repeated for all values ​​in the input (such as modified by weights) to generate a set of output values. However, since the kernel must read in adjacent values ​​multiple times and will cross or slide over the same input value multiple times to generate multiple outputs, it is computationally very expensive to perform using a naive approach.

[0004] Therefore, a system is desired that can more efficiently compute the convolution of input values ​​modified by the weights of a kernel to generate output values. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 A system 100 for convolving an input tensor with a spatially local transform (SLT) of a kernel to generate output activations is shown, according to one embodiment.

[0006] Figure 2 is an example of convolving a two-dimensional input with a kernel according to one embodiment.

[0007] Figure 3 is a flowchart illustrating a method of flattening an input tensor according to one embodiment.

[0008] Figure 4 An example of flattening an input tensor in the case of a one-dimensional kernel is shown according to one embodiment.

[0009] Figure 5A A first portion of an example of flattening an input tensor in the case of a two-dimensional kernel is shown according to one embodiment.

[0010] Figure 5BA second portion of an example of flattening an input tensor in the case of a two-dimensional kernel is shown according to one embodiment.

[0011] Figure 5C An example of flattening an input tensor for multiple input channels is shown according to one embodiment.

[0012] Fig. 6A A first portion of an example of flattening an input tensor in the case of a three-dimensional kernel is shown according to one embodiment.

[0013] Figure 6B A second portion of an example of flattening an input tensor in the case of a three-dimensional kernel is shown according to one embodiment.

[0014] Figure 7 is a flow chart illustrating a method of generating an extended core according to one embodiment.

[0015] Figure 8 An example of generating an extended kernel in the case of a one-dimensional kernel according to one embodiment is shown.

[0016] Fig. 9A An example of generating extended kernels for different two-dimensional kernels according to one embodiment is shown.

[0017] Fig. 9B An example of generating an extended kernel using column-major expansion according to one embodiment is shown.

[0018] Fig. 9C An example of generating an extended kernel in the case of multiple kernel filters according to one embodiment is shown.

[0019] Fig. 10A A first portion of an example of generating an extended kernel in the case of a three-dimensional kernel according to one embodiment is shown.

[0020] Fig. 10B A second part of an example of generating an extended kernel in the case of a three-dimensional kernel according to one embodiment is shown.

[0021] Fig. 10C A third portion of an example of generating an extended kernel in the case of a three-dimensional kernel according to one embodiment is shown.

[0022] Fig.11 is a flow chart illustrating a method of generating a control pattern according to one embodiment.

[0023] Fig. 12A A first portion of an example of a conceptual basis for generating a control pattern according to one embodiment is shown.

[0024] Fig. 12BA second part of an example of a conceptual basis for generating a control pattern according to one embodiment is shown.

[0025] Fig.13A An example of a portion of the generation of values ​​for a control pattern of a two-dimensional kernel according to one embodiment is shown.

[0026] Fig. 13B An example of a portion of the generation of values ​​for a control pattern of a three-dimensional kernel according to one embodiment is shown.

[0027] Fig. 13C An example of control mode for generation of cores of different dimensions according to one embodiment is shown.

[0028] Fig.14 is a flow chart illustrating a method for generating a convolved output using a flattened input, an expanded kernel, and a control mode according to one embodiment.

[0029] Fig.15A A first portion of an example of generating output activations using flattened inputs, expanded kernels, and control patterns in the case of a one-dimensional kernel, according to one embodiment, is shown.

[0030] Fig. 15B A second portion of an example of using flattened input, expanded kernel, and control patterns to generate output activations in the case of a one-dimensional kernel is shown according to one embodiment.

[0031] Fig. 15C A third portion of an example of using flattened input, expanded kernel, and control patterns to generate output activations in the case of a one-dimensional kernel, according to one embodiment, is shown.

[0032] Fig.16A A first portion of an example of generating output activations using flattened inputs, expanded kernels, and control patterns in the case of a two-dimensional kernel, according to one embodiment, is shown.

[0033] Fig. 16B A second portion of an example of using flattened input, expanded kernel, and control patterns to generate output activations in the case of a two-dimensional kernel, according to one embodiment, is shown.

[0034] Fig. 16C A third portion of an example of generating output activations using flattened inputs, expanded kernels, and control patterns in the case of a two-dimensional kernel is shown according to one embodiment.

[0035] Fig.16D A fourth portion of an example of generating output activations using flattened inputs, expanded kernels, and control patterns in the case of a two-dimensional kernel is shown according to one embodiment.

[0036] Fig.16EAn example of using a flattened input, an expanded kernel, and a control pattern to generate output activations with multiple channels in the case of a two-dimensional kernel is shown according to one embodiment.

[0037] Fig.16F A second portion of an example of using a flattened input, an expanded kernel, and a control pattern to generate output activations with multiple channels in the case of a two-dimensional kernel, according to one embodiment, is shown.

[0038] Fig.17A A first portion of an example of generating output activations using flattened inputs, expanded kernels, and control patterns in the case of a three-dimensional kernel is shown according to one embodiment.

[0039] Fig. 17B A second portion of an example of generating output activations using flattened inputs, expanded kernels, and control patterns in the case of a three-dimensional kernel is shown according to one embodiment.

[0040] Fig.18A A hardware diagram showing exemplary components for generating an extension core according to one embodiment is shown.

[0041] Fig.18B A hardware diagram is shown of an exemplary shifter circuit used in exemplary components for generating an extended core according to one embodiment.

[0042] Fig.19 A hardware diagram of exemplary components for generating a control pattern is shown according to one embodiment.

[0043] Fig. 20 A hardware diagram of exemplary components for performing multiply-add operations to generate output activations is shown according to one embodiment.

[0044] Fig.21 An exemplary layout of components for computing output activations in a machine learning processor is shown according to one embodiment.

[0045] Fig.22A An example machine learning processor is shown according to one implementation.

[0046] Fig. 22B An example machine learning processor according to another implementation is shown.

[0047] Fig.23 is a block diagram illustrating components of an example computing machine capable of reading instructions from a computer-readable medium and executing the instructions in a processor (or controller).

[0048] For purposes of illustration only, the drawings depict and the detailed description describes various non-limiting embodiments. DETAILED DESCRIPTION

[0049] The drawings and the following description relate to preferred embodiments by way of illustration only. Those skilled in the art may regard alternative embodiments of the structures and methods disclosed herein as viable alternatives that may be employed without departing from the principles disclosed.

[0050] Reference will now be made in detail to several embodiments, examples of which are shown in the accompanying drawings. It should be noted that similar or identical reference numerals may be used in the accompanying drawings where feasible, and these reference numerals may indicate similar or identical functions. These drawings depict embodiments of the disclosed systems (or methods) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein.

[0051] Exemplary Systems

[0052] Figure 1 A system 100 for convolving an input tensor with a spatial locality transform (SLT) of a kernel to generate an output activation is shown according to one embodiment. In one embodiment, the system 100 includes an input tensor 102, a kernel 110, an input stream generator 122, a flattened input stream 124, an extended kernel generator 126, an extended kernel 128, a control pattern generator 134, a control pattern 136, a multiply-add unit 140, and an output activation 142. However, in other embodiments, the system 100 includes different elements and / or the system 100 includes a different number of the elements shown.

[0053] The input tensor 102 is a collection of input values ​​108 that are modified by the kernel 110 in a convolution operation to generate an output activation 142. In one embodiment, the input tensor 102 is represented by a matrix. The matrix can have one, two, three, or more than three dimensions and can be stored in memory as an array. The number of dimensions of the array can be equal to the number of dimensions of the matrix. In another embodiment, the input tensor 102 has a plurality of input channels 104A-104N (commonly referred to as input channels 104). Each input channel 104 includes a matrix of one or more dimensions, such as dimension 1 106A to dimension 3 106C of the matrix in the input channel 104A shown.

[0054] In a convolution operation, kernel 110 is applied to input tensor 102 to generate an output activation, such as output activation 142. Figure 2Describes additional details about the convolution operation. The kernel 110 can be represented by a one-dimensional or more dimensional matrix. The matrix includes kernel values ​​114, which, in the case of a neural network, represent weights applied to the input values ​​108 of the input tensor 102. The number of dimensions of the input tensor 102, specifically the number of dimensions of the matrix for each input channel 104, should be at least equal to or greater than the number of dimensions of the kernel 110. The size (i.e., the number of elements of the kernel 110 throughout each dimension) can be smaller or larger than the input tensor 102. If the kernel 110 is larger than the input tensor 102, the input tensor 102 can be padded so that the size of the kernel 110 is less than or equal to the padded input tensor 102, thereby allowing the convolution operation to occur. The resulting output activation 142 is a matrix of the same size as the (padded) input tensor 102.

[0055] In one embodiment, the core 110 includes one or more core "filters" 112A-112M (generally referred to as core filters 112). The number of core filters 112 does not need to be equal to the number of input channels 104. Each core filter 112 includes a set of sub-filter cores, which are filters themselves and are equal to the number of input channels 104. A convolution operation is performed on each input channel 104 using each sub-filter core, and the resulting output matrix is ​​summed to generate a single output activation feature matrix or output features 144A-144M (generally referred to as output features 114). This operation is repeated for each core filter 112. The number of output features 144 generated is equal to the number of core filters 112 present in the core 110. Therefore, for example, if the core 110 includes one core filter 112, a single output feature 144 is generated. However, if the core 110 includes five core filters 112, the output activation 142 will have five output features 144. This enables the neural network to apply different kernel weights to different parts of the input (i.e., different input channels 104), and combine the results into a new output (i.e., different output features 144), which can then be used as further input in another layer of the neural network. FIG. 16E to FIG. 16F Additional details regarding the use of multiple input channels 104 and multiple nuclear filters 112 are described.

[0056] The input stream generator 122 converts the input tensor 102 into a flattened input stream 124. In order to read in the input tensor 102 more efficiently and avoid reading the same value of the input tensor 102 multiple times when the core 110 strides across the input tensor 102, the input stream generator 122 converts the input tensor 102 into a flattened input stream 124, which can then be fed into a buffer or other memory for access by the multiplication-addition unit 140 in the following efficient manner, which greatly reduces the number of times the necessary input values ​​are read.

[0057] To perform a transformation of the input tensor, the input stream generator 122 may first fill the input tensor 102. The input tensor 102 may be filled so that its size is the modulus (i.e., a multiple) of the size of the kernel 110. In addition, the kernel 110 has a "focal" point. The focal point is the location of the kernel 110 at which the output value of the convolution with the kernel 110 is generated. The input stream generator 122 fills the input tensor 102 so that the kernel 110 reaches every input value in the input tensor 102 when crossing the input tensor 102. Thus, for example, if the kernel is a 3×3 kernel, where the focal point is the center of the kernel, the two-dimensional matrix of the input tensor 102 may be further filled around the outer border of the matrix after being filled to the modulus of the kernel using a single vector of padding values ​​to allow the focal point of the kernel to slide over the input values ​​at the outer edge of the input tensor 102.

[0058] Finally, the input stream generator 122 may also pad the input tensor 102 to meet any requirements of the processor used to perform the convolution. Due to the size of the bus, the number of parallel processors, the memory size, or other constraints, the input stream generator 122 may also pad the input tensor 102 with padding values ​​so that the input tensor 102 meets the constraints of the processor after being flattened to become the flattened input stream 124. In one embodiment, the input stream generator 122 pads one side (e.g., the trailing edge) of each dimension of the input tensor 102 (or each input channel 104 of the input tensor 102) with a padding value whose size is equal to the size of the core 110 (or the size of each sub-filter core of each core filter 122). The padding values ​​described herein may be any null padding value, zero padding value, or standard padding value.

[0059] After padding the input tensor 102, the input stream generator 122 divides or segments the input tensor 102 into tiles using the padding. Each tile is the size of the core 110. Therefore, the padded matrix of the input tensor 102 is divided into multiple separate smaller matrices, each of which has the same size as the core 102. Although the input tensor 102 is described herein as being partitioned, this does not mean that the input stream generator 122 necessarily generates a new matrix for each tile. Instead, the input stream generator 122 can simply delineate the boundaries of each tile in the matrix of the input tensor 102.

[0060] In case of multiple input channels 104, the input stream generator 122 divides the matrix of each input channel 104 into its own set of multiple tiles. The size of the tile of each input channel 104 is equal to the size of the sub-filter kernel of each kernel filter 112 applied to the input channel 104.

[0061] After dividing the input tensor 102, the input stream generator 122 identifies the flattening order. This is the order in which the values ​​of each tile in the input tensor 102 are read. The order can be any order, and can include row priority, column priority, diagonal priority, channel priority, etc. In row priority, the values ​​in each row are read in a specific order (e.g., from left to right), and each row is also read in sequence in a specific order (e.g., from top to bottom). In column priority, instead of reading each row, each column is read in a specific order (e.g., from left to right), and the values ​​in each row are read in a specific order (e.g., from top to bottom) for each column. In diagonal priority, the tiles can be read along the diagonal. If the tiles include more than one layer, each layer can be processed continuously. Other orders are also feasible as long as the same sorting mode is subsequently used in the generation of the extension core 128 and the generation of the control mode 136.

[0062] The input stream generator 122 reads the values ​​of each tile in the identified flattening order and arranges the values ​​in the manner in which they are read for each tile on a single vector, thereby "flattening" the tile. The input stream generator 122 reads all tiles of the input tensor 102 and generates a corresponding number of vectors. These vectors are placed parallel to each other to generate a flattened input stream 124. The input stream generator 122 can read the tiles in a specific order, such as row-first order, column-first order, etc. As long as the order of the tiles is reflected in the generation of the extension kernel 128 and the control mode 136, any order can be used, and a valid output can be generated by the multiplication-addition unit 140.

[0063] If the input tensor 102 includes multiple input channels 104, the matrix of each input channel 104 is processed separately to generate a flattened input stream for each matrix of each input channel 104. The multiple flattened input streams may be combined together to form a (combined) flattened input stream 124. The multiple flattened input streams may be combined by "stacking" the multiple flattened input streams together, concatenating them together, or by some other combination method.

[0064] The flattened input stream 124 is the result of the input stream generator 122 flattening the input tensor 102. Regardless of the number of dimensions of the input tensor 102, the flattened input stream 124 of each matrix of the input tensor 102 is (at most) two-dimensional. This is because: as described above, each tile of the input tensor 102 is converted into a vector, and the vectors are placed parallel to each other. In one embodiment, if the input tensor 102 includes multiple input channels 104, the flattened input stream generated from each matrix of each input channel 104 can be combined with the flattened input streams of other matrices of other input channels 104 by laying each flattened input stream vertically or horizontally adjacent to each other (in a computer readable memory). The combined flattened input stream can also be represented in three dimensions, where each flattened input stream generated from each input channel is stacked on top of each other. In memory, this can be represented using a depth-first storage method, where the values ​​along the depth of the three-dimensional flattened input stream are stored as a priority order.

[0065] The flattened input stream 124 may be fed into a buffer or stored in a memory. The flattened input stream 124 may be read by the multiplication-addition unit 140 and referenced according to the tile number corresponding to each vector in the flattened input stream 124. In the example shown, the tiles of the flattened input stream 124 range from 0 to K to J. Here, tile #1+K represents the tile in the original input tensor 102, i.e., the first tile on the second row (or column) of tiles. The value K represents the number of tiles in the row or column of the filled input tensor 102 (depending on the order in which the tiles are read), and varies according to the width of the input tensor 102. For example, if the kernel and thus the tile is a 3×3 matrix, the K value of the input tensor filled with a width of 9 will be 3. In this case, three tiles fit along the width of the input tensor, so the number of the first tile on the second row (i.e., the fourth tile overall) will be tile number 3, because the tile count starts at 0. If the tensor 102 includes an additional dimension, the first tile of each second series of values ​​for that dimension indicates an additional tag. For example, if the input tensor 102 includes three dimensions, a separate tag M will indicate the number of tiles in a single layer of the input tensor 102, and tile #1+M will indicate the index value of the first tile in the second layer of the input tensor 102. As described below, these tags and tile numbers can be used by the multiplication and addition unit to reference or point to the correct tile in the flattened input stream 124 using indicators in the control mode 136. Alternatively, the tile number including the tag can be used as a clock cycle delay value that allows the multiplication and addition unit 140 to delay reading a value from a buffer including the flattened input stream 124. This enables the multiplication and addition unit 140 to similarly reference a specific tile in the flattened input stream 124.

[0066] As described in further detail below, this access method, in conjunction with other components of system 100, allows system 100 to perform only two reads on input tensor 102, compared to a standard convolution in which a number of values ​​in the input tensor would need to be read equal to the number of values ​​in kernel 110. s-1 times, where s is the number of dimensions of the input, thus saving a lot of resources. Figures 3 to 6B Additional details regarding the input stream generator 122 and the flattened input stream 124 are described.

[0067] The extended core generator 126 generates an extended core 128, which is used by the multiplication-addition unit 140 together with the flattened input stream 124 and the control mode 136 to generate the output activation 142. The extended core 128 is generated from the core 110. As shown below, the purpose of the "extended" core is to make it possible to use a simple dot product to multiply the selected vector of the input value of the flattened input stream 124 with the vector (e.g., column, row, channel) of the extended core 128 without having to make the original core 110 span the input tensor 102. This significantly simplifies the generation of the output activation 142. The expansion of the core 110 follows a specific pattern, including generating a rotation combination of the core 110 in a hierarchical manner. In the following description, reference is made to the first dimension, the additional dimension, and the last dimension. This is merely a way to refer to the dimensions of the core in an organized manner, because the core may have one to many dimensions, and is not intended to indicate the order or size of each dimension. For example, if the kernel is three-dimensional, the last dimension of the kernel does not necessarily refer to some three-dimensional representation of the kernel, but simply refers to the dimension number 3 of the kernel as long as it has 3 different dimension numbers (e.g., dimension 1, dimension 2, and dimension 3).

[0068] In one embodiment, the extended core generator 126 uses the first dimension of the core 110 (or a core filter 112 or a subfilter core of the core filter 112) and generates a square value block for each single-dimensional vector of the core, which includes all rotations of the single-dimensional vector. The block is generated by placing each of the rotations parallel to each single-dimensional vector. For example, a 3×3 two-dimensional core will have three 3×1 single-dimensional vectors for dimension 1. For each of these vectors, all possible rotations of the vector are generated, thereby creating two additional single-dimensional vectors for each. These vectors are placed parallel to the single-dimensional vectors used to generate the additional vectors, thereby creating square blocks for each single-dimensional vector. These square blocks can be called circulant matrices.

[0069] Thereafter, for each additional dimension of the kernel, the blocks of the previous dimension region or the lower dimensional region are grouped into sets. Each set includes blocks of the previous dimension aligned along vectors parallel to the axis of that dimension. So, returning to the example of a 3×3 matrix, if the previously generated square blocks are placed in the same position as the source that generated them (i.e., the single-dimensional vector), then for dimension 2, the vector can pass through all generated blocks. Therefore, the set of dimension 2 includes all blocks generated in the previous operation. In a 3×3 kernel, this includes three blocks, one block for each vector of the kernel.

[0070] For each set generated, the extended core generator 126 generates all rotations of the blocks in the set. Using the previous example, there are three blocks in the unique set of dimension 2. Therefore, the rotation of the set will generate two additional combinations of three blocks, for a total of 9 blocks. Similar to the method described above for single-dimensional vectors, these two additional combinations are placed parallel to the blocks in the set. Here, considering all dimensions of the two-dimensional kernel, the extended core generator 126 ends the generation of the extended kernel, and the combined results of the 9 blocks are output as the extended kernel 128.

[0071] However, if the kernel 110 includes additional dimensions, the above process is repeated, resulting in different sets of blocks being rotated and combined. This ultimately results in all dimensions being considered, and the resulting combination of all blocks with blocks of lower dimensions is output as an extended kernel 128. Therefore, as the dimensions increase, the number of sets also increases. In each additional dimension, after the blocks aligned along the vector as described above in the previous dimension are rotated, the combination of the rotated blocks is placed in a new block. These new blocks are used for the calculation of the next additional dimension and are combined together in various rotations. This continues until the last dimension, the number of dimensions of which is equal to the total number of dimensions of the kernel. On the last dimension, the last set of rotations is performed, and the resulting rotation combination is then output as an extended kernel. Therefore, for each additional dimension to be processed, the number of block sets is reduced in number, and after processing the rotation of the final dimension, only a single set of output blocks remains.

[0072] The number of vectors (e.g., rows or columns) of the extended kernel 128 is also equal to the number of elements in the kernel, or the number of elements in the tile of the input tensor 102. The actual size of the extended kernel 128 itself depends on the size of the kernel 110. The size of each dimension of the two-dimensional extended kernel 128 is equal to the product of the size values ​​of each dimension of the kernel 110. For example, if the kernel is a 2×2×2 matrix, the extended kernel 128 will have eight vectors (2^3) and therefore a size of 8×8, because the product of the values ​​of the kernel's dimensions is 8.

[0073] As described herein, the generated extended kernel 128 can be used in a dot product with selected values ​​of the flattened input stream 124 to generate output activations 142. As previously described, the values ​​in the flattened input stream 124 can be selected via delays or pointers using the indication of the control mode 136. After the values ​​are selected, the selected values ​​can be combined with the selected vectors of the extended kernel 128 by multiplying each vector of the extended kernel 128 by a different value selected from the flattened input stream 124 to generate output values ​​of the output activations 142. Since the number of vectors of the extended kernel 128 is equal to the number of elements in the tile (the same size as the kernel 110 as previously described), the number of output values ​​is also equal to the same number of elements per tile, thus comprising tiles of output activations 142 (or output matrices of values ​​in the case of multiple kernel filters). The positions of the tiles in the matrix of output activations 142 have positions corresponding to the same position tiles of the matrix of the input tensor 102.

[0074] The system 100 described herein allows the calculation of the value in each output tile without having to compute the value as in naive methods (e.g. Figure 2 As kernel 110 slides through input tensor 102 to each position corresponding to a position in an output tile, instead of repeatedly reading certain values ​​in the input tensor, each output tile is generated by extending a single pass of the kernel and selecting values ​​from the flattened input stream 124 a single time.

[0075] In the case of multiple kernel filters 112, similar to the above process, an extended kernel will be generated for each kernel filter 112 and applied to the input. In the case of multiple input channels 104, each kernel filter 112 has multiple sub-filter kernels, each sub-filter kernel corresponding to an input channel 104. In this case, an extended kernel will be generated for each sub-filter kernel, and each extended kernel generated from each sub-filter kernel will be applied to the relevant portion of the flattened input stream 124 corresponding to the input channel 104 for which the sub-filter kernel has been applied in the naive convolution implementation. Figures 7 to 10C to describe additional details about the generation of the extended kernel.

[0076] The control pattern generator 134 generates a control pattern 136 based on information about the kernel size in the kernel size input 132. The kernel size indicates the size of each dimension of the kernel 110 (or each kernel filter). The control pattern generator 134 takes this information and generates the same control pattern 136 for kernels of the same size and dimension. The control pattern generator 134 generates a value for each position of the control pattern 136 based on the coordinates of the position (e.g., row number, column number) and the size of the kernel's dimension. For each position, the control pattern generator 134 performs one or more test inequalities (equal to the number of kernel dimensions). Each test inequality is an inequality between the modulo operation of the row number and the modulo operation of the column number of the position in the control pattern 136. The result of the test inequality (i.e., true or false) is used to reference a table of control pattern values ​​to generate a value for the position in the control pattern.

[0077] Although the control pattern generator 134 is described herein as generating a control pattern 136 based on a core size input 132 using a programmatic approach, in other embodiments, the control pattern generator 134 accesses a pre-generated version of the control pattern 136 from a memory, non-volatile memory, a program instruction stack, or other source, and selects the correct pre-generated control pattern 136 from the source based on the core size input 132.

[0078] The control pattern 136 is a matrix that indicates to the multiplication-addition unit 140 which parts of the flattened input stream 124 to select from to generate selected values ​​that are multiplied (using dot products) with the vector of the extended kernel 128 to generate each output value. In a naive convolution implementation, for each time the kernel 110 crosses a certain position on the input tensor 102, the convolution operation is performed by adding the values ​​near the position representing the current focus of the kernel to the value at the focus itself, as weighted by the corresponding value in the kernel (at the same position). The resulting sum is the output value for the position corresponding to the focus. Therefore, for each focus position, a different value of the input tensor 102 is selected for summation. Since the flattened input stream 124 is divided into different tiles, each output position, in particular each output position in the output tile, is calculated using the value of the input tile at the same position as the output tile or the value of the input value in the adjacent input tile. Thus, the control pattern 136 indicates to the multiply-add unit 140, for each output position in the tile, a different specific input tile from which to extract input values ​​to perform the convolution calculation, i.e., the dot product mentioned previously. In particular, each vector of the control pattern 136 corresponds to a different position in the output tile and indicates the tile of the input matrix from which the input value is selected for calculating the value of the output position.

[0079] Since the output is generated tile by tile, the control pattern 136 may only need to indicate input tiles having positions corresponding to the positions of the current output tile being processed, and input tiles having positions corresponding to adjacent positions of the current output tile being processed. For example, in a two-dimensional matrix of the input tensor 102, the tiles are horizontally and vertically adjacent to the current output tile being processed. In a three-dimensional matrix, this can include tiles in adjacent layers. For matrices of higher dimensions, this will include other "adjacent" tiles. Therefore, the number of tile positions indicated by the control pattern 136 is a power of 2 of the dimension size of the input matrix. The size of the control pattern 136 is the same as the size of the extended kernel 128 generated from the kernel 110.

[0080] The control mode 136 indicates the tile according to the tile number of each tile in the flattened input stream 124. Since tiles adjacent to the input tile are not necessarily adjacent in tile number, the indicator in the control mode 136 does not directly indicate the tile by the tile number, but indicates the tile by the relative position relative to the tile number of the tile currently being processed. Therefore, for example, the relative position can indicate the current tile, the row (or column) below (or to the right) of the current tile, or the layer behind the current tile. Since the width / height and depth of the matrix of the input tensor 102 are known, the relative position can be calculated based on this information. For example, the tile in a row below the current tile will be the tile number of the current tile plus the width of the matrix. Therefore, the control mode 136 can indicate the value of the row width for such a tile, or a pointer or reference indicating the row width. The tile in a row below the current tile and to the right of it will be represented by the current tile number plus the row width plus 1. Therefore, the control mode 136 can indicate the row width +1, or an indicator of the row width +1 for such a tile. In one embodiment, the row width is a variable K as described above. Each of these combinations of indicators can be indicated by one or more bits. Because the control pattern 136 has a fractal pattern, each sub-portion of the matrix of the control pattern 136 can be indicated by a single first bit, additional sub-portions are indicated by additional bits, and each value in the sub-portion has a second bit. The combination of a single value and the sub-portion to which the value belongs indicates the relative tile position.

[0081] In the case of multiple core filters 112, if the core filters or sub-filter cores have different dimensions, multiple control patterns 136 can be selected. For each core of different dimensions, a corresponding control pattern 136 will be selected or generated according to the above method. The corresponding control pattern 136 will be used to select values ​​from the portion of the flattened input stream 124. Additional details about the generation of the control pattern 136 and its characteristics will be referred to below. Figure 11 to Figure 1 3 for description.

[0082] The multiplication-addition unit 140 performs the final calculation using the flattened input stream 124, the expansion kernel 128, and the control pattern 136 to generate the output activation 142. As described in more detail above, the multiplication-addition unit 140 selects values ​​from the flattened input stream 124 using indicators from the control pattern 136. As described above, each vector in the control pattern 136 indicates a specific tile of the flattened input stream 124 for accessing input values. The identifier at each position in the control pattern 136 corresponds to an indication of which tile, i.e., which vector, of the flattened input stream 124 the input value is selected from. The position within the vector of the control pattern 136 also corresponds to the position of the selected vector in the flattened input stream 124 that includes the correct input value. By parsing the entire vector of the control pattern 136, the multiplication-addition unit 140 generates a vector of selected input values.

[0083] The multiplication-addition unit 140 also selects a corresponding vector of the extended core 128 that matches the position (e.g., row number or column number) of the vector of the control pattern 136 used to select values ​​from the flattened input stream 124. The multiplication-addition unit 140 performs a dot product between the selected vector of the extended core 128 and the vector including the selected values ​​of the flattened input stream 124 to generate a single output value. The single output value is placed on the output tile that matches the position of the input tile currently being processed. In addition, the position of the single output value in the output tile corresponds to the position number of the vector in the control pattern 136 (or extended core 128) used to generate the single output value.

[0084] The multiply-add unit 140 repeats the process described herein for all vectors in the control pattern 136 (and the extended kernel 128), thereby generating a total number of output values ​​equal to the number of positions within the output tile. This allows the multiply-add unit 140 to generate an output tile of output activations 142 for each input tile of the input tensor 102. The multiply-add unit 140 also repeats the process for all input tiles of the input tensor 102 to generate the same number of output tiles for the output activations 142. After generating the entire set of output tiles, the multiply-add unit 140 outputs the complete set of output tiles as the final output of the output activations 142.

[0085] In the case of multiple input channels 104, the multiplication-addition unit 140 generates a "pre-output" for each input channel 104 using the kernel components of the particular sub-filter kernel or kernel 110 specified for that input channel 104. Each pre-output is generated in the same manner as the output described above. However, after all pre-outputs are generated for each input channel, the multiplication-addition unit 140 sums the values ​​of the pre-outputs into a single output matrix, which can be the output activations 142.

[0086] In the case of multiple kernel filters 112, the multiplication-addition unit 140 also applies each kernel filter 112 to the input tensor 102, and if there are multiple input channels 104, the multiplication-addition unit 140 applies each kernel filter 112 to all of the above input channels 104 and sums the pre-outputs. This creates a separate output feature 144 for each kernel filter 112. Each output feature 144 is a matrix of the same size as the input channel 104. The set of all output features 144 represents the output activation 142 and can be used as the input channel 104 in the next layer of the neural network (that is, the output activation 142 of one layer becomes the input tensor 102 of the next layer of the neural network). Refer to the following Figures 14 to 17B Additional details regarding the multiplication-addition unit 140 are described.

[0087] Example Convolution Operation

[0088] Figure 2 2 is an example of performing convolution on a two-dimensional input by a kernel according to one embodiment. Although the input values ​​202 and output values ​​214 described herein are represented using matrices, in practice, they can simply be represented using arrays, flat files, trees, or other methods without being arranged as matrices, as long as the calculation results are the same.

[0089] To perform a convolution on a set of input values ​​202, padding may first be applied to the input values ​​202 so that the "focal point" of the kernel may be applied to the values ​​at the edge of the input values ​​202 to allow the size of the output value 214 to be the same as the size of the input values. However, in other cases, padding is not applied to the input values ​​202, in which case the size of the output value 214 is smaller than the size of the input 202. Each side of the output values ​​214 is reduced by a number of elements equal to the number of elements between the following edge of the kernel and the focal point of the kernel, compared to the size of the input values ​​202, the edge being on the same side of the kernel as the side of the output values ​​214. The edge of the input values ​​202 is conceptually a location around the outer boundary of the matrix representing the input values ​​202, so that no other value is farther from the center of the matrix than the value on that edge. As described above, the "focal point" (also referred to as the "origin") of the kernel is the location of the kernel that is conceptually above the current output pixel. Typically, it will correspond to the location of one of the values / elements in the kernel, and for a symmetric kernel (such as a 3×3 kernel in the example shown), the focal point is typically the center element.

[0090] Therefore, in order to make the positions of the output values ​​214 match the positions of the input values ​​202, padding is added to the input values ​​so that the focus of the kernel when crossing the first position in the input values ​​causes the focus to be at positions of the input values ​​202 that match the edge values ​​on the matrix of the input values ​​202, and the output values ​​214 that are subsequently generated have positions that match the positions of these edge values. This ensures that the size of the output values ​​214 is the same as the size of the input values ​​202.

[0091] Thus, in the example shown, padding 204 of width 1 is added around the edges of the matrix representing the input values ​​202. Here, the size of the padded input values ​​202 is a multiple of the size of the kernel 210, so no additional padding is required to make the size a multiple of the kernel size.

[0092] After padding, the kernel 210 is applied to the input value 202. The kernel 210 is applied by striding (i.e., moving) the entire input value 202. The kernel is strided across the input value 202 according to the stride value. The stride value is a value that determines how far the kernel is moved for each stride. If the value is 1, the focus of the kernel strides across every possible position of the input value 202. Due to the padding, the kernel 210 will never exceed the boundaries of the padded input value 202 at any time, but the focus of the kernel can overlap with every original value of the input value 202. Note that if the stride value exceeds 1, the focus of the kernel will not be applied to every value of the input value 202.

[0093] For each input value that is the focal point of the kernel 210, a convolution operation 212 is performed on the input value 202 that is conceptually under the kernel 210. Thus, in the example shown, the focal point of the kernel is the selected convolution point 208. Here, as shown by the weighted dark square box 206 surrounding the selected convolution point 208, a total of 9 values ​​have been selected that are equal to the size of the kernel 210 and surround the selected convolution point 208. If the focal point of the kernel 210 is not at its center, the box will be shifted accordingly so that the position of the box will have its corresponding focal position at the selected convolution point 208. For example, if the focal point is at the upper left corner of the kernel 210, the box will be shifted one point down and one point right so that the selected convolution point 208 is at the upper left corner of the box.

[0094] The convolution operation 212 takes each of the input values ​​202 under the above bounding box 206 and performs a dot product operation between the input value 202 under the box and the value of the kernel 210 to generate a single output value of the output value 214. As shown, when the focus of the kernel 210 is located at the selected convolution point 208, the output is "8" for the kernel 210 and the input value 202. Here, due to the padding and the selected focus position, the position of the output value is the same as the position of the selected convolution point 208 in the input value 202.

[0095] The kernel is then spanned across one position in the input value 202 (horizontally or vertically) and the convolution operation 212 is repeated. The convolution operation 212 is complete when the focus of the kernel 212 has visited all possible input values ​​202. This creates a complete set of output values ​​that are the same size as the input values ​​(no padding). This convolution operation allows the input values ​​to be modified by the weights (kernel) and combined with other values ​​in the input to generate a new output.

[0096] Mathematically, the convolution operation can be expressed as:

[0097]

[0098] Here, A may be the kernel, B may be the kernel, and c is the convolution result.

[0099] This convolution operation allows a neural network to process complex data and generate a desired output based on that data. However, as shown here, this causes the same input value (e.g., the example repeatedly read value 218) to be read multiple times when the core 210 spans the input value 202. During one convolution of the input value, the example repeatedly read value 218 in the example shown will be read nine times because nine core positions will overlap with the value. If other cores are applied to the input value 202 as described above, even more repeated reads will be performed on the same value 218. Therefore, although the convolution operation 212 is a powerful tool in machine learning, in a naive approach such as shown here, it can potentially generate a very large number of reads, i.e., a very large number of I / O operations, which can become a problem as the input value 202 increases and the number of convolutions increases. Therefore, as disclosed herein, a more optimized method is provided that can significantly reduce the number of times the input value is repeatedly read. As described above, the number of reads can be reduced to 2 s-1 Reads, where s is the number of dimensions of the input. Thus, in the example here, instead of the 9 reads per value described above for example values ​​218, for the two-dimensional matrix of input values ​​202, only 2 reads are required.

[0100] Flattened input stream generation

[0101] Figure 3 300 is a flowchart illustrating a method of flattening an input tensor according to one embodiment. Although the flowchart shown may show an order of operations, the operations shown may be performed in any order and may have a greater or lesser number of operations. In one embodiment, Figure 3 The operations shown may be performed by the input stream generator 122 .

[0102] The input stream generator 122 receives 310 an input tensor to be convolved with a kernel. The input tensor may be the input tensor 102. The kernel may be the kernel 110. In one embodiment, the input stream generator 122 fills the input tensor with padding values ​​so that the size of the output of the convolution of the input tensor using the kernel is the same as the size of the input tensor. In one embodiment, for each padded input tensor dimension, the size of the input tensor with the padding value is an integer multiple of the corresponding dimension of the kernel. In another embodiment, the input stream generator 122 fills the trailing edge of each dimension of the input tensor with the following padding values, the width of the padding value being equal to the size of the kernel in the corresponding dimension. The trailing edge of each dimension is the edge (i.e., the face or the other end) of the input tensor with the largest index value. The padding value may be zero or empty, or some other value.

[0103] The input stream generator 122 divides 320 the input tensor into one or more tiles, each tile having a size equal to the size of the kernel. Thus, if the kernel is a 3×3× kernel, a 9×9 input tensor (including padding) will be divided into 9 (two-dimensional or 2D) tiles. Similarly, given a 3×3×3 kernel, a 9×9×9 input tensor will be divided into 27 (three-dimensional or 3D) tiles. In one embodiment, the kernel does not have square dimensions, and in this case, the input stream generator 122 divides the input tensor in an order aligned with the direction in which the kernel spans the input tensor. Thus, if the kernel spans in a row-first manner (from left to right), the input tensor will be divided along each row before entering the next row, and so on. Alternatively, in another embodiment, the input stream generator 122 divides the input tensor in an order orthogonal to the direction in which the kernel spans the input tensor. Thus, in the example above, the input tensor is first divided along each column (from top to bottom, then to the next column).

[0104] The input stream generator 122 flattens 330 the values ​​in one or more tiles into vectors to generate a flattened input stream. For each of the one or more tiles of the input tensor, this may include: accessing the values ​​of the tile in a defined order; arranging the values ​​in a vector according to the defined order; and arranging one or more vectors corresponding to each of the one or more tiles in a parallel arrangement to generate the flattened input stream. The defined order may be a row-first order, a column-first order, or a channel-first order. The channel-first order first accesses elements in a three-dimensional (3D) tile along an axis corresponding to the depth of the 3D tile, and then accesses elements in the 3D tile along axes corresponding to the width and height of the 3D tile. Although the flattened input is shown as two-dimensional here, in other embodiments, it includes more dimensions.

[0105] The flattened input stream may be stored in a buffer. The buffer may be read by a hardware acceleration processor to perform a multiply-add operation using 1) the values ​​in the flattened input stream selected by the control mode and 2) an expansion of the kernel to generate an output of the convolution operation as described herein without loading the values ​​of the input tensor into the buffer multiple times.

[0106] Additionally, the input tensor has a plurality of channels, wherein the kernel has a plurality of filters, and wherein the input channels are convolved with each kernel filter to generate an output having a plurality of output channels.

[0107] The following are the kernels of different dimensions Figure 4 , FIG. 5A to FIG. 5C and FIG. 6A to FIG. 6B Provides additional examples that flatten input streams.

[0108] Figure 4 An example 400 of flattening an input tensor in the case of a one-dimensional kernel according to one embodiment is shown. In one embodiment, the process described herein may be performed by the input stream generator 122 .

[0109] At 401, the input stream generator 122 receives an input 420. In the example shown, the input 420 is a set of input values ​​of 6×3, represented as a matrix. At 402, the input stream generator 122 pads the input 420 based on the kernel size. The input stream generator 122 pads the input 420 so that the focus (i.e., the center value b) of the kernel 422 can correspond to the same position of the input 420 so as to generate an output of the same size as described above. This causes the input stream generator 122 to add padding 426A. The input stream generator 122 also pads the input 420 so that it is a multiple (modulus) of the kernel size. This adds two columns of padding to padding 426B. Finally, in one embodiment, due to hardware requirements, the input stream generator 122 pads the trailing edge of the input 420 with padding equal to the width of the kernel. Here, the width of the kernel 422 is 3, so a padding of width 3 is added to the end of the input 420, resulting in three additional columns of padding at padding 426C.

[0110] At 403, the input stream generator 122 divides the now padded input 424 into tiles whose size is equal to the size of the kernel 422 to create a tiled input 428. Since the kernel is a matrix of size 3×1, the padded input 424 is divided into tiles of size 3×1 each. This will produce 12 tiles. Here, the tiles are divided and sorted in a row-major form so that the padded input 424 is divided row by row. However, the tiles can also be divided column by column, in which case tile 1 is [0,0,0], tile 2 is [1,7,13], and so on. As long as subsequent operations also follow the same orientation, the output produced will be the same.

[0111] At 404, the input stream generator 122 transforms the tiled input 428 into a flattened input 430. Here, the direction of the input 432 indicates the direction in which the flattened input 430 is input to the next step (the multiplication-addition unit 140). Therefore, tile 1 is placed first, then tile 2, and so on until tile 12. Each tile is transformed into a single vector and placed in parallel with other vectors transformed from other tiles. Since the tiles here are already vectors, no other transformations are performed. However, as shown in subsequent examples, tiles may not always be vectors, and in this case, they are flattened into vectors.

[0112] The flattened input 430 may be stored as an array, tree, or other structure, and may be stored in a cache, memory, or other storage medium. Each value in the tile may be accessed using a reference pointer, a memory address, or according to a clock cycle delay. With a clock cycle delay, the flattened input 430 may be read in one vector at a time, and different reads may be delayed by a certain number of clock cycles to access different vectors in the flattened input 430. For example, tile 7 may be accessed by delaying access by seven clock cycles.

[0113] Figure 5A FIG. 5 shows a first portion of an example 500 of flattening an input tensor in the case of a two-dimensional kernel, according to one embodiment. Figure 4 In contrast to the example in FIG. 5A to FIG. 5B In the example, a two-dimensional kernel is used to generate a two-dimensional tile. In one embodiment, the process can be performed by the input stream generator 122.

[0114] At 501, the input stream generator 122 receives an input and pads it based on the kernel size to generate a padded input 520. As previously described, padding satisfies three requirements: 1) in order to enable the size of the output values ​​to be the same as the size of the input values, padding can be added based on the focus of the kernel; 2) the input can be further padded to be modulo the size of the kernel; and 3) in some embodiments, due to hardware requirements, padding of an additional kernel width is added to the trailing edge of the input.

[0115] Thus, in the example shown, padding of the vector width is added to the outside of the input, and padding of the additional kernel width is added to the right and bottom (trailing edge) of the input. Since kernel 522 is a 3×3 kernel, the additional padding is 3 units wide on the right and 3 units high at the bottom.

[0116] At 502, the input stream generator 122 divides the padded input 520 into tiles whose size is equal to the kernel. Since the kernel is 3×3, the size of each tile is 3×3. This creates the tiled input 524. The row size of the tiled input 524 is indicated using a variable K, which is then used to index from the tile being processed to the tile in the second row. Therefore, the first tile on the second row of the tiled input 524 is tile 1+K, and the tile to its right is tile 1+K+1. Note that if the first tile is indexed starting at "0", K is set to [row tile size]+1, rather than the row tile size.

[0117] exist Figure 5B This process is further described in Figure 5B A second portion of an example of flattening an input tensor in the case of a two-dimensional kernel is shown according to one embodiment.

[0118] At 503, the input stream generator 122 flattens the tiles in the image into vectors according to a specific tile order. Here, the flattening order is row-first, as indicated by the flattening order 532 for a single tile 530 example. In other words, for each tile, as shown by the directional arrows, the values ​​in the tile are read row by row and placed in the vector of the tile in the flattened input 528. This vector, similar to the vector for the flattened input 430, is placed parallel to the vectors generated for other tiles (according to the same flattening order) and is used as an input to the multiplication-addition unit 140 according to the direction of the input 534. Although the vectors are shown as being vertical, in other embodiments, the orientation may be different.

[0119] In another embodiment, alternatively, the flattening order 538 is column-major, meaning that the values ​​in each tile are read column-by-column and then placed in a single vector and placed in parallel with other vectors generated from other tiles that populate the input. Thus, in contrast to the flattened input 528 (where the vectors for tile 1 are ordered [0,0,0,0,1,2,0,11,12]), here alternatively the vectors for tile 1 are ordered [0,0,0,0,1,11,0,2,12] because the values ​​in tile 1 are read column-by-column rather than row-by-row. The exact flattening order 538 does not affect the output as long as the order of other processes in the generation of the output values ​​corresponds to the same order.

[0120] Figure 5C An example of flattening an input tensor for multiple input channels according to one embodiment is shown. In one embodiment, the input tensor 102 may have Figure 1Multiple input channels 104A-104N are shown. Multiple input channels are convolved with the kernel. The outputs of these convolutions with the kernel are summed together to generate output activations. If there are multiple kernel filters in the kernel, each kernel filter is convolved with multiple input channels to generate output features. Each input channel can correspond to a different component of the input, such as a color channel, etc. In one embodiment, this process can be performed by the input stream generator 122.

[0121] At 504, the input stream generator 122 blocks any additional input channels with the first flattened input. For clarity, the core 542 is shown as a 2×2 core. Therefore, each tile is 2×2 and each vector is a 1×4 vector. Here, each input channel 540A (collectively referred to as input channel 540) is similar to the input from 501, so the input stream generator 122 fills each input channel 540 and flattens it, thereby generating flattened input channels 544A to 544N. In one embodiment, they are then "stacked" on top of each other to generate a flattened input stream 546. However, in other embodiments, they can be combined differently, such as via cascading, placement in a 3D array, etc. In any of these cases, the flattened input stream 546 is a combination of various flattened input channels 544, and the corresponding tiles of each of the flattened input channels 544 can be referenced together into the flattened input stream 546. Thus, for example, a reference to tile 5 using flattened input stream 546 (whether by delaying clock cycles or otherwise) will be able to reference tile 5 in all flattened input channels 544 that include flattened input stream 546 .

[0122] Fig. 6A The first part of an example of flattening an input tensor in the case of a three-dimensional kernel is shown according to one embodiment. Figures 4 to 5C Likewise, in one embodiment, the process described herein may be performed by the input stream generator 122 .

[0123] At 601, similar to the above reference Figures 4 to 5CIn the described method, the input stream generator 122 receives a three-dimensional input and fills the input to generate a filled input 626. For clarity, only a single width of the fill 628 is shown here. In addition, for clarity, the actual input values ​​630 are not shown here because they overlap in the perspective representation. Instead, the values ​​of the fill 628 are shown as cubes with a dark gray pattern, and the input values ​​630 are shown as white cubes. The same is true for the kernel 632 and the kernel value 634. Note that the filled input 626 has three dimensions 106: dimension 1 106A, dimension 2 106B, and dimension 3 106C. In some cases, dimension 1 can be referred to as width, dimension 2 can be referred to as height, and dimension 3 can be referred to as depth. In addition, dimension 1 can be referred to as having columns, dimension 2 can be referred to as having rows, and dimension 3 can be referred to as having channels or layers.

[0124] At 602, the input stream generator 122 blocks the padded input 626 into tiles whose size is equal to the size of the kernel 632 to generate the block input 626. Since the size of the exemplary kernel 632 is 2×2×2 and can be represented by a three-dimensional matrix, the size of the tile is also 2×2×2. In addition to the K variable 636 indicating the number of tiles in the row of the block input, the three-dimensional input also includes an M variable indicating the number of tiles in the layer of the block input. When calculating an output value corresponding to the position of the current tile, inputs from tiles one layer behind the current tile and tiles below the current tile may be required (as in the case of 2D input). Therefore, in addition to the K parameter that can be used to indicate the position of the tile below the current tile, the M parameter indicating the tile behind the current tile can also be used. Although directions such as below and behind are mentioned, in reality, the input tiles may not be arranged geometrically as shown here, but may be indicated abstractly in the data structure. However, the same K and M parameters will apply. Since the width of the padded input 626 is 6 and the height is 6, each layer of tiles includes 9 tiles, and the 2×2 layer of the kernel is divided nine times evenly into a 6×6 layer of the padded input 626. Therefore, the M parameter is 9 and the K parameter is 3 in the example.

[0125] The process Figure 6B Continue, Figure 6B The second part of the example of flattening the input tensor in the case of a three-dimensional kernel according to one embodiment is shown. Here, at 603, similar to the above reference Figure 4 and Figure 5BIn the described process, the input stream generator 122 flattens each tile into a vector according to the example flattening order 640, and places the vectors parallel to each other. Since the size of a single tile is now 2×2×2, it includes 8 values. An order is established so that the eight values ​​are read according to the order and arranged in a single vector. Here, the example flattening order 640 first reads a single tile row by row (row priority) and then each layer / channel (channel priority). In the example flattening order 640 showing the exploded view of a single tile, the order is indicated by bold and italic numbers. Therefore, the value at the tile position indicated by "1" is read first, followed by the value at the tile position indicated by "2", and so on, ending with the value at the tile position indicated by "8". Depending on the size of the tile, a different order can be established. As with the previous order, as long as the order remains consistent throughout the process, the output value will be the same regardless of which order is used.

[0126] The result of flattening by the input stream generator 122 is a flattened input 642. The example flattened order 640 is indicated by the order 644 in each vector of the flattened input. Thus, for each vector, the "1" position indicated by the order 644 has the value of the position from the corresponding tile that is the "1" position of the example flattened order 640 for that tile. Here, Fig. 6A Again, the dark gray area indicates the fill value, and the white area indicates the input value. For example, Fig. 6A Tile 3 in includes 4 input values ​​and 4 fill values. After following the example flattening order 640, it is transformed into a vector indicated at tile #3 at the flattened input 642, where the input values ​​alternate with the fill values. Figure 4 and Figure 5B Like the flattened input in , the flattened input is sent to the multiplication-addition unit 140 for further processing to generate an output value.

[0127] Extended Core Generation

[0128] Figure 7 700 is a flowchart illustrating a method of generating an extended core according to one embodiment. Although the flowchart shown may show an order of operations, the operations shown may be performed in any order and may have a greater or lesser number of operations. In one embodiment, Figure 7 The operations shown may be performed by the extended core generator 126 .

[0129] The extended kernel generator 126 receives 710 a kernel for convolving the input tensor. The kernel may be the kernel 110. For the first (smallest) dimension of the kernel, the extended kernel generator 126 generates a square value block (circulant matrix) for each single-dimensional vector of the kernel, which includes all rotations of the single-dimensional vector. Here, each single-dimensional vector is a unique vector that is a row of the kernel, a column of the kernel, a diagonal of the kernel, or a channel of the kernel. The channel of the kernel is a vector of the kernel aligned along an axis corresponding to the depth of the kernel (in this case, the third dimension).

[0130] If the core 110 has multiple core filters 112, each core filter is processed separately according to the flowchart shown in this article. If each core filter includes a sub-filter core, then the sub-filter core is also processed separately according to the flowchart. Although the dimension is referred to as "first" or "smallest" herein, this is not intended to indicate that the dimensions of the core have a particular order or size relationship. Alternatively, any dimension of the core can be selected as the first dimension. For example, in Fig. 6A In the three-dimensional kernel, the dimension along the X-axis 620 (X dimension), the dimension along the Y-axis 622 (Y dimension), or the dimension along the Z-axis 624 (Z dimension) can be selected as the first dimension.

[0131] Similarly, a kernel convolves with a single-channel or multi-channel input tensor to produce a single output channel. If the kernel includes multiple filters, each filter is applied to the input tensor to produce a separate output activation.

[0132] After processing the first dimension, for each additional dimension of the kernel, the extended kernel generator 126 groups 740 the blocks of the previous dimension into a set of blocks, each of which includes blocks of the previous dimension aligned along a vector parallel to the axis of the (additional) dimension. A vector parallel to the axis of a dimension is a vector parallel to the axis of the dimension currently being processed. Therefore, using the example above, if the X dimension is the first dimension, the next dimension to be processed may be the Y dimension (although it may also be the Z dimension). Along the Y dimension, one or more vectors may be drawn that are parallel to the Y axis and also aligned with the blocks from the previous dimension. This is because four blocks are generated in the previous dimension (X dimension), and each of these blocks, if conceptually "positioned" at the same position as the position where these blocks are generated, will form two "columns" of blocks. Each of these "columns" is aligned with or through a vector parallel to the Y axis. Aligned blocks are grouped together.

[0133] For additional dimensions, the extended core generator 126 also generates 750 one or more value blocks, each block including all rotations of the blocks within each of the block sets of the previous dimension. Therefore, in the above example, the blocks of each column rotate the blocks within the column to generate a new square value block. For the Y dimension, this will generate two square value blocks. In order to rotate a vector or a set of blocks, the extended core generator 126 rotates the values ​​in the blocks in the row or set (e.g., barrel shifts the values) to generate a new ordering of vectors or blocks for each rotation. Once all possible rotations are generated, the generated vectors or block ordering combinations are "stacked" together to generate an output block including all rotations of the base vector or base block set.

[0134] Continuing with the example, when processing the next dimension, the Z dimension, the vector is parallel to the Z axis. In this case, the two blocks are generated based on the rotation of the first two "columns" of each of the two blocks. Therefore, the two blocks are rotated in a way that aligns with the vector when they are "positioned". There are no other blocks that do not pass through this vector, so this is the only block required. However, if the kernel has higher dimensions (i.e., greater than 3 dimensions), there may be additional blocks that are not aligned with this vector, so additional vectors are needed to group the additional blocks.

[0135] When all dimensions are processed, the extended core generator 126 outputs 760 a block of values ​​corresponding to the last dimension in the additional dimensions of the core as an extended core. Using the previous example, after rotating two blocks from the Y dimension, one block remains. This block is output as an extended core.

[0136] Therefore, as in the example, the three-dimensional kernel has an extended kernel, which includes a first set of vectors, each first vector having a rotation arrangement of the first block in the first block group. Each first block in the first block group corresponds to a second vector in the second set of vectors, and each second vector corresponds to the second block in the second block group. Each second block in the second block group corresponds to a vector in the kernel (kernel vector), and each second block has a third set of vectors, which represent all rotation arrangements of the corresponding kernel vector of the second block, and each third vector represents a rotation arrangement of the corresponding kernel vector. Here, each second set of vectors includes all rotation arrangements of the second block in the corresponding second block group. In addition, the first set of vectors includes all rotation arrangements of the first block in the first block group.

[0137] In one embodiment, instead of a three-dimensional kernel as in the example, the kernel is a one-dimensional (1D) matrix, and the extended kernel includes a plurality of parallel vectors, each of which is a different rotation arrangement of the kernel. Here, the extended kernel is a square matrix, the size of each dimension of which is equal to the size of the kernel.

[0138] In another embodiment, the kernel is a two-dimensional (2D) matrix, so the extended kernel includes a first set of vectors, each first vector has a rotational arrangement of the blocks in the block group. Each block in the block group corresponds to a vector in the kernel (kernel vector), and each block has a second set of vectors, which represent all rotational arrangements of the corresponding kernel vectors. Each second vector represents a rotational arrangement of the corresponding kernel vector, and the first set of vectors includes all rotational arrangements of the blocks in the block group.

[0139] Once generated, the extended kernel is a two-dimensional square matrix whose length is equal to the product of the dimensions of the corresponding kernel used to generate the extended kernel. Figure 8 , 9A to 9C and FIG. 10A to FIG. 10C Additional examples of generating extended kernels for one-dimensional kernels, two-dimensional kernels, and three-dimensional kernels are described.

[0140] Figure 8 An example 800 of generating an extended kernel in the case of a one-dimensional kernel according to one embodiment is shown. In one embodiment, the process described herein may be performed by the extended kernel generator 126.

[0141] At 801, the extended kernel generator 126 receives a kernel 820. In the example shown, the kernel 820 has dimensions of 3×1 (lower case letters represent kernel values). Thus, it is a one-dimensional kernel. Figure 7 , at 802, the extended core generator 126 rotates each single-dimensional vector of the core 820. Since the core 820 has only a single vector, namely [a, b, c], only this vector can be rotated. Rotating this vector generates a total of three combinations, the original core 820 ([a, b, c]) and the combination shown in the rotation 822 ([c, a, b] and the rotation 824 ([b, c, a)). The method of generating the rotation is indicated by the arrows shown in 802. These combinations are grouped together to form a circulant matrix. At 803, since there is no need to process the other additional dimensions of the core 820, the extended core generator 126 outputs the grouped vectors generated during the rotation as the extended core 826.

[0142] Since the extended kernel includes various rotations of the kernel, different rows of the extended kernel can be used in dot products against tiles of the flattened input stream as selected by the control mode (e.g., control mode 136). Instead of having the original kernel span multiple times over a single value in the input, the kernel is extended here so that the above kernel can be rotated in 2 s-1 The number of times applied to the flattened input stream is not necessarily equal to the number of elements in the kernel. Since the size of the expanded kernel is much smaller than the possible input tensor 102, expanding the kernel in the process described herein is much more efficient than reading the values ​​of the input tensor multiple times.

[0143] Fig. 9A An example of generating extended kernels for different two-dimensional kernels according to one embodiment is shown. Figure 8 Likewise, in one embodiment, the process described herein may be performed by the extension core generator 126 .

[0144] At 901, the extended core generator 126 receives a core. Three examples of two-dimensional cores 920A to 920C with different sizes are shown here. The size of core 920A is 3×2, the size of core 920B is 2×3, and the size of core 920C is 3×3. At 902A, the single-dimensional vectors of core 920 are rotated with core rotations 922A to 922C using row priority order, that is, the single-dimensional vectors correspond to the rows of the rotated core. As shown in the figure, the number of single-dimensional vectors will vary depending on the size of the core. For example, core 920B with three rows has three rotation sets in core rotation 922B, while core 920A has only two rotation sets.

[0145] At 903A, an additional dimension, the second dimension, is rotated. Here, Figure 7 As shown, vectors aligned with the axis of the second dimension are selected, and those blocks passing through each vector are rotated together. In the example of kernel 920A, two blocks 926A to 926B are generated in kernel rotation 922A, and these blocks are aligned with a single vector along the axis of the second dimension. Block 926A is generated by combining the rotation of rows [a, b, c], and block 926B is generated by combining the rotation of rows [d, e, f]. These two blocks are rotated in kernel rotation 924A to generate extended kernel 932A. Since there are no more dimensions in the kernel, no further rotation is performed.

[0146] A similar process is performed for core 920B. However, this time there are three blocks 928A to 928C. Therefore, in core rotation 924B, three blocks 928A to 928C are rotated and combined to generate extended core 932B. In the case of core 920C, three blocks 930A to 930C are also generated in core rotation 922C (because core 920C has three rows). These blocks 930A to 930C are rotated in the second dimension at core rotation 924C to generate extended core 932C. Each extended core has equal sizes along two dimensions (e.g., a square matrix), and the size along each dimension is equal to the product of the size of each dimension of the core. For example, core 920B has a size of 3×2, so the size of each dimension of its extended core 932B is 3×2=6.

[0147] Fig. 9B An example of using column-major expansion to generate an extended kernel according to one embodiment is shown. Fig. 9A , the example shown here uses column-major order.

[0148] At 902B, the extended core generator 126 obtains the core 920C and rotates each single-dimensional vector. However, instead, the single-dimensional vectors are selected as columns of the core 920C. Therefore, the first vector shown here has the value [a, d, g] corresponding to the labeled 1st column in the figure, rather than a vector with the value [a, b, c]. The remaining two vectors correspond to the labeled 2nd and 3rd columns.

[0149] At 903A, the rotation vector from 902B (formed as blocks 938A to 938C) is rotated in kernel rotation 936, which is similar to Fig. 9A The kernel shown in is rotated 924. This generates an extended kernel 940, which is different from extended kernel 932C, although the same kernel is used. However, if the control pattern is also generated taking into account the column-major order, the output values ​​will not be different after applying the convolution.

[0150] Fig. 9C An example of generating an extended core in the case of multiple core filters according to one embodiment is shown. If the core has multiple core "filters", each core filter will be extended in the same manner as described above for a single core. If each filter has other sub-filter cores, these sub-filter cores will also be extended in the same manner. Therefore, as shown, core filters 942A to 942N are extended to extended core filters 944A to 944N.

[0151] Fig. 10A FIG. 1 shows a first portion of an example 1000 of generating an extended kernel in the case of a three-dimensional kernel according to one embodiment. Figure 8 and 9A to 9C Likewise, in one embodiment, the process described herein may be performed by the extension core generator 126 .

[0152] At 1001, the extended core generator 126 receives a core 1026. In the exploded view shown, the core 1026 can be seen to have layers 1028A and 1028B. Each layer has a different core value "a" through "h", so that for the 2×2×2 core 1026, eight values ​​are generated.

[0153] At 1002, the extended kernel generator 126 rotates all single-dimensional vectors of the kernel 1026 (using row-major order in this example). There are four vectors in the kernel 1026, corresponding to four rows in the kernel 1026, and two rows on each layer 1028. Therefore, four rotation sets are generated by the kernel rotation 1030.

[0154] At 1003A, the extended kernel generator 126 rotates the second dimension (i.e., the next dimension) by grouping the blocks generated in the kernel rotation 1030. Here, those blocks from the kernel rotation 1030 that are aligned along a vector parallel to the axis of the current dimension being processed (i.e., the second dimension) are grouped together. Here, the blocks forming set 1034A are aligned along a vector parallel to the second dimension axis, i.e., the Y axis 1022 (i.e., vector 1032A). The blocks forming set 1034B formed by values ​​on a different layer than the layer in set 1032A are also aligned along a vector parallel to the second dimension axis (i.e., vector 1032B).

[0155] Although two vectors are drawn here, they are presented to illustrate the conceptual reasoning of how to group the blocks. In practice, no vectors are drawn, and the extended core generator 126 groups the blocks by determining the index of the dimension from which it is generated. For example, the blocks of set 1034A are generated from layer 1028A (which may have index "0"), and those of set 1034B are generated from layer 1028B. Therefore, the extended core generator 126 knows to rotate the blocks generated from these different layers separately. For subsequent dimensions, the process will be repeated, and the index value of the current dimension is used to group the blocks generated in the previous dimension.

[0156] The process Fig. 10B Continue, Fig. 10B A second part of an example of generating an extended kernel in the case of a three-dimensional kernel according to one embodiment is shown.

[0157] Here, at 1003B, the extended core generator 126 generates rotations of the previously generated blocks that have been grouped into sets 1034A-1034B. The blocks of each set are rotated separately. As shown, this produces two blocks, a block produced by the rotation of the two blocks of set 1034A, and a block produced by the rotation of the two blocks of set 1034B.

[0158] At 1004A, the extended core generator 126 repeats step 1003A, but for the next dimension, the third dimension, which is also the last dimension of the three-dimensional core 1026. Here, the extended core generator 126 groups the blocks along the vectors that are aligned with the axis of the dimension. The vector shown is vector 1040, and the axis is the Z axis 1024. Each block generated in 1003B is generated by a block on a different layer of the core, so conceptually, the blocks generated in 1003B are also on different layers. Vector 1040 is aligned with the two blocks generated (i.e., it can pass through these two blocks), so these blocks are grouped into set 1038A. Since there are no other blocks to group, this is a unique set for the third dimension.

[0159] The process Fig. 10C Continue, Fig. 10C The third portion of an example of generating an extended kernel in the case of a three-dimensional kernel according to one embodiment is shown. Here, at 1004B, the extended kernel generator 126 generates all rotations of the blocks in the set of blocks generated when processing the previous dimension. In the example shown, there is only one set 1038A, so the blocks in that set are rotated to generate the final extended kernel 1044, because this is the last dimension of the kernel 1026.

[0160] Control pattern generation

[0161] Fig.11 1 is a flowchart 1100 illustrating a method of generating a control pattern according to one embodiment. Although the flowchart shown may show an order of operations, the operations shown may be performed in any order and may have a greater or lesser number of operations. In one embodiment, the control pattern generator 134 may perform Figure 7 The operation shown.

[0162] The control pattern generator 134 receives 1110 one or more dimensions of each of the dimensions of a kernel that is convolved with the input tensor to generate an output activation. The kernel may be Figure 1 The kernel 110 in . The kernel can be one-dimensional, two-dimensional, three-dimensional or more. Each dimension of the kernel has a size, and this is received by the control pattern generator 134.

[0163] At 1120 , for each position of the control pattern, the control pattern generator 134 generates 1130 a value for the position based on the location of the position in the control pattern and one or more dimensions of each of the dimensions of the kernel.

[0164] Once the values ​​for all positions are generated, the control pattern generator 134 outputs 1140 the complete control pattern square matrix. The size of each dimension of the control pattern is equal to the product of the width and height of the kernel. In addition, the value of each position of the control pattern indicates the following position from which the value in the flattened input tensor convolved with the kernel can be accessed.

[0165] In one embodiment, each position of the control pattern has a row number and a column number, and the generation of each value includes, for each dimension of the kernel, determining the result of a test based on the row number, column number, and one or more sizes of each dimension of the kernel. Each test generates a binary output indicating the relative offset of the value accessed from the flattened input stream, which, as described above, is generated from the input tensor in tiles of the size of the kernel of the flattened input tensor. For each dimension of the kernel, the value at that position of the control pattern is generated as a combination of the binary outputs of the tests. Each test can be an inequality test that compares a first modulo operation using the row number of the position with a second modulo operation using the column number of the position. The first modulo operation is a modulo between the row number and one or more sizes of each dimension in the dimension of the kernel. The second modulo operation is a modulo between the column number and one or more sizes of each dimension in the dimension of the kernel. In the case of a two-dimensional kernel, the first test is r mod h < c mod w, and the second test is floor((r mod (h * w)) / h) < floor((c mod (h * w)) / w), where r is the row number, c is the column number, h is the height of the kernel, and w is the width of the kernel.

[0166] In one embodiment, the control pattern includes a plurality of vectors, and the number of vectors in the plurality of vectors corresponds to the number of output value positions in the tile of the size of the kernel of the output activation. The values within each vector of the plurality of vectors correspond to delay values, where each delay value indicates the amount of delay in accessing each input value in the flattened input stream. As described above, the flattened input stream includes a set of parallel vectors generated from the input tensor, and the amount of delay specifies one of the parallel vectors within the flattened input stream.

[0167] In one embodiment, the delay value is used to generate an output value for each output value position of the tile of the size of the kernel of the output activation by selecting, for the output value position corresponding to the output value to be generated, the vector in the control pattern corresponding to that output value position. The delay values in the selected vector of the control pattern are accessed, and each delay value indicates the parallel vector in the flattened input stream from which each input value used to generate the output value is accessed. A dot product is performed between the accessed input values and the vectors of the extended kernel to generate the output value at the output value position of the tile of the size of the kernel of the output activation. In one embodiment, the extended kernel is generated by modifying the kernel to include a set of hierarchical blocks, where each block in the set of hierarchical blocks includes all rotational permutations of the block that are nodes of the block. The remaining blocks of the set of hierarchical blocks are blocks that include all rotational permutations of the respective vectors of the kernel, and each block in the set of hierarchical blocks has nodes that are blocks aligned with vectors along the axis parallel to the dimension represented by the block in the 0 kernel.

[0168] In one embodiment, the control pattern is a two-dimensional square matrix whose length is equal to the product of the dimensions of the corresponding kernel used to convolve the input tensor.

[0169] In one embodiment, the order in which the delay values ​​are arranged in the control mode is based on the defined order in which the kernel-size sized tiles of the input tensor are accessed to generate the flattened input stream.

[0170] In one embodiment, the defined order is row-first, column-first, or channel-first. The channel-first order first accesses elements in a tile of the three-dimensional (3D) kernel size of the input tensor along the axis corresponding to the 3D tile depth, and then along the axes corresponding to the 3D tile width and height.

[0171] Although the above flowchart shows a method of generating a control mode according to the size of the core dimension, in another embodiment, various control modes for various core sizes are stored in a memory and retrieved according to the core size used. Additional details about the generation of control modes are described below with reference to Figures 12 to 13.

[0172] Fig. 12A A first portion of an example 1200 of a conceptual basis for generating a control pattern is shown according to one embodiment.

[0173] As mentioned earlier, the purpose of the control pattern is to select values ​​from the correct tile in the flattened input stream to be combined with the expanded kernel to generate the output value. When the kernel strides across the input values ​​in a naive convolution implementation, different values ​​are read from the input and combined with the weights in the kernel to generate the output. The control pattern can simulate this situation by selecting different tiles from where different input values ​​are located to generate the correct output value. However, in contrast to the naive implementation, the input values ​​are not read multiple times, but only a minimum number of times. In order to determine the pattern in the control pattern that indicates which tile to select from, the following method can be used. However, although the following method can be used to generate the control pattern, in practice, the hardware will likely use a separate method, such as the one described above with reference to Fig.11 The method described here may lack efficiency. Therefore, the method described herein can be used more to provide a conceptual understanding of the control mode.

[0174] At 1201, similar to the above reference Figure 4 6, the input is flattened. Here the input tensor has a size of 4×4, and since the kernel 1220 has a size of 2×2, each tile has a size of 2×2, and there are 4 tiles in total (no padding is applied here). The flattened input 1218 thus has four vectors, each representing a tile. Similar to the above reference Figure 7 to Figure 10, the core is also expanded to generate extended core 1224.

[0175] At 1202, the first tile in the flattened input 1218, tile 1 (1222A), is multiplied with the extended kernel to generate element-wise products 1226A. Thus, the "1" in tile 1 1226A is multiplied with each of the four kernel values ​​"a", "b", "c", and "d" in the extended kernel 1224 in the corresponding row to generate four products "a1", "b1", "c1", and "d1". This operation is repeated for the remaining values ​​of the tile and the other tiles 2 to 4 (1222B-1222C). As shown, this creates element-wise products 1226A-1226D.

[0176] The process Fig. 12B Continue, Fig. 12B The second part of an example of a conceptual basis for generating a control pattern according to one embodiment is shown. As the element-by-element products 1226A-1226D are generated, at 1203, those tile values ​​required to generate element-by-element convolution products are determined for each position of the kernel that overlaps on a single tile of the original input (using the row-by-row span of the kernel in the illustrated example). Therefore, at position 1 (1230A), where the position of the kernel is located at the upper left corner of the input, as shown in the kernel position bounding box 1232, the kernel values ​​"a", "b", "c", and "d" overlaid on the input are multiplied with the values ​​below that are overlaid by the kernel values ​​of the input. Therefore, "a" is multiplied with "1" in the input, "b" is multiplied with "2", "c" is multiplied with "3", and "d" is multiplied with "4" to generate the element-by-element product pairs of position 1 (1234A) as shown. Depending on whether the flattened input stream or the extended kernel is generated using row-major or column-major format, these product pairs are shown as columns in the illustrated example, but can also be converted to be displayed as rows.

[0177] Based on the element-by-element pairs 1234A, it is determined from which of these element-by-element pairs 1226A-1226D these element-by-element pairs can be located. In the first example of position 1 (1230A), all of these pairs can be located in the element-by-element product 1226A and are labeled "A". Therefore, the control pattern after analyzing position 1 is shown at 1236A as a column of all "A". Here A can represent the tile currently being processed, which can be summarized as tile X.

[0178] For position 2 (1230B), the core position 1232 is shifted one to the right, although it still overlaps the first tile in the input. The element-wise product pair 1234B is "a2", "b5", "c4", and "d7". Since the "d" in the core now exceeds the value "7" in the input, this "d7" product pair cannot be found in the element-wise product 1226A, but is found in the element-wise product 1226B, represented by "B". Therefore, the next column of the control pattern represented at 1236B has a "B" at the last value corresponding to position "d7" in the element-wise product pair 1234B. Again, the B here corresponds to the current tile X plus 1, or tile X+1.

[0179] At position 3, the core position 1232 moves down one and to the left one. The core does not move to the right again because if this happens, it will no longer overlap the first tile. This is unnecessary because the control pattern is generated to determine only those values ​​from the flattened input stream that are used to generate output values ​​for a single tile corresponding to the same tile position in the input (i.e., the current tile). Additional tiles are calculated separately. Therefore, when the system calculates output values ​​tile by tile, the process described herein is only related to those core positions where the core overlaps on the single block in question. In addition, since the focus of core 1220 in the example shown is the upper left position of core 1220 (indicated by "a"), the core position 1232 on the current tile is the focus position of the core that covers each input value of the current tile. Therefore, position 3 is a position where the focus of the core covers the third value in the current tile (using row-by-row order). In this case, the kernel reads from the current tile and also from the tile below the current tile, which is tile X+K, where K (as described above) is the number of tiles in the row (or column). These values ​​are combined with the kernel to generate element-by-element product pair 1234C. As shown in element-by-element product pair 1234C, "c9" and "d10" include values ​​from the tile below the current tile and can be found in element-by-element product 1226C, i.e., "c9" and "d10" can be located in one of the values ​​of element-by-element product 1226C. The first two product pairs, "a3" and "b4", can be found in element-by-element product 1226A because they only include values ​​in the current tile. Therefore, as shown in 1236C, the next column of the control pattern after position 3 has two "A"s and two "Cs", and the order of the letters corresponds to the order of the product pairs.

[0180] Finally, at position 4 (1230D), the position 1232 of the kernel overlaid on the input touches the values ​​of the current tile and all three other tiles surrounding the current tile. Therefore, the product pairs 1234D are "a4", "b7", "c10", and "d13". These can be identified in each of the element-wise products 1226A-1226D, respectively. Therefore, the last column of the control pattern (as shown in 1236D) indicates all four element-wise products 1226A-1226D using letters "A" to "D", respectively. Here, D represents the current tile X, plus the row size variable K plus 1, or X+K+1, and therefore refers to the tile below and to the right of the current tile.

[0181] As shown here, because the values ​​in the control pattern indicate which tiles around the current tile to retrieve values ​​for to compute the convolution of the kernel focal position on the current tile, it can also be used to select values ​​from the flattened input stream and then combine them with the corresponding vector of the extended kernel to generate output values ​​for the corresponding focal position in the output.

[0182] Fig.13A An example 1300 of a portion of generation of values ​​for a control pattern for a two-dimensional core is shown, according to one embodiment. In one embodiment, the process described herein is performed by a control pattern generator 134 .

[0183] At 1131, the control pattern generator 134 generates all values ​​of the control pattern 1320. Here, most of the values ​​of the control pattern 1320 have been generated, but the remaining values ​​1324 have not yet been generated to illustrate the process. As shown, the value 1324 is at row 0 and column 8. At 1312, the control pattern generator 134 uses the row number and column number / coordinates of the position to determine the test result of the position. The test result refers to the inequality test 1330. The number of tests is equal to the number of kernel dimensions. Since the kernel here is two-dimensional, two tests are generated. Each test corresponds to one of the shift variables, which is used to identify the flattened size of each dimension in the input tensor. For example, K represents the size or number of tiles in the first dimension, that is, each row, and M represents the number of tiles in the second dimension, that is, each layer. Alternatively, each shift variable of a particular dimension can be defined as equal to the product of the tiled size of that dimension multiplied by the tiled size of any previous dimension. The tile size here means the number of tiles that span the length parallel to the dimensions in the input tensor.

[0184] Additionally, a test is generated for the first dimension that does not correspond to the shift variable, but rather to the current tile. Thus, test_k and test_1 are generated here. Alternatively, these could be called delay_k and delay_1, since, as described above, the shift variable can be used to indicate the amount of delay for accessing the flattened input stream.

[0185] As can be seen here, the generation of the test inequalities follows a pattern. The test inequality for the first dimension 1 is r mod h < c mod w, where r is the row number of the control pattern as shown in the control pattern 1320, h is the kernel height as shown in the kernel 1322, c is the column number, and w is the width of the kernel (mod is the modulo operation, and < means less than). The second test_k corresponding to the K variable is similar, but includes the area, i.e., h*w, in the calculation because this is the second dimension. Thus, the generated test inequality is floor((r mod (h*w)) / h) < floor((c mod (h*w)) / w), where floor is the floor or rounding down operation (to the nearest integer). Although the kernel 1322 here is a square kernel, in other cases, the kernel may not be square, so the height and width of the kernel are considered in the test inequality.

[0186] The results of two tests are referenced in the result table 1332. Here, different values (i.e., indicators) are placed at the corresponding positions in the control pattern according to the true or false (i.e., 1 or 0) results of each test. Here, since there are two tests, there are a total of 4 possible values, which are represented here as the letters A, B, C, and D, and are respectively equivalent to the binary values 00, 01, 10, and 11. As described herein, these values indicate the vectors / tiles of the flattened input stream from which values can be accessed to calculate the output activation. Here, as shown in the test result 1326, the row number is 0, the column number is 8, and the results of the two tests are true, so the value of 1324 is "D" or 11, which references the current tile x + k + 1. Although the row number and column number are sorted in the order shown, in another embodiment, this order can be reversed. As long as the test inequalities are modified (e.g., by inverting them), the results here will be the same. Thus, the process here is not limited by a specific method of labeling the rows or columns of the control pattern.

[0187] Fig. 13B An example 1301 showing a part of the generation of the values of the control pattern for a three-dimensional kernel according to one embodiment is presented. In one embodiment, the process described herein is performed by the control pattern generator 134.

[0188] Here, at 1351, the control pattern generator 134 generates the values of the control pattern 1360. Contrary to the control pattern in Fig.13A , the control pattern 1360 is generated for a three-dimensional kernel 1362 having a height of 2, a width of 2, and a depth of 2 (labeled h, w, and d respectively).

[0189] As previously in Fig.13AAs described in, each dimension of the kernel is associated with a test inequality. Thus, three test inequalities are generated here in test table 1370. Similar to Fig.13A , they can also be referred to as delay_m, delay_k, and delay_1 respectively, instead of test_m, test_k, and test_1. Both test_1 and test_k are the same as the test inequalities shown in Fig.13A . Test_m corresponding to M shift variables (i.e., how many tiles are there in one layer of the input tensor) follows a pattern similar to test_k, but now includes the kernel depth (d).

[0190] If the kernel will include additional dimensions, the test inequality will be further extended to include the additional dimensions. For example, for four dimensions, the test inequality can be floor((r mod(h*w*d*t)) / (h*d*t)<floor((c mod(h*w*d*t)) / (w*d*t), where t represents "intensity", or the size of the fourth dimension of the kernel. This pattern continues for higher dimensions, where additional dimensions are added to the test inequality. Thus, the test inequality takes the following form for a kernel with a number N of dimensions: floor((r mod(product of the sizes of the dimensions of the kernel)) / (h*product of the sizes of the dimensions of the kernel (excluding width)))<floor((c mod(product of the sizes of the dimensions of the kernel)) / (w*product of the sizes of the dimensions of the kernel (excluding height))), and the remaining test inequalities for this kernel are the test inequalities for lower-dimensional kernels N-1, N-2…1, where the test inequality for a kernel with 1 dimension is the same as test_1 shown here (and the test inequality for a kernel with 2 dimensions is the same as test_k shown here, and so on).

[0191] Since there are three tests for a three-dimensional kernel, there are a total of eight combinations, as shown in result table 1372. Each combination corresponds to an indicator A to H or a three-bit binary value. As further described with reference to Fig. 13C , in some embodiments, the pattern of binary bits can be used to efficiently store the control pattern. At 1352, the control mode generator 134 generates the final remaining value 1364 at row 0 and column 7 of the control mode 1360. As shown by the test result 1366, for this position, all three test inequalities are true, corresponding to indicator H or the binary value 111. Thus, H will be stored in the position at row 0 and column 7 of the control mode 1360. However, if not all of the test inequalities return true, a different indicator will be selected based on choosing a row in result table 1372, which indicates the three test inequality results that match the calculated test inequality results.

[0192] If the kernel includes additional dimensions, the size of the result table 1372 will increase by a factor similar to the number of inequalities tested. The number of rows in the result table will equal 2. x rows, where x is the number of dimensions of the kernel. It can be seen that if the results of the test inequality are organized to be sorted so that the result of the largest dimension forms the most significant bit of the binary value, and the result of the smallest dimension forms the least significant bit, each of the combinations of the results can be regarded as a binary value and can be sorted in order in the table (e.g., from top to bottom in ascending order). The indicators associated with each combination also increase in sequence and are also associated with combinations of shift variables of sequentially increasing dimensional sizes. For example, in the example shown, A and B are not associated with shift variables, but only with the current tile (i.e., the current tile x and x+1), C and D involve shift variables k (i.e., x+k and x+k+1), E and F involve the displacement variable m of the second highest dimension (i.e., x+m and x+m+1), and G and H involve the sum of k and m (i.e., x+m+k and x+m+k+1). If the kernel has additional dimensions, the shift variables will continue to be combined in increasing sizes. Each combination will include two indicators, one is the combination itself, the other is the combination plus 1. Thus, for a 4D kernel, the pattern will continue with I and J (x+t and x+t+1), K and L (x+t+k and x+t+k+1), M and N (x+t+m and x+t+m), and O and P (x+t+m+k and x+t+m+k+1). These will correspond to binary values ​​1000 to 1111 (or 8 to 15), with the most significant bit corresponding to the test inequality for the fourth dimension (i.e., "test_t" or "delay_t"), and the remaining bits corresponding to the same three tests shown here.

[0193] Note that to compute the test inequality, the variable r is replaced by the row number of the location of the control pattern whose value is to be generated (here 0), the variable c is replaced by the column number of the location of the control pattern whose value is to be generated (here 7), and the variables h, w, and d are replaced by the dimensions of the height, width, and depth of the kernel (here 2, 2, and 2).

[0194] Fig. 13C An example 1300 of generating control modes for cores of different dimensions is shown according to one embodiment. In FIG. 13 , a 3×1 1D core 1310, a 3×3 2D core 1314, and a 2×2×2 3D core 1318 are shown, with the 3D core 1318 shown in an exploded view for clarity. For these cores, corresponding control modes 1312, 1316, and 1320 are shown, respectively. They can be based on the control modes described above with reference to FIG. Fig.111. The method described above is used to generate. For a 1D kernel, the identifier "A" in the control mode 1312 indicates the current tile X, and "B" indicates tile X+1. For control mode 1316, "A" indicates the current tile X, "B" indicates tile X+1, "C" indicates tile X+K, where K is the row size variable previously described, and "D" indicates tile X+K+1. For control mode 1320, "A" to "D" represent the same tiles as control mode 1316. In addition, "E" corresponds to tile X+M (the tile behind the current tile), where M is the layer tile size (i.e., the number of tiles in the layer). "F" corresponds to tile X+M+1, i.e., the tile to the right of the tile behind the current tile. "G" is tile X+M+K, corresponding to the tile behind and below the current tile. "H" corresponds to tile X+M+K+1, and corresponds to the tile behind, below, and to the right of the current tile.

[0195] In one embodiment, instead of using indicators such as letters shown in Figure 13 to store control patterns, or storing control patterns as bit sequences representing these letters, the fractal properties of control patterns are used to store in a more efficient manner. For each of control patterns 1312, 1316, and 1320, the identifier begins with "A" along the upper and right edges, then leads to an alternating pattern between "A" and "B", and further divides the pattern until the bottom value on the left is the identifier indicating the tile farthest from the current tile (as shown, the largest letter in alphabetical order). Due to this fractal property, each control pattern can be hierarchically divided into one or more levels according to the number of unique identifiers in the control pattern. The number of levels corresponds to the number of dimensions of the core used to generate the control pattern. Therefore, control pattern 1312 has a single level. Control pattern 1316 has two levels, and control pattern 1320 has 3 levels.

[0196] Each level comprises a square matrix containing a single binary value (i.e., bit). The size of each dimension of the square matrix of each level is equal to the size of the dimension of the previous level multiplied by the size of the largest dimension of the core. The smallest level has a square matrix with a size of each dimension equal to the size of the largest dimension of the core. Thus, for example, for control mode 1316, the largest level has a square matrix of bits with a size of 9×9, while the size of each dimension of the second smallest level is 3×3 (i.e., 9 divided by 3, i.e., the size of the core dimension).

[0197] Each bit in a level that is not the largest level matches a plurality of bits in a larger level. In particular, the larger level is equally divided into a plurality of regions equal to the number of bits in the next smallest level, and the bits in the next smallest level are matched with the bits in the regions of the larger level. The position of the matching bit in the bits of the next smallest level corresponds to the position of the region in the region group of the larger level. The combination of matching bits across multiple levels allows the bit sequence of each position in the control pattern to reference the identifier. Therefore, in the example of control pattern 1316, each bit of the 3×3 matrix matches the 3×3 portion of the 9×9 larger matrix. The combination of the bits from the 9×9 matrix and the matching bits from the 3×3 matrix forms a two-bit value that can be used to reference one of the four unique indicators of the control pattern 1316. In the example of control pattern 1320, since there are 8 unique identifiers, there are three levels, the largest level is an 8×8 matrix, the second largest level is a 4×4 matrix, and the smallest level is a 2×2 matrix. In control pattern 1316, three bits can be used to reference the correct identifier (i.e., letters "A" to "H" as shown) among the eight unique identifiers. These positions were previously FIG. 13A to FIG. 13B The results of the examples are shown in the table.

[0198] This method allows the control pattern to be stored efficiently. To extract the correct bit sequence from each level, the multiplication and addition unit 140 can determine the position of the indicator in the control pattern, and then use it to determine the correct position in the largest level, and the matching positions in the remaining levels to determine the bit sequence corresponding to the indicator.

[0199] Generate output using flattened input, expanded kernel, and control mode

[0200] Fig.14 is a flowchart illustrating a method for generating a convolution output using a flattened input, an expanded kernel, and a control mode according to one embodiment. Although the flowchart shown may show an order of operations, the operations shown may be performed in any order and may have a greater or lesser number of operations. In one embodiment, Fig.14 The operations shown may be performed by a multiplication-addition unit 140 (also referred to as a multiplication-accumulation unit).

[0201] The multiplication-addition unit 140 accesses 1410 a flattened input stream from the buffer, the flattened input stream comprising a set of parallel vectors, each vector representing a set of input values ​​of a unique kernel-sized tile to be convolved by the kernel to generate an output activation input tensor. The flattened input stream may be the flattened input stream 124 and may be generated by the input stream generator 122 as described above.

[0202] The multiplication-addition unit 140 receives 1420 an extended kernel generated by rotating and permuting the values ​​of the kernel, the extended kernel having vectors, each vector corresponding to the output value position of a tile of the kernel size of the output activation. This may be the extended kernel 128, and may be generated by the extended kernel generator 126 as described above.

[0203] The multiply-add unit 140 receives 1430 a control pattern including a set of vectors, where each vector corresponds to an output value location of a tile of the kernel size of the output activation, and each vector includes a delay value indicating a parallel vector of a flattened input stream for accessing input values ​​for convolution. The control pattern may be the control pattern 136 and may be generated using the control pattern generator 134 as described above.

[0204] For each output value position of each kernel-sized tile of the output activation, the multiplication-addition unit 140 generates 1440 a dot product between a first vector including values ​​of the flattened input stream selected by the delay values ​​of the corresponding vector of the control pattern and a second vector corresponding to the vector in the extended kernel corresponding to the output value position. The multiplication-addition unit 140 repeats this operation for all output values.

[0205] In one embodiment, the input tensor has multiple channels and the kernel has multiple filters. Each channel of the input tensor is convolved with one or more filters of the kernel to generate an output activation having multiple output features. FIG. 15A to FIG. 15C , 16A to Fig.16F and 17A to Fig. 17B Additional examples for generating output values ​​by the multiplication-addition unit 140 are described.

[0206] Fig.15A A first portion of an example 1500 of generating output activations using flattened inputs, expanded kernels, and control patterns in the case of a one-dimensional kernel is shown, according to one embodiment. In one embodiment, the process described herein may be performed by the multiply-add unit 140 .

[0207] At 1501A, the multiplication-addition unit 140 processes the first tile #1 of the output activation (same tile position as in the input tensor). This is the tile currently being processed. At 1502A, the multiplication-addition unit 140 further processes output position 1 of the 3 output positions of the first tile (where 3 is the number of values ​​in each tile). In the processing of the first output position, at 1511A, the multiplication-addition unit 140 selects a value from the flattened input 1522 according to the tile position (e.g., delay amount) as indicated by the vector in the control pattern 1524 for that output position. For position 1, this vector is the rightmost column in the control pattern 1524. Therefore, as shown, the control pattern vector 1528A for position 1 in the output tile is all "A" indicating the current tile X, or tile 1 from which the value is selected. The value in the flattened input 1522 may be selected by referencing a memory address, an index value, or may be selected by delaying the clock cycle in which the multiply-add unit 140 reads from the flattened input 1522. Here, since the value of the control mode vector 1528A indicates "A", the selected input value 1530A is selected from tile 1 of the flattened input 1522 and is shown as [0, 1, 2].

[0208] At 1512A, the multiplication-addition unit 140 performs a multiplication-accumulation, i.e., a dot product, between the selection value 1530A and the extended kernel vector 1532A corresponding to position 1 in the output tile. For position 1, the extended kernel vector 1532A is the top row of the extended kernel 1526. The dot product operation results in the calculation a0+b1+c2 as shown, which generates an output 1534A at position 1, as shown.

[0209] The process Fig. 15B , which shows a second portion of an example of using flattened input, expanded kernel, and control mode to generate output activations in the case of a one-dimensional kernel according to one embodiment.

[0210] At 1503A, the second of the three output positions of the current tile is processed. At 1513A, similar to 1511A, a control mode vector 1536A corresponding to position 2 of the tile is selected. In this case, the vector is the middle column of the control mode 1524. The control mode vector 1536 is used to select a corresponding value from the flattened input 1522. For each identifier in the control mode vector 1536, the tile indicated by the identifier is selected from the flattened input 1522, and a value is selected from the flattened input 1522 at the position corresponding to the same position of the indicator in the control mode vector 1536 at the tile. Therefore, the first (top) position in the control mode vector 1536A has an identifier "A", which corresponds to tile X (as indicated in the legend) or tile 1 in the flattened input 1522 in the current example. Therefore, the value in tile 1 of the flattened input 1522 (which is the value "0") is selected from the same first position. The same operation is performed for the second position in the control mode vector 1536A, resulting in the selection of "1" from the second position of the first tile of the flattened input 1522. For the third and final position of the control mode vector 1536A, the identifier is now "B", which indicates (as indicated in the legend) that the tile to be referenced is tile x+1, or tile 2 in the current example. Therefore, the multiply-add unit 140 selects the same value in the third position from tile 2, or "5". Therefore, as shown, the selected input value 1538A is [0, 1, 5].

[0211] At 1514A, a multiply-accumulate operation is performed similar to the operation in 1512A. However, instead of selecting the first row from the extended kernel 1526, the row corresponding to position 2 of the output is selected, i.e., the vector. In the current example, this happens to be the second row of the extended kernel 1526. Therefore, the extended kernel vector 1524A at position 2 is [c, a, b]. The dot product between this extended kernel vector 1524A and the selected input value 1538A is c0+a1+b5, or the output 1544A at position 2 of the output tile.

[0212] The process Fig. 15C , which shows a third portion of an example of generating output activations using flattened inputs, expanded kernels, and control patterns in the case of a one-dimensional kernel according to one embodiment.

[0213] At 1504A, the process outputs the final position of the tile, 3. The process described here is similar to that of 1503A and 1502A, but all selections are moved appropriately for the third position.

[0214] Thus, at 1515A, the selected control mode vector 1546A now corresponds to position 3 of the output tile, and is therefore the leftmost column of the control mode 1524. This control mode vector 1546A has one indicator for tile 1 and two indicators for tile 2. Thus, the selected input values ​​1548A include "0" in tile 1 and "4" and "5" in tile 2.

[0215] At 1516A, a multiply-accumulate operation is performed on the selected input value 1548A and the extended kernel vector 1552 at position 3, which corresponds to row 3 of the extended kernel 1526 in the current example. The output of this dot product is output 1554A at position 3.

[0216] At 1505, the process is repeated for all tiles in the input to generate the same number of tiles for output. Thus, in the flattened input 1522 of the current example, for tile 2, the operations of 1501A are repeated from 1501B for tile 2 to 1501L for tile 12. If the flattened input 1522 includes more tiles, the process shown in 1501A is repeated until all tiles are processed.

[0217] Fig.16A The first portion of an example 1600 of generating output activations using flattened inputs, expanded kernels, and control patterns in the case of a two-dimensional kernel is shown, according to one embodiment. FIG. 15A to FIG. 15C In contrast, the process described herein is for a two-dimensional kernel rather than a one-dimensional kernel. In one embodiment, the processing described herein is performed by multiply-add unit 140 .

[0218] At 1601A, the first tile is processed by the multiplication-addition unit 140. Since each tile has nine positions, each position is processed to generate an output value for that position. At 1602A, output position 1 is processed. As with the one-dimensional kernel example, a control mode vector 1628A is selected from the control mode 1624 based on the output position value. Since the output position is 1, the column marked with the output position 1 of the control mode 1624 is selected as the control mode vector 1628A. The control mode vector 1628A has "A" as all of its identifiers. Therefore, the control mode vector 1628A indicates that the selected input value 1630A should all be selected from tile 1 of the flattened input 1622 (because the current tile X is tile 1).

[0219] The process Fig. 16B Continuing at, it shows a second part of an example of using flattened input, expanded kernel, and control mode to generate output activations in the case of a two-dimensional kernel according to one embodiment.

[0220] At 1612A, according to Fig.16A The output position labels of the extended kernel 1626 shown in FIG. 1 are multiplied and accumulated on the selected input value 1630A and the extended kernel vector 1632A selected from the extended kernel 1626. In this case, output position 1 corresponds to the top row of the extended kernel 1626. The top row is selected for the multiplication-accumulation operation. Therefore, the top row forms the extended kernel vector 1632A. The multiplication-addition unit 140 performs a dot product between the selected input value 1630A and the extended kernel vector 1632A to generate an output 1634A at position 1.

[0221] At 1603A, output position 2 is processed. Output position 2 corresponds to the second column from the right of control pattern 1624 (as marked). Note that the order of columns here does not have to follow the order shown. Instead, the order of columns can be in any order as long as the correct column corresponding to the correct output position is selected. In addition, control pattern 1624 does not need to be sorted by column, but can also be sorted by row. As long as the correct row corresponding to the correct output position is referenced, the output will be the same. Finally, the order of identifiers in the control pattern can also be reversed. As long as the identifiers are referenced in the correct order when using the identifiers to select input values, the output will be the same. This same concept also applies to extended core 1626, as its ordering can be different from the ordering described here.

[0222] As noted in the legend, identifier “A” in control pattern 1624 refers to tile x of flattened input 1622, where X is the current tile, and identifier “B” refers to tile x+1. Control pattern vector 1636A includes identifiers “A” and “B.” For each position in control pattern vector 1636A, multiplication-addition unit 140 determines which tile the identifier is referencing and selects an input value from that referenced tile in flattened input 1622 at the same input position of the vector of tiles in flattened input 1622 as the position of the identifier in control pattern vector 1636A. Thus, for example, position 6 in control pattern vector 1636A has identifier “B,” which indicates that a value should be selected from tile x+1, which is tile 2 of flattened input 1622 (current tile #1+1=tile 2). The multiply-add unit 140 selects the value at input position 6 from the tile 2 vector of the flattened input 1622 that corresponds to position 6 in the control mode vector 1636A and places it at position 6 of the selected input value 1638A. The remaining values ​​are selected in a similar manner.

[0223] The process Fig. 16C , which shows a third portion of an example of using flattened input, expanded kernel, and control mode to generate output activations in the case of a two-dimensional kernel according to one embodiment.

[0224] At 1614A, the selected input value 1638A generated in the previous operation is combined with the extended kernel vector 1642A in the dot product to generate an output 1644A at position 2. Here, the extended kernel vector 1642A is again selected based on the output position number, in this case 2. Therefore, the second row of the extended kernel 1626 is selected as the extended kernel vector 1642A.

[0225] The multiplication and addition unit 140 performs the above processing on the remaining output positions of the tile. As an example, at 1606A, the processing is performed on output position 5. Here, at 1615A, as Fig.16A As shown, the control mode vector 1646A is selected from the column of output position 5. Here, the control mode vector 1646A includes two new identifiers "C" and "D" corresponding to tiles X+K and X+K+1, respectively, where K represents the number of tiles in the row of the input. Therefore, as shown, position 9 in the control mode vector 1646A is selected from position 9 of tile 6 in the flattened input stream. This will generate a selected input value 1648A for this output position.

[0226] The process Fig.16D , which shows a fourth portion of an example of using flattened input, expanded kernel, and control mode to generate output activations in the case of a two-dimensional kernel according to one embodiment.

[0227] At 1615A, a multiply-accumulate operation is performed. The multiply-add unit 140 performs a dot product between the selected input value 1648A and the extended kernel vector 1652A selected for position 5. In the example shown, this is row 5 of the extended kernel 1626. The dot product result is output 1654A for position 5. The process is repeated at 1616 until all outputs for the current tile are generated. Furthermore, the process is repeated at 1617 until all tiles are processed.

[0228] Fig.16E An example of using a flattened input, an expanded kernel, and a control pattern to generate output activations with multiple channels in the case of a two-dimensional kernel is shown according to one embodiment.

[0229] As previously described, the input tensor 102 may have multiple channels, each channel being a separate matrix of input values. Figure 5C As described, multiple channels of input are flattened and "stacked" on top of each other. At 1618, according to Figure 4The method described in FIG. 6 flattens each of these input channels, and the values ​​of the flattened input channels 1656 are selected by the control mode 1624 to generate the selected input channel values ​​1658. Although a new matrix of selected input channel values ​​1658 is shown here, a new matrix may not actually be generated. Instead, the multiplication and addition unit 140 can dynamically select values ​​from the flattened input channels 1656. Therefore, the selected input channel values ​​1658 are shown here to conceptually illustrate the concept.

[0230] The process Fig.16F , which shows a second portion of an example of generating output activations through multiple channels using flattened inputs, expanded kernels, and control patterns in the case of a two-dimensional kernel according to one embodiment.

[0231] At 1618, the multiply-add unit 140 performs 1668 a dot product (i.e., a multiply-accumulate) using the selected input channel values ​​1658 and the kernel filters 1660A-1660N. Each input channel in the selected input channel values ​​1658 is convolved with a different sub-filter kernel 1666, which is itself a kernel. Each sub-filter kernel 1666 can have the same or different values. The convolution process can be similar to the above with respect to FIG. 16A to FIG. 16D The convolution output 1662A generated for a single kernel filter is summed 1670 to produce an output activation channel 1664A. This process is repeated for each of the remaining kernel filters 1660B-1660N, thereby generating multiple output activations 1664B-1664N. These output activations 1664A-1664N can be used as inputs to other convolutional layers in the neural network.

[0232] Fig.17A The first part of an example of generating output activations using flattened inputs, expanded kernels, and control patterns in the case of a three-dimensional kernel according to one embodiment is shown. In one embodiment, the process described herein may be performed by the multiply-add unit 140 .

[0233] At 1701, the multiplication-addition unit 140 is processing the input tile X. At 1702, the multiplication-addition unit 140 processes the output position 8 of the tile X. At 1711, the multiplication-addition unit 140 selects a value from the flattened input 1722 using the control pattern vector 1724 for the selected position 8. As with the above processing for the one-dimensional kernel and the two-dimensional kernel, the vector of control patterns corresponding to the output position (in this case, position 8) is selected as the control pattern vector 1724. The control pattern vector 1724 has eight different identifiers, from "A" to "H". As shown in FIG. Fig.17AAs shown in the legend in , each identifier indicates a different tile of the flattened input 1722. For example, identifier "H" corresponds to tile X+M+K+1, where M is the number of tiles in each layer and K is the number of tiles in each row of the input. For each identifier, multiplication-addition unit 140 selects a value at a position corresponding to the position of the identifier of the selected input value 1726 from the indicated tile of the flattened input 1722.

[0234] The process Fig. 17B Continue, Fig. 17B A second portion of an example of using flattened input, expanded kernel, and control patterns to generate output activations in the case of a three-dimensional kernel, according to one embodiment, is shown.

[0235] At 1712, the multiplication-addition unit 140 combines the selected input value 1728 of position 8 with the extended kernel vector 1730 of position 8 into a dot product to generate an output 1732 of position 8 of the current tile. The selected extended kernel vector 1730 corresponds to the position of the output.

[0236] At 1713 , the multiplication-addition unit 140 repeats the above process for all output positions for each tile in the input to generate output activations.

[0237] Exemplary Processor Components

[0238] Fig.18A A hardware diagram of an exemplary component for generating an extended core according to one embodiment is shown. In some embodiments, data values ​​1810 corresponding to an array (e.g., a core of weights) are stored in a data storage device circuit 1801 (e.g., a memory circuit). In some embodiments, the data values ​​corresponding to the array are stored as a sequence (e.g., stored as a single column). In this example, the data value 1810 is coupled to a shifter circuit 1820 on one or more inputs. The shifter circuit can receive the array as a sequence of data values. Each data value can include a digital data value of one or more bytes, for example, which can be an integer, a floating point number, or other type of data value. Therefore, each data value can be coupled to a shifter circuit 1802 on multiple conductors (e.g., wire interconnects on an integrated circuit), where, for example, a group of conductors can carry a data value, and multiple groups of conductors can carry a series of data values. In one embodiment, the data value is a weight, and the input data value is an activation value of a neural network.

[0239] Conductors carrying data values ​​may be received at the input of the shifter circuit 1802. For example, the input of the shifter circuit 1802 may receive conductors for carrying some or all of the sequence. As described above, data values ​​corresponding to the array may be received as a sequence. In one embodiment, the sequence includes all data values ​​of the corresponding array received from the data memory 1810 in parallel. However, in other embodiments, the sequence may include a portion of the data values ​​of the corresponding array (e.g., receiving and arranging a column at a time or receiving and arranging a row at a time). The shifter circuit 1802 may couple data values ​​from one or more inputs to one or more outputs in different orders to generate an arranged sequence. In some embodiments, each of the multiple outputs carries a sequence that may be coupled to the multiplier 1803. In other embodiments, the data values ​​of the sequence received at the first input are shifted to different positions and coupled to one or more outputs. In an example embodiment shown below, a sequence is received at the first input, the data values ​​are shifted to different positions, and the arranged sequence is sequentially generated at the second output.

[0240] In some embodiments, as described above, the arrangement generated by the shifter circuit 1802 can correspond to an extended core. Therefore, in some embodiments, only one sequence can be stored in the memory, rather than multiple arranged sequences being stored in the memory, thereby advantageously reducing the use of memory, for example. For example, the sequence received from the data storage device 1801 may include multiple subsequences based on the columns or rows of the array. The shifter circuit 1802 can arrange the sequence as follows. The data value can be coupled to one or more outputs of the shifter circuit 1802 from one or more inputs of the shifter circuit 1802 in the same order as the sequence. Therefore, the input sequence can include one of the sequences generated by the output. In addition, for multiple arrangements of subsequences, wherein the position of each subsequence is moved within the sequence, and for multiple arrangements of data values ​​in each subsequence, wherein the data value is moved within each subsequence, the data value is coupled to one or more outputs from one or more inputs in different orders. Examples of such arrangements for different arrays are shown above. In one embodiment, a data value is received on a first input of the shifter circuit 1802, and the data value is coupled from the first input to the first output to generate an input sequence on the first output. In addition, the data values ​​may be coupled from the first input to the plurality of second outputs in different orders to produce a plurality of permutation sequences on the plurality of second outputs. Fig.18B An example implementation of this method is shown in . In another embodiment, data values ​​are received at one or more inputs of the shifter circuit 1802 and the data values ​​are shifted to different positions between the inputs and one or more outputs to produce an input sequence and a plurality of permutation sequences.

[0241] The multiplier circuit 1803 may be coupled to one or more outputs of the shifter circuit 1802. Thus, the sequence generated on one or more outputs of the shifter circuit 1802 may be loaded into a plurality of register circuits in the multiplier 1803. The multiplier 1803 may also receive an input data value 1820. The input data value 1820 may be loaded into a plurality of second registers. Thus, a plurality of multiplication results 1830 may be generated. As described above, for example, the extended kernel may generate a plurality of convolution results, which are generated in parallel. In one embodiment, the data value corresponds to an M×N array (e.g., the kernel is applied to the input data value 1820). The shifter circuit 1802 may receive a sequence of length M*N from the data storage 1810 and generate M*N different sequences, each of which is of length M*N. The sequence may be coupled from the shifter circuit 1802 and loaded into the multiplier circuit 1803 (e.g., in parallel).

[0242] Fig.18B A hardware diagram is shown of an exemplary shifter circuit used in an exemplary component for generating an extended core according to one embodiment.

[0243] In this example, the input sequence SEQ1 corresponds to an M×N array, for example, where M and N are integers and one or more of M and N is greater than 1. Therefore, the sequence SEQ1 includes M*N data values ​​D0 to D(M*N). The sequence SEQ1 can be coupled from the input to a first output (e.g., stream S(1)) including unarranged M*N data values. In addition, the sequence SEQ1 can be arranged along M*N-1 different paths into M*N-1 different sequences SEQ2-SEQM*N, which are provided on M*N-1 outputs (e.g., streams S(2)-S(M*N), each of which includes M*N arranged data values, such as D0 to D(M*N).

[0244] More specifically, in this example, a sequence SEQ1 including a plurality of data values ​​D0 to D(M*N) is received on a first input stream S(1). Each data value in the sequence may, for example, include one or more bytes carried on a plurality of conductors. Here, the data values ​​are coupled from a first input (e.g., stream S(1)) to a plurality of different outputs (e.g., streams S(1) to S(M*N)) in different orders along different conductive paths to produce an original sequence SEQ1 and a plurality of permuted sequences SEQ2 to SEQM*N on a plurality of different outputs (e.g., streams S(1) to S(M*N)). For example, each data value D0 to D(M*N) in the input sequence may include one or more bytes carried on a plurality of conductors. The conductors carrying different data values ​​are reordered differently between the input (e.g., stream S(1)) and each of a plurality of outputs (e.g., streams S(2) to S(M*N)) to produce a different permuted sequence on each of the plurality of outputs. More specifically, in this example, SEQ1 on input S(1) is coupled to a plurality of rotators 1860 to 1862. For example, rotators 1860 to 1862 can reorder the position of data values ​​in each sequence by changing the position of conductors carrying data values. Two example rotations are shown at 1870 and 1871. In this example, after the rotation, each permutation sequence can be coupled to an input of a multiplexer circuit. An embodiment of the present disclosure may include an integrated circuit that includes a plurality of streams S. For example, a permutation sequence generated from SEQ1 via rotators 1860 to 1862 can be coupled to adjacent streams S(2) to S(M*N) using multiplexers 1851 to 1853. For example, each of the permutation conductors of different permutation sequences SEQ2 to SEQM*N carrying data values ​​D0 to D(M*N) can be coupled to an input of each multiplexer. A second input of each multiplexer is coupled to one of the streams S(2) to S(M*N). Thus, the output of each multiplexer couples one of the permutation sequences to each stream. Additionally, the wires used to input the original sequence SEQ1 carrying data values ​​D0 to D(M*N) are coupled to the first output (e.g., stream S(1)) in the same order to produce the original sequence carrying stream S(1) at the output. For example, streams S(1) to S(M*N), carrying SEQ1 to SEQM*N, respectively, can be coupled to a multiplier circuit. Thus, in this example, all permutations of sequence SEQ1 can advantageously be generated in parallel on the same cycle and loaded into the multipliers in parallel.

[0245] Fig.19A hardware diagram of an exemplary component for generating a control pattern according to one embodiment is shown. Features and advantages of some embodiments may include a control pattern generator circuit 1910 that can be integrated with, for example, a multiplier array circuit 1920 on the same semiconductor surface. In one embodiment, the control pattern generator 1910 is configured to receive (e.g., via software) the dimensions of the core, such as height (H) and width (W). For example, for each multiplier unit in the multiplier units 1921 to 1926 in the multiplier array 1920, the control pattern generator 1910 can generate two bits. One bit can select between a K-delay input or a non-K-delay input, while the other bit can select between a unit delay and a non-unit delay input (e.g., 1 delay or no delay). Each multiplier unit may include, for example, a storage unit (e.g., a register) for storing two mode control bits. The two bits together can select one of the following four data values ​​using a selection circuit: X, X+1, X+K, X+K+1, where X is the current tile and K is a variable indicating the tile size width of the input tensor, as described above. For kernels with additional dimensions, the control pattern generator 1910 generates an additional bit for each additional dimension, where the bit selects between an additional delay by shifting the variable of the additional dimension and no additional delay. Note that although the term delay is used here, in some cases, the access value of the flattened input stream is accessed not by delay but by a relative or absolute addressing system that can use the delay value as an index value into memory.

[0246] In this example, the control pattern generator 1910 includes a K delay logic circuit 1911 and a unit delay logic circuit 1912. The K delay logic circuit 1911 can, for example, receive a single dimension of the core and generate an H*H bit value for selecting between a K-delayed input and a non-K-delayed input. In this example, the K delay logic circuit 1911 receives the height H of the core and generates an H corresponding to the H×H triangular array of the control pattern. 2Bit value. Although H is used as an input to the K delay logic circuit 1911 in this example, other embodiments may use another dimension, such as width W, as an input to the K delay logic circuit. In addition, the unit delay logic circuit 1912 can receive two dimensions, such as a height H and a width W. The unit delay logic circuit 1912 generates an H*W bit value for selecting between inputs (e.g., no delay or K delay) such as a unit delay (+1) or no unit delay (e.g., 0). In this example, the unit delay logic circuit 1912 receives the height H of the core and the width W of the core, and generates an H*W bit value corresponding to an H*W triangular array (e.g., array 2220 in Figure 22). The bit value pairs for selecting no delay / K delay and unit delay / no unit delay are coupled to a plurality of multiplier units 1921 to 1926 and stored in, for example, a storage circuit 1930 such as a register in each multiplier unit. The selection is shown here with a multiplexer reference numeral 1931, but many different selection circuits are known to those skilled in the art. In this example, one bit from K delay logic circuit 1911 and one bit from unit delay logic circuit 1912 may be stored in a storage cell of each of multiplier cells 1921 to 1926 to, for example, select one of an undelayed, unit delayed, K delayed, or K+1 delayed input value. A pattern similar to pattern 2220 shown in FIG. 22 may correspond to a first bit value in storage circuit 1930 across multiplier array 1920, and a pattern similar to pattern 2210 in FIG. 22 may correspond to a second bit value in storage circuit 1930 repeated across multiplier arrays 1920 to 200 to control input selection across the multiplier array.

[0247] Fig. 20A hardware diagram of an exemplary component for performing a multiplication-addition operation to generate an output activation according to one embodiment is shown. In this example, a value from an extended core (M*N) is loaded into a first input register of a plurality of multiplication-accumulator circuits, and an input value aij is loaded into a second input register of a plurality of multiplication-accumulator circuits. For example, the multiplication-accumulator circuit may include a plurality of multipliers 2001, 2011, 2021, 2031, 2041, and 2051, wherein different values ​​wij may be loaded from the extended core to the plurality of multiplier circuits. Each multiplier may also load an input value from one of a plurality of tiles of the flattened input stream. For example, a11 corresponds to a value in the a11 position of the first tile (i.e., the first row and first column in the tile), which may be coupled to an input of a selection circuit 2006 (e.g., a multiplexer or "Mux"). The a11 input may also be coupled to a delay circuit 2003 to produce a delayed version of a11 (e.g., one cycle delayed) at a second input of a selection circuit 2006. The delayed version selects a cell from the current tile to form a tile. Additionally, a11 may be coupled to K delay circuits 2004 (e.g., a buffer or a first-in-first-out data storage unit ("first in first out" FIFO)) to produce, for example, a version of a11 delayed by K cycles at another input of the selection circuit 2006. This corresponds to a tile in a row below the current tile. Additionally, the output of the K delay circuit 2004 may be received at the input of another delay circuit 2005 to produce a K+1 delayed version of a11 at yet another input of the selection circuit 2006. Similarly, for example, multipliers 2011 to 2021 receive values ​​w12, ..., w1(M*N) corresponding to the first row of the extended array and input values ​​a11 ... aMN. Similarly, other rows of the extended array can be multiplied by input values ​​a11 ... aMN until the last row of the extended array w(M*N)1, ..., w(M*N)(M*N), for example, where the row includes the arranged values ​​as described above. Multipliers 2011, 2021, 2031, 2041 and 2051 include delays 2013, 2023, 2033, 2043 and 2053, K delays 2014, 2024, 2034, 2044 and 2054, delays 2015, 2025, 2035, 2045 and 2055, and selection circuits 2016, 2026, 2036, 2046 and 2056, respectively.

[0248] For example, a control mode can be used to select between various versions of a11 so that input values ​​from the appropriate tile are multiplied by weights stored in a particular multiplier. In this example, the input values ​​are multiplied by the sequence in the first row of the extended kernel, so a11w11…aMNw1(M*N) has a control mode value corresponding to A (the current tile undergoing convolution). However, in this example, calculating the convolution over the entire tile (including convolutions involving adjacent tiles) may include waiting for input values ​​in adjacent tiles to arrive. Therefore, calculating the convolution for a given "current tile" may include waiting for K+1 cycles for adjacent tile values ​​to be available. As described above, other control mode values ​​may be used depending on the dimensionality of the kernel. To calculate the convolution of the current tile, for example, the system may store input values ​​and wait for input values ​​from other tiles to be available for calculation. Thus, for example, the convolution of a particular tile may be calculated using K+1 delayed input values ​​for a particular tile, K delayed input values ​​for values ​​in an adjacent column tile, 1 delayed value for an input value in an adjacent row tile, and an undelayed value for an input value in a tile adjacent to an adjacent row and adjacent column tile. Although the present specification relates to current values ​​(e.g., for a current tile or undelayed values), unit (or 1) delay values, K delay values, and K+1 delay values, it should be understood that such delays may be relative to one another (e.g., at the input of a selection circuit). Various other delays may exist in the signal path, but are omitted here for purposes of illustration. In addition, if the core is to have additional dimensions, additional delay values, such as M delay values, etc., and corresponding circuitry will be included.

[0249] An array of input values ​​a11 to aMN (e.g., a11 to a33) can be loaded into an array of multiplier circuits 2001, 2011, ..., 2021, each of which is also loaded with different weights w11, w12, ..., w1 (M*N) (e.g., w33). As described above, for example, each multiplier receives an input selected from a different control mode input of a control mode array having the same dimensions as the extended core providing the weights. The output of each multiplier 2001, 2011, ..., 2021 can be coupled to one or more adder circuits (e.g., adders 2007, 2017, 2027) to sum the products and output the convolution result R1. Similarly, each row of the extended core can be loaded into multiple multipliers and multiplied by a vector of input values ​​that is masked by the corresponding row from the control mode array. Here, the last row of the extended kernel, including values ​​w(M*N)1, w(M*N)2, ..., w(M*N)(M*N), is loaded into the columns of multipliers 2031, 2041, and 2051 and combined with input values ​​a11, a12, ..., aMN, including values ​​such as positions (M*N, 1), (M*N, 2), ..., (M*N, M*N), controlled by the last row in the control pattern array. The outputs of the last column of the multipliers are added together in adders 2037, 2047, and 2057 to produce, for example, the M*Nth convolution result R(M*N). In an equivalent alternative embodiment, a delay can be coupled to the output of the multiplier.

[0250] Fig.21 An exemplary component layout 2100 for computing output activations in a machine learning processor according to one embodiment is shown. The illustrated layout represents one possible layout of components (logical components and / or physical components) in a machine learning processor (or other processor) for performing the operations described herein. The layout 2100 includes an ICU 2110, a plurality of MEM tiles 2115A-2115N, a plurality of SXM tiles 2120A-2120N, a plurality of NIM tiles 2125A-2125N, and an array of super cells 2130A-A to 2130N-N. In the following description, these elements may be generally referred to by omitting their alphabetic suffixes.

[0251] ICU 2110 issues instructions to NIM tile 2125. These instructions instruct NIM tile 2125 to configure super cells 2130 with the correct control mode values ​​in order to perform the convolution operations described herein. Super cells 2130 are arranged in a square grid. Each super cell 2130 has a size that indicates how many values ​​it can process in parallel. Super cells 2130 can all have the same size. For example, each super cell can be a 16×16 grid and can process 256 values.

[0252] In one embodiment, the input channels are divided vertically between the super-cells 2130, as shown at 2145, so that each input channel is processed by one or more layers (the layers indicated by the first letter in the naming scheme) of the super-cell 2130 after flattening. Since the input layer may be smaller than the size of the super-cell 2130, or when the vertical dimension of the input channel does not divide evenly into the vertical dimension of the super-cell 2130, one or more layers may include only a portion of the layer. In the same embodiment, the output features (i.e., the different output "channels") are divided horizontally between the super-cells 2130, as shown at 2155. Thus, one or more columns (or portions) of the super-cell 2130 (indicated by the second letter in the naming scheme) are used to process each output feature.

[0253] The super unit 2130 itself can be similar to the multiply-accumulate unit described above. After performing a convolution operation on the input value 2140, the super unit sends its output to the super unit 2130 above it. This allows the convolutions of different input layers to be added together to generate output features 2150 from multiple input layers. Using the above method (e.g., in Figures 16E to 16F ), if there are multiple channels for output, multiple sets of output features 2150 are generated.

[0254] On a first clock cycle, the ICU 2110 instructs the first NIM tile 2125A to configure one or more super cells 2130 associated with the NIM tile 2125A. In the example shown, the super cells 2130 on the same layer as the NIM tile 2125A are configured by the NIM tile 2125A. These are super cells 2130A-* (where * is any letter from A to N). On a second clock cycle, the NIM tile 2125A uses the super cells 2130A-* (where * is any letter from A to N). FIG. 13A to FIG. 13BThe method described in configures the correct control mode value for the super cell 2130A-A. For the first super cell 2130A-A, there is no offset for the row number and column number when the control mode value is generated. However, for any other super cell 2130 other than the super cell 2130A-A, the row number and column number used to generate the control mode value have an additional offset. For the column number, the offset is equal to the number of super cells 2130 to the left ("west") of the super cell 2130 being configured multiplied by the cell width of the super cell 2130. For the row number, the offset is equal to the number of super cells 2130 below ("south") the super cell 2130 being configured multiplied by the cell height of the super cell 2130. The padding offset value can be added based on how much of the super cell has been used from processing the previous input channel or output channel. The reason for using this offset is because the tile, control mode, and extended kernel sizes may not be multiples of the size of the super cell, so different tiles may span different super cells 2130. This means that the control mode values ​​may also span different super-units 2130, so the calculation of each control mode value needs to be offset by the location of the super-unit 2130 to generate the correct control mode value corresponding to one or more tiles or parts of tiles to be processed by the super-unit 2130.

[0255] In parallel (on the same clock cycle) with the super-cell 2130A-A being configured with the control mode value, the NIM block 2125A transmits the instruction to the subsequent NIM block 2125B (above or "north" of the current NIM tile 2125) along with the instruction to configure its super-cell 2130B-*. The instruction includes the offset value previously indicated, so that the NIM tile 2125B configures the super-cell 2130B-* with the correct control mode value accounting for the offset. This propagation of instructions is repeated until a number of super-cells 2130 equal to the size of the input are configured by the NIM tile 2125.

[0256] During each configuration of the super-cell 2130 in the super-channel, the NIM tile 2125 may configure the super-cell 2130 only for a specific channel of output (i.e., the output channel). The channel may be specified in the instruction from the ICU 2110. In order to configure the super-cell 2130 for additional channels of output, the ICU 2110 may issue additional instructions to the NIM tile 2125 at each clock cycle to configure the super-cell 2130 for additional channels of output. These instructions include the correct offset values ​​so that the super-cell 2130 processing the output layer has the correct row and column position values, so that the correct control mode value can be generated.

[0257] At each clock cycle, instructions from the ICU 2110 are propagated from one NIM tile 2125 "north" to the subsequent NIM tile 2125, while the ICU 2110 continues to issue instructions to the super-cell in the "east" direction. Thus, over multiple clock cycles, the super-cell 2130 is configured into a diagonal "wave". In this way, the configuration of the entire set of input channels and output channels is configured after multiple clock cycles. Once this process is completed, the input values ​​2140 will be received by the system described herein and processed by the super-cell 2130 tile by tile in a process similar to the multiply-accumulate operation described above to generate output features 2150.

[0258] Fig.22A An example machine learning processor according to one embodiment is shown. Features and advantages of the present disclosure include processor circuits capable of performing convolution operations at very high rates. In this example, a server computer 2201 (or host) may include configuration software 2210 that may receive one or more cores 2213 for performing convolution. Processor circuit 2200 may be a machine learning processor, which refers to a circuit that receives input data (e.g., activations such as images) and applies a core (e.g., neural network weights) to the input data as part of, for example, a machine learning (also known as artificial intelligence) algorithm. In this example, configuration software 2210 (sometimes referred to as a compiler) receives neural network weights 2213. In this example, an extended core or control mode (or both) may be generated in software. For example, weights 2213 may be analyzed by core generator 2212 and control mode generator 2211 to generate control mode 2225 and extended core 2226, which may be stored, for example, in a data storage device 2220 in processor circuit 2200, such as a memory or register circuit. For example, input value 2280 can be received in input / output (I / O) circuit 2222 and coupled to multiplication accumulator circuit 2276 via data bus 2224. Extension core 2226 and control mode 2225 can also be coupled to multiplication accumulator circuit 2221. Multiplication accumulator circuit 2221 can, for example, generate a convolution result as described above. In some embodiments, intermediate convolution result 2227 is stored in data storage device 2220 and applied to new activation input 2280. In other embodiments, for example, convolution result 2290 can be provided as output. Alternatively, the convolution result can be processed by other circuits 2223, such as a vector processor (not shown).

[0259] Fig.22AAn example machine learning processor according to another embodiment is shown. In this example, the core is analyzed by configuration software, but the extended core and control mode can be generated in hardware. For example, one or more cores (e.g., neural network weights) 2262 are received in configuration software (e.g., compiler) 2260 and analyzed by core analyzer component 2261 to generate configuration information for generating an extended core including a value K, which can be derived from the size of the core and other information (e.g., the size of the input to be processed). The server computer 2251 can send the configuration information and the value K to the processor circuit 2250. In this example, K can correspond to the number of tiles in each row of the input 2285, for example, where the size of the tile is the same as the size of the core. The configuration information can be used to configure the control mode generator 2277 and the shifter circuit 2278. The shifter circuit 2278 can be configured to arrange the core to generate an extended core from one or more cores 2275 stored in, for example, a data memory 2270.

[0260] In this embodiment, an extended core is generated in hardware. One or more cores 2275 can be stored in a data memory 2270. For example, the core can be stored in a data memory 2270 as a sequence of values, wherein the sequence can be accessed from the data memory 2270 and the sequence is coupled to a shift circuit 2278 (e.g., simultaneously). As described above, various shift circuits can be used to arrange the input sequence and generate an extended core. An example circuit for arranging the sequence to generate an extended core is described in more detail below. For example, the value of the extended core can be loaded into a multiplication accumulator 2271. Extending the core in hardware can reduce the amount of data storage required, which may be advantageous in certain embodiments.

[0261] Embodiments of the present disclosure include configuring K delay circuits in series with the multiplication accumulator 2271. In this example, the input value 2285 can be coupled to a delay circuit 2279, which can be, for example, a programmable delay circuit such as a programmable buffer. In one embodiment, the delay 2279 can be configured to have a delay of, for example, K, which can vary depending on the size of the core and the number of tiles along the dimension of the input array. In one embodiment, for example, the delay 2279 can be a first-in, first-out (FIFO) circuit, where the output is a delayed version of the input, and the delay is set to K. The input value can also be directly coupled to the multiplication accumulator circuit 2271 without delay. Therefore, the multiplication accumulator circuit 2271 can receive, for example, a current input value and an input value delayed by K cycles.

[0262] The control pattern generator 2277 can generate control signals for implementing control patterns to combine various delayed or undelayed input data values ​​with specific weights as described above. For example, a particular pattern control signal can correspond to a path for applying a kernel to each input tile and a path for flattening tiles and kernels. For example, as described above, the output of the control pattern generator 2277 can be used to select various delayed versions of input values.

[0263] Thus, the multiplication accumulator circuit 2271 receives the current input value, the K delayed input values, the extended core, and a control signal for selecting a different input. As described above, the multiplication accumulator 2271 may include an additional unit delay circuit for delaying both the current input value and the K delayed input values, and the control signal from the control pattern generator 2277 may be used to select an appropriate input value to be multiplied with, for example, a different value in the extended core. The output of the multiplication accumulator circuit 2271 may be an intermediate convolution 2276, which may be stored in the data memory 2270 and convolved with a subsequent input 2285, or the convolution result may be output, for example, at 2295. Alternatively, the convolution result may be processed by other circuits 2273, such as a vector processor (not shown).

[0264] Exemplary Computing Machine Architecture

[0265] Fig.23 is a block diagram illustrating components of an example computing machine capable of reading instructions from a computer-readable medium and executing those instructions in a processor (or controller). The computers described herein may include Fig.23 The single computing machine, virtual machine, including multiple nodes, i.e., multiple Fig.23 The computer described herein may be used to perform the functions described by any of the elements described in the previous figures.

[0266] By way of example, Fig.23 A schematic diagram of a computing machine in the example form of a computer system 2300 is shown within which instructions 2324 (e.g., software, program code, or machine code) may be stored in a computer-readable medium for causing the machine to perform any one or more of the processes discussed herein. In some embodiments, the computing machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate as a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0267] Fig.23The structure of the computing machine described in the figure may correspond to any software, hardware or combination of components shown in the figure above. Fig.23 Various hardware and software components are shown, but Figure 1 and 2 Each component described in the may include more or fewer elements.

[0268] By way of example, the computing machine may be a personal computer (PC), a tablet, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a smart phone, a network appliance, a network router, an Internet of Things (IoT) device, a switch or bridge, or any machine capable of executing instructions 2324 that specify actions to be performed by the machine. In addition, while only a single machine is shown, the term "machine" should also be taken to include any collection of machines that individually or collectively execute instructions 2324 to perform any one or more of the methodologies discussed herein.

[0269] The example computer system 2300 includes one or more processors (typically a processor 2302) (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application specific integrated circuits (ASIC), one or more radio-frequency integrated circuits (RFIC), or any combination thereof), a main memory 2304, and a static memory 2306, which are configured to communicate with each other via a bus 2308. The computer system 2300 may also include a graphics display unit 2310 (e.g., a plasma display panel (PDP), a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)). The computer system 2300 may also include an alphanumeric input device 2312 (e.g., a keyboard), a cursor control device 2314 (e.g., a mouse, trackball, joystick, motion sensor, or other pointing tool), a storage unit 2316, a signal generating device 2318 (e.g., a speaker), and a network interface device 2320, which are also configured to communicate via the bus 2308.

[0270] The storage unit 2316 includes a computer-readable medium 2322 on which are stored instructions 2324 that implement any one or more of the methods or functions described herein. The instructions 2324 may also reside in the main memory 2304 or in the processor 2302 (e.g., in a cache memory of the processor) in whole or in part during execution of the instructions by the computer system 2300, the main memory 2304, and the processor 2302, which also constitute computer-readable media. The instructions 2324 may be sent or received over the network 2326 via the network interface device 2320.

[0271] Although the computer-readable medium 2322 is shown as a single medium in the example implementation, the term "computer-readable medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) that can store instructions (e.g., instructions 2324). Computer-readable media can include any medium that can store instructions (e.g., instructions 2324) that are executed by a machine and cause the machine to perform any one or more of the methods disclosed herein. The computer-readable medium can include, but is not limited to, data repositories in the form of solid-state memory, optical media, and magnetic media. The computer-readable medium does not include transient media such as signals or carrier waves.

[0272] Other configuration considerations

[0273] Certain embodiments are described herein as, for example, including logic or multiple components, engines, modules, or mechanisms, as shown in the figure above. An engine can constitute a software module (e.g., code implemented on a computer-readable medium) or a hardware module. A hardware engine is a tangible unit that can perform certain operations and can be configured or arranged in a certain manner. In an example embodiment, one or more computer systems (e.g., independent client or server computer systems) or one or more hardware engines (e.g., a processor or a group of processors) of a computer system can be configured by software (e.g., an application or an application portion) as a hardware engine that operates to perform certain operations described herein.

[0274] In various embodiments, the hardware engine can be implemented mechanically or electronically. For example, the hardware engine may include dedicated circuits or logic that are permanently configured (e.g., as a dedicated processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) to perform specific operations. The hardware engine may also include programmable logic or circuits (e.g., as included in a general-purpose processor or another programmable processor) that are temporarily configured by software to perform certain operations. It will be understood that cost and time considerations may determine whether to implement the hardware engine mechanically, in a dedicated and permanently configured circuit, or in a temporarily configured circuit (e.g., configured by software).

[0275] The various operations of the example methods described herein may be at least partially performed by one or more processors (e.g., processor 202), which are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such a processor may constitute a processor-implemented engine that operates to perform one or more operations or functions. In some example embodiments, the engine referred to herein may include a processor-implemented engine.

[0276] The performance of certain operations can be distributed among one or more processors, not only residing in a single computer, but also deployed across multiple computers. In some example embodiments, one or more processors or processor-implemented modules can be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other example embodiments, one or more processors or processor-implemented modules can be distributed in multiple geographic locations.

[0277] By reading this disclosure, those skilled in the art will understand additional alternative structures and functional designs for similar systems or processes through the principles disclosed herein. Therefore, although specific embodiments and applications have been shown and described, it should be understood that the disclosed embodiments are not limited to the precise constructions and components disclosed herein. Various modifications, changes and variations that are obvious to those skilled in the art may be made to the arrangement, operation and details of the methods and equipment disclosed herein without departing from the spirit and scope defined by the appended claims.

[0278] Regarding the implementation methods including the above embodiments, the following technical solutions are also disclosed:

[0279] Solution 1. A method comprising:

[0280] Receives the input tensor to be convolved with the kernel;

[0281] Dividing the input tensor into one or more tiles, each tile having a size equal to the size of the kernel; and

[0282] The values ​​in the one or more tiles are flattened into vectors to generate a flattened input stream.

[0283] Solution 2. The method according to Solution 1 further includes:

[0284] The input tensor is padded with a padding value so that a size of an output of convolving the input tensor with the kernel is the same as the size of the input tensor.

[0285] Solution 3. The method according to Solution 1 further includes:

[0286] The input tensor is padded with a padding value so that the size of each dimension of the input tensor is an integer multiple of the corresponding dimension of the kernel.

[0287] Solution 4. The method according to Solution 1 further includes:

[0288] Pads the trailing edge of each dimension of the input tensor with a padding value whose width is equal to the size of the kernel in the corresponding dimension.

[0289] Option 5. The method according to Option 4, wherein the fill value is a zero value.

[0290] Option 6. A method according to Option 1, wherein the first side and the second side of the core are perpendicular, and the length of the first side is different from the length of the second side, and the method further includes: dividing the input tensor in an order aligned with the direction in which the core spans the input tensor.

[0291] Option 7. A method according to Option 1, wherein the first side and the second side of the core are perpendicular, and the length of the first side is different from the length of the second side, and the method further includes: dividing the input tensor in an order orthogonal to the direction in which the core spans the input tensor.

[0292] Solution 8. The method according to Solution 1, wherein flattening the values ​​in the one or more tiles further comprises: for each tile in the one or more tiles of the input tensor,

[0293] Access the values ​​of said tiles in a defined order;

[0294] arranging the values ​​in a vector according to the defined order; and

[0295] One or more vectors corresponding to each of the one or more tiles are arranged in a parallel arrangement to generate the flattened input stream.

[0296] Scheme 9. A method according to Scheme 1, wherein the defined order is at least one of row priority, column priority, and channel priority, wherein the channel priority order first accesses elements in a three-dimensional (3D) tile along an axis corresponding to the depth of the 3D tile, and then accesses elements in the 3D tile along axes corresponding to the width and height of the 3D tile.

[0297] Option 10. A method according to Option 1, wherein the flattened input stream is stored in a buffer, which is read by a hardware acceleration processor to perform multiplication and addition operations with 1) values ​​in the flattened input stream selected by a control mode and 2) an extension of the kernel to generate an output of a convolution operation without loading the values ​​of the input tensor into the buffer multiple times.

[0298] Option 11. A method according to Option 1, wherein the input tensor has multiple channels, wherein the kernel has multiple filters, and wherein the input channel is convolved with each kernel filter to generate an output having multiple output channels.

[0299] Solution 12. A system comprising:

[0300] A processor, the processor comprising an input stream generator, the input stream generator being configured to:

[0301] Receives the input tensor to be convolved with the kernel;

[0302] Dividing the input tensor into one or more tiles, each tile having a size equal to the size of the kernel; and

[0303] The values ​​in the one or more tiles are flattened into vectors to generate the flattened input stream.

[0304] Solution 13. The system according to Solution 12, wherein the input stream generator is further configured to:

[0305] The input tensor is padded with a padding value so that a size of an output of convolving the input tensor with the kernel is the same as the size of the input tensor.

[0306] Solution 14. The system according to Solution 12, wherein the input stream generator is further configured to:

[0307] The input tensor is padded with a padding value so that the size of each dimension of the input tensor is an integer multiple of the corresponding dimension of the kernel.

[0308] Solution 15. The system according to Solution 12, wherein the input stream generator is further configured to:

[0309] Pads the trailing edge of each dimension of the input tensor with a padding value whose width is equal to the size of the kernel in the corresponding dimension.

[0310] Option 16. A system according to Option 15, wherein the fill value is a zero value.

[0311] Embodiment 17. The system of embodiment 12, wherein the first side and the second side of the core are perpendicular, and the length of the first side is different from the length of the second side, and wherein the input stream generator is further configured to:

[0312] The input tensor is partitioned in an order aligned with the direction in which the kernel spans the input tensor.

[0313] Embodiment 18. The system of embodiment 12, wherein the first side and the second side of the core are perpendicular, and the length of the first side is different from the length of the second side, and wherein the input stream generator is further configured to:

[0314] The input tensor is partitioned in an order orthogonal to the direction in which the kernel spans the input tensor.

[0315] Solution 19. A system according to Solution 12, wherein the input stream generator is further configured to: for each tile of the one or more tiles of the input tensor,

[0316] Access the values ​​of said tiles in a defined order;

[0317] arranging the values ​​in a vector according to the defined order; and

[0318] One or more vectors corresponding to each of the one or more tiles are arranged in a parallel arrangement to generate the flattened input stream.

[0319] Option 20. A system according to Option 12, wherein the defined order is at least one of a row priority order, a column priority order, and a channel priority order, wherein the channel priority order first accesses elements in a three-dimensional (3D) tile along an axis corresponding to the depth of the 3D tile, and then accesses elements in the 3D tile along axes corresponding to the width and height of the 3D tile.

[0320] Solution 21. A method comprising:

[0321] Takes in a kernel to convolve with the input tensor;

[0322] For a first dimension of the kernel, generating a square block of values ​​(circulant matrix) for each single-dimensional vector of the kernel, the square block of values ​​including all rotations of the single-dimensional vector;

[0323] For each additional dimension of the kernel:

[0324] grouping the blocks of the previous dimension into block sets, each block set comprising blocks of the previous dimension aligned along a vector parallel to an axis of the dimension;

[0325] generating, for the additional dimension, one or more blocks of values, each block comprising all rotations of the blocks within each set of blocks in the previous dimension; and

[0326] A value block corresponding to a last dimension of the additional dimensions of the kernel is output as the extended kernel.

[0327] Option 22. A method according to Option 21, wherein the kernel is a one-dimensional (1D) matrix, and wherein the extended kernel includes multiple parallel vectors, each of the multiple parallel vectors is a different rotation arrangement of the kernel, and the extended kernel is a square matrix, and the size of each dimension of the square matrix is ​​equal to the size of the kernel.

[0328] Embodiment 23. The method of embodiment 21, wherein the kernel is a two-dimensional (2D) matrix, and wherein the extended kernel comprises:

[0329] a set of first vectors, each first vector having a rotated arrangement of blocks in a block group, wherein each block in the group of blocks corresponds to a vector in the kernel (kernel vector), and

[0330] Each block has a set of second vectors, the second vectors representing all rotational permutations of the corresponding kernel vector, each second vector representing one rotational permutation of the corresponding kernel vector,

[0331] The first vector set includes all rotation arrangements of the blocks in the block group.

[0332] Embodiment 24. The method of embodiment 21, wherein the kernel is a three-dimensional (3D) matrix, and wherein the extended kernel comprises:

[0333] a first set of vectors, each first vector having a rotated permutation of a first block in the first block set,

[0334] Each first block in the first block group corresponds to a second vector in the second vector set, and each second vector corresponds to a second block in the second block group.

[0335] wherein each second block in the second block group corresponds to a vector in the kernel (kernel vector), and

[0336] Each second block has a set of third vectors, the third vectors representing all rotational permutations of the corresponding kernel vector of the second block, each third vector representing one rotational permutation of the corresponding kernel vector,

[0337] wherein each second vector set includes all rotation arrangements of second blocks in the corresponding second block group, and wherein the first vector set includes all rotation arrangements of first blocks in the first block group.

[0338] Option 25. A method according to Option 21, wherein each single-dimensional vector is a unique vector, and the unique vector is at least one of the row of the core, the column of the core, the diagonal of the core, and the channel of the core, wherein the channel of the core is a vector of the core aligned along the axis corresponding to the depth (third dimension) of the core.

[0339] Option 26. The method according to Option 21, wherein the core includes a plurality of core filters, and the method further comprises:

[0340] A plurality of extended cores are generated, each extended core of the plurality of extended cores being generated from a core filter of the core, wherein each core filter of the core includes a core.

[0341] Option 27. A method according to Option 26, wherein each of the multiple extended kernels is used to convolve one or more channels of a flattened input stream, and the flattened input stream is a flattened version of the input tensor.

[0342] Option 28. A method according to Option 21, wherein the output of the convolution of the kernel with an input tensor having one or more channels generates multiple outputs, each of the multiple outputs being combined into a single output activation.

[0343] Solution 29. The method according to Solution 21, further comprising:

[0344] Accessing a flattened input stream from a buffer, the flattened input stream being a modified version of an input tensor, the flattened input stream having flattened parallel vectors, each flattened parallel vector being generated using input values ​​of a tile of a corresponding kernel size of the input tensor;

[0345] an access control pattern having a delay value indicating, for each output value position of a target kernel size tile of the input tensor, an amount of delay to be applied in the buffer to access a specified tile of the flattened input stream; and

[0346] For each output value position of a tile of the target kernel size of the input tensor, a dot product between a corresponding vector of the expanded kernel and a vector of values ​​selected from the flattened input stream using a set of corresponding delay values ​​of the control mode selected for the output value position is performed by a hardware acceleration processor to generate an output value for the output value position.

[0347] Option 30. A method according to Option 21, wherein the extended kernel is a two-dimensional square matrix, and the length of the two-dimensional square matrix is ​​equal to the product of the dimensions of the corresponding kernels used to generate the extended kernel.

[0348] Solution 31. A system comprising:

[0349] A processor having an extended core generator configured to:

[0350] Takes in a kernel to convolve with the input tensor;

[0351] For a first dimension of the kernel, generating a square block of values ​​(circulant matrix) for each single-dimensional vector of the kernel, the square block of values ​​including all rotations of the single-dimensional vector;

[0352] For each additional dimension of the kernel:

[0353] dividing the blocks of the previous dimension into sets of blocks, each set of blocks comprising blocks of the previous dimension aligned along a vector parallel to an axis of the dimension;

[0354] generating, for the additional dimension, one or more blocks of values, each block comprising all rotations of the blocks within each set of blocks in the previous dimension; and

[0355] A value block corresponding to a last dimension of the additional dimensions of the kernel is output as the extended kernel.

[0356] Option 32. A system according to Option 31, wherein the kernel is a one-dimensional (1D) matrix, and wherein the extended kernel includes a plurality of parallel vectors, each of the plurality of parallel vectors is a different rotation arrangement of the kernel, and the extended kernel is a square matrix, and the size of each dimension of the square matrix is ​​equal to the size of the kernel.

[0357] Embodiment 33. The system of embodiment 31, wherein the kernel is a two-dimensional (2D) matrix, and wherein the extended kernel comprises:

[0358] a first set of vectors, each first vector having a rotated arrangement of blocks in a block group, wherein each block in the block group corresponds to a vector in the kernel (kernel vector), and

[0359] Each block has a set of second vectors, the second vectors representing all rotational permutations of the corresponding kernel vector, each second vector representing one rotational permutation of the corresponding kernel vector,

[0360] The first vector set includes all rotation arrangements of the blocks in the block group.

[0361] Embodiment 34. The system of embodiment 31, wherein the kernel is a three-dimensional (3D) matrix, and wherein the extended kernel comprises:

[0362] a first set of vectors, each first vector having a rotated permutation of a first block in the first block set,

[0363] Each first block in the first block group corresponds to a second vector in the second vector set, and each second vector corresponds to a second block in the second block group.

[0364] wherein each second block in the second block group corresponds to a vector in the kernel (kernel vector), and

[0365] Each second block has a set of third vectors, the third vectors representing all rotational permutations of the corresponding kernel vector of the second block, each third vector representing one rotational permutation of the corresponding kernel vector,

[0366] wherein each second vector set includes all rotation arrangements of second blocks in the corresponding second block group, and wherein the first vector set includes all rotation arrangements of first blocks in the first block group.

[0367] Option 35. A system according to Option 31, wherein each single-dimensional vector is a unique vector, and the unique vector is at least one of the row of the core, the column of the core, the diagonal of the core, and the channel of the core, wherein the channel of the core is a vector of the core aligned along the axis corresponding to the depth (third dimension) of the core.

[0368] Embodiment 36. The system of embodiment 31, wherein the core comprises a plurality of core filters, and wherein the extended core generator is further configured to:

[0369] A plurality of extended cores are generated, each extended core of the plurality of extended cores being generated from a core filter of the core, wherein each core filter of the core includes a core.

[0370] Option 37. A system according to Option 36, wherein each of the multiple extended cores is used to convolve one or more channels of a flattened input stream, and the flattened input stream is a flattened version of the input tensor.

[0371] Option 38. A system according to Option 31, wherein the output of the convolution of the kernel with an input tensor having one or more channels generates multiple outputs, each of the multiple outputs being combined into a single output activation.

[0372] Embodiment 39. The system according to embodiment 31, wherein the extended core generator is further configured to:

[0373] Accessing a flattened input stream from a buffer, the flattened input stream being a modified version of an input tensor, the flattened input stream having flattened parallel vectors, each flattened parallel vector being generated using input values ​​of a tile of a corresponding kernel size of the input tensor;

[0374] an access control pattern having a delay value indicating, for each output value position of a target kernel size tile of the input tensor, an amount of delay to be applied in the buffer to access a specified tile of the flattened input stream; and

[0375] For each output value position of a tile of the target kernel size of the input tensor, a dot product is performed by a processor between a corresponding vector of the expanded kernel and a vector of values ​​selected from the flattened input stream using a set of corresponding delay values ​​of the control mode selected for the output value position to generate an output value for the output value position.

[0376] Option 40. A system according to Option 31, wherein the extended kernel is a two-dimensional square matrix, and the length of the two-dimensional square matrix is ​​equal to the product of the dimensions of the corresponding kernels used to generate the extended kernel.

[0377] Embodiment 41. A method comprising:

[0378] receiving one or more dimensions for each of the dimensions of a kernel that is convolved with an input tensor to generate an output activation;

[0379] Generate a control pattern for calculating an output value of the convolution of the input tensor, the control pattern being a square matrix, the size of each dimension of the square matrix being equal to the product of the width and height of the kernel, the control pattern being generated by the following operations:

[0380] A value for the position of the control pattern is generated based on the position of each position in the control pattern and one or more dimensions of each of the dimensions of the kernel, the value indicating the position from which a value of a flattened input tensor convolved with the kernel is accessed.

[0381] Embodiment 42. The method according to embodiment 41, wherein each position of the control pattern has a row number and a column number, and wherein generating a value for each position of the control pattern further comprises:

[0382] for each dimension of the kernel, determining a result of a test based on the row number, the column number, and one or more sizes for each dimension of the kernel, each test generating a binary output indicating a relative offset from which to access a value of the flattened input stream generated from the input tensor by flattening a tile of the kernel size of the input tensor; and

[0383] A value for the position of the control pattern is generated, the value being a combination of the binary outputs of the test for each dimension of the core.

[0384] Option 43. A method according to Option 42, wherein each test is an inequality test that compares a first modulo operation using the row number of the location with a second modulo operation using the column number of the location.

[0385] Option 44. A method according to Option 43, wherein the first modulo operation is the modulo between the row number and one or more dimensions of each dimension of the kernel, and the second modulo operation is the modulo between the column number and one or more dimensions of each dimension of the kernel.

[0386] Embodiment 45. The method of embodiment 44, wherein the kernel is a two-dimensional kernel, and wherein a first test in the tests corresponding to each dimension is:

[0387] r mod h <c mod w

[0388] The second test in the tests corresponding to each dimension is:

[0389] floor((r mod(h*w)) / h) <floor((c mod(h*w)) / w)

[0390] And wherein r is the row number, c is the column number, h is the height of the kernel, and w is the width of the kernel.

[0391] Embodiment 46. A method according to Embodiment 41, wherein the control pattern comprises a plurality of vectors, the number of vectors in the plurality of vectors corresponding to the number of output value positions in a tile of a kernel size of the output activation.

[0392] Scheme 47. A method according to Scheme 46, wherein the value within each of the multiple vectors corresponds to a delay value, each delay value indicates the delay amount for accessing a respective input value in a flattened input stream, the flattened input stream comprises a set of parallel vectors generated from an input tensor, and the delay amount specifies one of the parallel vectors within the flattened input stream.

[0393] Solution 48. The method according to solution 47, wherein the delay value is used to generate an output value for each output value position of a tile of the kernel size of the output activation by the following operation:

[0394] For an output value position corresponding to an output value to be generated, selecting a vector in the control mode corresponding to the output value position;

[0395] accessing delay values ​​in a vector of selections of the control mode, each delay value indicating a parallel vector in the flattened input stream from which each input value for generating the output value is accessed;

[0396] A dot product is performed between the accessed input value and the vector of the expanded kernel to generate an output value at the output value location in a tile of the kernel size of the output activation.

[0397] Scheme 49. A method according to Scheme 48, wherein the extended kernel is generated by modifying the kernel to include a hierarchical set of blocks, each block in the hierarchical set of blocks includes all rotational permutations of blocks that are nodes of the block, and the remaining blocks in the hierarchical set of blocks are blocks that include all rotational permutations of respective vectors of the kernel, and each block in the hierarchical set of blocks has a node, which is a block aligned with a vector parallel to an axis of a dimension represented by the block within the kernel.

[0398] Option 50. A method according to Option 41, wherein the control pattern is a two-dimensional square matrix, the length of which is equal to the product of the dimensions of the corresponding kernel used to convolve the input tensor.

[0399] Option 51. A method according to Option 41, wherein the order in which the delay values ​​in the control pattern are arranged is based on a defined order, and the tiles of the core size of the input tensor are accessed in the defined order to generate a flattened input stream.

[0400] Scheme 52. A method according to Scheme 41, wherein the defined order is at least one of row priority, column priority, and channel priority, wherein the channel priority order first accesses the elements in the tiles of the 3D kernel size of the input tensor along an axis corresponding to the depth of the three-dimensional (3D) tile, and then accesses the elements in the tiles of the 3D kernel size of the input tensor along axes corresponding to the width and height of the 3D tile.

[0401] Embodiment 53. A system comprising:

[0402] A processor having a control pattern generator configured to:

[0403] receiving one or more dimensions for each of the dimensions of a kernel that is convolved with an input tensor to generate an output activation;

[0404] Generate a control pattern for calculating an output value of the convolution of the input tensor, the control pattern being a square matrix, the size of each dimension of the square matrix being equal to the product of the width and height of the kernel, the control pattern being generated by the following operations:

[0405] A value for the position of the control pattern is generated based on the position of each position in the control pattern and one or more dimensions of each of the dimensions of the kernel, the value indicating the position from which a value of a flattened input tensor convolved with the kernel is accessed.

[0406] Embodiment 54. The system according to embodiment 53, wherein each position of the control pattern has a row number and a column number, and wherein the control pattern generator is further configured to:

[0407] for each dimension of the kernel, determining a result of a test based on the row number, the column number, and one or more dimensions of each of the dimensions of the kernel, each test generating a binary output indicating a relative offset from which to access a value of the flattened input stream generated from the input tensor by flattening tiles of the kernel size of the input tensor; and

[0408] A value for the position of the control pattern is generated, the value being a combination of the binary outputs of the test for each dimension of the core.

[0409] Embodiment 55. A system according to embodiment 54, wherein each test is an inequality test that compares a first modulo operation using the row number of the location with a second modulo operation using the column number of the location.

[0410] Option 56. A system according to Option 55, wherein the first modulo operation is the modulo between the row number and one or more dimensions of each dimension of the kernel, and the second modulo operation is the modulo between the column number and one or more dimensions of each dimension of the kernel.

[0411] Embodiment 57. The system of embodiment 56, wherein the kernel is a two-dimensional kernel, and wherein a first test of the tests corresponding to each dimension is:

[0412] r mod h <c mod w

[0413] The second test in the tests corresponding to each dimension is:

[0414] floor((r mod(h*w)) / h) <floor((c mod(h*w)) / w)

[0415] Where r is the row number, c is the column number, h is the height of the kernel, and w is the width of the kernel.

[0416] Embodiment 58. A system according to Embodiment 53, wherein the control mode includes a plurality of vectors, the number of vectors in the plurality of vectors corresponding to the number of output value positions in a tile of the kernel size of the output activation.

[0417] Option 59. A system according to Option 58, wherein the value within each of the multiple vectors corresponds to a delay value, each delay value indicates an amount of delay for accessing a respective input value in a flattened input stream, the flattened input stream includes a set of parallel vectors generated from an input tensor, and the delay amount specifies one of the parallel vectors within the flattened input stream.

[0418] Embodiment 60. The system of embodiment 59, wherein the delay value is used to generate an output value for each output value position of a tile of the kernel size of the output activation by the following operation:

[0419] For an output value position corresponding to an output value to be generated, selecting a vector in the control mode corresponding to the output value position;

[0420] accessing delay values ​​in a vector of selections of the control mode, each delay value indicating a parallel vector in the flattened input stream from which each input value for generating the output value is accessed;

[0421] A dot product is performed between the accessed input value and the vector of the expanded kernel to generate an output value at the output value location in a tile of the kernel size of the output activation.

[0422] Embodiment 61. A method comprising:

[0423] accessing from the buffer a flattened input stream comprising a set of parallel vectors, each vector representing a set of input values ​​of a unique kernel-sized tile of the input tensor that is convolved with the kernel to generate an output activation;

[0424] receiving an extended kernel generated by permuting values ​​of the kernel, the extended kernel having vectors of output value positions each corresponding to a kernel-sized tile of the output activation;

[0425] receiving a control pattern comprising a set of vectors, each vector corresponding to the output value locations of a kernel-sized tile of the output activations, each vector comprising delay values ​​indicating parallel vectors of the flattened input stream for accessing input values ​​for convolution; and

[0426] For each output value position of each kernel-sized tile of the output activation, a dot product between a first vector including values ​​of the flattened input stream selected by delayed values ​​of a corresponding vector of the control pattern and a second vector corresponding to a vector in the extended kernel corresponding to the output value position is generated using a hardware acceleration processor.

[0427] Option 62. A method according to Option 61, wherein the input tensor has multiple channels, wherein the kernel has multiple filters, and wherein each channel of the input tensor is convolved with one or more filters of the kernel to generate an output activation having multiple output features.

[0428] Embodiment 63. The method according to embodiment 61 further comprises:

[0429] The input tensor is padded with a padding value so that positions of values ​​of an output of the convolution of the input tensor have matching positions for all positions of values ​​in the input tensor and so that a size of the output of the convolution matches a size of the input tensor.

[0430] Embodiment 64. The method according to embodiment 61 further comprises:

[0431] The input tensor is padded with a padding value so that the size of each dimension of the input tensor is an integer multiple of the corresponding dimension of the kernel.

[0432] Embodiment 65. The method according to embodiment 61 further comprises:

[0433] Pads the trailing edge of each dimension of the input tensor with a padding value whose width is equal to the size of the kernel in the corresponding dimension.

[0434] Solution 66. The method according to Solution 61, wherein the flattened input stream is generated by the following operations:

[0435] For each tile of the one or more kernel-size tiles of the input tensor;

[0436] Access the values ​​of said tiles in a defined order;

[0437] arranging the values ​​in a vector according to the defined order; and

[0438] One or more vectors corresponding to each of the one or more tiles are arranged in a parallel arrangement to generate parallel vectors of the flattened input stream.

[0439] Scheme 67. A method according to Scheme 66, wherein the defined order is at least one of row priority, column priority, and channel priority, wherein the channel priority order first accesses elements in a three-dimensional (3D) tile along an axis corresponding to the depth of the 3D tile, and then accesses elements in the 3D tile along axes corresponding to the width and height of the 3D tile.

[0440] Embodiment 68. The method according to embodiment 61, wherein the extended core is formed by the following operations:

[0441] For a first dimension of the kernel, generating a square block of values ​​(circulant matrix) for each single-dimensional vector of the kernel, the square block of values ​​including all rotations of the single-dimensional vector;

[0442] For each additional dimension of the kernel:

[0443] grouping the blocks of the previous dimension into block sets, each block set comprising blocks of the previous dimension aligned along a vector parallel to an axis of the dimension;

[0444] generating, for the additional dimension, one or more blocks of values, each block comprising all rotations of the blocks within each set of blocks in the previous dimension; and

[0445] A value block corresponding to a last dimension of the additional dimensions of the kernel is output as the extended kernel.

[0446] Option 69. A method according to Option 68, wherein each single-dimensional vector is a unique vector, and the unique vector is at least one of the row of the core, the column of the core, the diagonal of the core, and the channel of the core, wherein the channel of the core is a vector of the core aligned along the axis corresponding to the depth (third dimension) of the core.

[0447] Embodiment 70. A method according to Embodiment 61, wherein the control pattern comprises a plurality of vectors, the number of vectors in the plurality of vectors corresponding to the number of output value positions in a tile of a kernel size of the output activation.

[0448] Option 71. A method according to Option 70, wherein the value within each vector of the multiple vectors corresponds to a delay value, each delay value indicates the delay amount for accessing a respective input value in a flattened input stream, the flattened input stream includes a set of parallel vectors generated from an input tensor, and the delay value specifies one of the parallel vectors within the flattened input stream.

[0449] Scheme 72. A method according to Scheme 70, wherein the value within each of the multiple vectors corresponds to a relative addressing value, each relative addressing value is an indication of the location of accessing a single input value in a flattened input stream, the flattened input stream includes a set of parallel vectors generated from an input tensor, and the relative addressing value is used to specify one of the parallel vectors within the flattened input stream.

[0450] Embodiment 73. A system comprising:

[0451] A processor and a multiply-accumulate unit, the processor and the multiply-accumulate unit being configured to:

[0452] accessing from the buffer a flattened input stream comprising a set of parallel vectors, each vector representing a set of input values ​​of a unique kernel-sized tile of the input tensor that is convolved with the kernel to generate an output activation;

[0453] receiving an extended kernel generated by permuting values ​​of the kernel, the extended kernel having vectors of output value positions each corresponding to a kernel-sized tile of the output activation;

[0454] receiving a control pattern comprising a set of vectors, each vector corresponding to the output value locations of a kernel-sized tile of the output activations, each vector comprising delay values ​​indicating parallel vectors of the flattened input stream for accessing input values ​​for convolution; and

[0455] For each output value position of each kernel-sized tile of the output activation, a dot product is generated between a first vector comprising values ​​of the flattened input stream selected by delayed values ​​of a corresponding vector of the control pattern and a second vector corresponding to a vector in the extended kernel corresponding to the output value position.

[0456] Embodiment 74. A system according to Embodiment 73, wherein the input tensor has multiple channels, wherein the kernel has multiple filters, and wherein each channel of the input tensor is convolved with one or more filters of the kernel to generate an output activation having multiple output features.

[0457] Embodiment 75. The system according to embodiment 73, wherein the multiplication-accumulation unit is further configured to:

[0458] The input tensor is padded with a padding value so that positions of output values ​​of the convolution of the input tensor have matching positions for positions of all values ​​in the input tensor and so that a size of the output of the convolution matches a size of the input tensor.

[0459] Embodiment 76. The system of embodiment 73, wherein the multiplication-accumulation unit is further configured to:

[0460] The input tensor is padded with a padding value so that the size of each dimension of the input tensor is an integer multiple of the corresponding dimension of the kernel.

[0461] Embodiment 77. The system of embodiment 73, wherein the multiplication-accumulation unit is further configured to:

[0462] Pads the trailing edge of each dimension of the input tensor with a padding value whose width is equal to the size of the kernel in the corresponding dimension.

[0463] Embodiment 78. The system according to embodiment 73, wherein the flattened input stream is generated by the following operations:

[0464] For each tile of the one or more kernel-size tiles of the input tensor;

[0465] Access the values ​​of said tiles in a defined order;

[0466] arranging the values ​​in a vector according to the defined order; and

[0467] One or more vectors corresponding to each of the one or more tiles are arranged in a parallel arrangement to generate parallel vectors of the flattened input stream.

[0468] Embodiment 79. The system according to embodiment 73, wherein the extended core is formed by the following operations:

[0469] For a first dimension of the kernel, generating a square block of values ​​(circulant matrix) for each single-dimensional vector of the kernel, the square block of values ​​including all rotations of the single-dimensional vector;

[0470] For each additional dimension of the kernel:

[0471] grouping the blocks of the previous dimension into block sets, each block set comprising blocks of the previous dimension aligned along a vector parallel to an axis of the dimension;

[0472] generating, for the additional dimension, one or more blocks of values, each block comprising all rotations of the blocks within each set of blocks in the previous dimension; and

[0473] A value block corresponding to a last dimension of the additional dimensions of the kernel is output as the extended kernel.

[0474] Embodiment 80. A system according to Embodiment 73, wherein the control pattern includes a plurality of vectors, the number of vectors in the plurality of vectors corresponding to the number of output value positions in a tile of the kernel size of the output activation.

Claims

1. A method comprising: receiving a corresponding size for each of a plurality of dimensions of a kernel that is convolved with an input tensor to generate an output value; as well as Generate a control pattern for calculating the output value, wherein the control pattern is a two-dimensional square matrix, wherein the size of each dimension is equal to the product of the width and height of the kernel, and the length of the two-dimensional square matrix is ​​equal to the product of the size of the dimension of the kernel, and the control pattern is generated by the following operations: A value is generated for each position of the control pattern, the value indicating a position for accessing a value from a flattened version of the input tensor for convolution with the kernel based on the position's position in the control pattern and a corresponding size of each of the dimensions of the kernel.

2. The method according to claim 1, wherein: Each position of the control pattern has a row number and a column number, and wherein generating a value of each position of the control pattern further comprises: for each dimension of the kernel, determining a result of a test based on the row number, the column number, and a corresponding size for each of the dimensions of the kernel, each test generating a corresponding binary output of a plurality of binary outputs indicating a relative offset for accessing a value from a flattened version of the input tensor generated from the input tensor by flattening tiles of the kernel size of the input tensor; and The value for the position of the control pattern is generated as a combination of the binary outputs of the tests for each dimension of the core.

3. The method according to claim 2, wherein: Each test is an inequality test that compares a first modulo operation using the row number of the position with a second modulo operation using the column number of the position.

4. The method according to claim 3, wherein: The first modulo operation is a modulo between the row number and a corresponding size of each of the dimensions of the kernel, and the second modulo operation is a modulo between the column number and a corresponding size of each of the dimensions of the kernel.

5. The method according to claim 4, wherein: The kernel is a two-dimensional kernel, and wherein a first of the tests corresponding to each dimension is: r mod h <c mod w wherein a second test in the tests corresponding to each dimension of the kernel is: floor((r mod(h*w)) / h) <floor((c mod(h*w)) / w) And wherein r is the row number, c is the column number, h is the height of the kernel, and w is the width of the kernel.

6. The method according to claim 1, wherein: The control pattern includes a plurality of vectors, the number of vectors in the plurality of vectors corresponding to the number of output value positions in a kernel-size-sized tile of the output values.

7. The method according to claim 6, wherein: The values ​​within each of the multiple vectors correspond to delay values, each delay value indicates an amount of delay for accessing a respective input value in a flattened version of the input tensor, the flattened version of the input tensor comprising a set of parallel vectors generated from the input tensor, and the delay amount specifies one of the parallel vectors within the flattened version of the input tensor.

8. The method according to claim 7, wherein: The delay value is used to generate an output value for each output value position of a tile of the kernel size of the output value by the following operation: For the output value position corresponding to the output value to be generated, selecting a vector in the control mode corresponding to the output value position; accessing delay values ​​in the selected vector of the control pattern, each delay value indicating a parallel vector within a flattened version of the input tensor for each input value accessed to generate the output value; as well as A dot product is performed between the accessed input value and the vector of the extended kernel to generate the output value at the output value location in a tile of the output value's kernel size.

9. The method according to claim 8, wherein: The extended kernel is generated by modifying the kernel to include a hierarchical set of blocks, each block in the hierarchical set of blocks including all rotational permutations of blocks as nodes of the block, the remaining blocks in the hierarchical set of blocks being blocks including all rotational permutations of respective vectors of the kernel, each block in the hierarchical set of blocks having nodes that are blocks aligned with vectors parallel to the axis of the dimension represented by the block within the kernel.

10. The method according to claim 1, wherein: The order in which the delay values ​​in the control pattern are arranged is based on a defined order in which the core-size-sized tiles of the input tensor are accessed to generate a flattened version of the input tensor.

11. The method according to claim 10, wherein: The defined order is at least one of a row priority order, a column priority order, and a channel priority order, and wherein the channel priority order first accesses elements in the 3D kernel size tile of the input tensor along an axis corresponding to the depth of the 3D kernel size tile of the input tensor, and then accesses elements in the 3D kernel size tile of the input tensor along axes corresponding to the width and height of the 3D kernel size tile.

12. A system comprising: A processor having a control pattern generator, the control pattern generator being configured to: receiving a corresponding size for each of a plurality of dimensions of a kernel that is convolved with an input tensor to generate an output value; and Generate a control pattern for calculating the output value, wherein the control pattern is a two-dimensional square matrix, wherein the size of each dimension is equal to the product of the width and height of the kernel, and the length of the two-dimensional square matrix is ​​equal to the product of the size of the dimension of the kernel, and the control pattern is generated by the following operations: A value is generated for each position of the control pattern, the value indicating a position for accessing a value from a flattened version of the input tensor for convolution with the kernel based on the position's position in the control pattern and a corresponding size of each of the dimensions of the kernel.

13. The system according to claim 12, wherein: The control pattern includes a plurality of vectors, the number of vectors in the plurality of vectors corresponding to the number of output value positions in a kernel-size-sized tile of the output values.

14. A system comprising: A processor having a control pattern generator, the control pattern generator being configured to: receiving a corresponding size for each of a plurality of dimensions of a kernel that is convolved with an input tensor to generate an output value; generating a control pattern for computing the output value, the control pattern being a square matrix having a size in each dimension equal to the product of a width and a height of the kernel, the control pattern being generated by: generating a value for each position of the control pattern, the value being based on a location of the position in the control pattern and a corresponding size of each of the dimensions of the kernel, the value indicating a location for accessing a value from a flattened version of the input tensor for convolution with the kernel, each position of the control pattern having a row number and a column number; for each dimension of the kernel, determining a result of a test based on the row number, the column number, and a corresponding size for each of the dimensions of the kernel, each test generating a corresponding binary output of a plurality of binary outputs indicating a relative offset for accessing a value from a flattened version of the input tensor generated from the input tensor by flattening tiles of the kernel size of the input tensor; as well as The value for the position of the control pattern is generated as a combination of the binary outputs of the tests for each dimension of the core.

15. The system of claim 14, wherein: Each test is an inequality test that compares a first modulo operation using the row number of the position with a second modulo operation using the column number of the position.

16. The system of claim 15, wherein: The first modulo operation is a modulo between the row number and a corresponding size of each of the dimensions of the kernel, and the second modulo operation is a modulo between the column number and a corresponding size of each of the dimensions of the kernel.

17. The system of claim 16, wherein: The kernel is a two-dimensional kernel, and wherein a first test of the tests corresponding to each dimension of the kernel is: r mod h <c mod w wherein a second test in the tests corresponding to each dimension of the kernel is: floor((r mod(h*w)) / h) <floor((c mod(h*w)) / w) And wherein r is the row number, c is the column number, h is the height of the kernel, and w is the width of the kernel.

18. The system of claim 13, wherein: The values ​​within each of the multiple vectors correspond to delay values, each delay value indicates an amount of delay for accessing a respective input value in a flattened version of the input tensor, the flattened version of the input tensor comprising a set of parallel vectors generated from the input tensor, and the delay amount specifies one of the parallel vectors within the flattened version of the input tensor.

19. The system of claim 18, wherein: The delay value is used to generate an output value for each output value position of a tile of the kernel size of the output value by the following operation: For the output value position corresponding to the output value to be generated, selecting a vector in the control mode corresponding to the output value position; accessing delay values ​​in the selected vector of the control pattern, each delay value indicating a parallel vector within a flattened version of the input tensor for each input value accessed to generate the output value; as well as A dot product is performed between the accessed input value and the vector of the extended kernel to generate the output value at the output value location in a tile of the output value's kernel size.

Images