Composing arbitrary convolutional neural network models from a fixed set of replicated pipeline components

Abstract

An application specific integrated circuit (ASIC) for computing a convolutional neural network (CNN) has a first input bus that receives an ordered stream of values from an array, where each location in the array has one or more channels, and a plurality of kernel processing tiles that receive the input via a configurable multiplexer. The kernel processing tiles and bus are arranged and connected such that the ASIC operates as a pipelined system, delivering an output stream synchronous with the input stream.

Description

[Technical Field]

[0001] This application is a continuation-in-part of co-pending U.S. patent application Ser. No. 17 / 742,245, filed May 11, 2022, which is a continuation-in-part of co-pending U.S. patent application Ser. No. 17 / 570,757, filed January 7, 2022 (now granted as U.S. Patent No. 11,354,571 on June 7, 2022), which is a continuation-in-part of co-pending U.S. patent application Ser. No. 17 / 373,497, filed July 12, 2021 (now granted as U.S. Patent No. 11,354,571 on July 7, 2022). No. 17 / 231,711, filed April 15, 2021 (now granted as U.S. Patent No. 11,099,854 on August 24, 2021), which is a continuation-in-part of co-pending U.S. Patent Application No. 17 / 071,875, filed October 15, 2020. The entire disclosure of the parent application is incorporated herein by reference, at a minimum.

[0002] The present invention belongs to the technical field of computer operations involving input and output of matrices, and more particularly to circuits designed for large scale multiplication in matrix operations. [Background technology]

[0003] The use of computers in matrix operations is widely known in the art, with specific examples being the development and use of image processing and neural networks. Neural networks are an important part of artificial intelligence and, as such, are a very popular subject in intellectual property development at the time of the filing of this patent application. Generally speaking, in this type of computer operation, a significant number of input values ​​are processed in a regular pattern, which is most often a matrix. The processing of the input values ​​may include the addition of biases and the application of weights by which individual input values ​​may be multiplied.

[0004] The inventors of the present invention believe that the sophisticated and computationally intensive operations in neural network technology, in which an incoming value is multiplied by each of a plurality of weight values, is an open step for technological innovation that will provide a clear advantage in the art. The inventors of the present invention also believe that there are advantages to be gained by modifying the order in which the mathematical operations are applied.

[0005] The inventors of the present invention believe that they have determined general changes in the order and manner of mathematical operations implemented in such applications, which changes can result in very significant reductions in the time and cost of such operations. Summary of the Invention [Means for solving the problem]

[0006] In one embodiment of the present invention, an application specific integrated circuit (ASIC) for computing a convolutional neural network (CNN) includes: a first input bus that receives an ordered stream of values ​​from an array, each location in the array having one or more data channels; a first ordered set of kernel processing tiles, from first to last, having a fixed number of parallel input connections and a fixed number of parallel output connections, each kernel processing tile in the first ordered set coupled to the input bus via one of a first set of configurable multiplexers, the kernel processing tiles adapted to compute convolutions for a common kernel size and pass the computed values ​​as inputs back to each configurable multiplexer in the first set of configurable multiplexers on a connected first output bus, as well as to adjacent downstream kernel processing tiles in the first ordered set; and a second ordered set of kernel processing tiles, from first to last, having a fixed number of parallel input connections and a fixed number of parallel output connections, the second ordered set coupled to the input bus via one of a first set of configurable multiplexers, the second ordered set adapted to compute convolutions for a common kernel size and pass the computed values ​​as inputs back to each configurable multiplexer in the first set of configurable multiplexers on a connected first output bus, as well as to adjacent downstream kernel processing tiles in the first ordered set. a second ordered set, each kernel processing tile of the ordered set coupled to the first output bus via one of the second set of configurable multiplexers, the kernel processing tiles of the second ordered set adapted to compute convolutions for a common kernel size and pass the computed values ​​to the connected second output bus as inputs returned to each configurable multiplexer of the second set of configurable multiplexers, and also to adjacent downstream kernel processing tiles of the second ordered set; and a third ordered set, from first to last, of kernel processing tiles having a fixed number of parallel input connections and a fixed number of parallel output connections, each kernel processing tile of the third ordered set coupled to the second output bus via one of the third set of configurable multiplexers, the kernel processing tiles of the third ordered set adapted to compute convolutions for a common kernel size and pass the computed values ​​to the connected third output bus as inputs returned to each configurable multiplexer of the third set of configurable multiplexers;a third ordered set adapted to also pass the third ordered set of downstream kernel processing tiles to adjacent downstream kernel processing tiles, the third output bus also connected via a single primary output multiplexer to a primary output circuit adapted to perform primary output processing and provide a final output.

[0007] In one embodiment, the ASIC further comprises an additional configurable multiplexer coupled to the first output bus for providing a selected value to a first kernel processing tile in the second ordered set, and an additional configurable multiplexer coupled to the second output bus for providing a selected value to a first kernel processing tile in the third ordered set. Also, in one embodiment, the ASIC further comprises one or more auxiliary function tiles for providing functions other than the functions of the kernel processing tiles. In one embodiment, one or more auxiliary function tiles accept inputs from a dual multiplexer connected to the input bus and the first output bus and provide outputs to the first output bus. Furthermore, in one embodiment, one or more auxiliary function tiles accept inputs from a dual multiplexer connected to the first output bus and the second output bus and provide outputs to the second output bus.

[0008] In one embodiment of the present invention, one or more auxiliary function tiles receive input from a dual multiplexer connected to the second and third output buses and provide an output to the third output bus. Also, in one embodiment, the ASIC further comprises external function circuitry that selects an input from the first output bus via a configurable multiplexer and provides an output to the first output bus. Also, in one embodiment, the ASIC further comprises external function circuitry that selects an input from the second output bus via a configurable multiplexer and provides an output to the second output bus. Also, in one embodiment, the ASIC further comprises external function circuitry that selects an input from the third output bus via a configurable multiplexer and provides an output to the third output bus. Also, in one embodiment, the common kernel size is a 3x3 kernel.

[0009] In one embodiment, the fixed number of parallel input connections is 16 and the fixed number of parallel output connections is 16. Also, in one embodiment, an ordered stream of values ​​is provided by one of a direct camera output of RGB values, a DMA interface suitable for access from a CPU bus, or a video stream decompression circuit to generate three parallel channels of red, green, and blue (RGB) values ​​for the image. Also, in one embodiment, the ASIC further includes combining the operations of two or more kernel processing tiles processing a 3×3 kernel of 16 parallel input and output connections to configure a 3×3 kernel having more than 16 inputs or more than 16 outputs, or both more than 16 inputs and more than 16 outputs. In one embodiment, the ASIC is adapted with additional circuitry to compute convolutions having kernels larger than 3×3 by combining the operations of multiple 3×3 kernel processing tiles. Furthermore, in one embodiment, the ASIC is adapted to compute 5×5 convolutions, 7×7 convolutions, or 9×9 convolutions.

[0010] In one embodiment, a kernel processing tile presents each input channel value to a large-scale multiplier, which calculates a full set of possible multipliers from the input and provides the multipliers, along with single-channel values ​​from the auxiliary parallel connection set, to a single convolution unit. The resulting outputs from each convolution unit are grouped in a set of 16 parallel output connections and made available to other kernel processing tiles on an output bus. In one embodiment, the kernel processing tile's input values ​​are processed by a local dual-input fixed multiplier. In one embodiment, each of the auxiliary function tiles receives parallel input connections from two separate buses via a dual multiplexer and outputs one of a MaxPool function, an Average function, a Sample function, and an Expand function selected by an output multiplexer. In one embodiment, the auxiliary function tile sums values ​​from the parallel input connections by individual channels and multiplexes the summed values ​​into a lookup table configured to provide any activation function that can be expressed in tabular form, including RELU, sigmoid, or tanh activation functions. In one embodiment, input channels received from separate parallel input connections are input to a first dedicated multiplexer that concatenates the parallel input connections to effectively reroute the data channels to specific parallel output connections and provides the concatenated output to an output multiplexer as a candidate for selection for the output of the auxiliary function tile. Further, in one embodiment, input channels received from separate parallel input connections are input to a second specialized multiplexer that concatenates two parallel input connections into one parallel output connection by alternating samples from each connection and provides the result to the output multiplexer as a candidate for selection for the output of the auxiliary function tile. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is an illustration of an embodiment in which the large scale multipliers applied to each common source are fixed and directly hardwired to the processing circuitry. [Figure 2]10 is an illustration of an embodiment in which the large scale multipliers applied to each common source are dynamic and routed through a multiplexer to processing circuitry. [Figure 3] 10 is an illustration of a simple embodiment in which shifted terms corresponding to set bits in each large multiplier are added to form a product. [Figure 4] 10 is an illustration of an enhanced embodiment in which additions and subtractions of shifted terms are mixed to form an equivalent solution of lower complexity. [Figure 5A] 1 is an illustration of a pipelined embodiment that maximizes clock frequency by building sub-synthesis from only pairwise operations. [Figure 5B] 1 is an illustration of an embodiment in which multiples are formed directly by a fixed set of cases without reference to standard arithmetic operations. [Figure 6] 1 is an illustration of a pipelined embodiment that maximizes circuit density by building sub-compositions out of up to every fourth operation. [Figure 7] FIG. 1 illustrates the structure and connectivity in one embodiment of the present invention that receives an input stream, pre-processes the input stream, and passes the results through a unique digital device to generate an output stream. [Figure 8A] FIG. 1 illustrates the structure and connectivity for generating a source-channel product. [Figure 8B] FIG. 10 illustrates additional details of the controllers and functions in one embodiment of the present invention. [Figure 9A] 1 is a partial illustration of a general case of pipelined operations in one embodiment of the present invention. [Figure 9B] 1 is another partial illustration of a general case of pipelined operations in one embodiment of the present invention. [Figure 9C] 1 is another partial illustration of a general case of pipelined operations in one embodiment of the present invention. [Figure 10A]FIG. 9C illustrates the internal structure of compositors 905a, 905b, and 905c of FIGS. 9A and 9B in one embodiment of the present invention. [Figure 10B] 9A and 9B in accordance with one embodiment of the present invention. [Figure 10C] FIG. 9B illustrates the internal structure of compositor 904 of FIG. 9A in one embodiment of the present invention. [Figure 10D] FIG. 9B illustrates the internal structure of compositor 901 of FIG. 9A in one embodiment of the present invention. [Figure 10E] 9B and 9C in accordance with one embodiment of the present invention. [Figure 10F] FIG. 9C illustrates the internal structure of compositors 907a, 907b, and 907c of FIGS. 9A and 9B in one embodiment of the present invention. [Figure 10G] FIG. 9B illustrates the internal structure of compositor 906 of FIG. 9A in one embodiment of the present invention. [Figure 11] FIG. 9D illustrates the internal structure and function of delay stages 908a, 908b, 908c, 908d, 908e, and 908f of FIG. 9C in one embodiment of the present invention. [Figure 12] FIG. 9D illustrates the operation of delay stage 909 of FIG. 9C in one embodiment of the present invention. [Figure 13] FIG. 9D illustrates the operation of delay stages 910a and 910b of FIG. 9C in one embodiment of the present invention. [Figure 14] FIG. 9D illustrates the operation of the finalization step 911 in FIG. 9C. [Figure 15] FIG. 1 illustrates a particular case of pipelined operations implementing a 5×5 convolution node in one implementation of the invention. [Figure 16] 1 is an illustration of an IC for a 4×4 aperture function in one embodiment of the present invention. [Figure 17A]FIG. 1 illustrates an IC with circuitry implementing part of a deep neural network that streams input channels independently. [Figure 17B] FIG. 1 shows an IC with circuitry implementing another part of a deep neural network. [Figure 18A] FIG. 1 shows an IC with circuitry implementing part of a deep neural network that streams four input channels simultaneously. [Figure 18B] FIG. 18B shows a circuit implementing another portion of the deep neural network of FIG. 18A. [Figure 19] FIG. 17B is a table showing array stream sizes for the DNNs of FIGS. 17A and 17B. [Figure 20] FIG. 18B is a table showing array stream sizes for the DNNs of FIGS. 18A and 18B. [Figure 21] FIG. 1 shows the circuitry of an IC implementing a 3×3 convolution node streaming four input channels simultaneously. [Figure 22] FIG. 10 illustrates the necessary configuration for a circuit to generate outputs for 4-up input channels for a "same" version of a 3x3 convolution. [Figure 23] FIG. 11 shows the layout required for a circuit to output two variants of a 1 row by 7 column convolution streaming four input channels simultaneously. [Figure 24A] FIG. 1 illustrates an arrangement of 2×2 MaxPool nodes across a 4-up data stream. [Figure 24B] FIG. 24B illustrates the arrangement of the 2×2 MaxPool nodes of FIG. 24A across a 2-up data stream. [Figure 25] FIG. 10 shows an unnatural example where reducing N is not possible. [Figure 26A] FIG. 1 illustrates a FIFO circuit used to repackage a 4-up stream into a 2-up stream. [Figure 26B] FIG. 1 illustrates repackaging a 3-up stream into a 5-up stream. [Figure 27A]FIG. 10 illustrates an implementation of a Concatenation node where the output contains all channels from all sources. [Figure 27B] FIG. 1 illustrates an implementation of a 4-upDense node. [Figure 28] FIG. 1 illustrates a 4-up implementation of a 3×3 Local Average node. [Figure 29] FIG. 10 illustrates another 4-up implementation of a 3×3 Local Average node. [Figure 30A] FIG. 1 illustrates an implementation of a 4-upSubset node. [Figure 30B] FIG. 1 illustrates a typical implementation of a 4-up Crop node. [Figure 31] FIG. 1 illustrates a system of interconnected ICs that implement a neural network. [Figure 32] FIG. 1 depicts the layout of a compositor on an integrated circuit configured to implement a 3×3×3 convolution as a 3D aperture function across 27 individual data samples. [Figure 33] FIG. 10 illustrates an IC that can buffer and simultaneously present data from multiple planes so that a single compositor can apply weights for multiple planes. [Figure 34] FIG. 1 depicts a typical 3×3×3 convolution implementation applied to a 4-up input stream. [Figure 35] FIG. 10 shows a fully inverted implementation of the IC applied to a 4-up data stream. [Figure 36] FIG. 36 illustrates the application of an aperture function IC circuit to an input array stream 3601 of ordered samples in one embodiment of the present invention. [Figure 37] FIG. 10 illustrates an example of receiving two independent input array streams in one embodiment of the present invention. [Figure 38] FIG. 2 illustrates a sequence of full and downscaled data rows for a series of four 2:1 data reductions in one embodiment of the present invention. [Figure 39] In one embodiment of the present invention

number

[0012] A variety of image and data algorithms make extensive use of the matrix form of linear algebra, both to prove propositions and to compute results. By "algorithm" in this application is meant a process or set of rules to be followed, especially in a computational or other problem-solving operation. Algorithm, in this application, should not be construed as software without exception. The algorithms described in this application may typically and preferably be implemented as hardware.

[0013] Matrix operations are defined as orthogonal sequences of one or more dimensions, and are generally recognized as having the same number of elements in all iterations of each given dimension. By way of example, an MxN matrix is ​​often represented by an array of values ​​such as:

[0014]

number

[0015] Conceptually, a matrix can have any number of dimensions, and can be represented as a set of tables indicating the values ​​for each dimension.

[0016] Subsets of matrices of the form Mx1 or 1xN are sometimes called vectors, which have their own defined properties and operations and are widely used in 2D and 3D graphics simulations.

[0017] Degenerate subsets of matrices of the form 1x1 are sometimes called scalars and constitute numbers that are very familiar to those skilled in the art.

[0018] Certain operations, such as multiplication, are well defined when the matrix values ​​are constants and the matrices are of compatible dimensions. A 3x4 matrix A can be multiplied with a 4x5 matrix B to form a 3x5 matrix C, which can often be written as: A×B=C

[0019]

number

[0020] However, the operation B×A is not well defined because the inner dimensions do not match (5≠3) and k cannot have a range of indices that are compatible with the indices of B and A.

[0021] A matrix whose elements are vectors or other matrices is known as a tensor (hence the name TensorFlow). A familiar form of a tensor can be an RGB image. One form of an RGB image is an HDMI frame as a 1080x1920 matrix of RGB values, where each pixel is a 3x1 vector of color components. A pixel is considered a true vector because a linear operation on the red component does not affect green or blue, and vice versa.

[0022] An HDMI frame is not generally thought of as a 5-dimensional matrix because processing pixel locations in an image is unrelated to processing color. Cropping an image by discarding parts of the image that are not of interest is valid and quite meaningful, but there is no corresponding operation for cropping color components. Similarly, there can be many operations on color with easily understandable effects that would not make sense if applied to the elements of the containing array. Thus, an HDMI frame is clearly a few tensors, not a 5D array.

[0023] Many image processing algorithms are known that can be expressed as matrix operations, which are a concise way of expressing recursive operations, and the rules of matrix mathematics are instrumental in proving certain propositions.

[0024] Execution of matrix-based algorithms in general-purpose computer processors is typically achieved through looping mechanisms, and both computer languages ​​and hardware CPUs may have features that make such loops efficient. However, there is nothing inherent in the mathematics of matrix definition that requires operations to be performed in a particular way or scheme to compute the correct result.

[0025] A modern hybrid of image processing and recognition is the convolutional neural network (CNN). Training such networks has long been extremely difficult, but actually running a trained network is relatively easy.

[0026] In a CNN, the output element of each convolution operates by passing an independent kernel against an input tensor to generate each component of the output tensor. Typically, when a neural network is used to process an image, the first layer of the network operates on an input array of the image's RGB pixels and generates an output array of related size containing an arbitrary vector of output components that is structurally unrelated to the RGB vector of the input components. The components of the output vector are generally described as features or activations, and represent the response strength (degree of recognition) of each kernel. Subsequent layers in a CNN take the output from the previous layer as their input, so only the first layer operates on pixel values; all remaining layers operate on features to generate more features. Each output feature of a convolution is independent and distinct from any other feature, just as color components are distinct from each other.

[0027] The usual form of a CNN layer is a 3x3 convolution. In operation, a 3x3 kernel of constant weights is applied element-wise to each particular location of the input tensor (i.e., the image); each of the weights is multiplied by the pixel component at the same relative location in the image, and the products are added to produce a single component of the output for that location. A bias constant (which can be zero) provides an initial value to make it easier to solve the model to arrive at optimal weight values.

[0028] If there are three input components, as there are in an RGB image, there are three separate sets of 3x3 weights (which in the first layer are colors) applied to each component value, but only one initial bias. Adding the bias to each convolution of the 3x3x3 weights forms a single output component value corresponding to the pixel's position at the center of the 3x3 patch. Each output channel then applies its own 27 weight values ​​until all output components for a given patch (a subset of input components at the same location as the output position and corresponding to the relative position of the kernel weights) have been calculated. It is common for a convolution to have between 64 and 256 output components, each with a unique set of 27 weights plus a bias.

[0029] In this example, each kernel multiplies its 27 weights with the same patch of nine pixels for the three RGB components. For a relatively small set of 64 output components, each separate input component is multiplied by 64 arbitrary, unrelated weights. After the output components for each patch are calculated, the adjacent patch is loaded from the image and the entire set of kernel weights is applied again. This process continues until the right edge of the image is reached, then the patch drops down one row and starts again at the left edge.

[0030] After the first layer is processed, the next convolutional layer processes the output of the first layer as input to the second layer. Thus, the 3x3 convolution now has 3x3x64 weights applied to the 3x3x64 input components of the patch. If this layer has 256 outputs, then 3x3x64x256 = 147,456 multiplications must be performed for each output position. Those skilled in the art will understand that this refers to a single layer in a deep neural network, which may include more than 40 layers.

[0031] The number of multiplications applied to each element of the patch is equal to the number of channels in the layer. On a standard CPU, these must always be done in some sequence. Many modern CPUs have the ability to perform multiple sets of multiplications simultaneously, especially when the data format is small (i.e., 8-bit). On a GPU or TPU, the number of available multipliers is much larger, but each multiplier is designed to produce the product of two distinct and unlimited factors.

[0032] Current state-of-the-art processors, CPUs, TPUs, or GPUs, do not take advantage of the simple fact that in CNN implementations, one of the multiplication factors is common to all weights applied to an input channel during processing for a patch.

[0033] The inventors of this application propose a large scale multiplier that performs all multiplications in a single step, which are conventionally performed sequentially instead. When the weights of a set of multiplications are all in a small precision (typically 8 bits for a TPU), the precision is limited (2 8 = 256) distinct weights, and there are a corresponding number of distinct multiples of the common input (which can be of any size; no matter what the precision of the common factor, there are still only 256 possible multiples when 8-bit weights are applied). In this case, there is a significant advantage to implementing a circuit that produces all the required outputs at once using many fewer elements than the same number of unlimited multipliers.

[0034] In one embodiment of the present invention, an equivalent large multiplication is dedicated to a single input channel, not necessarily shared, so the operation has the option of using several clock cycles and multiple register stages, allowing the operation to take a very simple and efficient form without impacting the overall throughput of the system.

[0035] In the general case where a single dynamic value is multiplied by many constants, replacing a single, multi-stage large multiplier circuit, such as in embodiments of the present invention, with an equivalent set of independent single-stage multiplier circuits results in a system that performs the same calculation with substantially higher throughput and substantially lower power and footprint. Even if the set of outputs is smaller than the number of actual multiples used, significant savings in power and space may still be possible.

[0036] Having established a clear advantage of the unique large scale multiplier in one implementation of the present invention compared to a standalone multiplier, this advantage can be further increased by changing the order of the sequence of operations.

[0037] The mathematics of algorithms in neural networks (or other similar image processing) does not require any particular sequence of operations: the same operations can be performed in any order and will produce the same exact calculation.

[0038] The inventors of the present invention have observed that the normal order for software running on a CPU, GPU, or TPU-based design is to simultaneously generate all output channels for a given position by multiplying the weights by the inputs and adding them immediately. Generating all output channels for a given position simultaneously by multiplying the weights by the inputs and adding them immediately minimizes the number of times the inputs must be read from RAM, and also limits the number of times the weights must be read from RAM. It does not eliminate multiple reads of the inputs, since there is no place other than RAM to keep them when processing the next row below.

[0039] However, in one embodiment of the present invention, if the operation order of a kernel or other aperture function defined to operate on an M×N patch of array inputs is reversed, i.e., effectively inverted, each input value is used only once and no RAM buffer is required. Instead of generating outputs one at a time by redundantly reading the inputs as the aperture function passes over each row, this unique operation processes multiple inputs at a time and keeps partial sums for all incomplete outputs only when first presented. The partial sums can be kept in hardware shift registers or standard hardware first-in-first-out registers (FIFOs), with the number of registers required to hold the kept values ​​proportional to the height of the kernel and the width of the input row.

[0040] Because the function implementing the aperture function can be decomposed into a series of subfunctions, each of which operates on the results of the previous subfunction, the implementation of the kernel can be achieved by sequentially composing the subfunctions over time, each of which operates immediately on the received data, resulting in the same series of operations as abstractly applying a kernel. We refer to this recomposed function, including any initialization, as the aperture function, and the individual steps as subfunctions. As used herein, aperture function refers to any M×N computation implemented at multiple locations over a sliding window or patch of M×N inputs of a larger R×C array of inputs. The aperture function may also include initialization and finalization operations, as in the implementation of a full CNN kernel. In the case of a CNN, initialization preloads bias values ​​into accumulators, and finalization transforms the raw output of the kernel via an optional activation function.

[0041] In this example of the invention, as the component of each new input position is presented, the component at that position represents the first element of the patch below and to the right, and simultaneously the last element of the patch above and to the left, and intermediate elements of all other patches that intersect with the current position. This allows, as one embodiment of the invention, a computational circuit to be developed that always has a fixed number of elements in progress (with some possible exceptions near the edges of the input), and that produces output as fast as it accepts input.

[0042] When the algorithm for guidance requires evaluation of the aperture function on patches that extend beyond the edges of the input array, many special cases and challenges arise, but they are not insurmountable. Special case logic can be added so that partial results for overlapping patches are compatible with the general case without affecting overall throughput.

[0043] In an implementation of the present invention, the operation of this inverted form of the aperture function receives input as a stream and produces output as a stream. The inputs do not need to be buffered in RAM because they are each referenced only once. Because the outputs also exist as a stream, they can be processed by subsequent layers without being buffered by RAM, a result attributable to the present invention that substantially improves processing speed compared to the required reads and writes to and from many other RAMs.

[0044] In one embodiment of the present invention, instead of many layers sharing a set of independent multipliers that operate, store, and then read the results back out to process the next layer sequentially, a pipeline can be created using a dedicated large multiplier that processes all layers simultaneously, feeding the output stream of each layer into the input of the next layer, without waiting for any layer to complete.

[0045] A fully implemented pipeline in one embodiment of the present invention can thus achieve an effective throughput measured two orders of magnitude better than traditional output-centric ordering processes, eliminating contention for RAM (because it does not use RAM), which forms the primary bottleneck for GPU- and TPU-based processing.

[0046] The latency of such a system, in one embodiment of the present invention, is reduced to the time from the input of the last pixel to the output of the final result. Because the last pixel of the image must, by definition of the algorithm, be the last data required to complete all of the final calculations for all layers, the latency of the system is exactly the product of the clock rate and the number of distinct clock stages in the pipeline that comprise the final output.

[0047] The use of a single dedicated massive multiplier for each input channel throughout the neural network in one embodiment of the present invention (instead of a limited set of independent multipliers that must be reused and dynamically allocated) makes it possible to build a pixel-synchronous pipeline in which all multiplications are performed in parallel, since only one massive multiplier is needed to handle any number of weights that are applied.

[0048] Having described above the essential features of the large scale multiplier innovation, and also the advantages of inversion, the inventors of the present invention now present a specific example: FIG. 1 illustrates one implementation of the present invention, in which each of a plurality of one or more source channels 1 through N is labeled 101a through 101d and assigned a dedicated large-scale multiplier 102a through 102d. Because each source channel in this example has a dedicated large-scale multiplication circuit that generates a set of multiples of that channel's value, the format of the source channels can vary between signed, unsigned, fixed, or floating point, at any precision convenient for the processing algorithm implemented in hardware. The specific output of each large-scale multiplication circuit, such as large-scale multiplication circuit 102c, can be fed directly into one or more computation units 103a through 103d, which can perform computations requiring multiples of any or all of the source channels. These computation units can be used to implement independent output channels of a single or unrelated algorithms computed on the same source channel. The output of the computation can be forwarded, as indicated at 104, for further processing, as may be required by one or more algorithms implemented in hardware. This situation arises, for example, when implementing neural networks in field programmable gate arrays (FPGAs) where the weight values ​​applied as multiplicands do not change.

[0049] Figure 2 illustrates one embodiment of the present invention in which the output of each massive multiplier, such as massive multiplier 102a of Figure 1, is fed into computation units 203a-203d via a set of multiplexers 201a-201d, such that the selected multiplier is chosen at system initialization or dynamically as the system operates. The output of the computation may then be forwarded for further processing at 204, as described above. This situation arises when implementing neural networks in application specific integrated circuits (ASICs), where the structure of the computation is left to chance but the weight values ​​used need to be changed.

[0050] FIG. 3 illustrates the internal structure of the large-scale multiplier 102a of FIGS. 1 and 2 in one embodiment. This structure may be common to large-scale multipliers 102b, 102c, and 102d, as well as other large-scale multipliers in other embodiments of the present invention. In this structure, products 303a through 303f of an A-bit source channel multiplicand 101a and all possible B-bit multipliers are created in parallel and sent to a multiplier 304. In this example, the A bits of the source multiplicand 101a are duplicated, shifted by adding a 0 bit to the least significant position, and padded by prepending a 0 bit to the most significant position so that all sets of all necessary shifted values ​​from 0 to B−1 are available in the form of vectors of A+B-bit terms 302a through 302d. These terms can be formed simply by routing circuit connections; no registers or logic circuits are required. If the clock period is sufficient to allow the maximum value of the B terms of the A+B bits to be combined in a single period, registers or sub-combinations may not be required. The individual products of the added terms 303a through 303f may be stored locally in registers or forwarded for further processing as combinatorial logic. B Each product of -1 can be formed by adding any or all of the corresponding terms 302a through 302d of B whenever a 1 bit occurs in each multiplier. Any source multiple 0 is an all-0 constant and may be included in multiple 304 for completeness when using multiplexers, but does not otherwise require circuitry. Any unused products 303a through 303f can be omitted, either by excluding them from the circuit specification to allow a synthesis tool to delete them, or by any other method. Unused terms 302a through 302d can also be omitted, but this generally has no effect because they do not occupy logic. In this way, all necessary multiples 304 of source multiplicand 101 can be formed as a single-stage pipeline or as combinational logic.

[0051] FIG. 4 illustrates an optimized embodiment in which a set of terms 401 is composed of all necessary individual terms 302a through 302e, including 0 through B, formed by A+B+1 bits. This allows products 402a through 402f to involve subtraction from larger terms instead of addition of smaller terms, which can be used to reduce the overall size of the circuit, potentially increasing the maximum allowable clock frequency. For example, for any given input a and multiplier 15, 8a + 4a + 2a + 1a = 15a combines four components, while 16a - 1a = 15a combines only two and is generally expected to be more compact and efficient. Each product 402a through 402f can be composed of any addition and subtraction of terms 302a through 302e that produces the correct result, and each particular variant can be chosen based on the optimal tradeoff for a particular implementation technology. For example, subtracting two N-bit quantities may require more logic than adding two N-bit quantities, but in general, adding three N-bit quantities will always require more logic than subtracting two. The processing of required multiples 304 is not altered by the details of combining the individual products 402a through 402f.

[0052] FIG. 5A illustrates one embodiment of a large-scale multiplier in which the clock cycle is such that only a single addition of an A+B bit value (or A+B+1 if subtraction is used) is possible per cycle. In this case, it is necessary to arrange the necessary elements in a multi-stage pipeline to accommodate multiples using more than two terms. Terms 401 are formed from each source channel 101 as described above, but are held one or more times in pipeline registers 501a and 501b for later reference. Pairs 502 of added two terms are calculated, registered, and then saved 503 as needed. Triples 504 are formed as the sum of pairs 502 and the held terms 501. Term value quads 505 are formed as the sums of pairs 502. Any unused elements may be omitted, and to increase overlap, only descending sequences of addends may be specified. This ensures that redundant sums, e.g., a+b and b+a, are not both used and kept in the final circuit. The products 506a-506f can use any addition or subtraction operation of any pair of registered subcomposites that meets timing constraints. Consistently using the largest available elements can reduce overall size, and therefore power, but any combination of operations that produces a correct result is acceptable.

[0053] The embodiment of Figure 5A is sufficient to generate all required multiples where B = 8. For larger sets of multiples, the sub-compositions shown can be re-composed in further pipeline stages so that all required multiples 506a-506f for any value of B consist of a single clock operation on an expanded set of sub-compositions including the previously disclosed retained term 501b, retained pair 503, triple 504, and quad 505, along with other sub-compositions as needed to form a set of terms sufficient to form multiples 506a-506f by a single clock operation.

[0054] FIG. 5B illustrates an embodiment in which multiples are formed directly by a fixed set of cases without reference to standard arithmetic operations. For each required multiple, a set of output values ​​a*b is enumerated for each source channel value a. This allows a hardware circuit synthesis tool to determine the optimal logic circuit 507 for generating the complete set of required multiples. The specification of the required output values ​​for any given input value is typically created by enumeration in a Verilog "case" or "casex" statement. This is distinct from a lookup table in which output values ​​are stored and accessed via an index formed from the input, because logic gates are used to implement the smallest subset of operations required to generate the complete set of output values, and redundant logic used to generate the associated sub-expressions is synthesized.

[0055] Whether Method 5A or 5B is most efficient in terms of space, frequency, and power depends on the core efficiency between arithmetic operations and arbitrary logic, as well as the particular values ​​of A and B. The choice of which method to use may be based on direct observation, simulation, or other criteria.

[0056] 6 illustrates an embodiment in which clock periods allow composition by addition and / or subtraction of four elements during each single clock period with sufficient levels of logic. By selecting from a set of sub-compositions, each of products 605a through 605f can be generated by combining as few as four registered elements. As before, terms are held in registers 501a and 501b, but triple 601 held in 602 is composed directly from term 401, without using pairs. Septet 603 and octet 604 are formed from triple 601 and the stored term 501a.

[0057] The exemplary embodiment of Figure 6 is sufficient to generate all required multiples where B = 32. For larger multipliers, the sub-combinations shown are recombined four at a time in further pipeline stages to generate all required multiples for any value of B. While the elemental sub-combinations shown are necessary and sufficient to generate all products where B = 32, other sub-combinations (perhaps chosen for consistency across different values ​​of B) are acceptable.

[0058] When the set of multipliers is fixed, as is common in FPGA applications, even a large coarse set of multipliers can be implemented efficiently because common elements can be merged and unused elements can be deleted. When a synthesis tool performs this function automatically, the circuit representation can include all possible elements without explicitly declaring which multipliers will be used.

[0059] If an operation on an A+B or A+B+1 bit value cannot be completed in a single clock cycle, a multi-stage pipeline adder can be inserted for any single stage of composition logic, with additional pipeline registers inserted as necessary so that all paths have the same number of clock periods. A pipeline stage period can be an instance of a single edge-to-edge clock change, or multiple clock cycles if throughput constraints allow. Neither multiple clock stages per operation nor the use of multiple clock cycles requires structural changes to any embodiment, other than the issues just mentioned.

[0060] An important object of the present invention is to provide the industry with a large scale multiplier implemented as an integrated circuit for use in a variety of applications. To this end, the inventors of the present invention provide, in one embodiment, a large scale multiplier implemented as an integrated circuit having ports that receive a stream of discrete values, circuitry for simultaneously multiplying each value received at the ports by a plurality of weight values, and an output channel that provides the generated large scale multiplier product.

[0061] In one version, the received discrete values ​​may be unsigned binary values ​​with a fixed width, the weight values ​​may be unsigned binary with a fixed width of 2 or more bits, and each multiple may be synthesized as a sum of bit-shifted replicas of the input. In another version, the set of shifted replicas may be increased to allow the use of subtraction operations to shrink or otherwise optimize the circuit. Unused outputs from the set may be deleted, either explicitly or implicitly.

[0062] In one embodiment, the output product set may be generated by combinational logic. In another embodiment, the output product set may be generated by a single-stage pipeline using a single or multiple clock cycles. In another embodiment, the output multiple set may be generated by a multi-stage pipeline by combining as few as two addends per stage. Unused elements of intermediate sub-compositions may be removed from the circuit either explicitly or implicitly.

[0063] In one embodiment, the output product set may be generated by a multi-stage pipeline combining three or more addends per stage, and the sub-compositions may be adjusted accordingly. Unused elements of intermediate sub-compositions may be removed from the circuit either explicitly or implicitly.

[0064] Another object of the present invention is to provide large-scale multiplication as an integrated circuit for implementing substantially improved convolutional neural networks as deep learning and artificial intelligence progress. In this effort, the inventors of the present invention provide the first convolutional neural network (CNN) node implemented as an integrated circuit, which has a first input channel defined as a stream of discrete values ​​of the first component of the elements of the array.

[0065] In this description, the inventors intend the term element of an array to mean an element that can have a single component or multiple components. A suitable example is an image, which may have pixels as elements, in one example a single color or three color values, or if the image is RGB color, each pixel may have a single component. Each color value in this example is a component of the element, which is a pixel.

[0066] Continuing with the above description of a first convolutional neural network (CNN) node implemented as an integrated circuit with a first input channel defined as a stream of discrete values ​​of a first component of an array of elements, the CNN further includes a first large scale multiplication circuit that simultaneously multiplies the received discrete value of the first component by multiple weight values, and an output channel that provides an output stream of discrete values.

[0067] In some embodiments of a CNN node, the initial output stream is formed from the product of an initial large multiplier circuit, in some circumstances by combining the product with a constant, and in some circumstances by applying an activation function.

[0068] In another embodiment, the CNN node further comprises a second input channel defined as a stream of discrete values ​​of a second component of the elements of the array, and a second large scale multiplier circuit that simultaneously multiplies the received discrete values ​​of the second component by multiple weight values. In another embodiment, there may be a third input channel defined as a stream of discrete values ​​of a third component of the elements of the array, and a third large scale multiplier circuit that simultaneously multiplies the received discrete values ​​of the third component by multiple weight values.

[0069] While CNN nodes having one, two, or three input component streams and dedicated large-scale multipliers have been described above, the inventors of the present invention further provide a convolutional neural network (CNN) having a first CNN node implemented as an integrated circuit with input channels defined as streams of discrete values ​​of the components of an array of elements, large-scale multiplication circuits dedicated to the individual input channels and simultaneously multiplying the received discrete component values ​​by multiple weight values, and output channels providing an output stream of discrete values, and a second CNN node having an input at least partially dependent on the output of the first node. This CNN may have successive nodes and may operate as a deep neural network (DNN). It is not required that successive nodes after the first node be CNN nodes.

[0070] Pipelined aperture function operation Referring back to the description provided earlier in this specification that discussed the order of operations in processing a CNN or other similarly selected aperture function that passes an array of computational subfunctions against an array of inputs to produce a net result, a specific description is now provided of the inverted form of operation of the aperture function in one embodiment of the present invention, which receives inputs as a stream and produces outputs as a stream. In this embodiment of the present invention, the inputs are not, and do not need to be, buffered in RAM because each input is referenced only once. Because the outputs are also generated as a stream, the output stream can be processed by subsequent layers without RAM buffering. The inventors of the present invention believe that this innovation substantially improves processing speed compared to the many otherwise required RAM read and write operations in other processing systems.

[0071] In one embodiment of the present invention, an apparatus and method are provided in which the operation of passing a two-dimensional aperture function over a two-dimensional array operates on an incoming stream of inputs, such that all inputs are processed immediately, partially completed computations are held until all necessary inputs have been received and processed, and outputs are generated as an adaptive stream, typically having the same or a lower data rate than the input stream. All inputs are received and processed at the rate provided and are not required to be stored or accessed in any order other than the given order. Even if the application of the aperture function is defined to generate more outputs than inputs, the circuitry can still operate at the rate of the incoming data by selecting a processing clock rate that is sufficiently increased so that the system never fails to receive and process an input at a given time.

[0072] The traditional way to implement the convolution of a kernel, or more general aperture function, with a larger input array is to collect the necessary input patches, apply the function to the input, and output the result. As the aperture is passed over the input array, each subsequent patch overlaps with the one just processed, so some input can be retained and reused. Various mechanisms, such as FIFOs, can be used to avoid redundantly reading input from source storage as the patch advances to each new row, but the source data will still be applied to each location in the kernel to generate each output in turn where that input patch overlaps each particular data input location.

[0073] If there are many output channels and many independent aperture functions to be calculated, a large scale multiplier can be used to consider all of the aperture functions in parallel and provide products of patches of input values. However, with this configuration and order of operations, each location of the source data will require a set of products for each location in the kernel as it is combined into various overlapping output locations.

[0074] The mechanism of the present invention is to invert, i.e., reverse the order of operations, for the special advantage of using a single large multiplier applied to a given input value only once per input channel. Rather than retaining or re-reading source values ​​for later use in the form of later product calculations, the process in one embodiment of the present invention calculates all necessary products for each input as they are given, and keeps current sums for each element of the aperture function that are complete up to the time the current input occurs.

[0075] Any aperture function that can be mathematically decomposed into a series of sub-functions applied sequentially can also be implemented in this way. This mechanism can be easily applied because the CNN kernel is nothing more than a series of sums of products of weights and inputs, and the order of operations is compatible with the order of the source inputs, which is obtained from left to right and top to bottom.

[0076] In one embodiment of the present invention, an array of compositors corresponding to sub-function elements of the aperture function is implemented on the IC, each maintaining a running total of the aperture function's values ​​as it progresses through the input stream. The final compositor in the array outputs the complete value of the function, and all other compositors output partial values ​​of the function.

[0077] In the simple case of applying a 3x3 kernel, the output of the top-left compositor reflects the sum of the first element of the kernel applied to the current input plus any initialization constants, the output of the top-middle compositor reflects the first two steps, and the output of the top-right compositor reflects the first three steps. The output of the top-right compositor needs to be delayed until it can be used again by the next line. The next line of the compositor continues the pattern of taking the partially completed function value, adding the contribution of each new input, and moving it forward. The last line of the compositor completes the last step of the function and outputs the completed value for any further processing.

[0078] Noting that the progression of partial values ​​of a function through compositors is generally left-to-right in the first row, and then in subsequent rows, ultimately to the final compositor in the last row, the flow of partial values ​​can be thought of as a stream, and compositors and flows can be referred to as upstream or downstream.

[0079] Over time, each compositor maintains a partial sum of aperture functions up to and including the current source input. Each compositor is always operating on a different patch position at the output, specifically the patch position where the current input occurs relative to the compositor's position in the aperture subfunction array.

[0080] The 3x3 kernel W is a function of the input A.

[0081]

number

[0082] The circuitry required to compute these subfunctions is then configured in the corresponding arrays of the compositor:

[0083]

number

[0084]

number

[0085] where a i is the current value from the input stream, and a i-1 From a i-8 In each case, a i is the previously processed input for a particular patch that occurs at a position relative to the output of each individual compositor. Each compositor will calculate the value of the aperture function up to and including its corresponding position in the aperture array. Each compositor takes the current value of the input stream and combines it with the previous value to produce a different partial sum that corresponds to a partially processed patch in the input array, where the current input value occurs at the relative position of that patch that corresponds to each compositor's position in the aperture function.

[0086] In this way, partial values ​​of the aperture function calculated in standard order and precision will be maintained in the input stream over time until the completed values ​​are ready to be output.

[0087] While this technique is quite straightforward within the input array, complications arise when applied to patches that overlap the edges of the input array because the aperture function is defined differently when not all inputs are available. In the case of CNN kernels, additional operations are omitted, as is the case when using zero as an input. The present invention is interested in maintaining a steady flow of partial sums through the compositor while handling such exceptions, as will be described below.

[0088] FIG. 7 is a diagram illustrating the structure and connectivity of one embodiment of the present invention that receives an input stream, pre-processes the input stream, and passes the results through a unique digital device to generate an output stream.

[0089] Input channel set 701 and associated control signals 702 are used by common circuitry 703 to generate any products of the input channel set with weights for subsequent sub-functions. The source channel products are then distributed to a bank of sub-function computation circuits 704a, 704b, and 704c, each of which generates a single channel of output channel set 705. Any number of independent output channels can be supported by common circuitry 703.

[0090] FIG. 8A illustrates large-scale multipliers 801a, 801b, and 801c in the common circuitry 703 of FIG. 7, which take each channel of the input channel set 701 and generate either a loose or complete set of multipliers, depending on the requirements of the sub-functions being defined. It should be noted that this illustration assumes three channels in the input channel set, such as for red, green, and blue pixel values ​​when processing an RGB image. In other embodiments, there may be one, two, or more than three channels. Any or all of the products 802 (multiples of the source input array values ​​constructed by the large-scale multipliers) may be available to a compositor, shown in FIGS. 9A, 9B, and 9C, described in enabling detail below. The compositor is an example of hardwired circuitry in the unique device of the present invention that performs sub-functions on the products of the source channels generated by the large-scale multipliers of FIG. 8A.

[0091] FIG. 8B is a diagram illustrating the structure of the synchronization circuitry that provides normal and exceptional processing signals to all compositors of all output channels.

[0092] Control circuit 803 synchronizes all output and control counters to the source input stream and ensures that output and control counters are set to their initial state whenever RST or INIT is asserted.

[0093] The colSrc counter 805 in this example is calculated by counting the interior dimensions of the array column by column across the rows, and advances as each pair of source channel products is processed. In this example, at the end of each row, the colSrc counter returns to the leftmost position (0) and the rowSrc counter 804 advances by 1. At the end of the source array stream, the rowSrc and colSrc counters return to their initial states, ready to receive a new array of inputs.

[0094] In this example, colDst counter 807 and rowDst counter 806 work together in a similar manner for all output channels. The colDst and rowDst counters are enabled by output enable signal (DSTEN) 813, which determines when post-processing enable signal (POSTEN) 812 is asserted.

[0095] It should be noted that although the system shown in this example produces a single output of the aperture function, it would typically be used to produce a set of channel output streams compatible with the dimensions of the source input stream. Each independent output channel would share at least some of the computational circuitry via a large-scale multiplier and common control logic.

[0096] The output enable (DSTEN) signal 813 controls when the finalization function receives and processes results from the compositor. The first few rows are received from the source input array, but valid results are not provided to the finalization function (see Figure 9C). The output enable signal 813 (DSTEN) is asserted when the rowDst and colDst counters indicate valid results are available, or alternatively, when processing delayed truncated results. The POSTEN signal 812 is asserted continuously or periodically to match the timing of the SRCEN signal 801. These signals are required to sequence the final output of all truncated compositors when processing the last row of the source input stream array.

[0097]

number

[0098] In this example, the POSTEN and DSTEN signals and colDst and rowDst counter values ​​are independent of the SRCEN signal and colSrc and rowSrc counter values, and processing of delayed results continues until all delayed results are finalized and sent to the output stream. The system can accept new input until the previous output is completed, allowing the system to process multiple frames of the source input stream without pausing between frames. POSTEN is not asserted while the source stream data has not yet reached the end of the array, and final results are retrieved from the compositor. Immediately after reaching the end of the source array, the POSTEN signal is asserted for each additional output, and final results are obtained from truncated delay lines 909, 910a, and 910b until the rowDst counter reaches the full number of output rows, as shown in Figure 9C below, at which point rowDst and colDst are reset to their initial states in preparation for the next frame of data.

[0099] The first row signal 808 (ROWFST) is asserted when the rowSrc counter indicates that the source data set from the stream represents the first row of the array.

[0100] The last row signal 809 (ROWLST) is asserted when the rowSrc counter indicates that the source data set from the stream represents the last row of the array.

[0101] The first column signal 810 (COLFST) is asserted when the colSrc counter indicates that the source data set from the stream represents the first column of each row of the array.

[0102] The last column signal 811 (COLLST) is asserted when the colSrc counter indicates that the source data set from the stream represents the last column of each row of the array.

[0103] Figures 9A, 9B, and 9C illustrate the unique device described above in a general case in which MxN subfunction elements of an aperture function are applied to each overlapping MxN patch of an array of RxC inputs, including those that overlap the edges, where these inputs are provided as streams of related components at regular or irregular time intervals to produce a corresponding stream of RxC outputs, each output being the collective effect of the MxN function elements applied to the input patches as specified by the aperture function rules. The function elements applied to each location in the array are, in this device, hardwired compositors for each of the MxN subfunctions, as shown in the composites of Figures 9A, 9B, and 9C.

[0104] The effect of this circuit is to calculate a recombined value of the aperture function at each location of the RxC input array, using the same series of operations that would be used to calculate the aperture function on each patch individually. If any locations are undesirable in the output stream, circuitry can be added to remove them, producing a tiled or spaced output rather than perfectly overlapping.

[0105] Source channel products 802 and source control signals 814 are available to each of compositors 901, 902a, 902b, 902c, 903a, 903b, 903c, 904, 905a, 905b, 905c, 906, 907a, 907b, and 907c. The source control signals are also connected to delays 908a, 908b, 908c, 908d, 908e, and 908f. Output channel controls and counters 815, along with delays 909, 910a, and 910b, are also available to finalization function 911. Additional pipeline stages may be inserted, either manually or by automated tools, to make the circuit routing appropriate for a given clock frequency, provided that, and only if, the order of operations is not changed. Timing control and counter signals are available to all elements of the circuit and are not shown separately.

[0106] Each compositor has a dedicated direct connection to either a specific input product or, alternatively, a programmable multiplexer that selects one of the products for each input value in the set and is preset before execution of the circuit. Each dedicated connection is a parallel path with enough wires to carry the bits representing the desired product in a single input interval. The optional use of preset multiplexers that select which product is sent to each compositor for each set element allows for field upgrades of weight values. Fixed connections are used when weights are not upgraded and remain fixed throughout the life of the device. Because the weight selection does not change during operation, choosing between fixed and variable product selections does not affect the operation of the circuit.

[0107] Each compositor receives a set of products corresponding to the subfunction weights from the large-scale multiplier, one for each input channel, and performs the subfunction calculations, typically by simply adding them all together, to form that compositor's contribution to the overall aperture function value. Each compositor also receives partially completed results from the adjacent compositor to its left, except for the one corresponding to the left column of the aperture function. Each compositor may also receive delayed, partially completed results from the compositor in the row above, except for the one corresponding to the top row of the aperture function. Each compositor has at most one connection from the left and one delayed connection from above, but each of these connections is a parallel path with enough conductors to carry the bits representing the partially completed result as input to that compositor. Depending on the subfunction definition for the position of the current input patch relative to the edges of the input array, each compositor performs one of three operations: combine this compositor's partial result with the initial value, if present, or combine this compositor's partial result with the partial result from the compositor to its left, or combine this compositor's partial result with a delayed partial result. The corrected result is placed in an output register of multiple bits sufficient to contain it, making it available to the right compositor and / or delay and finalization circuitry in successive input intervals. This corrected result can be either a partial result, a complete result, or a truncated result, depending on the compositor's position in the aperture function and the state of the input stream position.

[0108] Compositor (0,0) is unique in that there is no compositor to its left or above it in the aperture function, and therefore always initializes the calculation with each set of inputs received.

[0109] Compositor (M-1, N-1) is unique in that the result it produces is always the final result, but is structurally identical to all other compositors 903a, 903b, or 903c.

[0110] The output of some compositors is used for delay or post-processing, in which case the path through such delay or post-processing is wide enough to carry the bits representing this partial, truncated, or completed result. The output of some compositors is used only by the right-hand compositor. Calculations internal to the compositor and the output data format do not require changes depending on the use of the output.

[0111] The finalization circuit takes results from several possible sources and multiplexes them to select which one to process in any interval. After applying the finalization function, if any, the width of the final output may be reduced to form the output stream of this embodiment, which may either be the next input stream, the final output of the system including the present invention, or an output that can be used for further processing.

[0112] Data paths in unique devices in implementations of the present invention are shown in Figures 9A, 9B, and 9C by bold lines with directions indicated by arrows, and ellipsis symbols indicate where the last column or row in the range is repeated in its entirety. Data path (a) from the source channel products 802 is a set of parallel conductive paths, one path dedicated to each product of input components, each product being the input component multiplied by one of multiple weight values ​​of the aperture function. It should be clear that a 5x5 aperture function has 25 weight values ​​for each input component. In the context of an aperture function for an RxC input array of R, G, and B color pixels, there are 75 weight values. The illustrated line (a) has 75 parallel paths in this context, each path being a set of parallel conductors with a width accommodating the number of bits of precision desired. Line (a) is referred to in the art as a set of point-to-point connections, as opposed to a bus.

[0113] Data paths (b) in Figures 9A, 9B, and 9C are not extensions of lines (a), but are dedicated connections to a particular subset of the paths in lines (a). Lines (b) are not necessarily marked in every example in Figures 9A, 9B, and 9C, but any connection from line (a) directly to a particular one of the compositors is a dedicated line (b). By dedicated, we mean that each compositor is connected to its subset of the paths that carry the products of each input component and the weight values ​​required by that compositor.

[0114] Data paths (c) in Figures 9A, 9B, and 9C are point-to-point paths between the output register in each compositor and the next compositor to its right. These are dedicated paths of precise width that typically carry partial sums, as described in enabling detail elsewhere in this specification. While not every path (c) is marked in the drawing, in this example, any direct connection from one compositor to another can be assumed to be a path (c). Note that there are cases where output paths (c) branch off into alternative circuit configurations.

[0115] Another unique data path in one embodiment of the present invention is marked (d) in Figures 9A, 9B, and 9C. These are dedicated data paths from delay circuits, such as circuits 908A through 908f, either back to the compositor one row down and to the left, or directly to another delay circuit. The delay circuits are configured to receive partial sums at the right end of the compositor row, delay passage for the partial sums by a specific number of source intervals, and then forward those partial sums to other compositors and / or other processing at the appropriate times. The overall functionality is described in enabling detail elsewhere in this specification. Paths (d) between delay circuitry are also dedicated paths for partial sums typically forwarded at specific source intervals.

[0116] If either M or N is shrunk so that the last row or column of a range is not needed, the final element is omitted, and the implementation of the first row or column in the range is retained. In the degenerate case where one or both of M or N are shrunk to 2, the first and last row or column are retained, and the intermediate rows and columns are omitted. In the degenerate case where one of M or N is shrunk to 1, the first and last compositor implementations are combined, and no special initialization is required. In the special case where both M and N are 1, no inversion of the aperture function is required, but the use of a large multiplier still provides a clear advantage.

[0117] The source channel product 802 can be any set of binary values ​​simultaneously presented in relation to a specific position in the R×C array and in some predefined sequence. The source channels of the input stream can be any combination of integer or fractional values ​​in any format defined by any property for the aperture function input. One example is feature component values ​​generated as the output of a CNN layer, along with pixel values ​​from one or more video frames and / or any other sensor values ​​scaled to match the array size R×C. It is emphasized that each node implementing the present invention can accept outputs from other nodes in addition to or instead of primary source inputs. In one embodiment of the present invention, while it is common for the first node or nodes to accept image pixels as the primary input to the system, there are no restrictions on the nature of the data processed, as long as it can be formatted into a stream representing an R×C array.

[0118] In some embodiments of the present invention, the source stream element sets may be presented in row-first order, with each successive column presented in strictly ascending order. In some embodiments of the present invention, the rows and columns need not correspond to horizontal or vertical axes, but may be arbitrary, such as scanning columns up or down and right to left. Rows R and columns C here simply refer to the major and minor axes of the stream format. The circuitry need not be adjusted for input signals that produce input streams in an orientation other than the standard video left-to-right, top-to-bottom order. The aperture subfunction orientation can be followed to produce the same output for each input array position.

[0119] In this example, source inputs, which are the product of source values ​​and weights as required by the aperture function, are presented by a signal (SRCEN, see FIG. 8B) that indicates when each new set of elements is valid. Inputs can be stopped and restarted at any time. In some examples, a minimum spacing between inputs can be defined, so that the circuit can use a multi-cycle or faster clock to reduce size and power or gain benefits, and the output channel set can use the same minimum spacing.

[0120] Common control and synchronization circuitry 803 (FIG. 8B) provides counters and control signals that describe the current input position in the R×C array. The counters continue to operate for additional rows and columns after the final input, helping finalization function 911 (FIG. 9C) output the cumulative output generated beyond the input column by the last row of input (see FIGS. 12, 13, and 14 and the description below). Control signals are available to all other elements and are not shown in FIGS. 9A, 9B, and 9C.

[0121] Compositor circuits 901, 902a, 902b, 902c, 903a, 903b, 903c, 904, 905a, 905b, 905c, 906, 907a, 907b, and 907c each calculate that portion of the aperture function assigned to their position in the M×N function. All compositors operate on the same set of source channels and row and column counter states provided by control 803. Details of aperture function data processing are further described below with reference to additional figures.

[0122] As a set of source inputs is received from the input stream, partially completed calculations of an aperture function to be applied to all patches overlapping the current position in the input stream are sent from left to right and top to bottom within the compositor's M×N array. This operation accumulates complete calculations of the aperture function over time and outputs an accurate implementation of the aperture function on each patch of the input array, producing the same result with the same order of operations as if the aperture function were implemented by reading the input values ​​directly from the array. Replacing random accesses to the array with stream accesses is a key feature of the present invention, eliminating the need for redundant accesses to random-access memory.

[0123] Compositor right column

[0124]

number

[0125] When processing the last column C-1 of each input row,

[0126]

number

[0127] In this example, the compositor 903c at the (M-1, N-1) position always produces a complete accumulation of M by N subfunction elements, but is otherwise indistinguishable from the other compositors in that configuration 903c. As noted above, when processing the last column C-1 of each input row, the column on row M-1

[0128]

number

[0129] In this example, while processing the last row of the input, R-1,

[0130]

number

[0131]

number

[0132] FIG. 15 is a diagram illustrating the particular case of pipelined operations in one embodiment of the present invention implementing a 5×5 convolution node.

[0133] Source channel products 802 and source control signals (not shown here) are available to each of compositors 901, 902a, 902b, 903a, 903b, 904, 905a, 905b, 906, 907a, and 907b. The source control signals are also connected to delays 908a, 908b, 908c, and 908d. Output channel controls and counters are available to finalization 911, as well as delays 909 and 910a. Additional pipeline stages may be inserted, either manually or by automated tools, to make the circuit routing appropriate for a given clock frequency, if and only if the order of operations is not changed. Timing control and counter signals are available to all elements of the circuit and are not shown separately.

[0134] As each set of source channel products is presented in turn, each compositor selects the appropriate product to compute the sub-function corresponding to its position in the aperture function. Each 5x5 patch that intersects with the current position in the input array is corrected to include a calculation based on the product for that position. The net effect is that a single source stream of input is transformed into a parallel set of 5x5 streams of sub-computations that are sent between compositors, until each time all operations for the patch are completed, this occurs typically at compositor (4,4), and possibly at other compositors when processing the right or bottom edge of the input array.

[0135] Note that only the width of the input array affects the size of the delay elements, since each must delay the partial result for a number of source input intervals corresponding to receiving an input in one column and an input in the same column in the next row.

[0136] Figure 16 illustrates a 4x4 embodiment of the IC of the present invention. It is noted that the kernel can have an odd or even number of subfunctions in a row or column. This even version is degenerate in the sense that, because an additional line of output processing is omitted, element 910* shown in the general case of Figure 9C and shown in Figure 15 for the specific case of a 5x5 aperture function (odd number of rows and columns) does not occur at all.

[0137] Odd kernel sizes are symmetric around the center in both directions, but for even sizes the center is offset. In this implementation, the IC aligns the center to the position (

[0138]

number

[0139]

number

[0140] Other than these comments, the operation of the particular IC in FIG. 16 is as described for the other versions described.

[0141] Figure 10A illustrates the internal structure and operation of compositors 905a, 905b, and 905c of Figures 9A and 9B or 15 in one embodiment of the present invention. A source input set 1001 of stream values ​​in a channel set, either single or mixed data types as required by the aperture function, is used to calculate the contribution of each individual compositor by circuitry 1004.

[0142] Circuitry 1005 calculates an initial value of the sub-function using the output of 1004. Circuitry 1006 calculates an ongoing partial value of the sub-function using the output of 1004 and a previously calculated partial value 1002 by the adjacent compositor to the left. Circuitry 1007 calculates an ongoing partial value of the sub-function using the output of 1004 and a previously calculated and delayed partial value 1003 from one of 908a, 908b, 908c, 908d, 908e, and 908f in the row of the adjacent compositor above.

[0143] The operations of circuitry 1005, 1006, and 1007 may occur simultaneously (same clock cycle) with the operations of circuitry 1004, with their outputs shared, or may be implemented by a series of pipeline stages synchronized by the same clock.

[0144] Multiplexer 1008 selects which variant of the partial result is forwarded as the partial value of the subfunction as the output of compositor 1009. If COLFST 811 is not asserted then the output of 1006 is selected, otherwise if ROWFST 808 is not asserted then the output of 1007 is selected, otherwise the output of 1005 is selected.

[0145] This conditional processing is a natural consequence of allowing an M×N aperture function to extend across the edges of a source input stream representing a set of values ​​in an R×C array. A single location at the leftmost or topmost edge becomes the first computable element of the aperture function for several patches that touch or overlap those edges. Thus, each and every compositor at the first computable location of an overlapping patch is required to be initialized with the base value of the aperture function. Furthermore, each and every compositor at the first computable location of a subsequent row of that patch must be combined with the preceding value—the computed partial value of the same patch from the previous row. In this way, correct computation of all patches that overlap, touch, and are within the topmost and leftmost edges is guaranteed using a single circuit.

[0146] In Figures 10B to 10G, all elements introduced in Figure 10A and using the same reference numerals are functionally identical to those described with reference to Figure 10A.

[0147] Figure 10B is a diagram illustrating the internal structure and operation of compositors 902a, 902b, and 902c of Figures 9A and 9B or 15 in one embodiment of the present invention. A source input set 1001 of stream values ​​is used by circuitry 1004 to calculate the compositor's contribution to the aperture function.

[0148] Circuitry 1005 uses the output of 1004 to calculate the initial value of the sub-function, and circuitry 1006 uses the output of 1004 and the partial value 1002 previously calculated by the adjacent compositor to the left to calculate the ongoing partial value of the sub-function.

[0149] Multiplexer 1010 selects which variant of the partial result is forwarded as the partial value of the subfunction as the output of compositor 1009. If COLFST 811 is not asserted, the output of 1006 is selected, otherwise the output of 1005 is selected.

[0150] Figure 10C is a diagram illustrating the internal structure and operation of compositor 904 of Figure 9A or 15 in one embodiment of the present invention. A source input set of stream values ​​1001 is used by circuitry 1004 to calculate the contribution of each individual compositor.

[0151] Circuitry 1005 uses the output of 1004 to calculate the initial value of the sub-function, and circuitry 1007 uses the output of 1004 and a previously calculated and delayed partial value 1003 from one of 908a, 908b, 908c, 908d, 908e, and 908f in the adjacent compositor row above to calculate the ongoing partial value of the sub-function.

[0152] Multiplexer 1011 selects which variant of the partial result is forwarded as the subfunction's partial value as the output of compositor 1009. If ROWFST 808 is not asserted, the output of 1007 is selected, otherwise the output of 1005 is selected.

[0153] Figure 10D is a diagram illustrating the internal structure and operation of compositor 901 of Figure 9A or Figure 15 in one embodiment of the present invention. A source input set of stream values ​​1001 is used by circuitry 1004 to calculate the contribution of each individual compositor.

[0154] Circuitry 1005 uses the output of 1004 to calculate the initial value of the sub-function, which is forwarded as the partial value of the sub-function as the output of compositor 1009 .

[0155] Cell 901 (FIG. 9A, FIG. 15), if used, is always the first value in any full or truncated patch and therefore always generates the initialization value for that patch.

[0156] Figure 10E is a diagram illustrating the internal structure and operation of compositors 903a, 903b, and 903c of Figures 9B and 9C or 15 in one embodiment of the present invention. A source input set 1001 of stream values ​​is used by circuitry 1004 to calculate the contribution of each individual compositor.

[0157] Circuitry 1006 uses the output of circuitry 1004 and the partial value 1002 previously calculated by the adjacent compositor to the left to calculate the ongoing partial value of the sub-function, which is forwarded as the partial value of the sub-function as the output of compositor 1009.

[0158] Figure 10F illustrates the internal structure and operation of compositors 907a, 907b, and 907c of Figures 9A and 9B or 15 in one embodiment of the present invention. A source input set 1001 of stream values ​​is used to calculate the contribution of each individual compositor 1004.

[0159] Circuitry 1006 calculates the ongoing partial value of the sub-function using the output of circuitry 1004 and a previously calculated partial value 1002 by the adjacent compositor to the left. Circuitry 1007 calculates the ongoing partial value of the sub-function using the output of 1004 and a previously calculated and delayed partial value 1003 from one of 908a, 908b, 908c, 908d, 908e, and 908f in the row of the adjacent compositor above.

[0160] Multiplexer 1012 selects which variant of the partial result is forwarded as the partial value of the subfunction as the output of compositor 1009. If COLFST 811 is not asserted, the output of 1006 is selected, otherwise the output of 1007 is selected.

[0161] Figure 10G illustrates the internal structure and operation of compositor 906 of Figure 9A or 15 in one embodiment of the present invention. A source input set of stream values ​​1001 is used by circuitry 1004 to calculate the contribution of each individual compositor.

[0162] Circuitry 1007 computes the ongoing partial value of the sub-function using the output of 1004 and a previously computed and delayed partial value from one of 908a, 908b, 908c, 908d, 908e, and 908f in the upper adjacent compositor row in 1003. The output of circuitry 1007 is forwarded as the partial value of the sub-function as the output of compositor 1009.

[0163] 11 is a diagram illustrating the internal structure and operation of internal row delay lines 908a, 908b, 908c, 908d, 908e, and 908f (FIG. 9C). The delay lines are used to hold partially computed results from each row of the compositor for use in the next row.

[0164] When COLLST is asserted, the current position of the source input stream is at the right-most edge and

[0165]

number

[0166] The current position of the source input stream, colSrc, is

[0167]

number

[0168] The column position of the source input stream is

[0169]

number

[0170] The partial output selected by multiplexer 1106 is provided to a first-in-first-out (FIFO) circuit 1107 with CN positions, which is arranged so that source input stream positions are processed such that exactly one value is inserted, and the one value is removed in the same order that it was inserted. Because a partially completed result from one position is not required until the source input stream returns to the same patch position in the next row, this creates a delay so that the partial result calculated by one row is provided to the next row exactly when it is needed.

[0171] The partial output selected by multiplexer 1106 also provides the same value (1114) to delay lines 909, 910a, and 910b of the final result.

[0172] The partial output taken from FIFO 1107 is routed by 1108 to both the left-most compositor in the next row (1111) and to a series of parallel access registers 1109 to 1110 which further delay the partial output by one source input stream interval as the data is passed through the register chain.

[0173] When the current position of the source input stream is at the leftmost edge, the FIFO directs the output data at 1108 and the delayed results 1109 to 1110 are made available to the next row of cells at 1111, 1112 to 1113 respectively.

[0174] Note that when the source input array stream position is near the right edge, additional values ​​from the right side of the source input array stream inserted into FIFO 1107 by multiplexer 1106 are accessed only via path 1111, while when the source input array stream is in the leftmost position to access data inserted from path 1103 as usual, only additional parallel paths 1112 through 1113 are used. The apparent similarity in structure and requirements between processing the right edge and the left edge is a natural consequence of the symmetry of the overlap of subfunctions with the right and left edges of the source input stream array. When the value for N is even, the number of additional cells processed to support the right and left edges is not the same.

[0175] FIG. 12 is a diagram illustrating the internal structure and operation of the final truncated resultant delay line 909 (FIG. 9C).

[0176] When processing the last row of the source input stream array, the partial result from the auxiliary output 1201 of the internal row delay line 908d is considered to be the final result of the last row of the truncated patch and is kept in a FIFO 1202 whose number of elements C is equal to the width of the source input stream array.

[0177] Immediately after recording the final result of the truncated patch, the output of FIFO 1202 is forwarded to a further delay line 910a via 1203, or directly to final processing 911 if the value of M is such that no other delay lines intervene.

[0178] FIG. 13 is a diagram illustrating the internal structure and operation of the final truncated resultant delay lines 910a and 910b.

[0179] When processing the last row of the source input stream array, the partial results 1301 from the auxiliary outputs of the internal row delay lines 908e to 908f are considered to be the final result of the last row of the truncated patch and are kept in a FIFO 1304 whose number of elements C is equal to the width of the source input stream array.

[0180] When POSTEN is asserted, multiplexer 1303 switches between taking the value from 1302 and taking the value from the final truncated delay line of the row above, which has the effect of giving a final truncated result in the first order of the row that is compatible with the ordering of all previous output results.

[0181] Note that during the cycle of the input frame in which POSTEN is first asserted, the contents of FIFOs 1202 and 1304 are the final values ​​of the truncated patch that overlaps the last row of the source input stream array. Any suppression of execution when not processing the last row of the source input stream array is optional, since none of the data contained in FIFOs 1202 and 1304 is processed prior to that cycle.

[0182] Immediately after recording the final result of the truncated patch, the output of FIFO 1304 is forwarded to a further delay line via 1305, or directly to final processing 911 if the value of M is such that no other delay lines intervene.

[0183] FIG. 14 is a diagram illustrating the internal structure and operation of the final processing of all complete and truncated results.

[0184] As in Figure 11, the structure and functionality are identical, but if the current position of the source input stream is at the rightmost edge,

[0185]

number

[0186] The current position of the source input stream is

[0187]

number

[0188] During processing of the source input stream array, multiplexer 1402 provides 1403 the result selected by multiplexer 1106 directly to finalization. In the post-processing phase, the output of truncated result delay line 1401 is instead selected for finalization (1403).

[0189] The finalization circuit 1403 performs all additional computations, if any, and generates the final form of the output stream (1404) from the combined patch results. This may typically take the form of a rectified linear activation (RELU) function, where negative values ​​are set to zero and out-of-bounds values ​​are set to the maximum acceptable value, or any other desired conditioning function, such as sigmoid or tanh. The post-processing function is not required to complete within a single source input stream cycle, but is required to accept each final result at the rate of the source input stream array.

[0190] When DSTEN is asserted, the finalization circuit 1403 presents the final result as one value in the destination output stream. Any time DSTEN is not asserted, any partial or inaccurate values ​​produced by the finalization circuit 1403 are ignored, so any suppression of operations when the result is not used is optional.

[0191] In one implementation, the destination output stream array is processed by circuitry similar to that described above, in which case it is advantageous for the timing of the final truncated result to be identical to all previous final results. To that end, the control of FIFOs 1202 and 1304 is adjusted by control circuitry 702 to maintain an output rate identical to that of the primary output.

[0192] In another implementation, the destination output stream array is the final stage of the system and no further processing is required. In that case, it is advantageous to have the timing of the final truncated result completed as quickly as possible. To that end, the control of FIFOs 1202 and 1304 is adjusted by control circuitry 702 to output their results at the maximum frequency supported.

[0193] Note that the implementation described above generates a single output element from a full set of input elements. In a complete system that generates a larger set of output elements from an input set, the entire mechanism described would be replicated once for all output channels, with the notable exception of control circuitry 702, which may be shared by the output channels, since the timing of all individual sub-functions would be identical for the entire output set.

[0194] The inventors of the present invention have constructed a working prototype of an IC according to one embodiment of the present invention to test and confirm the details and features of the present invention, and the operation of this prototype confirms the above description. The inventors of the present invention have also developed a software-supported simulator, which has been used up until the time of filing this application to test and confirm the above details and description.

[0195] In another aspect of the invention, a system is provided for receiving an input stream of three-dimensional data, as is common in medical imaging, where additional circuitry and buffering is included to allow a three-dimensional aperture function to be passed over the three-dimensional input array, with corresponding calculations that accurately implement both interior and edge cases for the first and last planes.

[0196] In yet another aspect of the present invention, for the complex process of training a deep neural network (DNN), a hardware-assisted neural network training system is provided in which the training algorithm simply uses statistics collected from the forward inference, with the forward inference engine investing a significant amount of effort to periodically adjust the weights and biases for the entire network to converge the model to a desired state. By adding appropriate accumulators that sum the input states as the forward inference process is computed, the present invention forms a hardware-assisted neural network training system.

[0197] In yet another aspect of the present invention, in response to the widely known problem that floating-point precision limitations hinder the convergence of DNN models (known in the art as the "vanishing gradient problem"), a single large multiplier with limited bit-width precision is provided, which can be cascaded with additional adders to generate floating-point products of arbitrarily large precision. While this innovation is not typically required for forward inference calculations, it is crucial in DNN trainers to avoid problems that arise when the computed gradient becomes too small to measure.

[0198] N-up parallel processing The above-described embodiments and implementations of the present invention have focused on an apparatus and method for large-scale multiplication when performing functions that require multiplication, and on the implementation of aperture functions by novel ICs in convolutional neural networks (CNNs). However, it is well known in the art that a complete deep neural network (DNN) must necessarily implement a full set of entirely different aperture functions, many of which may require minimal computation.

[0199] To qualify as an embodiment of the present invention, each such implementation must conform to an overall system-wide pipeline format that accepts input as a stream of parallel values ​​representing an array in a consistent order, and simultaneously produces output as a stream of parallel values ​​representing the array in the same order. The final node of the DNN may return a conclusion that reflects an array of positions, or a conclusion regarding the entire input array. The embodiments of the present invention described below are for executing DNNs in novel ICs that support pipelined execution.

[0200] In one aspect of the present invention, the inventors have developed methods and apparatus for significantly accelerating pipelined computations in CNNs and DNNs. The inventors propose streaming multiple inputs to an IC in parallel in some embodiments of pipelined computations. In the above-described embodiments, inputs are typically streamed left to right across each column, then top to bottom across rows, in all implementations. Taking RGB data as an example, this takes the form of three separate channels at each pixel location, typically 8 bits each, representing each of the three independent RGB color values ​​observed at each pixel location. The inventors refer to this as a 1-up implementation. 1-up means streaming input values ​​for one pixel at a time, or, more generally, streaming values ​​for one input location at a time in an input array.

[0201] The inventors of the present invention believe that significant advantages can be achieved by streaming inputs at multiple input locations at once, as in the pixel example. To do so, circuitry must be added to the new IC that executes the input stream to generate the output stream. The circuitry implemented in the 1-up environment is repeated within the IC to process input values ​​in parallel for the additional input locations (pixels in this example), so the change is generally one of size rather than complexity.

[0202] Circuit configuration is least problematic when the width of each row is an integer multiple of the input count to be streamed in parallel, but this is not a necessary limitation of the present invention. In the pixel example, for a resolution of 1920 x 1080, the number of pixels across a row (1920) is evenly divisible by 1, 2, 3, 4, 5, 6, 8, 10, and 12. Therefore, streaming RGB values ​​for two pixels, called 2-up, as well as 3-up and 4-up, are efficient techniques. As the number of pixels increases, the overall size of the IC to handle all the processing increases by a factor directly related to the number of pixels to be considered in parallel, so users must make rational decisions.

[0203] However, as the stream progresses through the nodes of the DNN, the input array size is often reduced in dimension if the stride of the aperture function is other than 1 (not all input positions generate an immediate output position) or if the aperture function is defined to avoid overlap with the edges of the input array. In these general cases, the width of the input array cannot be constrained to an integer multiple of any given number N of parallel positions. One solution is to always align the left edge of each row of the input array with a specific position (nominally the left side) in the set of N positions. The right edge can then be represented by an incomplete set that always starts at the first position in the set of N positions. Additional circuitry is then used to avoid using invalid data in the computation and also to suppress outputs derived from that invalid data.

[0204] In an embodiment of the present invention, for a 2-up implementation of the pixel example, the R, G, and B values ​​for each of two adjacent pixels are streamed into the IC as pipelined inputs. The first two pixels are the first two pixels from the left in the top row. In the RGB example, there are six input values: the R, G, and B values ​​for each of the first two pixels. The next two pixels in that row are next in the stream, and so on, proceeding across the top row, then the R, G, and B values ​​for the first two pixels in the second row, and so on through the input array. The same general protocol is followed for 3-up or 4-up.

[0205] Figures 17A and 17B show a 1-up pipelined solution for a well-formed, minimal DNN model that can be used to understand images and respond with the relative excitation intensities of various objects the model is trained to recognize. The input channels 1701, as described above, are presented as input values ​​for individual pixels in a specific order, typically from left to right across each row, then from top to bottom. For RGB data, this takes the form of three separate channels, typically 8 bits each, representing the three independent color values ​​observed at that location. The 8-bit channels are not intended to limit the scope of the present invention.

[0206] If the input to this DNN circuit is the output of another DNN circuit, as naturally occurs when a large DNN is divided into smaller parts to aid in processing, the presented channels refer to every feature passed to the DNN. For example, if a particular segment of the model requires 64 feature channels as input, each value is presented in parallel in the specified format, as an unsigned or signed integer or floating-point value, with the desired bit precision.

[0207] It is important to understand that the blocks shown in Figure 17A (and other diagrams described) do not represent steps performed sequentially. Each block represents either an input channel or circuitry performing a function, such as an aperture function. The arrows between blocks represent a set of parallel conductors passing values ​​between processing circuitry. All processes are active simultaneously whenever an input to that block is presented. When an input stream is initiated, the circuitry represented by the block is activated in sequence until all processes are active, and an output stream is also generated on multiple channels. While input is still being accepted, the final output begins emitting from the first corner of the input array (nominally the upper left).

[0208] The first 7x7 convolutional node 1702 in this model is typical for RGB input in a DNN used for visual understanding. This 7x7 kernel can be applied only if the kernel patch falls within the bounds of the input array (typically in the case of RGB input), or it may be applied to every input position and synthesized missing values ​​(typically in the case of feature reprocessing). Generally, a fairly large number of output channels (typically 64) are produced, with the number of channels typically increasing throughout the rest of the system as feature values ​​pass through additional nodes.

[0209] Each of the subsequent convolution nodes 1703, 1704, 1705 also accepts and produces multi-channel array streams of the same dimensions as their inputs, with each having any number of output channels, which may be more, less, or the same as the number of input channels.

[0210] The concatenation node 1706 in this model accepts the parallel input array streams produced by nodes 1704 and 1705 and synchronizes them to produce a combined set of channels. The channels from the convolution nodes are not altered. However, because the nature of the pipeline is such that outputs corresponding to a particular array location from the 1x1 convolutions are produced before the outputs from the 3x3 convolutions, the concatenation function must provide buffering in the form of a first-in, first-out (FIFO) circuit to allow all channels to output data corresponding to the same location presented simultaneously.

[0211] The MaxPool node 1707 in this model utilizes an aperture function that compares all values ​​in the patch and outputs only the maximum value for each channel individually. The number of channels is unaffected, but the array dimensions of the input stream are reduced in the output stream. If, as is typical, the MaxPool node reduces the horizontal dimension by 2 and the vertical dimension by 2, the output array will be one-quarter the size of the input array.

[0212] Since the frame rates for the input and output streams must necessarily be the same (the output cannot be generated faster than the input on which it is based, and the output cannot be generated slower than the input, or data will be lost), the net effect is that the clock rate for the reduced output array stream is reduced proportionately.

[0213] In this MaxPool example, only one output is generated for a patch of four input positions, so the required output rate is only one-quarter of the input rate. Therefore, all subsequent nodes in the pipeline operate at a reduced effective throughput, which can be advantageous as the number of channels increases. Some resources that may be dedicated to each channel can be shared between channels when there are more cycles available to perform the required computations, resulting in an overall reduction in circuit size with only a small increase in power. Dimensionality reduction also forms an important cornerstone of the present invention.

[0214] Subsequent nodes in the illustrated model can utilize similar or different connection patterns, as long as each pattern supports a system-wide interface that simultaneously presents all channels of data corresponding to a given position in any input array stream.

[0215] As shown, in this model, after the value of MaxPool node 1707 streams to additional convolution, concatenation, and MaxPool nodes, these nodes are functionally the same as the nodes already mentioned, and therefore are not labeled with reference numbers.

[0216] The Global Average node 1708 in Figure 17B is unique in that the aperture function for node 1708 covers all remaining dimensions of the previous input array stream and simply returns the average value of each channel across the entire array. The output array is therefore dimensionally 1x1, forming the output channels 1709 of the overall circuit.

[0217] 18A and 18B show the overall structure and flow of a 4-up pipeline that implements a DNN model of the same form as that shown in FIGS. 17A and 17B.

[0218] The input channels 1801 are presented in parallel as four data sets for each channel. For RGB data, this takes the form of four individual pixels representing four adjacent columns of the input array, containing four RGB values ​​for each pixel, for a total of 12 inputs accepted simultaneously in parallel. Alternatively, the input channels may be from another DNN circuit, in which case the input channels take the form of four complete sets of input channels representing four adjacent columns of the input array. For example, if the model requires 64 feature channels as input, the four sets provide a total of 256 parallel inputs.

[0219] The initial 7x7 convolution node 1802 is typical for RGB inputs in DNNs used for visual understanding. In this 4-up implementation, node 1802 accepts inputs for four pixels at a time and generates outputs for four pixels at a time. Typically, the number of output channels is significantly larger than the number of input channels—64 or more—and no longer represents color information. Throughout the remainder of the DNN in this model, channels represent the detection strength of features or combinations of features found in the input array, with independent values ​​for each location. Each of the subsequent convolution nodes 1803, 1804, and 1805 also accepts and processes inputs for four pixels at a time per channel. Concatenation node 1806 accepts four sets of channels from convolution nodes 1804 and 1805 and outputs four sets of combined channels.

[0220] The first MaxPool node 1807 is labeled 4-up to 2-up. Node 1807 takes up to four samples representing a patch of input array locations that includes two adjacent columns in two consecutive rows. The effect is to reduce the dimensionality of the input array stream to produce an output array stream that is half the width and half the height, resulting in a net reduction in the effective throughput of all subsequent nodes by a factor of four. Benefits of using a single input processor include reducing subsequent processing clocks by utilizing more compact circuitry.

[0221] When using N-up parallel input processing, reducing the output array width is instead used to reduce the number of parallel outputs. Because parallel inputs represent adjacent columns in the same row of the input array stream, only the width reduction is relevant. While it is possible to maintain N-up parallel output at a reduced frequency, there is no advantage in size or power to doing so. The net effect of the MaxPool node 1807 is to reduce parallelism in the horizontal dimension from 4-up to 2-up (as labeled), reducing the processing frequency by a factor of 2, rather than a factor of 4 as in the 1-up case described above.

[0222] Nodes 1808, 1809, 1810, and 1811 process data in 2-up parallel tracks, each approximately half the size of their 4-up counterparts. This does not correspond to a reduction in power, as the total number of operations required by a 4-up, 2-up, or 1-up circuit is the same; only the overhead for managing N-up coordination is reduced.

[0223] A second 2x2 MaxPool node 1812 again obtains up to four samples representing a patch of input array locations comprising two adjacent columns in two consecutive rows. The net effect of node 1812 is to reduce parallelism in the horizontal dimension from 2-up to 1-up, reducing the processing frequency by a factor of two. All subsequent nodes shown in Figure 18B operate on a single set of input and output channels for each node, with the final output 1813 taking the form of a single sample for each channel presented simultaneously in parallel.

[0224] 19 and 20 are tables illustrating array stream sizes for a typical small DNN applied to an input stream compatible with images in HD RGB format. The table in Fig. 19 represents a DNN that implements only 1-up processing, as shown in Figs. 17A and 17B, while the table in Fig. 20 represents the same DNN that initially implements 4-up processing and transitions to 1-up processing at subsequent nodes, as shown in Figs. 18A and 18B.

[0225] Having described the nomenclature and general procedure for N-up parallel processing, the inventors now provide a specific example of an apparatus and method for applying a 3x3 convolution function to an input array using 4-up parallel processing. The input array in this example is an array of pixels in RGB color, as used in many other examples herein. Note that this is not intended to limit the scope of the present invention, as 3x3 convolution with 4-up parallel processing can be used for many other forms of input array. It should be understood that, again in this example, the 3x3 block shown represents a circuit configuration that performs a kernel function on an input stream.

[0226] Figure 21 shows an example of a circuit configuration on an IC that implements a 3x3 convolution node using a 4-up data stream. In Figure 21, one set of four inputs 2101 is a set retained from the previous input interval and is retained along with a current set of four inputs 2102 to provide all the inputs needed for all four output channels of the 3x3 convolution. As discussed above in enabling detail, pipelined processing requires the use of inputs from the previous input interval and inputs from the current interval to fully compute the outputs.

[0227] p0, p1, p2, and p3 represent the input channel values ​​for positions 0, 1, 2, and 3 in the first row of the input array, respectively. For simplicity, only a single symbol is used, but each symbol represents all channels at that input position. In the pixel context, each data point p x represents the R, G, and B values ​​for that pixel.

[0228] w 0,0 From w 2,2 represents the set of weights to be applied to the values ​​at the input channels. Since each weight applies to only one input channel, the number of input channels does not affect the structure of the circuit, and therefore multiple channels are not shown.

[0229] The weights in kernel rows 2103, 2104, and 2105 are applied in parallel (simultaneously) to input channels p0, p1, and p2, and the partial products for each set of weights for each row are summed immediately, following the rules for the aperture function of a 3x3 convolution. As described above in enabling details for pipelined processing, the partial sums are passed from each functional circuit to the next, and an output is generated when all necessary partials are achieved. The application of weights in row 2105 generates the kernel's final output for the current row by combining the partial products with the sum of the products from the application of weights in row 2104 from the previous row. The application of weights in row 2104 generates an intermediate value by combining the partial products with the sum of the application of weights in row 2103 from the previous row. The application of weights in row 2103 generates an initial value by summing the partial products and saving them for later use. Bias, if any, can be introduced at any stage. An activation function, if any, is applied to the final output 2105.

[0230] The complete circuit implementing the weights in rows 2103, 2104, and 2105, including any bias and activation functions, produces the first output channel of the 4-up set.

[0231] When the first 4-up set is presented from the input array stream, there is insufficient data to compute all four required outputs, so a second 4-up set is obtained and computations for all outputs are delayed until valid data is available for computation using inputs from both sets 2101 and 2102.

[0232] Circuits 2106, 2107, and 2108 apply weights in circuits that are copies of the previous circuits, the functions differing only in which position of the input the weights are applied. 0,0 From w 2,2 Note that while σ is the same for all output channels, each combination of one weight and one input channel is unique.

[0233] The output calculated using the weights in row 2105 of the first kernel copy generates the first parallel channel set q0 of the output array stream, and the outputs using the weights of the other kernel copies 2106, 2107, and 2108 generate the remaining parallel channel sets q1, q2, and q3, respectively, of the output stream 2109.

[0234] The first output, q0, corresponds to a 3x3 kernel centered at p1, so the circuit corresponding to the arrangement in Figure 21 is the solution for the inset or "valid" version of the 3x3 convolution. The width of the output array stream is therefore reduced by two positions from the width of the input array stream (typically the height is also reduced by two rows, but this is unrelated to horizontal processing) to fit the aperture function definition for the above variation.

[0235] Figure 22 shows the configuration required for a circuit to generate outputs for a 4-up input channel for a "same" version of a 3x3 convolution. That is, the dimensionality of the output array stream is not reduced, and one output position is generated for every different position in the input array stream. In this variation, input set 2203 represents the current values ​​of the 4-up input array stream, and input sets 2202 and 2201 represent the retained values ​​from the previous two sets.

[0236] Application of kernel circuits 2204, 2205, 2206, and 2207 produces values ​​q0, q1, q2, and q3, respectively, of a 4-up output array stream 2208, where the center of each kernel is aligned to correspond to one position in the 4-up input array stream.

[0237] When the first 4-up channel set is presented from the input array stream, there is insufficient data to compute all four required outputs, so a second 4-up set is presented and computation is delayed until valid data is available for both sets 2202 and 2203. Valid data is not yet available for set 2201, and kernel circuit 2204 suppresses the inclusion of weights applied to p3 in 2201, or forces uninitialized values ​​to zero, consistent with application of a 3×3 convolution aperture function to a location that causes the kernel to overlap the edge of the input array. This suppression mechanism is triggered for the first set in each row, but subsequent sets in that row utilize p3 value set 2201 to compute the complete kernel for that output position q0.

[0238] In the processing of a full DNN, situations arise where the 4-up stream technique is applied to an input array stream whose width is not an even multiple of 4. In such cases, invalid values ​​in the last 4-up set are suppressed by forcing the invalid values ​​to zero or by other means, and the final output position in the last 4-up set in the row is ignored. This is consistent with both the inset ("valid") and full ("identical") variations of the 3x3 aperture function.

[0239] In all cases, the first position of each row in the input array stream is always presented in the first position of the 4-up input set.

[0240] If the input row length is not an even multiple of the processing set width, the processing clock is increased so that the overall throughput of the N-up processing is compatible with that of the 1-up input source, and special buffering is required to pack the incoming values ​​into N-up sets, as described below.

[0241] Figure 23 shows the configuration required for a circuit that outputs two variants of a 1-row by 7-column convolution on 4-up data. From the previous discussion of 3x3 convolutions, those skilled in the art will understand that the particular number of rows in the kernel only affects the number of partial sums maintained over time, and does not affect the mapping of kernel weight columns to input set columns. Thus, the data arrangement shown in Figure 23 applies equally to 7x7, 3x7, or other kernels with a width of 7.

[0242] As mentioned above, input set 2303 is the 4-up dataset currently being presented from the input array stream, and sets 2302 and 2301 are previously presented and retained datasets from the immediately preceding and two sets before that, respectively.

[0243] Kernel processing circuits 2304, 2305, 2306, and 2307 represent the alignment required to produce the inset ("valid") convolution output 2308, and circuits 2309, 2310, 2311, and 2312 represent the alignment required to produce the perfect ("identical") convolution output 2313.

[0244] Circuit 2304 w 0,0 is aligned with p0 of the input set 2301 to generate an inset variant, and w of the circuit 2308 0,3 is aligned with p0 of input set 2302 to generate a full variant, where both circuits 2304 and 2309 generate output q0 for their respective use cases.

[0245] Those skilled in the art will appreciate that the two kernel sets have significant overlap of the same functions, and that it would be straightforward to deploy a single circuit using only five uniquely mapped kernel circuits to generate either variant as required. Those skilled in the art will appreciate that any M-up streaming data set (including 1-up) can also be repackaged into any other N-up streaming format (where M≠N) as needed to maintain a high enough overall throughput for the system to accept and process input array streams at the proposed rates. The cost of doing this is requiring N copies of a particular core processing circuit, but the overall effect is that the circuit can throttle back the processing clock to a reasonable limit for the implementation method while still accepting input streams at full speed.

[0246] 24A and 24B show a typical implementation of a 2x2 MaxPool node, where the maximum value for each channel is selected for a separate patch of two adjacent column positions across two adjacent rows.

[0247] Figure 24A shows the arrangement of 2x2 MaxPool nodes across a 4-up data stream 2401. As the first row of each pair is presented, comparator 2402 evaluates inputs p0 and p1 and passes the larger of these to FIFO circuit 2403, where it is held for use when the second row is presented. Comparator 2404 and FIFO 2405 simultaneously perform the same operation on inputs p2 and p3. As the second row of each pair is presented, comparator 2402 accepts the largest held value from FIFO 2403 for the same column location from the first row, compares it with inputs p0 and p1, and outputs the larger of the three values ​​as output q0. Comparator 2404 and FIFO 2405 perform the same operation on inputs p2 and p3 to generate output q1.

[0248] Output set 2406 includes two channel sets whose individual values ​​are the maximum of the four samples of each particular channel (values ​​from different channels do not interact with each other in this aperture function). Thus, output 2406 in Figure 24A is a 2-up output data stream generated from a 4-up input data stream.

[0249] Figure 24B shows the same 2x2 MaxPool node arrangement across a 2-up data stream 2407. Comparators 2408 and FIFOs 2409 are functionally identical to those described above, but only a single set is needed to accept 2-up inputs p0 and p1 and produce a single output channel set 2410. Thus, the output 2410 of the second example is a 1-up output data stream generated from the 2-up input data stream, and all downstream nodes can take on the smaller 1-up format.

[0250] The tiled MaxPool function, along with any other aperture function with a 2x2 stride, reduces the size of the input array by a factor of 2 in each dimension. Since the total width of an N-up array stream is N times the number of sets presented, the reduction can be affected by shrinking the set width or decreasing N, as long as N is evenly divisible by the horizontal stride. Since N is a replication factor for copies of the circuit running in parallel, it is preferable to reduce N as much as possible.

[0251] Figure 25 shows one possible example where reducing N is not possible. A 2x2 MaxPool node is applied, but in this case to a 5-up input stream. As with the 3x3 convolution described above, input set 2501 is retained and used in conjunction with the current input set 2502 to present the smallest set of values ​​such that all outputs can be generated in the same clock cycle. (Other configurations are possible, such as switching the first comparator to process p0 with p1 on the alternate input set, or p1 with p2 on the alternate input set, and setting the intermediate comparator to process p4 with p0 on the alternate input. This reduces the number of required copies of the aperture function from 5 to 3, which can be advantageous if the aperture function implementation is significantly more complex than a simple comparison.)

[0252] In this example, comparator 2503 and FIFO 2504 operate on the held values ​​of p0 and p1, comparator block 2506 operates on the held values ​​of p2 and p3, comparator block 2507 operates on the held value of p4 and the current value of p0, comparator block 2508 operates on the current values ​​of p1 and p2, and comparator block 2509 operates on the current values ​​of p3 and p4.

[0253] Since a 2.5-up data stream cannot be implemented within the constraints of the pipeline, the dimensionality reduction in this example must be applied to the width of the input array, and therefore the output 2510 is a 5-up output that reflects the 5-up input stream.

[0254] As mentioned above, in some circumstances it may be advisable to repackage an M-up stream as an N-up stream with the same array dimensions. Specialized FIFO circuitry can be used to perform this function. Figure 26A shows such a FIFO used to repackage a 4-up stream 2601 into a 2-up stream 2603. The FIFO 2602 accepts four inputs at a time and stores them as separate entries. Whenever two entries are available in the FIFO, outputs are produced two at a time. The data flow in Figure 26 (and in the following figures) is from the inputs through the circuitry to the outputs.

[0255] In the common situation where the input stream width is not an integer multiple of the input stream set size, a counter must be included to track the number of valid entries presented per row. For example, if the input array width is 10 and a 4-up input set is used, where four 3-sets are required to cover a complete row, the FIFO must ignore the last two entries of the presented third input set and output five 2-up output sets instead of six. After each row, the counter resets and begins counting entries in the next row. The array width limit may be fixed or presented via a preloaded register. If it is known that the array width will always be an integer multiple of both the input set size and the output set size, this logic can be omitted.

[0256] Figure 26B shows the repackaging of a 3-up stream 2604 into a 5-up stream 2606. The FIFO 2605 accepts three inputs at a time but stores them as separate entries. Whenever five entries are available in the store, the FIFO produces five outputs at a time.

[0257] As mentioned above, additional operations must be implemented to account for invalid entries that can occur at the end of a row when the array width is not an integer multiple of the input set size. Similar problems arise when the array width is not an integer multiple of the output set size. In this case, a final set must be emitted as each row is fully received, containing the last entry of the row in the first output, and the remaining channel sets containing invalid entries without specific values. For convenience, the practice of placing all zeros in invalid entries can be used to reduce the total circuit size in subsequent nodes where zeros have no effect, such as convolutions and MaxPools.

[0258] The size of the FIFO must be sufficient to hold as many input sets as necessary to ensure that no data is lost. To maintain overall system throughput, output is emitted as soon as enough entries are available to generate an output set.

[0259] Any set size may be repackaged into any other set size, but the required processing frequency will vary proportionally to the ratio of the sizes. For any M-up input repackaged as an N-up output, the required processing frequency is

[0260]

number

[0261] Throughout the system, for the simplest operations, each circuit that accepts rows should provide and ignore unused, invalid entries at the end of all rows whose row width is not an integer multiple of the set size. This is not a strict limitation, as the circuit can operate with additional logic not shown here. This ensures that every column position maps to the same channel set in the parallel set presented for every row, minimizing the complexity of the operations that combine values ​​at the same column position across multiple rows.

[0262] Figure 27A shows an implementation of a Concatenation node, in which channels from one source 2701 are concatenated position-wise with channels from another source 2702 or more sources (not shown) so that output 2706 contains all channels from all sources. Channel values ​​are not mixed or modified by this node. In the typical situation where sources have different timing, one or both of FIFOs 2703 and 2704 hold the input channel values ​​until the full set of output channels is available. Interleave circuit 2705 concatenates all channels from set p0 from each source to produce q0, sets p1 to produce q1, and so on.

[0263] A common example requiring this solution is the combination of the outputs of a 3x3 convolution node and a 1x1 convolution node, each applied to the same input array stream. Both nodes process the streams at the same rate, but the output of the 3x3 node cannot be finalized until the third row of the input stream is presented, while the output of the 1x1 node can be finalized as soon as any data from the input stream is presented. The net effect is that the output of the 1x1 node corresponding to a particular position in the input array stream is presented to the concatenation node well before the output of the 3x3 node for those same positions. Because the next node after the concatenation node needs all channels for any given position to be presented before it can perform any computation, the concatenation node must buffer the previously presented input stream and wait for the later presented input stream to arrive at the same position before it can present the full set of all channels for a given position at its output. This applies equally to 1-up or N-up data streams.

[0264] If every input array position of the slowest path is always presented after the same position by all other paths, the FIFO for that path can be omitted. If, under some conditions (typically the final position in the stream) the slowest path is not presented last, the data in the FIFO for that path must be kept to the minimum number of entries required to prevent data loss under those special conditions.

[0265] If the data path widths to the various sources are different, the path widths can be repackaged to match each other, as in Figures 26A and 26B, or the function can be merged with the FIFO used for concatenation buffering. Those skilled in the art will understand that any number of paths can be concatenated into a single operation by adjusting the size, and worst-case timing, of the FIFO for each previous path so that it holds as many values ​​as each path can present before the corresponding location is presented by the slowest path.

[0266] Figure 27B shows a 4-up Dense node implementation. A Dense node is mathematically equivalent to a convolution with a kernel size equal to the size of the input array. Thus, to create each output channel, one different weight is applied to each input position for each input channel. The number of output channels is independent of the number of input channels, and the output array produced is always a 1x1 array. In this exemplary implementation, inputs 2707 are sent in sets of four, so weights 2708 specific to each input position are loaded from a local store (not shown) and multiplied by the current input in circuitry 2709 to produce the partial products of the complete kernel. All partial products from all presented input channels are summed to produce a single 1-up set 2710 of output channels.

[0267] Figure 27C shows an implementation of a 4-up Global Average node that takes all values ​​for all positions of each input channel and averages them to produce the same number of output channels. The Global Average node is mathematically equivalent to a convolution with a kernel size equal to the size of the input array, applied only to each input channel individually (rather than to all input channels together as just described) with a common constant value equal to the inverse of the number of elements in the kernel. Because this is mathematically equivalent to multiplying by the inverse before or after the summation operation, circuit 2712 simply adds all values ​​at each position for each input channel 2711 and then multiplies by the inverse of the number of elements when all input values ​​are added to produce each output channel. Because all input positions are merged into a single value, output 2713 is a single 1-up channel set with an array size of 1 x 1.

[0268] Figure 28 shows a 4-up implementation of a 3x3 Local Average node that utilizes a sliding aperture function to calculate the average of each input channel over a subset of positions to generate output channels. This implementation forms an inset or "valid" set of outputs, where the aperture does not overlap the edges of the input array and the number of samples is the same for all output positions. Each output channel corresponds to a single input channel, and data is not mixed between channels. Similar to the convolution node implementation with similar size and input mapping shown in Figure 21 above, the current input set 2802 is held by register 2801 so that the current and previous input sets can be accessed simultaneously. Circuits 2803, 2804, and 2805 each apply the same summation of each input channel in sets p0, p1, and p2, but apply the summation to three different partial sums over time to generate set q0 of output array stream 2809. Circuit 2803 initializes the running sum for the first row, circuit 2804 generates the running sum for the intermediate rows using the output of circuit 2803 delayed by a FIFO (not shown), and circuit 2805 generates each final sum using the delayed output of circuit 2804. Circuit 2805 then multiplies the final sum by the reciprocal of the number of elements, in this case 1 / 9, to generate output set q0. Activation functions may be integrated into the circuits or, equivalently, placed between the nodes.

[0269] Equivalent circuit 2806 generates output set q1 from channel set p1, p2, and p3 of the previous input set. Similarly, circuit 2807 generates output set q2 from p2 and p3 of the previous input set 2801 together with p0 of the current input set 2802, and circuit 2808 generates output set q3 from p3 of the previous input set 2801 together with p0 and p1 of the current input set 2802.

[0270] If a local mean aperture function is generated for every valid position, the output 2809 has a reduced array size compared to the input; in this case, the width and height are each reduced by two positions, which is generally insufficient to significantly reduce a 4-up stream. If a horizontal step size other than 1 is used, i.e., not all possible output positions are utilized, the reduction in horizontal dimension may be implemented in the circuit as a reduction in N. For example, if the horizontal step size is 2, only every other value is needed, and the circuit can generate a 2-up output channel by computing only q0 and q2, omitting the unused circuitry for q1 and q3. Similarly, if the horizontal step size is greater than 4, the various circuits used to compute q0 through q3 can be utilized in sequence to generate a 1-up output stream.

[0271] Figure 29 shows another 4-up implementation of a 3x3 local average node that forms a full or "identical" set of inputs, where the aperture overlaps the edge of the input array and the dimensions of the output array are the same as the dimensions of the input array. In this case, the number of input locations sampled at the edge is not the same as the full set of samples taken internally, and therefore the final inverse used for each output location must reflect the number of samples used for that output location.

[0272] Similar to the exemplary circuit shown in FIG. 22, the variation in FIG. 29 utilizes input set 2903 to present the current values ​​of the 4-up input array stream, while input sets 2902 and 2901 present the retained values ​​from the previous two input sets.

[0273] Application of summing circuits 2904, 2905, 2906, and 2907 produces values ​​q0, q1, q2, and q3, respectively, of a 4-up output array stream 2908, where the center of each summation is aligned to correspond to one position in the 4-up input array stream. In this example, when the first 4-up input is presented at the beginning of each row, only summing circuit 2904 intersects the left edge of the input array, but depending on the number of sets populated at the end of the row, all four summing circuits may intersect the right edge of the input array, and therefore the selection of the inverse reflecting the number of samples taken will change accordingly.

[0274] Those skilled in the art will appreciate that the close correspondence between the exemplary circuits in Figures 21 and 28, and between the exemplary circuits in Figures 22 and 29, indicates that the structure and replication of the computations is not affected by the nature of the aperture function implemented, and further that the apparatus and method are equally applicable to any aperture function defined over a similar sliding window.

[0275] Figure 30A shows an implementation of a 4-up Subset node that passes only certain channels to the next node, but with equivalent array dimensions and timing. This node type is typically used to split incoming channels so that different processing styles can be applied to each group of incoming channels. If the set of channels routed to the output is fixed, the connection between the input 3001 and the output 3003 can be made by direct wiring of physical conductors. If not, routing circuitry 3002 affects the desired selection of channels using a multiplexer.

[0276] Figure 30B shows a typical implementation of a 4-up Crop node, which presents a subset of positions of the input array stream to the output array stream. Typically, entire rows at the top or bottom, or both, are omitted, along with columns at the left or right, or both. The current input set 3005 is combined with the previous input set 3004 in repackaging circuitry 3006 to generate channel set q0, q1, q2, and q3 for output 3007, such that the number of columns omitted at the left is not an integer multiple of the dataset size N, with q0 always used for the first column of each row. If omission of the left edge of the input array stream is not required or the number of omitted columns is an integer multiple of N, the previous input set 3004 may be omitted from the simplified circuit. If the output array is sufficiently reduced from the input array, the N-up input stream can be repackaged into an M-up output stream within the position selection circuitry.

[0277] In any of the nodes described above, either large multipliers or individual multipliers can be used with equal functionality. When many weights are applied to each input, large multipliers have advantages over individual multipliers based on the bit width of the multiplicand and product. In other situations, individual multipliers of equal precision may be smaller or have lower power usage. The N-up pipeline is independent of the type of multiplier used.

[0278] In another aspect of the invention, an IC may comprise one or more interconnected functional circuits and input and output ports, each IC implementing a portion of a neural network, as described above with reference to Figures 17A and B and 18A and B. The individual ICs of such ICs in a system embodiment may be connected in a linear order, output port to input port, or in an interconnected chain with parallel connections, from the first IC receiving primary input from a source array to the other ICs, with the output port of the last IC in the connected set then providing the output of the neural network including the functionality of all ICs.

[0279] 31 shows such a system 3100 of ICs interconnected to implement a neural network. IC 3101 has an input port 3102 that receives a stream of input values. The input values ​​may be any of the protocols described above for an input array that may have a single value per position in the array, or an input array that may have multiple values ​​per position, such as the example of an HDMI image with R, G, and B values ​​for each position in the input array, or the input stream may be ordered as an N-up stream as described in the above embodiment.

[0280] In Figure 31, five ICs 3101, 3105, 3106, 3107, and 3108 are shown interconnected between input and output ports. IC 3101 is shown with functional circuitry 3104, which is interconnected on the IC and leads to output port 3103, which connects to the input port of IC 3105. The functional circuitry implements the aperture function described in the various embodiments above. In this example, ICs 3105, 3106, 3107, and 3108 show functional circuits with the same interconnections as IC 3101, but it should be emphasized that these ICs are different and the functional circuits and interconnections between them are not the same. The diagram is for illustrative purposes only.

[0281] IC3105 connects by its output port to input ports for both IC3106 and IC3107, illustrating that there may not be a simple linear connection between ICs. The output ports of IC3106 and IC3107 are both shown connected to the input port of IC3108. Again, this diagram is illustrative only. In any system of interconnected ICs, the interconnections may be more complex. IC3108, as the last IC in the system, outputs an output stream for the neural network implemented by the system of interconnected ICs. The connections between the input and output ports are parallel paths of conductors delivering bits of value for each output interval. The system of ICs implements a neural network of some depth. In this embodiment of the invention, an infinite variety of neural networks can be implemented by interconnecting individual ICs with different nodes and interconnections on each individual IC.

[0282] Application to 3D image data In the embodiments and examples of the invention described above in enabling detail in many instances, the primary input data source is in the form of values ​​for pixels of a two-dimensional image, such as an RGB image as an HDMI frame, a 1080x1920 matrix of RGB values, each pixel of which is a 3x1 vector of color components. However, in the operation of the apparatus and methods in various embodiments of the invention, not all input data sets are in the form of two-dimensional (2D) arrays such as HDMI frames. However, page 41 above introduced a situation in which the primary input data source is a three-dimensional array of data points.

[0283] As an example of a three-dimensional image that can serve as input in a neural network, many medical devices, such as magnetic resonance imaging (MRI) devices, capture three-dimensional (3D) image data. The following description in this application extends the unique 2D processing apparatus and method described above in enabling detail to process streams of values ​​representing the 3D image data.

[0284] To understand the application of embodiments of the present invention to 3D data, we can revisit the 2D data sample used in the previous discussion, such as the HDMI frame shown above as a series of pixels arranged in a 2D plane. We have described aperture functions based on patches, such as a 3x3 patch, meaning that the patch is 3 pixels wide and 3 pixels high. A 3x3 aperture function governs calculations involving the data values ​​of nine pixels or features from the previous node, where the pixel or feature depends on the location of the patch relative to a particular pixel in the image. Now, considering a third dimension orthogonal to the plane of the 2D array, we can consider a 3x3x3 3D ​​patch with 27 data points. A 3D patch must have multiple data points in each dimension, with a common number such as 3x3x3 being typical. In 3D images, data points are called voxels rather than pixels. It is not necessary for the three dimensions of a 3D aperture function to be the same. This is not a limitation of embodiments of the present invention. Therefore, for purposes of this discussion, while the 2D patch was referred to above as MxN, the 3D aperture function has dimensions LxMxN.

[0285] Figure 32 shows the layout of a compositor on an integrated circuit configured to implement a 3x3x3 convolution as a 3D aperture function across 27 individual data samples. In this 3D aperture function, L, M, and N are all equal to 3. As in the example using HDMI frames, the data samples may be monochrome, have three color values, or may be features with more than three values ​​per sample. In embodiments of the present invention, operations are always pipelined; therefore, values ​​for data samples are input as a stream in a predetermined order. In one protocol, data values ​​are presented in the input stream first across the columns of the data array, then down each row, and from there to each plane in the three dimensions. The values ​​at each data point are processed only once, and partial sums representing the intersections of each 3D aperture location with each data point are calculated and forwarded for further processing, similar to the operation of the device in a 2D implementation.

[0286] Each 3x3x3 array in Compositor 3201 applies one of the 27 weights of a single output channel to a data point and forwards the partial sums for further processing. Row Buffer FIFO 3202 presents the partial sums from the previous row, appropriately delayed, and Plane Buffer FIFO 3203 presents the partial sums from the previous plane, also appropriately delayed. The final weight w 2,2,2 Once applied, the summation is complete and the full sum is passed to the finalization circuit 3204. Those skilled in the art will appreciate that the weight subscripts in each compositor refer to plane, row, and column, respectively.

[0287] It does not matter whether the input data point contains a single scalar value or multiple values, such as features from a previous neural network node. By extending the parallel weights and sums throughout the circuit, any number of parallel output channels can be generated for the same input source array.

[0288] Special case logic is embedded in the compositor 3201 and FIFO 3202 to handle edge cases for the first and last columns, first and last rows, and first and last planes. Those skilled in the art will appreciate that the edge cases for the third dimension correspond closely to the edge cases for the second dimension, and similar solutions will suffice.

[0289] It will also be appreciated that a fully functional IC with an array of compositors as shown in FIG. 32 further has input ports, output ports, and control circuitry coupled to the compositor, delay circuit, and finalization circuit for operating at least one counter and generating control signals, as described above for a system operating on a 2D data array.

[0290] Figure 33 shows an embodiment in which data from multiple planes can be buffered and presented simultaneously so that a single compositor can apply weights for multiple planes. When the number of input channels is much smaller than the number of output channels, this implementation may be preferred to reduce the total size of the required plane buffers, at the expense of increasing the number of shared large multipliers required.

[0291] In the example of a 3x3x3 convolution, a FIFO 3302 of size equal to RC-1 holds input voxels for each of the three most recently presented planes in registers 3301. Data from these registers is aggregated onto bus 3303, which distributes source input data for the same 2D location across the three planes to all compositors 3304.

[0292] Each compositor 3304 applies one weight for each plane of the 3x3x3 convolution to the corresponding source input for that plane and sums the results. The partial sums are passed to other compositors, appropriately delayed by FIFOs 3305, as in the 2D implementation. Edge case conditions, including those for the first and last planes, are embedded in the various compositors and closely match those implemented by the fully inverted form shown in Figure 32.

[0293] Note that the use of the partially inverted form in Figure 33 breaks the strict sequence of the decomposed aperture function in that the upper-left corner source values ​​from three different planes are combined as the first action in the calculation. If the aperture function is a convolution and all products can be added in any order to get the same result, then this technique is equivalent to the fully inverted variant. For any aperture function that relies on a time sequence of operations to compute a valid result, the fully inverted variant of Figure 32 must be used instead.

[0294] Considering again the 3×3×3 example on a P×R×C input array volume where the input is a single scalar value for every voxel, the first form in FIG. 32 requires only a single large-scale multiplier for all output channels, but two R×C FIFOs per channel to transfer partial sums. In contrast, the alternative embodiment in FIG. 33 requires three large-scale multipliers, but only two R×C FIFOs total to buffer the raw scalar data. In this case, the second form may be preferable if the product of R and C is large enough so that the size of the FIFOs dominates the size of the large-scale multipliers. However, if the number of input channels is large compared to the number of output channels, both the total number of large-scale multipliers required and the total size of the FIFOs are reduced when using the embodiment in FIG. 32 compared to the embodiment in FIG. 33. The choice between the two embodiments can be made solely based on the number of input and output channels and the combined size of the buffers and multipliers. If the order of operations is not important, then both embodiments are numerically equivalent, and so long as the aperture function is calculated accurately and completely, the choice between them is purely one of overall cost and convenience.

[0295] Figure 34 shows a typical 3x3x3 convolution implementation applied to a 4-up input stream. As previously described, the 4-up input stream presents data for four input array locations in parallel. A source input set is presented by register 3402, which is then held by register 3401, and the two sets are presented simultaneously in presentation order to all four compositor sets 3403. Each compositor set sequences data between rows and planes (FIFOs not shown), similar to the case for the 1-up embodiment described with reference to Figure 32, with the main difference being that columns are processed simultaneously rather than sequentially.

[0296] Each compositor set 3403 has its own edge conditions embedded in it, since each may be exposed to different subsets in the horizontal direction. This embodiment is applicable to any aperture function, since the sequence of operations can still be the same even if the horizontal operations are accomplished simultaneously.

[0297] Figure 35 shows a fully inverted embodiment of the same 3x3x3 convolution example applied to a 4-up data stream described above with reference to Figure 34. In this embodiment, each source voxel is processed only once, and a total of four large multipliers per input component are required to support all output channels in the parallel set.

[0298] Source input data is presented to register 3501 and distributed to all compositor sets. Compositor set 3502 is similar to compositor set 3403 described with reference to Figure 34. The compositor in set 3504 accepts data from P2 and P3 and forwards its partial sums to the compositor in set 3503, where data from P0 of the next source input data set is applied. The compositor in set 3506 accepts data from P3 and forwards its partial sums to the compositor in set 3505, where data from P0 and P1 of the next source input data set is applied. Inter-row FIFOs provide standard delay functionality in compositor sets 3503 and 3505, but are not included in compositor sets 3504 and 3506. Inter-plane FIFOs provide equivalent delay functionality to compositors 3502, 3503, and 3505.

[0299] The final output is the weights w 2,2,2 These compositors implement the weights w for all four outputs. 0,2,2 It is shown as the bottommost compositor.

[0300] Note that compositor set 3502 produces its output one input source interval before compositor sets 3503 and 3505, and therefore must be delayed to produce all four outputs simultaneously.

[0301] Those skilled in the art will appreciate that, with appropriate restrictions to fit the requirements of a particular aperture function, any such function that can be decomposed into a series of step sequences can be computed continuously over time by such circuitry. Furthermore, while the number and format of compositor sets is determined by the particular combination of N-up (including 1-up) data representation and a three-dimensional array of sample sizes per output, the basic format, timing, and exception rules are universal for all combinations.

[0302] Other aperture functions typically used in deep neural networks, such as MaxPool, can be included within the general form described.

[0303] Synthetic scaling applied to shared neural networks In yet another aspect of the present invention, a system is provided that uses a single instance of an aperture function implemented as an IC circuit to process multiple sources of input data presented as independent parallel input array streams or as dynamically synthesized scales of the input array streams.

[0304] The nature of applying an aperture function across a multi-dimensional data array requires that partially completed subfunction values ​​from each column location in each row be retained and combined with the subfunction value at the corresponding column location in the subsequent row. If the number of input dimensions is greater than two, corresponding values ​​must also be retained from the first plane to the next plane, from the first volume to the next volume, and so on. The specific data items that need to be retained for recombination during later processing are referred to in this discussion as the context for the location under consideration. In the above discussion, the context required for processing an ordered stream of input values ​​row-by-row, and the circuitry for retaining the context values ​​in an ordered manner (such as FIFO circuitry or other registers) have been described as being part of the IC that also implements the aperture function.

[0305] In the following description for applying aperture functions to multiple independent parallel input array streams or as multiple dynamically synthesized scales of an input array stream, the context for the current row of the input stream being processed by the aperture function circuit (based on results held from the previous row) is synchronously provided to the aperture function circuit by separate IC circuitry.

[0306] Splitting the data into rows and presenting the input data in a continuous stream means that there is a discontinuity at the end of each row, and the implementation circuitry must set aside the subfunction value calculated for the right edge and resume processing the value at the left edge. When this context switch occurs, it does not matter which row is processed next; the only requirement for correct operation is that the presented context corresponds to the input value currently being presented.

[0307] Thus, a single aperture function circuit can continuously process rows of data presented as a series of input positions, and the rows can be from different sources, subject only to the constraint that the context for the rows is also presented. Furthermore, the rows need not all be the same width.

[0308] A common technique in computer graphics is to use a pre-computed set of downscaled 2D textures to reduce the workload and improve visual quality when applied to a 3D model. This is commonly referred to as the MIPMAP technique, which utilizes a series of sample sets, each set being half the width and half the height of the previous set. In this form of MIPMAP, both the height and width are reduced by a factor of two for each subsequent set, and the area of ​​each subsequent scaled image is reduced by a factor of four. The sum of the infinite series shown below can be considered as 4 / 3, and any finite sequence of truncated series will always be smaller.

[0309]

number

[0310] As a result, for a single circuit to compute aperture functions over an interleaved set of half-scale input arrays (in both dimensions), the circuit only needs to operate at a 33% increased frequency to ensure complete computation of all scales in synchronization with the presentation of 1:1 scale data.

[0311] A typical example is processing an HD-compatible input video stream running at 60 frames per second (FPS) and presenting individual RGB pixel samples at 148.5 MHz. If a processing frequency of 150 MHz is sufficient to process a stream at 1:1 original scale, then a frequency of 200 MHz is sufficient to process any finite series of half-scale images at the same input rate.

[0312] If the aperture function is implemented in a CNN node to make it possible to understand images, 2:1 scaling requires the model to learn to recognize each object classification across a 2:1 size range. If using a narrower range is more accurate or is otherwise desirable, the model can be scaled to

[0313]

number

[0314]

number

[0315]

number

[0316] Thus, for the example referenced above:

[0317]

number

[0318] In a CNN that processes image data, the first layer accepts input in the form of an array of pixels or samples, such as the typical RGB or grayscale format. Subsequent layers accept output from the upstream layer in the form of feature intensities, typically in the range [0.0,1.0).

[0319] Multiple sets of scaled image arrays for the first layer may be synthesized in real time as the data is presented. For 2:1 scaling, multiple valid sampling schemes exist and are compatible with the proposed circuit. For smooth scaling, as each pair of pixels is received, the individual components, e.g., RGB, are summed and retained separately. For each odd-numbered row, the sum of the current pair is combined with the sum from the previous row, and the final sum is divided by 4 to fit the original value range. This scheme avoids sampling artifacts and is usually preferable for images presented to a human viewer. However, simply discarding every other row and column is also effective. This results in a smaller and simpler circuit, which may be preferable as long as it matches the model's training plan. Subsequent layers receive features already created in subscale format and require no modification or manipulation. The circuit operates by interleaving the data streams for each scale in real time at each layer.

[0320] One such configuration, illustrated by the example of 2:1 scaling, switches scale as needed at each context change, i.e., at the end of each row at a given scale. Initially, full-scale pixels are presented to the circuitry implementing the first layer, and at the end of the first row, a context switch occurs to continue processing at the left end of the second row. While the aperture function processes the full-scale samples, the scaling circuitry simultaneously samples the original data as needed. At the end of the second row, instead of switching context to the third row at 1:1 scale, the context switches to the first row at 2:1 scale, where the scaler begins subsampling the 2:1 data, which is then fed to the aperture function in preparation for calculating the 4:1 scale.

[0321] After processing the first row of the input array at 2:1 scale, the context switches back to the third row at 1:1 scale, and after the fourth row of 1:1 data is processed, the context switches to the second row at 2:1 scale, after which the first row at 4:1 scale becomes available. After processing the first row at 4:1 scale, the context switches back to the fifth and sixth rows at 1:1 scale, then the third row at 2:1 scale, then the seventh and eighth rows at 1:1 scale, then the fourth row at 2:1 scale, then the second row at 4:1 scale, then the first row at 8:1 scale, and so on.

[0322] The above method can be continued to any desired finite degree of subsampling. A fixed buffer is used to hold incoming samples for the 1:1 scale while processing lower scales. The sampling buffer performs this function for all lower scales. The circuit implementation of the aperture function must operate at a high enough frequency to process the pending results along with the pending input before any single buffer overflows. The increase indicated by the above equation is the minimum possible. Higher speed operations can also be used to further simplify the buffering logic.

[0323] The output of the above configuration is a single stream of an interleaved set of feature rows in the ordering described above. Subsequent layers do not need to subsample or buffer the 1:1 feature input. Lower-scale features are passed through at the expected times so that the pattern of context switches occurs in exactly the same pattern as those generated.

[0324]

number

[0325]

number

[0326]

number

[0327] Other rational or non-rational sampling patterns can be implemented for subscale ordering as well, and circuit designs can be made to accommodate any scaling configurations discovered in the training process for the CNN or other considerations.

[0328] Although the examples described so far produce a downscaled sample stream, this is not a limitation of the present invention. In some embodiments, sampling can also produce an upscaled stream.

[0329] The configuration of the scales does not change the core implementation of the aperture function. The difference between this variation and the previous single-scale implementation described above is that the single context switch from right-most to left-most based on the column counter is replaced with an external mechanism that manages multiple contexts for multiple scales. The calculations and edge rules for the aperture function that combine the current input value with the context for the current position are unchanged, and the overall circuit size is increased minimally.

[0330] Another application of context switching is to process interleaved rows from unrelated streams using the same aperture function. As with the scaling example above, the rows of inputs to the various arrays do not all need to be of equal width. Any input whose processing must be delayed must be buffered, and the overall frequency of the aperture function implementation must be increased so that the entire set of input streams is processed before a single buffer overflows.

[0331] Single-scale and multi-scale processing of different streams can be freely mixed, and different input streams do not need to be synchronized to start and end, or even at the same height, since the complete context of any given position at any given scale includes whether that position is at the top or bottom row.

[0332] Figure 36 illustrates the application of an aperture function IC circuit 3602 to an input array stream 3601 of ordered samples, such as image pixels or features, computed by an upstream CNN node. For any column position in the currently presented row, the context for the current row is several values ​​calculated for the corresponding column position in the previous row, assuming a previous row exists, along with exception flags maintained by the context manager, including the first and last column and first and last row signals. These values ​​are synchronously presented to the aperture function circuit 3602 by a separate context management circuit 3603 and combined with the values ​​calculated for the previous column position to generate data for the aperture function's final output stream 3604. Partial subfunction values ​​for the current position are also calculated and maintained by the context management circuit 3603 as context to be combined in subsequent rows. In the standard left-to-right, top-to-bottom sensor presentation order for individual samples, values ​​for individual elements of the aperture subfunction from the previous column are immediately used. In edge cases processing the first column of each row, values ​​are omitted from the calculation or combined according to the aperture function definition.

[0333] Other non-standard based presentation orders are possible as well, such as sensors that deliver data in interlaced rows or serpentine format (alternating left to right, right to left), which requires context and computational circuitry to enable this presentation order.

[0334] The partially computed subfunction values ​​for each column position are retained and become the context for subsequent rows. Both retaining and presenting may be combined in the IC circuitry implementation of aperture function 3602, but can be separated from the aperture function implementation without affecting the overall system size or efficiency.

[0335] Whenever a given span of input samples is being processed and processing of a different, discontinuous span must begin, the context of the last column processed must be preserved and set aside so that the context of the first column of the next span can be accessed. In a standard single-sample, left-to-right, top-to-bottom presentation order, this occurs as the column moves from the rightmost edge to the leftmost edge. For simplicity, the remainder of this discussion will refer to potentially any span as a row. In this simplest example, the value held as context in the previous row is presented as context to the next row in capture order.

[0336] Because no other data is required to process the currently presented row, the aperture function implementation has no requirement or dependency on the immediately following row being processed next, and therefore any row can be presented at any time as long as the context for that row is presented at the same time.

[0337] Figure 37 shows a simple example that accepts two independent input array streams, a first stream 3701 and a second stream 3702, and alternately presents rows from each stream through a store-and-forward multiplexer 3703 that holds the input from one stream while processing the other. Because values ​​from one stream are never mixed with values ​​from the other, the aperture function circuit 3602 is unchanged from the first example (Figure 36), in which context management is external to the aperture function circuit. In the system of Figure 37, context management is performed by a context circuit 3704 that is separate from the aperture function circuit.

[0338] Here, the retained context contains values ​​for each alternately captured stream, and the presented context presents the context for each stream alternately. If both streams have the same dimensions, capture and presentation can be combined into a single FIFO with twice the width and the same number of rows as required by the aperture function, which satisfies the requirement that the context for each row is presented simultaneously with the input array sample for that row. The final output 3705 is then presented with the two streams interleaved in time, and any subsequent nodes must accept this time-interleaved input array format. Because the input streams are independent, the complexity required to calculate the aperture function for a single stream of interleaved rows does not apply to any subsequent aperture nodes.

[0339] The two streams need not be of the same dimension, nor do they need to start and end at the same time. Any differences in array dimensions and frame starts and ends must be accommodated by capture and presentation contexts operable in context circuitry 3704 that take a form sufficient to satisfy the simultaneity requirement, but do not change aperture function 3602. The streams need not be presented at the same frame or pixel rate, in which case interleaving circuitry 3703 buffers the incoming data and passes completed rows at a processing frequency sufficient to keep up with all input streams without data loss.

[0340] This method of combining two or more independent streams of an input array can be applied to a variety of other processing situations, most notably a system that accepts a single input video stream and synthesizes each frame into a set of sampled scales, all of which are processed by the same aperture function.

[0341] Figure 38 shows the sequence of full and downscaled data rows for a series of four 2:1 reductions. The first unscaled substream 3801 shows 16 rows of pixel data labeled 1:1, meaning no reduction. In operation of a system such as that represented by Figure 37, rows R0 through R1 in substream 3801 16(and thereafter) may be presented to multiplexer 3703 as a first data stream. Once data from substream 3801 is presented, it may be reduced, for example, by averaging pairs of columns in each pair of rows of substream 3801 (a 2:1 reduction), to produce a set of eight half-width data rows as substream 3802. The resulting array of data presented by substream 3802 as R'0 through R'7 is one-quarter the area of ​​the array of data presented by substream 3801. Once downscaled data is created for substream 3802, it may be presented to multiplexer 3703 as a second parallel data stream. Substream 3803 represents data from substream 3802 further downscaled at a 2:1 ratio as R''0 through R''3, and substream 3804 represents data from substream 3803 further downscaled at a 2:1 ratio as R'''0 through R'''1. Data from sub-streams 3803 and 3804 are also presented to multiplexer 3703 as separate data streams.

[0342] Figure 39 shows

[0343]

number

[0344]

number

[0345] Figure 40, similar to Figure 36, shows the application of full-scale and scaled data to an aperture function 3602 in a situation where the data is presented as full-scale samples, typically an array of pixels 3601, and is reduced to be processed by a common aperture function. The output of the aperture function is a time-interleaved stream of differently scaled outputs in the same pattern as the interleaved sub-streams described in detail above.

[0346] A single input array stream 3601 is accepted by a cascaded multi-scale sampler 4001, which presents any desired scale of samples as a time-interleaved sequence 4002. The aperture function 3602 does not require modification. The presentation of sample rows and their associated contexts is maintained as needed. A context management circuit 4003 manages the partial outputs generated by the aperture function so that context data at various scales is accepted and presented at times corresponding to subsequent rows of the same scale. This can be achieved by embedding knowledge of the expected sequence, such as in the 2:1 case, or by direct signaling (not shown) from the sampler, such as in the case of any non-rational scaling ratio. The final output from the aperture function is presented as an interleaved multi-scale array stream 4004 of function-specific calculations, typically detected features.

[0347] Figure 41 illustrates the processing of the interleaved stream by a subsequent CNN node, which no longer requires scaling logic. The stream can be any mix of independent streams and synthesized sub-scaled frames. Context management circuitry 4102 performs syncopated context switching based on the established pattern of row representation. The output interleaved input stream 4101 from the previous node is fed to the aperture function 3602 unchanged, while the context management circuitry 4102 manages the partial output of the aperture function by either directly embedding the sequence of presentation or by accepting a signal (not shown) describing the row dimensions and transitions. The final output 4103 is a compatible interleaved stream that may or may not have the same dimensions as the input stream 4101 and may not be presented in the same order.

[0348] As an example, if the aperture function is a 2x2 tiled MaxPool node, the output stream will produce a half-width output row for every two input rows. For the 2:1 scaling case, the sequence of stream elements is the same. For any non-rational case, the sequence for a particular scale is not the same.

[0349] Figure 42A shows the sampling and syncopation logic required to generate interleaved streams of 1:1, 2:1, 4:1, and 8:1 scales. The complete input array stream 3601 of scalable values ​​is passed directly to a FIFO 4202, which holds any values ​​received during the processing of other scales. A 2:1 sampler 4201 accepts multiple samples from multiple rows, generates a single sample value, and then, upon completion, transfers it to a FIFO 4203 where it is held until processed. Another 2:1 sampler 4201 accepts 2:1 samples and transfers completed 4:1 samples to a FIFO 4204. Another 2:1 sampler 4201 accepts 4:1 samples and transfers completed 8:1 samples to a FIFO 4205. The syncopated multiplexer 4206 selects from the available data as each scaled row ostensibly becomes available and forwards that data to the interleaved sample stream 4002 along with any signals needed to describe it when no interleaving pattern is embedded in the holding logic.

[0350] Figure 42B shows a 1:1 ratio.

[0351]

number

[0352]

number

[0353]

number

[0354]

number

[0355]

number

[0356] Figure 43A shows the subsampling and syncopation logic required to generate interleaved streams from variously downscaled streams. The complete input array stream 3601 of scalable values ​​is passed directly to FIFO 4202, which holds values ​​received while other scales are being processed. U:1 sampler 4301 accepts multiple samples from multiple rows of the 1:1 stream, generates a single sample value, and then transfers it to FIFO 4304 upon completion to hold it until processed. V:1 sampler 4302 accepts U:1 samples and transfers completed UV:1 samples to FIFO 4305. W:1 sampler 4303 accepts UV:1 samples and transfers completed UVW:1 samples to FIFO 4306.

[0357] The syncopated multiplexer 4307 selects from the available data as each scale row ostensibly becomes available and forwards the data to the interleaved sample stream 4002 along with any signals needed to describe it when no interleaving pattern is embedded in the holding logic.

[0358] The interleaved data can take any form or ordering that is convenient for downstream processing. Complete row spans may reduce the complexity of the holding logic, but it is also feasible to generate sequences of individually scaled samples as they become available. The only requirement that must be met is that context for each individual sample must be captured and presented, which adds additional circuitry to the holding section (since results from the previous column are not always available and must be included in the context), but can be achieved if there is some advantage for certain node types. If it is desired to generate output for the full-scale input array before proceeding to other scales, it is also feasible to replace FIFO 4202 with an immediate pass-through while holding the individual input arrays with FIFOs 4304, 4305, and 4306.

[0359] Figure 43B illustrates the generation of interleaved multi-scale sample streams in another embodiment. In Figure 43B, W:1 sampler 4303 samples the U:1 downscaled stream instead of the V:1 downscaled stream, generating a UW:1 downscaled stream in FIFO 4309. V:1 sampler 4302 samples the 1:1 stream instead of the U:1 downscaled stream, generating a V:1 downscaled stream in FIFO 4308. The downscaled stream is processed by syncopated multiplexer 4307 to generate interleaved stream 4002.

[0360] FIG. 43C illustrates the generation of interleaved multi-scale sample streams in yet another embodiment. FIG. 43C shows all of the elements of FIG. 43A but adds an additional 1:T upscale sampler 4310. As a result, in the embodiment of FIG. 43C, there is a 1:1 full-scale stream held in FIFO 4202, three downscaled streams generated by samplers 4301, 4302, and 4303, and one upscaled stream generated by sampler 4310 and held in FIFO 4311, with all five streams interleaved by multiplexer 4312 to generate multi-scale sample stream 4002. Those skilled in the art will appreciate that in other embodiments there may be more upscaled streams and multiple sampling of a single upscaled or downscaled stream.

[0361] Implementing CNN models on a fixed ASIC with configurable elements In yet another aspect of the present invention, a system is provided in which a fixed-form circuit utilizing multiple replicated pipelined circuit elements may be configured to compute a CNN over an input array stream. One or more convolution nodes having a particular kernel size may be combined with some particular number of parallel input and output connections and replicated to provide the basis for other combinations of parallel input and output connections and kernel sizes other than those directly implemented.

[0362] In one embodiment of the present invention, an application-specific integrated circuit (ASIC) is provided that contains many copies of identical kernel processing tiles arranged in a pattern selected to enable unimpeded forward connections required to implement both stock and custom deep neural networks. In other embodiments of the present invention, the selection of kernel sizes present in the set of kernel processing tiles is optimized to support a particular model format. Additional auxiliary tiles can be included to perform computations other than convolutions. Since results do not need to flow upstream, it is not necessary for every tile to have access to every other tile. For efficiency, tiles are arranged in groups on the ASIC, which may be in linear rows or other formats that favor an efficient layout.

[0363] An example of an ASIC in accordance with an embodiment of the present invention is shown in Figure 44, which illustrates an ASIC in which kernel processing tiles comprise pipelined convolution circuits for a common kernel size (3x3 in this example) and are configured with selected parallel input and output connection sizes, i.e., 16 each in this example. Primary input 4401 to the system is presented as a multi-channel input stream. The channels in some embodiments are three channels for RGB or YUV video, provided on input bus 4402 and made available to a first group of tiles 4405 via multiplexers 4403 and 4404.

[0364] Those skilled in the art will appreciate that interconnection buses shown as single wires are actually a number of parallel conductors determined at least in part by the desired precision with which they are implemented in a system. Where branches of a bus connect, the connection is shown with an enlarged dot. Elsewhere, where buses are shown crossing, there is no connection between the crossing buses.

[0365] The primary input 4401 may comprise at least one of a direct camera interface, a DMA interface suitable for access from the CPU bus, a video stream decompression circuit, or other stream interface limited only by the parallel channel streams streamed onto the input bus 4402. The input stream need not be a multiple of the nominally 16 parallel connection size, as unused connections are disabled and ignored by parts of the circuit that do not utilize the connections.

[0366] FIG. 44 shows kernel processing tiles 4405, which in this example have identical physical structures, arranged in four columns. In this example, there are six kernel processing tiles shown in the first row (top). This first number of kernel processing tiles differs from the other groups of kernel processing tiles on the ASIC in that each of these first number is coupled to input bus 4402 via multiplexer 4403 and performs processing for the first layer of the CNN. It should be understood that the number of kernel processing tiles in this top row connected to input bus 4402 via multiplexer can be any number. In some minimal embodiments, it is sufficient, and is a goal of the ASIC in embodiments of the present invention, to enable the ASIC to be used for situations involving a variety of input and output channels and different kernel sizes.

[0367] The kernel processing tiles in this example are identical with 16 parallel input connections and 16 parallel output connections, and the tiles implement a 3x3 kernel. Although the interconnections and methods are described below, there may be any number of input and output channels, and the kernel size may be other than 3x3, such as 5x5, 7x7, or 9x9. Compatibility is provided by combining functions of multiple 3x3 kernels.

[0368] Each kernel processing tile 4405 in the first row receives a primary input in the form of a set of 16 parallel input connections from an input bus 4402 via a multiplexer 4403. Those skilled in the art will appreciate that the multiplexers providing input to each kernel processing tile 4405 are identical and therefore not all have element numbers. Each kernel processing tile also accepts input in the same format from the immediately preceding kernel processing tile, which may or may not be physically to its left as shown in FIG. 44. The first kernel processing tile in each row accepts an auxiliary input from a multiplexer 4404 from the output bus of the row above (except for the first row, which has no row above), allowing composite kernels to be split across multiple rows. Each kernel processing tile 4405 provides a primary output to an output bus 4406 and provides the same output to the next adjacent kernel processing tile, except for the last tile in the row.

[0369] The input bus 4402 and all output buses 4406 are unidirectional connections with a single driver for each connection set, providing inputs to multiple multiplexers 4403, 4407, 4409, and 4412, as well as to kernel processing tiles and other components. Buses are never used for bidirectional data flow; the depiction in Figure 44 simply shows that the output of one row is available as an input to the same row.

[0370] The values ​​on output bus 4406 are time-multiplexed. When the configured model is such that the values ​​presented to the set of physical connections are the same as the system's processing frequency, each physical connection carries only a single data channel for each array stream position. When the configured model presents values ​​to the set of physical connections at a frequency several times lower than the processing frequency, each physical connection may carry multiple time-multiplexed values ​​for each array stream position, or may hold a single value constant for multiple processing cycles. Each form of time multiplexing is advantageous for implementing model nodes with a large number of input and output channels.

[0371] As an illustrative example, a model in which a processing path flows through four 2x2 MaxPool nodes has a data rate that is reduced by exactly 256, allowing each physical connection to carry 256 data channels. Provision for exploiting this potential configuration is built into each kernel processing tile and auxiliary function tile by adding configuration logic and multiple indexes so that each tile can adapt to the proposed configuration. The net result is that the set of physical tiles required to implement a large model is reduced by several orders of magnitude. A detailed description of the use of configurable time multiplexing is provided below with respect to Figure 54.

[0372] The dual multiplexer 4407 provides two 16 parallel input connections to the auxiliary function tiles 4408 in this example, equivalent to two of the 4403 multiplexers. The outputs of the 16 parallel connections are provided by each auxiliary function tile 4408 to one of a set of 16 connections on the output bus 4406.

[0373] One or more sets of 16 parallel output connections (three are shown) are provided by multiplexer 4409 to external function circuitry 4410, which may or may not reside on the same ASIC die or in the same IC package. One or more sets of 16 parallel input connections are returned from external function circuitry 4410 as separate bus connections and made available on output bus 4406.

[0374] The output bus 4406 of each row of kernel processing tiles provides input to the next row, replacing the function of the input bus 4402 to the first row, and is combined with the output bus of that row and available to the kernel processing tile of that row via multiplexers 4403, 4404, and 4407, and to the external function interface of that row via multiplexer 4409.

[0375] Because data does not flow upstream due to the inherent nature of temporal pipelines that process data streams, not all possible connections are necessarily available. If the allocation of each particular tile is achieved by assigning usage in left-to-right order, unavailable connections on each bus 4406 may be omitted, effectively reducing overall circuit size without sacrificing functionality. The output of each kernel processing tile is then available to all kernel processing tiles in the row below it and to any kernel processing tiles to the right of the kernel processing tile that produces the output in the same row.

[0376] Multiplexer 4412 selects one or more sets of the 16 parallel input connections and provides them to primary output circuitry 4413. Primary output circuitry 4413 may include final output processing (typically SoftMax) and can export results via direct parallel connections, standard SERDES (Serializer / Deserializer), DMA (Direct Memory Access), or any other interface required to create a known result for the model.

[0377] Figure 45 shows a configuration in which kernel processing tiles processing a fixed 3x3 kernel of 16 parallel input and output connections are combined to output connections that make up a 3x3 kernel with more inputs, outputs, or both. The first two kernel processing tiles 4501 and 4502 on the top left are configured to take the same set of 16 input connections (by selecting them via a multiplexer, not shown) and generate 32 output channels on two different sets of 16 parallel output connections. The next two kernel processing tiles 4503 and 4504 are configured to compute 16 output channels from different weights applied to two different sets of 16 input channels by passing partial sums from the first kernel processing tile 4503 to the second kernel processing tile 4504, where the partial sums are combined with the remaining partial sums to generate the output on a single set of parallel output connections. The final four tiles 4505, 4506, 4507, and 4508 are configured as two sets of partial sum pairs that take inputs from two sets of 16 parallel input connections to produce a 3x3 convolution with a total of 32 inputs and 32 outputs.

[0378] The combination of the 3x3 kernel processing tiles in Figure 45 is actually achieved by the operation of multiplexers connecting the kernel processing tiles in each arrangement to buses. Those skilled in the art will understand that a 3x3 convolution pipelined node can be configured with any number of input and output channels by combining arbitrarily sized sets of kernel processing tiles together. Any node with inputs that are not an integer multiple of the data channel set size (nominally 16) simply ignores unused channels by setting their weights to 0 or otherwise disabling them in the calculation. Any node with outputs that are not an integer multiple of the channel set size may avoid calculating values ​​for unused channels, which will be ignored by downstream nodes.

[0379] The choice of parallel connection set size (16 in this example) is arbitrary, and the only impact of the granularity chosen is the proportion of unused connections in the complete configuration for a given model. There is no need, nor does it appear to be beneficial, for set sizes to be restricted to powers of two. While some greater efficiency in physical layout can be achieved by making all connections the same width, even this is not a requirement, and in some cases irregular or varying width connection sizes may be optimal.

[0380] Figure 46 shows the internal structure of a kernel processing tile 4405. A set of parallel input connections 4601 selected by a multiplexer 4403 (not shown, see Figure 44) is split into different single connections, in this example 16 connections, and each individual connection is provided to a large scale multiplier 4603 which computes the full set of possible multiples from its input. All multiples of all inputs are provided to each convolution unit 4604 along with a single connection from the auxiliary connection set 4602. The resulting outputs of each convolution unit are grouped into a set of 16 parallel output connections 4605 and made available to other tiles on an output bus 4406 (not shown, see Figure 44).

[0381] Figure 47 shows the internal structure of each convolution unit 4604. A full set of input products 4701 is provided by internal bus 4703 to each of the nine weight summation cells 4704 used in this 3x3 example. As with previously disclosed versions of pipelined circuitry for convolution, a pair of array stream-wide FIFOs 4705 delay the partial product sums to accommodate the requirement that a single circuit process the data as a 3x3 patch presented by the input stream. In this example, the final sum of each convolution is combined with a bias value in adder 4707, along with an optional value received from auxiliary input 4702 after a delay by FIFO 4706. Once the final sum has been combined with the bias and optional auxiliary input, the sum is output 4708 to the containing tile, where it forms one element of a parallel output connection set.

[0382] All FIFOs 4705 and 4706 delay values ​​by input stream position rather than by clock cycles to pass through a given time-multiplexed data. In the example described above in Figure 45, FIFO 4706 may be configured to pass data through without delay. FIFO 4706 exists to support the construction of the synthesis kernel shown in Figure 51 and described below.

[0383] Figure 48 shows a single summing cell 4704 that applies each of the presented input channel multiples 4801 and selects one particular multiple via configuration register 4802 and multiplexer 4803. The selected multiple is then passed through a variable shift register 4805 controlled by configuration register 4804 before being transferred to adder 4806, where it is combined with all the other selected and scaled multiples of the individual input channels.

[0384] The combination of configurable index 4802 and configurable scale 4804 allows floating-point weight values ​​to be applied to fixed-point feature or pixel values. The addition of one weight per input channel, combined with a configured bias value 4807, completes the computation for this step of the convolution over time. The summation cell 4704 can also add a delayed value 4808 from the previous row via FIFO 4705 and a forwarded partial sum 4809 from the cell to its left. If both are present, they are combined by adder 4810. In the first row of cells, there is no row above, so there is no delayed connection 4808, adder 4810 is omitted, and the forwarded value 4809 is connected directly to adder 4811, where it is combined with the bias value 4807. In the first column of cells, there is no column to the left, so there are no forwarded values, and adder 4810 is again omitted. In the single cell at the top left, there are no delay or forward connections, so adders 4810 and 4811 are both omitted and the bias value 4807 is routed directly to the final adder 4812 where it is combined with the sum of the adder tree to form the output 4813.

[0385] It is not essential that the present invention work with the large multipliers used. If the combination of large bus widths and large multiplexers is cumbersome, the actual input values ​​can be distributed (over a much smaller bus) and a local dual-input fixed multiplier can be used to set the actual value to be multiplied instead of the index. Shifter and scale registers are maintained to allow full floating-point weights to be applied from the fixed-point products.

[0386] If the selected precision of the system is very large, for example, 12 bits of precision in the mantissa, the large scale multiplier and transport bus may become excessively large. In this case, a smaller large scale multiplier can be used with a correspondingly smaller bus, which is then indexed and summed multiple times to generate each required product. In this example, a 6-bit large scale multiplier is used with two index values ​​and two (much smaller) multiplexers to select two independent multipliers, which are then combined with appropriate scaling to produce the same result as a 12-bit large scale multiplier.

[0387] The operation of the entire system is not affected by the selection of the multiplication method, only the path from the tile input 4601 to the scaler 4805 of each summation cell. The selection of the multiplication method and the configuration of the large multiplier bus width (if used) may be delayed until the ASIC is laid out and the optimal format in terms of size and power is determined.

[0388] Figure 49 shows the internal structure of auxiliary function tile 4408. The primary parallel input connection set 4901 selected by dual multiplexer 4407 is, in this example, routed through a variable FIFO 4902, with a configured delay, and optionally combined by adder 4904 with an auxiliary input 4903, also from multiplexer 4407. If summation is not required in the current configuration, multiplexer 4905 can feed the delayed input values ​​directly to a common lookup table 4906.

[0389] The common lookup table 4906 is a set of static registers configured with a mapping for each possible index value to some arbitrary output value. Each input channel controls a multiplexer to this set of register outputs so that the same function is applied to all input channels independently. This allows for the implementation of any activation function required by the model, such as, but not limited to, RELU, sigmoid, or tanh, without the need to include different circuitry to compute each such function.

[0390] If the configuration for this tile does not use a lookup table, the input can bypass it via multiplexer 4907, disabling lookup 4906 to save power.

[0391] Only one of the MaxPool 4908, Average 4909, Sample 4910, Expand 4911, and bypass or channel concatenation functions can be selected by a configurable multiplexer 4914 for every channel to form an output 4915 that drives a set of parallel output connections on the output bus 4406.

[0392] The MaxPool function block 4908 provides for the calculation of the maximum value over a configurable patch of the input stream over time, and also directly implements the stride mechanism, which has the effect of reducing the downstream data rate.

[0393] The Average function block 4909 provides calculation of average values ​​over configurable patches and may share circuitry with MaxPool 4908. The Sample function block 4910 provides cropping of input arrays and skip strides and may also share circuitry with the MaxPool 4908 and Average 4909 blocks.

[0394] Each function block that implements a non-unitary write mechanism effectively reduces the dimensionality of the input array stream. Because the frame rate of the entire stream remains constant, this proportionally increases the time that downstream nodes can process each input stream position. This provides the basis for time-multiplexed use of bus connections and determines the degree of time multiplexing available. Kernel processing tiles can utilize the increased time interval to transport and process multiple data channels across a fixed set of parallel connections. While weights are expanded to accommodate the additional values, multipliers and accumulators are reused, allowing for the expansion of a node's input and / or output channels without requiring the use of multiple kernel processing tiles. Control logic coordinates the placement of values ​​over time among connected tiles.

[0395] The Expand function block 4911 provides padding of the input arrays and duplication of values ​​to align the reduced input array stream with the unreduced input array stream.

[0396] Dedicated multiplexer 4912 concatenates channels obtained from primary parallel input connection 4901 with channels obtained from auxiliary parallel input connection 4903, effectively rerouting the channels to specific parallel output connections. In this example, where a fixed 16 input channels are used per tile, multiplexer 4912 can take the lowest channels 1 through 15 from input 4901 and the corresponding lowest channels 15 through 1 from input 4903 to generate a single set of 16 channels to be selected by multiplexer 4914. Because the assignment of data channels to specific positions in the kernel is arbitrary (any input data channel and corresponding weight can occupy any position within the kernel without changing the result), when all used channels are placed in the lowest available positions in each set of 16 parallel connections, this configuration is sufficient to perform all rerouting required by the CNN model to be implemented. Use of this multiplexer is limited to data channels with equivalent time multiplexing.

[0397] Another specialized multiplexer 4913 combines two input channels into one output channel by alternating samples from each set of parallel input connections. To use this circuit, each input value on both channels must be held for two or more stream processing cycles. The output value is held for half the length of the incoming value and occupies only a single parallel output connection set on the bus.

[0398] When neither a patch function nor a concatenation is selected by the configurable multiplexer 4914, the tile operates in bypass mode, which can be useful to provide a delay path between a set of parallel input connections and the kernel processing tile it will be used in the subsequent row.

[0399] Figure 50 shows the elements required to implement any of the patch functions contained in the auxiliary tile. A single input channel 5001 is fed into patch function circuitry 5002, along with the delay value emitted by that patch circuitry from the previous row, and output 5004 as the final value for each patch. This allows any function to be computed that treats all input stream positions uniformly.

[0400] In the case of MaxPool 4908, when a patch function is defined to span multiple rows, the function selects the largest value seen and uses FIFO 5003 to associate the patch value from the previous row with the current row. In the case of Average 4909, the function accumulates the sum of all values ​​in each patch and uses FIFO 5003 to present the sum from the previous row. In the case of Sample 4910, one selected location within each patch is output as the value for the entire patch, and when the selected value does not fall in the last row of the patch being processed, FIFO 5003 is used to hold the selected value from the previous row. Each of these patch functions produces only a single value per patch, which has the net effect of reducing both the area of ​​the input stream array passed downstream and the throughput required to maintain stream processing.

[0401] In the case of Expand4911, the value from each distinct position in the input stream is repeated one or more times in the output stream, and a FIFO5003 is used to hold the values ​​encountered before iterating again for subsequent rows where the function is defined as producing multiple rows. This function has the effect of increasing the area of ​​the input stream array passed downstream and the throughput required to sustain stream processing, however, different paths through the model may have different sizes and the array sizes must match when they need to be recombined.

[0402] Figure 51 shows the configuration of connections to external circuitry. Inputs 5101 represent one set of external parallel input connections, which may be in the form of parallel buffers, SERDES, or any other format that may be convenient for connection to other circuitry, and provide a set of parallel output connections to output bus 4406. Configurable multiplexer 5102 selects one set of parallel input connections from bus 4406 and passes it to external output 5103, which may be in any output format. It is expected that at least some of the external inputs and outputs will be compatible with each other and can be used to connect multiple units of the ASIC to construct models that are larger than can fit in the provided tiles.

[0403] An important use of external circuitry is to provide arbitrary functionality that was not included in the auxiliary function tile 4408. This feature effectively "future-proofs" the ASIC, eliminating the requirement to predict the results of future developments in the model design.

[0404] Figure 52 shows the configuration of four 3x3 kernel processing tiles 4405 employed to compute a 5x5 convolution. All elements not involved in this configuration are omitted from this figure and are disabled by the configuration.

[0405] The kernel processing tile in the top left position is configured to compute a 3x3 convolution with the top left 3x3 value of the 5x5 patch of the input stream, and the kernel processing tile in the top right position is configured to compute a 2x3 convolution with the top right 2x3 value of the 5x5 patch.

[0406] The bottom left kernel processing tile is configured to compute a 3x2 convolution with the bottom left 3x2 value, and the bottom right kernel processing tile is configured to compute a 2x2 convolution with the bottom right 2x2 value of the 5x5 patch.

[0407] Both FIFOs 4706 are configured for a delay of two positions, since the left column of tiles produces final results corresponding to two input stream positions at a time, before the right column of tiles produces their final results.

[0408] The output of the top row adder 4707 is selected from output bus 4406 and routed as the primary input to FIFO 4902. Because the top row of tiles completes calculations for the top 5x3 section of the 5x5 convolution, corresponding to two array widths, at a time before the bottom row of tiles completes, FIFO 4902 must be configured to delay the partial sums by two array widths of input stream positions.

[0409] The output of adder 4707 in the bottom row is selected from output bus 4406 and routed as an auxiliary input to adder 4904, where it is combined with the delayed partial sum from the top row to produce the final result of the 5x5 convolution, which is then passed through lookup table 4906, where an activation function is applied.

[0410] Figure 53 shows the configuration of nine similarly arranged 3x3 kernel processing tiles to implement a 7x7 kernel. The timing difference between the kernel processing tiles is the same as in the previous example, so the FIFOs are configured for the same delay. The auxiliary adder 4904 on the left combines the partial results from the first two rows, and the adder 4904 on the right combines those values ​​with the partial result from the lower row and forwards it to the lookup table as the final result.

[0411] Those skilled in the art will appreciate that any convolution kernel size can be configured with any kernel processing tile kernel size using a similar configuration of partially disabled elements and appropriate delays. Combining this method with the tile configuration shown in Figure 45 allows for the configuration of any kernel with any number of input and output channels.

[0412] The inclusion of multiplexers 4404 in each row of tiles is specifically intended to accommodate large convolutions that may span multiple groups of tiles arranged within an ASIC. While it is anticipated that regular rows of replicated kernel processing tiles may be optimal under certain circumstances, the operation of the present invention is by no means limited to regular configurations. When designing an ASIC implementing the present invention, any combination of kernel processing tile size and tile grouping can be selected to reflect the needs of a particular model, or a statistically determined combination can be selected to cover the widest range of models within a single ASIC die. If a model requires more resources than can be addressed by the tiles of a single ASIC, multiple ASICs can be chained together to further expand the ability to cover arbitrarily large models.

[0413] Figure 54 shows an abstract example of a functionally complete convolutional neural network. Input 5401 provides RGB pixel data to three of the 16 parallel connections in the exemplary implementation. The other 13 parallel connections are unused.

[0414] Convolution 5402 is a 3-input, 32-output, 7x7 convolution with a RELU activation function, which in this example is implemented by two different sets of nine 3x3 kernel processing tiles and two auxiliary tiles to produce a total of 32 output data channels. The RELU activation function is implemented by a lookup table in the final auxiliary tile for each set of 16 parallel output connections.

[0415] Sample 5403 is a 2x2 subsampling of the convolution output, a typical configuration found in ResNet and YOLO model variations. The Sample function can be selected at each of the final auxiliary tiles included in the 7x7 convolution, thus no additional tiles are required. The output of the Sample function generates an array stream of 1 / 4 the area at 1 / 4 the processing rate, per position. Each output value is held constant for four stream processing cycles until the next value is presented to the parallel output connections. This provides an opportunity for four channels of time-multiplexed data at downstream nodes.

[0416] The convolution 5404 is a 32-input, 64-output, 5x5 convolution with a RELU activation function. The 32 input channels are presented to the multiplexer as two sets of 16 parallel input connections, so that two sets of four 3x3 kernel processing tiles are stitched together to apply weights for each individual output channel to the input. However, because the input is presented at one-quarter the stream processing rate, four sets of output channel values ​​may be calculated and placed onto a single set of 16 parallel output connections, each carrying time-multiplexed data.

[0417] The MaxPool5405 is a 64-input, 64-output, 2x2 max function configured to accept and process four time-multiplexed values ​​across each of 16 parallel input connections. The output is the maximum value of each individual channel found within a patch that is two positions wide and two positions high. The outputs are placed onto a single set of 16 parallel output connections, with each value held constant for four stream processing cycles before the next value is presented. Only a single auxiliary tile is required to process all 64 channels of time-multiplexed data. This provides an opportunity for 16 channels of time-multiplexed data in downstream nodes.

[0418] Convolution 5406 is a 64-input, 128-output, 3x3 convolution with a sigmoid activation function, accepting 64 time-multiplexed inputs across a single set of 16 parallel input connections and producing 128 time-multiplexed output values ​​across a single set of 16 parallel output connections. Each output value is held for two stream processing cycles instead of the incoming four. Auxiliary tiles with appropriate values ​​loaded into lookup tables implement the sigmoid activation function with equivalent timing on the data channels.

[0419] The MaxPool5407 is a 128-input, 128-output, 2x2 max function configured to accept and process eight time-multiplexed values ​​across each of 16 parallel input connections. The output is the maximum value of each individual channel found within a patch two positions wide and two positions high. The outputs are placed onto a single set of 16 parallel output connections, with each value held constant for eight stream processing cycles before the next value is presented. Only a single auxiliary tile is required to process all 128 channels of time-multiplexed data. This provides an opportunity for 64 channels of time-multiplexed data at downstream nodes.

[0420] The convolution 5408 is a 128-input, 256-output, 3x3 convolution with arbitrary activation functions, accepting 128 time-multiplexed inputs over a single set of 16 parallel input connections and producing 256 time-multiplexed output values ​​over a single set of 16 parallel output connections. Each output value is held for four stream processing cycles instead of the incoming eight. Auxiliary tiles with appropriate values ​​loaded into lookup tables implement arbitrary activation functions with equivalent timing on the data channels.

[0421] Expand 5409 is a 256-input, 256-output, 2x2 expansion that replicates each input value twice in each of two rows. The 256 time-multiplexed output channels are arranged onto a single set of 16 parallel output connections, but each replicated value is held constant for only one stream processing cycle instead of the incoming four. This function is needed to make the output array size compatible with the output of Convolution 5410, described below, so that concatenation of multiple channels at equivalent locations can be implemented as defined in the model. This reduces the chance of 16 channels of time-multiplexed data at downstream nodes.

[0422] The convolution 5410 is a 128-input, 256-output, 3x3 convolution that accepts 128 time-multiplexed inputs over a single set of 16 parallel input connections and produces 256 time-multiplexed output values ​​over a single set of 16 parallel output connections. Each output value is held for only one stream processing cycle, rather than two incoming ones. Auxiliary tiles with appropriate values ​​loaded into lookup tables implement activation functions with equivalent timing on the data channels.

[0423] Concatenation 5411 is implemented by two different streams of 256 input channels with compatible timing and array stream positions, concatenation into a single set of 512 output channels. Because each value in the two 16 parallel connections is only held for one stream processing cycle, no further multiplexing is required or feasible. Therefore, the concatenation function is represented in the circuit purely by the organization of time-multiplexed data on the bus connection; in this case, no auxiliary tiles are required.

[0424] The MaxPool5412 is a 512-input, 512-output, 2x2 max function configured to accept and process 16 time-multiplexed values ​​across two sets of 16 parallel input connections. Because the incoming stream is presented to two different sets of parallel input connections, the outputs are initially placed onto two sets of 16 parallel output connections, with each value held constant for four stream processing cycles before the next value is presented. These two sets of 16 parallel output connections are then routed to another auxiliary tile, where they pass through a time-multiplexer section to generate 512 data channels carried by a single 16 parallel output connection, with each data value held constant for two stream processing cycles instead of four.

[0425] Convolution 5413 is a 512-input, 1024-output, 1x1 convolution that accepts 512 time-multiplexed inputs across a single set of 16 parallel input connections and produces 1024 time-multiplexed output values ​​across a single set of 16 parallel output connections. Each output value is held for only one stream processing cycle, rather than two incoming ones. The 1x1 convolution is implemented by utilizing only the bottom-right output summation cell of the nine cells in a 3x3 kernel processing tile.

[0426] Average 5414 is configured to accept 1024 time-multiplexed data channels over a single set of 16 parallel input connections and calculate the average of the input values ​​over a patch size equal to the entire reduced array stream.

[0427] The output 5415 of the model is delivered as 1024 time-multiplexed values ​​on 16 external connections representing the presence or absence of 1024 different categories.

[0428] This model form is not intended as a robust example, but is devised to take advantage of many attributes of the present invention and to show how the circuit actually works.

[0429] Those skilled in the art will appreciate that all of the embodiments shown in the drawings and described above are exemplary and do not detail every possible form the invention may take, as there may be various other forms that may be realized within the scope of the invention.

[0430] The scope of the present invention is limited only by the claims.

Claims

1. 1. An application specific integrated circuit (ASIC) for computing a convolutional neural network (CNN), comprising: a first input bus for receiving an ordered stream of values ​​from an array, each location in the array having one or more data channels; a first ordered set, from first to last, of kernel processing tiles having a fixed number of parallel input connections and a fixed number of parallel output connections, each kernel processing tile in the first ordered set coupled to an input bus via one of a first set of configurable multiplexers, the kernel processing tiles adapted to compute convolutions for a common kernel size and pass the computed values ​​as inputs back to each configurable multiplexer in the first set of configurable multiplexers, to a connected first output bus, and also to adjacent downstream kernel processing tiles in the first ordered set; a second ordered set, from first to last, of kernel processing tiles having a fixed number of parallel input connections and a fixed number of parallel output connections, each kernel processing tile in the second ordered set coupled to a first output bus via one of the second set of configurable multiplexers, the kernel processing tiles of the second ordered set adapted to compute convolutions for a common kernel size and pass the computed values ​​as inputs back to each configurable multiplexer in the second set of configurable multiplexers to the connected second output bus, as well as to adjacent downstream kernel processing tiles in the second ordered set; a third ordered set, from first to last, of kernel processing tiles having a fixed number of parallel input connections and a fixed number of parallel output connections, each kernel processing tile in the third ordered set coupled to a second output bus via one of the third set of configurable multiplexers, the kernel processing tiles of the third ordered set adapted to compute convolutions for a common kernel size and pass the computed values ​​to the connected third output bus as inputs back to each configurable multiplexer in the third set of configurable multiplexers, and also to adjacent downstream kernel processing tiles in the third ordered set, the third output bus also connected via a single primary output multiplexer to a primary output circuit adapted to perform primary output processing and provide a final output; An application specific integrated circuit (ASIC) comprising:

2. 10. The ASIC of claim 1, further comprising: an additional configurable multiplexer coupled to the first output bus, the additional configurable multiplexer providing a selected value to a first kernel processing tile in the second ordered set; and an additional configurable multiplexer coupled to the second output bus, the additional configurable multiplexer providing a selected value to a first kernel processing tile in the third ordered set.

3. The ASIC of claim 1 further comprising one or more auxiliary function tiles that provide functions other than those of the kernel processing tiles.

4. 4. The ASIC of claim 3, wherein one or more auxiliary function tiles accept inputs from a dual multiplexer connected to the input bus and the first output bus, and provide outputs to the first output bus.

5. 4. The ASIC of claim 3, wherein one or more auxiliary function tiles accept inputs from a dual multiplexer connected to a first output bus and a second output bus, and provide outputs to the second output bus.

6. 4. The ASIC of claim 3, wherein one or more auxiliary function tiles accept inputs from a dual multiplexer connected to the second output bus and the third output bus, and provide outputs to the third output bus.

7. 10. The ASIC of claim 1, further comprising external function circuitry that selects an input from the first output bus via a configurable multiplexer and provides an output to the first output bus.

8. 8. The ASIC of claim 7, further comprising external function circuitry that selects an input from a second output bus via a configurable multiplexer and provides an output to the second output bus.

9. 9. The ASIC of claim 8, further comprising external function circuitry that selects an input from a third output bus via a configurable multiplexer and provides an output to the third output bus.

10. 2. The ASIC of claim 1, wherein the common kernel size is a 3x3 kernel.

11. 2. The ASIC of claim 1, wherein the fixed number of parallel input connections is 16 and the fixed number of parallel output connections is 16.

12. 10. The ASIC of claim 1, wherein the ordered stream of values ​​is provided by one of a direct camera output of RGB values, a DMA interface suitable for access from a CPU bus, or a video stream decompression circuit to produce three parallel channels of red, green, and blue (RGB) values ​​for the image.

13. 12. The ASIC of claim 11, comprising combining operations of two or more kernel processing tiles processing a 3x3 kernel of 16 parallel input and output connections to form a 3x3 kernel with more than 16 inputs, or more than 16 outputs, or both more than 16 inputs and more than 16 outputs.

14. 11. The ASIC of claim 10 adapted by additional circuitry to compute convolutions having kernels larger than 3x3 by combining operations of multiple 3x3 kernel processing tiles.

15. 15. The ASIC of claim 14 adapted to compute a 5x5 convolution, a 7x7 convolution, or a 9x9 convolution.

16. 2. The ASIC of claim 1, wherein a kernel processing tile presents each input channel value to a large scale multiplier, which computes a full set of possible multipliers from the inputs and provides the multipliers, along with single channel values ​​from an auxiliary set of parallel connections, to a single convolution unit, and the resulting outputs from each convolution unit are grouped in a set of 16 parallel output connections and made available to other kernel processing tiles on an output bus.

17. 2. The ASIC of claim 1, wherein input values ​​of a kernel processing tile are processed by a local dual-input fixed multiplier.

18. 4. The ASIC of claim 3, wherein each one of the auxiliary function tiles receives parallel input connections from two separate buses via a dual multiplexer and outputs one of a MaxPool function, an Average function, a Sample function, and an Expand function selected by an output multiplexer.

19. 20. The ASIC of claim 18, wherein the auxiliary function tile sums values ​​from parallel input connections by individual channels and multiplexes the summed values ​​into a lookup table adapted to provide any activation function that can be expressed in tabular form, including RELU, sigmoid, or tanh activation functions.

20. 20. The ASIC of claim 18, wherein input channels received from separate parallel input connections are input to a first dedicated multiplexer that concatenates the parallel input connections to effectively reroute data channels to specific parallel output connections and provides the concatenated output to an output multiplexer as a candidate for selection for the output of the auxiliary function tile.

21. 20. The ASIC of claim 18, wherein input channels received from separate parallel input connections are input to a second specialized multiplexer, which combines the two parallel input connections into one parallel output connection by alternating samples from each connection, and provides the results to an output multiplexer as candidates for selection of an output of an auxiliary function tile.

Images