Combining arbitrary convolutional neural network models from fixed set of repeating pipeline components

Abstract

An application specific integrated circuit (ASIC) for computing a convolutional neural network (CNN) has a first input bus that receives a stream of ordered values from an array, each location in the array having one or more channels, and a plurality of kernel processing blocks that receive inputs through a configurable multiplexer. The arrangement and connection of the kernel processing blocks and the bus enable the ASIC to operate as a pipeline system and transmit the output stream in a manner synchronized with the input stream.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation-in-part of co-pending application 17 / 742,245 filed on May 11, 2022, which is a continuation-in-part of application 17 / 570,757 filed on January 7, 2022, now published as US11,354,571 on June 7, 2022, which is a continuation-in-part of application 17 / 373,497 filed on July 12, 2021, now published as US11256981 on February 22, 2022, which is a continuation-in-part of application 17 / 231,711 filed on April 15, 2021, now published as US11099854 on August 24, 2021, which is a continuation-in-part of co-pending application 17 / 071,875 filed on October 15, 2020. All disclosures of the parent application are incorporated herein by reference at least in part. Technical Field

[0003] The present invention belongs to the technical field of computer operations involving matrix input and output, and more particularly, to a circuit designed for batch multiplication in matrix operations. Background Art

[0004] The use of computers in matrix operations is well known, and specific examples include image processing and the development and use of neural networks. Neural networks are an important component of artificial intelligence and are therefore a very popular subject for intellectual property development at the time of filing this patent application. Generally speaking, in such computer operations, a large number of input values ​​are processed according to a regular pattern, which in most cases is a matrix. The processing of the input values ​​may involve biasing and applying weights, by which the individual input values ​​may be multiplied.

[0005] The inventors believe that in neural network technology, the complex and computationally intensive operation of multiplying an incoming value by each of a plurality of weight values ​​is a step to be innovated that can provide unique advantages to the technology. The inventors also believe that modifying the order in which the mathematical processes are applied can bring advantages.

[0006] The inventor believes that he has identified general changes in the order and manner of carrying out mathematical processes in such applications that are likely to significantly reduce the time and cost of such operations. Summary of the invention

[0007] In an embodiment of the present invention, an application-specific integrated circuit (ASIC) for computing a convolutional neural network (CNN) is provided, which includes: a first input bus for receiving an ordered stream of values ​​from an array, each position in the array having one or more data channels; a first ordered set, from the first to the last, comprising kernel processing blocks having a fixed number of parallel input connections and a fixed number of parallel output connections, each kernel processing block of the first ordered set being coupled to the input bus via a configurable multiplexer in a first group of configurable multiplexers, the kernel processing block being adapted to compute convolutions of a common kernel size and pass the computed values ​​to a first output bus, the first output bus being connected back as an input to each configurable multiplexer in the first group of configurable multiplexers, and to an adjacent downstream kernel processing block of the first ordered set; a second ordered set, from the first to the last, comprising kernel processing blocks having a fixed number of parallel input connections and a fixed number of parallel output connections, each kernel processing block of the second ordered set being coupled to the input bus via a configurable multiplexer in a second group of configurable multiplexers. A first output bus is coupled, the core processing blocks of the second ordered set are suitable for calculating the convolution of the common kernel size and passing the calculated values ​​to the second output bus, the second output bus is connected back as an input to each configurable multiplexer in the second group of configurable multiplexers, and is passed to the adjacent downstream core processing blocks of the second ordered set; and a third ordered set, from the first to the last, contains core processing blocks with a fixed number of parallel input connections and a fixed number of parallel output connections, each core processing block of the third ordered set is coupled to the second output bus through a configurable multiplexer in the third group of configurable multiplexers, the core processing blocks of the third ordered set are suitable for calculating the convolution of the common kernel size and passing the calculated values ​​to the third output bus, the third output bus is connected back as an input to each configurable multiplexer in the third group of configurable multiplexers, and is passed to the adjacent downstream core processing blocks of the third ordered set, and the third output bus is also connected to the main output circuit through a single main output multiplexer, and the main output circuit is suitable for performing main output processing and providing a final output.

[0008] In one embodiment, the ASIC further comprises an additional configurable multiplexer coupled to the first output bus to provide a selected value to a first core processing block in a second ordered set, and an additional configurable multiplexer coupled to the second output bus to provide a selected value to a first core processing block in a third ordered set. In addition, in one embodiment, the ASIC further comprises one or more auxiliary function blocks to provide additional functionality beyond the functionality of the core processing blocks. In one embodiment, the one or more auxiliary function blocks receive inputs from a dual multiplexer connected to the input bus and the first output bus and provide outputs to the first output bus. In another embodiment, the one or more auxiliary function blocks receive inputs from a dual multiplexer connected to the first output bus and the second output bus and provide outputs to the second output bus.

[0009] In one embodiment of the present invention, one or more auxiliary function blocks accept inputs from a dual multiplexer connected to a second output bus and a third output bus and provide outputs to the third output bus. In addition, in one embodiment, the ASIC further includes an external function circuit that selects inputs from the first output bus through a configurable multiplexer and provides outputs to the first output bus. In addition, in one embodiment, the ASIC further includes an external function circuit that selects inputs from the second output bus through a configurable multiplexer and provides outputs to the second output bus. In one embodiment, the ASIC further includes an external function circuit that selects inputs from the third output bus through a configurable multiplexer and provides outputs to the third output bus. In one embodiment, the common core size is a 3×3 core.

[0010] In one embodiment, the fixed number of parallel input connections is 16 and the fixed number of parallel output connections is 16. Further, in one embodiment, the ordered stream of values ​​is provided by one of a direct camera output of RGB values, a DMA interface adapted for access from a CPU bus, or a video stream decompression circuit, thereby producing three parallel channels of red, green, and blue (RGB) values ​​for an image. Further, in one embodiment, the ASIC further comprises combining the operation of two or more kernel processing blocks, the kernel processing blocks processing a 3×3 kernel with 16 parallel input and output connections to configure a 3×3 kernel with inputs greater than 16 or outputs greater than 16 or both inputs and outputs greater than 16. In one embodiment, the ASIC is adapted by additional circuitry to compute convolutions with kernels greater than 3×3 by combining the operation of multiple 3×3 kernel processing blocks. And in one embodiment, the ASIC is adapted to compute 5×5 convolutions, 7×7 convolutions, or 9×9 convolutions.

[0011] In one embodiment, the core processing block presents each input channel value to a mass multiplier, which computes a complete set of possible multiples from the input and provides the multiples together with the single channel values ​​from the auxiliary parallel connection set to a single convolution unit, each of which produces a final output consisting of a set of 16 parallel output connections and provided to other core processing blocks on an output bus. In one embodiment, the input values ​​of the core processing block are processed by a local dual input fixed multiplier. In addition, in one embodiment, one of the auxiliary function blocks receives parallel input connections from two separate buses through a dual multiplexer and outputs one of the MaxPool, Average, Sample and Expand functions selected by the output multiplexer. In one embodiment, the auxiliary function block sums the values ​​from the parallel input connections on a single channel basis and multiplexes the summed values ​​into a lookup table, which is suitable for providing any activation function that can be represented in a tabular form, including RELU, sigmoid or tanh activation functions. In one embodiment, the input channels received from different parallel input connections are inputs to a first dedicated multiplexer, which cascades the parallel input connections to effectively reroute the data channels to specific parallel output connections and provides the cascaded outputs to an output multiplexer as selection candidates for the auxiliary functional block output. And in one embodiment, the input channels received from different parallel input connections are inputs to a second dedicated multiplexer, which cascades the two parallel input connections into one parallel output connection by alternately sampling from each connection and provides the result to the output multiplexer as a selection candidate for the auxiliary functional block output. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 An embodiment is shown in which the bulk multipliers applied to each common source are fixed and connected directly into the processing circuitry.

[0013] Figure 2 An embodiment is shown in which the bulk multipliers applied to each common source are dynamic and routed to the processing circuitry through a multiplexer.

[0014] Figure 3 A simple embodiment is shown in which the shift terms corresponding to the bit settings in each bulk multiplier are summed to form the product.

[0015] Figure 4 An enhanced embodiment is shown in which addition and subtraction of shift terms are intermixed to form an equivalent solution of lower complexity.

[0016] Figure 5AA pipelined embodiment is shown that builds subcombinations only by pairwise operations, thereby maximizing clock frequency.

[0017] Figure 5B Embodiments are shown in which multiples are formed directly from a fixed set of cases without reference to standard arithmetic operations.

[0018] Figure 6 A pipelined embodiment is shown that builds subcombinations with a maximum of four operations to maximize circuit density.

[0019] Figure 7 is a schematic diagram of the structure and connections of an embodiment of the present invention, which receives an input stream, pre-processes the input stream, and feeds the result through a unique digital device to produce an output stream.

[0020] Fig. 8A is a schematic diagram showing the structure and connections for generating source channel products.

[0021] Figure 8B is a schematic diagram showing additional details of controls and functionality in an embodiment of the invention.

[0022] Fig. 9A It is a partial schematic diagram of a general situation of pipeline operation in an embodiment of the present invention.

[0023] Fig. 9B is another partial schematic diagram of a general situation of pipeline operation in an embodiment of the present invention.

[0024] Fig. 9C is another partial schematic diagram of a general situation of pipeline operation in an embodiment of the present invention.

[0025] Fig. 10A This is an example of an embodiment of the present invention. Fig. 9A and 9B Schematic diagram of the internal structure of synthesizers 905a, 905b and 905c.

[0026] Fig. 10B This is an example of an embodiment of the present invention. Fig. 9A and Fig. 9B Schematic diagram of the internal structure of synthesizers 902a, 902b and 902c.

[0027] Fig. 10C This is an example of an embodiment of the present invention. Fig. 9A Schematic diagram of the internal structure of synthesizer 904.

[0028] Fig. 10D This is an example of an embodiment of the present invention. Fig. 9ASchematic diagram of the internal structure of synthesizer 901.

[0029] Fig.10E This is an example of an embodiment of the present invention. Fig. 9B and Fig. 9C Schematic diagram of the internal structure of synthesizers 903a, 903b and 903c.

[0030] Fig.10F This is an example of an embodiment of the present invention. Fig. 9A and Fig. 9B Schematic diagram of the internal structure of synthesizers 907a, 907b and 907c.

[0031] Figure 10G This is an example of an embodiment of the present invention. Fig. 9A Schematic diagram of the internal structure of synthesizer 906.

[0032] Fig.11 The embodiments of the present invention are described in Fig. 9C Schematic diagram of the internal structure and function of delay stages 908a, 908b, 908c, 908d, 908e and 908f.

[0033] Fig.12 This is an example of the present invention. Fig. 9C Schematic diagram of the operation of the delay stage 909.

[0034] Fig.13 This is an example of an embodiment of the present invention. Fig. 9C Schematic diagram of the operation of delay stages 910a and 910b.

[0035] Fig.14 It is shown Fig. 9C Schematic diagram of the operation of completing step 911.

[0036] Fig.15 It is a schematic diagram showing the specific situation of implementing the pipeline operation of 5×5 convolution nodes in an embodiment of the present invention.

[0037] Fig.16 An integrated circuit (IC) for a 4×4 aperture function in an embodiment of the present invention is shown.

[0038] Fig.17A An IC with circuitry implementing a portion of a deep neural network capable of streaming input channels individually is shown.

[0039] Fig. 17B An IC with circuitry implementing another portion of a deep neural network is shown.

[0040] Fig.18AAn IC is shown having circuitry implementing a portion of a deep neural network capable of streaming four input channels simultaneously.

[0041] Fig.18B Shows the implementation Fig.18A Another part of the deep neural network circuit.

[0042] Fig.19 It is shown Fig.17A and Fig. 17B Table of array flow sizes for DNNs.

[0043] Fig. 20 It is shown Fig.18A and Fig.18B Table of array flow sizes for DNNs.

[0044] Fig.21 The circuit of an IC capable of streaming four input channels simultaneously and performing 3×3 convolution nodes is shown.

[0045] Fig. 22 The circuit layout required to produce an output for 4-up input channels for the “same” version of a 3×3 convolution is shown.

[0046] Fig.23 The layout required for two variants of a convolution with a circuit outputting 1 row and 7 columns (streaming four input channels simultaneously) is shown.

[0047] Fig.24A The figure shows the layout of 2×2 MaxPool nodes on a 4-up data flow.

[0048] Fig. 24B Shows Fig.24A Layout of 2×2 MaxPool nodes in 2-up data flow.

[0049] Fig.25 An artificially designed example where N cannot be reduced is shown.

[0050] Fig.26A A FIFO circuit is shown for repackaging a 4-up stream into a 2-up stream.

[0051] Fig.26B Repackaging of a 3-up stream into a 5-up stream is shown.

[0052] Fig.27A An implementation of cascading nodes is shown so that the output contains all channels from all sources.

[0053] Fig.27B An implementation of a 4-up dense node is shown.

[0054] Fig.27C An implementation of a 4-up Global Average node is shown.

[0055] Fig.28 A 4-up implementation of 3×3 Local Average nodes is shown.

[0056] Fig.29 Another implementation of 4-up of 3×3 local averaging nodes is shown.

[0057] Fig. 30A An implementation of a 4-up Subset node is shown.

[0058] Fig. 30B A typical implementation of a 4-up crop node (Crop Node) is shown.

[0059] Fig.31 A system of interconnected ICs implementing a neural network is shown.

[0060] Fig.32 A layout of a synthesizer on an integrated circuit is described, the layout being configured to perform a 3×3×3 convolution used as a 3D aperture function to process twenty-seven independent data samples.

[0061] Fig.33 An IC is shown in which data from multiple planes can be buffered and rendered simultaneously so that weights for multiple planes can be applied by a single compositor.

[0062] Fig.34 An implementation of a typical 3×3×3 convolution applied to a 4-up input stream is shown.

[0063] Fig.35 A fully everted implementation of the IC applied to 4-up data flow is shown.

[0064] Fig.36 The application of an aperture function IC circuit to an input array stream 3601 of ordered samples in an embodiment of the present invention is shown.

[0065] Fig.37 An example of receiving two independent input array streams in an embodiment of the present invention is shown.

[0066] Fig.38 A sequence of full and scaled data rows for a series of four 2:1 data reductions in an embodiment of the present invention is shown.

[0067] Fig.39 A sequence of full and scaled rows is shown for irrational scaling in an embodiment of the present invention.

[0068] Fig.40 The application of full-scale and zoomed data to an aperture function in an embodiment of the present invention is shown.

[0069] Fig.41 The processing of the interleaved stream by subsequent CNN nodes in an embodiment of the present invention is shown.

[0070] Fig.42A The sampling and splitting logic required to generate an interleaved stream in an embodiment of the present invention is shown.

[0071] Fig.42B Another example of sampling and slicing logic required to generate an interleaved stream in an embodiment of the present invention is shown.

[0072] Fig.43A The sub-sampling and slicing logic required to generate an interleaved stream from various scaled streams in an embodiment of the present invention is shown.

[0073] Fig.43B Generating an interleaved multi-scale sample stream in another embodiment is shown.

[0074] Fig.43C In yet another embodiment, generating an interleaved multi-scale sample stream is shown.

[0075] Fig.44 is a schematic diagram of an ASIC in an embodiment of the present invention.

[0076] Fig.45 The layout of blocks used to provide more inputs and outputs in one embodiment of the present invention is shown.

[0077] Fig.46 The internal structure of the core processing block in the embodiment of the present invention is shown.

[0078] Fig.47 The internal structure of the convolution unit in an embodiment of the present invention is shown.

[0079] Fig.48 A single summing unit of a core processing block in an embodiment of the present invention is shown.

[0080] Fig.49 The internal structure of the auxiliary function block in the embodiment of the present invention is shown.

[0081] Fig.50 The elements required to implement any block function contained in an auxiliary block in an embodiment of the present invention are shown.

[0082] Fig.51The connection layout with external circuits in an embodiment of the present invention is shown.

[0083] Fig.52 Schematic diagram showing the layout of four 3×3 kernel processing blocks used to compute 5×5 convolutions in an embodiment of the present invention.

[0084] Fig.53 1 shows a layout of nine 3×3 kernel processing blocks arranged in a manner to implement a 9×9 kernel in an embodiment of the present invention.

[0085] Fig.54 An abstract example of a fully functional convolutional neural network in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0086] Various image and data algorithms widely use the matrix form of linear algebra to prove propositions and calculate results. In this application, "algorithm" refers to a process or set of rules to be followed in computing or other problem-solving operations, especially in computing or problem-solving operations by computers. In this application, algorithms should not be generally interpreted as software. The algorithms described in this application can and are usually preferably implemented in hardware.

[0087] Matrix operations are defined as orthogonal sets of one or more dimensions, usually assuming that each iteration of each given dimension has the same number of elements. For example, an M×N matrix is ​​usually described by an array of values, such as:

[0088]

[0089] Conceptually, a matrix can have any number of dimensions, and a matrix can be depicted as a set of tables showing the values ​​for each dimension.

[0090] A subset of matrices in the form of M×1 or 1×N can be called vectors. They have their own defined specific properties and operations and are widely used in 2D and 3D graphics simulations.

[0091] A degenerate subset of matrices of the 1×1 form may be referred to as scalars, and constitute numbers that are very familiar to those skilled in the art.

[0092] When the values ​​of the matrices are constants, and the matrices have compatible dimensions, certain operations, such as multiplication, are well-defined. A 3×4 matrix A can be multiplied with a 4×5 matrix B to form a 3×5 matrix C, which can generally be written as:

[0093] A×B=C

[0094]

[0095] The operation B × A is ill-defined because the inner dimensions do not match (5 ≠ 3), and k does not have a single range that is compatible with the indices of B and A.

[0096] When the elements of a matrix are vectors or other matrices, the matrix is ​​called a tensor (hence the name TensorFlow). A common form of a tensor might be an RGB image. One form of an RGB image is an HDMI frame, which is a 1080×1920 matrix of RGB values, where each pixel is a 3×1 vector of color components. Pixels are treated as true vectors because linear operations on the red component do not affect the green or blue, and vice versa.

[0097] HDMI frames are not usually viewed as 5D matrices, because the manipulation of pixel locations in the image is not linked to the manipulation of color. Cropping an image by cutting out parts of it that are not of interest is efficient and makes perfect sense, but there is no corresponding operation for cropping color components. Likewise, many operations on color may produce easily understandable effects, but would make no sense if applied to the elements of an array containing them. Thus, HDMI frames are clearly 2,3 tensors, not 5D arrays.

[0098] It is known that many image processing algorithms can be expressed as matrix operations. Matrix operations are a concise way to express repeated operations, and the rules of matrix mathematics help to prove certain propositions.

[0099] Execution of matrix-based algorithms on general-purpose computer processors is usually done via loops, and both the computer language and the hardware CPU may have features that make such loops efficient. However, the mathematics defined by matrices does not itself require that operations be performed by any particular method or plan in order to compute correct results.

[0100] Convolutional neural networks (CNNs) are a modern hybrid of image processing and cognition. Although training such networks has been challenging for many years, actually executing a trained network is relatively simple.

[0101] In a CNN, each convolution output element produces each component of the output tensor by moving a separate kernel over the input tensor. Typically, when a neural network is used to process an image, the first layer of the network operates on the image's RGB pixel input array and produces an output array of related size, containing arbitrary output component vectors that are structurally independent of the RGB vectors of the input components. The output vector components are often described as features or activations, representing the strength of the response (degree of recognition) of each kernel. Subsequent layers of a CNN take the output of the previous layers as their input, so only the first layer works on pixel values, and the remaining layers all work on features to produce more features. Each output feature of the convolution is uncorrelated and distinct from the other features, just as color components are distinct from each other.

[0102] A common form of CNN layer is a 3×3 convolution. In the operation, a 3×3 constant weight kernel is applied to each specific position of the input tensor (i.e., image) element; that is, each weight is multiplied by the pixel component at the same relative position in the image, and the products are added to produce a single output component at that position. The bias constant (which may be zero) provides an initial value to facilitate solving the model to find the optimal weight value.

[0103] If there are three input components (as in an RGB image), then there are three different sets of 3×3 weights applied to each component value (color in the case of the first layer), but there is only one initial bias. Each convolution of the 3×3×3 weights plus the bias results in an output component value that corresponds to the location of the center of a 3×3 pixel block. Each output channel in turn has its own 27 weight values ​​applied until all output components for a given block are calculated (the subset of input components that are in the same position as the output location and correspond to the relative position of the kernel weights). Convolutions typically have between 64 and 256 output components, each with a unique specific set of 27 weights plus the bias.

[0104] In this example, each kernel multiplies its 27 weights with the same block of 9 pixels for the 3 RGB components. Each individual input component is multiplied by 64 arbitrary and unrelated weights for a relatively small set of 64 output components. After each block's output components are calculated, the adjacent block is loaded from the image and the kernel's full set of weights is applied again. This process continues until the right edge of the image is reached, then the blocks are moved down one row and start again from the left edge.

[0105] After the first layer is processed, the next convolutional layer processes the output of the first layer as the input of the second layer. Therefore, the 3×3 convolution now has 3×3×64 weights applied to the 3×3×64 input components of the block. If this layer has 256 outputs, 3×3×64×256=147,456 multiplications must be performed for each output position. Those skilled in the art will understand that this refers to a single layer in a deep neural network that may contain more than 40 layers.

[0106] The number of multiplications applied to each element of the block is equal to the number of channels in the layer. In a standard CPU, these multiplications must be done in some order. Many modern CPUs have the ability to perform multiple sets of multiplications simultaneously, especially if the data format is small (i.e. 8 bits). In a GPU or TPU, the number of multipliers available is much higher, but each multiplier is designed to produce a product from two different and unrestricted factors.

[0107] In prior art processors, CPUs, TPUs or GPUs, no exploitation is made of the simple fact that in CNN implementations, one of the factors used for the multiplication is common to all weights applied to the input channels during a processing block.

[0108] The inventors of this application propose a batch multiplier that can complete all multiplication operations in one step, while traditional multiplication operations are performed sequentially. When a group of multiplication weights are all small precision (usually 8 bits for TPU), there will be a finite (2 8 = 256) different weights, and a corresponding number of different common input multiples (which can be of any size; no matter what the precision of the common factor, there are still only 256 possible multiples when 8-bit weights are applied). In this case, there is a clear advantage in using far fewer components than the same number of unrestricted multipliers to implement a circuit that produces all the desired outputs at once.

[0109] In an embodiment of the present invention, the equivalent bulk multiplier is dedicated to a single input channel and is not always shared. Therefore, the operation can choose to use multiple clock cycles and multiple register stages. In this way, the operation form is very simple and efficient without affecting the overall throughput of the system.

[0110] In the common case of a single dynamic value multiplied by multiple constants, replacing an equivalent set of independent single-stage multiplier circuits with a single multi-stage bulk multiplier circuit in an embodiment of the present invention enables the system to perform the same calculation with higher throughput, lower power consumption and footprint. Even if the output set is smaller than the multiple actually used, significant power and space savings can still be achieved.

[0111] In the embodiments of the present invention, the unique batch multiplier has obvious advantages over independent multipliers, and rearranging the operation order can further enhance the advantages.

[0112] The mathematics of neural network (or other similar image processing) algorithms do not require any particular order of operations. If the same operations are performed in any order, the same correct calculation results will be obtained.

[0113] The inventors have observed that the usual order of software executed on a CPU, GPU, or TPU-based design is to produce all output channels at a given position simultaneously by multiplying the weights by the inputs and summing them immediately. Producing all output channels at a given position simultaneously by multiplying the weights by the inputs and summing them immediately minimizes the number of times the inputs must be read from RAM and limits the number of times the weights must be read from RAM. It does not eliminate the need to read the inputs multiple times because there is no place to keep the inputs except RAM while processing the next row.

[0114] However, in an embodiment of the present invention, if the order of operations of a kernel or other aperture function defined to operate on an M×N array of input blocks is reversed, i.e., it is actually turned inside out, each input value is used only once and no RAM buffer is required. Rather than generating outputs one at a time by redundantly reading the inputs as the aperture function traverses each row, this unique operation processes inputs one at a time only as they initially appear, and retains partial sums of all incomplete outputs. The partial sums can be retained in hardware shift registers or standard hardware first-in-first-out registers (FIFOs), and the number of registers required to hold the retained values ​​is proportional to the height of the kernel and the width of the input row.

[0115] Since the function implementing the aperture function can be decomposed into a series of subfunctions, each operating on the results of the immediately preceding subfunction, the implementation of the kernel can be realized by combining the subfunctions sequentially over time, so that each subfunction operates on the received data at once and produces the same sequence of operations as applying the kernel abstractly. We call this recombined function (including any initialization) the aperture function, and the individual steps subfunctions. As used in this paper, an aperture function refers to any M×N computation implemented at multiple locations of a sliding window or block of M×N inputs of a larger R×C input array. As with the implementation of a full CNN kernel, the aperture function can also include initialization and finalization operations. In the case of a CNN, initialization preloads bias values ​​into the accumulator, and finalization transforms the raw output of the kernel through an arbitrary activation function.

[0116] In this example of the invention, as the component of each new input position appears, the component of that position represents the first element of the block to the bottom and right, the last element of the block to the top and left, and the middle elements of all other blocks that intersect the current position. This allows the development of a computational circuit as an embodiment of the invention that always has a fixed number of elements in process (with some possible exceptions near the edges of the inputs) and produces output at the same rate as it accepts input.

[0117] There are a number of special cases and problems that arise when the guided algorithm needs to evaluate the aperture function on blocks that extend beyond the edge of the input array, but they are not insurmountable. Special case logic can be added to make partial results for overlapping blocks compatible with the normal case without affecting overall throughput.

[0118] In an embodiment of the present invention, this flipped aperture function operation accepts input in the form of a stream and produces output in the form of a stream. The input does not need to be buffered in RAM because each input is only referenced once. Since the output is also streamed, subsequent layers can also process it without RAM buffering, because the present invention significantly improves the processing speed and reduces many other necessary RAM read and write operations.

[0119] In an embodiment of the present invention, instead of many layers sharing a set of independent multipliers to execute, store, and then read back the results to process the next layer in sequence, dedicated batch multipliers are used to produce a pipeline that processes all layers simultaneously, streaming the output of each layer into the input of the next layer without waiting for any layer to complete.

[0120] Therefore, the fully implemented pipeline in an embodiment of the present invention can achieve an effective throughput two orders of magnitude higher than the traditional output-centric sorting process, and eliminates the contention for RAM (because it does not use RAM). It is this contention for RAM that forms the main bottleneck of GPU- and TPU-based processing.

[0121] In an embodiment of the present invention, the latency of such a system is reduced to the time from the last pixel input to the last result output. Since the last pixel of the image must be the last data required to complete the final calculation of all layers according to the definition of the algorithm, the latency of the system is strictly the clock frequency multiplied by the number of different clock stages in the pipeline including the final output.

[0122] In an embodiment of the present invention, by using a single dedicated batch multiplier for each input channel throughout the neural network (instead of a finite set of independent multipliers that must be reused and dynamically allocated), a pixel-synchronous pipeline can be constructed in which all multiplications are performed in parallel because only one batch multiplier is required to handle any number of weights applied.

[0123] After describing the basic features of the batch multiplier innovation and the advantages of flipping, the inventors propose the following specific examples:

[0124] Figure 1 1 is a schematic diagram illustrating an embodiment of the present invention, wherein each of a plurality of one or more source channels 1 to N, labeled 101a to 101d, is assigned a dedicated batch multiplier 102a to 102d. Since each source channel in this example has a dedicated batch multiplier circuit to create a set of multiples of the channel value, the source channel format can be changed between signed, unsigned, fixed or floating point, and the precision can be arbitrarily selected to facilitate the implementation of the processing algorithm in hardware. The specific output of each batch multiplier circuit (e.g., batch multiplier circuit 102c) can be directly fed to one or more computing units 103a to 103d, which can perform calculations requiring multiples of any or all source channels. Such computing units can be used to implement independent output channels of a single algorithm or unrelated algorithms to be calculated on the same source channel. The output of the calculation can be forwarded for further processing (as shown in 104), which may be the need of one or more algorithms implemented in hardware. This occurs, for example, when implementing a neural network in a field-programmable gate array (FPGA), where the values ​​of the weights applied as multipliers do not change.

[0125] Figure 2 An embodiment of the present invention is shown, wherein each batch multiplier (such as Figure 1 The output of the batch multiplier 102a in is fed to the calculation units 203a to 203d through the multiplexer group 201a to 201d, so that the selected multiples can be selected at system initialization or dynamically selected when the system is running. The output of the calculation can then be forwarded to 204 for further processing as before. This happens when a neural network is implemented in an application-specific integrated circuit (ASIC), where the structure of the calculation is determined, but the weight values ​​used need to be changeable.

[0126] Figure 3 An embodiment is shown Figure 1 and Figure 2The internal structure of the batch multiplier 102a of the present invention. This structure may be common to batch multipliers 102b, 102c and 102d and other batch multipliers in other embodiments of the present invention. In this structure, multiple products 303a to 303f are generated in parallel by multiplying all possible B-bit multipliers with the source channel multiplicand 101a of the A bit in parallel, and are transmitted to the multiple 304. In this example, the A bit of the source channel multiplicand 101a is copied, and it is shifted upward by adding a 0 bit in the least significant position, and a 0 bit is added in the most significant position for filling, so that a complete set of all required shift values ​​from 0 to B-1 is provided in the form of a vector of A+B bit items 302a to 302d. These items can be simply formed by routing circuit connections without registers or logic circuits. If the clock cycle is sufficient to allow a maximum of B A+B bit items to be combined in a single cycle, registers or sub-combinations may not be required. The single product 303a to 303f of the added item can be locally registered or forwarded to be further processed as a combinational logic. Each product from 1 to the Bth power of 2 minus 1 multiplied by the source multiplicand 101a can be formed by adding any or all of the B corresponding terms 302a to 302d (as long as a 1 bit appears in each multiplier). Any and all source multiples of 0 are constants of all 0 bits and can be included in the multiples 304 for completeness when multiplexers are used, but no circuitry is required otherwise. Any unused products 303a to 303f can be omitted by excluding them from the circuit specification, allowing the synthesis tool to delete them, or by any other method. Unused terms 302a to 302d can also be omitted, but since they do not occupy logic, they generally have no effect. In this way, all required multiples 304 of the source multiplicand 101 can be formed as a single stage of pipeline or combinational logic.

[0127] Figure 4An optimized embodiment is shown in which the item set 401 consists of all the required individual items 302a to 302e from 0 to B (including B), consisting of A+B+1 bits. This allows the products 402a to 402f to replace the addition of smaller items with the subtraction of larger items, and can be used to reduce the overall size of the circuit, thereby increasing the maximum clock frequency allowed. For example, for any given input a and multiplier 15, 8a+4a+2a+1a=15a combines four components, while 16a-1a=15a only combines two components, and is therefore generally more compact and efficient. Each product 402a to 402f can be composed of any addition and subtraction of items 302a to 302e that produces the correct result, and each specific variant can be selected based on the best trade-offs for a specific implementation technology. For example, the subtraction of two N-bit numbers may require more logic than the addition of two N-bit numbers, but in general, the addition of three N-bit numbers always requires more logic than the subtraction of two. The processing of the required multiple 304 does not change depending on the details of forming the individual products 402a through 402f.

[0128] Figure 5A An embodiment of a batch multiplier is shown in which the clock cycle is such that only one addition of the A+B bit value (or A+B+1 if subtraction is used) can be performed per cycle. In this case, in order to accommodate the use of multiples of more than two terms, the required elements must be arranged into a multi-stage pipeline. Term 401 is formed by each source channel 101, as previously described, but is retained once or multiple times in pipeline registers 501a and 501b for later reference. A pair 502 of two terms added is calculated and registered, and then saved 503 as needed. A triplet 504 is formed by the sum of the pair 502 and the retained term 501. A quadruple 505 of term values ​​is formed by the sum of the pair 502. Any and all unused elements can be omitted, and to increase overlap, only a descending sequence of addends can be specified. This ensures that redundant sums (e.g., a+b and b+a) are not both utilized and retained in the final circuit. Products 506a to 506f can utilize any addition or subtraction operation on any pair of registered subcombinations that meet the timing constraints. The overall size, and thus power, can be reduced by consistently using the largest element available, but any combination of operations that produces the correct result is acceptable.

[0129] Figure 5AThe embodiment of is sufficient to generate all desired multiples when B = 8. For larger sets of multiples, the subcombinations shown can be recombined in further pipeline stages so that, for any value of B, all desired multiples 506a to 506f can be composed by single clock operations on the expanded set of subcombinations, including the previously disclosed reserved items 501b, reserved pairs 503, triplets 504, and quadruplets 505, as well as other subcombinations as needed to form a set of items sufficient to form multiples 506a to 506f by single clock operations.

[0130] Figure 5B An embodiment is shown in which multiples are formed directly from a fixed set of cases without reference to standard arithmetic operations. For each desired multiple, a set of output values ​​a*b is enumerated for each source channel value a. This allows the hardware circuit synthesis tool to determine the optimal logic circuit 507 to produce a complete set of desired multiples. For any given input value, the specific specification of the desired output value is typically specified in Verilog by enumeration using a "case" or "casex" statement. This is different from a lookup table, in which the output values ​​are stored and accessed by an index formed by the input values. Because logic gates are used to implement the minimum subset of operations required to produce a complete set of output values, and redundant logic for generating related sub-expressions is merged.

[0131] Which of the two methods, 5A and 5B, is most efficient in terms of space, frequency, and power depends on the specific values ​​of A and B and the core efficiency of arithmetic operations versus arbitrary logic operations. The choice of which method to use may be based on direct observation, simulation, or other criteria.

[0132] Figure 6 An embodiment is shown in which the clock cycle has enough logic levels to form four elements by addition and / or subtraction in each clock cycle. Each product 605a to 605f can be generated by combining no more than four registered elements by selecting from the subset of combinations. As before, the items are retained in registers 501a and 501b, but the triple 601 retained in 602 is composed directly from the item 401 and no pairs are used. Septets 603 and octets 604 are formed from triples 601 and the retained item 501a.

[0133] Figure 6The exemplary embodiment of is sufficient to produce all required multiples for B = 32. For larger multipliers, the subcombinations shown can be recombined four at a time in further pipeline stages to produce all required multiples for any value of B. For the case of B = 32, the subcombination elements shown are necessary and sufficient to produce all products, but other subcombinations are also acceptable (perhaps chosen for consistency across different values ​​of B).

[0134] When the set of multipliers is fixed (as is common in FPGA applications), even large and sparse sets of multipliers can be implemented efficiently because common elements are merged and unused elements are omitted. When synthesis tools perform this function automatically, the representation of the circuit may contain all possible elements without explicitly stating which multiples are used.

[0135] If the operation on the A+B or A+B+1 bit value cannot be completed in a single clock cycle, a multi-stage pipeline adder can be inserted for any single stage of combinational logic, but additional pipeline registers must be inserted to make the number of clock cycles the same for all paths. The pipeline stage cycle can be an instance of a single edge-to-edge clock transition or an instance of a multi-cycle clock if throughput constraints permit. Other than the issues mentioned above, multiple clock stages per operation and the use of multi-cycle clocks do not require architectural changes to any embodiment.

[0136] An important object of the present invention is to provide the industry with a batch multiplier implemented in an integrated circuit for use in a variety of applications. To this end, the inventors provide, in one embodiment, a batch multiplier implemented in an integrated circuit having a port for receiving a stream of discrete values, circuitry for simultaneously multiplying each value received at the port by a plurality of weight values, and an output channel for providing the resulting product of the batch multiplier.

[0137] In one version, the received discrete values ​​may be fixed-width unsigned binary values, the weight values ​​may be fixed-width unsigned binary values ​​of two or more bits, and each multiple may consist of the sum of shifted copies of the inputs. In another version, the set of shifted copies may be increased to allow the use of subtraction operations to reduce or otherwise optimize the circuit. Unused outputs in the set may be omitted explicitly or implicitly.

[0138] In one embodiment, the output product set may be generated by combinatorial logic. In another embodiment, the output product set may be generated by a single stage pipeline using one or more clock cycles. In another embodiment, the output multiple set may be generated by a multi-stage pipeline by combining no more than two addends per stage. Unused elements in intermediate sub-combinations may be explicitly or implicitly eliminated from the circuit.

[0139] In one embodiment, an output product set may be produced by a multi-stage pipeline by combining three or more addends per stage, and the sub-combinations may be adjusted accordingly. Unused elements in intermediate sub-combinations may be explicitly or implicitly eliminated from the circuit.

[0140] Another object of the present invention is to provide batch multiplication in an integrated circuit to enable significantly improved convolutional neural networks in the continued development of deep learning and artificial intelligence. The inventors in this work provide a first convolutional neural network (CNN) node implemented as an integrated circuit, having a first input channel defined as a stream of discrete values ​​of a first component of an array element.

[0141] In this specification, the inventor intends to refer to the term array element as an element that can have a single component or multiple components. A good example is an image, which can have pixels as elements, and if the image is monochrome, each pixel can have a single component, or in one example, if the image is RGB color, each pixel can have three color values. Each color value in this example is a component of an element, i.e., a pixel.

[0142] Continuing with the above description, a first convolutional neural network (CNN) node is implemented as an integrated circuit, wherein a first input channel is defined as a discrete value stream of a first component of an array element, and a first batch multiplier circuit is further present in the CNN for multiplying the received discrete value of the first component with a plurality of weight values ​​simultaneously. The output channel provides an output stream of discrete values.

[0143] In one embodiment of the CNN node, a first output stream is formed from products of a first batch of multiplier circuits by, in some cases, combining the products with a constant and, in some cases, by applying an activation function.

[0144] In another embodiment, the CNN node further includes a second input channel, the second input channel being defined as a stream of discrete values ​​of a second component of the array elements, and a second batch multiplier circuit, the second batch multiplier circuit multiplies the received discrete values ​​of the second component with a plurality of weight values ​​simultaneously. In another embodiment, there may be a third input channel, the third input channel being defined as a stream of discrete values ​​of a third component of the array elements, and a third batch multiplier circuit, the third batch multiplier circuit multiplies the received discrete values ​​of the third component with a plurality of weight values ​​simultaneously.

[0145] Having described a CNN node with one, two or three input component streams and a dedicated batch multiplier, the inventors also provide a convolutional neural network (CNN) having a first convolutional neural network (CNN) node implemented as an integrated circuit, including an input channel of a discrete value stream of components defined as array elements, a batch multiplier circuit dedicated to each input channel for multiplying the received discrete values ​​of the components with multiple weight values ​​simultaneously, and an output channel providing an output stream of discrete values, and a second CNN node whose input depends at least in part on the output of the first node. This CNN can have consecutive nodes and can operate as a deep neural network (DNN). There is no requirement that consecutive nodes after the first node must be CNN nodes.

[0146] Pipeline Aperture Function Operation

[0147] Returning now to the previous description in this specification, discussing the order of operations for processing a CNN or other similarly selected aperture function that passes an array of computational sub-functions over an input array to produce a net result, a specific description of the flipped form of the aperture function operation in an embodiment of the present invention is now provided, which accepts inputs as a stream and produces outputs as a stream. In this embodiment of the present invention, the inputs are not and do not need to be buffered in RAM because each input is referenced only once. The outputs are also produced in a stream, so the output stream can be processed by subsequent layers without RAM buffering. The inventor believes that this innovation significantly increases processing speed compared to the many necessary read and write operations to RAM in other processing systems.

[0148] In an embodiment of the invention, an apparatus and method are provided in which operations are implemented by applying a two-dimensional aperture function to a two-dimensional array, the operations being implemented by acting on an incoming stream of inputs such that all inputs are processed immediately and partially completed computations are retained until all required inputs have been received and processed, and outputs are produced in a stream-compliant manner, typically at the same or lower data rate than the input stream. All inputs are accepted and processed at the rate provided and need not be stored or accessed in any order other than the order presented. If the application of the aperture function is defined to produce more outputs than inputs, the circuitry can still run at the incoming data rate by selecting a sufficiently increased processing clock rate so that the system is not unable to accept and process inputs as they are presented.

[0149] The traditional way to implement a kernel, or more generally an aperture function, for convolution with a larger input array is to collect the desired chunks of input, apply the function to the input, and output the result. As the aperture passes through the input array, each subsequent chunk will overlap with the chunk just processed, so some of the input can be retained and reused. Various mechanisms (such as FIFOs) can be used to avoid redundantly reading inputs from source storage as chunks move to each new row, but the source data is still applied to each location in the kernel in turn to produce each output whose chunk of input overlaps with each particular data input location.

[0150] If you want to compute multiple output channels and multiple independent aperture functions, you can use batch multipliers to feed the products of blocks of considered input values ​​to all aperture functions in parallel. However, with this layout and order of operations, each position of the source data requires a set of products for each position in the kernel as it is combined into overlapping individual output positions.

[0151] The mechanism of the present invention is to flip (i.e., reverse the order of operations) to take advantage of the particular use of a single batch multiplier per input channel to apply a given input value only once. Rather than retaining or re-reading source values ​​for later use in computing subsequent products, the process in an embodiment of the present invention computes all necessary products as each input is computed, and retains a running total for each element of the aperture function that is complete when the current input appears.

[0152] Any aperture function that can be mathematically decomposed into a series of sub-functions applied sequentially can be implemented in this way. Since CNN kernels are just an additive sequence of products of weights times inputs, and the order of operations is compatible with the order in which source inputs are taken from left to right and top to bottom, this mechanism can be easily applied.

[0153] In an embodiment of the present invention, an array of synthesizers is implemented on the IC, corresponding to sub-functional elements of the aperture function, each synthesizer maintaining a running sum of the value of the aperture function as it progresses on the input stream. The last synthesizer in the array outputs the complete value of the function, while all other synthesizers output partial values ​​of the function.

[0154] In the simple case of applying a 3×3 kernel, the output of the top-left combiner reflects the first element of the kernel applied to the current input plus any initialization constants, the output of the top-middle combiner reflects the first two steps, and the output of the top-right combiner reflects the first three steps. The output of the top-right combiner is delayed until it can be used again by the next row. The combiners in the next row continue to accept partially completed function values, add the contribution of each new input, and pass it on. The last row of combiners completes the final steps of the function and outputs the completed value for further processing.

[0155] Note that the transfer of partial values ​​of functions between compositors is typically from left to right in the first row, then from left to right in subsequent rows, and finally to the last compositor in the last row. The flow of partial values ​​can be thought of as a flow, and compositors and flows are referred to as upstream or downstream.

[0156] At any time, each synthesizer maintains a partial sum of the aperture functions, up to and including the current source input. Each synthesizer is always working on a different block position of the output, specifically the block where the current input appears at the synthesizer's relative position in the array of aperture sub-functions.

[0157] If we express the 3×3 kernel W as a function of input A,

[0158]

[0159] u=k+a 11 w 11 +a 12 w 12 +a 13 w 13 +a 21 w 21 +a 22 w 22 +a 23 w 23 +a 31 w 31 +a 32 w 32 +a 33 w 33

[0160] Then the function that implements the kernel can be decomposed into equivalent sub-functions.

[0161] v0(a)=k+aw 11

[0162] v1(t,a)=t+aw 12

[0163] ν2(t,a)=t+aw 13

[0164] ν3(t,a)=t+aw 21

[0165] ν4(t,a)=t+aw 22

[0166] v5(t,a)=t+aw 23

[0167] v6(t,a)=t+aw 31

[0168] v7(t,a)=t+aw 32

[0169] v8(t,a)=t+aw 33

[0170] u=v8(v7(v6(v5(v4(v3(v2(v1(v0(a 11 ), a 12 ), a 13 ), a 21 ), a 22 ), a 23 ), a 31 ), a 32 ), a 33 )

[0171] u=((((((((k+a 11 w11)+a 12 w 12 )+a 13 w 13 )+a 21 w 21 )+a 22 w 22 )+a 23 w 23 )+a 31 w 31 )+a 32 w 32 )+a 33 w 33

[0172] u=k+a 11 w 11 +a 12 w 12 +a 13 w 13 +a 21 w 21 +a 22 w 22 +a 23 w 23 +a 31 w 31 +a 32 w 32 +a 33 w 33 =u(A,W)

[0173] The circuitry required to compute these subfunctions is then laid out in the arrays of the corresponding synthesizers.

[0174]

[0175] and the partially completed sum is retained as the output value of the synthesizer

[0176]

[0177] Among them, a i is the current value from the input stream, and a i-1 to a i-8 are the previously processed inputs of a particular block, where a i appears in a position relative to the output of each individual synthesizer. Each synthesizer will compute values ​​of the aperture function up to and including the position corresponding to that synthesizer in the aperture array. Each synthesizer takes the current value of the input stream and combines it with the previous value to produce a different partial sum corresponding to a partially processed block in the input array, where the current input value appears in the relative position of that block corresponding to each synthesizer's position in the aperture function.

[0178] In this way, partial values ​​of the aperture function computed in standard order and precision are maintained in the input stream over time until the completed value is ready for output.

[0179] While this technique is very simple inside the input array, complications arise when applied to blocks that overlap the edge of the input array, because the aperture function is defined differently when all inputs are not available. In the case of a CNN kernel, additional operations are discarded, which is equivalent to using zero as input. The present invention is concerned with maintaining a steady flow of partial sums through the synthesizer while handling these exceptional cases, as described below.

[0180] Figure 7 is a schematic diagram of the structure and connections of an embodiment of the present invention, which receives an input stream, pre-processes the input stream, and feeds the result through a unique digital device to produce an output stream.

[0181] Common circuit 703 uses input channel set 701 and related control signal 702 to generate any and all products of input channel set and weights for subsequent sub-functions. The source channel products are then distributed to a set of sub-function calculation circuits 704a, 704b and 704c, each of which generates a single channel of output channel set 705. Common circuit 703 can support any number of independent output channels.

[0182] Fig. 8A yes Figure 7Schematic diagram of batch multipliers 801a, 801b and 801c in common circuit 703 of the embodiment of the present invention, which receive each channel of the input channel set 701 and generate a sparse or complete set of multiples as required by the defined sub-function. It should be noted that this illustration assumes three channels in the input channel set, such as the case when processing red, green and blue pixel values ​​in an RGB image. In other embodiments, there may be one, two or more than three channels. Any or all products 802 (multiples of the source input array values ​​constructed by the batch multipliers) can be used in the synthesizer, such as Fig. 9A , 9B 9C, which will be described in detail below. The synthesizer is an example of a hard-wired circuit in the unique device of the present invention, which is Fig. 8A The batch multiplier generates the source channel product execution subfunction.

[0183] Figure 8B is a schematic diagram of the structure of a synchronization circuit, which provides normal and abnormal processing signals for all synthesizers of all output channels.

[0184] The control circuit 803 synchronizes all output and control counters to the source input stream and sets the output and control counters to the initial state when RST or INIT is asserted.

[0185] In this example, the column source (colSrc) counter 805 calculates the internal dimensions of the array column by column in the row and advances as each set of source channel products is processed. At the end of each row, the colSrc counter returns to the leftmost position (0) in this example, and the row source (rowSrc) counter 804 advances one position. At the end of the source array stream, the rowSrc and colSrc counters return to their initial states and are ready to receive new input arrays.

[0186] In this example, the column destination (colDst) counter 807 and the row destination (rowDst) counter 806 cooperate to function in a similar manner as the counters for all output channels. The colDst and rowDst counters are enabled by the output enable signal (DSTEN) 813 and determine when the post-processing enable signal (POSTEN) 812 is asserted.

[0187] It is worth noting that the system described in this example produces a single aperture function output, but is generally used to produce a set of channel output streams that are compatible with the size of the source input stream. Each independent output channel will share at least some computational circuitry through bulk multipliers and common control logic.

[0188] Output Enable (DSTEN) signal 813 controls when the finalizer function accepts and processes results from the synthesizer. Although the first few rows are accepted from the source input array, no valid results are presented to the finalizer function (see Fig. 9C ). When the rowDst and colDst counters indicate that a valid result is available, or when a delayed truncated result is being processed, the output enable signal 813 (DSTEN) is asserted. The POSTEN signal 812 is asserted continuously or periodically to match the timing of the SRCEN signal 801. These signals are needed to sequence the final output of all truncated synthesizers when processing the last row of the source input stream array. Each row synthesizer to M-2 will produce the final truncated output concurrently with the last complete output, which in turn must be preserved and output sequentially after all complete block outputs to conform to the array stream format.

[0189] In this example, the values ​​of the POSTEN and DSTEN signals and the colDst and rowDst counters are independent of the values ​​of the SRCEN signal and the colSrc and rowSrc counters, and processing of delayed results continues until all delayed results are finalized and sent to the output stream. The system can accept new inputs while previous outputs are completing, allowing the system to process multiple frames of the source input stream without pausing between frames. When the source stream data has not yet reached the end of the array, POSTEN is not asserted and the final result is obtained from the synthesizer. Immediately after reaching the end of the source array, the POSTEN signal is asserted for each additional output, and the final result is obtained from the truncated delay lines 909, 910a, and 910b, as described below Fig. 9C As shown, the counter in rowDst is incremented until the rowDst counter reaches the full number of output rows, and then rowDst and colDst are reset to their initial conditions in preparation for the next data frame.

[0190] When the rowSrc counter indicates that the source data set from the stream represents the first row of the array, the first row signal 808 (ROWFST) is asserted.

[0191] When the rowSrc counter indicates that the source data set from the stream represents the last row of the array, the last row signal 809 (ROWLST) is asserted.

[0192] When the colSrc counter indicates that the source data set from the stream represents the first column of each row of the array, the first column signal 810 (COLFST) is asserted.

[0193] When the colSrc counter indicates that the source data set from the stream represents the last column of each row of the array, the last column signal 811 (COLLST) is asserted.

[0194] Fig. 9A , Fig. 9B and Fig. 9C The unique arrangement described above is shown in the general case where M×N sub-function elements of an aperture function are applied to each overlapping M×N block of an array of R×C inputs (including those overlapping edges) presented as a stream of related components at regular or irregular time intervals to produce a corresponding stream of R×C outputs, where each output is the aggregate effect of the M×N function elements applied to the input block as specified by the rules of the aperture function. In this arrangement, the function element applied to each position in the array is a hard-wired synthesizer of each M×N sub-function, such as Fig. 9A , Fig. 9B and Fig. 9C As shown in the composite diagram.

[0195] The circuit works by computing the reassembled value of the aperture function at each position in the R×C input array, using the same sequence of operations as would be used to compute the aperture function for each block individually. If certain positions are not needed in the output stream, circuitry can be added to ignore those positions, thereby producing a tiled or spaced output, rather than a fully overlapping output.

[0196] Source channel products 802 and source control signals 814 are available to each synthesizer 901, 902a, 902b, 902c, 903a, 903b, 903c, 904, 905a, 905b, 905c, 906, 907a, 907b, and 907c. The source control signals are also connected to delays 908a, 908b, 908c, 908d, 908e, and 908f. Output channel control and counter 815 are provided to delays 909, 910a, and 910b and finalization function 911. Additional pipeline stages can be inserted by manual or automated tools to make the circuit routing feasible at a given clock frequency if and only if the order of operations does not change. Timing control and counter signals are available to all elements of the circuit and are not shown separately.

[0197] Each synthesizer has a dedicated direct connection to a specific input product or a selective programmable multiplexer that selects one of the products for each input value and is preconfigured before the circuit is executed. Each dedicated connection is a parallel path with multiple lines sufficient to carry the bits representing the desired product in a single input interval. The weight values ​​can be upgraded in the field using an optional preconfigured multiplexer to select which product of each set element is sent to each synthesizer. A fixed connection is used when the weights are not upgraded and this connection remains unchanged for the life of the device. Since the choice of weights does not change during operation, the choice of fixed or variable product selection does not affect the operation of the circuit.

[0198] Each synthesizer receives a set of products corresponding to the sub-function weights from the batch multiplier (one for each input channel) and performs the sub-function calculation, typically simply adding them all together to form this synthesizer's contribution to the overall aperture function value. Each synthesizer (except the synthesizer corresponding to the left column of the aperture function) also receives a partially completed result from the synthesizer immediately to the left. Each synthesizer (except the synthesizer corresponding to the top row of the aperture function) can also receive a delayed partially completed result from the synthesizer in the upper row. Each synthesizer has at most one connection from the left and one delayed connection from the top, each of which is a parallel channel with multiple conductors, the multiple conductors being sufficient to carry bits representing the partially completed result as input to the synthesizer. Depending on the definition of the sub-function (the position of the current input block relative to the edge of the input array), each synthesizer performs one of three operations: combines its partial result with the initialization value (if any), combines its partial result with the partial result of the synthesizer to the left, or combines its partial result with the partial result of the delay. The modified result is placed in an output register having a sufficient number of bits to contain the result and make it available to the right-hand synthesizer and / or delay and finalization circuits in subsequent input intervals. Depending on the synthesizer's position in the aperture function and the state of the input stream position, this modified result may be a partial result, a complete result, or a truncated result.

[0199] The synthesizer (0,0) is unique in that there is no synthesizer to the left or above the aperture function, and therefore is always initialized for each set of inputs received.

[0200] The synthesizer (M-1, N-1) is also unique in that the result it produces is always the final result, but is structurally identical to all other synthesizers 903a, 903b, or 903c.

[0201] The outputs of some synthesizers are used for delay or post-processing, in which case the path through such delay or post-processing is wide enough to transmit bits representing a partial, truncated, or complete result. The outputs of some synthesizers are used only by the synthesizer to the right. The computations within the synthesizer and the output data format do not need to change depending on the purpose of the output.

[0202] The finalization circuit takes results from multiple possible sources and multiplexes them to select which result to process at any interval. After applying the finalization function (if any), the width of the final output may be reduced and will form the output stream of this embodiment, which can be the next input stream, the final output of the system containing the present invention, or it can be used for further processing.

[0203] exist Fig. 9A , Fig. 9B and Fig. 9C In , the data paths on the unique device in the embodiment of the present invention are represented by bold lines, arrows indicate direction, and ellipsis indicates the position in the range where the last column or last row is completely repeated. The data path (a) from the source channel product 802 is a set of parallel conductive paths, one path dedicated to each product of the input components, each product being the value of the input component multiplied by one of multiple weight values ​​of the aperture function. Obviously, the 5×5 aperture function has 25 weight values ​​for each input component. For the aperture function of the R×C input array of R, G, and B color pixels, there are 75 weight values. Therefore, in this case, line (a) has 75 parallel paths, each of which is a set of parallel conductors whose width can accommodate the required number of bits to achieve accuracy. In the art, line (a) is referred to as a set of point-to-point connections, as distinguished from a bus.

[0204] Fig. 9A , Fig. 9B and Fig. 9C The data paths (b) in the circuit are not extensions of the circuit (a), but are dedicated paths that connect to a specific subset of the paths in the circuit (a). Fig. 9A , Fig. 9B and Fig. 9C Each instance in is marked, but each connection from line (a) directly to a single synthesizer is a dedicated line (b). Dedicated means that each synthesizer is connected to a subset of paths that transmit the product of each input component and weight value required by the synthesizer.

[0205] Fig. 9A , Fig. 9B and Fig. 9C The data paths (c) in the diagram are point-to-point paths between the output register of each synthesizer and the next synthesizer to the right. These are precision-width dedicated paths that typically carry partial sums, as described elsewhere in the specification. Not every path (c) is labeled in the diagram, but it can be assumed that every direct connection from one synthesizer to another in this example is a path (c). Note that there are cases where the output path (c) branches to alternative circuits.

[0206] Another different data path in the embodiment of the present invention is Fig. 9A , Fig. 9B and Fig. 9CThese are dedicated data paths from the delay circuits (e.g., circuits 908A through 908f) that can either be returned to the synthesizer on the left side of the next row or can be directly connected to other delay circuits. The delay circuits are designed to accept partial sums at the right end of the synthesizer row, delay delivery of the partial sums for a specific number of source intervals, and then pass those partial sums to another synthesizer and / or other processor at the appropriate time. The overall functionality is described in detail elsewhere in this specification. The paths (d) between the delay circuits are also similar dedicated paths for passing typical partial sums for a specific source interval.

[0207] If either M or N shrinks to the point where the last row or column of the range is not needed, the ending element is omitted and the implementation of the first row or column in the range is retained. In the degenerate case where either or both M or N shrinks to 2, the first row or last column is retained and the middle rows or columns are omitted. In the degenerate case where either M or N shrinks to 1, the implementations of the first and last synthesizers are combined and no special initialization is required. In the special case where both M and N are 1, there is no need to flip the aperture function, but there is still a clear advantage to using batched multipliers.

[0208] The source channel product 802 can be any set of binary values ​​that are associated with specific positions of the R×C array and are presented simultaneously in some predefined order. The source channels of the input stream can be any combination of integer or fractional values, and the format of these values ​​can be any format defined for the input of the aperture function. An example is pixel values ​​from one or more video frames and / or any other sensor values ​​scaled to match the array size R×C and feature component values ​​produced as outputs of the CNN layer. It is emphasized that each node embodying the present invention can accept outputs from other nodes in addition to (or in place of) the main source input. Although in embodiments of the present invention, the first node or multiple nodes typically accept image pixels as the main input to the system, there is no restriction on the nature of the data processed if the data can be formatted as a stream representing an R×C array.

[0209] In one embodiment of the invention, the source stream element set may be presented in row-first order, with each subsequent column presented in strictly ascending order. In some embodiments of the invention, the rows and columns need not correspond to the horizontal or vertical axes, but may be arbitrary, such as above or below the columns and scanning from right to left. Rows R and columns C refer to the major and minor axes of the stream format in this article only. No adjustment circuit is required for input signals that produce input streams in directions other than the standard video left-to-right, top-to-bottom order. The direction of the aperture subfunction may be adjusted to produce the same output at each input array position.

[0210] In this example, the source input is the product of the source value required by the aperture function and the weight, given by the signal (SRCEN see Figure 8B ) indicates when each new set of elements is valid. Inputs can be paused and resumed at any time. In some cases, a minimum spacing between inputs can be defined, and the circuit can use multi-cycle or higher speed clocks to reduce size, power, or otherwise gain advantage, and the output channel sets can use the same minimum spacing.

[0211] Common control and synchronization circuit 803 ( Figure 8B ) provides counters and control signals for describing the current input position in the R×C array. The counters can continue to run additional rows and columns after the final input to assist in finalizing function 911 ( Fig. 9C ) Outputs the cumulative output produced by the last row of input over the input column. (See Fig.12 , Fig.13 and Fig.14 and the description below) control signals are available to all other elements and are not Fig. 9A , Fig. 9B and Fig. 9C Shown in.

[0212] Synthesizer circuits 901, 902a, 902b, 902c, 903a, 903b, 903c, 904, 905a, 905b, 905c, 906, 907a, 907b and 907c each calculate the portion of the aperture function assigned to its position in the M×N function. All synthesizers operate on the same set of source channels and on row counter and column counter states, which are provided by controller 803. The data processing details of the aperture function will be further described below with reference to other figures.

[0213] As a source input set is received from an input stream, the portion of the aperture function applied to all blocks overlapping the current position in the input stream is computed and passed from left to right and top to bottom in the M×N array of synthesizers. This operation accumulates the full computation of the aperture function over time and outputs the correct implementation of the aperture function on each block of the input array, producing the same results through the same sequence of operations as if the aperture function were implemented by reading the input values ​​directly from the array. An important feature of the present invention is that random access to the array is replaced by streaming access, thereby eliminating the requirement for redundant access to random access memory.

[0214] In the right column of the synthesizer To N-1 (excluding the bottom row), the partial output is passed to delay stages 908a, 908b, 908c, 908d, 908e and 908f where it is retained for the required number of input intervals so that it can be used for further calculations at the same logical block position when input corresponding to a lower row of the block is received.

[0215] When processing the last column C-1 of each input line, All synthesizers to N-1 and rows 0 to M-2 also represent the last computation of that row of the block containing the last column of the input array, and their values ​​are forwarded to delay stages 908a, 908b, 908c, 908d, 908e and 908f and require special processing to be inserted into the sequence so that they are available at the correct time to continue computing the aperture function when subsequent input rows are received. Fig.11 and related descriptions.

[0216] In this example, the synthesizer 903c at position (M-1, N-1) always produces a complete accumulation of M×N subfunction elements, but otherwise is no different from the other synthesizers in this configuration 903c. As described above, when processing the last column C-1 of each input row, the number of columns on row M-1 from All synthesizers through N-1 also represent a completed but truncated accumulation of aperture function elements and are sent directly to the finalization function 911 for processing for insertion into the output stream.

[0217] In this example, when processing the last row R-1 of input, the The synthesizer to M-1 also represents the completed but truncated accumulation of the sub-function element calculations and is sent to the truncated output delay lines 909, 910a and 910b and retained to the main output of row M-1 completed at 911. Figure 8B The control signal shown cuts off the output of the additional The rows are transmitted from delay lines 909, 910a and 910b and completed at 911 and are ultimately provided to the output stream receiver 705 at any desired time interval.

[0218] Fig.15 It is a schematic diagram showing the specific situation of implementing the pipeline operation of 5×5 convolution nodes in an embodiment of the present invention.

[0219] Source channel products 802 and source control signals (not shown here) are available to each synthesizer 901, 902a, 902b, 903a, 903b, 904, 905a, 905b, 906, 907a and 907b. The source control signals are also connected to delays 908a, 908b, 908c and 908d. Output channel controls and counters are provided to delays 909, 910a and finalization 911. Additional pipeline stages can be inserted manually or by automated tools to adapt the circuit routing to a given clock frequency without changing the order of operations. Timing control and counter signals are available to all elements of the circuit and are not shown separately.

[0220] As each set of source channel products appears in turn, each synthesizer selects the appropriate products to compute for the subfunction corresponding to the position in the aperture function. Each 5×5 block that intersects the current position in the input array is modified to include computation of the products based on that position. The net effect is that a single source input stream is converted into a set of parallel 5×5 partial computation streams that are passed between synthesizers until all operations on each block are completed, which typically happens in synthesizer (4,4) and sometimes happens when processing the right or bottom edge of the input array.

[0221] Note that only the width of the input array affects the size of the delay elements, since each delay element must delay the partial result by the number of source input intervals corresponding to receiving input for one column and input for the same column in the next row.

[0222] Fig.16 A 4×4 embodiment of the IC of the present invention is shown. As is well known, a kernel can have an odd number of sub-functions in a row or column, or an even number of sub-functions. Since the extra output processing rows are omitted, Fig. 9C In general, the elements 910* and Fig.15 The element 910* in the special case of a 5×5 aperture function (with odd numbers of rows and columns) does not appear at all, so this even-numbered version is degenerate.

[0223] The odd-sized kernels are symmetrical about the center in both directions, but the even-sized kernels have their centers offset. The IC in the embodiment of the present invention places the centers of the even-sized kernels to the right and below the natural division, at positions In an alternative embodiment of the invention, the circuit may be modified to place the center above and to the left of the natural division.

[0224] In addition to the above instructions, Fig.16 The operation of a particular IC is the same as the operation of the other versions of the IC.

[0225] Fig. 10AThis is an example of an embodiment of the present invention. Fig. 9A and 9B or Fig.15 Schematic diagram of the internal structure and operation of synthesizers 905a, 905b and 905c. The source input stream value set in channel set 1001 (which can be a single or a mixture of data types required by the aperture function) is used by circuit 1004 to calculate the contribution of each individual synthesizer.

[0226] Circuit 1005 calculates the initial value of the sub-function using the output of 1004. Circuit 1006 calculates the continuing partial value of the sub-function using the output of 1004 and the partial value previously calculated by the synthesizer 1002 immediately to the left. Circuit 1007 calculates the continuing partial value of the sub-function using the output of 1004 and the partial value previously calculated and delayed from the synthesizer row (one of 908a, 908b, 908c, 908d, 908e, or 908f) immediately above 1003.

[0227] The operation of circuits 1005, 1006, and 1007 (which use the shared output of circuit 1004) can be performed simultaneously with the operation of circuit 1004 (within the same clock cycle), or the operation of circuits 1005, 1006, and 1007 can be implemented through a series of pipeline stages synchronized by the same clock.

[0228] Multiplexer 1008 selects which variable of the partial result is forwarded as the partial value of the sub-function as the output of synthesizer 1009. If COLFST 811 is not asserted, the output of 1006 is selected, otherwise if ROWFST 808 is not asserted, the output of 1007 is selected, otherwise the output of 1005 is selected.

[0229] This conditional processing is a natural consequence of allowing the M×N aperture function to extend to the edges of the source input stream (representing an array of R×C value sets). A single position at the leftmost or topmost edge will be the first computable element of the aperture function for multiple blocks adjacent to or overlapping those edges. Therefore, each synthesizer at the first computable position of an overlapping block needs to be initialized with a reference value for the aperture function. In addition, each synthesizer at the first computable position of a subsequent row of blocks must be combined with a prior value of the partial value of the same block calculated in the previous row. In this way, a single circuit can be used to ensure that all blocks overlapping, adjacent to, or located inside the topmost and leftmost edges are correctly calculated.

[0230] exist FIG. 10B to FIG. 10G In, all Fig. 10A Components introduced in the same manner and using the same component numbers are functionally equivalent to those in reference Fig. 10A The elements described are the same.

[0231] Fig. 10B This is an example of an embodiment of the present invention. Fig. 9A and 9B or Fig.15 Schematic diagram of the internal structure and operation of synthesizers 902a, 902b and 902c. A source input stream value set 1001 is used by circuit 1004 to calculate the synthesizer contribution to the aperture function.

[0232] Circuit 1005 uses the output of 1004 to calculate the initial value of the sub-function, and circuit 1006 uses the output of 1004 and the partial value previously calculated by synthesizer 1002 immediately to the left to calculate the continuing partial value of the sub-function.

[0233] Multiplexer 1010 selects which variable of the partial result is forwarded as the partial value of the sub-function as the output of synthesizer 1009. If COLFST 811 is not asserted, the output of 1006 is selected, otherwise the output of 1005 is selected.

[0234] Fig. 10C This is an example of the present invention. Fig. 9A or Fig.15 Schematic diagram of the internal structure and operation of synthesizer 904. A source input stream value set 1001 is used by circuit 1004 to calculate the contribution of each individual synthesizer.

[0235] Circuit 1005 uses the output of 1004 to calculate the initial value of the sub-function, while circuit 1007 uses the output of 1004 and the previously calculated and delayed partial value obtained from the synthesizer row (one of 908a, 908b, 908c, 908d, 908e or 908f) immediately above 1003 to calculate the continuous partial value of the sub-function.

[0236] Multiplexer 1011 selects which variable of the partial result is forwarded as the partial value of the child function as the output of synthesizer 1009. If ROWFST 808 is not asserted, the output of 1007 is selected, otherwise the output of 1005 is selected.

[0237] Fig. 10D This is an example of the present invention. Fig. 9A or Fig.15 Schematic diagram of the internal structure and operation of synthesizer 901. A source input stream value set 1001 is used by circuit 1004 to calculate the contribution of each individual synthesizer.

[0238] Circuit 1005 calculates the initial value of the sub-function using the output of 1004 and forwards it as the output of synthesizer 1009 as a partial value of the sub-function.

[0239] Unit 901( Fig. 9A , Fig.15) is always the first value used in any complete or truncated block, and therefore always generates an initialization value for the block.

[0240] Fig.10E This is an example of an embodiment of the present invention. Fig. 9B and Fig. 9C Schematic diagram of the internal structure and operation of synthesizers 903a, 903b and 903c. A source input stream value set 1001 is used by circuit 1004 to calculate the contribution of each individual synthesizer.

[0241] Circuit 1006 uses the output of circuit 1004 and the partial value previously calculated by synthesizer 1002 immediately to the left to calculate a continuous partial value of the sub-function, which is forwarded as the output of synthesizer 1009 as the partial value of the sub-function.

[0242] Fig.10F This is an example of an embodiment of the present invention. Fig. 9A and 9B or Fig.15 Schematic diagram of the internal structure and operation of synthesizers 907a, 907b and 907c. A source input stream value set 1001 is used to calculate the contribution of each individual synthesizer 1004.

[0243] Circuit 1006 calculates the continuing partial value of the sub-function using the output of circuit 1004 and the partial value previously calculated by the synthesizer 1002 immediately to the left. Circuit 1007 calculates the continuing partial value of the sub-function using the output of 1004 and the partial value previously calculated and delayed from the synthesizer row immediately above 1003 (one of 908a, 908b, 908c, 908d, 908e, or 908f).

[0244] Multiplexer 1012 selects which variable of the partial result is forwarded as the partial value of the sub-function as the output of synthesizer 1009. If COLFST 811 is not asserted, the output of 1006 is selected, otherwise the output of 1007 is selected.

[0245] Figure 10G This is an example of the present invention. Fig. 9A or Fig.15 Schematic diagram of the internal structure and operation of synthesizer 906. A source input stream value set 1001 is used by circuit 1004 to calculate the contribution of each individual synthesizer.

[0246] Circuit 1007 calculates a continuous partial value of the sub-function using the output of circuit 1004 and a previously calculated and delayed partial value from the synthesizer row (one of 908a, 908b, 908c, 908d, 908e or 908f) immediately above 1003. The output of circuit 1007 is forwarded as the output of synthesizer 1009 as the partial value of the sub-function.

[0247] Fig.11 is a diagram showing intra-row delay lines 908a, 908b, 908c, 908d, 908e, and 908f ( Fig. 9C ) is a schematic diagram of the internal structure and operation of the . These delay lines are used to hold part of the computational results from each row of synthesizers for use in the next row.

[0248] When COLLST is asserted, the current position of the source input stream is at the rightmost edge, and the line The outputs of the synthesizers (1101) to N-2 (1102) are retained by registers 1104 to 1105 respectively for future reference.

[0249] If the current position of the source input stream, colSrc, is less than Then multiplexer 1106 selects from the retained values ​​in reverse order from right to left as defined by the index calculation (n-2)-colsrc, otherwise it selects the current value (1103) from the last synthesizer in row m.

[0250] Note that when the source input stream column position is less than The rightmost synthesizers in the row will contain no valid data, making these time slots available for inserting reserved data.

[0251] The partial output selected by the multiplexer 1106 is fed to a first-in, first-out (FIFO) circuit 1107 having CN positions, which is configured to process the source input stream positions so that exactly one value is inserted and one value is extracted in the same order as it was inserted. Since the partially completed result from a position is not needed until the source input stream returns to the position of the same block in the next row, this creates a delay so that the partial result of one row of calculation will be present to the next row exactly when needed.

[0252] The selected partial output of multiplexer 1106 also feeds the same value (1114) to the final result delay lines 909, 910a and 910b.

[0253] The partial output extracted from FIFO 1107 is routed at 1108 to the leftmost synthesizer in the next row (1111) and to a series of parallel access registers 1109 to 1110 which further delay the partial output by one source input stream interval as the data passes through the register chain.

[0254] When the current position of the source input stream is at the leftmost edge, the FIFO directs the output data at 1108, and delayed results 1109 to 1110 are provided to the units 1111, 1112 to 1113 of the next row, respectively.

[0255] It is noteworthy that the additional values ​​on the right side of the source input array stream inserted into FIFO 1107 by multiplexer 1106 are only accessed via path 1111 when the source input array stream is located near the right edge, while the additional parallel paths 1112 to 1113 are only used when the source input array stream is located at the leftmost position to access the data normally inserted from path 1103. The apparent similarity in structure and requirements between right and left edge processing is a natural consequence of the symmetry of the subfunction overlapping the right and left edges of the source input stream array. When the value of N is even, the number of additional cells processed to support the left and right edges is different.

[0256] Fig.12 is a diagram showing the final truncated result delay line 909 ( Fig. 9C )'s internal structure and operation.

[0257] When processing the last row of the source input stream array, the partial result from the auxiliary output 1201 of the intra-row delay line 908d is considered as the final result of the last row of the truncation block and is retained in the FIFO 1202, whose number of elements C is equal to the width of the source input stream array.

[0258] After recording the final result of the truncation block, the output of FIFO 1202 will be immediately transmitted to further delay line 910a via 1203, or directly to final processing 911 if the value of M is such that no other delay line intervenes.

[0259] Fig.13 is a schematic diagram showing the internal structure and operation of the final truncated result delay lines 910a and 910b.

[0260] When processing the last row of the source input stream array, the partial results 1301 from the auxiliary outputs of the intra-row delay lines 908e to 908f are considered as the final results of the last row of the truncation block and are retained in the FIFO 1304, whose number of elements C is equal to the width of the source input stream array.

[0261] When POSTEN is asserted, multiplexer 1303 switches between taking values ​​from 1302 and taking values ​​from the final truncated delay line of the row above, which will have the effect of presenting the final truncated result in row-first order, compatible with the order of all previously output results.

[0262] Note that in an input frame cycle, when POSTEN is first asserted, the contents of FIFOs 1202 and 1304 are the final values ​​of the truncated block overlapping the last row of the source input stream array. Any data contained in FIFOs 1202 and 1304 before that cycle will not be processed, so any execution suppression is optional if the last row of the source input stream array is not processed.

[0263] After recording the final result of the truncation block, the output of FIFO 1304 will be immediately transmitted to further delay lines via 1305, or directly to final processing 911 if the value of M is such that no other delay lines intervene.

[0264] Fig.14 is a schematic diagram showing the internal structure and operation of the final processing of all complete and truncated results.

[0265] like Fig.11 As shown, and has the same structure and function, if the current position of the source input stream is at the rightmost edge, then The outputs of the cells of row M-1, (1101) to N-2 (1102) are retained by registers 1104 to 1105, respectively, for future reference.

[0266] If the current position of the source input stream is less than Then multiplexer 1106 selects from the retained values ​​in reverse order from right to left, otherwise it selects the current value from the last synthesizer (1103) in row M-1.

[0267] When processing the source input stream array, multiplexer 1402 feeds the result selected by multiplexer 1106 directly to finalization (1403).When in the post-processing stage, the output of truncated result delay line 1401 will be selected for finalization (1403).

[0268] Finalization circuit 1403 performs all additional computations (if any) to produce the final form of the output stream (1404) from the constituent block results. This can typically take the form of a rectified linear activation (RELU) function with negative values ​​set to zero and out-of-bounds values ​​set to the maximum acceptable value, or any other desired conditioning function (e.g., sigmoid or tanh). The post-processing function need not be completed within a single source input stream cycle, but needs to accept each final result at the rate of the source input stream array.

[0269] When DSTEN is asserted, the finalization circuit 1403 presents the final result as a value of the target output stream. At any time when DSTEN is not asserted, any partial or incorrect value produced by the finalization circuit 1403 will be ignored, so any suppression of the operation when the result is not used is optional.

[0270] In one embodiment, the target output stream array is processed by circuits similar to those described above. In this case, it is advantageous for the timing of the final truncated result to be the same as all previous final results. To this end, the control circuit 702 coordinates the control of FIFOs 1202 and 1304 to keep the output rate the same as the main output rate.

[0271] In another embodiment, the target output stream array is the final stage of the system and does not require further processing. In this case, it is advantageous to complete the timing of the final truncated result as soon as possible. For this reason, the control circuit 702 coordinates the control of FIFO 1202 and 1304 to output these results at the maximum frequency supported.

[0272] Note that the above embodiment produces a single output element from a complete set of input elements. In a complete system generating a large number of output elements from an input set, the entire mechanism described is repeated once for each output channel, with the obvious exception of control circuit 702, which may be shared by the output channels, since the timing of all the individual sub-functions is the same for the entire output set.

[0273] The inventor has built a working prototype of the IC in the embodiment of the present invention to test and confirm the details and features of the present invention, and the operation of the prototype confirms the above description. The inventor has also developed a software-supported simulator, which has been used to test and confirm the above details and description before submitting this application.

[0274] In another aspect of the invention, a system is provided for accepting an input stream of three-dimensional data (as typically presented in medical imaging), including additional circuitry and buffers to allow a three-dimensional aperture function to pass through the three-dimensional input array and perform corresponding calculations to correctly implement interior and edge conditions for the first and last planes.

[0275] In yet another aspect of the present invention, a hardware-assisted neural network training system is provided for the complex process of training a deep neural network (DNN), wherein most of the work is done by the forward reasoning engine, and the training algorithm only needs to use the statistical data collected from the forward reasoning to periodically adjust the weights and biases of the entire network so that the model converges to the desired state. By adding appropriate accumulators to sum the input states when calculating the forward reasoning process, the present invention forms a hardware-assisted neural network training system.

[0276] In yet another aspect of the present invention, a single batch multiplier with limited bit width precision is provided to address the well-known problem that floating point precision limitations hinder convergence of DNN models (known in the art as the "vanishing gradient problem"), which can be cascaded with additional adders to produce arbitrarily high precision floating point products. While this innovation is typically not required for forward reasoning calculations, it may be critical in DNN trainers to avoid problems that occur when calculated gradients become too small to measure.

[0277] N-up parallel processing

[0278] In the above-described embodiments and implementations of the present invention, the focus is on the apparatus and method for batch multiplication when executing functions that require multiplication, and the implementation of aperture functions by a novel IC in a convolutional neural network (CNN). However, it is well known in the art that a complete deep neural network (DNN) must implement a whole set of completely different aperture functions, many of which may require only a minimal amount of computation.

[0279] To comply with embodiments of the invention, each such embodiment must comply with a system-wide pipeline format that accepts inputs in the form of parallel streams of values ​​representing arrays in a consistent order, and simultaneously produces outputs in the form of parallel streams of values ​​representing arrays in the same order. The final node of the DNN can return a conclusion reflecting the position array, or return a conclusion about the entire input array. The embodiments of the invention described below are for executing DNNs in a novel IC that supports pipelined execution.

[0280] In aspects of the present invention, the inventors have developed a method and apparatus to significantly accelerate pipeline operations in CNNs and DNNs. The inventors have proposed pipeline operations in some embodiments to stream inputs to ICs in multiples in parallel. In the above embodiments, the input typically flows through each column from left to right in all embodiments, and then flows through each row from top to bottom. Taking RGB data as an example, this will take the form of three separate channels at each pixel position, typically 8 bits per channel, representing each of the three independent RGB color values ​​observed at each pixel position. The inventors call this a 1-up implementation. 1-up means streaming input values ​​for one pixel at a time. Or more generally, streaming values ​​for one input position at a time in the input array.

[0281] The inventors have recognized that considerable advantages can be gained by streaming inputs through multiple input locations at a time, such as in the case of pixels. To do this, circuitry must be added to the new IC to perform the input streaming and produce the output streaming. This change is typically in scale rather than complexity, as circuitry implemented in a 1-up environment is repeated in the IC to process input values ​​from additional input locations (pixels in this case) in parallel.

[0282] While the circuitry is least problematic when the width of each row is an integer multiple of the input count to be transferred in parallel, this is not a necessary limitation of the invention. Using pixels as an example, for a resolution of 1920×1080, the number of pixels per row (1920) is divisible by 1, 2, 3, 4, 5, 6, 8, 10, and 12. Therefore, streaming the RGB values ​​of two pixels (called 2-up) is an effective approach, as are 3-up and 4-up. As the number of pixels increases, the size of the IC to handle all the processing also increases, and the factor of increase is directly related to the number of pixels that need to be processed in parallel, so the user must make a reasonable decision.

[0283] However, as data flows down through the nodes of a DNN, the size of the input array is often reduced in size in cases where the aperture function has a stride other than 1 (not every input position immediately produces an output position) or the aperture function is defined to avoid overlapping the edges of the input array. In these common cases, the width of the input array cannot be restricted to an integer multiple of any given number of parallel positions N. One solution is to always align the left edge of each row of the input array with a specific position (nominally the left side) of the set of N positions. The right edge can be represented by an incomplete set, always starting at the first position of the set of N positions. Additional circuitry is then used to avoid computation with invalid data and to suppress any output derived from said invalid data.

[0284] In an embodiment of the present invention, for a 2-up implementation in a pixel instance, the R, G, and B values ​​for each of two adjacent pixels are streamed to the IC as pipeline inputs. The first two pixels are the first two pixels on the left side of the top row. For the RGB instance, there will be six input values, which are the R, G, and B values ​​for each of the first two pixels. The next two pixels in the row are the next pixels in the stream, and so on, across the top row, followed by the R, G, and B values ​​for the first two pixels in the second row, and so on, until across the input array. For 3-up or 4-up, the same general protocol is followed.

[0285] Fig.17A and Fig. 17BA 1-up pipeline solution is shown for a well-built minimal deep DNN model that might be used to understand images and respond to the relative excitation intensities of various objects recognized by the trained model. The input channels 1701 are presented as input values ​​for each pixel in a specific order, typically from left to right across each row, then from top to bottom, as described above. For RGB data, this takes the form of three separate channels, typically 8 bits per channel, representing three independent color values ​​observed at that location. The scope of the present invention is not limited to eight-bit channels.

[0286] If the input to this DNN circuit is the output of another DNN circuit (which will naturally happen if a large DNN is broken into smaller parts to aid processing), the channels presented will be one channel for each feature passed into the DNN. For example, if a particular part of the model requires 64 feature channels as input, each value will be presented in parallel as an unsigned or signed integer or floating point value in the specified format with the desired number of bits of precision.

[0287] It is important to understand Fig.17A The boxes depicted in (and in other figures described) do not represent steps that are performed sequentially. Each box represents an input channel or a circuit that performs a function (such as an aperture function). The arrows between the boxes represent sets of parallel conductors that pass values ​​between processing circuits. As long as input is provided to the box, all processes are activated simultaneously. As the input flow begins, the circuits represented by the boxes are activated one by one until all processes are activated and output flows are also produced in multiple channels. While still accepting input, the emission of the final output at the first corner of the input array (nominally the upper left corner) begins.

[0288] The first 7×7 convolution node 1702 in this model is a typical RGB input in a DNN for visual understanding. This 7×7 kernel may be applied only when the kernel block fits within the input array boundaries (typical for RGB input), or it may be applied to every input position and synthesize missing values ​​(typical for reprocessing features). Typically, a large number of output channels are generated (usually 64), and the number of channels throughout the rest of the system typically increases as feature values ​​pass through additional nodes.

[0289] Subsequent convolution nodes 1703, 1704, 1705 also accept and produce multi-channel array streams of the same dimension as their input. The number of output channels of each node is arbitrary and can be greater than, less than, or equal to the number of input channels.

[0290] The concatenation node 1706 in this model accepts the parallel input array streams generated by nodes 1704 and 1705 and synchronizes them to produce a combined set of channels. The channel values ​​of the convolution node do not change. However, due to the characteristics of the pipeline, each output from the 1×1 convolution corresponding to a specific array position is produced before the output from the 3×3 convolution. Therefore, the concatenation function must provide buffering in the form of a first-in-first-out (FIFO) circuit so that all channels can output data corresponding to the same position at the same time.

[0291] The MaxPool node 1707 in this model compares all values ​​of the block using an aperture function and outputs the maximum value for each channel separately. The number of channels is not affected, but the array dimensions of the input stream will be reduced in the output stream. If the MaxPool node reduces the horizontal dimension by 2 and the vertical dimension by 2, the size of the output array will be one-fourth the size of the input array.

[0292] Since the frame rate of the input and output streams must be the same (the output can't go faster than the input, nor slower than the input, or you'll lose data), the net effect is that the clock frequency of the output array stream will be proportionally reduced.

[0293] In this MaxPool example, since only one output is produced for a block of four input positions, the required output rate is only one-quarter the input rate. Therefore, all subsequent nodes in the pipeline will operate at a reduced effective throughput. As the number of channels continues to increase, the reduced effective throughput may be advantageous. When more cycles are available to perform the required calculations, some resources that may be dedicated to each channel can be shared between channels, resulting in an overall reduction in circuit size with only a slight increase in power. The reduction of dimensionality is also an important basis for the present invention.

[0294] Subsequent nodes in the model may use similar or different connection patterns, as long as each pattern supports simultaneous presentation of all data channels corresponding to a given position in any input array stream at the system-wide interface.

[0295] As shown, the values ​​after the MaxPool node 1707 flow to additional Convolution, Cascade, and MaxPool nodes in this model, but since these nodes are functionally identical to those already described, they do not have component numbers.

[0296] Fig. 17BThe uniqueness of the global average node 1708 in is that the aperture function of node 1708 covers the entire remaining dimension of the previous input array stream and only returns the average value of each channel over the entire array. Therefore, the output array dimension is 1×1 and forms the output channel 1709 of the entire circuit.

[0297] Fig.18A and Fig.18B Shows the implementation and Fig.17A and Fig. 17B The overall structure and process of the 4-up pipeline of the same form of DNN model is shown.

[0298] The input channels 1801 are presented in parallel as four sets of data for each channel. For RGB data, this would take the form of four individual pixels, representing four adjacent columns of the input array, with each pixel containing four RGB values, for a total of 12 inputs received in parallel at the same time. Alternatively, the input channels may come from another DNN circuit, in which case they take the form of four complete sets of input channels, representing four adjacent columns of the input array. For example, if the model requires 64 feature channels as input, the four sets of inputs would contain a total of 256 parallel inputs.

[0299] The first 7×7 convolution node 1802 is a typical node for processing RGB input in a DNN for visual understanding. In this 4-up implementation, node 1802 accepts input of four pixels at a time and produces output of four pixels at a time. The number of output channels is typically quite large compared to the number of input channels, 64 or more, and no longer represents color information. In this model, in the rest of the DNN, channels represent the detection strength of a feature or feature combination found in the input array, and each position has an independent value. Each subsequent convolution node 1803, 1804, and 1805 also receives and processes input of four pixels per channel at a time. Cascade node 1806 accepts four groups of channels from convolution nodes 1804 and 1805 and outputs combined channels in groups of four.

[0300] The first MaxPool node 1807 is labeled 4-up to 2-up. Node 1807 takes the maximum of four samples representing a block of input array positions consisting of two adjacent columns on two consecutive rows. Since the effect is to reduce the dimensionality of the input array stream so that the width and height of the output array stream are half of the original, the effective throughput of all subsequent nodes is reduced by a factor of four. When single-input processing is used, the subsequent processing clock can be reduced to advantage by utilizing more compact circuits.

[0301] When N-up parallel input processing is used, the reduction in output array width serves to reduce the number of parallel outputs. Since the parallel inputs represent adjacent columns in the input array stream on the same row, only a reduction in width makes sense. While the N-up parallel outputs can be retained at a reduced frequency, there is no advantage in size or power to doing so. The net effect of the MaxPool node 1807 is to reduce the parallelism in the horizontal dimension from 4-up to 2-up (as indicated by the markings) and reduce the processing frequency by a factor of two, rather than by a factor of four as in the 1-up case described above.

[0302] Nodes 1808, 1809, 1810, and 1811 process data in 2-up parallel tracks, and each node is approximately half the size of its 4-up counterpart. This does not represent a reduction in power, as the total number of operations required for a 4-up, 2-up, or 1-up circuit is the same, it is just that the overhead of managing N-up coordination is reduced.

[0303] The second 2×2MaxPool node 1812 again takes the maximum of four samples representing a block of input array positions consisting of two adjacent columns on two consecutive rows. The net effect of node 1812 is to reduce the parallelism in the horizontal dimension from 2-up to 1-up and reduce the processing frequency by a factor of two. Fig.18B As shown, all subsequent nodes operate on a single set of their respective input and output channels, with the final output 1813 taking the form of a single sample per channel presented simultaneously in parallel.

[0304] Fig.19 and Fig. 20 is a table describing the array stream sizes of typical small DNNs applied to input streams compatible with images in HDRGB format. Fig.19 The table describes a DNN that only implements 1-up processing, such as Fig.17A and Fig. 17B shown; Fig. 20 The table of describes the same DNN that initially implements 4-up processing and transitions to 1-up processing in subsequent nodes, as Fig.18A and Fig.18B shown.

[0305] Having described the terminology and general procedures of N-up parallel processing, the inventors now provide examples of specific apparatus and methods for applying a 3×3 convolution function to an input array using 4-up parallel processing. The input array in this example is an array of pixels in RGB color, as used in many other examples in this specification. It should be noted that this is not a limitation on the scope of the invention, as 3×3 convolution with 4-up parallel processing can be used for input arrays of many other formats. In this example, it should be understood again that the 3×3 blocks shown represent circuits that perform a kernel function on an input stream.

[0306] Fig.21 An example of an IC circuit that performs a 3×3 convolution node using a 4-up data flow is shown. Fig.21 In , a set of four inputs 2101 is a set of inputs retained from the immediately previous input interval and retained together with the current set of four inputs 2102 to provide all the inputs required for all four output channels of the 3×3 convolution. As described in detail above, in pipeline processing, the inputs of the immediately previous input interval and the inputs of the current interval are required to fully calculate the output.

[0307] p0, p1, p2, and p3 represent the input channel values ​​at positions 0, 1, 2, and 3 in the first row of the input array, respectively. For brevity, only a single symbol is used, but each symbol represents all channels at an input position. For the pixel case, each data point p x Represents the R, G, and B values ​​of the pixel.

[0308] w 0,0 to w 2,2 Represents the set of weights to be applied to the values ​​in the input channels. Since each weight is applied to only one input channel, the number of input channels does not affect the structure of the circuit, so multiple channels are not shown.

[0309] The weights in kernel rows 2103, 2104, and 2105 are applied in parallel (simultaneously) to input channels p0, p1, and p2, and the partial products of each set of weights in each row are summed immediately as per the rules of the aperture function of the 3×3 convolution. As described above in the implementation details of pipeline processing, the partial summation is passed from each functional circuit to the next functional circuit, and an output is generated when all necessary partial summations are completed. After the weights of row 2105 are applied, the partial products are combined with the sum of the products of the weights of row 2104 applied in the previous row, resulting in the final output of the kernel for the current row. The weights of row 2104 are applied, and the partial products are combined with the sum of the weights of row 2103 applied in the previous row, resulting in an intermediate value. The weights of row 2103 are applied, and an initial value is generated by summing the partial products and retaining the partial products for later use. Bias (if any) may be introduced at any stage. The activation function (if any) is applied to the final output 2105.

[0310] A complete circuit implementing the weights of rows 2103, 2104, and 2105, including bias and activation functions, produces the first output channel of the 4-up set.

[0311] When the first 4-up set is presented from the input array stream, there is not enough data to compute all four outputs required, so computation of all outputs is delayed until the second 4-up set is fetched and valid data is available for computation using inputs from both sets 2101 and 2102.

[0312] Circuits 2106, 2107, and 2108 apply weights in circuits that are copies of the previous circuit, and these functions differ only in where the weights are applied to the inputs. Note that the weight set w 0,0 to w 2,2 The weights are the same for all output channels, but the combination of a weight and an input channel is unique.

[0313] The output calculated using the weights of row 2105 of the first kernel copy produces the first parallel channel set q0 of the output array stream, while the output using the weights of the other kernel copies 2106, 2107 and 2108 produces the remaining parallel channel sets q1, q2 and q3 of the output stream 2109 respectively.

[0314] Since the first output q0 corresponds to a 3×3 kernel centered at p1, it is the same as Fig.21 The circuit corresponding to the layout in is the solution of an inserted or "valid" version of the 3×3 convolution. Therefore, the width of the output array stream is reduced by two positions compared to the width of the input array stream, which meets the definition of the changing aperture function. The height is also usually reduced by two rows, but this is irrelevant for horizontal processing.

[0315] Fig. 22 The layout required for the circuit to produce the output of a 4-up input channel for the "same" version of a 3×3 convolution is shown, i.e., the dimensionality of the output array stream is not reduced, and one output position is produced for each distinct position of the input array stream. In this variation, input set 2203 represents the current values ​​of the 4-up input array stream, while input sets 2202 and 2201 represent the retained values ​​of the previous two sets.

[0316] Application of kernel circuits 2204, 2205, 2206 and 2207 produces values ​​q0, q1, q2 and q3 of the 4-up output array stream 2208, respectively, and is now aligned so that the center of each kernel corresponds to a position in the 4-up input array stream.

[0317] When the first 4-up channel set is presented from the input array stream, there is not enough data to compute all four outputs required, so computation is delayed until the second 4-up set is presented and valid data is available for both sets 2202 and 2203. Valid data for set 2201 is not yet available, and kernel circuit 2204 will suppress the application of weights to p3 of 2201, or force uninitialized values ​​to zero, consistent with how a 3×3 convolutional aperture function is applied to a location that causes the kernel to overlap the edge of the input array. The first set in each row triggers this suppression mechanism, but subsequent sets in that row will use p3 value set 2201 to compute the full kernel for that output position q0.

[0318] When processing full DNNs, there are cases where the 4-up stream technique is applied to a stream of input arrays whose width is not an even multiple of 4. In this case, invalid values ​​in the final 4-up set are suppressed by forcing them to zero or otherwise, and the final output position in the last 4-up set of the row is ignored. This is consistent with the inserted ("valid") and full ("same") variants of the 3×3 aperture function.

[0319] In all cases, the first position of each row of the input array stream always appears in the first position of the 4-up input set.

[0320] If the length of the input line is not an even multiple of the processing set width, the processing clock needs to be increased so that the total throughput of the N-up processing matches the throughput of the 1-up input source, and special buffering is required to pack the incoming values ​​into N-up sets. This special buffering is described below.

[0321] Fig.23The required layout for two variations of a 1 row by 7 column convolution on the circuit output 4-up data is shown. From the previous discussion of 3 rows by 3 rows of convolutions, those experienced in the art should know that the specific number of rows in the kernel affects only the number of partial sums preserved over time, and does not affect the mapping of kernel weight columns to input set columns. Therefore, Fig.23 The data layout shown in is equally applicable to 7×7, 3×7, or any other kernel with a width of 7.

[0322] As described above, input set 2303 is the currently presented 4-up data set from the input array stream, while sets 2302 and 2301 are the previously presented and retained data sets from the immediately preceding and second previous sets, respectively.

[0323] Kernel processing circuits 2304, 2305, 2306 and 2307 represent the alignment required to produce the inserted (“valid”) convolution output 2308, while circuits 2309, 2310, 2311 and 2312 represent the alignment required to produce the complete (“identical”) convolution output 2313.

[0324] Circuit 2304 W 0,0 is aligned with P0 of input set 2301 to produce an insertion variant, and w of circuit 2308 0,3 Aligned with P0 of input set 2302 to produce a complete variant where circuits 2304 and 2309 both produce output q0 for their respective use cases.

[0325] It will be appreciated by those skilled in the art that there is considerable overlap in the common functionality of the two sets of cores, and that it is straightforward to arrange a single circuit to produce either variant as desired using only five uniquely mapped core circuits. It will also be appreciated by those skilled in the art that any M-up stream data set (including 1-up) can be repackaged into any other N-up stream format (where M≠N) as desired to keep the overall throughput of the system high enough to accept and process the input array stream at the given rate. The cost of this is the need for N copies of certain core processing circuits, but the overall effect is to allow the circuit to constrain the processing clock to within reasonable limits for the implementation approach while still accepting input streams at full speed.

[0326] Fig.24A and Fig. 24B A typical implementation of a 2x2 MaxPool node is shown, where the maximum value of each channel is selected for different blocks of two adjacent column positions on two adjacent rows.

[0327] Fig.24AThe layout of a 2×2 MaxPool node on a 4-up data flow 2401 is shown. When the first row of each pair is presented, comparator 2402 evaluates inputs p0 and p1 and passes the larger input to FIFO circuit 2403 for retention for use when the second row is presented. At the same time, comparator 2404 and FIFO 2405 perform the same operation on inputs p2 and p3. When the second row of each pair is presented, comparator 2402 accepts the retained maximum value for the same column position of the first row from FIFO 2403, compares the maximum value with inputs p0 and p1 and outputs the larger of the three values ​​as output q0, while comparator 2404 and FIFO 2405 perform the same operation on inputs p2 and p3 to produce output q1.

[0328] The output set 2406 includes two sets of channels, with a single value for each channel being the maximum of the four samples for each particular channel (in this aperture function, values ​​from different channels do not affect each other). Fig.24A Output 2406 is a 2-up output data stream generated from a 4-up input data stream.

[0329] Fig. 24B The same 2×2 MaxPool node is shown laid out on a 2-up data stream 2407. The comparators 2408 and FIFOs 2409 are functionally identical to those described above, but only a single set is required to accept the 2-up inputs p0 and p1 to produce a single set of output channels 2410. Therefore, the output 2410 of the second example is a 1-up output data stream produced from the 2-up input data stream, and all downstream nodes can take on a smaller 1-up form.

[0330] The tiled MaxPool function, as well as any other aperture function with a 2×2 stride, reduces the size of the input array by a factor of 2 in each dimension. Since the total width of the N-up array stream is N times the number of sets presented, reduction can be achieved by either reducing the set width or reducing N, as long as N is divisible by the horizontal stride. Since N is the replication factor of the copies of the circuit executed in parallel, it is preferred to reduce N as much as possible.

[0331] Fig.25A contrived example where N cannot be reduced is shown. It applies a 2×2 MaxPool node, but in this case to a 5-up input stream. As in the previous 3×3 convolution case, input set 2501 is retained and used in conjunction with the current input set 2502 to present a minimum set of values ​​so that all outputs can be produced on the same clock cycle. (Other layouts are possible, such as switching the first comparator to process p0 vs. p1 or p1 vs. p2 on alternating input sets, while setting the middle comparator to process p4 vs. p0 on alternating inputs. This would reduce the number of copies required for the aperture function from five to three, and would be advantageous in cases where the implementation of the aperture function is much more complex than a simple comparator.)

[0332] In this embodiment, comparator 2503 and FIFO 2504 operate on the retained values ​​of p0 and p1, comparator block 2506 operates on the retained values ​​of p2 and p3, and comparator block 2507 operates on the retained value of p4 and the current value of p0. Comparator block 2508 operates on the current values ​​of p1 and p2, and comparator block 2509 operates on the current values ​​of p3 and p4.

[0333] Since a 2.5-up data flow is not possible within the constraints of the pipeline, the dimensionality reduction in this example must be applied to the width of the input array, so the output 2510 is a 5-up output reflecting a 5-up input flow.

[0334] As mentioned above, in some cases it may be wise to repack the M-up stream into an N-up stream with the same array dimensions. A specialized FIFO circuit can be used to perform this function. Fig.26A Such a FIFO is shown for repackaging a 4-up stream 2601 into a 2-up stream 2603. FIFO 2602 accepts 4 inputs at a time and stores them as separate entries. Whenever 2 entries are available in the FIFO, 2 outputs are produced at a time. The data flow in Figure 26 (and the data flow in the following figure) goes from the input down through the circuit to the output.

[0335] In the common case where the width of the input stream is not an integer multiple of the input stream set size, a counter must be included to track the number of valid entries presented per row. For example, if the input array width is 10, using 4-up input sets, where 3 4-up input sets are required to cover a full row, the FIFO must ignore the last two entries of the third set of input presented, and output 5 2-up input sets, instead of 6. After each row, the counter is reset and begins counting entries for the next row. Array width limits can be fixed or presented via preloaded registers. This logic can be omitted if it is known that the array width is always an integer multiple of the input set size and the output set size.

[0336] Fig.26B The 3-up stream 2604 is shown repacked into a 5-up stream 2606. The FIFO 2605 accepts 3 inputs at a time but stores them as separate entries. The FIFO produces 5 outputs at a time whenever 5 entries are available in memory.

[0337] As mentioned above, in the case where the array width is not an integer multiple of the input set size, additional operations must be performed to handle invalid entries that may appear at the end of a row. A similar problem occurs when the array width is not an integer multiple of the output set size. In this case, a final set must be published when each row is fully received, containing the final entries for the row in the first output and invalid entries that do not have a specific value in the remaining channel sets. For convenience, the practice of placing all zeros in invalid entries can be used to reduce the total circuit size in subsequent nodes where zeros have no effect, such as in convolutions and MaxPool.

[0338] The size of the FIFO must be large enough to hold as many input sets as possible so that no data is lost. To maintain overall system throughput, outputs are emitted as soon as enough entries are available to generate an output set.

[0339] While any set size can be repacked into any other set size, the required processing frequency will change in proportion to the size. For any M-up input that is repacked into an N-up output, the required processing frequency can be described as

[0340]

[0341] Throughout the system, to simplify operation, each circuit that accepts a row should provide and ignore all unused invalid entries at the end of a row where the row width is not an integer multiple of the size of the set. This is not a strict restriction, as the circuits may use additional logic not shown in this article anyway. This guarantees that each column position maps to the same set of channels in the parallel set presented for each row, and minimizes the complexity of operations that group together values ​​from the same column position in multiple rows.

[0342] Fig.27AAn embodiment of a cascade node is shown where channels from one source 2701 are positionally cascaded with channels from another source 2702 or more sources (not shown) so that the output 2706 contains all channels from all sources. This node does not mix or change channel values. In the common case where the sources have different timings, one or both of the FIFOs 2703 and 2704 will hold the input channel values ​​until a complete set of output channels is available. Interleaving circuit 2705 will cascade all channels from set p0 from each source to generate q0, and then cascade all channels from set p1 to generate q1, and so on.

[0343] A common instance where this solution is needed is combining the output of a 3×3 convolution node and the output of a 1×1 convolution node, each of which is applied to the same input array stream. Although both nodes process the streams at the same rate, the output of the 3×3 node cannot be finalized until the third row of the input stream is presented, while the output of the 1×1 node can be finalized as soon as any data from the input stream is presented. The net effect is that the output of the 1×1 node corresponding to a particular position of the input array stream will be presented to the cascade node significantly before the output of the 3×3 node. Since the next node after the cascade node will require all channels at any given position to be presented before any computation can be done, the cascade node must buffer the earlier presented input stream and wait for the later presented input stream to arrive at the same position before it can present the full set of all channels for a given position on the output. The same is true for 1-up or N-up data streams.

[0344] If each input array position of the slowest path is always presented after the same position through all other paths, then the FIFO for that path can be omitted. If under certain conditions (usually the final position of a stream) the slowest path does not appear last, then the data in the FIFO for that path must retain a minimum number of entries to prevent data loss in these special cases.

[0345] If the data path widths of the sources are different, then Fig.26A and Fig.26B The path widths are repacked to match each other as shown, or this function can be merged with the FIFO for cascade buffering. It should be understood by those skilled in the art that any number of paths can be cascaded into a single operation by adjusting the size of the FIFO of each earlier path to retain as many values ​​as each path can present in the worst case before the slowest path presents the corresponding position.

[0346] Fig.27BAn implementation of a 4-up dense node is shown. The dense node is mathematically equivalent to a convolution where the kernel size is the same as the size of the input array. Therefore, to create each output channel, a different weight is applied to each input position of each input channel. The number of output channels is independent of the number of input channels, and the resulting output array is always a 1×1 array. Since the inputs 2707 are submitted in groups of four in this exemplary implementation, the weights 2708 specific to each input position are loaded from local storage (not shown) and multiplied with the current input in circuit 2709 to form partial products of the complete kernel. All partial products of all input channels presented are added to produce a single 1-up set of output channels 2710.

[0347] Fig.27C An implementation of a 4-up global average node is shown that takes all values ​​at all positions for each input channel and averages them to produce the same number of output channels. The global average node is mathematically equivalent to a convolution where the kernel size is the same as the size of the input array and is applied to each input channel individually (rather than all input channels together as described above) with a common constant equal to the inverse of the number of elements in the kernel. Since multiplying by the inverse before or after the summation operation is mathematically equivalent, circuit 2712 simply sums all values ​​at each position of each input channel 2711 and then multiplies by the inverse of the number of elements after all input values ​​have been summed to produce each output channel. Since all input positions are merged into a single value, the output 2713 is a single 1-up set of channels with an array size of 1×1.

[0348] Fig.28 A 4-up implementation of a 3×3 local average node is shown that uses a sliding aperture function to compute the average of each input channel over a subset of positions to produce an output channel. This implementation forms an interpolated or "valid" output set where the aperture does not overlap the edges of the input array and the number of samples is the same for all output positions. Each output channel corresponds to a single input channel and data is not mixed between channels. Fig.21As with the implementation of the convolution node with similar size and input map shown in , the current input set 2802 is retained by register 2801 so that the current and immediately previous input sets can be accessed simultaneously. Circuits 2803, 2804, and 2805 perform the same summation on each input channel set p0, p1, and p2, respectively, but apply the summation to three different partial sums that vary over time to produce a set q0 of output array streams 2809. Circuit 2803 initializes the running sum of the first row, circuit 2804 uses the output of circuit 2803 delayed by a FIFO (not shown) to produce the running sum of the intermediate rows, and circuit 2805 uses the delayed output of circuit 2804 to produce each final sum. Circuit 2805 then multiplies the final sum by the inverse of the number of elements (1 / 9 in this example) to produce the output set q0. The activation function can be integrated into the circuit or equivalently placed between nodes.

[0349] Equivalent circuit 2806 generates output set q1 from channel set p1, p2 and p3 of the previous input set. Similarly, circuit 2807 generates q2 from p2 and p3 of the previous input set 2801 and p0 of the current input set 2802, and circuit 2808 generates q3 from p3 of the previous input set 2801 and p0 and p1 of the current input set 2802.

[0350] If a local mean aperture function is to be generated for each valid position, the array size of the output 2809 is reduced compared to the input, in this case by two positions each in width and height, but this is generally not enough to significantly reduce the 4-up stream. If a horizontal step size other than 1 is used, i.e., not every possible output position is utilized, the reduction in horizontal size can be implemented in the circuit as a reduction in N. For example, if the horizontal step size is 2, only every other value is needed, and the circuit can generate a 2-up output channel by calculating only q0 and q2, and omitting the unused circuitry for q1 and q3. Similarly, if the horizontal step size is greater than 4, the various circuits used to calculate q0 to q3 can be used in sequence to generate a 1-up output stream.

[0351] Fig.29 Another 4-up implementation of a 3×3 Local Average node is shown that forms a complete or "identical" set of inputs, where the aperture overlaps the edge of the input array, and the output array dimensions are the same as the input array dimensions. In this case, the number of input positions sampled at the edge is different from the complete set of samples taken in the interior, so the final inverse used for each output position must reflect the number of samples used for that output position.

[0352] and Fig. 22 Similar to the exemplary circuit shown in, Fig.29The variant in utilizes input set 2903 to present the current values ​​of the 4-up input array stream, while input sets 2902 and 2901 present the retained values ​​of the previous two input sets.

[0353] The application of summation circuits 2904, 2905, 2906 and 2907 produces values ​​q0, q1, q2 and q3 of the 4-up output array stream 2908 respectively, and is now aligned so that the center of each summation corresponds to a position in the 4-up input array stream. In this example, when the first 4-up input is presented at the beginning of each row, only summation circuit 2904 will intersect the left edge of the input array, but all four summation circuits may intersect the right edge of the input array, depending on the number of sets of padding at the end of the row, so the choice of reflecting the inverse of the number of samples taken will vary accordingly.

[0354] observe Fig.21 and Fig.28 The close correspondence between the exemplary circuits of Fig. 22 and Fig.29 Given the close correspondence between the exemplary circuits of , those skilled in the art will appreciate that the structure and replication of the computations are unaffected by the nature of the aperture function implemented, and that the apparatus and method are equally applicable to any aperture function defined on a similar sliding window.

[0355] Fig. 30A An implementation of a 4-up subset node is shown that passes only specific channels to the next node, but passes the specific channels with equivalent array dimensions and timing. This node type is typically used to split incoming channels so that different processing can be applied to each set of incoming channels. If the set of channels routed to the output is fixed, the connection between input 3001 and output 3003 can be achieved by direct wiring of physical conductors. Otherwise, routing circuit 3002 will use multiplexers to affect the desired channel selection.

[0356] Fig. 30BA typical implementation of a 4-up clipping node is shown, which presents a positional subset of an input array stream to an output array stream. Typically, entire rows of the top edge or bottom edge (or both) and columns of the left edge or right edge (or both) are omitted together. In order to allow the number of columns omitted at the left edge to be not an integer multiple of the data set size N, the current input set 3005 is combined with the previous input set 3004 in a repackaging circuit 3006 to produce a channel set q0, q1, q2, and q3 of the output 3007, so that q0 is always used for the first column of each row. When it is not necessary to omit the left edge of the input array stream, or the number of columns omitted is an integer multiple of N, the previous input set 3004 can be omitted from the simplified circuit. If the output array is greatly reduced relative to the input array, the N-up input stream can be repacked into an M-up output stream in the position selection circuit.

[0357] In any of the nodes described above, either batched or single multipliers can be used with the same effect. Batched multipliers have advantages over single multipliers based on the bit width of the multipliers and products when multiple weights are applied to each input. In other cases, a single multiplier with equivalent precision may be smaller or use less power. N-up pipelining does not depend on the type of multipliers used.

[0358] In another aspect of the invention, an IC may be provided with one or more interconnected functional circuits and input and output ports, each IC implementing a portion of a neural network, as described above with reference to Fig.17A and Fig. 17B as well as Fig.18A and Fig.18B In a system embodiment, a single IC of these ICs may be connected from the first IC receiving the primary input from the source array to the other ICs, from the output port to the input port, in a linear sequence or in an interconnected chain with parallel connections. The output port of the last IC in the connected set will provide the output of the neural network including the functions of all ICs.

[0359] Fig.31 Such an interconnected IC system 3100 implementing a neural network is shown. IC 3101 has an input port 3102 for receiving a stream of input values. The input values ​​may be in any of the protocols described above for input arrays, which may have a single value for each position in the array, or multiple values ​​for each position, such as RG and B values ​​for each position of the input array in an HDMI image, or the input stream may be ordered as an N-up stream as described in the above embodiments.

[0360] Fig.313101, 3105, 3106, 3107, and 3108 are shown interconnected between input and output ports. IC 3101 is shown interconnected with functional circuit 3104 leading to output port 3103, which is connected to the input port of IC 3105. The functional circuit can implement the aperture function described in the different embodiments above. In this example, IC 3105, 3106, 3107, and 3108 show functional circuits with the same interconnections as IC 3101, but it is emphasized that the ICs are different and the interconnections between the functional circuits are different. The figure is representative.

[0361] IC 3105 is connected to the input ports of ICs 3106 and 3107 through an output port to illustrate that a simple linear connection between ICs may not exist. The output ports of ICs 3106 and 3107 are both connected to the input port of IC 3108. Again, the diagram is representative. In any system of interconnected ICs, the interconnections may be more complex. IC 3108, as the last IC in the system, outputs an output stream for a neural network implemented by the system of interconnected ICs. The connections between the input ports and the output ports are parallel paths of conductors that provide a bit of value for each output interval. The IC system implements a neural network of a certain depth. In this aspect of the invention, an unlimited variety of neural networks can be implemented by interconnecting the individual ICs and providing different nodes and interconnections on the single IC.

[0362] Application to 3D image data

[0363] In the embodiments and examples of the present invention described in detail in many of the above embodiments, the primary input data source is in the form of two-dimensional image pixel values, for example, an RGB image as an HDMI frame, which is a 1080×1920 matrix of RGB values, and each pixel is a 3×1 color component vector. However, in the operation of the apparatus and methods in various embodiments of the present invention, not all input data sets are in the form of two-dimensional (2D) arrays (such as HDMI frames). However, on page 41 above, a case where the primary input data source is a three-dimensional array of data points is introduced.

[0364] As an example of a three-dimensional image that can be used as an input to a neural network, many medical devices (e.g., magnetic resonance imaging (MRI) devices) can capture three-dimensional (3D) image data. The following description in this application expands on the above-described unique 2D processing device and method to enable detailed processing of a value stream representing 3D image data.

[0365] In order to understand the application of embodiments of the present invention to 3D data, we can reconsider the 2D data samples used in the previous description, such as HDMI frames, which have been described above as a series of pixels arranged in a 2D plane. The aperture function is described as being based on blocks (e.g., 3×3 blocks), i.e., three pixels wide and three pixels high. The 3×3 aperture function manages the calculation of data values ​​involving nine pixels or features from the previous nodes, which depend on the position of the block centered on a particular pixel in the image. If we now consider the third dimension orthogonal to the 2D array plane, we can consider a 3×3×3 3D block with 27 data points. A 3D block must have multiple data points in each dimension, and a common number such as 3×3×3 is typical. In a 3D image, data points are called voxels rather than pixels. The three dimensions of the 3D aperture function are not required to be the same. This is not a limitation of embodiments of the present invention. Therefore, for the purposes of this specification, since the 2D blocks are referred to as M×N above, the size of the 3D aperture function is L×M×N.

[0366] Fig.32 A layout of a synthesizer on an integrated circuit is described, the layout being configured to perform 3×3×3 convolutions used as a 3D aperture function to process twenty-seven independent data samples. In this 3D aperture function, L, M, and N are all equal to three. As with the example using HDMI frames, the data samples may be monochrome, or may have three color values, or may feature more than three values ​​per sample. In an embodiment of the present invention, operations are always pipelined so that the values ​​of the data samples are input as a stream in a predetermined order. In one protocol, the data values ​​are first presented in the input stream in columns of a data array, then move down to each row, and then continue to each plane in the third dimension. The value at each data point is processed only once, and the partial sum representing the intersection of each 3D aperture position overlapping each data point is calculated and forwarded for further processing, just as when the device is operated in a 2D embodiment.

[0367] The 3×3×3 array of synthesizers 3201 each applies one of the 27 weights for a single output channel to a data point and forwards the partial sum for further processing. The row buffer FIFO 3202 displays the portion of the previous row after appropriate delay, and the plane buffer FIFO 3203 displays the portion of the previous plane after appropriate delay. Once the last weight W is applied 2,2,2 , the summation is complete and the sum is passed to the final processing circuit 3204. Those skilled in the art will appreciate that the subscripts of the weights at each combiner refer to the plane, row, and column in sequence.

[0368] It does not matter whether the input data point contains a single scalar value or multiple values ​​(such as features from a previous neural network node). By extending the parallel weights and sums throughout the circuit, any number of parallel output channels can be generated for the same input source array.

[0369] Special case logic is embedded in the synthesizer 3201 and FIFO 3202 to handle the edge cases of the first and last columns, the first and last rows, and the first and last planes. Those skilled in the art will appreciate that the edge cases in the third dimension are very similar to those in the second dimension, and similar solutions are sufficient.

[0370] It should also be understood that Fig.32 The fully functional IC for the synthesizer array shown will also have input ports, output ports, and control circuitry that operates at least one counter and generates control signals coupled to the synthesizer, delay circuits, and finalization circuits as described above for a system operating on a 2D data array.

[0371] Fig.33 In the illustrated embodiment, data from multiple planes can be buffered and rendered simultaneously, allowing a single compositor to apply weights for multiple planes. This implementation may be preferred when the number of input channels is much lower than the number of output channels, reducing the total size of the required plane buffers at the expense of increasing the required shared batch multipliers.

[0372] For the example of a 3×3×3 convolution, the input voxels are retained in registers 3301 of the three most recently rendered planes via a FIFO 3302 of size equal to RC-1. The data from these registers are merged into a bus 3303 that distributes source input data of the same 2D position on the three planes to all synthesizers 3304.

[0373] Each synthesizer 3304 applies one weight per plane of the 3×3×3 convolution to the corresponding source input for that plane and sums the results. The partial sums are passed to the other synthesizers and delayed appropriately by the FIFO 3305, just as in the 2D implementation. Edge case conditions (including the first and last plane cases) are embedded in the various synthesizers and are used with Fig.32 The fully flipped form shown in Figure 1 matches this very well.

[0374] Please note that using Fig.33The partially flipped form of in breaks the strict order of the decomposition aperture function, since the source values ​​from the top left corners of the three different planes are combined together as the first action of the computation. If the aperture function is a convolution, where all products can be added in any order to obtain the same result, then this technique is equivalent to the fully flipped variant. For any aperture function that relies on a temporal sequence of operations to compute a valid result, must be used instead. Fig.32 A completely flipped variant of .

[0375] Taking the 3×3×3 input array volume P×R×C (the input of each voxel is a scalar value) as an example, Fig.32 The first form in requires only one bulk multiplier for all output channels, but each channel requires two R×C FIFOs to forward the partial sums. In contrast, Fig.33 The alternative embodiment requires three batch multipliers, but only two R×C FIFOs are needed to buffer the raw scalar data. In this case, if the R×C product is large enough that the size of the FIFO is larger than the size of the batch multiplier, the second form may be preferred. However, if the number of input channels is greater than the number of output channels, then the second form may be preferred relative to Fig.33 Example, using Fig.32 The total number of batch multipliers required and the total size of the FIFO are reduced when the aperture function is implemented. The choice between the two embodiments can be made based solely on the number of input and output channels and the total size of the buffers and multipliers. In cases where the order of operations does not matter, the two embodiments are numerically equivalent, and the choice between the two is purely a matter of overall cost and convenience as long as the aperture function is calculated correctly and completely.

[0376] Fig.34 An implementation of a typical 3×3×3 convolution applied to a 4-up input stream is shown. As previously described, the 4-up input stream presents data for four input array locations in parallel. The source input set is presented by register 3402 and then retained by register 3401 so that both sets are presented to all four synthesizer sets 3403 simultaneously in the order in which they were presented. Each synthesizer set sorts the data between rows and planes (FIFO not shown), just as in reference 3404. Fig.32 The 1-up embodiment described above is the same as above, with the main difference being that the columns are processed simultaneously rather than sequentially.

[0377] Each synthesizer set 3403 has unique edge conditions built in, since each synthesizer set may be exposed to a different subset in the horizontal direction. Even though the horizontal operations are done simultaneously, the order of operations can still be the same, so this embodiment is applicable to any aperture function.

[0378] Fig.35shows the same 3×3×3 convolution example applied to the above reference Fig.34 A fully flipped embodiment of the 4-up data flow is described. In this embodiment, each source voxel is processed only once, and a total of four batch multipliers are required per input component to support all output channels in the parallel set.

[0379] Source input data is presented in registers 3501 and distributed to all synthesizer sets. Synthesizer set 3502 is referenced Fig.34 The synthesizer set 3403 described is similar. The synthesizers in set 3504 accept data from P2 and P3 and forward portions to the synthesizers in set 3503, where data from the next source input data set P0 is applied. The synthesizers in set 3506 accept data from P3 and forward portions to the synthesizers in set 3505, where data from the next source input data sets P0 and P1 are applied. The inter-row FIFOs provide standard delay functionality in synthesizer sets 3503 and 3505, but are not included in synthesizer sets 3504 and 3506. The inter-plane FIFOs provide equivalent delay functionality for synthesizers 3502, 3503, and 3505.

[0380] The final output is achieved by weight W 2,2,2 The synthesizer generates, which is described in this paper as the weights W of all four outputs 0,2,2 The furthest synth below.

[0381] Note that synthesizer set 3502 produces its output one input source interval earlier than synthesizer sets 3503 and 3505, and therefore must be delayed to produce all four outputs simultaneously.

[0382] With appropriate restrictions to meet the requirements of a particular aperture function, one skilled in the art will appreciate that any such function that can be decomposed into a sequence of sequential steps can be computed continuously over time by such a circuit. Furthermore, while the number and form of the synthesizer sets are determined by the particular combination of N-up (including 1-up) data presentation and the three-dimensional array of sample sizes for each output, the basic form, timing, and exception rules are common to all combinations.

[0383] Other aperture functions commonly used in deep neural networks (such as MaxPool) can be accommodated in the general form described.

[0384] Synthetic Scaling for Shared Neural Networks

[0385] In yet another aspect of the invention, a system is provided for processing multiple input data sources presented as independent parallel input array streams or as dynamically synthesized scales of input array streams using a single instance of an aperture function implemented as an IC circuit.

[0386] The property of applying an aperture function on a data array with more than one dimension is that the partially completed sub-function value from each column position on each row must be retained and combined with the sub-function value of the corresponding column position on the subsequent row. If the number of input dimensions is greater than 2, corresponding values ​​must also be retained from the first plane to the next plane, from the first volume to the next volume, and so on. In this discussion, the specific data items that need to be retained for reassembly during post-processing are referred to as the context of the position under consideration. In the above description, the context required to process an ordered stream of input values ​​(i.e., the context required when processing row by row), and the circuits for retaining context values ​​in order (such as FIFO circuits and other registers), are described as being part of an IC that also implements the aperture function.

[0387] In the following description of applying an aperture function to multiple independent parallel input array streams or as multiple dynamically synthesized scaled input array streams, the context of the current row of the input stream processed by the aperture function circuit (based on the results retained from the previous row) is synchronously provided to the aperture function circuit by a separate IC circuit.

[0388] Presenting the input data as a continuous stream and breaking the data into rows means that there is a discontinuity at the end of each row, at which point the implementation circuitry must set aside the sub-function value computed for the right edge and restart processing the value for the left edge. When this context switch occurs, it does not matter which row is processed next. The only requirement for correct operation is that the context presented corresponds to the currently presented input value.

[0389] Thus, a single aperture function circuit can successively process rows of data presented at a range of input positions, and the rows can come from different sources, provided that the context of the rows is also presented. Furthermore, the rows do not have to all be of equal width.

[0390] A common technique in computer graphics is to use a set of pre-computed scaled 2D textures to reduce the amount of work and improve the visual quality when applied to 3D models. This is often called the MIPMAP technique, which utilizes a series of sample sets, each with a width and height of 1 / 2 of the previous set. In this form of MIPMAP, each subsequent set is reduced in height and width by a factor of 2, and the area of ​​each subsequent scaled image is reduced by a factor of 4. The sum of the infinite series shown in the figure below is taken to be 4 / 3, while the sum of any truncated finite sequence of this series is always less than this series.

[0391] in

[0392] Therefore, for a single circuit to compute the aperture function on a set of interleaved half-scale (in two dimensions) input arrays, the circuit only needs to operate at a 33% increase in frequency to ensure that complete calculations at all scales are synchronized with the presentation of 1:1 scale data.

[0393] A typical example is processing an HD-compatible input video stream running at 60 frames per second (FPS) and rendering a single RGB pixel sample at 148.5MHz. If 150MHz is sufficient for a 1:1 native scale stream, 200MHz is sufficient for any limited series of half-scaled images at the same input rate.

[0394] When the aperture function is implemented as a CNN node to provide understanding of the image, a 2:1 scale would require the model to learn to recognize every object classification within a 2:1 size range. In cases where using a narrower range is more accurate or otherwise desirable, the model can Learning recognition tasks in a large and small range, and as the area is reduced by a factor of 1 or 2 The required frequency increase is given by

[0395] in

[0396] Therefore, a processing frequency of 300 MHz is sufficient to complete the above example. Calculation of group proportions.

[0397] In a CNN that processes image data, the first layer accepts input in the form of an array of pixels or samples, typically in RGB or grayscale format. Subsequent layers accept output from upstream layers in the form of feature intensities (usually in the range [0.0, 1.0)).

[0398] Multiple sets of scaled image arrays for the first layer can be synthesized in real time as the data is presented. For 2:1 scaling, there are multiple effective sampling schemes that are compatible with the proposed circuit. To achieve smooth scaling, each time a pair of pixels is received, the individual components (e.g., RGB) are summed and retained separately. On each odd row, the sum of the current pair is added to the sum of the previous row, and the final sum is divided by four to fit the original value range. This scheme avoids sampling artifacts and is often preferred for images presented to human viewers. But simply discarding every other row and every other column is also effective. This results in a smaller and simpler circuit, which may be preferred as long as it matches the training scheme of the model. Subsequent layers receive features that have been created in sub-scaled form without modification or manipulation. The circuit operates by interleaving the data stream for each scale in real time at each layer.

[0399] Using a 2:1 scaling ratio as an example, one layout would be to switch the scaling ratio as needed at each context change, i.e. at the end of each row for a given ratio. Initially, full-size pixels are presented to the circuitry implementing the first layer, and at the end of the first row, a context switch is made to continue processing at the left edge of the second row. While the aperture function processes the full-size samples, the scaling circuitry samples the raw data as needed. At the end of the second row, instead of switching the context to the third row at the 1:1 ratio, the context is switched to the first row at the 2:1 ratio, and the scaler now begins subsampling the 2:1 data input to the aperture function in preparation for calculating the 4:1 ratio.

[0400] After processing the first row of the 2:1 scale input array, the context switches back to the third row of the 1:1 scale, and after processing the fourth row of the 1:1 data, the context switches back to the second row of the 2:1 scale, and then the first row of the 4:1 scale is available. After processing the first row of the 4:1 scale, the context switches back to the fifth and sixth rows of the 1:1 scale, then the third row of the 2:1 scale, then the seventh and eighth rows of the 1:1 scale, then the fourth row of the 2:1 scale, then the second row of the 4:1 scale, and the first row of the 8:1 scale.

[0401] The above method can be continued to any desired finite degree of subsampling. A fixed buffer is used to retain incoming samples at a 1:1 ratio while processing lower ratios. The sampling buffer performs this function for all lower ratios. The circuit implementation of the aperture function must run at a high enough frequency to process pending results and pending inputs before any single buffer overflows. The increase indicated by the above formula is the minimum possible; higher operating speeds can be used to further simplify the buffering logic.

[0402] The output of the layout is a single stream of feature rows interleaved in the order described above. Subsequent layers do not have to subsample or buffer the 1:1 feature input; lower-scale features are passed through at the expected times, making the pattern of context switches exactly the same as that produced.

[0403] accomplish Scaling is much more complicated. Each pixel is combined into weighted subsamples to produce each row samples and frames Instead of producing a fixed pattern of scaled input arrays by combining or dropping discrete pixels, the order in which the rows are presented to the aperture function circuit will no longer appear in a simple regular pattern; instead, the rows are processed as each row of sampled data is completed. Subsequent layers must process the data at each sampling scale in exactly the same order. This can be achieved by duplicating the counter logic used to produce the sampling pattern, or by sending signals in parallel with the composite feature stream to indicate when to switch context and to which scale, or by other means.

[0404] Other reasonable or unreasonable sampling patterns can be implemented for subscale ordering in a similar manner. Circuit design can be performed based on any scaling layout or other considerations discovered during CNN training.

[0405] Although the examples described thus far produce a downscaled sample stream, this is not a limitation of the present invention. In some embodiments, the sampling may produce an upscaled stream.

[0406] The permutation of scales does not change the core implementation of the aperture function. The difference between this variant and the previous single-scale implementation described above is that the single context switch from the right edge to the left edge based on the column counter is replaced by an external mechanism that manages multiple contexts at multiple scales. The calculation of the aperture function and the edge rules by which it combines the current input value with the context of the current position remain unchanged, and the increase in overall circuit size is minimal.

[0407] Another application of using context switching is to process interleaved rows from unrelated streams using the same aperture function. As with the scaling example described above, the input rows to the various arrays do not all have to be the same width. All inputs that must be processed late are buffered, and the overall frequency of the aperture function implementation must be increased so that the entire set of input streams can be processed before any single buffer overflows.

[0408] Single-scale and multi-scale processing of different streams can be mixed freely. And, since the full context of any given position at any given scale includes whether that position is in the top or bottom row, different input streams do not need to start and end in sync, or even have the same height.

[0409] Fig.36An aperture function IC circuit 3602 is shown applied to an input array stream 3601 of ordered samples (e.g., image pixels or features) computed by an upstream CNN node. For any column position of the currently presented row, the context of the current row (i.e., multiple values ​​computed for the corresponding column position of the previous row, and exception flags maintained by the context manager, including first column, last column, first row, and last row signals—assuming there is a previous row) is synchronously presented to the aperture function circuit 3602 by a separate context management circuit 3603 and combined with the value computed for the immediately previous column position to produce data for the final output stream 3604 of the aperture function. Partial sub-function values ​​for the current position are also computed and retained by the context management circuit 3603 as context for combination on subsequent rows. In standard sensor presentation order (left to right, top to bottom for a single sample), the values ​​of the individual elements of the aperture sub-function of the previous column are used immediately; in the edge case of processing the first column of each row, these values ​​are either omitted from the calculation or synthesized according to the definition of the aperture function.

[0410] Other non-standards-based presentation orders are also possible, such as sensors that transmit data in staggered rows or in a serpentine fashion (alternating left to right and right to left), which requires context and computational circuitry to allow such presentation orders.

[0411] The partially calculated sub-function values ​​for each column position are retained and become the context for subsequent rows. Both retention and presentation can be combined into the implementation of the IC circuit of the aperture function 3602, but can be separated from the implementation of the aperture function without affecting the size or efficiency of the overall system.

[0412] Whenever processing of a given range of input samples is complete and processing of a different, non-contiguous range must begin, the context of the last processed column must be preserved and set aside, and the context of the first column of the next range must be made accessible. In the standard single-sample left-to-right, top-to-bottom presentation order, this occurs when the column transitions from the rightmost edge to the leftmost edge. For clarity, in this discussion, the potentially arbitrary ranges will be referred to as rows. In this simplest instance, the values ​​preserved as context in the previous row are rendered as context for the next row, in the order they were captured.

[0413] Since no additional data is needed to process the currently rendered row, there is no requirement or dependency in the implementation of the aperture function to process the next row next, so any row may be rendered at any time as long as the context of that row is rendered simultaneously.

[0414] Fig.37A simple example is shown which receives two independent input array streams, a first stream 3701 and a second stream 3702, and alternately presents rows from each stream via a store and forward multiplexer 3703 which retains input from one stream while processing the other. Since values ​​from one stream are never mixed with values ​​from the other stream, the aperture function circuit 3602 is similar to the first example ( Fig.36 ) where the context management is external to the aperture function circuit. Fig.37 In the system, context management is handled by context circuit 3704 which is separate from the aperture function circuit.

[0415] The retained context now contains the values ​​of each stream captured alternately, while the presented context presents the context of each stream alternately. If the dimensions of the two streams are the same, the capture and presentation can be combined into a single FIFO with twice the width and the same number of rows as required by the aperture function, so that the requirement that the context of each row is presented simultaneously with the input array samples of that row is met. The final output 3705 is then presented with the two streams interleaved in time, and any subsequent nodes must accept this time-interleaved form of the input array. Since the input streams are independent, the complexity required to compute the aperture function for a single interleaved row stream will not be applicable to any subsequent aperture nodes.

[0416] The two streams need not have the same dimensions, nor need they begin and end together. Any changes in the array dimensions and the beginning and end of the frame must be accommodated by the capture context and presentation context operable in context circuit 3704, which will take a form sufficient to meet the simultaneity requirement, but with aperture function 3602 unchanged. The streams need not be presented at the same frame or pixel rate, in which case interleaving circuit 3703 will buffer incoming data and deliver completed rows at a processing frequency sufficient to keep up with all incoming streams without losing data.

[0417] This approach of combining two or more independent input array streams can be applied to a variety of other processing situations. Of particular note, the system accepts a single input video stream, synthesizes each frame into a set of sampling ratios, and processes them through the same aperture function.

[0418] Fig.38 A sequence of complete and scaled data rows for a series of four 2:1 reductions is shown. The first unscaled substream 3801 depicts 16 rows of pixel data labeled 1:1, meaning there is no reduction. Fig.37 In the operation of the system shown, rows R0 to R 16(and later) data may be presented to multiplexer 3703 as a first data stream. When presenting the data of substream 3801, the data may be reduced by averaging the column pairs on each pair of rows of substream 3801 (2:1 reduction), etc., to generate a set of 8 rows of half-width data as substream 3802. The area of ​​the R'0 to R'7 data array represented by substream 3802 is one-fourth the area of ​​the data array represented by substream 3801. When creating the scaled data of substream 3802, these data may be presented to multiplexer 3703 as a second parallel data stream. Substream 3803 represents the data from substream 3802 further reduced by a ratio of R"0 to R"3 at a ratio of 2:1, and substream 3804 represents the data from substream 3803 further reduced by a ratio of R"'0 to R"'1 at a ratio of 2:1. Data from substreams 3803 and 3804 are also presented to multiplexer 3703 as separate data streams.

[0419] Fig.39 A sequence of full (substream 3901) and scaled (substream 3902) rows is shown for irrational scaling of . Note that the irregularity exhibited by each row reflects the irrational sampling of the row. Subsequent scales are not irregular in the same way: the next subscale will be a 2:1 reduction of the 1:1 scale, and the next subscale will be 2:1 downscaling. Again, the scaled substreams are dynamically generated and presented as independent data streams to the multiplexer 3703. By directly signaling that a specific subscale is always present, the initial irregular row ordering does not need to be replicated in subsequent nodes.

[0420] Fig.40 The application of full scale and scaled data to an aperture function 3602 is shown, where the data is presented in the form of an array 3601 of full scale samples, typically pixels, such as Fig.36 As shown, these data are reduced for processing by a common aperture function. The output of the aperture function is a time-interleaved stream of outputs at different scales, in the same pattern as the interleaved sub-streams described in detail above.

[0421] A single input array stream 3601 is accepted by a cascaded multi-scale sampler 4001, which presents any and all desired scales of samples as a time-interleaved sequence 4002. The aperture function 3602 need not change; the display of sample rows and their associated context is maintained as desired. Context management circuitry 4003 manages the partial output generated by the aperture function so that context data at various scales is accepted and presented at times corresponding to subsequent rows at the same scale. This can be accomplished by embedding knowledge of the expected sequence (e.g., for the 2:1 case) or by a direct signal from the sampler (not shown), e.g., for the case of any irrational scaling ratio. The final output of the aperture function is presented in the form of an interleaved multi-scale array stream 4004 of function-specific computations, typically detected features.

[0422] Fig.41 The processing of interlaced streams by subsequent CNN nodes is shown, which no longer require any scaling logic. These streams can be any mixture of independent streams and synthesized sub-scaled frames. The context management circuit 4102 implements segmentation context switching based on the established row rendering mode. The interlaced input stream 4101 from the output of the previous node is fed to the aperture function 3602 unchanged, while the context management circuit 4102 manages the partial output of the aperture function by directly embedding the rendering sequence or by accepting signals (not shown) describing the row dimensions and transformations. The final output 4103 is a compatible interlaced stream whose dimensions may be the same as the input stream 4101, or different, and may not be presented in the same order.

[0423] For example, if the aperture function is a 2×2 tiled MaxPool node, the output stream will produce one half-width output row for every two input rows. The order of the stream elements will be the same for the 2:1 scaling case, and different for any irrational case for a particular scaling.

[0424] Fig.42AThe sampling and slicing logic required to produce interleaved streams of 1:1, 2:1, 4:1, and 8:1 scales is shown. The complete input array stream 3601 of scalable values ​​is passed directly to FIFO 4202, which retains any values ​​received while processing other scales. A 2:1 sampler 4201 accepts multiple samples from multiple rows to produce a single sample value, and then forwards the completed sample value to FIFO 4203, which is retained until processing is completed. Another 2:1 sampler 4201 accepts 2:1 samples and forwards completed 4:1 samples to FIFO 4204. Another 2:1 sampler 4201 accepts 4:1 samples and forwards completed 8:1 samples to FIFO 4205. A slicing multiplexer 4206 selects from the available data (ostensibly when each scaled row is available) and forwards the data to the interleaved sample stream 4002 along with any signals required to describe the interleaving pattern (when it is not embedded in the retention logic).

[0425] Fig.42B It shows the generation of 1:1, 2:1, and 4:1 scaled interleaved streams. The complete input array stream 3601 of scalable values ​​is passed directly to the FIFO 4202 which holds any values ​​received while processing other scaling. Sampler 4207 accepts multiple samples from multiple rows to produce a single value, then forwards the completed value to FIFO 4208 where it is held until processed. The actual number of rows and samples used to make up the scaled value will vary depending on the irrational width and height of the sampling region; some input rows and columns are used to produce multiple outputs. 2:1 sampler 4201 accepts 1:1 samples (bypassing sampler 4207) and forwards the completed sample to FIFO 4203. Another 2:1 sampler 4201 accepts 1:1 samples from 4207. Sample and completed The samples are forwarded to FIFO 4209. Another 2:1 sampler 4201 accepts 2:1 samples and forwards the completed 4:1 samples to FIFO 4204. A slicing multiplexer 4210 selects from the available data (ostensibly when each scaled row is available) and forwards the data to the interleaved sample stream 4002 along with any signals needed to describe the interleaving pattern (when it is not embedded in the retention logic).

[0426] Fig.43AThe subsampling and slicing logic required to produce an interleaved stream from the various scaled streams is shown. The complete input array stream 3601 of scalable values ​​is passed directly to FIFO 4202, which holds the received values ​​while processing other scales. U:1 sampler 4301 accepts multiple samples from multiple lines of the 1:1 stream to produce a single value, and then forwards the completed value to FIFO 4304, which is held until processed. V:1 sampler 4302 accepts U:1 samples and forwards completed UV:1 samples to FIFO 4305. W:1 sampler 4303 accepts UV:1 samples and forwards completed UVW:1 samples to FIFO 4306.

[0427] The slicing multiplexer 4307 selects from the available data (ostensibly every scaled row when available) and forwards the data to the interleaved sample stream 4002 along with any signals required to describe the interleaving pattern (when it is not embedded in the retention logic).

[0428] The interleaved data may be in any form or order that is convenient for downstream processing. Completed row ranges may reduce the complexity of the retention logic, but it is also feasible to generate a sequence of scaled samples when available; the only requirement that must be met is that the context of each sample must be captured and presented, which adds additional circuitry to the retention portion (since the result of the immediately previous column is not always available and must be included in the context), but this may be achieved if there are certain advantages for a particular node type. If it is desired to generate output for the full-scale input array before proceeding with additional scaling, it is also feasible to retain the respective input arrays through FIFOs 4304, 4305 and 4306 while replacing FIFO 4202 with an immediate pass-through approach.

[0429] Fig.43B Another embodiment of generating an interlaced multi-scale sample stream is shown. Fig.43B 4308. In the example, W:1 sampler 4303 samples the U:1 scaled stream instead of the V:1 scaled stream, thereby generating a UW:1 scaled stream to FIFO 4309. V:1 sampler 4302 samples the 1:1 stream instead of the U:1 scaled stream, thereby generating a V:1 scaled stream to FIFO 4308. The scaled streams are processed by a slicing multiplexer 4307, thereby generating an interleaved stream 4002.

[0430] Fig.43C In yet another embodiment, generating an interleaved multi-scale sample stream is shown. Fig.43C Shows Fig.43A , but with the addition of an additional 1:T amplifying sampler 4310. Fig.43CIn the embodiment of FIG. 400 , a 1:1 full-scale stream is retained in FIFO 4202, three scaled streams are generated by samplers 4301, 4302, and 4303, a magnified stream is generated by sampler 4310 and retained in FIFO 4311, and all five streams are interleaved by multiplexer 4312, thereby generating a multi-scale sample stream 4002. Those skilled in the art will appreciate that in other embodiments, there may be more magnified streams, as well as multiple sampling of a single magnified scale stream or a scaled stream.

[0431] Implementing CNN models on fixed ASICs with configurable elements

[0432] In yet another aspect of the invention, a system is provided that utilizes a fixed form circuit of multiple replicated pipeline circuit elements that can be configured to compute a CNN on a stream of input arrays. One or more convolution nodes having a specific kernel size can be combined and replicated with a specific number of parallel input and output connections to provide a basis for other combinations of parallel input and output connections and kernel sizes beyond the size that can be directly implemented.

[0433] In one embodiment of the invention, an application specific integrated circuit (ASIC) is provided that includes many copies of the same kernel processing block, arranged in a pattern selected to allow unimpeded forward connections as needed to implement conventional and custom deep neural networks. In other embodiments of the invention, the selection of kernel sizes present in the set of kernel processing blocks is optimized to support specific model forms. Additional auxiliary blocks can be included to perform calculations other than convolutions. Since results are not required to flow upstream, not all blocks need to be accessible to all other blocks. For efficiency, blocks are arranged on the ASIC in groups, which can be linear rows or other forms that facilitate efficient layout.

[0434] Fig.44 An example of an ASIC in an embodiment of the present invention is shown, wherein an ASIC is shown, wherein the kernel processing block includes a pipeline convolution circuit for a common kernel size (3×3 in this example), consisting of a selected parallel input and output connection size, namely 16 each in this example. The main input 4401 of the system is presented as a multi-channel input stream. In some embodiments, the channels are three channels for RGB or YUV video and are provided to the input bus 4402 and provided to the first group block 4405 through multiplexers 4403 and 4404.

[0435] Those skilled in the art will appreciate that the interconnecting buses (shown as single lines) are actually parallel conductors, the number of which is determined at least in part by the desired precision implemented in the system. Where bus branches connect, the connection is represented by an enlarged point. Elsewhere the buses cross, there is no connection between the crossing buses.

[0436] The primary input 4401 may include at least one of a direct camera interface, a DMA interface suitable for access from the CPU bus, a video stream decompression circuit, or other stream interface, with the only restriction being that a parallel channel stream is transferred into the input bus 4402. The input stream need not be a multiple of the nominal 16 parallel connection size, as the unused connections are disabled and ignored in any portion of the circuit that does not use them.

[0437] exist Fig.44 , in this example, the core processing blocks 4405 with the same physical structure are shown as arranged in four rows. In this example, six core processing blocks are shown in the first row (top). This first number of core processing blocks is different from other core processing block groups on the ASIC because the first number of core processing blocks are each coupled to the input bus 4402 through a multiplexer 4403 and process the first layer of the CNN. It should be understood that the number of core processing blocks in the upper row connected to the input bus 4402 through the multiplexer is an arbitrary number. In some minimalist embodiments, one core processing block is sufficient, but in embodiments of the present invention, the goal of the ASIC is to be able to use the ASIC with a wide range of input and output channels and different core sizes.

[0438] The kernel processing blocks in this example are identical with 16 parallel input connections, 16 parallel output connections, and these blocks implement a 3×3 kernel. The interconnect and method are described below, where there can be any number of input and output channels, and the kernel size can be other than 3×3, such as 5×5, 7×7, or 9×9. Adaptability is provided by combining the functionality of multiple 3×3 kernels.

[0439] Each core processing block 4405 in the first row takes its primary input from input bus 4402 in the form of a set of 16 parallel input connections via multiplexer 4403. Those skilled in the art will appreciate that the multiplexers providing input to each core processing block 4405 are identical and therefore not all multiplexers are labeled with component numbers. Fig.44 As shown, each core processing block also accepts input in the same form from the immediately preceding core processing block, which may or may not be physically located to the left. The first core processing block in each row accepts auxiliary input from the output bus of the previous row (except for the first row, as there is no previous row) at multiplexer 4404, which allows the composite core to be split into multiple rows. Each core processing block 4405 provides a primary output to the output bus 4406 and provides the same output to the next adjacent core processing block (except for the last one in the row).

[0440] The input bus 4402 and all output buses 4406 are unidirectional connections with one driver per connection group and provide input to multiple multiplexers 4403, 4407, 4409 and 4412, which then provide input to the core processing blocks and other components. The bus is never used for bidirectional data flow, Fig.44 The description in just means that the output of a row can be used as input for the same row.

[0441] The values ​​on output bus 4406 are time-division multiplexed. When the model is configured so that the values ​​presented on the set of physical connections are the same as the processing frequency of the system, each physical connection carries only a single data channel for each array stream position. When the model is configured to present values ​​on the set of physical connections at some multiple less than the processing frequency, each physical connection can carry multiple time-division multiplexed values ​​for each array stream position, or can hold a single value unchanged over multiple processing cycles. Each form of time-division multiplexing is advantageous for the implementation of model nodes with a large number of input and output channels.

[0442] As an illustrative example, a model where the processing path flows through four 2×2 MaxPool nodes has a data rate that is exactly 256 times lower, and each physical connection can carry 256 data channels. Through the addition of configuration logic and multiple indexes, each core processing block, as well as auxiliary function blocks, has built-in measures to take advantage of this potential layout, allowing each block to adapt to the configuration presented. The end result is that the set of physical blocks required to implement large models is reduced by multiple orders of magnitude. Fig.54 To fully describe the use of configurable time multiplexing.

[0443] In this example, dual multiplexer 4407 provides two 16 parallel input connections to auxiliary function block 4408, equivalent to two 4403 multiplexers. Each auxiliary function block 4408 provides 16 parallel connected outputs to a set of 16 connections on output bus 4406.

[0444] Multiplexer 4409 provides one or more sets of 16 parallel output connections (3 shown) to external functional circuits 4410, which may or may not be located on the same ASIC chip or IC package. One or more sets of 16 parallel input connections are returned from external functional circuits 4410 as different bus connections and are available on output bus 4406.

[0445] The output bus 4406 of each row of core processing blocks provides input for the next row, replacing the function of the input bus 4402 providing input for the first row, and is combined with the output bus of the row through multiplexers 4403, 4404 and 4407 for use by the core processing blocks of the row. It is also connected to the external function interface of the row through multiplexer 4409.

[0446] Because the inherent nature of the time pipeline that processes data flow is that data does not flow upstream, not all possible connections are necessarily available. If the allocation of each specific block is done by allocating usage in order from left to right, the unavailable connections for each bus 4406 can be omitted, thereby reducing the overall circuit size without affecting functionality. The output of each core processing block is then available to the entire row of core processing blocks below it and any core processing blocks to the right in the same row that produce outputs.

[0447] Multiplexer 4412 selects one or more groups of 16 parallel input connections and provides them to the main output circuit 4413, which may include final output processing (typically SoftMax) and can output the results through direct parallel connections, standard SERDES (serializer / deserializer), DMA (direct memory access), or any other interface required to obtain known model results.

[0448] Fig.45 Depicted is a layout for combining kernel processing blocks that handle 16 parallel input and output connections of a fixed 3×3 kernel to configure a 3×3 kernel with more inputs, outputs, or both. The first two kernel processing blocks 4501 and 4502 in the upper left corner are configured to take the same set of 16 input connections (selected by a multiplexer not shown) to produce 32 output channels on two different sets of 16 parallel output connections. The next two kernel processing blocks 4503 and 4504 are configured to calculate 16 output channels from different weights applied to two different sets of 16 input channels by passing partial sums from the first kernel processing block 4503 to the second kernel processing block 4504, where the partial sums are combined with the remaining partial sums to produce outputs on a single set of parallel output connections. The last four blocks 4505, 4506, 4507 and 4508 are configured as two sets of partial sum pairs that take inputs from two sets of 16 parallel input connections to produce 3×3 convolutions with a total of 32 inputs and 32 outputs.

[0449] Fig.45The combination of 3×3 kernel processing blocks in is actually achieved by the operation of the multiplexer connecting the kernel processing blocks in each layout to the bus. Those skilled in the art will understand that by combining arbitrarily large sets of kernel processing blocks together, the 3×3 convolution pipeline nodes can be configured to have any number of input and output channels. Any node whose input is not an integer multiple of the data channel set size (nominally 16) will simply ignore the unused channels, which can be achieved by setting their weights to zero or disabling them in the calculation. Any node whose output is not an integer multiple of the channel set size can avoid calculating values ​​for unused channels, which will be ignored by downstream nodes.

[0450] The choice of the parallel connection set size (16 in this example) is arbitrary, and the only effect of the chosen granularity is the fraction of unused connections in the full configuration for a given model. There is no requirement, nor expected advantage, for set sizes to be constrained to powers of 2. Some greater physical layout efficiency can be achieved by making all connections the same width, but this is not required, and in some cases irregular or varying widths of connection size may be optimal.

[0451] Fig.46 The internal structure of the core processing block 4405 is shown. Fig.44 ) The selected parallel input connection set 4601 is split into different individual connections, 16 in this example, and each individual connection is provided to a batch multiplier 4603, which calculates the entire set of multiples possible for that input. All multiples of all inputs and the individual connection set 4602 from the auxiliary connections are provided to each convolution unit 4604. The resulting outputs of each convolution unit are grouped into a set of 16 parallel output connections 4605 and provided to other blocks on the output bus 4406 (not shown, see Fig.44 ).

[0452] Fig.47 The internal structure of each convolution unit 4604 is shown. An internal bus 4703 provides a complete set of input products 4701 to each of the nine weight sum units 4704 used in this 3×3 example. As with previously disclosed versions of pipeline circuits for convolution, a pair of array stream width FIFOs 4705 delay the partial product sums to meet the requirement of a single circuit to process data in the form of 3×3 blocks presented by the input stream. In this example, the final sum of each convolution is combined with a bias value in adder 4707 and an optional value received from auxiliary input 4702 after a delay in FIFO 4706. Once the final sum is combined with the bias and optional auxiliary inputs, the sum is output 4708 to the containing block, where it will form an element of the parallel output connection set.

[0453] All FIFOs 4705 and 4706 delay values ​​by input stream position rather than by clock cycles and pass time-division multiplexed data in a given manner. Fig.45 In the example described in , FIFO 4706 can be configured to pass data without delay. FIFO4 706 is used to support the construction of complex cores, such as Fig.51 As shown below.

[0454] Fig.48 A single summing unit 4704 is shown which applies each presented input channel multiple 4801 and selects a particular multiple via a configuration register 4802 and a multiplexer 4803. The selected multiple then passes through a variable shift register 4805 controlled by the configuration register 4804 and is then forwarded to an adder 4806 where the selected multiple is combined with all other selected and scaled multiples for the respective input channels.

[0455] The combination of configurable index 4802 and configurable scale 4804 allows floating point weight values ​​to be applied to fixed point features or pixel values. One weight for each input channel is added in combination with the configured bias value 4807 to complete the calculation of the convolution over time step. The summing unit 4704 can also add the delayed value 4808 from the previous row through FIFO 4705 and add the partial sum 4809 forwarded from the left unit. When both are present, they are added by adder 4810. In the cells of the first row, there is no row above, so the delayed connection 4808 will not be present, adder 4810 will be omitted, and the forwarded value 4809 will be directly connected to adder 4811, where it is added with the bias value 4807. In the cells of the first column, there is no column to the left, and the forwarded value does not exist, and adder 4810 is omitted again. In the single upper left cell, there are neither delay connections nor forward connections, adders 4810 and 4811 are both omitted, and bias value 4807 is routed directly to the final adder 4812 where it is combined with the sum of the adder tree to form output 4813.

[0456] The present invention does not require the use of bulk multipliers. If the combination of large bus widths and large multiplexers is burdensome, the actual input values ​​may be distributed (over smaller buses) and local two-input fixed multipliers used, where the actual values ​​to be multiplied are configured rather than indices. Shifters and scale registers are retained to apply full floating point weights from fixed point products.

[0457] If the chosen system precision is very large, such as 12 bits of mantissa precision, the bulk multiplier and transmission bus may be too large. In this case, a smaller bulk multiplier and correspondingly smaller bus can be used, which is then indexed multiple times and summed to produce each required product. In this case, a 6-bit bulk multiplier can be used, with two index values ​​and two (much smaller) multiplexers to select two independent multiples, which are then combined with appropriate scaling to produce the same result as a 12-bit bulk multiplier.

[0458] The operation of the entire system is not affected by the choice of multiplication method, only the path from the input 4601 of the block to the scaler 4805 of each summing unit is affected. The choice of multiplication method and the configuration of the bus width of the bulk multiplier (if used) can be postponed until ASIC layout to determine the best form in terms of size and power consumption.

[0459] Fig.49 The internal structure of auxiliary function block 4408 is shown. In this example, the primary parallel input connection set 4901 selected by dual multiplexer 4407 is routed through variable FIFO 4902, delayed by configuration, and optionally combined by adder 4904 with auxiliary input 4903 also from multiplexer 4407. If the sum is not required in the current configuration, multiplexer 4905 can feed the delayed input values ​​directly into common lookup table 4906.

[0460] The common lookup table 4906 is a set of static registers configured with a mapping of every possible index value to some arbitrary output value. Each input channel controls a multiplexer going into the output of this set of registers so that the same function is applied to all input channels individually. This allows any activation function required by the model to be implemented, such as but not limited to RELU, sigmoid, or tanh, without having to include various circuits to calculate each such function.

[0461] If the lookup table is not used in the configuration of this block, the input can bypass the lookup table through multiplexer 4907 and the lookup 4906 can be disabled to save power.

[0462] A configurable multiplexer 4914 can select only one of the MaxPool 4908 , Average 4909 , Sample 4910 , Expand 4911 , and bypass or channel cascade functions for all channels and form an output 4915 that drives a set of parallel output connections on the output bus 4406 .

[0463] MaxPool function block 4908 provides calculation of a maximum value over configurable blocks of the input stream over time, and directly implements a stride mechanism that has the effect of reducing the downstream data rate.

[0464] The Average function block 4909 provides calculation of the average value of a configurable block and may share circuitry with the MaxPool 4908. The Sample function block 4910 provides clipping of the input array and skipping strides and may also share circuitry with the MaxPool 4908 and Average 4909 blocks.

[0465] Each functional block that implements a non-single step mechanism effectively reduces the dimensionality of the input array stream. Since the frame rate of the entire stream is constant, this proportionally increases the time allowed by downstream nodes to process each input stream position. This provides the basis for time-division multiplexing of the bus connections and dictates the extent to which time-division multiplexing can be used. The core processing blocks can take advantage of the increased time intervals to transmit and process multiple channels of data through a fixed set of parallel connections. Weights are expanded to accommodate additional values, but multipliers and accumulators are reused, allowing the input and / or output channels of a node to be expanded without the use of multiple core processing blocks. Control logic coordinates the distribution of values ​​over time between connected blocks.

[0466] The expansion function block 4911 provides padding of the input array and copies values ​​to align the reduced input array stream with the non-reduced input array stream.

[0467] A dedicated multiplexer 4912 connects the channels obtained from the primary parallel input connection 4901 with the channels obtained from the auxiliary parallel input connection 4903 to effectively reroute the channels to specific parallel output connections. In this example, each block uses a fixed 16 input channels, and multiplexer 4912 can obtain the 1 to 15 lowest channels from input 4901 and the corresponding 15 to 1 lowest channels from input 4903 to produce a set of 16 channels for selection by multiplexer 4914. Since the assignment of data channels to specific positions in the kernel is arbitrary (any input data channel and corresponding weight can occupy any position in the kernel without changing the result), this layout is sufficient to perform all the rerouting required for the CNN model to be implemented when all used channels are placed in the lowest available position of each set of 16 parallel connections. The use of this multiplexer is limited to data channels with equivalent time multiplexing.

[0468] Another dedicated multiplexer 4913 cascades two input channels into one output channel by alternately sampling samples from each parallel input connection set. Using this circuit requires retaining each input value on both channels for two or more stream processing cycles. The output value will be retained for half the time of the input value and only occupies a single parallel output connection set on the bus.

[0469] If a block function or cascade is not selected by configurable multiplexer 4914, the block will operate in bypass mode, which may be helpful in providing a delay path between a parallel set of input connections and a core processing block to be used in a subsequent row.

[0470] Fig.50 The elements required to implement any block function contained in an auxiliary block are shown. A single input channel 5001 is fed into the block function circuit 5002, along with a delayed value from the previous row, and output 5004 as the final value for each block. This allows the calculation of any function that uniformly processes all input stream positions.

[0471] For MaxPool 4908, when a block function is defined to span multiple rows, the function selects the maximum value seen and uses FIFO 5003 to associate the value of the block from the previous row with the current row. For Average 4909, the function accumulates the sum of all values ​​within each block and FIFO 5003 is used to present the sum of the previous row. For Sample 4910, where a selected position within each block is to be output as the value of the entire block, FIFO 5003 is used to retain the selected value of the previous row when the selected value does not fall on the last row of the block being processed. Each of these block functions generates only a single value for each block, and the net effect is to reduce the area of ​​the input stream array passed downstream and the throughput required to maintain stream processing.

[0472] For Expand 4911, the value of each individual position of the input stream is repeated one or more times to the output stream, and a FIFO 5003 is used to retain previously encountered values ​​for repetition again in subsequent rows when the function is defined to produce multiple rows. This function has the effect of increasing the area of ​​the input stream array passed downstream and maintaining the throughput required for stream processing, but requires matching array sizes when different paths through the model have different sizes but need to be reassembled.

[0473] Fig.51The connection layout to the external circuit is shown. Input 5101 represents a set of external parallel input connections, which can be parallel buffers, SERDES or other forms that are convenient for connection to other circuits, and provides a set of parallel output connections to the output bus 4406. The configurable multiplexer 5102 selects a set of parallel input connections from the bus 4406 and passes it to the external output 5103, which can be any output form. It is expected that at least some of the external inputs and outputs are compatible with each other so that they can be used to connect multiple units of the ASIC to form models that exceed the existing block capacity.

[0474] An important use of external circuitry is to provide arbitrary functionality not contained in auxiliary function block 4408. This feature effectively makes the ASIC "future-proof" and avoids the requirement to anticipate future developments in model design.

[0475] Fig.52 The layout of four 3×3 kernel processing blocks 4405 for computing a 5×5 convolution is shown. All elements not participating in this configuration are omitted from this figure and are disabled by configuration.

[0476] The kernel processing block at the upper left position is configured to perform a 3×3 convolution calculation on the upper left 3×3 values ​​of the 5×5 block of the input stream, while the kernel processing block at the upper right position is configured to perform a 2×3 convolution calculation on the upper right 2×3 values ​​of the 5×5 block.

[0477] The lower left kernel processing block is configured to calculate the lower left 3×2 values ​​for 3×2 convolution calculation, and the lower right kernel processing block is configured to calculate the lower right 2×2 values ​​of the 5×5 block for 2×2 convolution calculation.

[0478] Since the blocks in the left column will produce final results at a time corresponding to two input stream positions before the blocks in the right column produce final results, both FIFOs 4706 are configured for a two position delay.

[0479] The output of upstream adder 4707 is selected from output bus 4406 and routed as the primary input to FIFO 4902. Since the upstream block will complete the computation of the upper 5×3 portion of the 5×5 convolution a time corresponding to two array widths before the downstream block completes, FIFO 4902 must be configured to delay the partial sum by an array width of two input stream positions.

[0480] The output of downstream adder 4707 is selected from output bus 4406 and routed as an auxiliary input to adder 4904, where it is combined with the upstream delayed partial sum to produce the final result of the 5×5 convolution. The final result then passes through lookup table 4906, where an activation function is applied.

[0481] Fig.53 The layout of nine 3×3 core processing blocks arranged in a similar manner to implement a 7×7 core is shown. Since the timing differences between the core processing blocks are the same as the previous example, the FIFOs are configured for the same latency. The auxiliary adder 4904 on the left combines the partial results of the first two rows together, and the adder 4904 on the right combines these values ​​with the partial results of the next row and forwards it to the lookup table as the final result.

[0482] Those skilled in the art will appreciate that any arbitrary convolution kernel size can be composed of any kernel processing block kernel size using a similar arrangement of partially disabled elements and appropriate delays. Fig.45 The layout of the blocks shown in can be combined to allow configuration of any kernel with an arbitrary number of input and output channels.

[0483] The multiplexers 4404 included in each row of blocks are specifically designed to provide large convolutions that can span multiple groups of blocks arranged within the ASIC. Although it is foreseeable that in some cases, regular rows of repeating kernel processing blocks may be optimal, the operation of the present invention is by no means limited to regular arrangements. In designing an implementation ASIC of the present invention, any mixture of kernel processing block sizes and block groups may be selected to reflect the needs of a particular model, or a statistically determined mixture may be selected to cover the widest range of models within a single ASIC chip. If a model requires more resources than can be accommodated by the blocks of a single ASIC, multiple ASICs may be linked together to further expand the ability to cover arbitrarily large models.

[0484] Fig.54 An abstract example of a fully functional convolutional neural network is shown. In the exemplary embodiment herein, input 5401 provides RGB pixel data for 3 of the 16 parallel connections, and the other 13 parallel connections are unused.

[0485] Convolution 5402 is a 3-input, 32-output, 7×7 convolution with a ReLU activation function, implemented in this example by two sets of 9 3×3 kernel processing blocks and two auxiliary blocks to produce a total of 32 output data channels. The RELU activation function is implemented by a lookup table of the final auxiliary block for each set of 16 parallel output connections.

[0486] Sample 5403 is a 2×2 subsampling of the convolution output, a typical arrangement in ResNet and YOLO model variants. The Sample function can be selected in each final auxiliary block included in the 7×7 convolution, so no additional blocks are required. The output of the Sample function generates an array stream with a quarter of the original area, and the processing rate of each position is one quarter of the original. Each output value remains unchanged for four stream processing cycles before the next value is presented to the parallel output connection. This provides an opportunity for four time-division multiplexed data channels in downstream nodes.

[0487] Convolution 5404 is a 32-input, 64-output, 5×5 convolution with a ReLU activation function. Since the 32 input channels are presented as two sets of 16 parallel input connections to the multiplexer, two sets of four 3×3 kernel processing blocks are combined to apply the weights of each individual output channel to the input. But since the input is presented at one-quarter the stream processing rate, four sets of output channel values ​​can be calculated and placed on a set of 16 parallel output connections, each carrying time-division multiplexed data.

[0488] The MaxPool 5405 is a 64-input, 64-output, 2×2 maximum function configured to accept and process four time-division multiplexed values ​​on each of the 16 parallel input connections. The output is the maximum value of each individual channel found within a block that is two positions wide and two positions high. The output is placed on a single set of 16 parallel output connections, and each value remains constant for four stream processing cycles before the next value is presented. Only a single auxiliary block is required to process all 64 channels of time-division multiplexed data. This provides the opportunity for 16 channels of time-division multiplexed data in downstream nodes.

[0489] Convolution 5406 is a 64-input, 128-output, 3×3 convolution with a sigmoid activation function that accepts 64 time-division multiplexed inputs through a set of 16 parallel input connections and produces 128 time-division multiplexed output values ​​through a set of 16 parallel output connections. Each output value is retained for two stream processing cycles instead of four stream processing cycles for the input. An auxiliary block with appropriate values ​​loaded into a lookup table implements the sigmoid activation function on the data path with equivalent timing.

[0490] The MaxPool 5407 is a 128-input, 128-output, 2×2 maximum function configured to accept and process eight time-division multiplexed values ​​on each of the 16 parallel input connections. The output is the maximum value of each individual channel found within a block that is two positions wide and two positions high. The output is placed on a single set of 16 parallel output connections, and each value remains constant for eight stream processing cycles before the next value is presented. Only a single auxiliary block is required to process all 128 channels of time-division multiplexed data. This provides the opportunity for sixty-four channels of time-division multiplexed data in downstream nodes.

[0491] Convolution 5408 is a 128-input, 256-output, 3×3 convolution with an arbitrary activation function that accepts 128 time-multiplexed inputs through a set of 16 parallel input connections and produces 256 time-multiplexed output values ​​through a set of 16 parallel output connections. Each output value is retained for four stream processing cycles instead of eight stream processing cycles for the input. Auxiliary blocks with appropriate values ​​loaded into a lookup table implement the arbitrary activation function with equivalent timing on the data path.

[0492] Expand 5409 is a 256-input, 256-output, 2×2 Expand with each input value replicated twice in each of the two rows. The 256 time-multiplexed output channels are placed on a set of 16 parallel output connections, but each replicated value remains constant for only one stream processing cycle, rather than four stream processing cycles for the input. This function requires making the output array size compatible with the output of convolution 5410 described below so that the cascading of multiple channels in equivalent positions can be achieved as defined by the model. This reduces the opportunity for 16 time-multiplexed data channels in downstream nodes.

[0493] Convolution 5410 is a 128-input, 256-output, 3×3 convolution that accepts 128 time-multiplexed inputs through a set of 16 parallel input connections and produces 256 time-multiplexed output values ​​through a set of 16 parallel output connections. Each output value is retained for only one stream processing cycle, rather than two stream processing cycles for the input. Auxiliary blocks with appropriate values ​​loaded into the lookup table implement the activation function with equivalent timing on the data path.

[0494] The Cascade 5411 has two different 256 input channel streams with compatible timing and array stream positions, so they can be cascaded into a set of 512 output channels. Since each value on the two 16 parallel connections only holds one stream processing cycle, no further multiplexing is required or feasible. Therefore, the cascade function is expressed in the circuit simply by arranging the time-multiplexed data on the bus connections, and no auxiliary blocks are required in this case.

[0495] MaxPool 5412 is a 512-input, 512-output, 2×2 maximum function configured to accept and process 16 time-division multiplexed values ​​on two sets of 16 parallel input connections. Since the incoming streams are presented on two different sets of parallel input connections, the outputs are initially placed on two sets of 16 parallel output connections, and each value is held for four stream processing cycles before the next value is presented. The two sets of 16 parallel output connections are then routed to another auxiliary block, where they pass through a time-division multiplexer section to produce 512 data channels carried by a single 16 parallel output connection, with each data value held for two stream processing cycles instead of four.

[0496] Convolution 5413 is a 512-input, 1024-output, 1×1 convolution that accepts 512 time-division multiplexed inputs through a set of 16 parallel input connections and produces 1024 time-division multiplexed output values ​​through a set of 16 parallel output connections. Each output value is retained for only one stream processing cycle, rather than the two passed in. The 1×1 convolution is implemented by utilizing only the bottom right output summing unit of the nine units of the 3×3 kernel processing block.

[0497] The Average 5414 accepts 1024 channels of time-division multiplexed data through a set of 16 parallel input connections and is configured to calculate the average of the input values ​​over a block size equal to the entire reduced array stream.

[0498] The output 5415 of the model is transmitted over 16 external connections in the form of 1024 time-division multiplexed values, indicating the presence or absence of 1024 different categories.

[0499] This model form is not intended to be a reliable example, but rather to exploit many of the properties of the invention and to illustrate how the circuit works in practice.

[0500] Those skilled in the art will appreciate that the embodiments shown in the drawings and described above are exemplary and do not describe in detail every possible form that the present invention may take. Within the scope of the present invention, there may be various other possible forms.

[0501] The scope of the present invention is limited only by the claims.

Claims

1. An application specific integrated circuit (ASIC) for computing a convolutional neural network (CNN), comprising: a first input bus that receives an ordered stream of values ​​from an array, each position in the array having one or more data channels; a first ordered set, from first to last, comprising kernel processing blocks having a fixed number of parallel input connections and a fixed number of parallel output connections, each kernel processing block in the first ordered set being coupled to an input bus via a configurable multiplexer in a first set of configurable multiplexers, the kernel processing blocks being adapted to compute convolutions of a common kernel size and to pass the computed values ​​to a first output bus connected as an input back to each configurable multiplexer in the first set of configurable multiplexers and to an adjacent downstream kernel processing block in the first ordered set; a second ordered set, from first to last, comprising kernel processing blocks having a fixed number of parallel input connections and a fixed number of parallel output connections, each kernel processing block of the second ordered set being coupled to the first output bus via a configurable multiplexer of a second set of configurable multiplexers, the kernel processing blocks of the second ordered set being adapted to compute convolutions of a common kernel size and pass the computed values ​​to a second output bus connected as an input back to each configurable multiplexer of the second set of configurable multiplexers and to an adjacent downstream kernel processing block of the second ordered set; as well as A third ordered set, from first to last, comprises core processing blocks having a fixed number of parallel input connections and a fixed number of parallel output connections, each core processing block of the third ordered set being coupled to the second output bus via a configurable multiplexer in a third group of configurable multiplexers, the core processing blocks of the third ordered set being adapted to compute convolutions of a common kernel size and pass the computed values ​​to a third output bus, the third output bus being connected back as input to each configurable multiplexer in the third group of configurable multiplexers and to adjacent downstream core processing blocks of the third ordered set, the third output bus being further connected to a primary output circuit via a single primary output multiplexer, the primary output circuit being adapted to perform primary output processing and provide a final output.

2. The ASIC of claim 1 further comprising an additional configurable multiplexer coupled to the first output bus to provide a selected value to the first core processing block in the second ordered set, and further comprising an additional configurable multiplexer coupled to the second output bus to provide a selected value to the first core processing block in the third ordered set.

3. The ASIC of claim 1, further comprising one or more auxiliary function blocks to provide functions different from the functions of the core processing blocks.

4. The ASIC of claim 3, wherein the one or more auxiliary function blocks accept inputs from a dual multiplexer connected to the input bus and the first output bus and provide outputs to the first output bus.

5. The ASIC of claim 3, wherein the one or more auxiliary function blocks accept inputs from a dual multiplexer connected to the first output bus and the second output bus and provide outputs to the second output bus.

6. The ASIC of claim 3, wherein the one or more auxiliary function blocks accept inputs from a dual multiplexer connected to the second output bus and the third output bus and provide outputs to the third output bus.

7. The ASIC of claim 1, further comprising an external function circuit that selects input from the first output bus through a configurable multiplexer and provides output to the first output bus.

8. The ASIC of claim 7, further comprising an external function circuit that selects input from the second output bus through the configurable multiplexer and provides output to the second output bus.

9. The ASIC of claim 8, further comprising an external function circuit that selects input from the third output bus through the configurable multiplexer and provides output to the third output bus.

10. The ASIC of claim 1, wherein the common core size is a 3x3 core.

11. The ASIC of claim 1, wherein the fixed number of parallel input connections is 16 and the fixed number of parallel output connections is 16.

12. The ASIC of claim 1 , wherein the ordered stream of values ​​is provided by one of direct camera output of RGB values, a DMA interface suitable for access from a CPU bus, or a video stream decompression circuit to produce three parallel channels of red, green, and blue (RGB) values ​​for an image.

13. The ASIC of claim 11, comprising combining the operations of two or more core processing blocks that process a 3×3 core with 16 parallel input and output connections to configure a 3×3 core with greater than 16 inputs or greater than 16 outputs or greater than 16 inputs and outputs.

14. The ASIC of claim 10 adapted by additional circuitry to compute convolutions with kernels larger than 3x3 by combining operations of multiple 3x3 kernel processing blocks.

15. The ASIC of claim 14, wherein the ASIC is adapted to compute a 5x5 convolution, a 7x7 convolution, or a 9x9 convolution.

16. An ASIC as claimed in claim 1, wherein the core processing block presents each input channel value to a batch multiplier, which calculates the complete set of possible multiples from the input and provides the multiples along with the individual channel values ​​from an auxiliary set of parallel connections to a single convolution unit, each convolution unit producing a final output consisting of a set of 16 parallel output connections and provided to other core processing blocks on an output bus.

17. The ASIC of claim 1, wherein the core processing block input values ​​are processed by local two-input fixed multipliers.

18. The ASIC of claim 3, wherein one of the auxiliary function blocks receives parallel input connections from two separate buses through a dual multiplexer and outputs one of the MaxPool, Average, Sample, and Expand functions selected by an output multiplexer.

19. An ASIC according to claim 18, wherein the auxiliary function block sums the values ​​from the parallel input connections in a single channel and multiplexes the summed value into a lookup table, wherein the lookup table is adapted to provide any activation function that can be represented in tabular form, including RELU, sigmoid or tanh activation functions.

20. The ASIC of claim 18, wherein input channels received from different parallel input connections are inputs to a first dedicated multiplexer, the first dedicated multiplexer cascades the parallel input connections to effectively reroute the data channels to specific parallel output connections, and provides the cascaded outputs to an output multiplexer as selection candidates for auxiliary function block outputs.

21. The ASIC of claim 18, wherein the input channels received from different parallel input connections are inputs to a second dedicated multiplexer, which cascades the two parallel input connections into one parallel output connection by alternately sampling from each connection and provides the result to the output multiplexer as a selection candidate for the auxiliary function block output.

Images