processor

Abstract

A processor is disclosed. The processor includes a register, a non-zero weight value selector, and a multiplier. The register stores a first set of weight values ​​and a second set of weight values. Each set of weight values ​​includes at least one weight value, and each weight value in the first set of weight values ​​corresponds to a weight value in the second set of weight values. The non-zero weight value selector selects a non-zero weight value from the weight values ​​in the first set of weight values ​​or the non-zero weight values ​​in the second set of weight values ​​that correspond to the weight values ​​in the first set of weight values. The multiplier multiplies the selected non-zero weight value by an activation value corresponding to the selected non-zero weight value to form an output product value.

Description

[0001] This application is a continuation-in-part of U.S. Patent Application No. 16 / 446,610, filed on June 19, 2019, entitled “Neural Processor,” which claims priority to and the benefit of: (i) U.S. Provisional Application No. 62 / 689,008, filed on June 22, 2018, entitled “Single-Pass Neural Processor Accelerator Architecture,” (ii) U.S. Provisional Application No. 62 / 798,297, filed on January 29, 2019, entitled “Single-Pass NPU,” (iii) U.S. Provisional Application No. 62 / 798,297, filed on May 1, 2019, entitled “Mixed Precision with Depthwise Convolutions,” No. 62 / 841,590, filed on May 1, 2019, entitled “Mixed Precision Neural Processing Unit Block,” (iv) U.S. Provisional Application No. 62 / 841,606, filed on May 1, 2019, entitled “Mixed Precision Neural Processing Unit Block,” (v) U.S. Provisional Application No. 62 / 835,496, filed on April 17, 2019, entitled “Hardware Channel-Parallel Data Compression / Decompression,” and (vi) U.S. Provisional Application No. 62 / 841,819, filed on May 1, 2019, entitled “Mixed Precision Compression,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] One or more aspects of embodiments according to the present disclosure relate to processing circuits, and more particularly, to processing circuits for performing a combination of multiplication and addition. Background Art

[0003] In operation, neural networks may perform tensor operations (e.g., tensor multiplications and convolutions) involving a large number of multiplications and additions. If these operations were performed by a general-purpose central processing unit (CPU) or even a graphics processing unit (GPU), which may be better suited for such tasks, these operations would be relatively slow and incur a relatively high energy cost per operation. Particularly in small devices (e.g., mobile, handheld devices), which may have a strictly constrained power budget, the power consumption associated with the use of a general-purpose CPU or a graphics processing unit (GPU) can be a significant disadvantage.

[0004] Therefore, there is a need for an improved processing circuit for neural network computations. Summary of the Invention

[0005] According to some embodiments of the present disclosure, a processor is provided, comprising: a first block, a second block, a memory and a bus, wherein the bus is connected to the memory, the first block and the second block, the first block comprising: a first weight register, a second weight register, an activation buffer, a first multiplier and a second multiplier, the first block being configured to perform convolution of an activated array with a kernel of weights, the steps of performing convolution sequentially comprising: forming a tensor product of the kernel and a first subarray of the activated array; forming a tensor product of the kernel and a second subarray of the activated array, the second subarray being offset from the first subarray by n array elements in a first direction, where n is a positive integer; and forming a tensor product of the kernel and a third subarray of the activated array, the third subarray being offset from the second subarray by one array element in a second direction perpendicular to the first direction.

[0006] In some embodiments, the step of performing convolution further includes, in sequence: after forming the tensor product of the kernel and the third subarray, forming the tensor product of the kernel and a fourth subarray of the activated array, the fourth subarray being offset from the third subarray by m array elements in a third direction opposite to the first direction, where m is a positive integer; and forming the tensor product of the kernel and a fifth subarray of the activated array, the fifth subarray being offset from the fourth subarray in the second direction by one array element.

[0007] In some embodiments, m is equal to n.

[0008] In some embodiments, n is equal to 1.

[0009] In some embodiments, the step of performing the convolution further includes, in sequence, forming, after forming the product of the kernel and the first subarray, n-1 products of the kernel and n-1 corresponding subarrays of the activated array, the subarray in the kth of the n-1 products being offset from the first subarray in the first direction by k+1 array elements.

[0010] In some embodiments, the processor further comprises a cache connected to the activation buffer and configured to supply activations to the activation buffer, the cache having a size sufficient to store H+(H+n)×(W-1)-1 activations, where H is the size of the core in the first direction and W is the size of the core in the second direction.

[0011] In some embodiments, the activation buffer is configured to include: a first queue, connected to a first multiplier, and a second queue, connected to the second multiplier, the first queue including a first register and a second register adjacent to the first register, the first register being an output register of the first queue, and the first block is further configured to: in a first state, multiply the first weight by the activation from the output register of the first queue in the first multiplier, and in a second state, multiply the first weight by the activation from the second register of the first queue in the first multiplier.

[0012] In some embodiments, in the second state, the output register of the first queue contains zero.

[0013] In some embodiments, the processor further includes: a first adder configured to be connected to the output of the first multiplier and the output of the second multiplier in a first state and to add a product received from the output of the first multiplier and a product received from the output of the second multiplier.

[0014] In some embodiments, the processor further comprises: a second adder configured to be connected to the output of the first multiplier in a second state.

[0015] According to some embodiments of the present disclosure, a method for performing calculations using a processing circuit is provided, wherein the processing circuit includes: a first block, a second block, a memory, and a bus, the bus being connected to the memory, the first block, and the second block, the first block including: a first weight register, a second weight register, an activation buffer, a first multiplier, and a second multiplier, the method including: performing convolution of an activated array with a kernel of weights, the steps of performing convolution sequentially including: forming a tensor product of the kernel with a first subarray of the activated array; forming a tensor product of the kernel with a second subarray of the activated array, the second subarray being offset from the first subarray by n array elements in a first direction, where n is a positive integer; and forming a tensor product of the kernel with a third subarray of the activated array, the third subarray being offset from the second subarray by one array element in a second direction perpendicular to the first direction.

[0016] In some embodiments, the step of performing convolution further includes, in sequence: after forming the tensor product of the kernel and the third subarray, forming the tensor product of the kernel and a fourth subarray of the activated array, the fourth subarray being offset from the third subarray by m array elements in a third direction opposite to the first direction, where m is a positive integer; and forming the tensor product of the kernel and a fifth subarray of the activated array, the fifth subarray being offset from the fourth subarray in the second direction by one array element.

[0017] In some embodiments, m is equal to n.

[0018] In some embodiments, n is equal to 1.

[0019] In some embodiments, the step of performing the convolution further includes, in sequence, forming, after forming the product of the kernel and the first subarray, n-1 products of the kernel and n-1 corresponding subarrays of the activated array, the subarray in the kth of the n-1 products being offset from the first subarray in the first direction by k+1 array elements.

[0020] In some embodiments, the processing circuit further comprises a cache connected to the activation buffer and configured to supply activations to the activation buffer, the cache having a size sufficient to store H+(H+n)×(W-1)-1 activations, where H is the size of the core in the first direction and W is the size of the core in the second direction.

[0021] In some embodiments, the activation buffer is configured to include: a first queue, connected to a first multiplier, and a second queue, connected to the second multiplier, the first queue including a first register and a second register adjacent to the first register, the first register being an output register of the first queue, and the first block is further configured to: in a first state, multiply the first weight by the activation from the output register of the first queue in the first multiplier, and in a second state, multiply the first weight by the activation from the second register of the first queue in the first multiplier.

[0022] In some embodiments, in the second state, the output register of the first queue contains zero.

[0023] In some embodiments, the processor further includes a first adder, and the method further includes: connecting the first adder to the output of the first multiplier and the output of the second multiplier in a first state, and adding the product received from the output of the first multiplier and the product received from the output of the second multiplier through the first adder.

[0024] According to some embodiments of the present disclosure, a method for performing calculations using a device for processing is provided, wherein the device for processing includes: a first block, a second block, a memory and a bus, the bus being connected to the memory, the first block and the second block, the first block including: a first weight register, a second weight register, an activation buffer, a first multiplier and a second multiplier, the method including: performing convolution of an activated array and a kernel of weights, the steps of performing convolution sequentially including: forming a tensor product of the kernel and a first subarray of the activated array; forming a tensor product of the kernel and a second subarray of the activated array, the second subarray being offset from the first subarray by n array elements in a first direction, where n is a positive integer; and forming a tensor product of the kernel and a third subarray of the activated array, the third subarray being offset from the second subarray by one array element in a second direction perpendicular to the first direction.

[0025] According to some embodiments of the present disclosure, a processor is provided, comprising: a first block, a second block, a memory and a bus, wherein the bus is connected to the memory, the first block and the second block, the first block comprising: a first weight register, a second weight register, an activation buffer, a first multiplier and a second multiplier, the processor being configured to perform a first convolution of an activated array and a first kernel of weights, the steps of performing the first convolution comprising: broadcasting a first subarray of the activated array to the first block and the second block; forming a first tensor product, which is a tensor product of a first subarray of the first kernel of weights and a first subarray of the activated array; storing the first tensor product in the memory; broadcasting a second subarray of the activated array to the first block and the second block; forming a second tensor product, which is a tensor product of a second subarray of the first kernel of weights and a second subarray of the activated array; and adding the first tensor product and the second tensor product.

[0026] In some embodiments, the first block further includes: a weight decompression unit configured to: decompress a data word encoding a plurality of weights in a compressed form to extract a first weight and a second weight; input the first weight into a first weight register; and input the second weight into a second weight register.

[0027] In some embodiments, the first block is further configured to: perform a second convolution of the activated array with a second kernel of weights, the second steps of performing the convolution sequentially including: forming a tensor product of a first portion of the second kernel and a first sub-array of the activated array, the first portion of the second kernel including weights stored in a first weight register; forming a tensor product of a second portion of the second kernel and a first sub-array of the activated array, the second portion of the second kernel including weights stored in a second weight register; and forming a tensor product of the first portion of the second kernel and a second sub-array of the activated array, the first portion of the second kernel including weights stored in the first weight register.

[0028] In some embodiments, the activation buffer is configured to include: a first queue, connected to a first multiplier, and a second queue, connected to the second multiplier, the first queue including a first register and a second register adjacent to the first register, the first register being an output register of the first queue, and the first block is further configured to: in a first state, multiply the first weight by the activation from the output register of the first queue in the first multiplier, and in a second state, multiply the first weight by the activation from the second register of the first queue in the first multiplier.

[0029] In some embodiments, in the second state, the output register of the first queue contains zero.

[0030] In some embodiments, the processor further includes: a first adder configured to be connected to the output of the first multiplier and the output of the second multiplier in a first state; and to add the product received from the output of the first multiplier and the product received from the output of the second multiplier.

[0031] In some embodiments, the processor further comprises: a second adder configured to be connected to the output of the first multiplier in a second state.

[0032] In some embodiments, the processor further includes: a first accumulator connected to the first adder, and a second accumulator connected to the second adder, the first accumulator including a register and configured to add the sum received from the first adder to the value in the register of the first accumulator in a first state to form an accumulated value of the first accumulator, and store the accumulated value of the first accumulator in the register of the first accumulator.

[0033] In some embodiments, the second accumulator includes a register and is configured to add the sum received from the second adder to the value in the register of the second accumulator in the second state to form an accumulated value of the second accumulator, and store the accumulated value of the second accumulator in the register of the second accumulator.

[0034] In some embodiments, the processor further comprises an activation zeroskip control circuit configured to determine whether the output register of the first queue contains zero, and in response to determining that the output register of the first queue contains zero, cause the first block to operate in the second state.

[0035] According to some embodiments of the present disclosure, a method for performing calculations using a processing circuit is provided, wherein the processing circuit includes: a first block, a second block, a memory and a bus, the bus being connected to the memory, the first block and the second block, the first block including: a first weight register, a second weight register, an activation buffer, a first multiplier and a second multiplier, the method including: performing a first convolution of an activated array and a first kernel of weights, the step of performing the first convolution including: broadcasting a first subarray of the activated array to the first block and the second block; forming a first tensor product, the first tensor product being the tensor product of the first subarray of the first kernel of weights and the first subarray of the activated array; storing the first tensor product in the memory; broadcasting a second subarray of the activated array to the first block and the second block; forming a second tensor product, the second tensor product being the tensor product of the second subarray of the first kernel of weights and the second subarray of the activated array; and adding the first tensor product and the second tensor product.

[0036] In some embodiments, the first block further includes a weight decompression unit, and the method further includes: decompressing a data word encoding multiple weights in a compressed form by the weight decompression unit to extract a first weight and a second weight; inputting the first weight into a first weight register; and inputting the second weight into a second weight register.

[0037] In some embodiments, the method further includes: performing a second convolution of the activated array with a second kernel of weights, the second convolution step sequentially including: forming a tensor product of a first portion of the second kernel and a first subarray of the activated array, the first portion of the second kernel including weights stored in a first weight register; forming a tensor product of a second portion of the second kernel and a first subarray of the activated array, the second portion of the second kernel including weights stored in a second weight register; and forming a tensor product of the first portion of the second kernel and a second subarray of the activated array, the first portion of the second kernel including weights stored in the first weight register.

[0038] In some embodiments, the activation buffer is configured to include: a first queue, connected to a first multiplier, and a second queue, connected to the second multiplier, the first queue including a first register and a second register adjacent to the first register, the first register being an output register of the first queue, and the first block is further configured to: in a first state, multiply the first weight by the activation from the output register of the first queue in the first multiplier, and in a second state, multiply the first weight by the activation from the second register of the first queue in the first multiplier.

[0039] In some embodiments, in the second state, the output register of the first queue contains zero.

[0040] In some embodiments, the processing circuit further includes: a first adder, and the method further includes: connecting the first adder to the output of the first multiplier and the output of the second multiplier in a first state, and adding the product received from the output of the first multiplier and the product received from the output of the second multiplier through the first adder.

[0041] In some embodiments, the processing circuit further comprises a second adder, the method further comprising connecting the second adder to the output of the first multiplier in the second state.

[0042] In some embodiments, the processing circuit further includes: a first accumulator connected to the first adder, and a second accumulator connected to the second adder, the first accumulator including a register, and the method further includes: adding the sum received from the first adder to the value in the register of the first accumulator by the first accumulator in the first state to form an accumulated value of the first accumulator, and storing the accumulated value of the first accumulator in the register of the first accumulator by the first accumulator.

[0043] In some embodiments, the second accumulator includes a register, and the method further includes: adding the sum received from the second adder and the value in the register of the second accumulator by the second accumulator in the second state to form an accumulated value of the second accumulator, and storing the accumulated value of the second accumulator in the register of the second accumulator by the second accumulator.

[0044] According to some embodiments of the present disclosure, a method for performing calculations using a device for processing is provided, wherein the device for processing includes: a first block, a second block, a memory and a bus, the bus being connected to the memory, the first block and the second block, the first block including: a first weight register, a second weight register, an activation buffer, a first multiplier and a second multiplier, the method including: performing a first convolution of an activated array and a first kernel of weights, the step of performing the first convolution including: broadcasting a first subarray of the activated array to the first block and the second block; forming a first tensor product, the first tensor product being the tensor product of the first subarray of the first kernel of weights and the first subarray of the activated array; storing the first tensor product in the memory; broadcasting a second subarray of the activated array to the first block and the second block; forming a second tensor product, the second tensor product being the tensor product of the second subarray of the first kernel of weights and the second subarray of the activated array; and adding the first tensor product and the second tensor product.

[0045] According to some embodiments of the present disclosure, a processor is provided, comprising: a first block, a second block, a memory, an input bus, and an output bus, wherein the input bus is connected to the memory, the first block, and the second block; the first block comprises: a first weight register, a second weight register, an activation buffer, a first multiplier, and a second multiplier; the first block is configured to perform a first convolution of an activated array with a kernel of weights; the memory comprises: a first memory bank group (memory bank set) and a second memory storage body group; the input bus includes: a first segmented bus for broadcasting data in a first direction, and a second segmented bus for broadcasting data in a second direction opposite to the first direction; the first segmented bus includes: a first switch block and a second switch block; the first switch block is connected to the first block and the first memory storage body group; the second switch block is connected to the second block and the second memory storage body group; the second segmented bus includes: a third switch block and a fourth switch block; the third switch block is connected to the first block and the first memory storage body group; the fourth switch block is connected to the second block and the second memory storage body group; the input of the first switch block is connected to the output of the second switch block; the output of the third switch block is connected to the input of the fourth switch block.

[0046] In some embodiments, the first segmented bus is configured to connect a first group of memory banks to a first bank through a first switch block and to connect a second group of memory banks to a second bank through a second switch block in a first bus state.

[0047] In some embodiments, the first segmented bus is further configured to connect the second memory bank group to the first bank through the first switch block and through the second switch block in the second bus state, and to connect the second memory bank group to the second bank through the second switch block.

[0048] In some embodiments, the activation buffer is configured to include: a first queue, connected to a first multiplier, and a second queue, connected to the second multiplier, the first queue including a first register and a second register adjacent to the first register, the first register being an output register of the first queue, and the first block is further configured to: multiply the first weight by the activation from the output register of the first queue in the first multiplier in a first state, and multiply the first weight by the activation from the second register of the first queue in the first multiplier in a second state.

[0049] In some embodiments, in the second state, the output register of the first queue contains zero.

[0050] In some embodiments, the processor further includes: a first adder configured to be connected to the output of the first multiplier and the output of the second multiplier in a first state; and to add a product received from the output of the first multiplier and a product received from the output of the second multiplier.

[0051] In some embodiments, the processor further comprises: a second adder configured to be connected to the output of the first multiplier in a second state.

[0052] In some embodiments, the processor further includes: a first accumulator connected to the first adder; and a second accumulator connected to the second adder, the first accumulator including a register and configured to add a sum received from the first adder to a value in the register of the first accumulator in a first state to form an accumulated value of the first accumulator, and store the accumulated value of the first accumulator in the register of the first accumulator.

[0053] In some embodiments, the second accumulator includes a register and is configured to add the sum received from the second adder to the value in the register of the second accumulator in the second state to form an accumulated value of the second accumulator, and store the accumulated value of the second accumulator in the register of the second accumulator.

[0054] In some embodiments, the processor further comprises: activating a zero skip control circuit configured to: determine whether the output register of the first queue contains zero, and in response to determining that the output register of the first queue contains zero, cause the first block to operate in the second state.

[0055] In some embodiments, the processor further comprises a multiplexer having an input connected to the first multiplier on a single-port side of the multiplexer, a first output connected to the first adder on a multi-port side of the multiplexer, and a second output connected to the second adder on the multi-port side of the multiplexer.

[0056] According to some embodiments of the present disclosure, a method for performing calculations using a processing circuit is provided, wherein the processing circuit includes: a first block, a second block, a memory, an input bus, and an output bus, wherein the input bus is connected to the memory, the first block, and the second block, the first block includes: a first weight register, a second weight register, an activation buffer, a first multiplier, and a second multiplier, and the first block is configured to perform a first convolution of an array of activations and a kernel of weights; the memory includes: a first memory bank group and a second memory bank group; the input bus includes: a first segmented bus for broadcasting data in a first direction; and a second segmented bus for broadcasting data in a second direction opposite to the first direction; A segmented bus includes: a first switch block and a second switch block; the first switch block is connected to the first block and the first memory storage body group; the second switch block is connected to the second block and the second memory storage body group; the second segmented bus includes: a third switch block and a fourth switch block; the third switch block is connected to the first block and the first memory storage body group; the fourth switch block is connected to the second block and the second memory storage body group; the input end of the first switch block is connected to the output of the second switch block; the output of the third switch block is connected to the input of the fourth switch block, and the method includes: in a first bus state, connecting the first memory storage body group to the first block through the first switch block, and connecting the second memory storage body group to the second block through the second switch block.

[0057] In some embodiments, the method further includes: in the second bus state, connecting the second memory bank group to the first bank through the first switch block and the second switch block, and connecting the second memory bank group to the second bank through the second switch block.

[0058] In some embodiments, the activation buffer is configured to include: a first queue, connected to a first multiplier; and a second queue, connected to the second multiplier, the first queue including a first register and a second register adjacent to the first register, the first register being an output register of the first queue, and the first block being further configured to: multiply the first weight by the activation of the output register from the first queue in the first multiplier in a first state, and multiply the first weight by the activation of the second register from the first queue in the first multiplier in a second state.

[0059] In some embodiments, in the second state, the output register of the first queue contains zero.

[0060] In some embodiments, the processing circuit further includes a first adder, and the method further includes: connecting the first adder to the output of the first multiplier and the output of the second multiplier in a first state; and adding the product received from the output of the first multiplier and the product received from the output of the second multiplier by the first adder.

[0061] In some embodiments, the processing circuit further comprises a second adder, and the method further comprises: connecting the second adder to the output of the first multiplier in the second state.

[0062] In some embodiments, the processing circuit further includes: a first accumulator connected to the first adder; and a second accumulator connected to the second adder, the first accumulator including a register, and the method further includes: adding the sum received from the first adder to the value in the register of the first accumulator by the first accumulator in the first state to form an accumulated value of the first accumulator, and storing the accumulated value of the first accumulator in the register of the first accumulator by the first accumulator.

[0063] In some embodiments, the second accumulator includes a register, and the method further includes: adding the sum received from the second adder and the value in the register of the second accumulator by the second accumulator in the second state to form an accumulated value of the second accumulator, and storing the accumulated value of the second accumulator in the register of the second accumulator by the second accumulator.

[0064] According to some embodiments of the present disclosure, a method for performing calculations using a device for processing is provided, wherein the device for processing includes: a first block, a second block, a memory, an input bus, and an output bus, wherein the input bus is connected to the memory, the first block, and the second block, the first block includes: a first weight register, a second weight register, an activation buffer, a first multiplier, and a second multiplier, and the first block is configured to perform a first convolution of an array of activations and a kernel of weights; the memory includes: a first memory bank group and a second memory bank group; the input bus includes: a first segmented bus for broadcasting data in a first direction; and a second segmented bus for broadcasting data in a second direction opposite to the first direction. ; The first segmented bus includes: a first switch block and a second switch block; the first switch block is connected to the first block and the first memory storage body group; the second switch block is connected to the second block and the second memory storage body group; the second segmented bus includes: a third switch block and a fourth switch block; the third switch block is connected to the first block and the first memory storage body group; the fourth switch block is connected to the second block and the second memory storage body group; the input of the first switch block is connected to the output of the second switch block; the output of the third switch block is connected to the input of the fourth switch block, the method includes: in the first bus state, connecting the first memory storage body group to the first block through the first switch block, and connecting the second memory storage body group to the second block through the second switch block.

[0065] According to some embodiments of the present disclosure, a processor is provided, comprising: a first block, a second block, a memory and a bus, wherein the bus is connected to the memory, the first block and the second block, the first block comprising: a first weight register, a second weight register, an activation buffer, a first multiplier and a second multiplier, the activation buffer being configured to comprise: a first queue connected to the first multiplier; and a second queue connected to the second multiplier, the first queue comprising a first register and a second register adjacent to the first register, the first register being an output register of the first queue, the first block being configured to: in a first state, multiply the first weight by the activation from the output register of the first queue in the first multiplier, and in a second state, multiply the first weight by the activation from the second register of the first queue in the first multiplier.

[0066] In some embodiments, in the second state, the output register of the first queue contains zero.

[0067] In some embodiments, the processor further includes: a first adder configured to be connected to the output of the first multiplier and the output of the second multiplier in a first state and to add a product received from the output of the first multiplier and a product received from the output of the second multiplier.

[0068] In some embodiments, the processor further comprises: a second adder configured to be connected to the output of the first multiplier in a second state.

[0069] In some embodiments, the processor further includes: a first accumulator connected to the first adder; and a second accumulator connected to the second adder, the first accumulator including a register and configured to add the sum received from the first adder to the value in the register of the first accumulator in a first state to form an accumulated value of the first accumulator, and store the accumulated value of the first accumulator in the register of the first accumulator.

[0070] In some embodiments, the second accumulator includes a register and is configured to add the sum received from the second adder to the value in the register of the second accumulator in the second state to form an accumulated value of the second accumulator, and store the accumulated value of the second accumulator in the register of the second accumulator.

[0071] In some embodiments, the processor further comprises: activating a zero skip control circuit configured to: determine whether the output register of the first queue contains zero, and in response to determining that the output register of the first queue contains zero, cause the first block to operate in the second state.

[0072] In some embodiments, the processor further comprises a multiplexer having an input connected to the first multiplier on a single-port side of the multiplexer, a first output connected to the first adder on a multi-port side of the multiplexer, and a second output connected to the second adder on the multi-port side of the multiplexer.

[0073] In some embodiments, the activation zero skip control circuit is configured to control the multiplexer to connect the input to the first output in a first state and to connect the input to the second output in a second state.

[0074] In some embodiments, the second queue includes a first register and a second register adjacent to the first register, the first register being an output register of the second queue; and the first block is further configured to: multiply the first weight by the activation of the second register from the second queue in the first multiplier in the third state.

[0075] According to some embodiments of the present disclosure, a method for performing calculations using a processing circuit is provided, wherein the processing circuit includes: a first block, a second block, a memory, and a bus, wherein the bus is connected to the memory, the first block, and the second block, wherein the first block includes: a first weight register, a second weight register, an activation buffer, a first multiplier, and a second multiplier, wherein the activation buffer is configured to include: a first queue connected to the first multiplier; and a second queue connected to the second multiplier, wherein the first queue includes a first register and a second register adjacent to the first register, wherein the first register is an output register of the first queue, and wherein the method includes: multiplying a first weight by an activation of the output register from the first queue by the first multiplier in a first state, and multiplying the first weight by an activation of the second register from the first queue by the first multiplier in a second state.

[0076] In some embodiments, in the second state, the output register of the first queue contains zero.

[0077] In some embodiments, the processing circuit further includes a first adder, and the method further includes: connecting the first adder to the output of the first multiplier and the output of the second multiplier in a first state, and adding the product received from the output of the first multiplier and the product received from the output of the second multiplier through the first adder.

[0078] In some embodiments, the processing circuit further comprises a second adder, the method further comprising connecting the second adder to the output of the first multiplier in the second state.

[0079] In some embodiments, the processing circuit further includes: a first accumulator connected to the first adder; and a second accumulator connected to the second adder, the first accumulator including a register, the method further including adding, by the first accumulator in a first state, the sum received from the first adder and the value in the register of the first accumulator to form an accumulated value of the first accumulator, and storing, by the first accumulator, the accumulated value of the first accumulator in the register of the first accumulator.

[0080] In some embodiments, the second accumulator includes a register, and the method further includes adding, by the second accumulator in the second state, the sum received from the second adder and the value in the register of the second accumulator to form an accumulated value of the second accumulator, and storing, by the second accumulator, the accumulated value of the second accumulator in the register of the second accumulator.

[0081] In some embodiments, the processing circuit further comprises activating a zero skip control circuit, and the method further comprises determining whether the output register of the first queue contains zero by activating the zero skip control circuit, and in response to determining that the output register of the first queue contains zero, causing the first block to operate in the second state.

[0082] In some embodiments, the processing circuit further comprises a multiplexer having an input connected to the first multiplier on a single-port side of the multiplexer, a first output connected to the first adder on a multi-port side of the multiplexer, and a second output connected to the second adder on the multi-port side of the multiplexer.

[0083] In some embodiments, the method further includes controlling the multiplexer to connect the input to the first output in a first state and to connect the input to the second output in a second state by activating the zero skip control circuit.

[0084] According to some embodiments of the present disclosure, a method for performing calculations using a device for processing is provided, wherein the device for processing includes: a first block, a second block, a memory and a bus, the bus being connected to the memory, the first block and the second block, the first block including: a first weight register, a second weight register, an activation buffer, a first multiplier and a second multiplier, the activation buffer being configured to include: a first queue connected to the first multiplier; and a second queue connected to the second multiplier, the first queue including a first register and a second register adjacent to the first register, the first register being an output register of the first queue, the method including: multiplying a first weight by an activation from an output register of the first queue in the first multiplier in a first state, and multiplying the first weight by an activation from the second register of the first queue in the first multiplier in a second state. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] These and other features and advantages of the present disclosure will be appreciated and understood with reference to the specification, claims, and accompanying drawings, in which:

[0086] Figure 1A is a block diagram depicting a neural processor according to the subject matter disclosed herein;

[0087] Figure 1B is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0088] Figure 1C depicts data flow in a portion of a neural processor according to the subject matter disclosed herein;

[0089] Figure 1D depicts data flow in a portion of a neural processor according to the subject matter disclosed herein;

[0090] Figure 1E depicts data flow in a portion of a neural processor according to the subject matter disclosed herein;

[0091] Figure 1F depicts data flow in a portion of a neural processor according to the subject matter disclosed herein;

[0092] Figure 1G depicts data flow in a portion of a neural processor according to the subject matter disclosed herein;

[0093] Figure 1H depicts data flow in a portion of a neural processor according to the subject matter disclosed herein;

[0094] Figure 1I is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0095] Figure 1J is a block diagram depicting a portion of a neural processor for three scenarios according to the subject matter disclosed herein;

[0096] Figure 1K is a schematic diagram of a portion of a neural processor according to the subject matter disclosed herein;

[0097] Figure 1L is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0098] Figure 1MA is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0099] Figure 1MB is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0100] Figure 1Nis a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0101] Figure 1O is a block diagram depicting a neural processor according to the subject matter disclosed herein;

[0102] Figure 1P is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0103] Figure 1Q is a size table based on the subject matter disclosed herein;

[0104] Figure 1R is a tensor graph according to the subject matter disclosed herein;

[0105] Figure 1S is a tensor graph according to the subject matter disclosed herein;

[0106] Figure 1T depicts data flow in a portion of a neural processor according to the subject matter disclosed herein;

[0107] Figure 1U depicts data flow in a portion of a neural processor according to the subject matter disclosed herein;

[0108] Figure 1V is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0109] Figure 1WA is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0110] Figure 1WB depicts data flow in a portion of a neural processor according to the subject matter disclosed herein;

[0111] Figure 1WC depicts data flow in a portion of a neural processor according to the subject matter disclosed herein; Figure 1WD depicts data flow in a portion of a neural processor according to the subject matter disclosed herein; Figure 1WE depicts data flow in a portion of a neural processor according to the subject matter disclosed herein;

[0112] Figure 1X is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0113] Figure 2AA is a convolutional graph according to the subject matter disclosed herein;

[0114] Figure 2AB is a convolutional graph according to the subject matter disclosed herein;

[0115] Figure 2ACis a convolutional graph according to the subject matter disclosed herein;

[0116] Figure 2AD is a convolutional graph according to the subject matter disclosed herein;

[0117] Figure 2BA is a convolutional graph according to the subject matter disclosed herein;

[0118] Figure 2BB is a convolutional graph according to the subject matter disclosed herein;

[0119] Figure 2BC is a convolutional graph according to the subject matter disclosed herein;

[0120] Figure 2BD is a convolutional graph according to the subject matter disclosed herein;

[0121] Figure 2BE is a convolutional graph according to the subject matter disclosed herein;

[0122] Figure 2BF is a convolutional graph according to the subject matter disclosed herein;

[0123] Figure 2BG is a convolutional graph according to the subject matter disclosed herein;

[0124] Figure 2BH is a convolutional graph according to the subject matter disclosed herein;

[0125] Figure 2BI is a convolutional graph according to the subject matter disclosed herein;

[0126] Figure 2BJ is a convolutional graph according to the subject matter disclosed herein;

[0127] Figure 2BK is a convolutional graph according to the subject matter disclosed herein;

[0128] Figure 2BL is a convolutional graph according to the subject matter disclosed herein;

[0129] Figure 2BM is a convolutional graph according to the subject matter disclosed herein;

[0130] Figure 2C is a convolutional graph according to the subject matter disclosed herein;

[0131] Figure 2DA is a convolutional graph according to the subject matter disclosed herein;

[0132] Figure 2DB is a convolutional graph according to the subject matter disclosed herein;

[0133] Figure 2DC is a convolutional graph according to the subject matter disclosed herein;

[0134] Figure 2DD is a convolutional graph according to the subject matter disclosed herein;

[0135] Figure 2DE is a convolutional graph according to the subject matter disclosed herein;

[0136] Figure 2DF is a convolutional graph according to the subject matter disclosed herein;

[0137] Figure 2DG is a convolutional graph according to the subject matter disclosed herein;

[0138] Figure 2DH is a convolutional graph according to the subject matter disclosed herein;

[0139] Figure 2DI is a convolutional graph according to the subject matter disclosed herein;

[0140] Figure 2DJ is a convolutional graph according to the subject matter disclosed herein;

[0141] Figure 2DK is a convolutional graph according to the subject matter disclosed herein;

[0142] Figure 2DL is a convolutional graph according to the subject matter disclosed herein;

[0143] Figure 2DM is a convolutional graph according to the subject matter disclosed herein;

[0144] Figure 2DN is a convolutional graph according to the subject matter disclosed herein;

[0145] Figure 2DO is a convolutional graph according to the subject matter disclosed herein;

[0146] Figure 2DP is a convolutional graph according to the subject matter disclosed herein;

[0147] Figure 2DQ is a convolutional graph according to the subject matter disclosed herein;

[0148] Figure 2DR is a convolutional graph according to the subject matter disclosed herein;

[0149] Figure 2DS is a convolutional graph according to the subject matter disclosed herein;

[0150] Figure 2DT is a convolutional graph according to the subject matter disclosed herein;

[0151] Figure 2DV is a convolutional graph according to the subject matter disclosed herein;

[0152] Figure 2DWis a convolutional graph according to the subject matter disclosed herein;

[0153] Figure 2DX is a convolutional graph according to the subject matter disclosed herein;

[0154] Figure 2E is a read table according to the subject matter disclosed herein;

[0155] Figure 2F is a read table according to the subject matter disclosed herein;

[0156] Figure 2GA is a convolutional graph according to the subject matter disclosed herein;

[0157] Figure 2GB is a convolutional graph according to the subject matter disclosed herein;

[0158] Figure 2HA is a convolutional graph according to the subject matter disclosed herein;

[0159] Figure 2HB is a convolutional graph according to the subject matter disclosed herein;

[0160] Figure 2HC is a convolutional graph according to the subject matter disclosed herein;

[0161] Figure 2HD is a convolutional graph according to the subject matter disclosed herein;

[0162] Figure 3AA is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0163] Figure 3AB depicting data flows according to the subject matter disclosed herein;

[0164] Figure 3AC depicting data flows according to the subject matter disclosed herein;

[0165] Figure 3AD depicting data flows according to the subject matter disclosed herein;

[0166] Figure 3AE depicting data flows according to the subject matter disclosed herein;

[0167] Figure 3AF depicting data flows according to the subject matter disclosed herein;

[0168] Figure 3AG depicting data flows according to the subject matter disclosed herein;

[0169] Figure 3AH depicting data flows according to the subject matter disclosed herein;

[0170] Figure 3AIdepicting data flows according to the subject matter disclosed herein;

[0171] Figure 3AJ depicting data flows according to the subject matter disclosed herein;

[0172] Figure 3AK depicting data flows according to the subject matter disclosed herein;

[0173] Figure 3BA A block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0174] Figure 3BB is a data graph based on the subject matter disclosed herein;

[0175] Figure 3BC is a data graph based on the subject matter disclosed herein;

[0176] Figure 3CA is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0177] Figure 3CB is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0178] Figure 3DA is a data graph based on the subject matter disclosed herein;

[0179] Figure 3EA is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0180] Figure 3EB is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0181] Figure 3FA is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0182] Figure 3FB is a data graph based on the subject matter disclosed herein;

[0183] Figure 3FC is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0184] Figure 3GA is a data graph based on the subject matter disclosed herein;

[0185] Figure 3GB is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0186] Figure 3GC is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0187] Figure 3GD is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0188] Figure 3HA is a data graph based on the subject matter disclosed herein;

[0189] Figure 3HB is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0190] Figure 3HC is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0191] Figure 3HD is a data graph based on the subject matter disclosed herein;

[0192] Figure 3IA is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0193] Figure 3IB is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0194] Figure 3IC is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0195] Figure 3ID is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0196] Figure 3IE is a data graph based on the subject matter disclosed herein;

[0197] Figure 3IF is a data graph based on the subject matter disclosed herein;

[0198] Figure 3JA depicting data flows according to the subject matter disclosed herein;

[0199] Figure 3JB depicting data flows according to the subject matter disclosed herein;

[0200] Figure 3JC depicting data flows according to the subject matter disclosed herein;

[0201] Figure 3JD depicting data flows according to the subject matter disclosed herein;

[0202] Figure 3KA is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0203] Figure 3KB is a data graph based on the subject matter disclosed herein;

[0204] Figure 3LA is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0205] Figure 3LB is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0206] Figure 3LC is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0207] Figure 3LD is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0208] Figure 3MA is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0209] Figure 3MB is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0210] Figure 3NA is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0211] Figure 3OA is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0212] Figure 3OB is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0213] Figure 3OC is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0214] Figure 3PA is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0215] Figure 3PB is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0216] Figure 3PC is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0217] Figure 4AA is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0218] Figure 4AB is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0219] Figure 4AC is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0220] Figure 4AD is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0221] Figure 4AE is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0222] Figure 4AF is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0223] Figure 4AG is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0224] Figure 4AH is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0225] Figure 4AJ is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0226] Figure 4AK is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0227] Figure 4AL is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0228] Figure 4AM is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0229] Figure 4AN is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0230] Figure 4BA is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0231] Figure 4BB is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0232] Figure 4BC is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0233] Figure 4BD is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0234] Figure 4CAis a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0235] Figure 4CB is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0236] Figure 4CC is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0237] Figure 4DA is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0238] Figure 4DB is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0239] Figure 4DC is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0240] Figure 4EA is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0241] Figure 4EB is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0242] Figure 4EC is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0243] Figure 4FA is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0244] Figure 4FB is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0245] Figure 4G is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0246] Figure 4H is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0247] Figure 5A is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0248] Figure 5B is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0249] Figure 5C is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0250] Figure 5D is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0251] Figure 5E is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0252] Figure 5F is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0253] Figure 5G is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0254] Figure 6 is a block diagram depicting a portion of a neural processor according to the subject matter disclosed herein;

[0255] Figure 7A depicts an example of IFM data having a relatively uniform distribution of zero values ​​distributed between IFM slices and in pathways within an IFM slice;

[0256] Figure 7B depicts another example of IFM data in which zero values ​​are clustered in the same IFM lane of adjacent IFM slices;

[0257] Figure 7C A block diagram depicting an example embodiment of a system for using an IFM shuffler to pseudo-randomly permute values ​​within each IFM slice to disperse clusters of non-zero values ​​within the IFM slice in accordance with the subject matter disclosed herein;

[0258] Figure 7D depicts a block diagram of an example embodiment of a 16-channel butterfly shuffler according to the subject matter disclosed herein;

[0259] Figure 7E A block diagram depicting an example embodiment of a pseudo-random generator connected to a butterfly shuffler according to the subject matter disclosed herein;

[0260] Figure 8A depicts a block diagram of an example embodiment of a baseline multiplier unit according to the subject matter disclosed herein;

[0261] Figure 8B A block diagram depicting an example embodiment of a multiplier unit supporting dual sparsity for both zero-valued activation skipping and zero-valued weight skipping according to the subject matter disclosed herein; and

[0262] Figure 8C A block diagram depicting an example embodiment of a system that uses an IFM shuffler to pseudo-randomly permute values ​​within each IFM slice to even out the distribution of zero-valued activations and zero-valued weights in accordance with the subject matter disclosed herein. DETAILED DESCRIPTION

[0263] The specific embodiments described below in conjunction with the accompanying drawings are intended as descriptions of exemplary embodiments of neural processors provided in accordance with the present disclosure and are not intended to represent the only forms in which the present disclosure may be constructed or utilized. The description, in conjunction with the depicted embodiments, illustrates features of the subject matter disclosed herein. However, it should be understood that the same or equivalent functions and structures may be implemented by different embodiments that are also intended to be included within the scope of the subject matter disclosed herein. As shown elsewhere herein, the same element numbers are intended to indicate the same elements or features. In addition, as used herein, the word "exemplary" means "used as an example, instance, or illustration." Any embodiment described herein as "exemplary" should not be construed as necessarily preferred or advantageous over other embodiments.

[0264] As used herein, the term "module" refers to any combination of software, firmware, and / or hardware configured to provide the functionality described herein in conjunction with the modules. Software may be implemented as a software package, code, and / or instruction set or instructions, and the term "hardware" as used in any embodiment described herein may, for example, include, alone or in any combination, hard-wired circuitry, programmable circuitry, state machine circuitry, and / or firmware that stores instructions executed by programmable circuitry. Modules may be implemented collectively or individually as circuits that form part of a larger system (e.g., but not limited to, an integrated circuit (IC), a system on a chip (SoC), etc.). The various components and / or functional blocks disclosed herein may be implemented as modules that may include software, firmware, and / or hardware that provide the functionality described herein in conjunction with the various components and / or functional blocks.

[0265] Figure 1A Depicts a high-level block diagram of a neural processor 100 in accordance with the subject matter disclosed herein. The neural processor 100 can be configured to efficiently determine or compute convolutions and / or tensor products of input feature maps (IFMs) (or tensors of "activations") with multidimensional arrays (or tensors) of weights to form output feature maps (OFMs). The neural processor 100 can also be configured to determine or compute feature map pooling and / or activation functions; however, for the purposes of clarity and brevity, pooling and activation functions are not generally covered herein.

[0266] A plurality of memory bank groups 109 (each memory bank group includes a number of memory banks (eg Figure 4AB and 4ACThe four memory banks 108 in the MR block 102 can be connected to a multiply-and-reduce (MR) tile 102 (described in further detail below) via an IFM delivery fabric 104. The IFM delivery fabric 104 brings the input activation map stored in a memory bank group 109 to the tile 102 for subsequent computation. As will be discussed in further detail below, the tile 102 includes an array of multiplier units (MUs) 103. The tile 102 is also connected to the memory bank group 109 via an OFM delivery fabric 106. The OFM delivery fabric 106 sends the results of the computations from the tile 102 to the memory bank group 109 for storage. In one embodiment, the memory bank group 109 can be a static random access memory (SRAM) memory bank group. Therefore, the memory bank group 109 may be referred to herein as an SRAM bank group 109, or simply as SRAM 109. In another embodiment, the memory bank group 109 may include a group of volatile and / or non-volatile memory banks.

[0267] The IFM transport fabric 104 can be a segmented bus (as discussed below), and thus, each of the SRAM bank groups 109 can be associated with one of the tiles 102. A central controller 110 can supply control words for controlling registers in the system via a common bus 112. Data can be transferred to the neural processor via an AXI (ARM Advanced eXtensible Interconnect) interconnect 114, and the results of processing operations performed by the neural processor 100 can similarly be retrieved via the AXI interconnect 114. An MCU (microcontroller) 116 can be used to schedule computations in a timely manner by appropriately configuring the central controller 110, and to coordinate and execute data transfers between the neural processor 100 and external memory 120 using a DMA controller 118. Each of the different components and / or functional blocks of the neural processor described herein can be implemented as separate components and / or modules.

[0268] Each block 102 may include a multiply-reduce (MR) array 122 having a multiply-reduce (MR) column 133 . Figure 1B The MR arrays 122 are depicted as they may be configured in some embodiments. Each MR array 122 may include eight MR columns 133, of which only two are depicted for clarity. Each MR column 133 may include sixteen MUs 103 and two adder trees 128A and 128B, of which only four are depicted for clarity.

[0269] Each MU 103 may include a plurality of registers (e.g., a register file 127 containing eighteen 9-bit registers, which may be referred to as "weight registers") and a multiplier 126. The multiplier 126 multiplies the input activations by the weights in the register file 127. Subsequently, the adder trees 128A and 128B in each MR column 133 sum (i.e., reduce) the resulting products from the sixteen MUs 103 in the column to form a dot product. As described below, the summation may be performed in a specific manner.

[0270] Each tile 102 may also include an IFM cache 139 and an activation broadcast unit (ABU) 141. The IFM cache 139 may reduce SRAM reads for input feature maps by caching IFM values ​​received from the SRAM 109. Just as each MR column 133 may include sixteen MUs 103, the IFM cache 139 may include sixteen parallel "activation lanes," where each activation lane 137 effectively corresponds to a "row" of MUs 103 in the MR array 122.

[0271] The activation broadcast unit 141 may be responsible for the preparation of the input activations. The first step in the preparation process may include fetching the input activations from the IFM cache 139 into the IFM activation buffer 124 according to the convolution sequence, while also omitting zero-valued activations when possible to implement sparse activation computation functionality. The sparse activation computation functionality may optionally be disabled, resulting in a "dense" tensor computation mode. The second step in the preparation process may include converting activations of numeric types into a sign-and-8bit-magnitude format, the step of converting may include using a type converter 135 to divide data types having a bit width exceeding 8 bits into a series of signed 8-bit magnitude values. When the activations have been encoded using a "zero-point" encoding as supported by, for example, Google TensorFlow, a zero-point constant value Z may be added to the activations before converting the values ​​to a signed 8-bit magnitude format.

[0272] Just as each MR column 133 may contain sixteen MUs 103, the ABU 141, IFM buffer 124, and type converter 135 may each contain sixteen channels. The resulting sixteen converted activation values ​​are broadcast in parallel to the MR array 122, such that each activation channel brings the input activation value to a corresponding row of eight MUs 103.

[0273] Each MR column 133 may also include an accumulator 130A and 130B, one for each of the adder trees 128A and 128B. As used herein, an "accumulator" is a combination of an adder and a register that can be configured to add an input value to the contents of a register and overwrite the contents of the register with the resulting sum.

[0274] As previously described, the MUs 103 in the MR array 122 may be arranged into a plurality of rows (e.g., 16 rows) and columns (or "OFM channels") (e.g., eight columns), where, for clarity, Figure 1B Only four of the 16 lines are depicted, in Figure 1B Only two columns labeled "O0" and "O7" are depicted.

[0275] An IFM vector having a length of sixteen values ​​may be referred to herein as an "IFM slice." An IFM slice may have associated plane coordinates (x, y) and an associated depth channel index d as an index into an associated IFM tensor (e.g., IFM[x, y, d: d+15]). In general, block 102 receives one IFM slice at a time from an on-chip memory or SRAM containing a 3D IFM tensor, in which each input IFM slice contains values ​​for sixteen depth channels from index d to d+15 (inclusive) at a plane location (x, y) in the input layer.

[0276] Similarly, an OFM vector having a length of eight values ​​may be referred to herein as an "OFM slice". An OFM slice may have associated plane coordinates (x, y) and an associated depth channel index d as an index into the associated OFM tensor (e.g., OFM[x, y, d: d+7]). In general, block 102 produces an OFM slice as output. In some embodiments, when the block is not stalled, as will be seen below, the output rate may vary from one OFM slice per clock to, for example, a maximum of two OFM slices per clock. Note that the OFM output vector (OFM slice) output from block 102 may need to be further reduced by a reduction fabric 111 to complete the OFM vector calculation before sending the final OFM vector result through the OFM transmit fabric 106 for storage in SRAM 109.

[0277] Note that both the IFM tensor and the OFM tensor may also have a fourth "batch" dimension; however, as opposed to neural network model training, the primary purpose of the neural processor 100 is to accelerate neural network model real-time inference, and real-time inference is typically performed based on a batch size of 1. To simplify the explanation, the batch dimension will be omitted in most of the following discussion, and the batch dimension details will be described separately later.

[0278] The neural processor 100 may be implemented in synchronous logic, and each MR column 133 may be entirely within a single clock domain. In some embodiments, during each operating cycle (e.g., during each clock cycle), each of the sixteen multipliers 126 may form a corresponding product from two multiplicands (or operands) at its inputs. Each of the adders 128A and 128B may form a sum of some or all of the sixteen products at the inputs of the adders 128A and 128B (e.g., Figure 1B 130B) and the adder of each accumulator 130A and 130B may form the sum of (i) the current value of the register of the corresponding accumulator in 130A and 130B plus (ii) the output of the corresponding adder in adders 128A and 128B. At the beginning of the next clock cycle, the output of each adder of each accumulator 130A and 130B may be written to the register of accumulator 130A and 130B.

[0279] In some embodiments, the computations provided by block 102 may be pipelined, and additional registers (i.e., flip-flop arrays) may be present in Figure 1B , to provide, for example, sufficient timing margin at the clock speed at which the circuit operates. In such an embodiment, the throughput may be the same (i.e., the same as in the absence of the additional registers (e.g., one multiplication and addition per clock cycle)), but the delay between (i) the input data being input to multiplier 126 and (ii) the final result of the multiplication and addition being written to the registers of accumulators 130A and 130B may be greater (e.g., a few clock cycles).

[0280] Figures 1C to 1H Depicting an example of such operation, the neural processor 100 exploits sparsity in the IFM data to utilize the multiplier 126 by advancing certain multiplication and addition operations out of sequence (if an element of the IFM data is equal to zero, the multiplier 126 will additionally perform a multiplication with zero), thereby accelerating the completion of the computation. The IFM data may be stored in the SRAM bank group 109, and the retrieval of the IFM data from the SRAM bank group 109 may be scheduled such that the activation buffer 124 operates as a plurality of queues. Each queue formed by the activation buffer 124 corresponds to a plurality of queues such as Figure 1B As depicted in FIG. 1 , each queue outputs IFM data to a respective lane of the MR array 122 .

[0281] For clarity, assume that the IFM cache 139 between the SRAM bank group 109 and the activation buffer 124 has been disabled and bypassed. Also assume that the data type of activation is uint8 (8-bit unsigned integer) and the data type of weight is int8 (8-bit signed integer), in which case the type converter 135 is used to pass the activation value unchanged and the multiplication in the MU 103 takes one clock cycle. Another assumption is that the SRAM bank group 109 contains the following at the beginning of the example operation: Figure 1B Some sample IFM values ​​are depicted in , and only one block is being used.

[0282] Another assumption is that the weight tensor W[0...15,0...7,a...j] corresponds to 16 IFM lanes, 8 IFM columns, and 10 IFM input vectors, and a to j have been preloaded into the corresponding MU register file (ie, register file 127).

[0283] Once the example operation starts, Figure 1C It can be seen that, in the depicted example, the two IFM vectors a[] and b[] in the two rightmost columns of the SRAM bank group 109 have been fetched into the activation buffer 124, such that the first column (i.e., the right column a[]) of the activation buffer 124 contains the first vector of IFMs (i.e., elements a0 to a3), and the second column (i.e., the left column b[]) of the activation buffer 124 contains the second vector of IFMs (i.e., elements b0 to b3, with b1=0). Figure 1C , the second queue contains a1 as its first element (closest to the MR array 122), and the second queue contains zero (0) as its second element (ie, b1=0).

[0284] Prior to the activation buffer 124, the IFM vector a[0...3] is broadcast to the MR array 122 (i.e., the IFM value a0 is broadcast as an input to each of the eight multipliers 126 in the top row via the topmost activation path 137). Simultaneously, the top row of multipliers 126 in columns 0 through 7 receive weights W[0,0...7,a] from their respective local register files 127 as the second input to each multiplier 126.

[0285] Similarly, the value a1 is broadcast as an input to the second row of multipliers 126 from the top through the second activation path 137 from the top. At the same time, the second row of multipliers 126 from the top in columns 0 to 7 receive weights W[1, 0 . . . 7, a] from their respective local register files 127 as the second input of each multiplier 126.

[0286] In operation, the product of the first vector of the IFM (i.e., elements a0 to a3) and the corresponding weight may be formed in each of the 16×8 array of multipliers 126, and the sum of the products corresponding to the desired dot product may be formed in the first adder 128A and stored in the first accumulator 130A. That is, the contents of the first accumulator 130A contain:

[0287] Σ A,0 =a0*w 0,0,a +a1*w 1,0,a +a2*w 2,0,a +a3*w 3,0,a ...

[0289] Σ A,7 =a0*w 0,7,a +a1*w 1,7,a +a2*w 2,7,a +a3*w 3,7,a .

[0290] In this regard, the determination or calculation of the OFM output vector corresponding to IFM a[] utilizes the results available in accumulator 130A (in Figure 1C Depicted as Σ A,0...7 ) is complete and ready for output to the OFM transmit structure 106. The accumulator 130A for each column may then be cleared.

[0291] exist Figure 1D , after the first vector of the IFM has been processed, the third vector of the IFM (i.e., elements c0 to c3, with c2=0) can be read into the activation buffer 124. Instead of forming products of weights with all elements of the second vector of the IFM (i.e., elements b0 to b3, with b1=0), which would require forming products of zero with corresponding weights in each multiplier 126 of the second pass, the second element of the third vector of the IFM (i.e., element c1) is advanced out of sequence and multiplied with the corresponding weight in each multiplier 126 of the second pass.

[0292] At the same time, the multipliers 126 in paths 0, 2, and 3 are receiving weights W[0, 0 ... 7, b], W[2, 0 ... 7, b], and W[3, 0 ... 7, b], respectively, from their respective local register files. Because path 1 is operating out of sequence due to activation b1 = 0 being skipped, the multipliers in path 1 receive weights W[0, 0 ... 7, c] associated with IFM vector ("pixel") c, rather than weights associated with IFM vector (pixel) b.

[0293] Since block 102 is now processing two pixels simultaneously (a portion of pixel c and pixel b), adding products in the column may produce incorrect results. To obtain the correct result, one of the two adder trees 128A and 128B is used to calculate the dot product of pixel b, while the other of the two adder trees 128A and 128B is used to start calculating the dot product of pixel c.

[0294] The product formed by each multiplier 126 of the second path is input to the second adder 128B (at Figure 1D The indicator is Σ B,0...7 ), while the product formed by the multiplier 126 of the other path is input to the first adder 128A. The out-of-sequence advancement of element c1 forms a "hole" in the activation buffer 124, which can be opened in a subsequent clock cycle by advancing another element (such as Figure 1E As depicted, it is utilized when element d1 is advanced out of sequence).

[0295] Once the products of the non-zero elements of the second vector of IFM data and the corresponding weights have been determined or calculated, and their sum is in the first accumulator 130A for each column, the first accumulator 130A for each column contains the dot product of the second vector of IFM (b[]) and the weight vector for that column, and the dot product can be output to the OFM transmission structure 106. The first accumulator 130A for each column can be cleared. That is, the contents of the first accumulator 130 for each column before being cleared contain:

[0296] Σ A,0 =b0*w 0,0,b +b2*w 2,0,b +b3*w 3,0,b ...

[0298] Σ A,7 =b0*w 0,7,b +b2*w 2,7,b +b3*w 3,7,b .

[0299] At this time, the second accumulator 130B of each column only contains one item of the dot product of the third vector (c1) of the IFM and the corresponding weight vector. In other words, the content of the second accumulator 130B includes:

[0300] Σ B，0 =c1*w 1，0，c ...

[0302] Σ B,7 =c1*w 1,7,c .

[0303] Reference Figure 1EIn a subsequent operation (e.g., during the next clock cycle), the product of the element (c0, c3) of the third vector of the IFM and the corresponding weight vector may be formed by the first multiplier 126 and the fourth multiplier 126 of each column of the MR array 122. Each product may be added to a product already stored in the second accumulator 130B to complete the dot product of the third vector of the IFM (c[]) and the corresponding weight vector in the second accumulator 130B. That is, the contents of the second accumulator 130B include:

[0304] Σ B,0 =c0*w 0,0,c +c1*w 1,0,c +c3*w 3,0,c ...

[0306] Σ B,7 =c0*w 0,7,c +c1*w 1,7,c +c3*w 3,7,c .

[0307] The dot product of the fourth vector of the IFM (i.e., elements d0 through d3, with d0=d3=0) and the weight vector can be determined or calculated simultaneously by advancing both element d1 out of sequence (because the product involving c1 was performed in the previous cycle, leaving a "hole" in the activation buffer 124) and element d2 (because c2=0). The contents of the first accumulator 130A include:

[0308] Σ A,0 =d1*w 1,0,d +d2*w 2,0,d ...

[0310] Σ A,7 =d1*w 1,7,d +d2*w 2,7,d .

[0311] At this point, the calculation of the OFM data of the IFM vector c[] and the IFM vector d[] is completed.

[0312] In a similar manner, when the activation buffer contains Figure 1F When two vectors e[] and f[] have complementary sparsity as depicted in FIG, each column of the MR column 122 can form two dot products simultaneously. Figure 1F In the example depicted in , for example, “the dot product of the fifth vector of IFM data (i.e., elements e0 to e3, and e0=e1=0) and the corresponding weight vector” and “the dot product of the sixth vector of IFM data (i.e., elements f0 to f3, and f2=f3=0) and the corresponding weight vector” are formed simultaneously, and the two non-zero elements of the sixth vector are not advanced in sequence.

[0313] Figure 1G Depicts the state of the seventh vector g[] of IFM data (i.e., elements g0 to g3, and g1=g2=g3=0) in the first column of the activation buffer 124 and the eighth vector of IFM data (i.e., elements h0 to h3, and h2=h3=0) in the second column of the activation buffer 124. Figure 1G Describing how the (non-zero) elements of the eighth vector of IFM data are processed simultaneously with the (non-zero) elements of the seventh vector of IFM data by advancing the (non-zero) elements of the eighth vector out of sequence, the “dot product of the eighth vector of IFM data h[] with each corresponding weight” is formed simultaneously with the “dot product of the seventh vector of IFM data with each corresponding weight.” Because one of the (non-zero) elements (h0) of the eighth vector of IFM data and the (non-zero) element (g0) of the seventh vector of IFM data are in the same path, each of the (non-zero) elements of the eighth vector of IFM data is shifted to an adjacent path of MR column 122 so that these elements can be advanced out of sequence.

[0314] "Inputting the first element (h0) of the eighth vector h[] of IFM data to the second multiplier 126 from the top of each column (which is not used for the seventh vector g[] of IFM data because it has a zero element at that position), and inputting the second element (h1) of the eighth vector h[] of IFM data to the third multiplier 126 of each column (which is also not used for the seventh vector g[] of IFM data)" allows the (non-zero) elements of the eighth vector of IFM data to be processed simultaneously with the (non-zero) elements of the seventh vector of IFM data. The corresponding elements of the weight vector of the eighth vector h[] are also shifted. More specifically, each MU 103 associated with the topmost path obtains two weights, one weight associated with g0 (in Figure 1G Marked as w 0，0..7，g , where 0..7 indicates the corresponding column), and another weight is associated with h0 (in Figure 1G Marked as w 0，0..7，h ). Each weight w 0，0..7，g is input to the corresponding multiplier 126 in the topmost path that is receiving g0. However, each weight w 0，0..7，h is shifted down one path and is input to the multiplier 126 of the second path from the top in the same column that is receiving h0. Finally, the MUs 103 in the second path from the top each acquire a weight w 1，0..7，h (associated with h1), and shift these weights down one pass, to the third pass from the top in the same column that is receiving h1.

[0315] exist Figure 1GIn the state depicted in FIG, each multiplier 126 of each MR column 122 in the bottom path is unused for one cycle. In some embodiments, the likelihood of such an event can be reduced by configuring the MR block 102 with a deeper (e.g., 3-deep) activation buffer 124 so that each activation path can have more (e.g., three) values ​​from the same path selected from it to fully utilize all multipliers 126. Taking non-zero activations away (shifting) from paths that are more than one path away also provides greater flexibility in replacing zero-valued activations with non-zero activations. Having more than two sets of adder trees and associated accumulators can also improve multiplier utilization.

[0316] Figure 1H Depicted in Figure 1G , wherein the first column of activation buffer 124 contains the ninth vector of IFM data (containing entirely zeros), and the second column of activation buffer 124 contains the tenth vector of IFM data (i.e., elements j0 through j3). Figure 1H In the depicted state, all elements of the tenth vector of IFM data can be advanced out of sequence, and the dot product of the tenth vector j[] of IFM data and each weight vector can be calculated without causing a one-cycle delay in the processing of the ninth vector of IFM data.

[0317] As depicted in the above example, the output of multiplier 126 may be input to adder tree 128A during some clock cycles and may be input to adder tree 128B during other clock cycles. When the output of multiplier 126 is not input to adder tree 128A or 128B, the inputs to the adder trees may be set to zero. Figure 1I Depicts the use of multiplexer 132 to direct the output of any multiplier 126 to either the first adder 128A or the second adder 128B to support, for example, Figures 1D to 1H Here, the multiplexer control signals sel_adder_tree[0...15] come from the block control FSM 144 ( Figure 1O ) to coordinate the computation within block 102, which includes fetching IFM vectors from the cache, selecting non-zero activations from the activation buffer and multiplexing the non-zero activations into the activation paths, selecting the adder tree to use with each IFM vector, multiplexing the multiplier unit outputs into the correct adder tree, and clearing the column accumulators.

[0318] Since the output of multiplier 126 is always input to either adder tree 128A or adder tree 128B, but never to both adder trees 128A and 128B simultaneously, both adder trees 128A and 128B may be implemented using less logic. Figure 1JDescribe how both the first adder 128A and the second adder 128B can be logical concepts implemented using a single physical adder tree and suitable multiplexers (not shown). For clarity, consider configuring two adder trees, each including four inputs. A four-input adder tree can be implemented using three adders. In a simple approach, each adder tree will use three adder elements, so configuring two four-input adder trees will use six adder sub-elements. Two four-input adder trees can be constructed using only three adder elements with the help of some additional multiplexers. There are three cases worth considering. (i) In the first case, all four inputs are summed by the first logical adder 128A (and the output of the second logical adder 128B is zero). (ii) In the second case, three of the inputs are summed by the first logical adder 128A (and the output of the second logical adder 128B is equal to the remaining inputs). (iii) In the third case, two of the inputs are summed by the first logical adder 128A, and two of the inputs are summed by the second logical adder 128B. In the other two cases ( Figure 1J (not depicted), the second logical adder 128B sums three or all four of the inputs, respectively, and the first logical adder 128A sums the remaining inputs or zero. As used herein, Figure 1J As in the example of , an "adder" is a physical circuit for adding at least two numbers to form a sum, or one of a plurality of logical adders formed by a combination of a physical adder and a multiplexer. Figure 1J As can be seen, only three adder elements (and some additional multiplexers not shown) instead of six adder elements are sufficient to realize all possible cases.

[0319] Figure 1K Depicted is an internal circuit diagram of a multiplier unit 103 according to the subject matter disclosed herein. The multiplier unit 103 may include an unsigned 8-bit by unsigned 8-bit multiplier 126, a register file 127 that may hold local weights, a multiplexer 143 that may select input weights for the multiplier 126, multiplexers 149 and 151 that may shift the local weights to adjacent lanes, logic 145, 136, 157, 155, and 159 that may detect a multiply-by-zero condition and idle the multiplier 126 to reduce dynamic power consumption, and weight loading logic 157.

[0320] Register file 127 holds weights. One register corresponds to a single int8 or uint8 weight. Weights with a larger bit width occupy more than one register, for example, an int16 (16-bit signed integer) or uint16 (16-bit unsigned integer) weight may occupy two registers. Register file 127 can hold 18 int8 or uint8 weights or correspondingly 9 int16 or uint16 weights. As described later, the number of registers can be selected to enable 3 by 3 convolutions to be calculated using 16-bit weights without resorting to generating partial results.

[0321] The register file 127 includes a register file for registers loaded via the vertical weight load bus 101 ( Figure 1N ) loads the weights {swt_in[C], wt_abs_ld_in[7:0][C]} from a single input port. Each MR column 133C receives its own weight load bus, where C is in the range of 0 to 7. Each MR column 133C is loaded from the weight decompression unit 138 ( ) by placing the weight values ​​{swt_in[C], wt_abs_ld_in[7:0][C]} on the vertical weight load bus 101, specifying the index of the target register (from zero to seventeen) on the weight register index bus wt_ld_idx[4:0], and determining the lane weight load enable wt_ld_en_lane[L] to load the weights into lane L. Figure 1N ) loads weights in an entire path (i.e., in all eight columns in a single path simultaneously).

[0322] from Figure 1K , loading all weights in a single path takes eighteen cycles, and loading all weights in the entire MU array 122 requires a total of 18*16=288 clock cycles. In some cases, this weight loading speed may be insufficient, especially when computing fully connected (FC) layers. Unlike convolutional layer computations, during FC layer computations, each weight is used only once and is discarded thereafter. Therefore, in order to maintain maximum utilization of the multipliers 126 when computing FC layers, it is necessary to load one weight in each multiplier unit 103 per clock, which is slower than Figure 1K In this case, the embodiment may be modified to include additional weight loading buses 101 {swt_in[C0], wt_abs_ld_in[7:0][C0]}, {swt_in[C1], wt_abs_ld_in[7:0][C1]}, etc., to speed up weight loading.

[0323] exist Figure 1KIn

[0044] , the weight register file 127 includes three output ports to enable simultaneous access to three weights in the event that one of the weights is to be shifted up one way while a second weight is shifted down one way and a third weight is being consumed locally.

[0324] Multiplexer 147 is used to implement the retrieval of weights from the local register file for local consumption. Figure 1C In the example, multiplexer 147 selects the locally stored weight w to be multiplied by the IFM value a0. 0,0,a As another example, in Figure 1D In the example, the multiplexer 147 selects the locally stored weight w to be multiplied by the IFM value c1. 1,0,c .

[0325] Multiplexer 149 is used to implement the process of getting the weight from the local register file 134 and shifting the weight to the path below. Figure 1G In the local storage weight w 0,0,h Shift down one channel to multiply by the IFM value h0.

[0326] Finally, multiplexer 151 is used to implement the process of fetching weights from the local register file 127 and shifting the weights to the upper path.

[0327] Because the activation broadcast unit (ABU) 141 has complete information about the shift of each activation path and the offset in the activation buffer associated with each IFM value being broadcast (to the activation path), the ABU 141 controls all three register file fetch multiplexers 147, 149 and 151 using signals sel_wt_self[4:0], sel_wt_dn1[4:0] and sel_wt_up1[4:0] respectively.

[0328] To reduce the area of ​​MR column 133, the number of output ports in register file 127 can be reduced from three to two, for example, by prohibiting simultaneous up and down shifting of weights from the same register file. The number of output ports in register file 127 can be further reduced to one, for example, by prohibiting all weight shifts or allowing a single shift of weights or locally consuming weights. However, limiting shifts and maximum shift distances may reduce multiplier utilization to some extent. Multiple variations and combinations of shift target path selection and activation buffer depth can be designed to optimize multiplier utilization while reducing the complexity, area, and power of MR column 133 and activation broadcast unit 141. As described in the related disclosure (Attorney Docket No. 1535-467CON2), a specific effective method and apparatus for achieving optimized multiplier utilization involves shuffling (permuting) activation paths in a pseudo-random manner while loading the associated weights accordingly.

[0329] exist Figure 1K , multiplexer 143 selects the input weights to be used in the multiplication of multiplier 126. As previously discussed, the input weights may come from the local weight register file 127, or from the weight register file in the adjacent upper path (and in some embodiments, the same column) to become "shifted down", or from the weight register file in the adjacent lower path (and in some embodiments, the same column) to become "shifted up", as represented by signals {swt_self, wt_abs_self[7:0]}, {swt_dn1, wt_abs_dn1[7:0]}, and {swt_up1, wt_abs_up1[7:0]}, respectively. Because the activation broadcast unit 141 has complete information about the shift of each active path and the activation buffer offset associated with each IFM value being broadcast (to the active path), the ABU 141 uses the signal sel_mult_wt[1:0] to control the multiplexer 143.

[0330] For example, in Figure 1C In the example, multiplexer 143 selects the weight w that will be multiplied by the IFM value a1. 0,0,a {swt_self,wt_abs_self[7:0]}. exist Figure 1D In the example, multiplexer 143 selects the weight w that will be multiplied by the IFM value c1. 1,0,c {swt_self,wt_abs_self[7:0]}. exist Figure 1G , multiplexer 143 selects the weight w that will be multiplied by the IFM value h0 through the second multiplier 126 from the top in column zero. 0,0,h {swt_self,wt_abs_self[7:0]}.

[0331] Note that, if Figure 1K As shown in , each register file 127 has a bit width of nine, where eight bits hold the weight magnitude stored in a signed 8-bit size format and one bit holds the weight sign stored in a signed 8-bit format (and with a "zero point" constant Z pre-incremented when applicable). The bit width of the register file 127 can be reduced to eight bits by adding logic that converts the signed int8 type to a signed 8-bit size representation (including zero point addition when applicable) on the fly when the weight is retrieved from the register file 127. Such on-the-fly conversion may be of interest when the size of the register file 127 has been chosen to be large enough to result in the described area savings.

[0332] The activation broadcast unit 141 broadcasts the activation {sact, act_abs[7:0]} to be used as the input to the multiplier 126. Logic gates 145 and 159 use the signals wt_zero and act_zero (auxiliary signals from the ABU) to check for a multiplication by zero condition where the weight (to be multiplied) is equal to zero or the activation (to be multiplied) is equal to zero, or both. If a multiplication by zero condition occurs, the resulting signal mult_by_zero is asserted, causing the clocks of the weight and activation multiplier input registers to be gated using the mult_in_ce signal. Gating the clocks of the input multiplier registers keeps the multiplier inputs and the multiplier internal signals (freezing) at their previous states, thereby preventing switching activity from degrading dynamic power. In parallel with this activity, the weight loading logic 157 delays the mult_in_ce signal corresponding to the multiplication by zero by one cycle to generate the mult_out_zero signal, which causes the logic gate 155 to reset the multiplier output mult_result[15:0] to zero. As discussed later, the ABU 141 also sends a signal en_mult to idle all multipliers 126 whenever computations in an entire block are to be stopped.

[0333] Figure 1K The signal names in follow the convention where “act” stands for activation, “wt” stands for weight, “s” in “sact”, “swt”, “mult_out_s”, “s_in_a”, etc. stands for “sign”, and “abs” in “wt_abs”, “act_abs”, etc. stands for absolute value (magnitude).

[0334] ABU 141 broadcasts the activation {sact, act_abs[7:0]} in a signed 8-bit magnitude format. Similarly, the selected weight (for multiplication) {mult_swt, mult_wt_abs[7:0]} is also supplied in a signed 8-bit magnitude format. Registers 136a and 136b latch the activation and weight, respectively, which are multiplied to create the input signals {s_in_a, mult_in_a[7:0]}, {s_in_b, mult_in_b[7:0]} to multiplier 126. In some embodiments, multiplier 126 computes the product by multiplying two absolute 8-bit values ​​and XORing the two signs, producing a signed 16-bit magnitude output {mult_out_s, mult_out_abs[15:0]}. Logic 153 converts the signed 16-bit magnitude result to a 16-bit signed output to be input to the adder tree by negating the product absolute magnitude mult_out_abs[15:0] to generate the signal mult_out[15:0] when the product sign is determined (i.e., the product result is negative). Finally, logic 155 resets mult_out[15:0] to zero in the case of multiplication by zero, as previously described.

[0335] To summarize the role of the ABU 141 for multiplication control, the ABU 141 provides: input IFM data in a signed 8-bit format, weight selection control for both the upshift path and the downshift path, and an auxiliary signal, act_zero, indicating that the current active being broadcast is equal to zero. When the act_zero signal is asserted, the actual value of {sact, act_abs[7:0]} may remain unchanged to reduce active path switching activity. While it is possible for a zero-valued active to be broadcast, some embodiments may minimize this occurrence.

[0336] Figures 1B to 1H The computation supporting sparse activations is depicted by fetching non-zero valued activations from the IFM buffer 124 within the ABU 141 whenever possible and reusing the associated weights to the multiplier 126 to obtain the correct dot product. The IFM buffer 124 fetches IFM values ​​from the cache 139 and stages the fetched IFM values ​​in the activation staging FIFO (or IFM staging FIFO) 165 (see Figure 1L and Figure 1MA ). Subsequently, a plurality of activation multiplexers 166 are used to obtain non-zero activations from the IFM staging FIFO 165 (when possible), so that activations can be "shifted" up or down from adjacent paths, as well as obtaining activations out of sequence.

[0337] exist Figure 1MA and Figure 1MBIn the examples (discussed below), the "look-ahead" distance (h) is the search distance along the same lane, the "look-aside" distance (d) is the sideways search distance, and the FIFO depth (F) represents the depth of the activation FIFO 165. For clarity of terminology, a plurality of activation multiplexers 163-166 accept IFM lanes as inputs from the IFM staging FIFO 165, apply forward and sideways lookups to retrieve activations, and output the resulting values ​​to activation "lanes" (not lanes). The use of the term "lane" helps distinguish the concept of logical indexing of depth "lanes" within a tensor from the concept of activations flowing along physical hardware "lanes."

[0338] Register 161 within IFM staging FIFO 165 may be optional and is shown for clarity of explanation. In some cases, area and power may be reduced by eliminating the activation of staging FIFO register 161, connecting IFM multiplexer 163 directly to the multi-port cache output, and modifying the IFM cache read logic to fetch IFM values ​​from cache 139 directly to multiplexer 163 in the correct order.

[0339] Figure 1MA Depicts the configuration of a multiplexer 163 that can be used to select activations from the activation staging FIFO register 161 from any of a number of possible values ​​stored in the activation FIFO 165, including values ​​in the same lane and values ​​in other lanes, for broadcast to the MR array 122 (via the type converter 135) and input to the multiplier 126 in any of the multiple lanes of the block (e.g., a total of 16 lanes in the block). For the more general case, each cell can enter a 2*d multiplexer, and each destination can have an equal number of sources (2*h*d), except that lanes 1 and 16 have h*(d+1) sources due to being at the ends.

[0340] Let the output cache size (C) be defined as the accumulate-and-return unit (ARU) 167 residing in each MR column ( Figure 1N ) in the output cache. Let the input bandwidth (I) be defined as the IFM streaming bandwidth (the number of 16-byte long IFM vectors per clock cycle); and let the output bandwidth (O) be defined as the OFM transport fabric bandwidth (the number of 8-byte long OFM vector results per clock cycle). In addition, the original sparsity (s r %) can be defined as the sparsity based on the observation that the number of zero elements in the activation tensor is proportional to the total number of activations in the activation tensor. a%) can be defined as the actual number of zero elements applied during the two-dimensional convolution (conv2d) processing on the activation tensor (proportional to the total number of activations in the activation tensor), which takes into account the convolution stride (e.g., the convolution stride may not use a specific zero-valued activation or may include a specific zero-valued activation multiple times), and this takes into account convolution padding. Multiplier utilization (U M ) can be defined as the percentage of cycles in which the multiplier performs a valid multiplication (multiplication by a non-zero activation). For a 1×1 convolution, for example, if the activation tensor has the original sparsity s r %, then if a simple, naive approach is used (i.e., the "dense" computation mode without zero skipping), the multiplier utilization will be (1-s r %), and for non-1×1 convolutions, when using simple, naive (dense) computation, the multiplier utilization is (1-s a %).

[0341] Figure 1MB Describe: (i) Figure 1MB The first configuration on the left side (this is Figure 1MA The configuration shown in Figure 1MA (ii) an enlarged view of the four rows of the circuit; Figure 1MB The second configuration of the center Figure 1MA an enlarged view of four rows of the circuit; and (iii) Figure 1MB In the third configuration on the right Figure 1MA A magnified view of four rows of the circuit of . In the first configuration, the lateral lookup multiplexer inputs come from the rows above and below, and there are no forward lookup multiplexer inputs from the same row. The first configuration typically has fewer lines than the other two configurations and extends the search for non-zero activation values ​​to other channels (i.e., to fewer rows), which may be advantageous if one channel tends to have consecutive zeros. Furthermore, if a factor of two speedup is targeted, two positions may be sufficient, and channels 1 and 16 have the same number of candidates in the h=2, d=1 configuration. The second configuration may be referred to as a "full multiplexing scheme." In this configuration, the lateral lookup multiplexer inputs come from the channels above and below, and the forward lookup input comes from the same channel at the next depth. In the third configuration, the lateral lookup multiplexer inputs are not used, and the forward lookup multiplexer inputs only come from the same channel (i.e., lateral lookup d=0). The third configuration has relatively low complexity (ie, less than half the multiplexers and wires are required) and may allow for simpler weight skipping at the expense of slightly reduced multiplier utilization.

[0342] Figure 1NA top-level diagram depicts a block 102 comprising an MR array 122 containing a grid of MUs 126 organized in eight MR columns 133 and 16 rows. Each MU 126 element includes a subscript (MU) corresponding to the row and column coordinates of the MU within the MR array 122. row,col ). The weight decompression unit 138 may receive compressed weights from the SRAM bank group 109 located locally in the block and decompress the weights during the process of writing the weights to the weight registers 127. The weights may be compressed to take advantage of sparsity in the weights, thereby reducing the memory used to store the weights and reducing the bus bandwidth used to send the weights to the multiplier unit 126. Alternatively, the weights may be stored uncompressed in the SRAM bank group 109. The IFM cache 139 may be used to cache IFM data to reduce the bottleneck effect of the IFM transmission structure 104, such as in Figures 1D to 1H , and ABU 141 may be used to implement skipping of zero-value activations (or "activation skipping").

[0343] Figure 1O Describes the hierarchy of neural processor control. The neural processor 100 may have controllable Figure 1A 102. The state machines or "control finite state machines" (control FSMs) or "control logic" of the various elements depicted in FIG. The control hierarchy may have two levels including a "global" level and a "local" level. In operation, the global control (GC) FSM 140 arranges the operations of the SRAM control FSM 142 and the block control FSM 144 (including starting the weight loading phase, and starting and controlling the calculation phase), where the SRAM control FSM 142 and the block control FSM 144 may be local control state machines. Because the blocks 102 support skipping zero value activations, the output rate of the blocks 102 may vary slightly depending on the actual sparsity of the IFM slices received by each block 102. Therefore, the calculations in the blocks 102 may run several clocks ahead or behind. Thus, the global control FSM 140 coordinates the operation of the tile control FSMs 144 to bring the outputs from the multiple tiles 102 back into synchronization to complete the reduction using the reduce structure 111 and send the final OFM result to the SRAM bank group 109 via the OFM transfer structure 106. This can be done, for example, using a small output FIFO 198 (also 179) within the ARU 167 ( Figure 1X ) to synchronize the outputs of multiple blocks 102, and in the extreme case that the block output FIFO 198 becomes full, synchronization of the outputs of multiple blocks 102 is achieved by throttling (stopping) the block 102 with the full output FIFO to allow other blocks to catch up.

[0344] Each of the plurality of SRAM control (SC) FSMs 142 may generate SRAM addresses and read / write signals for each SRAM bank within the SRAM bank group 109. Each of the plurality of tile control (TC) FSMs 144 may skip activation when activation has a value of zero. To prepare for operation, the host CPU (not shown) loads the starting address and size (height, width, depth, batch size) of each IFM and OFM tensor into the SRAM control FSM 142; loads the operation type (i.e., fully connected (FC) or convolution) and IFM, OFM, and weight data types into the global control FSM 140, and loads the IFM and OFM weight loop configuration, the order of IFM traversal, the number of IFM passes (explained later) and other compute mapping settings, activation functions and pooling (if any) selection; enables or disables partial result generation; loads weight tensor size (height, width, number of input and output depth channels); loads zigzag Z height (discussed below); and loads options for convolution padding and convolution stride into the configuration registers of the FSM. The host CPU also writes to registers associated with the IFM transfer structure 104, the OFM transfer structure 106, and the reduce structure (RF) 111 to configure connectivity according to the operating parameters, including the addresses of the IFM tensors and OFM tensors within each SRAM bank group 109. To begin the operation, the host CPU writes to registers in the global control FSM 140. The global control FSM 140 then signals the SRAM control FSM 142 and the block control FSM 144 to start.

[0345] In some embodiments, the global control FSM 140 controls the scanning within the convolution window, shifts the convolution window, and iterates over the IFM tensor to produce a stream of IFM slices. The global control FSM 140 sends the planar pixel (x, y) coordinates; the depth channel index d; and the IFM slice; along with a read signal, to the SRAM control FSM 142. Each of the SRAM control FSMs 142 adds a starting address, fetches the appropriate IFM data, and outputs the data to the IFM transfer structure 104. Typically, the IFM (and OFM) tensor size is too large to fit within a single SRAM bank group 109, resulting in the IFM (and OFM) tensor being subdivided into multiple portions to be stored across multiple SRAM bank groups 109. During computation, the global control FSM 140 arranges the IFM and (accordingly) OFM tensors to be traversed (fetched or stored in a specific sequence), while also generating on-the-fly reconfigurations of the IFM transport structure 104 and the OFM transport structure 106 to fetch IFM data from and write OFM data to the correct SRAM memory bank groups 109.

[0346] All tile caches 139 can receive data at substantially the same time. The global control FSM 140 calculates and provides to all tile control FSMs 144: (i) the address of the IFM cache 139 register file that holds each incoming data entry, and (ii) a write enable signal for writing data from the IFM transport structure 104 into the cache 139. The write enable signal is active when an IFM slice is coming from the SRAM bank group 109 via the IFM transport structure 104, and is inactive when an IFM slice is already cached. As the global control FSM 140 traverses the IFM layers (tensors) in a specific order, it also keeps track of which IFM slices required for a computation are already cached, and signals the SRAM control FSM 142 when to read data that is not already in the IFM cache 139. If the data is already cached in the tile cache 139, the global control FSM 140 deactivates the read signal, causing the SRAM control FSM 142 to skip the SRAM read. To simplify management of the IFM cache, each IFM slice from the IFM transfer structure is written to all associated target blocks at the same address in the IFM cache 139 (specified by a mapping, as discussed later) and their respective IFM caches, regardless of the block's target number. Since block computations run at slightly different rates due to uneven activation sparsity, the control logic for each block manages IFM cache 139 reads locally, independently of other blocks.

[0347] In some embodiments, the process of writing OFM results is similar to reading IFM values. However, due to activation skipping, the computation latency may vary. Each block control FSM 144 has information indicating when all columns in that block have completed computation. The block control FSM 144 of each block sends an ofm_ready signal to the global control FSM 140, which instructs the SRAM control FSM 142 to write the resulting OFM slice from the OFM transfer structure 106 to the SRAM memory bank at the appropriate (x, y, d) index in the OFM tensor. During the OFM tensor traversal, the global control FSM 140 generates OFM (x, y, d) slice coordinates in a manner similar to how it generates IFM (x, y, d) slice coordinates during the IFM tensor traversal. Once the computation is complete, the global control FSM 140 sends an interrupt to the host CPU.

[0348] As previously mentioned, due to activation skipping, tile 102 can produce, for example, up to two output results per clock. Therefore, IFM transfer structure 104 should be able to supply up to two IFM slices per clock to avoid reduced multiplier utilization. Therefore, tile control FSM 144 can inform global control FSM 140 of the amount of data remaining in the cache to be processed, so that global control FSM 140 can instruct SRAM control FSM 142 to resume fetching IFM data to avoid IFM cache underflow. When any of the tile IFM caches 139 becomes full, global control FSM 140 instructs SRAM control FSM 142 to suspend IFM tensor traversal, which includes reading IFM slices from SRAM 109 and writing IFM slices to the tile cache 139.

[0349] Reference Figure 1P In some embodiments, the IFM cache 139 includes sixteen lanes 170. Each lane contains a register file 169 with two input ports and two output ports. Because the system block 102 can process up to two activations per clock (when there are enough zero activations) due to activation skipping (and having two adder trees per MU column), dual ports can be used. To process activations faster (e.g., three IFM slices per clock), three input ports, three output ports, three IFM transport fabric bandwidths, three OFM transport fabric bandwidths, and three adder trees per MU column can be used.

[0350] Activations are input from the SRAM 109 at up to double the rate through the IFM transfer structure 104. The bank control FSM 144 tracks the amount of IFM data remaining to be processed in each way control logic 146. When any of the cache ways is about to become full, the bank control FSM 144 can notify the global control FSM 140 that at least one cache way is about to become full, and the global control FSM 140 can throttle (stop) the IFM reads controlled by the SRAM control FSM 142 to avoid overflowing one or more bank cache ways until cache space is freed.

[0351] When the convolution window scan is complete (and the window is translated to the next position) and when the IFM cycle is complete, the global control FSM 140 can also notify the block control FSM 144 so that the block can reset the column accumulator appropriately and not mix the convolution at one position with the convolution at the next position. The concept of an IFM cycle is defined and discussed later.

[0352] The block control FSM 144 generates the signals necessary to read IFM data from each cache way register file 169, including the read address and read enable for each register file's output port. Every clock cycle, unless a block 102 has completed processing and is waiting for other blocks to complete processing, the block control FSM 144 reads one or two data values ​​(from one or both cache ports, respectively) to make the results available for reduction by the reduce structure 111. Whether one or two bytes are read per single clock cycle depends on activation sparsity. The IFM buffer 124 within the ABU 141 checks whether activation is sparse and can notify the block control FSM 144, causing it to load one byte if one slot is free in the ABU IFM staging FIFO 165, and two bytes if two slots are free.

[0353] Figure 1Q The table in depicts the cache size sufficient to hold all IFM slices while performing convolution operations using convolution window sizes 1×1, 2×2, 3×3, and 4×4 to avoid repeated reads from the SRAM 109 as the convolution window slides plane by plane from one (x,y) position to the next. The data in the table assumes that the register file 134 of the multiplier unit 103 contains 18 weight registers and that the convolution window scans the input tensor in a "zigzag" sequence (as discussed below) because: since a single read from the SRAM 109 typically consumes significantly more power than a single read from the local register file 169, the "zigzag" scan sequence can be used to maximize the use of the IFM cache 139, thereby minimizing reads from the SRAM 109 and power consumption.

[0354] For example, where the zigzag scan value or parameter Z (discussed further below) is set to 2 and the MU 103 holds 18 weights (enough to hold two 3×3 8-bit convolution kernels or one 3×3 16-bit convolution kernel), the register file 169 should have a size of 20 bytes.

[0355] A neural network may have between 50% and 90% of its multiplicands where at least one multiplicand (activation and / or weight) is zero. This may be the case, for example, for an Inception v3 neural network after weight pruning has been applied. If MR block 102 can efficiently skip occurrences of multiplication by zero, MR block 102 may be able to process data in, for example, 100% - 80% = 20% of the time it would take to process without zero skipping, which is five times faster. As previously mentioned, in some embodiments, MR implementations may be configured for caching to allow for faster delivery of data (either multiplied or skipped) using more than two multiplicand inputs. In some block diagrams herein, for simplicity and clarity of explanation, only double input bandwidth (and activation buffer 124 depth of only two) is depicted. However, it will be understood that the depth of IFM activation buffer 124 may be greater than two, and for sufficiently sparse data, the corresponding speed increase (in a configuration without zero skipping) may be greater than a factor of two.

[0356] As in Figures 1B to 1H As described in the context of and as described in the following paragraphs, data sparsity can be used to achieve significant improvements in processing throughput through appropriate operation of the IFM cache 139 and the ABU 141. Figure 1R A 3×3 convolution window is depicted at the starting position within the IFM tensor (stored in SRAM 109) to initiate the input layer convolution. To start the layer convolution operation, nine IFM slices a0[0..15] to i0[0..15] are read from SRAM 109, transferred to the target tiles 102 via the IFM transfer structure 104, and written to the IFM cache 139 of each target tile 102. Figure 1S Another example of such data is depicted, where several elements are zero.

[0357] Figure 1T Depicts how data can be logically stored in the IFM cache 139 just before the layer convolution operation begins, where the values ​​are ordered in the order of arrival (from the SRAM) and do not necessarily show their arrangement according to the actual storage address of the values. Although the cache can store more activation values ​​to accommodate the movement of the convolution window, in this example, a 3×3 convolution is performed and for clarity, the figure depicts nine (3×3=9) 8-bit activation values. Similarly, Figure 1U Depicts from Figure 1T This example explicitly has some activations with zero value.

[0358] Figure 1V A single lane 171 of the activation broadcast unit 141 is depicted according to some embodiments. Each ABU lane 171 includes an IFM lane staging FIFO 173, which may be implemented using a register file; a lane multiplexer 163; lane control logic 146; and activation lane digital type conversion circuitry 148. Each ABU lane 141, along with the block control FSM 144 and other ABU lanes, can control activation skipping (i.e., skipping activation elements with a zero value) within that lane.

[0359] The activation path digital type conversion circuit 148 can also convert activations from a signed two's complement digital encoding to a signed 8-bit size format to simplify multiplier circuits that handle signed and unsigned data of various bit widths, including uint8, int8, uint16, int16, uint24 (24-bit unsigned integer), int24 (24-bit signed integer), uint32 (32-bit unsigned integer), int32 (32-bit signed integer), etc. Each ABU path 171 can also broadcast activations to the multiplier units 126 of the associated row within the MR column 133 as part of the signal set of the activation path 137.

[0360] Lane IFM staging FIFO 173 has two input ports, two output ports, and can be two-valued deep. The two input ports can be used to draw activations from IFM cache 139 at a rate of up to two activations (bytes) per clock cycle. Thus, when sufficient zero-valued activations are present, up to two activations can be processed per clock cycle as a result of having two adder trees in an MU column, a lane cache with two input ports and two output ports, and a staging buffer 173 depth of two. In some embodiments, if the IFM data is expected to be sparse enough to justify a larger number of activations per clock (e.g., three activations per clock), activations can be processed using a circuit having three adder trees per MU column, three lane cache input / output ports, three staging FIFO input ports, and a staging FIFO depth of three (where "staging FIFO" in this context refers to IFM lane staging FIFO 173).

[0361] The lane control logic 146 can broadcast a set of control signals as part of the signal group for activating lanes 137 to the relevant rows of multipliers 126 to inform the multipliers 126 whether the activation is zero. If the activation is zero, the control signal indicates which non-zero activation is being multiplexed to replace the zero (including from which lane and how deep in the staging FIFO (offset in the staging FIFO)) so that each multiplier 126 can select the correct weight and adder tree for multiplication. Similarly, the lane control logic 146 also controls the lane multiplexer 163 to multiplex activations from the correct staging FIFO 173 depth offset located in the correct adjacent IFM lane onto the activation lane 137.

[0362] Figure 1V An IFM lane hierarchical FIFO 173 is depicted with four output logical connections, sufficient to provide either of the two buffer activations to the adjacent lane above, either of the two buffer activations to the adjacent lane below, and both buffer activations to the lane activation multiplexer 163. Although Figure 1V A hierarchical FIFO 173 is depicted with four output logical connections, but because the FIFO 173 is only two values ​​deep in the depicted embodiment and therefore only holds two values ​​available for simultaneous output, the FIFO 173 has only two physical output ports.

[0363] Figure 1WA Depicts that the first two vectors in the IFM have been read in (also as Figure 1C 165 having four separate IFM lane staging FIFOs 173 (not 16 for clarity). In this state, the FIFOs can check which activation values ​​are zero and which are not. In some embodiments, each FIFO register has a zero detector (e.g., 8-input NOR logic). Each lane staging FIFO 173 reports which activations are zero to the corresponding lane control logic 146, which tracks which activations in that lane have been used (e.g., borrowed, which results in the creation of a queue such as Figure 1D ). The lane control logic 146 of each lane sends this information about the lane staging FIFO occupancy (including which activations are zero) to the block control FSM 144. The activations a0, a1, a2, and a3 undergo digital format conversion (if the activations are signed activations such as int8 or int16), become subdivided into 8-bit values ​​(if the activations are wider than 8, such as uint16, int16, uint24, int24, uint32, int32, etc.), and are broadcast to the corresponding row of multiplier units 126.

[0364] On the next clock cycle, the IFM staging FIFO 165 may contain Figure 1WB (and Figure 1D ). At this point, activations a0..a3 have been processed, and b0, b2, and b3 are being broadcast to their respective rows of multiplier units 126. Since b1 is 0, the lane for b1 is unused. The lane control logic 146 for each lane sends this information (which activations are zeros or "holes") to the block control FSM 144. The block control FSM 144 then makes decisions about: (i) which data to multiplex out (in Figure 1WB and Figure 1D In the example embodiment, the bank control logic 144 causes the cache to fetch two values ​​(instead of one) and (ii) to accept two values ​​(instead of one) from the FIFO, thereby skipping the entire FIFO column of holes and / or zeros. In the example embodiment, the bank control logic 144 causes the cache to fetch two values ​​(instead of one) and (iii) to accept two values ​​(instead of one) from the FIFO, thereby skipping the entire FIFO column of holes and / or zeros. In addition, the bank control logic 144 causes the cache to fetch two values ​​if multiple values ​​in the IFM lane staging FIFO 173 associated with the lane (as opposed to the entire column) include zeros and / or holes.

[0365] For example, lane 1 (output c1) has six output options: c0, c1, c2 (all zero) and b0, b1 (also all zero), and b2. Multiplexer 163 outputs one of these six options. Which option is output is determined by block control FSM 144. To enable this, multiplexer 163 can be configured to retrieve data from two FIFO columns in the lane above, two FIFO columns in the lane below, and two FIFO columns in the same lane as multiplexer 163. This can be done using, for example, Figure 1MA and Figure 1MB. As previously mentioned in the description of those figures, the ability to retrieve (and multiplex) data from one lane above and below may be referred to as "look-aside of 1," and the ability to retrieve (and multiplex) data from up to the second FIFO column from the right may be referred to as "look-ahead of 2." Each IFM hierarchical FIFO 165 column and lane combination may have a separate look-ahead value and / or look-ahead value associated with it; however, for clarity and simplicity, it may be assumed that all columns and lanes in an IFM hierarchical FIFO 165 have the same associated look-aside value and the same look-ahead value. Furthermore, other variations may be employed based on how many inputs each multiplexer 163 has and where those inputs are connected, which are not covered by the look-ahead and look-ahead concepts, including, for example, prohibiting inputs from the hierarchical FIFO from being sent to the same active lane and connecting lanes 0 and 15 in a more flexible manner to compensate for lanes 0 and 15 not having one of two adjacent lanes.

[0366] The number of side lookups and / or forward lookups can be greater than 2. Larger numbers can result in better performance by more optimally skipping zero activations, resulting in further reductions in block computation time. This benefit can be achieved because each path has more choices about where to retrieve non-zero activations when the number of side lookups and / or forward lookups is large. More choices of non-zero activations help to spread the non-zero activations more evenly across all paths, so that each path ends up having approximately the same number of non-zero activations, as opposed to some paths having more and others having fewer, potentially causing block processing to wait to complete until the path with the most activations has completed computation. As previously mentioned, spreading the non-zero activations can be achieved by pseudo-randomly shuffling the activation paths and associated weights, as described in a separate related disclosure (Attorney Docket No. 1535-467CON2).

[0367] Figure 1WC A configuration is depicted where the forward lookup is 2, the side lookup is 2, and the multiplexer 163 has 10 inputs for each FIFO column. In such an embodiment, the FIFOs may be two deep and, accordingly, may have two output ports.

[0368] Figure 1WD A configuration is depicted where the forward lookup is 3, the side lookup is 1, and multiplexer 163 has 9 inputs. In such an embodiment, the FIFO may be three deep and may have three output ports.

[0369] Figure 1WEA configuration is depicted where both the forward and side lookups are 3 and multiplexer 163 has 15 inputs. In such an embodiment, the FIFO may be three deep and may have three output ports.

[0370] Activating the broadcast unit 141 and the block control FSM 144 may similarly involve Figure 1E to Figure 1G For example, Figure 1E The diagram depicts the "hole" created by lane control logic 146 (in the lane where c1 was originally located) when c1 was already borrowed (multiplexed in the second column from the right) in the previous clock cycle. Each lane control logic 146 notifies block control FSM 144 which data cells in IFM staging FIFO 165 are zero or empty, allowing block control FSM 144 to appropriately control activation multiplexer 163. Block control FSM 144 determines multiplexer control to distribute activations to increase or optimize throughput. Optimal throughput is achieved when all lanes have the same number of non-zero activations, as opposed to having all lanes unbalanced, such that some lanes have many non-zero activations while other lanes (in the same block) have mostly zeros. In such an unbalanced situation, lanes with mostly zeros may complete their computation faster (i.e., may output all non-zero activations more quickly) than lanes with many non-zero activations, which can delay the completion of computation for that block and result in reduced multiplier utilization in zero-rich lanes.

[0371] As another example, in Figure 1G In the state depicted in FIG, the lane control logic 146 also receives multiplexer select signals from the block control FSM 144 to keep track of (i) the holes created and (ii) where the activations are multiplexed from. The lane control logic 146 then broadcasts this information to the multiplier units 126 of the relevant row so that when the activations have been multiplexed out of order (where Figure 1G The “ordered” in , for example, means that g0 from the activation buffer is output to the activation path labeled g0), each multiplier unit 126 in the row can multiply the out-of-order activation by its corresponding weight.

[0372] For example, as depicted, if an activation is multiplexed one way up from the second rightmost hierarchical FIFO column, then the corresponding weight by which the activation is multiplied is located in the multiplier unit of the one way up (in each column).

[0373] When the forward lookup is greater than 2 (for example, 3), and the activation is retrieved from the third column from the rightmost, the corresponding weight to be retrieved is forward 3-1=2, which means that if the ordered activation has been multiplied with the weight w[row,col,i], then the appropriate weight to be multiplied is changed to w[row,col,i+2].

[0374] Figure 1H The figure depicts the situation (which is advantageous from a throughput perspective) when activations are multiplexed (advanced out of order) such that an entire FIFO column (all 16 lanes) becomes empty (containing only zeros or holes). Because two FIFO columns are consumed simultaneously, the rightmost all-zero column is skipped (discarded), and the second column from the rightmost is broadcast and exhausted for computation. The tile control FSM 144 detects this situation and instructs the IFM cache 139 to load two values ​​into the FIFO. This reduces computation latency in the tile by one clock cycle.

[0375] Figure 1X The accumulation and return unit (ARU) 167 is depicted. The role of the ARU 167 is to complete the dot product calculation and apply the activation function (when applicable) to produce a completed output feature map (OFM), which is ready to be transferred back to the SRAM for storage via the OFM transfer structure. Figure 1N As shown in FIG, each MR column 133 includes two ARUs 167, one ARU 167 for each adder tree 128A and 128B.

[0376] ARU 167 has two inputs, one from the local adder tree 128A (or 128B) and one from the reduction structure 111. As explained later, at the heart of each ARU 167 are adders 181 and accumulators 130A, which perform dot product computations by accumulating (over time). To complete the OFM computation, the fully reduced dot product may (optionally) be truncated (via rounding) using unit 187, scaled by factor 191 using multiplier 189, summed with OFM bias term 195 using adder 193, and passed through activation function 197. Activation function 197 may be a module that may support one or more activation functions, such as rectified linear units (ReLUs), sigmoid functions, hyperbolic tangents, and the like. If (for reasons explained later) the dot product reduction cannot be completed, the partial dot product, or just the “partial product,” from the accumulator 130A (130B) may bypass the scaling, bias, and activation functions on its way to the OFM delivery structure 106 via multiplexer 199 and output FIFO 198. Multiplexer 183 bypassing adder 181 may allow adder tree values ​​to be loaded directly into the accumulator 130A, e.g., to initiate accumulation.

[0377] Multiplexer 174 may select the input source for ARU 167, which is used for "returning" (scaling, biasing, and activation application, along with partial paths, as applicable) between (i) adder trees within the same (local) block where ARU 167 is located and (ii) reduction structure 111, which includes a configurable adder tree that combines local ("intra-block") adder trees 128A and 128B into an even larger ("inter-block") adder tree capable of reducing multiplier unit products from multiple blocks (e.g., from 32 or 64 or 128 or 256 multiplier units).

[0378] Because the block control FSM keeps track of which lanes and adder trees in each MR column 133 are used to obtain each partial IFM reduction, the block ARU 167 is controlled by the block control FSM 144. The ARU 167 has two outputs, including one connected to the OFM transfer structure 106 via FIFO 198 and on-the-fly pooling logic 196, and one connected to the reduction structure 111 via FIFO 179. The block control FSM 144 also keeps track of the status of the output FIFOs 198 and 179. Because each block 102 performs computations at a slightly different speed due to the unpredictability of zero-activation skipping, each output FIFO 198 and 179 is used to restore synchronization of block outputs by delaying outputs from blocks that eventually finish running earlier (faster) than other blocks. It may be necessary to synchronize the block outputs with FIFO 179 because the block outputs may undergo further reduction in reduce structure 111, which can be considered as a set of additional adder tree stages and therefore may require its inputs (from the block) to arrive in parallel and synchronously. Similarly, it may be necessary to synchronize the block outputs with FIFO 198 so that all paths of an OFM slice are output to the OFM transfer structure simultaneously. Output FIFOs 198 and 179 sizes of four or fewer entries may be sufficient in many cases. In the event that the output FIFO 198 or 179 is about to overflow in one or more blocks, the block control FSM 144 may halt computation until the output FIFO 198 or 179 is empty. The output FIFO 198 or 179 may have two input ports to merge the results from two adder tree paths (A and B).

[0379] Finally, the block control FSM 144 and the SRAM control FSM 142 work together to read data from the output FIFO 198 , perform reduce structure processing, and transfer the results through the OFM transfer structure 106 for storage in the SRAM 109 .

[0380] The activation number type converter 135 works with the accumulation and return unit 167 to support signed and unsigned input and output data types of various bit widths (including the ability to arbitrarily use one data type for activations and another data type for weights, referred to below as "mixed data types").

[0381] In some embodiments, the following data types may be used: int8, uint8, int16, uint16, int24, uint24, int32, and uint32 for IFM data, OFM data, and weight data. As described below, IFM data and weight data types may be freely mixed. For example, convolutional or fully connected layer computations may be performed using uint8 activations and int8 weights, or int8 activations and int8 weights, or int16 activations and int8 weights, or int16 activations and int16 weights, etc. OFM data types including uint8, int8, uint16, int16, uint24, int24, uint32, int32, etc. may also be arbitrarily selected by applying a combination of scaling, rounding, and activation function selection.

[0382] The activation may be prepared for operation as follows. The activation may be stored in SRAM 109, for example, as int8 or uint8 or int16 or uint16 as specified by the user. The IFM data may be retrieved into cache (i.e., into IFM cache 139) and then broadcasted by the activation broadcast unit 141, as Figure 1L As shown in , the activation broadcast unit 141 includes an activation digital type converter 135. As a first step, if the activation is quantized using a "zero point" offset quantization as used in Google TensorFlow, the type converter 135 adds a "zero point" offset to the activation. The digital type converter 135 then prepares the activation by applying a suitable transformation (or "conversion"), which enables multiplication using data types wider than 8 bits (e.g., 16-bit weights and / or 16-bit activations, signed or unsigned) to be performed using the 8-bit unsigned multiplier 126. Figure 1K As shown in , for each lane, the activation broadcast unit 141 broadcasts the 8-bit absolute value of the activation, act_abs[7:0], accompanied by a 1-bit sign, sact. The transformation applied by the activation digital type converter 135 converts the int8 / unit8 to "sign and 8-bit absolute value". If the input activation is uint8, the type converter 135 sets the output broadcast 8-bit absolute value equal to the input uint8 value (i.e., no transformation) and sets the broadcast sign to zero (which indicates that a non-negative value is represented).

[0383] If the input activation data type is int8, the activation number type converter 135 sets the output absolute value to the absolute value of activation and sets the output sign to 1 if activation is negative and to 0 otherwise.

[0384] The weights may be prepared for the following operations. The weights may be stored in the SRAM 109 as int8 or uint8 or int16 or uint16 as specified by the user. When the weights are loaded into the MU registers, the weights are transformed in the weight decompression unit 138 (using the same transform as used to activate the digital type converter 141). The weights are stored as 8-bit absolute values ​​and 1-bit sign. Figure 1K and Figure 1N , when the weights are loaded from the SRAM 109 into the MU register and input into the multiplier unit 103 through the vertical weight load bus 101, the values ​​represented as int8 and uint8 are converted into 8-bit absolute values ​​wt_abs_Id_in[7:0][C] and 1-bit sign representation swt_in[C].

[0385] 8-bit multiplication can be performed as follows. Multiplier 126 can be an unsigned 8-bit by unsigned 8-bit multiplier. The multiplication operation can take activations and weights represented by 8-bit absolute values ​​and 1-bit signs as input. Multiplier 126 then multiplies the two 8-bit absolute values ​​and XORs the two signs. If the product of the two 8-bit absolute values ​​is zero, the output sign is set to zero. The output of multiplier 126 (16-bit absolute value accompanied by its sign) is then converted to int17 and transmitted to adder tree 128A (or 128B). Subsequently, adder tree 128A (or 128B) reduces the signed int17 value received from the column multiplier unit and transmits the signed sum to ARU 167 associated with the adder tree.

[0386] In some embodiments, 16-bit and 8-bit input data types can be mixed as follows. 8-bit weights and 8-bit activations can be multiplied in one cycle. In some embodiments, all possible combinations of 8-bit digital data types are supported (e.g., uint8 activations × int8 weights, int8 activations × int8 weights, uint8 activations × uint8 weights, and int8 activations × int8 weights). Two cycles may be used to determine or calculate: (i) the product of a 16-bit weight and an 8-bit activation or (ii) the product of a 16-bit activation and an 8-bit weight. Four cycles may be used to determine or calculate the product of a 16-bit activation and a 16-bit weight. All possible combinations of 8-bit and 16-bit digital data types may be supported (e.g., uint16 activations × int8 weights, int16 activations × int8 weights, uint16 activations × int16 weights, uint8 activations × int16 weights, int16 activations × int16 weights, etc.).

[0387] In some embodiments, 16-bit activations may be processed as follows. When the activation is uint16 or int16, the type converter 135 may prepare the data by applying a transform (similar to the 8-bit transform described above). Values ​​in uint16 or int16 format may be transformed into 16-bit absolute value and sign format. If 8-bit (uint8 or int8) weights are used, the first cycle output of the activation broadcast unit 141 may be the least significant byte (LSB) of the 16-bit absolute value and sign resulting from the transform (for multiplication with the 8-bit weights), and the second cycle output of the activation broadcast unit 141 may be the most significant byte (MSB) of the 16-bit absolute value and sign resulting from the transform (also for multiplication with the 8-bit weights). The two partial product results (both converted to signed int17) can then be sent to the column accumulator 130A or 130B (as usual, via the column adder tree 128A or 128B to the column accumulate and return unit 167), except that the most significant byte product can also be shifted up by 8 bits using the sign extension shift 175 (and multiplexer 177) before being added.

[0388] If the weight is 16 bits (uint16 or int16), four clock cycles can be used to perform the multiplication of the (16-bit) activation and the weight. The first cycle output of the activation broadcast unit 141 can be the least significant byte of the 16-bit absolute value and sign produced by the transformation of the activation, and the multiplier 126 can be input with the least significant byte of the 16-bit absolute value of the weight at the same time, and the first multiplication can be performed. During the second cycle, the product of the same part of the activation (i.e., the least significant byte of the 16-bit absolute value and sign produced by the transformation of the activation) can be input to the multiplier again together with the most significant byte of the 16-bit absolute value of the weight, and the second multiplication can be performed.

[0389] The third cycle output of the activation broadcast unit 141 can be the most significant byte of the 16-bit absolute value and sign produced by the activated transformation, the multiplier can be simultaneously input with the least significant byte of the 16-bit absolute value of the weight, and the third multiplication can be performed. During the fourth cycle, the product of the same portion of the activation (i.e., the most significant byte of the 16-bit absolute value and sign produced by the activated transformation) can be input to the multiplier 126 again together with the most significant byte of the 16-bit absolute value of the weight, and the fourth multiplication can be performed. In addition to the sign-extended upshifter 175 and the multiplexer 177 that can be used to pre-shift the second and third partial products by 8 bits before addition and pre-shift the fourth partial product by 16 bits, all four partial product results can be output to the column accumulator 130A (or 130B) (as usual, via the associated adder tree 128A or 128B for the column to the accumulation and return unit for the column) and added together.

[0390] Performing a convolution operation involves traversing an IFM tensor stored in SRAM 109 and streaming the contents of the IFM tensor to one or more blocks 102 as a series of IFM slices transmitted through the IFM transmission structure 104. The IFM tensor has three dimensions (and a batch index (which is omitted for clarity of explanation)) using coordinates represented as (x, y, d), where the x and y indices correspond to the plane coordinates of the activations, and the index d corresponds to the depth channel. The neural processor 100 traverses the IFM tensor by looping through the (x, y, d) index values ​​in a specific order. As used herein, looping over the (x, y) coordinates represents a "planar" traversal, and looping over the d coordinate represents a "depth-wise" traversal.

[0391] The following sections describe a plane traversal including the use of the IFM cache 139. Figure 1N , the IFM transport fabric 104 can be connected to the IFM tiles 102 via IFM caches 139. Each tile 102 can have one IFM cache 139, with each IFM cache 139 being local to the associated tile. Utilizing an IFM cache 139 (per tile) helps reduce the number of reads from the SRAM 109. Reducing the number of reads from the SRAM 109 can be beneficial in three ways: (i) reducing the contribution of the SRAM 109 to the overall power consumption of the neural processor, (ii) reducing the chances of SRAM read or write stalls, and (iii) reducing the amount of traffic flowing through the IFM transport fabric 104.

[0392] The SRAM power consumption reduction aspect may be of interest when the SRAM 109 consumes relatively high power compared to the flip-flop register power consumption, which may occur in practice. The SRAM stall aspect may be particularly important when the number of SRAM banks located in each SRAM unit 109 is low compared to the number of input / output (I / O, read or write) operations to be performed. For example, as will be described later, each SRAM bank group unit 109 may contain four SRAM banks and thus be able to perform up to four I / O operations simultaneously (per clock cycle). These I / O operations may be IFM slice reads, writes to one or two OFM slices, partial result reads or writes, and slice reads or writes requested by the AXI interconnect 114.

[0393] When more than four such I / O operations must simultaneously access data residing in the same SRAM bank 109, or one or more I / O operations must access data in the same bank, a bank access conflict may occur, causing the SRAM bank arbitration logic to stall AXI accesses or IFM data fetches or OFM data writes or partial result I / O, potentially also causing computational stalls. Therefore, the IFM cache 139 can reduce IFM reads from the SRAM cells 109, thereby reducing the likelihood of these types of stalls.

[0394] As will be discussed in more detail later, in the case where the weight kernel size is particularly large, the computation may be split into multiple parts, and partially completed computation results ("partial results" or "partials") may be stored in SRAM 109. In order to maintain acceptable computational accuracy, partial results typically have a relatively long bit width (e.g., 4 bytes or 6 bytes) compared to IFM data and OFM data. Writing partial results with long bit widths to SRAM and reading partial results with long bit widths from SRAM consumes correspondingly higher SRAM bandwidth, which may increase the likelihood of SRAM bank access conflicts and, therefore, the likelihood of AXI and / or computation stalls. Therefore, the IFM cache 139 may help alleviate SRAM I / O bottlenecks (particularly for computations using partial results).

[0395] When communication bus area is at a premium, reducing IFM transport fabric traffic may be of interest. Figure 1PThe IFM transfer structure 104 depicted in FIGURE 1 can transfer up to two IFM slices per clock to the IFM cache 139. When the IFM transfer structure transfers N slices to the IFM cache 139 simultaneously (e.g., per single clock), the IFM transfer structure 104 can be referred to as having a width of "N slices." By caching IFM slices locally within each tile, the IFM transfer structure 104 can remain idle when the IFM slices required for a computation are already locally cached by the tile and readily available for processing. An IFM transfer structure 104 with idle cycles (having less than 100% utilization) allows the idle cycles to be used to transfer additional IFM slices, thereby increasing the overall "effective" IFM transfer bandwidth by more than a factor of two. Thus, when the area of ​​the IFM transfer structure 104 is at a premium, the width of the IFM transfer structure 104 can be reduced from, for example, two slices to one slice, while still maintaining the overall IFM transfer bandwidth at a factor of one or greater, and sometimes at a factor of two or greater.

[0396] As will be seen below, the IFM cache 139 provides the greatest benefit for convolution operations with a kernel planar width and / or height greater than 1. "Depth-wise" convolutions (convolutions with a kernel width and height both equal to 1) and fully connected computations can also benefit from the IFM cache, but typically only in rare cases.

[0397] To understand the solution provided by one embodiment (referred to herein as a "zigzag" plane traversal, which is designed to increase IFM cache hit rates), as shown in FIG. Figures 2AA to 2AD As depicted in

[15] , we first consider a simple, naive planar-wise traversal of the IFM tensor using a 2×2×16×16 weight kernel. Here, 2×2 represents the planar height and width of the weight kernel, 16 represents the IFM depth (i.e., one slice), and 1 represents the OFM depth. However, for clarity of explanation, the convolution can be considered as purely planar (i.e., 2×2×1×1). Figure 2AA Depicts a convolution operation starting with a convolution (kernel weight) window placed at the top left corner of the IFM tensor. After computing the 2×2 convolution at that location, the window slides one pixel to the right. The computation is repeated by sliding until the window reaches the top right corner of the IFM tensor. Figure 2AB As depicted in , once in the top right corner, the convolution is computed and the convolution window now slides down one row instead of to the right. Figure 2AC The same computation and sliding steps are further repeated as depicted in , except that the convolution window now keeps sliding to the left until Figure 2ADAs depicted in , it reaches the left edge of the IFM tensor, where it slides down one row again. Repeating these steps ultimately results in a complete plane scan (traversal) of the IFM tensor. Because the window slides primarily horizontally (i.e., the inner loop loops over the x-coordinate), such a scan can be referred to as a horizontal scan (as opposed to a vertical scan).

[0398] Consider using IFM cache 139 with Figures 2BA to 2BL At the beginning of the convolution operation, the IFM cache 139 is cleared, the 2×2 convolution window is placed at the top left corner of the IFM tensor, and then the four IFM values ​​required to calculate the convolution at that starting position are retrieved. Figure 2BA As depicted in , the first of the four IFM values ​​is retrieved from the top leftmost position in the IFM tensor. This position may be referred to as being at row 0, column 0. Because the cache has been flushed, the IFM value at row 0, column 0 must be retrieved from SRAM 109 rather than IFM cache 139, resulting in Figure 2BA A cache miss is marked with an "M" in the . IFM value is cached once it is retrieved. Figure 2BB Depicted is the second IFM value (of four values) retrieved at row 0, column 1. The cache does not contain a value associated with that location (row 0, column 1), resulting in another cache miss marked by an "M." The light shading of the row 0, column 0 location indicates that the IFM value retrieved in the previous step was already cached. Figure 2BC and Figure 2BD The retrieval of the remaining two IFM values ​​is depicted, each resulting in a cache miss. At this point, all four IFM values ​​have been retrieved, the convolution computation at the current position can be completed, all four IFM values ​​have also been cached, and the convolution window can slide one column to the right.

[0399] Figures 2BE to 2BH Depicts retrieving four more IFM values ​​to compute the convolution at the new location. Figure 2BE In , retrieving the IFM value at row 0 column 1 results in a cache hit (H), thus avoiding an SRAM read. Similarly, Figure 2BG Another cache hit is depicted at row 1, column 1, while retrieving the other two IFM values ​​each results in a cache miss.

[0400] When the convolution window continues to slide, as Figures 2BI to 2BL (as well as Figures 2BE to 2BH), the leftmost previously cached IFM values ​​will not participate in the computation for an extended period of time, or not participate at all, until the convolution window slides all the way to the rightmost edge of the IFM tensor, slides down one row, and slides all the way back to the cached values. Therefore, once the convolution window slides, these values ​​can be cleared from the cache to keep the cache size small.

[0401] Figures 2BI to 2BL Depicts retrieving the next four IFM values ​​to compute the convolution at the next position (one step to the right), resulting in two cache hits and two cache misses. Figure 2BM As shown in , because two-quarters of the IFM values ​​(marked with light shading) are reused each time the convolution window slides one step to the right, horizontally caching IFM values ​​during a 2x2 convolution results in a cache hit probability (rate) of approximately 50%. More generally, a convolution using a H x W planar kernel size in conjunction with a horizontal cache and assuming a cache of sufficient size results in a cache hit rate of H*(W-1) / (H*W). A cache size sufficient for such a convolution might be (W-1) bytes per channel per tile. However, as will be explained later, the neural processor 100 can also use "IFM weight cycling" to accumulate multiple IFM channels into a dot product by sequentially cycling the weights of the multiplier units during the dot product computation. Therefore, as will become clear later, in the most general case, the maximum cache size is equal to the number of weights stored in the MU weight register file 127 per channel per tile (for an 8-bit weight data type, this number is equal to 18).

[0402] exist Figures 2BA to 2BM Keeping the cache size relatively small requires aggressive purging of cached values. Figure 2BM , when the convolution window slides over row R (row 2), the IFM value from the previous row R-1 (row 1) has long been cleared from the cache (indicated as a cache miss "M" at row 1, column 2). In order to increase the cache hit rate to above H*(W-1) / (H*W), one may consider caching the values ​​of, for example, one or more rows of the IFM tensor. However, caching an entire IFM tensor row would require an increase in the cache size, so that the cache size generally becomes a function of the IFM tensor width. Since the IFM tensor width is typically unknown at ASIC design time, and since the IFM tensor width can be relatively large, caching IFM rows appears to be expensive in terms of silicon area and is therefore undesirable. The same reasoning applies to the symmetric case when the convolution window is scanned primarily vertically (i.e., the inner loop in the plane coordinates iterates over row numbers) rather than horizontally.

[0403] In contrast to simple, naive plane scans, some embodiments perform a plane traversal of the IFM tensor in a "zig-zag" shape during the convolution operation. Zig-zag plane traversal can help increase cache hit probability while still keeping cache size small. Figure 2C Depicts a down-right-up-right zigzag path along which the convolution window may shift (slide) in such an embodiment. Unlike a simple, naive horizontal traversal, Figure 2C The convolution window in slides to the right after computing two convolutions (in vertically adjacent positions) instead of just one. Thus, a single complete left-to-right, edge-to-edge sweep of the IFM tensor by the convolution window produces two rows of convolution results, as opposed to a single row of results from a simple, naive horizontal pass.

[0404] In the more general case, the zigzag traversal can be parameterized using a "Z number" corresponding to the number of output rows processed in a single horizontal IFM tensor sweep. For example, in Figure 2C , the Z number is equal to 2. As we will see later, a higher Z number results in a higher cache hit rate.

[0405] exist Figure 2C In , a zigzag traversal that produces two rows of results per single horizontal scan can be thought of as performing a naive horizontal traversal on an IFM tensor that is twice as wide but half as high. More generally, the zigzag traversal path can be thought of as "unfolding" a single (horizontal) scan of length H*Z columns using a total of H / Z scans to complete the IFM tensor convolution, where H and W are the height and width of the IFM tensor, respectively. For example, in Figure 2C In the example, Z=2, so instead of traversing the HxW IFM layer with a simple, naive scan, the length of the arrow path is approximately H*Z=W*2, so the logical IFM layer width becomes W*Z=2W, and the logical IFM layer height becomes H / Z=H / 2. A simple, naive horizontal line is equivalent to a zigzag traversal with Z=1.

[0406] Figures 2DA to 2DD A zigzag traversal using Z=1 is depicted for the first position of the convolution window. Retrieving all four IFM values ​​results in a cache miss, causing four SRAM reads to occur. Figures 2DE to 2DH As depicted, for the next position of the convolution window, fetching another two IFM values ​​results in a cache miss, while the other two IFM fetch positions overlap with the previous position of the convolution window, thus resulting in two cache hits.

[0407] like Figure 2DI to Figure 2DLAs depicted in , for the next position of the convolution window, two IFM values ​​are cache misses, and two overlap with the previous position of the convolution window, both resulting in cache hits. Figures 2DM to 2DP As depicted, for the next position of the convolution window, one IFM value is a cache miss, and three overlap with the previous position of the convolution window and are cache hits. Thus and with further reference to Figure 2DQ to Figure 2DX ,The use of zigzag paths significantly improves the cache hit to cache miss ratio.

[0408] Figure 2E is a table showing the actual number of SRAM reads associated with a zigzag traversal relative to the number of SRAM reads in an ideal cache (i.e., a cache with infinite capacity and that never flushes any values). Figure 2E The table in is a measure of the efficiency of a zigzag traversal. The table assumes that the cache size is sufficient for a given Z when performing a single sweep (i.e., the values ​​from the previous sweep are cleared). Lower numbers in the table correspond to higher efficiency, with 1.0 being the ideal case. The Conv Size represents the planar dimension of the square weight kernel. For example, a 3x3 convolution with a zigzag traversal of Z=2 results in 2xmore SRAM reads than a 3x3 convolution using an ideal cache. However, a 3x3 convolution using a zigzag traversal with Z=1 (i.e., a simple, naive (e.g., horizontal) traversal) results in 3x more SRAM reads than using an ideal cache. Thus, as calculated by the formula described below, in this case, a zigzag traversal with Z=2 reduces the SRAM read count by a factor of 3 / 2=1.5 (3 / 2=1.5x) compared to a simple, naive traversal, while the cache size for the zigzag traversals with Z=2 and Z=1 remains small compared to a simple, naive traversal. Note that larger Z numbers yield larger SRAM read count savings. For example, for a 3x3 convolution, increasing cache Z to 4 results in a 3 / 1.5=2x (3 / 1.5=2x) SRAM read savings.

[0409] Figure 2F A table depicts the average expected IFM SRAM reads per clock for feeding the IFM cache, assuming one IFM slice is processed per clock. For example, a 5x5 convolution with cache Z=4 performs an SRAM read only 8% of the time on average, compared to 100% of the time (i.e., every clock) without the cache and 20% of the time with cache Z=1 (i.e., a simple, naive traversal scheme).

[0410] Figure 2GA to Figure 2GB Derivation of cache hit / miss counts and cache sizes is depicted. Zigzag traversal involves the repetition of a two-step sequence where the convolution window slides vertically by Z-1 rows and then sideways by one column. For simplicity, special cases at the edges of the IFM tensor are ignored and the convolution window of planar size W x H is slid sideways ( Figure 2GA ) sliding one column results in H cache misses (marked "m") and H*(W-1) hits. The next vertical ( Figure 2GB The step of sliding Z-1 rows down in (Z-1) results in (Z-1) cache misses and (Z-1)*(H*W-1) cache hits.

[0411] Accordingly, once the convolution window has slid horizontally by one column, the convolution window can use the previously cached values ​​within the kernel window (cached during the previous vertical translation, in Figure 2GA Marked as "c" in the current calculation. Because the window will start to slide vertically (at Figure 2GA , down), so outside the kernel window (in Figure 2GA The previously cached value (marked as "c" in the middle, below) should also remain in the cache to be used. In addition, after the convolution window slides down by Z-1 rows, right by one column, and back up, the value obtained from SRAM (marked as "m") should also be added to the cache to be used in the calculation at the current position. Next, each time the convolution window slides down by one row, one cache value (top left) can be cleared and one value from SRAM can be added (marked as "m"). Therefore, for Figure 2GB Counting the number of "c" tags in the cache, the required cache size is (H+(H+Z-1)*(W-1)).

[0412] As explained later, if weight rotation is used, the cache size can be increased by the same factor as the number of kernels stored simultaneously in any tile. As mentioned above, when the convolution kernels are small, the system can store multiple planar kernels in each MU 103. For example, if MU 103 has 18 weight registers and the convolution is 2x2, four 2x2 kernels can be stored in MU weight registers 127. For example, the dot product of IFM data with 64 channels 0...63 can be calculated into OFMs 0...7 by looping over the four stored kernels over time. The system can take the IFM slice holding channels 0...15, multiply it by the first kernel (of four), and store the result in the tile's accumulator; take the IFM slice holding channels 1...31, multiply it by the second 2x2 kernel (of four), and add the result to the already stored accumulator value; and repeat a third and fourth time. These IFMs can also be cached, resulting in a corresponding increase in cache size. The IFM cache size has an upper limit, regardless of the choice of plane switching method (naive or zigzag or some other method), however, the IFM cache size is a function of the size of the multiplier unit weight register file 127. This is because each cached IFM slice must have a corresponding weight in the weight register file to be multiplied, and the weight register file itself is limited (e.g., limited to 18 weights). Note that the same reasoning also translates to the IFM cache size having a lower limit equal to the weight register file size.

[0413] Therefore, the IFM cache size should be set to (H+(H+Z-1)*(W-1)-1) and take the maximum value of MU_WEIGHTS of all possible supported H and W combinations, where MU_WEIGHTS is equal to the size of the multiplier unit weight register file 127 (e.g., 18). For example, if the neural processor 100 has 18 weights per multiplier unit 103, supports zigzag traversals of all natural H and W for Z=2 and kernel weight plane sizes such that H*W≤18 (e.g., 1x1, 1x2, 2x1, ..., 4x4, 9x2, 2x9), then the IFM cache size is the maximum of (1+(1+2-1)*(1-1)-1)=0, (1+(1+2-1)*(2-1)-1)=2, (2+(2+2-1)*(1-1)-1)=2, ..., (4+(4+2-1)*(4-1)-1)=18, (2+(2+2-1)*(9-1)-1)=25, (9+(2+2-1)*(2-1)-1)=11 and 18, i.e., 25.

[0414] In some embodiments, the MU weight register file capacity is equal to 18 8-bit weights (uint8 or int8) or equivalently, equal to 9 16-bit weights (uint16 or int16). When the IFM data is 16 bits (uint16 or int16), the IFM cache can store 16-bit IFM data by allocating two bytes per 16-bit IFM. Thus, similar to how the MU weight register 127 is capable of storing nine 16-bit weights, the IFM cache 139 can store nine 16-bit IFM values. Zigzag (and simple, naive) plane traversal can be applied to 16-bit IFM values ​​in a manner similar to how it is applied to 8-bit values. In this case, the cache size calculation described above should also include additional W and H terms in the maximum function (such as (H+(H+Z-1)*(W-1)–1)*size_of(IFM_DATA_TYPE), where size_of(IFM_DATA_TYPE) represents the byte size of the data type of the IFM value (e.g., 3 bytes for 24-bit IFM values ​​and 4 bytes for 32-bit IFM values). In cases where the IFM data type is 24-bit, 32-bit, or larger, zigzag (and simple, naive) caching can be used, however, it is recommended to increase the size of the MU weight register file 127 (and the size of the IFM cache 139) to 3 x 3 x size_of(IFM_DATA_TYPE). As explained later, this ensures that convolutions can be performed on weight kernels of the popular 3x3 plane size without resorting to using potentially undesirable partial results.

[0415] As previously described, the global control FSM 140, SRAM control FSM 142, block control FSM 144, and lane control logic 146 work together to perform appropriate control of SRAM IFM fetches, transfers of IFM slices across the IFM transport fabric 104, caching of IFM values ​​in local blocks 102, retrieval of cached IFM values ​​(typically at a slightly different rate for each active lane), and resynchronization of OFM results between blocks 102. To configure the IFM and OFM plane traversal, the host CPU loads calculation parameters, including the zigzag height Z, into the global control FSM 140 and SRAM control FSM 142. The global control FSM 140 then coordinates the SRAM control FSM 142 and block control FSM 144 to start and execute the calculations.

[0416] As the convolution window traverses the input and output layers in a zigzag plane manner, each accumulation and return unit 167 can receive OFM values ​​to advantageously calculate pooling on the fly without having to save the pre-pooling results to SRAM and read the values ​​back later to apply pooling. Figure 2HA to Figure 2HDAs shown, when the pooling windows do not overlap, ARU 167 can perform pooling by not sending each convolution OFM result, but instead holding the convolution results in registers in pooling logic 196 until each pooled output is complete. ARU 167 writes the pooled output to SRAM 109 only after each pooled output is complete. For max pooling, an output register of ARU 167 registers can hold a maximum value, which is compared to the convolution output and updated when the latest OFM output exceeds the current maximum. Once the pooling window slides, ARU 167's output register is reset to restart the max operation. For average pooling, ARU 167's accumulator continues to sum the OFM outputs until the pooling window is about to slide. The accumulator is then multiplied by 1 / (POOLING_WIDTH * POOLING_HEIGHT) to calculate the average, which is rounded and written to SRAM 109. Once the pooling window slides, the accumulator is reset to restart averaging.

[0417] For example, Figure 2HA A zigzag plane traversal with Z=2 is depicted in conjunction with 2x2 plane pooling, where the IFM layer is traversed in such a way that the OFM values ​​(in each pooling window) are calculated sequentially. Because the output of the ARU 167 generates four OFM values ​​to calculate each pooling one after another, the ARU pooling logic 196 can take the maximum of the four consecutive results to calculate the maximum pooling. Figure 2HB Depicted is a zigzag plane traversal and 3x3 plane pooling with Z = 3. Since the Z value is equal to the pooling kernel height, traversing the IFM layer in a zigzag manner naturally results in generating OFM data within each pooling window in an order suitable for maximum pooling and average pooling. Figure 2HC An additional illustration is provided where Z=H=4, where H represents the height of the pooling kernel.

[0418] Figure 2HD The case is depicted when the Z value does not match the height of the pooling kernel, such that Z=4 and the height of the pooling kernel is 2. In this case, the pooling logic 196 may subdivide the pooling into two regions (as depicted, upper 2x2 and lower 2x2), and use an additional register to temporarily store data from one of the two pooling regions ( Figure 2HDIn the embodiment of the present invention, the zigzag pooling window width and stride are respectively the height of the zigzag traversal and the height of the zigzag pooling window. ...

[0419] In cases where pooling windows overlap vertically, making running pooling problematic, and / or where custom pooling (beyond max and average pooling) is desired, pooling can be accomplished by (i) placing read-modify-write logic near the SRAM bank 109 (not depicted) and / or (ii) reading the SRAM out to the CPU, GPU, DSP, or other type of compute core via an AXI interface, performing pooling, and writing the results back to the NPU SRAM via the AXI interface. The custom read-modify-write logic near the SRAM bank 109 can also be reused to efficiently add partial results without sending the partial results back to the tile.

[0420] To configure the neural processor 100 to perform a particular operation (e.g., a convolutional or fully connected layer computation), the IFM and OFM tensor sizes should be considered and the computation “mapped” onto the available hardware in conjunction with the parameters of the operation (e.g., operation type, stride, etc.). Each individual block 102 may only have a fixed number of 16 IFM depth channel inputs and 8 OFM depth channel outputs, while the number of depth channels in a deep learning neural network model layer varies and typically far exceeds 16 and 8. A mapping algorithm can be run offline (during compile time as opposed to run time) to break down the large IFM and OFM tensors into parts (sub-tensors), assign these parts to the available blocks for computation, and generate a description (configuration) of how the outputs from the available blocks can be reassembled to complete the computation. As will be explained in more detail below, the mapping algorithm can also determine the order in which the IFM (and accordingly OFM) tensors are traversed, both in the planar direction and, in particular, in the depth direction. Because there may be multiple solutions to a particular mapping problem (i.e., for given IFM, OFM and weight tensor sizes and operation parameters), the mapping algorithm may also accept parameters indicating whether the solution is optimized for lowest power, lowest SRAM size, lowest computational latency (achieved by maximizing multiplier utilization) and / or a combination of these (e.g., lowest power given a fixed available SRAM size).

[0421] Aspects of the mapping operation of some embodiments can be understood from a set of examples that progress from simple to increasingly more advanced examples. Because activation skipping does not affect the mapping to a large extent, for clarity of explanation, features associated with zero activation skipping should be ignored, and it is assumed that each OFM column has only one adder tree and accumulator (i.e., the computation is "intensive"). Because caching does not affect the mapping to a large extent, caching including zigzag plane translation methods should also be ignored, and the convolution window is assumed to move in a raster manner (sliding in the plane direction). Figures 3AA to 3AK In the first example depicted in , a single block 102 is used to compute a 3x3x16x8 convolution. Figure 3AA Block 102 is depicted as accepting as input an IFM slice having 16 depth channels and producing an OFM slice having 8 depth channels. For this example, Figure 3AB As shown in , the size of the IFM tensor 304 is 64x64x16, the size of the OFM tensor 303 is 64x64x8, and the size of the weight tensor 302 is 3x3x16x8.

[0422] Initially, as Figure 3AC As depicted in FIG, weights are preloaded from SRAM 109 into MU weight register file 127. The size of weight core 302 is 3x3x16x8. A weight core 302 with a plane size of 3x3 has 3*3=9 plane "positions" which are Figure 3AC Indicated as A to I in FIG. Each plane position is associated with a 16-long weight vector used to calculate the dot product with the 16-long IFM value vector of one OFM channel. Figure 3AC As depicted in , since there are 8 OFM channels, the weight kernel 302 can be considered to have one 3D tensor for each OFM channel.

[0423] Specifically, weights may be loaded into the MU weight register file 127 as follows. The multiple MU weight register files in the entire MR array 122 may be considered as a tensor with dimensions 18x16x8 (18 weights per MU, 16MU rows and 8MU columns), which is more than sufficient to hold the entire weight kernel of size 3x3x16x8. The 18x16x8 weight register file tensor size may also be rewritten as (3x3)x16x8, where each MU weight register file at row R, column C stores all 9 weights for 3x3=9 plane positions (x, y) in the weight tensor W x H x R x C, where W and H are the weight kernel plane width and height (i.e., W=3, H=3). For example, referring to Figure 3ACThe weight register file in row 0, column 0 stores weights {A0[0], B0[0], C0[0], D0[0], E0[0], F0[0], G0[0], H0[0], I0[0]}, where the notation is "A…I", followed by OFM columns "0…7" and IFM rows "[0…15]". Correspondingly, the weight register file in row 15, column 0 stores weights {A0

[15] , B0

[15] , C0

[15] , D0

[15] , E0

[15] , F0

[15] , G0

[15] , H0

[15] , I0

[15] }. The weight register file in row 15, column 7 stores weights {A7

[15] , B7

[15] , C7

[15] , D7

[15] , E7

[15] , F7

[15] , G7

[15] , H7

[15] , I7

[15] }, etc. Since block 102 computes dot products "vertically" using a column-wise adder tree, it can be seen that the described order of loading weights allows the dot product of the IFM input to be computed at each plane position A...I.

[0424] Reference Figure 3AD , the convolution window may then be positioned at the start position, and the eight accumulators 130 (as previously mentioned, there is one accumulator for each of the eight OFM channels for clarity of mapping explanation) may be cleared.

[0425] Reference Figure 3AE , block 102 may then read the IFM a[0..15] (where a...z represents the plane position of the IFM and 0..15 represents the IFM depth channel) from SRAM 109 and broadcast the values ​​to the 8 columns of block 102. The first column may element-wise multiply a[0..15] with the preloaded weights A0[0]...A0

[15] , the second column may element-wise multiply a[0..15] with the preloaded weights A1[0]...A1

[15] , and so on. The resulting products may be vertically summed (reduced) using a tree of adders per column and added to the corresponding accumulator 130. Because there are another 8 plane positions (out of 3*3=9) to be processed to complete the 3x3 convolution at a single position, the resulting dot product is still not final.

[0426] Reference Figure 3AF, block 102 may then read the IFM b[0..15] from SRAM 109 and broadcast the values ​​to the 8 columns of block 102. The first column may element-wise multiply b[0..15] with the preloaded weights B0[0]...B0

[15] , the second column may element-wise multiply b[0..15] with the preloaded weights B1[0]...B1

[15] , and so on. The resulting products may be vertically summed and added to the corresponding accumulators 130. Figure 3AG , block 102 may then read IFM c[0..15] from SRAM 109 and broadcast the values ​​to the 8 columns of block 102. The first column may element-wise multiply c[0..15] with the preloaded weights C0[0]....C0

[15] , the second column may element-wise multiply c[0..15] with the preloaded weights C1[0]...C1

[15] , etc. The resulting products may be vertically summed and added to the corresponding accumulators 130.

[0427] Reference Figure 3AH , block 102 may then read IFMg[0..15] from SRAM and broadcast the values ​​to the 8 columns of block 102. The first column may element-wise multiply g[0..15] with the preloaded weights D0[0]...D0

[15] , the second column may element-wise multiply g[0..15] with the preloaded weights D1[0]....D1

[15] , and so on. The resulting products may be summed vertically and added to the corresponding accumulators 130. Figure 3AI , block 102 may then read the IFM h[0..15] from the SRAM and broadcast the values ​​to the 8 columns of block 102. The first column may element-wise multiply h[0..15] with the preloaded weights E0[0]...E0

[15] , the second column may element-wise multiply h[0..15] with the preloaded weights E1[0]...C1

[15] , etc. The resulting products may be summed vertically and added to the corresponding accumulators 130.

[0428] Reference Figure 3AJ , similar operations may be performed for the remaining positions in the nine positions of the kernel (labeled as a to o). The value stored in the accumulator 130 may then be rounded to form an 8-bit output OFM result, and all eight OFM results may be written to the SRAM 109. This completes the calculation of one convolution. Figure 3AK As depicted in , the convolution window plane can then be translated by one column, and the operation can be repeated.

[0429] exist Figures 3BA to 3BC In the second example depicted in , a single block is used to determine or calculate the 3x3x16x128 convolution. As mentioned before, for convenience, as Figure 3BA As depicted in , the term "IFM slice" may be defined to mean 16 IFM depth channels (i.e., the unit of IFM read and block input), and the term "OFM slice" may be defined to mean 8 OFM depth channels (i.e., the unit of OFM block output). Figure 3BB As depicted in , it may be convenient to depict the operation mapping in rectangles, where the height of the rectangle corresponds to the number of IFM channels and the width of the rectangle represents the number of OFM channels. A 3x3x16x128 convolution can be accomplished by splitting the convolution into 16 3x3x16x8 convolutions, so that the previous example of performing a 3x3x16x8 convolution can be repeated 16 times. In the first step, the 3x3x16x8 convolution of OFM[0..7] can be calculated. In the second step, the 3x3x16x8 convolution of OFM[8..15] can be calculated, and so on, until in the sixteenth step, the 3x3x16x8 convolution of OFM[120..127] can be calculated. The processing of the next subset of OFM channels may be referred to herein as "stepping the OFM". The sixteen steps may correspond to sixteen rectangles, the first, second, and sixteenth of the sixteen rectangles being in the order of Figure 3BC Depicted in, and from Figure 3BB and Figure 3BC As can be seen in Figure 2, when the sixteen steps are completed, a 3x3x16x128 convolution has been calculated.

[0430] Hypothetically, an infinite number of OFM channels could be processed in this manner by simply splitting the OFM into sufficiently small chunks. Each time the system "steps the OFM," the IFM is completely reread (in this example, sixteen times). Each read of the (entire) IFM may be referred to herein as an "IFM pass," and if the operation is performed repeatedly, each such IFM pass may consume a considerable amount of energy (or power). Reducing power consumption is often highly desirable, particularly for battery-powered devices such as mobile smartphones. The next example depicts a method for avoiding some of this energy cost.

[0431] exist Figure 3CA and 3CBIn the third example depicted in FIG, instead of using one block, sixteen blocks are used to determine or calculate a 3x3x16x128 convolution. Each block 102 has 16*8=128 multipliers 126, so the 16 blocks have a total of 128*16=2048 multipliers. IFM[0..15] can be broadcast to all 16 blocks 102, such that block 1 will calculate OFM[0..7], block 2 will calculate OFM[8..15], and so on, and block 16 will calculate OFM[120..127]. As used herein, the term IFM "broadcast" means that the IFM is input to multiple MR blocks 102 simultaneously, as opposed to the description of block 102, where broadcast means that the ABU output is input to all MU columns with a single block.

[0432] The neural processor 100 has a plurality of SRAM memory banks 109 ( Figure 1A and Figure 3AC ). In this way, refer to Figure 3CB , the input IFM[0..15] can be input from SRAM bank group 0. The output of block 1 (OFM[0..7]) can be concatenated with the output of block 2 (OFM[8..15]) into the 16-channel OFM[0..15] and stored in SRAM bank group 1. Similarly, the output of block 3 can be concatenated with the output of block 4 and stored in SRAM bank group 2, and so on, the output of block 15 is concatenated with the output of block 16 and stored in SRAM bank group 8. It can be seen that in this third example, as a result of using IFM broadcasting, all OFMs are calculated in a single "pass" (i.e., reading the entire IFM data once), and because the IFM data is only read once, most of the energy consumption incurred by performing multiple IFM passes in the second example above is avoided.

[0433] exist Figure 3DA In the fourth example depicted in , sixteen tiles are used to determine or calculate a 3x3x16x256 convolution. 16 tiles may only be able to generate up to 16*8=128 OFM channels in a single pass. In this example, 256 OFM channels will be generated. Therefore, two OFM steps may be run, the first step to calculate OFM[0..127], and the second step to calculate OFM[128..255]. Two IFM passes may be used, thereby fully reading the IFM twice. In Figure 3DA The OFM is formed in two steps.

[0434] exist Figure 3EA and Figure 3EBIn the fifth example depicted in , sixteen tiles are used to determine or calculate a 3x3x32x64 convolution. Unlike the previous example with 16 IFM channels, this example involves 32 IFM channels. All 32 IFM channels (2 slices) can be read from the SRAM 109 simultaneously. The neural processor 100 can have multiple SRAM memory bank groups. Each memory bank group (in the mapping example) can stream 1 slice per clock cycle. Thus, in order to read (stream) 2 slices (32 IFM channels) simultaneously, two memory bank groups can be used, a first memory bank group of the two memory bank groups can stream IFM[0..15], and a second memory bank group of the two memory bank groups can stream IFM[16..31].

[0435] Reference Figure 3EB , the computation of OFM[0..7] can be split across block 1 and block 9. Block 1 can reduce (add) IFM[0..15] to the unfinished OFM[0..7]. Block 2 can reduce IFM[16..31] to the unfinished OFM[0..7]. The computation of OFM[0..7] can then be completed by adding the outputs of block 1 and block 2 (and applying biases, activation functions, etc.). To perform this addition, the adder trees of block 1 and block 2 can be "connected" using more than one additional hardware adder stage. The reduction fabric 111 provides such additional hardware adder stages. Similar operations can be used for OFM[8..15] (adding block 2 and block 10), ..., OFM[56..63] (adding block 8 and block 16). Reference Figure 3EB , in this example, there is no output from banks 1..8 to SRAM 109. As will be explained later, only banks 9..16 save OFM to SRAM 109.

[0436] exist Figures 3FA to 3FC In the sixth example depicted in , sixteen blocks are used to determine or calculate a 3x3x32x512 convolution. Figure 3FA , as in the fifth example, two IFM slices (IFM[0..31]) can be read from two SRAM banks, and each of the two IFM slices can be broadcast to 8 blocks. Two such groups of 8 blocks together can calculate OFM[0..63], and the results can be saved to 4 SRAM bank groups. Figure 3FB, 64 OFMs can be computed per IFM pass (i.e., the entire IFM can be read to compute the 64 OFMs). In this way and in a manner similar to that of the fourth example, 512 OFMs can be computed in 8 IFM passes (and, equivalently, 8 OFM "steps"). OFM[0..63] can be computed during the first IFM pass. OFM[64..127] can be computed during the second IFM pass, and so on, OFM[448..511] is computed during the eighth IFM pass. In this example, the "2 IFM slices multiplied by 64 OFM slices" operation has been split into 8 OFM steps. Each OFM step convolves "2 OFM slices multiplied by 8 OFM slices". Referring to Figure 3FC In some embodiments, a virtual SRAM bank may be used to handle situations where an SRAM bank (which may have a capacity of approximately 32 kB) is either exhausted of IFM data or filled with OFM data.

[0437] In such cases, the data structures of the neural processor 100 can be switched transparently (to the block receiving the IFM stream) to connect to another SRAM bank group. As previously mentioned, the IFM and OFM tensors may be too large to be stored in a single SRAM bank group 109 and therefore may need to be split into multiple sub-tensors, each of which is small enough to fit in the SRAM bank group 109 for storage. The global control FSM 140 contains configuration registers that specify how the IFM and OFM tensors are split and stored in the SRAM bank groups, including the IFM and OFM sub-tensor index, size, index of the SRAM bank group where each sub-tensor is stored, and the address at which each sub-tensor is stored within the associated SRAM bank group.

[0438] As computation proceeds and IFM (OFM) traversals move from a subtensor stored in one SRAM bank group 109 to another subtensor stored in another SRAM bank group 109, the global control FSM 140 orchestrates a running reconfiguration of the IFM and OFM transport structures, switching the IFM source (and OFM destination) SRAM bank group from the current SRAM bank group to the next. In some embodiments, this reconfiguration is accomplished in a manner that is transparent to the blocks consuming the IFM (and blocks generating output) and does not halt or slow down computation during bus switching.

[0439] As mentioned previously, a piece of software, referred to herein as a "mapper," statically (at compile time) decides how to partition the entire IFM and OFM memory across SRAM bank groups and physical SRAM banks, as well as weight the core memory and partial results. Figure 3FCAs depicted in , for clarity of mapping explanation, details of physical IFM and OFM storage across multiple SRAM bank groups may be ignored, and the SRAM bank groups may be considered to be a "virtual" or "logical" view 306 in the IFM and OFM.

[0440] exist Figures 3GA to 3GD In the seventh example depicted in FIG, sixteen blocks are used to determine or calculate a 3x3x32x512 convolution. In this example, fewer IFM passes are used to calculate the same convolution as in the sixth example to save energy. Figure 3GA , each multiplier unit weight register file 127 may have 18 weights, of which only 9 are used for the 3x3 convolution in the sixth example. Thus, two sets of 3x3 weights may be stored (as opposed to one), and the two sets of 3x3 weights may be "cycled" over time. Specifically, a 3x3x32x512 convolution may be split into two 3x3x16x512 convolutions that are interleaved in time. Referring to Figure 3GB In a similar manner to the third example, a 3x3x16x512 convolution can be mapped to 16 physical blocks. For each IFM pass, one IFM slice can be read from the SRAM bank group and broadcast to the 16 physical blocks, which output 128 OFM channels to 8 SRAM bank groups. In this example, it takes four IFM passes (and four OFM steps) to complete the OFM calculation.

[0441] Reference Figure 3GC In some embodiments, in the first step, IFM[0..15] may be input to compute the convolution of OFM[0..127] at OFM position (x,y), but the OFM result may be kept in the accumulator instead of writing the result to SRAM. Figure 3GD In the second step, each multiplier unit weight register file 127 may then switch to the second set of 3x3 weights and input IFM[16..31] to complete the calculation of OFM[0..127]. This process may be referred to herein as an "IFM weight loop." Then, in the third step, OFM[0..127] may be saved to SRAM and the accumulator cleared. These three steps may be repeated until the calculation is complete.

[0442] Reference Figure 3GAIn some embodiments, a logical block can be defined as a physical block that stores multiple sets of weights. As can be seen, in this example (the seventh example), two groups of 16 such logical blocks (interleaved in time) (i.e., 32 logical blocks) are formed by storing two sets of 3x3 weights. In the seventh example, the 32 logical blocks can physically calculate more (e.g., wider) OFMs in each IFM pass, reducing the number of IFM passes (and SRAM IFM read energy) by a factor of two compared to the sixth example.

[0443] exist Figure 3HA to Figure 3HC In the eighth example depicted in FIG, a 3x3x512x256 convolution is first determined or calculated using sixteen physical blocks. Note that in this example both the number of IFM and OFM channels (512 and 256, respectively) are quite large. As discussed in further detail below, when the convolution kernel is too large to be calculated otherwise, a partial result or "partial" may be used. However, this example shows how a convolution with a large weight kernel can still be performed without using a partial. The 3x3x512x256 convolution may be performed as shown in FIG. Figure 3HB . For a 3x3 8-bit convolution, two sets of 3x3 8-bit weights can be stored in each multiplier unit, resulting in (2 sets of weights) * (16 physical blocks) = 32 logical blocks. 32 logical blocks can be reduced to 32 IFM slices, resulting in a maximum number of IFM channels that can be processed without using fractions of (32 slices) * (16 IFM channels per slice) = 512 IFM channels. Thus, a 3x3x512xN convolution can be calculated without using fractions of N, where N is any positive integer.

[0444] Reference Figure 3HB and 3HC , a block adder tree combined with the reduction structure 111 can be used to reduce 256 IFM channels per clock. In order to reduce all 512 IFM channels (and generate 8 OFM channels), two weight cycles are performed. In weight cycle 1, as Figure 3HB , IFM[0...15] may be input to bank 1, IFM[16..31] may be input to bank 2, etc., and IFM[240..255] may be input to bank 16. The hardware tree may be connected across all 16 banks (per column) using hardware adder stages provided by the reduce structure 111. The adder tree root may end at bank 16 (as discussed later in the context of the reduce structure 111, OFM transfer structure, and adder tree) so that only bank 16 generates a result, and the accumulators of banks 1...15 are not used in this configuration. Figure 3HCIn the weight loop 2 depicted in , IFM[256...271] may be input to block 1, IFM[272...287] may be input to block 2, and so on, and IFM[496...511] may be input to block 16. Block 16 may then write the completed OFM[0..7](x,y) result to SRAM bank 16. Finally, 32 IFM passes (32 OFM steps) may be performed to calculate OFM[0...7], then OFM[8..15], and so on, to OFM[248..255]. Note that while the number of IFM passes and OFM steps are the same in this particular example, the difference between IFM passes and OFM steps will become clearer in later examples.

[0445] Figure 3HD It also depicts how to convert the IFM pass to OFM simply by performing 64 IFM passes (64 OFM steps) instead of 32 IFM passes (32 OFM steps). Figure 3HA to Figure 3HC The 3x3x512x256 convolution depicted in

[15] is changed to a 3x3x512x512 convolution.

[0446] exist Figures 3IA to 3IF In the ninth example depicted in , 16 blocks are used and partial results are used to determine or calculate a 3x3x512x256 convolution. In some cases, using portions can make energy savings possible by reducing the number of SRAM reads (compared to, for example, the eighth example). When using portions, the mapping algorithm can split the weight tensor into multiple parts, specifically, in the depth channel direction, converting a single convolution operation (including loading the weight tensor, traversing the IFM, and writing to the OFM) into two or more convolution operations. The outputs of these two or more resulting convolutions are then combined to produce the final result.

[0447] First, recall Figures 3HB to 3HC Depicted is a 3x3x512x256 convolution computed without parts. Figures 3IA to 3IB and Figure 3IC to Figure 3ID Depicts the hardware resource mapping associated with the weight tensor (and corresponding IFM and OFM) after the 512 IFM channels are split into 256 and 256 corresponding to two separate convolutions, each of size 3x3x256x256.

[0448] Figures 3IA to 3IB The first of two 3x3x256x256 convolutions is depicted. Because the weight kernel plane size is 3x3=9, each MU weight register file, which can hold 18 8-bit weights, has enough capacity to store two sets of 3x3 weights, thus making 32 logic blocks available for computation.

[0449] Eight IFM slices can then be loaded. Each IFM slice can then be broadcast to 2 physical blocks. 16 OFM steps (16 IFM passes) can be performed. During the first weight cycle, Figure 3IA As depicted in , the 3x3 IFM[0..127] can be input, convolved with the first set of 3x3 weights, reduced using the adder tree, and accumulated in the accumulator registers of blocks 8 and 16. Figure 3IB During the second weight cycle, the 3x3 IFM[128..255] can be input, convolved with the second set of 3x3 weights, reduced using the adder tree, and further accumulated in the accumulator registers in blocks 8 and 16. At this point, the convolution of the 3x3 IFM[0..255] with the corresponding 3x3x256x16 weight kernel is completed for OFM channels 0...15 and can be written to virtual SRAM banks 8 and 9 as a partial result. Because this is a partial result, the value of accumulator (Acc) 130 bypasses activation function module 197 on the way to SRAM, as opposed to the completed result. Optionally, to reduce SRAM size requirements and power consumption, bit range selection module 187 can reduce the bit width of the partial result to an integer, for example, down to 4 bytes when using 8-bit activations and weights, or down to 6 bytes when using 16-bit activations and weights.

[0450] The above steps are repeated until the entire IFM[0...255] (i.e., for all desired planar (x,y) positions) has been processed in one pass over IFM[0...255], resulting in a corresponding set of partial results calculated for OFM[0...15]. The partial results for the remaining OFM channels [16...255] are calculated by performing another 15 passes over IFM[0...255] (corresponding to another 15 OFM steps).

[0451] Note that in this mapping example, the OFM generated physically and simultaneously in one pass is widened (expanded) by a factor of two (from one OFM slice to two OFM slices) using two partial passes. In addition, the size of the IFM tensor processed during each partial pass is shortened by a factor of two from HxWx512 to HxWx256.

[0452] If respectively Figure 3IC and 3ID , the second portion of the IFM pass may be the same as the first portion, except that IFM[256..383] may be input during the first weight cycle, and IFM[384..511] may be input during the second weight cycle.

[0453] Similar to ARU 167, completing the original 3x3x512x256 convolution involves adding the partial results (from the two 3x3x256x256 convolutions, element-wise) and applying scaling, biasing, and activation functions. There are multiple ways to accomplish this final step, including: (i) reading the partial results generated by the first partial convolution, sending the partials to tile ARU 167 via IFM transport fabric 104 to be element-wise summed with the second set of partial results so that ARU 167 will generate the final result during the second partial convolution; (ii) having the partial outputs of ARU 167 during both partial convolutions, while having additional logic in SRAM bank group 109 to perform a read-modify-write to add the partials and apply the activation function. More specifically, additional logic for completing the partial will receive the partial results during the second partial convolution, read the results of the first partial convolution from the SRAM, sum the results and apply the activation function on the fly and write the final result back to the SRAM; (iii) having additional logic in the SRAM memory bank group 109 that is capable of performing read-add-write operations on the partials so as to continue to add the partial results from two or more partial operations element by element without applying the activation function, and then be completed during the final partial operation rounding by reading the partial results and sending the partial results to the block ARU 167.

[0454] Unlike the case where sections are not used, when sections are used, OFM height and width should be considered when arranging convolution operations. Figure 3IE , four bytes can be used to store each partial result (assuming that both IFM and OFM are 8 bits). In this case, the SRAM storage size of the partial result is equal to (OFM height) * (OFM width) * (OFM depth) * (4 bytes). As depicted, if the SRAM (on-chip) storage capacity is not sufficient for the partial results, the OFM data can be divided into sub-windows and processed one at a time. However, each time a sub-window is processed, it may be necessary to load (or reload) the core weights of the entire group, which may increase energy consumption. For example, assume that the OFM plane size is set to 10x10 and the IFM plane size is set to be equal to the OFM plane size. In this case, the core weight size is relatively large, at 3*3*512*256=1.2 megabytes. The SRAM size for storing the entire partial result of the entire IFM plane size without subdividing it into plane sub-windows is 10*10*256*4=102400 bytes. For simplicity, it is further assumed that the SRAM has sufficient capacity so that sub-windows do not need to be used.

[0455] Figure 3IFThis summarizes the process of computing the convolution in this example, whereby a first set of portions of IFM[0..255] and all OFM[0..255] are determined or calculated and saved, a second set of portions of IFM[0..255] and all OFM[0..255] are determined or calculated (but not written to SRAM as this is the last partial pass), and the portions are added element-wise when determining or calculating the second partial convolution, applying the activation function on the fly and writing it to SRAM.

[0456] As previously mentioned, using MR block 102 to add parts element-by-element and apply activation functions is optional. Instead, an Auxiliary Planar and Activation Processing (APAP) unit dedicated to element-by-element and planar (without cross-channel reduction) operations can be used. These units can be located within SRAM bank group 109 and can access parts stored locally in SRAM as well as parts that reach the SRAM bank group. The APAP unit then writes the completed results to SRAM 109.

[0457] The determination or calculation performed according to the ninth example can save a lot of energy by performing two passes. Because the number of IFM passes is reduced from 32 to 16, the amount of IFM data read is (IFM height) * (IFM width) * (IFM channel) * (IFM pass) = 10 * 10 * 512 * (32-16) = 819200 bytes (ignoring cache). The amount of partial data written to SRAM is (OFM height) * (OFM width) * (OFM channel) * (partial convolution number - 1) * (4 bytes) = 10 * 10 * 256 * (2-1) * 4 = 102400 bytes. In other words, if the second partial pass saves the result to SRAM 109 instead of directly inputting the result to the plane / activation unit, it will result in twice the amount. Furthermore, the amount of partial data read from SRAM 109 is (OFM height) * (OFM width) * (number of partial convolutions - 1) * (4 bytes) = 10 * 10 * 256 * (2 - 1) * 4 = 102400 bytes. In other words, if the second partial pass saves the result to SRAM 109 instead of directly inputting the result to the plane / activation unit, twice this amount will be incurred. Thus, in the example, performing a 3x3x512x256 (8-bit) convolution with versus without a partial results in 819,000 fewer IFM bytes being read from SRAM, while incurring an additional 102,400 bytes to write the partial to SRAM and another 102,400 bytes to read the partial from SRAM.

[0458] Assuming that the energy of one SRAM write is about twice the energy of one SRAM read, the total SRAM energy saved is equal to 819000-2*102400-102400=511800*(energy per SRAM read).

[0459] exist Figures 3JA to 3JD In the tenth example depicted in FIG, four blocks are used to determine or calculate an 8x8x16x64 convolution. An 8x8 convolution has 8*8=64 weights, which may not fit into a single multiplier unit. A single multiplier unit 103 may store, for example, only 18 weights. Therefore, Figure 3JA As depicted in , the 64 weights can be divided among the four blocks 102 such that block 1 stores W[0..1, 0..7,*,*], block 2 stores W[2..3, 0..7,*,*], block 3 stores W[4..5, 0..7,*,*], and block 4 stores W[6..7, 0..7,*,*], where the weight kernel notation is W[row, column, IFM channel, OFM channel] and the “*” indicates the entire applicable range. The system can then add (reduce) across blocks to compute OFM[0..7] such that each block effectively performs a 2x8x16x64 convolution and four 2x8x16x64 convolutions performed simultaneously using four blocks, with the four 2x8x16x64 convolutions being aggregated into one 8x8x16x64 convolution. Each 2x8x16x64 convolution also consists of two 1x8x16x64 convolutions combined together using an IFM weight cycle.

[0460] Figure 3JB Depicted is the first step of the IFM weight loop, where the even (not yet odd) rows within the convolution window are convolved. Here, block 1 convolves row 0W[0,*,*,*] of the convolution window with IFM values ​​“a0,b0,c0,d0,e0,f0,g0,h0”, while block 2 convolves row 2W[2,*,*,*] of the convolution window with IFM values ​​“a2,b2,c2,d2,e2,f2,g2,h2”. Block 3 convolves row 4W[4,*,*,*] of the convolution window with IFM values ​​“a4,b4,c4,d4,e4,f4,g4,h4”, and block 4 convolves row 6W[6,*,*,*] of the convolution window with IFM values ​​“a6,b6,c6,d6,e6,f6,g6,h6”. The products of the multiplier units 103 are reduced using the block adder tree within the block and using the addition adder tree levels provided by the reduction structure 111, and are accumulated in the accumulator 130 of block 4 (streamed to the four blocks as IFM values ​​"a*, b*, ..., h*" on the IFM transmission structure 104).

[0461] Figure 3JCDepicts the second step of the IFM weight loop, where odd-numbered rows within the convolution window are convolved. Here, block 1 convolves row 1W[1,*,*,*] of the convolution window with IFM values ​​“a1, b1, c1, d1, e1, f1, g1, h1”, while block 2 convolves row 3W[3,*,*,*] of the convolution window with IFM values ​​“a3, b3, c3, d3, e3, f3, g3, h3”. Block 3 convolves row 5W[5,*,*,*] of the convolution window with IFM values ​​“a5, b5, c5, d5, e5, f5, g5, h5”, and block 4 convolves row 7W[7,*,*,*] of the convolution window with IFM values ​​“a7, b7, c7, d7, e7, f7, g7, h7”. Similar to the first IFM weight round robin step, the products of the multiplier units 103 are reduced using the block adder tree within the block and using the add adder tree level provided by the reduce structure 111, and are accumulated in the accumulator 130 of block 4 (streamed to the four blocks as IFM values ​​"a*, b*, ..., h*" on the IFM transmit structure 104). However, unlike during the first IFM weight round robin step, the accumulator 130 is not cleared at the beginning of the second IFM weight round robin step, so that once both IFM weight round robin steps are completed, the accumulator 130 contains the dot products of both the even and odd rows.

[0462] The resulting OFM[0...7] can then be written to SRAM 109, completing the convolution of the 8x8x16x8 window for one OFM position. Figure 3JD To continue the computation, the convolution window can then be shifted to compute the next 8x8 convolution, as depicted in . The process can be repeated until the entire OFM is completed.

[0463] exist Figure 3KA and Figure 3KB In the eleventh example depicted in FIG, sixteen blocks are used to determine or calculate 8x8x64x64 convolution. 8x8 convolution can be applied to 16 blocks, and more IFM and OFM channels can be used. Figure 3KA As depicted in , splitting the 8x8 convolution over four physical blocks reduces the number of "logical" blocks by a factor of four, e.g., (16 physical blocks) / (4 physical blocks per operation) = 4 logical blocks. As used herein, the term "physical grouping" of physical blocks is defined by connecting the block adder trees into a single adder tree (per column) to perform operations that are too large for a single physical block 102.

[0464] Reference Figure 3KA, because the 8x8 convolution may be too large to fit into a single block 102, the 8x8 convolution may be split into four blocks. By connecting the adder trees from the four blocks to a single adder tree, the four blocks may be physically grouped into one logical block. Figure 3KB , mapping 8x8x64x64 to 16 physical blocks is logically transformed into mapping 8x8x64x64 to 4 logical blocks, where each logical block has 18*4=72 weights, which is enough to fit 8x8=64 convolution weights.

[0465] Figure 3KB Depicting the mapping of an 8x8x64x64 convolution operation to 4 logical blocks (thus 16 physical blocks). The operations of the transform can be performed as follows. First, four IFM slices can be read. All IFM channels can be read at once to avoid partials. Second, each IFM slice can be "broadcast" to a logical block. Third, 8 OFMs (one OFM slice) can be calculated in one IFM pass. This can be repeated so that (64 OFMs) / (8 OFMs per pass) = 8 OFM passes (8 OFM steps) can be performed to calculate all OFM channels.

[0466] In some cases, more OFM channels may be needed, for example, to determine or calculate an 8x8x64x1024 convolution. This is made possible without using sections by adding more OFM steps that perform more IFM passes to re-read the IFM. In some cases, more IFM channels may be needed, for example, to determine or calculate an 8x8x128x64 convolution. In such cases, it may be necessary to use sections unless (i) the number of physical blocks is increased, or (ii) the number of weights per multiplier is increased. However, in some applications, large size convolutions like 8x8 may only be applied to RGB images or images with a small number of IFM channels. The MU weight register file 127, which holds N weights, can accommodate convolution kernels of up to H*W≤N, where H and W represent the plane height and width of the weight kernel. For example, a MU 103 with 18 8-bit weight capacities can hold convolution kernels including 4x4, 5x3, 3x5, 6x2, 2x6, 7x2, 2x7, 8x2, 2x8, 9x2, 2x9, 18x1, and 1x18. In practice, the need to compute an 8x8x128x64 convolution may be rare and thus may be performed by the CPU rather than the neural processor 100, making the associated neural processor additional hardware logic optional. For clarity, the IFM, OFM, and reduction structure descriptions omit cases where a connection of H*W>N is required (such as the case described in this example).

[0467] exist Figures 3LA to 3LDIn the twelfth example depicted in , sixteen blocks are used to determine or calculate a 1x1x1024x64 convolution. Each MU may have 18 weights. Since a 1x1 convolution only requires 1x1=1 weight, (18 weights per multiplier) / (1 weight per convolution window)=18 groups of 1x1 convolution weights may be fitted into each block. The number of logical blocks may be calculated as (16 physical blocks)*(18 groups of convolution weights per multiplier)=288 logical blocks. The calculation of a 1x1x1024x16 convolution using 16 physical blocks may be transformed into the calculation of a 1x1x1024x16 convolution using 288 logical blocks. All (1024) IFM channels may be read in one IFM pass to avoid partial. With 288 logic blocks, an IFM size of up to (16 IFM channels per IFM slice) * (288 logic blocks) = 4608 channels is acceptable. A 1x1x1024x64 convolution, without using any of the available 4608 IFM channels, only requires 1024 of them. Therefore, the number of OFM slices that can be computed per IFM pass is 4 ((maximum 4608 IFM channels) / (1024 IFM channels)) = 4 OFM slices.

[0468] The determination or calculation can be performed as follows. First, 16 sets of 1x1 weights can be stored in each MU. During each OFM step (IFM pass), 64 slices (all 1024 IFM channels) can be read. Physically, this corresponds to reading (64 IFM slices) / (16 sets of 1x1 weights per MU) = 4 IFM slices each time. Each of the four IFM slices can be broadcast to (16 physical blocks) / (4 IFM slices) = blocks to calculate 4 OFM slices in one OFM step (and one IFM pass). The OFM can be calculated using (8 OFM slices) / (broadcast over 4 blocks) = 2 OFM steps (and 2 IFM passes). The IFM weights can be looped 16 times.

[0469] Specifically, refer to Figure 3LA, the convolution calculation can be performed along the following steps. In the first step, the accumulator is cleared. In the second step, IFM[0..15], IFM[16..31], IFM[32..47], and IFM[48..63] are obtained and IFM[0..15], IFM[16..31], IFM[32..47], and IFM[48..63] are broadcast to blocks 1, 5, 9, and 13, blocks 2, 6, 10, and 14, blocks 3, 7, 11, and 15, and blocks 4, 8, 12, and 16, respectively. In the third step, the system accumulates the dot products calculated by blocks 1..4 and OFM[0..7], blocks 5..8 and OFM[8..15], blocks 9..12 and OFM[16..23], and blocks 13..16 and OFM[24..31] as intermediate (unfinished) results into the accumulator registers of blocks 4, 8, 12 and 16 respectively.

[0470] Reference Figure 3LB In the fourth step, the accumulator is not cleared and MU 103 is switched to use the next set of 1x1 weights corresponding to the step in the IFM weight loop. In the fifth step, IFM[64..79], IFM[80..95], IFM[96..111] and IFM[112..127] are obtained and broadcast to blocks 1, 5, 9 and 13, blocks 2, 6, 10 and 14, blocks 3, 7, 11 and 15, and blocks 4, 8, 12 and 16 respectively. In the sixth step, the system accumulates the dot products calculated by blocks 1..4 and OFM[0..7], blocks 5..8 and OFM[8..15], blocks 9..12 and OFM[16..23], and blocks 13..16 and OFM[24..31] as intermediate (unfinished) results in the accumulator registers of blocks 4, 8, 12 and 16 respectively.

[0471] Reference Figure 3LC, the calculation can proceed, continuing to loop over the IFM weights (a total of 16 IFM weight loop steps), fetching and broadcasting the IFM, and computing and accumulating dot products until the last IFM slice is reached (channel 960 to channel 1023). At this step, the accumulator is not cleared, and the MU 103 is switched to the next (last 16th) set of 1x1 weights corresponding to the last step in the IFM weight loop. In the next step, IFM[960..975], IFM[976..991], IFM[992..1007] and IFM[1008..1023] are obtained and IFM[960..975], IFM[976..991], IFM[992..1007] and IFM[1008..1023] are broadcast to blocks 1, 5, 9 and 13, blocks 2, 6, 10 and 14, blocks 3, 7, 11 and 15, and blocks 4, 8, 12 and 16 respectively. Next, the system accumulates the dot products calculated by blocks 1..4 with OFM[0..7], blocks 5..8 with OFM[8..15], blocks 9..12 with OFM[16..23], and blocks 13..16 with OFM[24..31] into the accumulator registers of blocks 4, 8, 12, and 16, respectively, to obtain the final dot product results. In the next step, activation is applied to the dot product results accumulated in the accumulator registers of blocks 4, 8, 12, and 16, and the four resulting OFM slices are written to SRAM. This completes the calculation of OFM[0..31].

[0472] Reference Figure 3LD , the system then proceeds to the next OFM step (by performing another IFM pass) and repeats the computation, this time for OFM[32..63]. The system loads the slice of weights for the next OFM step: W[0,0,0..1023,32..63]. Figure 1K and Figure 1N As depicted in , weight loading can occur concurrently with calculations using the vertical weight loading bus 101, in which case there is no additional delay caused by the weight loading process. The system can clear the accumulator and switch MU 103 to the first set of 1x1 weights. The system can then repeat as in Figures 3LA to 3LC The operations described in the context of to compute OFM[32..63].

[0473] like Figure 3LD Depicted in (similar to Figure 3LC), once the system has gone through 15 of the 16 IFM weight cycles, obtained the corresponding IFM slices, and calculated and accumulated the intermediate dot product results, the system reaches the last (16th) round of the IFM weight cycle. In this round, the accumulator is not cleared, and MU 103 is switched to the next (last 16th) set of 1x1 weights (the last, 16th IFM weight cycle step). The system obtains IFM[960..975], IFM[976..991], IFM[992..1007], and IFM[1008..1023] and broadcasts them to blocks 1, 5, 9, and 13, blocks 2, 6, 10, and 14, blocks 3, 7, 11, and 15, and blocks 4, 8, 12, and 16, respectively. Next, the system accumulates the dot products calculated by blocks 1..4 with OFM[32..39], the dot products calculated by blocks 5..8 with OFM[40..47], blocks 9..12 with OFM[48..55], and blocks 13..16 with OFM[56..63], respectively. At the end of this process, the system applies the activation function 197 (in blocks 4, 8, 12, and 16) to the completed dot products stored in the accumulator 130 (in blocks 4, 8, 12, and 16), and writes the final OFM[32..63] results to the SRAM to complete the convolution operation.

[0474] Now consider the fully connected (FC) layer calculation, which is the opposite of a convolution operation. First consider the simple case of a 16x8 FC calculation using a single tile and a single IFM sample. Note that the FC layer calculation is similar to the 1x1 convolution (described in the previous example), except that the weights are discarded after multiplication with the IFM. A single 16x8 FC calculation can be completed by loading 1 weight into each MU, taking a single IFM[0...15] slice, calculating the dot product using the tile's adder tree, applying the activation function to the resulting dot product, and writing the completed IFM[0...7] result to SRAM 109.

[0475] Consider the case of determining or calculating a 16x16 FC using a single block 102 and a single IFM sample. A single 16x16 FC calculation can be accomplished by loading two weights into each MU 103, obtaining a single IFM[0...15], and having the MU 103 select the first of the two preloaded weights for multiplication. As described above, OFM[0...7] can be calculated. MU 103 can select the second of the two preloaded weights for multiplication and calculate M[8..15]. This process of looping through MU weights to calculate multiple OFMs from the same IFM is referred to herein as "OFM weight looping."

[0476] Note that the 16x16 FC calculation is done using one IFM pass, not two OFM steps (corresponding to two OFM weight cycles). Therefore, as observed in most other examples, the number of OFM steps is generally equal to the number of IFM passes unless OFM weight cycles are used.

[0477] Consider another simple case of determining or calculating a 16x128 FC using a single block and a single IFM sample. This can be achieved by loading 16 weights into each MU 103 and obtaining a single IFM slice. The 16 OFM steps can be performed by looping through the OFM weights (i.e., by looping through the MU weights to calculate OFM[0...7], OFM[8..15], ..., OFM[120..127] one by one).

[0478] Consider the simple case of determining or computing a 16x8 FC using a single block for a batch of 18 IFM samples (i.e., the IFM tensor shape can be represented as 1x16x18). As a side note, because the neural processor 100 performs inference (not training), the mapping examples implicitly assume an IFM batch size of 1, as is typical for inference applications. Computations with IFM batch sizes greater than 1 can also be mapped to the hardware. For example, the computation can be repeated as described for each sample in the IFM batch. However, a single-block computation of 16x8 FC for a batch of 18 IFM samples can leverage the MU weight register file capacity to preload 18 weights into each MU 103, one for each IFM sample. The computation can then be completed by fetching the first IFM[0..15][0] sample (of the 18 in the batch), computing the dot product of the fetched IFM sample with the first of the 18 weights in each MU, applying the activation function, and writing the resulting OFM[0..7][0] to SRAM. Next, the IFM[0..15][1] sample is taken and multiplied by the second of the 18 weights in each MU 103 to obtain OFM[0..7][1] after the activation function is applied. This sequence continues until an entire batch of IFM[0..15][0..17] samples (18 total) has been processed, resulting in a batch of OFM[0..7][0..17] samples. Looping over the MU weights to process a batch of IFM samples may be referred to herein as "IFM batch cycling". Note that IFM weight cycling, OFM cycling, and IFM batch cycling may be combined to perform the calculations as long as the MU weight register file capacity is sufficient.

[0479] exist Figure 3MA and Figure 3MBIn the thirteenth example depicted in FIG, a single block is used to perform a 288x8 fully connected determination or calculation. Figure 3MA As previously described, the computation of the full connection can be similar to a 1x1 convolution, where the convolution window is not shifted and the weights are not reused and must be discarded after a single use. One block 102 can compute 8 OFM channels (i.e., 1 OFM slice) in parallel. 288 IFM channels can correspond to 288 / (16 rows per MR block) = 18 slices. The system can use 18 weights in each MU 103 to store all 18 slices of FC weights.

[0480] To perform a fully connected computation, the system may perform the following steps (which may be performed concurrently to some extent, that is, they may overlap in time). In the first step, weights may be loaded from SRAM 109. Figure 1K and 1N As depicted in , weights may be loaded concurrently with the computation using, for example, a vertical weight load bus 101. This way, the system can ensure that the FC weights are placed into SRAM 109. In a second step, the accumulator for OFM[0..7] may be cleared. In a third step, one sample of IFM[0..15] may be input into the block, and the result may be added to OFM[0..7] accumulator 130 to form an intermediate (unfinished) result.

[0481] In the fourth step, the OFM[0..7] accumulator may be left unflushed and the system may switch to the next set of FC weights (round-robin IFM weights). In the fifth step, IFM[16..31] may be input into the block and the result may be added to the OFM[0..7] accumulator. Figure 3MB , where IFM[280..287] is the last slice, the steps can be repeated until all IFM channels (and associated weights) have been cycled through. Finally, the activation function can be applied to the accumulated dot products, and the final OFM[0..7] results can be written to SRAM. This completes the fully connected computation.

[0482] exist Figure 3NA In the fourteenth example depicted in FIG, a 288x64 fully connected determination or calculation is performed. In this example, the OFM channel count is increased from 8 (in the thirteenth example) to 64. If the system splits the FC 288x64 calculation into eight smaller FC calculations of size 288x8 and performs the calculations one by one (e.g., in eight OFM steps), this is equivalent to the thirteenth example. This results in eight IFM passes.

[0483] exist Figures 3OA to 3OCIn the fifteenth example depicted in , a 1024x32 fully connected determination or computation is performed on a single IFM sample (i.e., a batch size of 1). Figure 3OA Since FC determination is similar to a 1x1 convolution, there can be (18 weights per MU) * (16 physical blocks) = up to 288 logical blocks, each performing a 1x1 convolution. This way, the system can read all 1024 IFM channels (1024 / 16 = 32 IFM slices) in a single pass to avoid partial retrieval.

[0484] To read all 32 IFM slices, 32 logical blocks can be used. The calculation may involve computing 32 OFMs (4 OFM slices). To compute 32 OFMs in a single pass (computing all OFMs simultaneously), (32 OFM slices) * (4 OFM slices) = 128 logical blocks can be used. Thus, the available number of logical blocks (288) is sufficient. The number of logical blocks can be reduced to the required 128 by storing 8 weights in each MU 103 (instead of storing up to 18 weights per MU 103).

[0485] The calculation can be performed as follows. The system can store 8 sets of IFM FC weights per MU 103 and use 128 logical blocks (as described above). The entire calculation can be completed in a single IFM pass by calculating four OFM slices. Each of the four IFM slices can be obtained and broadcast to four blocks. Because 8 sets of IFM weights are stored in each MU, the weights can be recycled eight times. The sequence can include the following steps. In the first step, the OFM accumulator can be cleared. In the second step, IFM[0..63] (4 IFM slices) can be obtained and each slice can be broadcast to four blocks. In the third step, the remaining OFM[0..31] (4 OFM slices) can be calculated and added to the OFM accumulator.

[0486] Reference Figure 3OB , in the fourth step, the OFM accumulator may be left uncleared and the next set of weights may be used. In the fifth step, IFM[64..127] (4 IFM slices) may be obtained. In the sixth step, the system may continue to calculate the (unfinished) OFM[0..31] (4 OFM slices) by adding the sum of the products to the OFM accumulator. Figure 3OC , the system may continue to loop over the weights and accumulate OFM results until all IFMs have been processed. As a final step, the system may take IFM[960..1023] and accumulate it into OFM[0..31], then apply the activation function to the accumulated OFM[0..31] and write the result to SRAM 109.

[0487] exist Figure 3PA to Figure 3PC In the sixteenth example depicted in FIG, sixteen blocks and a batch size of 1 are used to perform a 4096x1024 fully connected determination or calculation. This calculation may use (4096 / 16 IFM channels per block) = 256 IFM slices and (1024 / 8 OFM channels per block) = 128 OFM slices. As in some of the other examples described above, it may be advantageous to read the entire IFM to avoid partials. Up to (18 weights per MU) * (16 physical blocks) = 288 logical blocks may be used to perform the calculation. To read the entire IFM, 256 logical blocks may be used. Thus, the available number of logical blocks (288) is sufficient. The system may be configured to use 256 logical blocks by loading 16 sets of weights into each MU 103. To read 256 IFM slices (without partials) in one round, all 256 logical blocks may be used. Therefore, each IFM pass will generate (256 logic blocks / 256 IFM slices) = 1 OFM slice, and in order to complete the calculation, (128 OFM slices) / (1 OFM slice per IFM pass) = 128 OFM steps will be performed (hence 128 IFM passes).

[0488] exist Figure 3PA The physical configuration is depicted in . The reduction structure 111 can be configured to reduce the output of all 16 blocks into a single OFM slice. 16 IFM slices (from 16 virtual SRAM banks) will be obtained, and each IFM slice will be "broadcasted" to only one block 102.

[0489] The calculation can be performed in several steps as follows. In the first step, the OFM[0..7] accumulator is cleared. In the second step, 16 IFM slices (IFM[0..255]) are obtained and reduced to the OFM[0..7] accumulator as intermediate (unfinished) results.

[0490] In the third step, the OFM[0..7] accumulator is left unflushed and the system switches to the next IFM weight group in MU 103. In the fourth step, the next 16 IFM slices (IFM[256..511]) are obtained, reduced and added to the OFM[0..7] accumulator. Figure 3PB As depicted, these steps may continue until all IFMs (up to and including IFM[4080..4095]) have been processed. An activation function may be applied to the accumulated dot products (in block 16), and the final result may be written to SRAM 109. This completes the calculation of OFM[0..7]. Figure 3PC, to perform the next OFM step, the system may repeat the previous calculation for OFM[8..15], load the weights W[0..4095,8..15], and continue stepping OFM until all OFMs are calculated (until OFM[1016..1023]) to complete the entire FC calculation.

[0491] When the IFM already has more than (18 weights) * (16 IFM channels per IFM slice) * (16 physical blocks) = 4608 channels, there may be a FC calculation case. In this case, parts can be used by splitting the IFM channels into parts (of sufficient size to be mapped onto existing physical hardware), calculating the FC for each part separately, adding the partial results element-wise as described above (stored in SRAM), and completing the calculation by applying the activation function.

[0492] In the case where the weight is 16 bits, as previously described, the MU weight register file capacity becomes 9 (16-bit weight) instead of 18 (8-bit weight), and the calculation can be performed using multiple cycles. Similar reasoning applies to larger weight bit lengths (e.g., 24 bits or 32 bits, where, for example, the MU weight register file 127 has sufficient capacity to hold six 24-bit weights or four 32-bit weights).

[0493] Optionally, in addition to mapping operations to all available physical blocks, the neuroprocessor can be logically subdivided into several neuroprocessors, each with a smaller number of blocks. For example, a neuroprocessor with 16 physical blocks can be logically viewed as two neuroprocessors, each with half the original number of blocks (e.g., 8 blocks per neuroprocessor), or as four neuroprocessors, each with a quarter of the original number of blocks (e.g., 4 blocks per neuroprocessor), and so on. Given the number of physical blocks remaining after the partitioning, each neuroprocessor resulting from such a subdivision follows essentially the same mapping principles as described above. Subdividing a neuroprocessor into multiple smaller neuroprocessors may be desirable for operations that require relatively fewer IFM reductions and generate relatively fewer OFM channels (more specifically, their products). For example, a 1x1x32x32 convolution mapping requires only 4 blocks. If mapped to 16 tiles, a 1x1x32x32 convolution would result in 12 of the 16 tiles being unused, thus significantly reducing multiplier utilization. In a case like this, a neuroprocessor with 16 physical tiles could be subdivided into four neuroprocessors, each with 4 tiles, mapping the 1x1x32x32 convolution to each of the four resulting neuroprocessors, subdividing an IFM tensor of, for example, size HxWx32 into four non-overlapping IFM tensors of size (H / 2xW / 2x32), assigning one such quarter-sized IFM tensor to one of the four smaller neuroprocessors, and thus computing the convolution on all four IFM sub-tensors in parallel. Note that such small weight tensor sizes may be relatively uncommon, and that operation modes like this need to be appropriately supported by the IFM, OFM, and reduction structures.

[0494] The various mappings of neural network layer operations onto the available hardware require support from the IFM transport structure 104, the OFM transport structure 106, and the reduction structure 111. Figure 4AAA sketch of the physical layout of a neural processor is depicted with 16 hardware blocks 102 and 16 SRAM bank groups 109. In one embodiment, the SRAM bank group 109 memories may be placed in a distributed fashion, where each SRAM bank group 109 is located adjacent to (local to) exactly one of the blocks 102 forming the block and SRAM bank group unit 401. This allows for streaming of IFM and OFM data between each block 102 and its local SRAM 109 in a highly parallel fashion, i.e., having up to 16 IFMs and / or streams running in parallel, in order to avoid bandwidth bottlenecks between the SRAM and the compute blocks, which may exist if the SRAM is aggregated into a larger memory array and placed further away from the blocks (i.e., when the memory is not distributed).

[0495] Figure 4AB and Figure 4AC The connection between the block 102 and its local SRAM bank group 109 is depicted, as well as the contents of the SRAM bank group 109. Each SRAM bank group 109 may have four SRAM banks B0, B1, B2, B3 to provide sufficient bandwidth for concurrent read and write operations to service IFM, OFM transfer structures, CPU access through AXI ports (not shown), read and write partial results, and load weights. Figure 4AB The path between memory banks B0, B1, B2, and B3 is depicted via multiplexer 403 to the IFM transfer structure 104. This path can transfer up to two IFM slices per compute clock to supply sufficient IFM data to the tile capable of activating zero skipping. The IFM transfer structure 104 connects to the tile 102 to bring in IFM data from its local SRAM bank group as well as the other 15 SRAM bank groups. Each SRAM bank group 109 also supplies weights directly to its local tile 102, specifically, to the weight decompression unit 138 within the local tile 139. To make weight loading fast, all four SRAM banks B0 through B3 can fetch weights in parallel and input them to the WDU 139. Unlike in convolution, because the FC weights must be discarded after each multiplication, it is particularly important to load weights into the tile as quickly as possible during fully connected layer computations.

[0496] The multiple MU weight register files 127 in each MR block 102 can accept weight kernels of size 18*16*8=2304 bytes=144 words, where each word has 128 bits. For example, if the total SRAM capacity available to the neural processor 100 is 2M (mega) bytes, each SRAM bank group has (2M bytes) / (16 SRAM bank groups)=128K (kilo) bytes. Furthermore, if each SRAM bank group contains 4 SRAM banks, the size of each SRAM bank is (SRAM bank group size) / (number of SRAM banks per SRAM bank group)=128K / 4=32K bytes. Therefore, each of the four local SRAM banks can store 144 / 4=36 words (of the 2048 available words).

[0497] Figure 4AC The local OFM connection between the block and its local SRAM bank group is depicted. The block 102 outputs completed or partial results to the OFM transfer structure, which transfers the data to the local SRAM bank group and other SRAM bank groups elsewhere, and makes the data available to SRAM banks B0 to B3 via demultiplexer 405.

[0498] The following paragraphs discuss the IFM transfer structure 104 and the OFM transfer structure 106. The IFM transfer structure 104 forms connections and transfers data from the SRAM memory bank group 109 to the block 102, while the OFM transfer structure 106 forms connections and transfers data from the block 102 back to the SRAM memory bank group 109.

[0499] Given the task of bringing IFM data from the SRAM bank groups to the blocks and bringing OFM from the blocks back to the SRAM, it might seem that the connections between the SRAM bank groups and the blocks must be all-to-all, and the connections between the blocks and the SRAM bank groups must also be many-to-many. Having many-to-many connections may require the use of a cross-bar switch (e.g., 16 to 16), which in a case like this may consume excessive silicon area and is therefore highly undesirable. More specifically, the area of ​​a full crossbar switch is proportional to O(NM), where N is the number of switch inputs and M is the number of switch outputs. In the case where N=M=T=16, where T is the number of physical blocks, this makes O(NM)=O(T 2 ), T 2 is the square of the number of tiles, and makes increasing (scaling up) the number of tiles (e.g., from 32 to 32 or 64) extremely expensive relative to silicon area.

[0500] However, as discussed in detail below, a many-to-many connection between the blocks and the SRAM bank groups is not required. To reduce the size and complexity of the communication structure, some embodiments aim to store the OFM locally where the OFM will be generated (by each of the physical blocks) by partitioning the SRAM into non-overlapping stores. IFM data is still transferred from the respective SRAM bank groups 109 to each block 102, however, the IFM transfer structure configuration can be reduced to 5 necessary modes corresponding to the 5 main modes of reduction between blocks. Note that instead of storing the OFM locally and retrieving the IFM in a distributed (global) manner, the IFM transfer structure 104 and the OFM transfer structure 106 can also be constructed to extract the IFM locally while writing the OFM results in a distributed (global) manner.

[0501] Typically, convolutional or fully connected layer computations can be decomposed into one of the following five configurations regarding inter-block reduction: (1) Figure 4AD As depicted in FIG, an IFM slice is input by broadcasting the IFM slice to all 16 blocks 102 resulting in a total of 16 OFM slices; (2) as Figure 4AE As depicted in , two IFM slices are input in parallel by broadcasting each of the two IFM slices to 8 blocks; (3) as Figure 4AG As depicted in , four IFM slices are input in parallel by broadcasting each of the four IFM slices to four blocks; (4) as Figure 4AJ As depicted in , 8 IFM slices are input in parallel by broadcasting each of the four IFM slices to 2 blocks; (5) as Figure 4AL As depicted in , 16 IFM slices are input in parallel by broadcasting each of the 16 IFM slices to 1 block.

[0502] Because each IFM slice is broadcast to 8 blocks and the outputs of 2 blocks are reduced by the reduction structure 111 to obtain a complete (or partial) result, case (2) can be referred to as a "broadcast 8 reduce 2" case. Similarly, because each IFM slice is broadcast to 4 blocks 102 and the outputs of 4 blocks 102 are reduced, case (3) can be referred to as a "broadcast 4 reduce 4" case. Because each IFM slice is broadcast to 2 blocks 102 and the outputs of 8 blocks 102 are reduced, case (4) can be referred to as a "broadcast 2 reduce 8" case. Because each IFM slice is broadcast to only one block 102 (i.e., no broadcasting) and the outputs of all 16 blocks 102 are reduced, case (5) can be referred to as a "broadcast 1 reduce 16" case. Because the IFM slice is broadcast to 16 tiles 102 and the output of 1 tile 102 is reduced (ie, there is no reduction), case (1) may be referred to as a "broadcast 16 reduce 1" case.

[0503] The five inter-tile reduction configurations may be considered in more detail regarding what connection patterns the IFM transport structure 104 and the OFM transport structure 106 must support in each of the five reduction configuration cases. For additional clarity, the term "inter-tile reduction" is referred to herein as specifying the use of the reconfigurable adder tree provided by the reduction structure 111 to reduce the reduce block outputs, as opposed to "intra-tile reduction," which is referred to herein as specifying the use of the adder trees 128A, 128B internal to the block 102 to reduce the multiplier unit products.

[0504] The following notation can be used to identify situations where the interconnect structure can be put into use. The notation Bm-Rn- represents the situation where each IFM slice is broadcast to m blocks and the outputs of n blocks are reduced by the inter-block reduction structure 111 to obtain the result. In the case where 16 physical blocks are available, the five inter-block reduction situations include Figure 4AD B16-R1 depicted in Figure 4AF B8-R2 depicted in Figure 4AH B4-R4 depicted in Figure 4AK B2-R8 as depicted in Figure 4AM B1-R16 depicted in.

[0505] The maximum number of inter-block reduce cases is equal to LOG2(N), where N is the number of physical blocks in the neural processor 100. The inter-block reduce configurations available in a neural processor with N blocks are constructed starting with configuration BN-R1 (m=N and N=1), then dividing m by two and multiplying n by two for each next configuration until m reaches 1. For example, if the neural processor 100 has only 8 blocks, there are four available inter-block configurations, including B8-R1, B4-R2, B2-R4, and B1-R8. A neural processor 100 with 32 blocks can provide up to six inter-block configurations, including B32-R1, B16-R2, B8-R4, B4-R8, B2-R16, and B1-R32.

[0506] Since calculations can produce both a final result (e.g., when an activation function is applied) and a partial result, each inter-block configuration may have two cases to consider regarding the OFM transmission path. These two cases include a case where the final result is Bm-Rn-F and a case where the partial result is Bm-Rn-P.

[0507] Figure 4AE 、 4AG4AJ, 4AL, and 4AN further depict block outputs being added together through the reduction structure 111 in each of the five reduction configurations. For example, Figure 4AL Depicts the B2-R8 configuration, where the outputs of the eight blocks T0, T8, T4, T12, T10, T2, T14, and T6 are fed through an adder tree ( Figure 4AK The outputs of the eight blocks T7, T15, T3, T11, T13, T5, T9 and T1 are summed through another adder tree ( Figure 4AK The right adder tree in is summed.

[0508] Note that the configurable adder tree of the reduction structure 111 is designed to add the outputs of adjacent blocks 102 rather than adding the outputs of blocks 102 that are spread apart from each other, thereby making the configurable adder tree of the reduction structure compact and the tree itself "distributed." Note also that, unlike in the previous example, the 16 blocks here are identified as T0 to T15, and the ordering of the block identification numbers has been changed (compared to the notation used in the mapping example) to simplify the notation in the following examples.

[0509] Each inter-block reduction configuration can be examined in detail one by one. The first example case includes B16-R1 operation. Following the principle of getting IFM globally (from any SRAM bank group) while storing OFM as locally as possible, in this configuration, the input IFM can be streamed from any SRAM bank group S0..S15. Figure 4BA As shown in FIG, the SRAM bank group S10 provides an IFM slice stream to all 16 blocks T0 to T15 via the IFM transmission structure 104 (eg, Figure 4AD , one IFM slice is broadcast to all 16 tiles. When one SRAM bank group (e.g., S10), for example, runs out of IFM data, another SRAM bank group (e.g., S11) can become the data source and continue streaming IFM data to the tiles. These steps can continue until the entire IFM tensor has been streamed in. If multiple IFM passes are required, the IFM tensor streaming sequence can be repeated as needed.

[0510] In the B16-R1 configuration, there are no inter-block reductions, so that the adder unit of each block 102 only accumulates the results of that block, and as described below, the OFM complete or partial results will be written to the nearby SRAM memory bank group 109. Therefore, when the result is a stream of final or partial results, each of the 16 blocks 102 in the B16-R1 configuration generates a stream of OFM slices. Specifically, in some cases, when operating with 8-bit IFM and OFM, each value can be up to 32 bits wide, or when assuming 16-bit IFM and OFM data, each value can be up to 48 bits wide, and as Figure 4BB As shown by the arrows in , each partial result can be stored locally. In this case, each SRAM memory bank group 109 acts as a destination for storing partial results. In addition, each SRAM memory bank group 109 receives data from its local block, for example, SRAM memory bank group S8 receives data from block T8, S0 receives data from T0, and so on. Since each SRAM memory bank group 109 has 4 SRAM memory banks 108, each SRAM memory bank group 109 can typically store 16 4-byte partial results per clock. However, the current source SRAM memory bank group 109 must simultaneously extract IFM data and write partial results, which in some cases may exceed the total available bandwidth of the SRAM memory bank group. When the convolution plane kernel size is larger than 1x1, the IFM cache 139 can help reduce the IFM reads from the source SRAM memory bank group 109 in cases like this. Furthermore, operations using IFM weight loops and / or convolution plane kernel sizes larger than 1x1 generate output once in several clocks (as opposed to one result per clock), thus reducing requirements on OFM bandwidth and avoiding SRAM access bottlenecks.

[0511] When the final results are generated, each final value may be quantized to 8 bits (or 16 bits, etc.) and the value may be written to the SRAM bank group [S0..S7] or [S8..S15]. Figure 4BC and Figure 4BD The OFM transmission structure connections and configuration options are depicted. Since the OFM slice width is half the IFM slice width (8 deep channels vs. 16 deep channels), the outputs of two vertically adjacent tiles ("tile columns") can be sent to either the upper SRAM bank group or the lower SRAM bank group via short local connections. Each SRAM bank group is capable of processing a slice with 16 channels (since an IFM slice has 16 channels), so each SRAM bank group 109 can also accept two OFM slices. For example, the outputs of tiles T0 and T8, which together form a tile column, can be grouped together and sent to, for example, Figure 4BC The SRAM bank group S8 shown in FIG is located immediately below T8 or located as shown in FIG. Figure 4BD Similarly, block column T4 T12 outputs can be combined and sent locally to S4 or S12, block column T10 T2 outputs to S10 or S2, block column T14 T6 outputs to S14 or S6, block column T7 T15 outputs to S7 or S15, block column T3 T11 outputs to S3 or S11, block column T13 T5 outputs to S13 or S5 and block column T19 T1 outputs to S9 or S1.

[0512] The second example case depicts the B8-R2 operation. Figure 4CA As shown in FIG, one IFM slice can be supplied from the upper SRAM bank group 109, where the term "upper" is defined to include S0, S4, S10, S14, S7, S3, S13, and S9, and one IFM slice can be supplied from the lower SRAM bank group 109, where the term "lower" is defined to include S8, S12, S2, S6, S15, S11, S5, and S1. More specifically, any one of the upper SRAM bank groups 109 can act as a source for sending (broadcasting) an IFM slice to all upper blocks T0, T4, T10, T14, T7, T3, T13, and T9. For example, the IFM transmission structure 104 can be configured to read an IFM slice from S10 and broadcast the IFM slice to T0, T4, T10, T14, T7, T3, T13, and T9. Alternatively, for example, the IFM transport structure 104 may be configured to read the IFM slice from S3 and broadcast the IFM slice to T0, T4, T10, T14, T7, T3, T13, and T9.

[0513] Similarly, any one of the lower SRAM banks 109 can act as a source for sending (broadcasting) an IFM slice to all lower blocks T8, T12, T2, T6, T15, T11, T5, and T1. For example, the IFM transfer structure 104 can be configured to read an IFM slice from S11 and broadcast the IFM slice to T8, T12, T2, T6, T15, T11, T5, and T1. Alternatively, for example, the IFM transfer structure 104 can be configured to read an IFM slice from S8 and broadcast the IFM slice to T8, T12, T2, T6, T15, T11, T5, and T1.

[0514] In addition, refer to Figure 4CA , the SRAM bank groups 109 can be paired to send IFM slices so that data is received from one of the following pairs in one (clock) cycle: [S0, S1], [S2, S3], [S4, S5], [S6, S7], [S8, S9], [S10, S11], [S12, S13], and [S14, S15]. For example, in Figure 4CA , the IFM slice originates from the [S10, S11] pair of the SRAM bank group 109.

[0515] Figure 4CB Depicts the input of two IFM slices, where each IFM slice is broadcast to 8 blocks, and the output of the two blocks is reduced in a column-wise manner. For example, in Figure 4AFAfterwards, the output of T0 and the output of T8 are reduced to generate one result; the outputs of T4 and T12 are reduced to generate another result; the output of T10 and T2 are reduced to generate yet another result; the output of T14 and T6 are reduced to generate yet another result; the output of T7 and T15 are reduced to generate yet another result; the output of T3 and T11 are reduced to generate yet another result; the output of T13 and T5 are reduced to generate yet another result; and the output of T9 and T1 are reduced to generate yet another result.

[0516] In the case of partial results, the eight reduction results can be stored in one of the two groups [S0..S7] and [S8..15] of SRAM memory banks. For example, Figure 4CB Depicted are eight partial results stored in the SRAM bank group [S0..S7]. In the case of a final result, the OFM transfer structure 106 may merge the results of two adjacent block columns stored in one of the four SRAM bank groups including [S0..S3], [S4..S7], [S8..S11], and [S12..S15]. For example, Figure 4CC Depicted are the eight final results stored in the SRAM bank group [S4..S7].

[0517] The third example case depicts the B4-R4 operation. Figure 4DA As shown, one IFM slice can be supplied from each quarter of the floor plan. Figure 4DB , which may involve broadcasting four IFM slices and generating four results after reduction. As long as the IFM slices are from one of the four groups including [S0..S3], [S4..S7], [S8..S11], and [S12..S15], and as Figure 4DB As depicted in the case where the result is partial, as long as the output is written to one of the four groups [S0..S3], [S4..S7], [S8..S11] and [S12..S15], and as Figure 4DC As depicted in FIG, if the result is one of the eight groups [S0 S1], [S2S3], [S4 S5], [S6 S7], [S8 S9], [S10 S11], [S12 S13], and [S14 S15], the IFM transport structure 104 and the OFM transport structure 106 can manage to send inputs and receive outputs in one (clock) cycle.

[0518] Reference Figure 4AJ , note that each reduction group 407 generates one output result. Two results can be stored in the top column, and two results can be stored in the bottom column. Because the OFM slice containing the final result has a size of 8 bytes, the OFM transfer structure 106 can merge the results of two adjacent columns. Figure 4AHIt also depicts that four IFM slices are broadcasted to form four output results after the reduction.

[0519] The fourth example case depicts the B2-R8 operation. Figure 4EA As depicted in , an IFM slice can be provided from every eighth of the floor plan. Figure 4EB , the operation may involve broadcasting two IFM slices to generate two results after the reduction.

[0520] The IFM transfer structure 104 and the OFM transfer structure 106 can manage to send inputs and receive outputs in one (clock) cycle as long as the inputs come from one of the two groups including [S0..S7] and [S8..S15], and as long as the outputs are written to one of the eight groups [S0 S1], [S2 S3], [S4 S5], [S6 S7], [S8 S9], [S10 S11], [S12 S13] and [S14 S15] if the result is partial, and any SRAM memory bank group 109 if the result is final.

[0521] Figure 4EA It is depicted that for a fourth example case source data is broadcast. Figure 4EB depicts that for the fourth example case partial results are formed, and Figure 4EC The final result is formed for the fourth example case. Figure 4AJ , each part 407 generates a result after reduction. One of the two results can be stored at the top and the other result can be stored at the bottom. Because the OFM slice containing the final result has a size of 8 bytes, the OFM transfer structure 106 can merge the results of two adjacent columns. Figure 4AK It also depicts that four IFM slices are broadcasted to form two output results after the reduction.

[0522] The fifth example case depicts the B1-R16 operation. Figure 4FA As depicted in , one IFM slice may be supplied from each SRAM bank group 109 corresponding to one broadcast. Figure 4FB , the operation may involve reducing the outputs of all 16 blocks 102 to generate one result that may be stored in any SRAM bank group 109 when the result is partial and when the result is final.

[0523] Because the OFM slice containing the final result has a size of 8 bytes, the OFM transfer structure 106 can merge the results of two adjacent columns. Figure 4AM Also depicted are 16 IFMs slicing the input to form a single output result after reduction.

[0524] The IFM transfer structure 104 and the OFM transfer structure 106 can be designed in a manner, including the examples described above, such that for an operation, calculations can always be performed and stored to the SRAM 109 in such a way that subsequent operations that consume the results of the previous operation can obtain those results for all permutations of the reduction configurations of the current and subsequent operations. For example, the current operation can use the B4-R4 reduction configuration and store its results to the SRAM bank group 109 after selecting the OFM transfer structure 106 connectivity associated with the B4-R4 reduction configuration. The next operation can use the B2-R8 reduction configuration with the associated selection of the OFM transfer structure 106 connectivity while being able to successfully obtain the data calculated and stored by the previous B4-R4 operation.

[0525] Figure 4G One possible implementation of the IFM transport fabric 104 that supports all of the previously described IFM transport fabric connectivity options for all of the reduction configurations is depicted. The fabric includes four bidirectional multi-drop buses, two of which are placed between the upper SRAM bank group and the upper block, and the other two bidirectional buses are placed between the lower SRAM bank group and the lower block. The buses can be connected in a round-robin fashion through registers 411 so that data from the upper bus can flow to the lower bus and back. Note that for clarity of explanation, in Figure 4G Additional pipeline registers that may be present in the IFM transfer structure 104 have been omitted.

[0526] Figure 4H One possible implementation of the OFM transport fabric 106 that supports all OFM transport fabric connectivity options for all previously described reduction configurations is depicted. The fabric consists of four bidirectional 16-byte wide multi-drop buses to support reduction configurations B2-R8 and B1-R16. Note that for clarity of explanation, Figure 4H Pipeline registers that may be present in the OFM transfer structure 106 have been omitted.

[0527] The reduction structure 111 can perform "inter-block" reductions (instead of the intra-block reductions performed by adder trees 128A and 128B) for all reduction configurations (e.g., B8-R2, B4-R4, B2-R8, and B1-R16 configurations) except configuration R1 (when there is no inter-block reduction). The reduction structure 111 includes Figure 5A, a reconfigurable adder tree consisting of reduce and accumulate (RAA) nodes 520 as depicted in FIG. Each RAA node 520 operates on a partial reduction result (i.e., the linear result before the activation function is applied). The RAA node 520 receives input from the ARU 167 in the same block column as the RAA node or from other RAA nodes. The RAA node 520 sends the output to a higher RAA node in the adder tree or back to the ARU 167. Then, if the result is final, the ARU 167 applies the activation function and sends the final result to the OFM transport structure 106. Alternatively, if the result is partial, the ARU 167 sends the partial result to the OFM transport structure 106 while bypassing the activation function.

[0528] Figure 5B The reduction structure 111 for the R16 configuration is depicted. Here, the ARU module 167 generates the partial reduction results (from the intra-block adder trees 128A and 128A) via Figure 1X The "to-reduce structure" output shown in FIG streams the results of the partial reduction to the first-level RAA node 502. The first-level RAA node 502 reduces the 16 ARU partially reduced data streams (ARU streams of partially reduced data) into 8 partially reduced data streams (streams of partially reduced data) in pairs. The second-level RAA 504 also reduces the 8 streams generated by the first-level RAA node 502 into 4 partially reduced data streams in pairs. The third-level RAA node 506 and the fourth-level RAA node 508 complete the reduction process to produce a fully reduced data stream (stream of fully-reduced data), which is sent to the ARU 167 of the block T14 for activation function application (when the final result is generated) and output to the OFM transmission structure 106. Note that the block T14 is physically located near the tree root RAA node 508 and corresponds to Figure 4FB ARU 167 in block T14.

[0529] Figure 5CA reduction structure 111 for an R8 configuration is depicted. Unlike the R16 configuration, the R8 configuration includes two adder trees (instead of one adder tree), wherein each adder tree has three stages instead of four stages. The first adder tree reduces the partially reduced data from the ARUs of blocks T0, T8, T4, T12, T10, T2, T14, and T6, and sends the fully reduced results to the ARU 167 of block T12 for data return. The second adder tree reduces the partially reduced data from the ARUs 167 of blocks T7, T15, T3, T11, T13, T5, T9, and T1, and sends the fully reduced results to the ARU 167 of block T13 for data return. Note that in Figure 4FB In FIG, each of the tiles T12 and T13 is physically located near the corresponding tree root RAA node 506 and corresponds to the ARU 167 of the tiles T12 and T3, respectively.

[0530] Figure 5D A configuration R4 is depicted with four adder trees, where each adder tree reduces the outputs of the partial reductions from four blocks. Figure 4DB The physical location of the ARU 167 is depicted in relation to the four tree root nodes.

[0531] Figure 5E A configuration R2 is depicted having eight adder trees, where each adder tree reduces the outputs of the partial reductions from two blocks 102 . Figure 4CB The physical locations of the ARUs relative to the eight tree root nodes are depicted.

[0532] at last, Figure 5F Configuration R1 is depicted without an adder tree, and the block ARU 167 outputs the result directly to the OFM transmit structure 106 without the need for a reduce structure 111. Figure 4BB Depicted is the physical location of ARU 167 in this case. Note that Figure 4BB 、 4BC The digital indications inside the ARU 167 in 4BD, 4CB, 4CC, 4DB, 4DC, 4EB, 4EC and 4DB are as follows Figures 5B to 5F The RAA tree node levels are indicated in , where level 0 corresponds to configuration R1 (no reduction structure is used). Configuration R1 is implemented by the ARU multiplexer 174 in the ARU, which directly sends data from the accumulator 130A (or 130B) to the activation function and the partial path (starting from the bit range selection unit 187), so that Figure 1X Note that for clarity of general explanation, some auxiliary logic that may be required to properly bypass the reduction structure 111 in the case of sparse activation support is not shown.

[0533] Figure 5G The reduction structure 111 formed by RAA nodes 502, 504, 506, and 508 is depicted. Note again that each RAA node is physically located near exactly one block 102. Each RAA node 502 receives input from two blocks in the block column where node 502 resides. There is exactly one RAA node 502 per block column. RAA node 508 receives its input from node 506, which in turn receives its input from node 504, which in turn receives its input from node 502. Note that block T12 does not have an RAA node 502 associated with it because there are 15 tree nodes, while the number of physical blocks is 16.

[0534] like Figure 5A As shown in FIG, each RAA node 520 has two functions, including reducing two inputs A and B using an adder 512 and accumulating the results of the reduction using an accumulator 518 and an adder 514. A multiplexer 516 allows the reduction result from the adder 512 to be loaded directly into the accumulator 518 when accumulation begins, for example, to start an IFM weight cycle. The multiplexer 516 also allows the reduction results to be accumulated as, for example, an IFM weight cycle progresses over time.

[0535] Storing weights in a compressed format can be beneficial in reducing the amount of SRAM (and off-chip DDR) memory required to store the weights, to reduce the SRAM (and off-chip DDR) power associated with extracting weights, and to speed up weight loading, particularly during fully connected layer computations. In some embodiments, idle cycles can be used to load multiplier unit weights. Additionally, in some embodiments, multiple vertical load buses 101 can be used to speed up weight loading, rather than depicting only one weight load bus per MR column. Figure 1K .

[0536] More specifically, as previously mentioned Figure 4AB As depicted in , weights are stored in four SRAM banks 108 local to each block 102, and each block 102 is able to read all four banks in parallel. Each SRAM bank 108 retrieves 16 8-bit weights. Because each block 102 has 8 MR columns, when the weights are not compressed, it takes (8 MR columns per block) / (4 local SRAM banks per block) = 2 clocks to load one 8-bit weight for each activation path. Each block 102 also includes a weight decompression unit 138 per block, which can be used to decompress FC and convolution weights. For example, each multiplier unit 103 may have 18 weights, and it may take (18 weights per MU) * (2 clocks per weight load) = 36 clock cycles to load all MU weights. Smaller cores that do not use all 18 weights can load faster.

[0537] Weight streaming concurrently with FC computations can be used to improve throughput in fully connected computations to keep multiplier utilization high during large FC computations. As previously mentioned, FC computations do not reuse weights. Therefore, it may be necessary to stream weights quickly during FC computations. Specifically, an FC computation with an IFM weight cycle of 1 will require providing one weight to each MU per clock to keep all multipliers 126 fully utilized. An IFM weight cycle of 2 requires providing one weight to each MU 103 every two clocks to keep all multipliers fully utilized. More generally, an IFM weight cycle of N requires providing one weight to each MU 103 every N clocks to keep all multipliers 126 fully utilized.

[0538] According to various deep learning research publications, the fully connected layer weights can be compressed, sometimes by a factor of 2 or more. In this case, one decompressed weight can be loaded into each MU 103 per clock, rather than one uncompressed weight per two clocks.

[0539] However, in addition, the IFM data must also be fetched from SRAM 109 along with the weights, thus reducing the SRAM bandwidth available for fetching the weights. The amount of IFM data fetched from SRAM 109, in turn, depends on the map-reduce configuration. A large reduction number (e.g., R16) requires more paths to fetch the IFM data than a smaller reduction configuration (e.g., R1).

[0540] Because all 64 SRAM banks may be busy fetching FC weights, IFM data may not be read from SRAM 109 at the same time. To increase multiplier utilization, IFM data may be stored spliced ​​across all 64 banks. In some embodiments, to fetch IFM data, weight reading stops for one clock cycle, and all 64 banks cause one IFM data to be read into a 1-deep cache register located next to the output of SRAM 109. The IFM data is then streamed from 64 16-byte rows of the cache. More specifically, fetching one IFM data from all 64 banks in parallel can fetch enough data at a time to equal the number of IFM data reads (R = (64 SRAM banks) * (broadcast configuration number B) / (number of physical blocks)). Therefore, as Figure 6 As shown in for some embodiments, the maximum multiplier utilization for fully connected layer computations may be calculated based on R / (1+R) as a function of the number of broadcast configurations B.

[0541] As previously described, the global control FSM 140, as well as the SRAM control FSM 142 and the block control FSM 144, may have various configuration registers. In some embodiments, the contents of some of these configuration registers may be capable of being switched on the fly to change the configuration of the neural processor 100 on the fly, for example, when the neural processor 100 transitions from one operation to another, or when one SRAM bank 109 runs out of data and the IFM transfer structure 104 must switch on the fly (without delay) to stream IFM data from another SRAM bank 109. Following commonly known design practices, such on-the-fly reconfiguration may be achieved by double-buffering the configuration registers and taking the new configuration into effect by switching between the two buffers. Figure 1A As depicted in FIG, the central controller 110 may receive configuration data from the CPU via the AXI bus and pass that configuration data via the common bus 112, which in turn may send and load the configuration values ​​from the CPU into the configuration registers of the control logic (such as the global control FSM 140, the SRAM control FSM 142, and the block control FSM 144), as well as various other registers including the ARU bias register 195, the scaling register 191, the activation function 197 configuration register, etc. In order to coordinate on-the-fly configuration changes involving a large number of double-buffered registers that are switched at various times as needed, the common bus 112 may be loaded with not only the configuration register values, but also the time (clock count) at which the double-buffered register must have its configuration switched to take effect.

[0542] Figure 1A Also depicted are SRAM bank groups 109, each with an AXI slave interface that enables the CPU to write to the IFM and weight tensors, and read back the OFM results. Because the SRAM bank groups service I / O requests from the IFM transport fabric 104 and OFM transport fabric 106, as well as local weight load connections, CPU I / O requests on the AXI interface 114 can be arbitrated and assigned a lower priority, allowing neural network computations to continue without delay while the CPU waits for results.

[0543] Additionally, the subject matter disclosed herein provides a scalable multiplexer circuit or module, referred to herein as a "butterfly shuffler," that efficiently scrambles data for purposes including homogenizing sparse data. There may be situations where sparse data, such as, in particular, data associated with an input feature map, may include non-zero values ​​that are clustered together. That is, the data may be non-uniform sparse data. In such situations, a system that can process sparse data in parallel, such as by multiplying input feature map (IFM) values ​​in parallel, may leave many multipliers idle (i.e., multipliers having at least one operand equal to 0), while a small group of multipliers may provide a large number of multiplications, thereby leading to a bottleneck condition.

[0544] For example, refer to Figure 7A , the IFM data in the memory or SRAM 109 has zero values ​​relatively evenly distributed in the IFM slices and in the paths in the IFM slices. The IFM buffer 141 may receive data from Figure 7A , and uses a look-ahead of 1 in conjunction with a look-aside of 1 to successfully multiplex non-zero activations in an out-of-order manner to implement activation skipping. For example, non-zero value 701 may be multiplexed one pass down and one position ahead to replace the zero value at position 702. Similarly, IFM buffer 141 may pass other non-zero values ​​out of order as indicated by the arrow marks.

[0545] Figure 7B The IFM data depicted in Figure 7A The same number of zero values; however, Figure 7B The zero values ​​in position 704 are clustered in the same IFM lanes of adjacent IFM slices. IFM buffer 141 would have to support lateral lookup 4 to successfully multiplex non-zero activations 703 in place of the zero value occupying position 704 to implement activation skipping. Supporting a large lateral lookup range (e.g., greater than 1) may be prohibitively expensive in terms of silicon area because multiplexer 163 would have more inputs to bring activation values ​​from lanes located further away.

[0546] Reference Figure 7C The IFM shuffler 720 may be used to pseudo-randomly shuffle the values ​​within each IFM slice to disperse the clustering of non-zero values ​​within the IFM slice, so that, for example, Figure 7B The arrangement of the data shown in the conversion bit Figure 7A The arrangement of the data is shown in .

[0547] Note that the pseudo-random scrambling of activations must be accompanied by scrambling of weights in the same manner so that the scrambled activations will be multiplied by the correct weights. It should also be noted that since the pseudo-random scrambling sequence can be known before the calculation, the weights can be scrambled offline per channel for each incoming IFM slice and loaded into the MR block 102 before the calculation begins.

[0548] In addition to shuffling the IFM slice values ​​on a per-pass basis, the IFM shuffler 720 can also reorder the temporal sequence of IFM slices. Note that the MR block weights must be reordered offline accordingly for the steps in the dot product calculation to match the changed order in which the IFM slices will arrive.

[0549] The IFM shuffler 720 can be efficiently implemented using a butterfly shuffler. Figure 7D The 16-channel (path) butterfly shuffler 740 can be composed of 64 2 (i.e., a and b) to 1 multiplexers M arranged in an array of 16 rows 0...15 and 4 columns 0...3. row,col 730. As shown, the butterfly shuffler 740 can flexibly scramble or rearrange the IFM slice values ​​arriving through 16 input paths (d0 to d15) into another IFM slice to be output through the output paths (o0 to o15).

[0550] Reference Figure 7D , the multiplexers 730 in each column are paired to create a 2×2 crossbar. More specifically, in the 16-way butterfly shuffler 740, the 16 multiplexers 730 in each column are grouped in pairs to form eight 2×2 crossbar switches. The control signals of the multiplexers belonging to a pair are connected together. The 16 multiplexers 730 in column 0 are paired to form eight 2×2 crossbars as follows: {M 0,0 ,M 1,0}、{M 2,0 ,M 3,0}、{M 4,0 ,M 5,0}、{M 6,0 ,M 7,0}、{M 8,0 ,M 9,0}、{M 10,0 ,M 11,0}、M 12,0 ,M 13,0}、{M 14,0 ,M 15,0 The eight result pairs are represented by the signals Sel0… 7,0 Control. Reset (de-asserting) sel x,col Make the corresponding crossbar pass the input to the output in an uncrossed manner. Asserting sel x,colCauses the corresponding crossbar to pass the input to the output (i.e., the input signal is switched at the output of the crossbar). For example, reset sel 0,0 So that the multiplexer {M 0,0 ,M 1,0 The 2×2 crossbar formed by} passes through channels 0 and 1 as channels 0 and 1 without change. Set sel 0,0 Use the multiplexer {M 0,0 ,M 1,0} Output paths 0 and 1 as paths 1 and 0 (i.e., swap (cross)).

[0551] Note that the multiplexer pairs in column 0 can be controlled by sel x,0 Controlled pairs of multiplexers {M x*2 ,0,M x*2+1 ,0}, where x is an integer from 0 to 7. More generally, in a butterfly shuffler with N lanes and M=log2(N) columns, the multiplexers in column c are paired as follows: x,c Controlled mod(x,k)+floor(x,k)*2,c ,M mod(x,k)+floor(x,k)*2+k,c}, where k = 2 c ,x∈[0,2 M-1 ], each column has 2 M-1 control signals, and there are a total of S=2 M-1 *M=N*log2(N) / 2 control scrambling signals, resulting in a total of 2 N*log2(N) / 2 A mess.

[0552] The butterfly shuffler 740 disclosed herein is not a full cross-bar multiplexer configuration. A full cross-bar configuration has a large area O(N 2 ), where N is the number of paths for the data. In contrast, the area of ​​the butterfly shuffler 740 is O(N*log(N)), where N is the number of paths for the data. In general, a full crossbar provides n! unique shuffles, while a butterfly shuffler with N paths produces 2 N*log2(N) / 2 For example, a 16-way butterfly shuffler has 2 per 16 ways. 16*4 / 2 =2 32 A mess.

[0553] Figure 7EA pseudo-random generator 741 (e.g., a linear feedback shift register) is shown controlling the scrambling of the butterfly shuffler data path 740. Before a calculation begins (e.g., to calculate a convolution at a specific location), the MR block's control logic can initialize the pseudo-random generator 741 to generate a known pseudo-random scrambling sequence to shuffle the data into the IFM slice. As previously mentioned, the weights preloaded into the MR block 102 to be used in this calculation must be pre-shuffled offline so that the post-shuffle order of the paths in each IFM slice is consistent with the weight's path index.

[0554] As described above, zero activation sparsity can be supported by lateral lookup and forward lookup mechanisms and is also enhanced by an IFM shuffler (such as a butterfly shuffler 740). Zero activation skipping using two adder trees per MU column can produce a maximum speedup of approximately 2 times and an average speedup of approximately 1.5 times. However, the input feature map structure and memory (SRAM) bandwidth may be limited. As previously described, the input feature map structure bandwidth in the example embodiment can be limited to 2 times to match the 2 times maximum speedup obtained by zero activation skipping. Therefore, the 2 times maximum speedup due to zero activation skipping can make the OFM structure throughput 2 times compared to the calculation with zero activation skipping disabled. The OFM structure throughput should also match the calculation throughput, thereby providing 2 times the bandwidth.

[0555] If the memory (SRAM) and / or IFM transport structure is limited to 2 times, for example due to the SRAM clock frequency or area or power constraints associated with the bandwidth of the IFM transport structure, further increases in zero activation skipping may be capped because the SRAM and / or IFM transport structure becomes a bottleneck in data transfer and the MR block multipliers become idle while waiting for data processing. More generally, computational speedup by any mechanism including zero activation and zero weight skipping may become capped. As previously mentioned, methods and devices for zero activation skipping have been proposed. However, convolutional and fully connected layer weights also typically exhibit sparsity (i.e., weight kernels can have a large number of zero weights). Therefore, it may be advantageous to explore zero weight multiplication skipping in addition to zero activation skipping while being mindful of the limited bandwidth constraints imposed by the bandwidth of the IFM transport structure and / or memory (SRAM).

[0556] For example, consider methods and apparatus for supporting weight sparsity, including combining weight sparsity with activation sparsity. Assuming that the IFM transport fabric bandwidth is capped at 2 times the baseline bandwidth (i.e., with all multiplication skipping methods disabled), the overall throughput of the weight sparsity scheme may also be capped at 2 times the baseline throughput. For this reason, for weight sparsity support, especially when combined with activation sparsity support to further increase computational speedup by more than 2 times, it may be advantageous to utilize another approach orthogonal to IFM transport (i.e., an approach that does not require a further increase in IFM transport fabric bandwidth).

[0557] One such method may be output feature map computation. More specifically, while keeping the IFM transmission structure bandwidth unchanged, the MU train may generate more than one output per OFM cycle. Figure 8A A baseline MU 810 is depicted with zero-activation skip logic omitted for clarity and also without zero-weight skip logic. Here, the weight register file 805 has 18 weights 815. Multiplier 822 receives the activations and register file 805 weights using an 18-to-1 multiplexer 820 to compute the product of the terms, which feeds into an adder tree to continue the dot product computation. Figure 8B MU 850 is depicted that supports double sparsity (i.e., both zero-valued activations and zero-valued weight skipping). Note that the weight register file 805 has been logically divided into two groups 811 and 812, each containing nine weights. Here, the first group of nine weights belongs to one output channel, while the second group of nine weights belongs to the second output channel. In other words, the output loop is typically kept at least 2. Mapping experiments conducted by the inventors have shown that keeping the output loop at least 2 can be feasible for most layers of popular neural network models, while for the remaining layers, logical weight register grouping can be disabled.

[0558] Zero-valued weight skipping may continue to check whether the weight values ​​in group 0 scheduled for the upcoming multiplication are equal to zero, and in that case, instead use the weights in group 1. If the weights in both group 0 and group 1 have zero values, the MU may process the next pixel.

[0559] In another aspect of the subject matter disclosed herein, reference is made to Figure 8C , the ABU may broadcast an additional activation group 850 corresponding to the next activation (i.e., the activation that typically follows the currently broadcasted activation 750) with reference to the order of activations scheduled by the IFM buffer 124 as a result of the application of the zero-skip forward lookup and the lateral lookup. Figure 8B, and accordingly, MU 850 may receive two sets of activation broadcast buses. In particular, the additional activation bus allows faster columns (i.e., columns with all MUs that have been able to skip multiplication due to zero activations and / or zero weights) to advance to the next pixel. Also note that while some columns can continue to calculate the next pixel in this out-of-order manner, the number of activation buses per MU row limits how far the columns can continue out of order (i.e., in Figure 8B Only one pixel in the example depicted).

[0560] Note that, as previously mentioned, the IFM shuffle can be particularly helpful in enabling two sets of activations to be sent in each cycle when the aggregation of non-zero values ​​becomes dispersed (ie, evened out).

[0561] In summary, the proposed dual sparsity approach may have the following advantages: exploiting weight sparsity in addition to activation sparsity, not requiring higher IFM and / or SRAM bandwidth, while improving computational speedup by more than a 2x upper limit (i.e., computing 2x faster relative to the baseline (with sparsity support disabled)), while receiving IFM data no faster than 2x. Another advantage of the proposed dual sparsity approach may be the reuse of the weight selection multiplexer 820 when weights are grouped logically rather than physically. A specific embodiment may choose not to use lateral lookups for zero activation skips, thereby not requiring lateral lookup logic and multiplexers to bring (borrow) weights from neighboring MUs. Note that having IFM shuffling in the absence of lateral lookup logic would be particularly advantageous for such an embodiment. Finally, logically, for computational mapping purposes, such computations may be viewed essentially as processing 16 output columns per block using 16×8 multipliers instead of 8.

[0562] As used herein, the terms "multiplexer" and "demultiplexer" are used interchangeably; each term means a switchable device having multiple data terminals (e.g., data input or data output) on one side (the "multi-port" side) and a single data terminal (e.g., data output or data input) on the other side (the "single-port" side), the device being configured to connect one of the multiple data terminals on one side to the single data terminal on the other side, selected based on a control signal received at a control input of the device.

[0563] The term "processing unit" is used herein to include any combination of hardware, firmware, and software for processing data or digital signals. Processing unit hardware may include, for example, an application-specific integrated circuit (ASIC), a general-purpose or dedicated central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), and a programmable logic device (such as a field programmable gate array (FPGA)). In a processing unit as used herein, each function is performed by hardware that is configured (i.e., hardwired) to perform that function, or by more general hardware (such as a CPU) that is configured to execute instructions stored in a non-transitory storage medium. A processing unit may be manufactured on a single printed circuit board (PCB) or distributed across several interconnected PCBs. A processing unit may contain other processing units; for example, a processing unit may include two processing units, an FPGA, and a CPU, interconnected on a PCB.

[0564] It will be understood that although the terms "first," "second," and "third" may be used herein to describe various elements, components, regions, layers, and / or parts, these elements, components, regions, layers, and / or parts should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer, or part from another element, component, region, layer, or part. Thus, a first element, first component, first region, first layer, or first part discussed herein could be referred to as a second element, second component, second region, second layer, or second part without departing from the spirit and scope of the present inventive concept.

[0565] For ease of description, spatially relative terms (e.g., "below," "beneath," "beneath," "above," "above," etc.) may be used herein to describe the relationship between one element or feature and another element or feature depicted in the accompanying drawings. It will be understood that such spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the accompanying drawings. For example, if the device in the accompanying drawings is turned over, an element described as being "below" or "beneath" another element or feature would thereafter be positioned "above" the other element or feature. Thus, the example terms "below" or "beneath" can include both the orientations "above" and "below." The device may be positioned otherwise (e.g., rotated 90 degrees or in other orientations), and the spatial descriptions used herein should be interpreted accordingly. Furthermore, it will be understood that when a layer is referred to as being "between" two layers, it can be the only layer between the two layers, or one or more intervening layers may also be present.

[0566] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the subject matter disclosed herein. As used herein, the terms "substantially," "about," and similar terms are used as terms of approximation rather than terms of degree, and are intended to account for the inherent variations in measurements or calculations that one of ordinary skill in the art would recognize.

[0567] As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. It will also be understood that the term "comprising", when used in this specification, indicates the presence of the described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the relevant listed items. A statement such as "at least one of...", when following a list of elements, modifies the entire list of elements without modifying the individual elements in the list. In addition, when describing embodiments of the subject matter disclosed herein, the use of "may" refers to "one or more embodiments of the present disclosure". In addition, the term "exemplary" is intended to mean an example or illustration. As used herein, the term "using" may be considered synonymous with the term "utilizing".

[0568] It will be understood that when an element or layer is referred to as being “on,” “connected to,” “coupled to,” or “adjacent to” another element or layer, the element or layer can be directly on, directly connected to, directly coupled to, or directly adjacent to the other element or layer, or one or more intervening elements or layers may be present. In contrast, when an element or layer is referred to as being “directly on,” “directly connected to,” “directly coupled to,” or “immediately adjacent to” another element or layer, there are no intervening elements or layers present.

[0569] Any numerical range listed herein is intended to include all subranges of the same numerical precision contained within the listed range. For example, the range of "1.0 to 10.0" is intended to include all subranges between (and including) the listed minimum value of 1.0 and the listed maximum value of 10.0 (i.e., having a minimum value equal to or greater than 1.0 and a maximum value equal to or less than 10.0 (e.g., 2.4 to 7.6)). Any maximum numerical limit listed herein is intended to include all lower numerical limits contained therein, and any minimum numerical limit listed in this specification is intended to include all higher numerical limits contained therein.

[0570] Although exemplary embodiments of a neural processor have been specifically described and illustrated herein, many modifications and variations will be apparent to those skilled in the art. Therefore, it will be understood that a neural processor constructed according to the principles of the present disclosure may be implemented in ways other than as specifically described herein. The present invention is further defined by the following claims and their equivalents.

Claims

1. A processor, comprising: a register storing a first set of weight values ​​and a second set of weight values, each set of weight values ​​including at least one weight value, and a weight value in the first set of weight values ​​corresponding to a weight value in the second set of weight values ​​in one-to-one correspondence; a non-zero weight value selector that selects a non-zero weight value from the weight values ​​in the first set of weight values ​​or the weight values ​​in the second set of weight values ​​that correspond to the weight values ​​in the first set of weight values; as well as A multiplier that multiplies the selected non-zero weight value and the activation value corresponding to the selected non-zero weight value to form an output product value.

2. The processor according to claim 1, wherein: The weight values ​​in the first set of weight values ​​and the weight values ​​in the second set of weight values ​​that correspond to the weight values ​​in the first set of weight values ​​both include zero-valued weight values, and The non-zero weight value selector controls the multiplier to prevent the multiplier from forming an output product value.

3. The processor according to claim 1, wherein: A first weight value in the first set of weight values ​​and a first weight value in the second set of weight values ​​corresponding to the first weight value in the first set of weight values ​​both include a zero-valued weight value, and In which, the non-zero weight value selector selects a non-zero weight value from the second weight value in the first group of weight values ​​and the second weight value in the second group of weight values ​​corresponding to the second weight value in the first group of weight values, and the second weight value in the first group of weight values ​​is different from the first weight value in the first group of weight values.

4. The processor according to claim 1, wherein: The first set of weight values ​​includes nine weight values, and the second set of weight values ​​includes nine weight values.

5. The processor of claim 1 , further comprising: Multiplexer, combined between register and multiplier, The non-zero weight value selector controls the multiplexer to couple the selected non-zero weight value to the multiplier. The processor according to claim 1 , wherein: The processor is part of a neural processor.

7. The processor according to claim 1, wherein: The selected non-zero weight value consists of a uint8 value.

8. A processor, comprising: a register receiving N weight values, wherein N is a positive even integer greater than 1, the N weight values ​​being logically arranged into a first group and a second group, the first group and the second group being of equal size, and the weight values ​​in the first group corresponding one-to-one to the weight values ​​in the second group; a multiplexer coupled to the register, the multiplexer selecting and outputting a non-zero weight value from either the weight values ​​in the first group or the weight values ​​in the second group corresponding to the weight values ​​in the first group; and The multiplier multiplies the non-zero weight value output from the multiplexer and the activation value corresponding to the non-zero weight value output from the multiplexer to form an output product value.

9. The processor of claim 8, further comprising: A weight value selector controls the multiplexer to output a non-zero weight value based on whether the weight values ​​in the first group are equal to zero and whether the weight values ​​in the second group corresponding to the weight values ​​in the first group are equal to zero.

10. The processor of claim 9, wherein: The weight values ​​in the first group and the weight values ​​in the second group corresponding to the weight values ​​in the first group both include zero-valued weight values, and The weight value selector further controls the multiplier to prevent the multiplier from forming an output product value.

11. The processor according to claim 9, wherein: The first weight value in the first group and the first weight value in the second group corresponding to the first weight value in the first group both include a zero-valued weight value, and The weight value selector selects a non-zero weight value from the second weight value in the first group and the second weight value in the second group corresponding to the second weight value in the first group, and the second weight value in the first group is different from the first weight value in the first group.

12. The processor according to claim 8, wherein: The first group includes nine weight values, and the second group includes nine weight values.

13. The processor according to claim 8, wherein: The processor is part of a neural processor.

14. The processor according to claim 8, wherein: Non-zero weight values ​​output from the multiplexer consist of uint8 values.

15. A processor, comprising: A first register receives N weight values, where N is a positive even number greater than 1, and the N weight values ​​are logically arranged into a first group and a second group, the first group and the second group have equal sizes, and the weight values ​​in the first group correspond one-to-one with the weight values ​​in the second group; a multiplexer coupled to the first register, the multiplexer selecting and outputting a non-zero weight value from either the weight values ​​in the first group or the weight values ​​in the second group corresponding to the weight values ​​in the first group; a second register receiving a plurality of activation values; and A multiplier is coupled to the multiplexer and the second register, and the multiplier multiplies the non-zero weight value output from the multiplexer and the activation value corresponding to the non-zero weight value output from the multiplexer and received from the second register to form an output product value.

16. The processor of claim 15, further comprising: A weight value selector controls the multiplexer to output a non-zero weight value based on whether the weight values ​​in the first group are equal to zero and whether the weight values ​​in the second group corresponding to the weight values ​​in the first group are equal to zero.

17. The processor of claim 16, wherein: The weight values ​​in the first group and the weight values ​​in the second group corresponding to the weight values ​​in the first group both include zero-valued weight values, and The weight value selector further controls the multiplier to prevent the multiplier from forming an output product value.

18. The processor of claim 16, wherein: The first weight value in the first group and the first weight value in the second group corresponding to the first weight value in the first group both include a zero-valued weight value, and The weight value selector selects a non-zero weight value from the second weight value in the first group and the second weight value in the second group corresponding to the second weight value in the first group, and the second weight value in the first group is different from the first weight value in the first group.

19. The processor of claim 15, wherein: The first group includes nine weight values, and the second group includes nine weight values.

20. The processor of claim 15, wherein: The processor is part of a neural processor.

Images