A computer architecture for a convolutional neural network

Abstract

A computer architecture for a convolutional neural network (CNN) is disclosed comprising a systolic array of processing elements. The processing elements are configured to: load at least one weight and a plurality of inputs; selectively multiply the at least one weight with each of the plurality of inputs to produce a product; accumulate the product with a partial sum input from another processing element, when available; and output a partial sum output resulting from the accumulation. The processing elements are configured for triangular movement of inputs between processing elements in the systolic array, which results in the plurality of inputs. The computer architecture also comprises at least one storage register configured to temporarily store an input during said triangular movement of inputs; and an adder configured to accumulate the partial sum outputs from the systolic array and to output a result of the accumulation.

Description

[0001] A Computer Architecture for a Convolutional Neural Network

[0002] FIELD OF DISCLOSURE

[0003] The disclosure relates to a computer architecture for a convolutional neural network (CNN).

[0004] BACKGROUND

[0005] Artificial Intelligence (Al) is a pervasive paradigm that has changed the way devices can assist everyday activities. However, in order to continuously meet high standards of accuracy, Al models are becoming ever more data-intensive, particularly when Deep Neural Networks (DNNs) are considered. Indeed, in addition to demanding a huge number of computations, DNNs also exhibit high memory requirements to store learned weights, as well as inputs and outputs.

[0006] A Convolutional Neural Network (CNN) is an example of a data-intensive DNN, since it executes convolutions on multidimensional arrays, known as feature maps (fmaps), to carry out tasks like image classification, image segmentation, object detection and speech recognition. Conventionally, Central Processing Units (CPUs) and Graphics Processing Units (GPUs) manage CNN workloads. However, such known architectures suffer from the well-known Von Neumann bottleneck, owing to the physical separation between the computing core and the memory. This significantly impacts energy efficiency, given that data needs to be first fetched from an external Dynamic Random Access Memory (DRAM), then buffered on-chip, and finally processed by the computing core. By way of example, it has been shown that a normalized DRAM energy cost from a commercial 65 nanometer (nm) process can be 200 times higher than the energy cost associated with a Multiply-Accumulation (MAC) operation, which is the elementary operation performed by the computing core. Data reuse is one way to mitigate such cost, by allowing inputs, weights, or partial sums (psums) to be held on- chip as long as they need to be consumed.

[0007] More specifically, a CNN consists of a sequence of convolutional layers (CLs) that extract features of interest from input data, by emulating the behavior of humans’ visual cortex to retrieve patterns, such as shapes and edges, from images. These fmaps can be eventually processed by Fully-Connected Layers (FCLs) for classification. As illustrated in Figure 1, each CL performs N three-dimensional convolutions between M input fmaps (ifmaps), each consisting of a Hi x Wi plane, and N filters, each consisting of M kernels of KHXKW weights. In other words, each filter scans the M ifmaps through sliding windows with stride S, where S is the distance between consecutive sliding windows.

[0008] For simplification, this disclosure assumes KH = Kw = K and S = 1. As a result, N HoxWo output fmaps (ofmaps) are generated, with Ho = Hi -K +1 and Wo = Wi -K +1. Each output element is named activation and follows equation 1 below: where O identifies the generic output activation, I is the generic input activation and W is the generic weight belonging to a kernel of one filter; n iterates over the N ofmaps, h0iterates over the Ho rows of each ofmaps, w0iterates over the Wo elements belonging to each ofmap’s row, m iterates over the M ifmaps, kh and kwiterate over the kernels’ rows and columns, respectively. It is worth underlining that N biases can be eventually added to each output activation if required by the layer.

[0009] Figure 2 illustrates the way in which a CL behaves, and provides an example of convolution between a 5 x 5 ifmap (which may also be referred to as an input matrix), and a 3 x 3 kernel (which may also be referred to as a weight matrix), which produces a 3 x 3 ofmap (which may also be referred to as an output matrix).

[0010] Systolic Arrays (SAs) are representative architectures that implement data reuse by using an array of Processing Elements (PEs) interconnected with each other. Data moves rhythmically through the array, thus evoking the blood flow in the cardiovascular system, hence the term systolic. Conceived in the late 1970s, SAs have been mainly used for matrix multiplications, thus not being directly compatible with CNNs. In recent years, the research community has made an effort to allow SAs to meet the CNN’s workflow, either by rearranging ifmaps differently and using data redundancy, or by proposing specific dataflows to directly cope with CLs’ workflow.

[0011] For instance, General Matrix Multiplication (GeMM) rearranges ifmaps and introduces data redundancy to make the inputs’ scheduling compliant with matrix multiplication. However, this reflects in higher memory capacity and, in turn, a higher number of memory accesses, thus negatively impacting area and energy. Weight Stationary (WS) SAs are examples of architectures using GeMM. Basically, inputs are supplied from the main memory and moved horizontally along the array, while weights are kept stationary on-chip. A known Tensor Processing Unit (TPU) refers to this case and consists of 256 x 256 PEs, able to outperform CPUs and GPUs in terms of energy efficiency by 30 times at least. In order to avoid any rearrangement of ifmaps, the Row Stationary (RS) SA has been proposed subsequently. Specifically, rows of inputs and weights are stored and reused at the PE level through dedicated memory blocks. However, this requires a more complex micro-architecture, also considering that control logic is required to orchestrate said reuse of inputs and weights. In addition, since inputs and weights circulate cycle-by-cycle inside each PE, higher switching activity is exhibited impacting, in turn, the energy consumption. Moreover, the area covered by the SA depends on the ifmaps’ sizes, thus making the deployment of large-scale architectures a challenge. Eyeriss is the pioneer RS-based SA and consists of 168 PEs, outperforming prior SAs from 1.4 to 2.5 times in terms of energy efficiency when dealing with convolutions.

[0012] Other SAs include Input Stationary (IS) SAs where a batch of inputs is retained inside the PEs, while weights and psums move between adjacent PEs throughout the process; and Output Stationary (OS) SAs where psums are reused at the PE level until the final sum is generated and inputs and weights move between adjacent PEs throughout the process.

[0013] It is an aim of the present disclosure to provide a computer architecture that address one or more of the problems above or at least provides a useful alternative.

[0014] SUMMARY

[0015] In general, this disclosure proposes a novel dataflow for SAs based on a triangular input movement (TrIM) between PEs, which is particularly suitable for convolutional layers (CLs) for CNNs. Overall, TrIM aims at maximizing local data reuse to reduce the number of memory accesses.

[0016] According to a first aspect of the present disclosure, there is provided a computer architecture for a convolutional neural network (CNN) comprising: a systolic array of processing elements, the processing elements being configured to: load at least one weight and a plurality of inputs; selectively multiply the at least one weight with each of the plurality of inputs to produce a product; accumulate the product with a partial sum input from another processing element, when available; and output a partial sum output resulting from the accumulation; wherein the processing elements are configured for triangular movement of inputs between processing elements in the systolic array, which results in the plurality of inputs; at least one storage register configured to temporarily store an input during said triangular movement of inputs; and an adder configured to accumulate the partial sum outputs from the systolic array and to output a result of the accumulation.

[0017] Thus, the disclosure provides a computer architecture which is suitable for a convolutional neural network (CNN) and which is configured to reuse inputs after they have been fetched from memory, by circulating the inputs between processing elements (PEs), by way of a triangular movement.

[0018] For example, the inputs may flow in path forming a right-triangular shape when viewed from above a plane of the array, by the inputs initially being supplied vertically, then being moved from right to left, and finally being moved diagonally upwardly and to the right to close the triangular shape. Of course, in other examples, the inputs may flow in a path forming a left-triangular shape or the array may be rotated, for example, by 90 degrees so that the vertical direction becomes a horizontal direction.

[0019] Notably, the at least one storage register is provided to assist with the triangular movement of the inputs, for example, from an end PE in a second row of the array to an end PE in a first row, via a storage register provided adjacent to the end PE in the second row.

[0020] Advantageously, the design of the computer architecture is not limited by a size or shape of the CNN or any input matrix or weight matrix. Furthermore, the applicants have performed a design space exploration varying the kernel size and the ifmap size, and the TrIM architecture has shown a limited number of memory accesses, as well as noticeable computing efficiency given that PEs work at their peak throughput. This, in turn is advantageous in terms of energy consumption, cost and latency.

[0021] For example, when the proposed computer architecture is made to process 256x256 ifmaps, it requires a number of memory accesses that practically corresponds to the number of inputs (by experiencing only a 1 .5% overhead with 3x3 kernels). In addition, every PE performs 2 operations-per-cycle fully utilizing multipliers and adders.

[0022] The plurality of inputs may comprise a first input received from a first direction; a second input received from a second direction, orthogonal to the first direction; and a third input received from a processing element, or one of the at least one storage registers, disposed diagonally with respect to a given processing element, when available. For example, a top row of processing elements may receive inputs diagonally from processing elements and storage registers in a row below but will not be required to pass any inputs upwardly. Similarly, a bottom row of processing elements and storage registers may transfer inputs diagonally to processing elements in a row above but will not be required to receive inputs from below.

[0023] The at least one storage register may be configured to temporarily store an input prior to providing the input diagonally to a processing element in an adjacent row of the systolic array (e.g. in a row above or below the row in which the storage register is provided).

[0024] The processing elements may be configured for one or more of: loading of weights in the first direction, optionally wherein the first direction is a vertical column direction; loading of partial sums in the first direction, optionally wherein the first direction is a vertical column direction; loading of the first input in the first direction, wherein the first direction is a vertical column direction; either loading of the second input from a right processing element to a left processing element or storage register in a same row or loading of the second input from a left processing element to a right processing element or storage register in a same row; and either loading of the third input from a processing element or storage register disposed in an adjacent row and left column with respect to a given processing element; or loading of the third input from a processing element or storage register disposed in an adjacent row and right column with respect to a given processing element, wherein the adjacent row is a row above or below the given processing element.

[0025] The at least one storage register may comprise K-1 storage registers, wherein K is an integer of 2 or more and the systolic array comprises K x K processing elements.

[0026] The at least one storage register may comprise a shift register buffer. This is advantageous because shift register buffers are more efficient when compared to other types of storage registers, for example, based on static random access memory (SRAM). However, in some cases, the type of storage register may not be limited and any suitable storage register may be used.

[0027] The at least one storage register may comprise a plurality of storage registers provided adjacent to each row of the systolic array, except for a first (e.g. uppermost) row.

[0028] The plurality of registers in each row may be configured to be loaded successively from a same row until each of the plurality of registers is loaded with an input, prior to transferring the respective inputs from each of the plurality of registers, diagonally to a respective one of the processing elements in an adjacent row.

[0029] The computer architecture may further comprise control circuitry configured for selecting one or more of the plurality of registers for use in each row depending on a size of an input matrix. The plurality of registers in each row may comprise a maximum number of registers required in relation to a maximum width of an input matrix and the plurality of registers may be split into a plurality of sub-buffers from which inputs are selected for diagonal transfer to respective ones of the processing elements in an adjacent row (e.g. a row above or below). Note, each processing element is configured to receive one diagonal input from an adjacent row. The diagonal input may be received from a processing element in an adjacent row or a storage register in an adjacent row. In some cases, all of the diagonal inputs may be received from storage registers in an adjacent row.

[0030] The processing elements may comprise at least one multiplexer configured for the selection of the plurality of inputs for use in the multiplication step.

[0031] The processing elements may be configured to store and reuse the at least one weight once loaded into a desired processing element. For example, in pre-processing steps, the weights may be sequentially loaded into the array and then stored for subsequent use in the respective processing elements.

[0032] According to a second aspect of the present disclosure, there is provided a processing element for use in a systolic array of processing elements in a convolutional neural network (CNN), the processing element being configured to: load at least one weight and a plurality of inputs; selectively multiply the at least one weight with each of the plurality of inputs to produce a product; accumulate the product with a partial sum input from another processing element, when available; and output a partial sum output resulting from the accumulation; wherein the processing element is configured for triangular movement of inputs between processing elements in the systolic array, which results in the plurality of inputs.

[0033] According to a third aspect of the present disclosure, there is provided a computer architecture for a convolutional neural network (CNN) comprising: a plurality of the computer architectures as described above, wherein each of said computer architectures is configured to process a specific input matrix and weight matrix; and an adder configured to accumulate the outputs from each of said computer architectures and to output a result of the accumulation.

[0034] According to a fourth aspect of the present disclosure, there is provided a computer architecture for a convolutional neural network (CNN) comprising: a plurality of the computer architectures of the third aspect; and a controller configured to control operation of the plurality of computer architectures of the third aspect and to output a plurality of output matrices.

[0035] The computer architecture of the fourth aspect may further comprise an adder and a partial sum buffer for each of the computer architectures of the third aspect, wherein the adder and the partial sum buffer are configured to accumulate the output matrices from the respective computer architectures of the third aspect for a volumetric convolution.

[0036] At least some of the above and other features of the invention are set out in the claims.

[0037] These and other aspects will be apparent from the embodiments described in the following. The scope of the present disclosure is not intended to be limited by this summary nor to implementations that necessarily solve any or all of the disadvantages noted.

[0038] Any features described in relation to one aspect of the disclosure may be applied to any one or more other aspect of the disclosure.

[0039] Brief Description of Certain Embodiments

[0040] Some embodiments of the disclosure will now be described by way of example only and with reference to the accompanying drawings, in which: Figure 1 shows a block diagram for a convolutional layer in which N three-dimensional filters are subjected to as many three-dimensional convolutions with M input feature maps to generate N output feature maps;

[0041] Figure 2 illustrates a two-dimensional convolution between a 5x5 input feature map and a 3x3 kernel to generate a 3x3 output feature map;

[0042] Figure 3 illustrates a generic block diagram of the computer architecture for a convolutional neural network (CNN), according to the disclosure;

[0043] Figure 4 illustrates a block diagram of a processing element according to the disclosure;

[0044] Figure 5 illustrates a block diagram showing the triangular movement of inputs between processing elements in accordance with the disclosure;

[0045] Figure 6 illustrates a specific block diagram of the computer architecture for a convolutional neural network (CNN), according to the disclosure;

[0046] Figure 7 illustrates the computations performed by each processing element in the architecture of Figure 6, when processing a 5x5 input matrix and a 3x3 kernel;

[0047] Figure 8 illustrates graphs showing the number of memory accesses for the present architecture;

[0048] Figure 9 illustrates a graph showing the throughput per processing element for the present architecture;

[0049] Figure 10 illustrates a graph showing the number of registers required for the present architecture;

[0050] Figure 11 illustrates a generic block diagram of the computer architecture for a slice for a convolutional neural network (CNN), including reconfigurable shift register buffers, according to the disclosure; Figure 12 illustrates a block diagram of a reconfigurable shift register buffer for use the in architecture of Figure 11 ;

[0051] Figure 13 illustrates a block diagram of the computer architecture for a core for a convolutional neural network (CNN), according to the disclosure; and

[0052] Figure 14 illustrates a block diagram of the computer architecture for an engine for a convolutional neural network (CNN), according to the disclosure.

[0053] Detailed Description of the Preferred Embodiments

[0054] Generally speaking, the disclosure proposes a triangular input movement (TrIM) systolic array (SA), with a unique dataflow and architecture configured to maximise the reuse of inputs and minimise the time and energy spent fetching inputs from memory.

[0055] In the following description, the computer architecture is generally described in the orientation shown in the Figures. However, it will be understood that the orientation of the elements is not limited to that shown and, in practice, for example, the computer architecture itself may be rotated or configured in a mirror image and / or data flowing through the computer architecture may be configured to flow in a different direction to that described.

[0056] Figure 3 illustrates a computer architecture 100 for a convolutional neural network (CNN) comprising a systolic array (SA) 102 of processing elements (PE). The processing elements PEs are configured to: load at least one weight (W) and a plurality of inputs (I); selectively multiply the at least one weight (W) with each of the plurality of inputs (I) to produce a product; accumulate the product with a partial sum (psum) input from another processing element (PE), when available; and output a partial sum (psum) output resulting from the accumulation. The processing elements (PE) are configured for triangular movement of inputs (I) between processing elements (PE) in the systolic array (SA) 102, which results in the plurality of inputs (I). The computer architecture 100 also comprises at least one storage register in the form of a shift register buffer (SRB) configured to temporarily store an input (I) during said triangular movement of inputs (I); and an adder 104 configured to accumulate the partial sum (psum) outputs from the systolic array (SA) and to output a result of the accumulation (output). As shown in Figure 3, the SA 102 comprises K * K processing elements (PEs), where K is an integer of 2 or more. The SA 102 is therefore formed of PEs arranged in rows (i) denoted 0 to K-1 and columns (j) denoted 0 to K-1. However, it should be noted that a square SA 102 may not be necessary and in some cases, the SA 102 may have more rows than columns or more columns than rows.

[0057] During convolution, the weights (W) are retained (i.e. stationary) at the PE level, while inputs (I) and partial sums (psums) are moved across the SA 102 to maximize data reuse. However, before any computation, the weights (Wext) are fetched from a memory (not shown) and loaded into the SA 102 via the PEs along a first (e.g. top) row of the SA 102, as illustrated. The weights (W) are then progressively moved from the first row to each subsequent row until all weights (W) from a given kernel are loaded into the SA 102.

[0058] The movement of the inputs (I) can be summarized in three steps: (i) first, inputs (lext) are fetched from the memory and vertically loaded into the SA 102 via the PEs; (ii) then, the inputs (I) in each PE (apart from the first PEo.o) are moved horizontally, for example, from right (IR) to left (II), until they reach the leftmost PEs (i.e. in the first column j=0); (iii) finally, they move diagonally (ID) to the PEs in an adjacent row (e.g. a row above).

[0059] According to a width (Wi) of the input matrix being processed, the leftmost PEs in rows 1 to K-1 are connected to Shift Register Buffers (SRBo to SRBK-2) with depths Wi - K - 1 , to accommodate the inputs (I) moved horizontally in step (ii) to ensure the correct execution of the diagonal movement over the time. More specifically, with each cycle of the SA 102, the inputs (I) are sequentially moved horizontally to fill the Shift Register Buffers (SRBo to SRBK-2) before being transferred from each individual register of the Shift Register Buffers (SRBo to SRBK-2) to respective ones of the PEs in an adjacent row (e.g. a row above). Thus, the input (I) in the leftmost register of the Shift Register Buffer (e.g. SRBo) will be transferred to the leftmost PE in a row above (e.g. PEo.o) and the input (I) in the next leftmost register of the Shift Register Buffer (e.g. SRBo) will be transferred to the next leftmost PE in a row above (e.g. PEo.i) etc. As such, steps (i)-(iii) form a data flow having a right-triangular shape, hence the name Triangular Input Movement (TrIM). Figure 3 schematizes a generic architecture 100 accommodating the proposed dataflow. Each PE is labelled as PEij, with 0 < i,j < K, and, as illustrated in Figure 4, consists of a MAC unit, comprising a multiplier 400 and an accumulator 402; four registers 404 to store, respectively, a weight (Wext), a vertical input (lext), an input to be transferred left (II) and a partial sum (psum) output from the MAC unit; and two multiplexers 406 to establish which of the plurality of inputs is to be used by the MAC unit. A first multiplexer 406 is arranged for selection between the vertical input (lext) and the input (ID) transferred diagonally, the output from the first multiplexer 406 is an input to the second multiplexer along with an input received from a PE to the right (IR). The output from the second multiplexer 406 is fed into the multiplier 400 along with the weight (Wext). The output from the multiplier 400 is fed into the accumulator 402 along with a partial sum (psum) received from a PE above (when available). The output from the accumulator 402 is the partial sum (psum) output fed into the PE below. In addition, the output from the second multiplexer 406 may be stored in the register 404 as the input to be transferred left (II).

[0060] Thus, as illustrated in Figure 5, the inputs are loaded vertically (e.g. from the main memory to PEj+i.o as illustrated) in a first step; transferred horizontally, e.g. from right to left (for example, as illustrated, from PEj+i.o to SRBi) in a second step; and then forwarded diagonally from a leftmost register in SRBi to a PE in an adjacent row (e.g. PEi.o) in a third step. In general, the inputs may first be provided to the SRBs from each PEi.o, except for the uppermost PE with i=0, then shifted along the registers in the SRBs, and finally the inputs stored into the leftmost K registers are supplied diagonally to the PEij elements in the row above.

[0061] However, for small input matrices, the number of registers in the SRBs may be lower than K (but at least 1 ), and thus diagonal connections for diagonal transfer of inputs (ID) may be provided from some of the PEs in one row (e.g. below) to PEs in an adjacent row (e.g. above). Generally, this is true when Wi < 2K. Otherwise, the general case described above is valid.

[0062] Psums are accumulated vertically, thus from each PEij to each PEi+ij. Eventually, the adder tree 104 accumulates the psums coming from the PEK-IJ elements in the bottom row. Figure 6 illustrates a specific example for a 5 * 5 input matrix, with I = 1 ,2.25 being the inputs, and a 3 x 3 kernel, with W = A,B . I being the weights. The illustrated TrIM architecture 600 comprises a SA made of 3 rows (named Rowo, Rowi, and R0W2), each having 3 PEs. The leftmost PEs belonging to Rowi and R0W2 are connected respectively to SRBo and SRB1, each with depth 1 . A final adder tree 602, consisting of two adders, is provided to accumulate the psums coming from the PEs of R0W2 to provide a final output. In what follows, the detailed functionality of the dataflow is explained cycle-by-cycle (i.e. for each clock pulse corresponding to a movement of data within the SA). In a preliminary phase, K cycles are needed to load the KxK weights, arranged in rows of K weights per cycle. After this preliminary phase, the actual computations and input movements can begin as described below and as illustrated in Figure 7. According to Figure 7, for every cycle: (i) PEs with numbers in bold and underlined refer to inputs subjected to right to left movements; (ii) PEs with a black triangle refer to inputs subjected to diagonal movements; (iii) in the other PEs, inputs are fetched from the main memory.

[0063] • Cycle 1. Inputs I = 1 ,2,3 are supplied vertically to Rowo and are multiplied by the stored weights W = A,B,C.

[0064] • Cycle 2. Rowo multiplies inputs I = 2,3,4 with weights W = A,B,C, where I = 2,3 are reused through right-to-left movements, while I = 4 is supplied externally. Rowi multiplies the new inputs I = 6,7,8 with stored weights W = D,E,F, and accumulates these with the partial sums (psums) coming from Rowo.

[0065] • Cycle 3. Rowo multiplies I = 3,4,5 with W = A,B,C, where I = 3,4 are reused through right-to-left movements, while I = 5 is supplied externally. Rowi multiplies the inputs I = 7,8,9 with W = D,E,F, and accumulates these with the psums coming from Rowo. While I = 7,8 are reused through right-to-left movements, I = 9 is supplied externally. R0W2 multiplies the new inputs I = 11 ,12,13 with W = G,H,I, and accumulates these with the psums coming from Rowi. At this cycle, SRBo receives I = 6 to be reused later.

[0066] • Cycle 4. Rowo multiplies I = 6,7,8 with W = A,B,C, where I = 6 is reused diagonally from SRBo, while I = 7,8 are reused diagonally from PEi o and PEi.i, respectively. Rowi multiplies the inputs I = 8,9,10 with W = D,E,F, and accumulates these with the psums coming from Rowo. While I = 8,9 are reused through right-to-left movements, I = 10 is supplied externally. R0W2 multiplies the inputs I = 12,13,14 with W = G,H,I, and accumulates these with the psums coming from Rowi. While I = 12,13 are reused through right-to-left movements, I = 14 is supplied externally. At this cycle, SRBo and SRBi receive I = 7 and 1 = 11 , respectively. Finally, the psums coming from PE2,o, PE2.1, and PE2,2 are accumulated through the adder tree to get the very first output activation Oo,o.

[0067] • Cycle 5. Rowo multiplies I = 7,8,9 with W = A,B,C, where I = 7,8 are reused through right-to-left movements, while I = 9 is reused diagonally from PEi . Rowi multiplies the inputs I = 11 ,12,13 with W = D,E,F, and accumulates these with the psums coming from Rowo. 1 = 11 is reused diagonally from SRBo, while I = 12,13 are reused diagonally from PEi o and PEi , respectively. R0W2 multiplies the inputs I = 13,14,15 with W = G,H,I, and accumulates these with the psums coming from Rowi. While I = 13,14 are reused through right-to-left movements, I = 15 is supplied externally. At this cycle, SRBo and SRB1 receive I = 8 and I = 12, respectively. Finally, the psums coming from PE2,o, PE2.1, and PE2.2 are accumulated through the adder tree to get the second output activation Oo,i.

[0068] • Cycle 6. Rowo multiplies I = 8,9,10 with W = A,B,C, where I = 8,9 are reused through right-to-left movements, while I = 10 is supplied externally. Rowi multiplies the inputs I = 12,13,14 with W = D,E,F, and accumulates these with the psums coming from Rowo. While I = 12,13 are reused through right-to-left movements, I = 14 is reused diagonally from PE2,I. R0W2 multiplies the new inputs l = 16,17,18 with W = G,H,I, and accumulates these with the psums coming from Rowi. At this cycle, SRBo and SRB1 receive 1 = 11 and I = 13, respectively. Finally, the psums coming from PE21O, PE2,I, and PE2.2 are accumulated through the adder tree to get the third output activation Oo,2.

[0069] • Cycle 7. Rowo multiplies I = 11 ,12,13 with W = A,B,C, where 1 = 11 is reused diagonally from SRBo, while I = 12,13 are reused diagonally from PEi,o and PEi , respectively. Rowi multiplies the inputs I = 13,14,15 with W = D,E,F, and accumulates these with the psums coming from Rowo. While I = 13,14 are reused through right-to-left movements, I = 15 is supplied externally. R0W2 multiplies the inputs I = 17,18,19 with W = G,H,I, and accumulates these with the psums coming from Rowi. While I = 17,18 are reused through right-to-left movements, I = 19 is supplied externally. At this cycle, SRBo and SRB1 receive I = 12 and 1 = 16, respectively. Finally, the psums coming from PE2,o, PE2.1, and PE2.2 are accumulated through the adder tree to get the fourth output activation Oi,o.

[0070] • Cycle 8. Rowo multiplies I = 12,13,14 with W = A,B,C, where I = 12,13 are reused through right-to-left movements, while I = 14 is reused diagonally from PEi . Rowi multiplies the inputs I = 16,17,18 with W = D,E,F, and accumulates these with the psums coming from Rowo. I = 16 is reused diagonally from SRBi, while I = 17,18 are reused diagonally from PEi o and PEi,i, respectively. R0W2 multiplies the inputs I = 18,19,20 with W = G,H,I, and accumulates these with the psums coming from R0W1. While I = 18,19 are reused through right-to-left movements, I = 20 is supplied externally. At this cycle, SRBo and SRB1 receive 1 = 13 and 1 = 17, respectively. Finally, the psums coming from PE2,o, PE2.1, and PE2.2 are accumulated through the adder tree to get the fifth output activation Ou .

[0071] • Cycle 9. Rowo multiplies I = 13,14,15 with W = A,B,C, where I = 13,14 are reused through right-to-left movements, while I = 15 is supplied externally. R0W1 multiplies the inputs I = 17,18,19 with W = D,E,F, and accumulates these with the psums coming from Rowo. While I = 17,18 are reused through right-to-left movements, I = 19 is reused diagonally from PE2,I. R0W2 multiplies the new inputs I = 21 ,22,23 with W = G,H,I, and accumulates these with the psums coming from R0W1. At this cycle, SRBo and SRB1 receive I = 16 and I = 18, respectively. Finally, the psums coming from PE21O, PE2,I, and PE2,2 are accumulated through the adder tree to get the sixth output activation O12.

[0072] • Cycle 10. Rowo has completed its current computations, so it can work on new inputs. R0W1 multiplies the inputs I = 18,19,20 with W = D,E,F, and accumulates these with the psums coming from Rowo. While I = 18,19 are reused through right- to-left movements, I = 20 is supplied externally. R0W2 multiplies the inputs I = 22,23,24 with W = G,H,I, and accumulates these with the psums coming from R0W1. While I = 22,23 are reused through right-to-left movements, I = 24 is supplied externally. From this cycle, SRBo and SRB1 do not need to receive data anymore. Finally, the psums coming from PE2,o, PE2,-I, and PE2.2 are accumulated through the adder tree to get the seventh output activation 02,0.

[0073] • Cycle 11. Rowo and R0W1 have completed their current computations, so they can work on new inputs. R0W2 multiplies the inputs I = 23,24,25 with W = G,H,I, and accumulates these with the psums coming from R0W1. While I = 23,24 are reused through right-to-left movements, I = 25 is supplied externally. Finally, the psums coming from

[0074] PE21O, PE2,I, and PE2.2 are accumulated through the adder tree to get the eighth output activation 02,1. • Cycle 12. All the rows have completed their current computations, so they can work on new inputs. Finally, the psums coming from PE2,o, PE2.1, and PE2.2 are accumulated through the adder tree to get the final ninth output activation 02,2.

[0075] It will be understood that the operation of the architecture 600 is controlled by a controller (not shown).

[0076] The generic architecture of Figure 3 was characterized using different kernel sizes and ifmap sizes considering the number of memory accesses, throughput and number of registers required. The kernel size was varied using K = 3,5,7, while ifmaps were considered as square input matrices, with Hi = Wi = I, where I = 16,32,64,128,256.

[0077] Figure 8 depicts the number of memory accesses required by TrIM for the kernel sizes and the ifmap sizes defined above, and considering: (a) the total number of memory accesses for each case; and (b) the percentage overhead that relates to the number of extra memory accesses required in order to retrieve the inputs more than once. The total number of memory accesses is given by equation 2:

[0078] From Figure 8(a), the larger the ifmap size, the higher the number of memory accesses because of the higher number of inputs to be read from the main memory. However, it can be seen that at a fixed ifmap size, the influence of larger kernel sizes (K) is limited. For example, when a 256x256 ifmap is considered, the number of memory accesses when K=7 is only 11 .9% higher than the case K=3.

[0079] Figure 8(b) highlights the significant advantage introduced by TrIM to maximize data reuse locally. In fact, for every I and K, the overhead in terms of additional memory accesses is a limited fraction of the entire contribution. The larger the ifmap size, the smaller the overhead. For example, when 256x256 ifmaps are considered, the overhead is only 1.5% with K=3. This means that inputs are mostly only read once and extensively reused locally, thus positively impacting the energy efficiency of the TrIM architecture. Figure 9 illustrates the throughput exhibited by the TrIM architecture for the kernel sizes and the ifmap sizes defined above. The throughput is given by the number of operations that each PE actually performs per clock cycle and as reported in equation 3:

[0080] Figure 9 spotlights a relevant message: the throughput, other than being practically independent by the ifmap size and kernel size, is equal to the theoretical peak throughput achievable by each PE. In fact, the peak throughput is 2 operations per cycle (OPs / cycle), considering that each PE can perform one multiplication and one addition every clock cycle. Therefore, the TrIM architecture shows significant computing efficiency.

[0081] Figure 10 illustrates the number of registers required by the TrIM architecture. These include the registers in each PE, the registers of the SRBs, as well as the register included into the adder tree 104 in Figure 3. Equation 4 summarizes the number of registers:

[0082] Registers = 4 x K2+ (K - 1 ) x (Wi - K - 1) + 1 (4)

[0083] From Figure 10, the higher the kernel size and the ifmap size, the higher the number of registers. For example, when K=3, a TrIM architecture dealing with 256x256 ifmaps requires 8.9 times more registers than the case of 16x16 ifmaps. The higher contribution is mainly dictated by the SRBs, considering that the number of PEs is the same in both cases. The higher number of registers with larger ifmap sizes is welcomed to allow TrIM to maximize data reuse, thus alleviating the significant energydraining contribution required by memory accesses. In fact, as previously stated, with 256x256 ifmaps the inputs are practically only read once from the memory (just 1.5% overhead) and extensively reused locally. Furthermore, considering that SRBs consist of shift registers, their energy consumption is many times lower than the energy required to satisfy the memory accesses.

[0084] A hardware architecture based on TrIM, which deals with multi-dimensional CLs for

[0085] CNNs is also considered. This architecture consists of three levels: • The TrIM Slice (TS). This module is responsible for performing 2-D K * K convolutions. At this stage, weights are kept stationary in each Processing Element (PE), while inputs are reused between PEs by exploiting the innovative triangular movement described above. Specifically, each TS processes an independent ifmap and a specific kernel of one of N three-dimensional filters.

[0086] • The TrIM Core (TC). This higher-level module accommodates multiple TSs, each managing a specific ifmap and a group of PM kernels (with PM being the number of kernels processed in parallel) belonging to one of the three-dimensional filters. Additional adders allow the provisional outputs from each slice to be accumulated and generate an ofmap.

[0087] • The TrIM Engine (TE). This top-level module consists of PN multiple TCs (with PN being the number of filters processed in parallel), each dealing with an independent three-dimensional filter. All the TCs use the same set of ifmaps.

[0088] Each level is described in detail in the following sections.

[0089] The TrIM Slice

[0090] Figure 11 illustrates a generic block diagram of the computer architecture for a TrIM Slice (TS) for a convolutional neural network (CNN), including reconfigurable shift register buffers, according to the disclosure. The TS includes a SA of K*K PEs, an adder tree and K-1 Reconfigurable Shift Register Buffers (RSRBs). The SA is similar to that described above and shown in Figure 3, with K rows (Rowi, with 0 < i < K), each having K PEs. In a preliminary process, weights (Wext) are fetched from the memory and provided to the PEs of Rowo as groups of K elements per cycle, thus shifted from the top row down to the bottom row. This process lasts K cycles to ensure that the current kernel is completely stored at the PE level.

[0091] As described previously, the inputs are then loaded initially from memory (lext) in a vertical direction, moved within each row from right (IR) to left (II), forwarded to the closest RSRB and dispatched via a bus connected to each RSRB to transfer up to K inputs diagonally (ID) to the ROW to complete the triangular movement. Psums are first accumulated vertically and then through an adder tree that finalizes the K x K convolution.

[0092] The generic PE is as described above and shown in Figure 4. However, slight modifications may be made to the PE placed at the boundaries of the array. Those belonging to Rowo does not receive any psum from previous rows, thus the current products are accumulated with zeros. In the same row, the leftmost PE is missing the II connection since no RSRB is interfaced with the first PE. Regarding the bottom row, no diagonal connections (i.e., ID) are provided to the PEs, since no RSRBs are placed below. Finally, for each row, the rightmost PEs does not receive any IR, thus missing those connections.

[0093] The main difference between the TS of Figure 11 and the architecture of Figure 3 is that each SRB is reconfigurable and therefore adaptable for different ifmap sizes, thus ensuring that a single hardware instance is capable of processing any CL belonging to the target CNN.

[0094] Each RSRB consists of WIM registers, with WIM being the width of the largest ifmaps. To manage other ifmaps’ sizes, where Wi < WIM, the RSRB is split into multiple SubBuffers (SBs), each consisting of L registers, as shown in Figure 12. Generally, L>K, with K being the kernel size, although for small ifmaps L may be less than K (i.e. where some of the diagonal movement is from PEs to PEs and not all from RSRBs to PEs). While all of the SBs serially forward data from right to left and eventually into the next SB, selected SBs (which may include some or all of the SBs) are configured to provide the data stored in the last K registers to a multiplexer 120. The multiplexer 120 selects a group of K values from one of the SBs to feed into respective PEs in Rowi-i, according to the selector’s value, thus finalizing the triangular movement. It is worth underlining that the parameter L and the number of SBs feeding the multiplexer 120 are established at design time, according to the characteristics of the CNN in question.

[0095] Returning to Figure 11 , the Adder Tree is supplied by the K psums coming from the bottom row of the array for the final accumulation. The Adder Tree consists of [log2(K)] stages, with the last stage followed by an output register.

[0096] In the current TS implementation, the computational units support B-bit unsigned integer inputs and B-bit signed integer weights. As a result, signed integer psums are provided. In particular, the psums coming from the bottom row of the array have a bit width equal to 2*B+K. Then, after the adder tree, the current output is a signed integer consisting of 2xB+K+[log2(K)] bits. However, other implementations are contemplated and the disclosure is not limited by this example.

[0097] The TrIM Core

[0098] A TrIM Core (TC) architecture may deliver three-dimensional convolutions by accommodating PM TSS that operate in parallel, followed by an adder tree 130 that further accumulates the outputs from the PM TSS, as depicted in Figure 13. In other words, with reference to a generic convolutional layer, as depicted in Figure 1 , the TSs are fed by PM ifmaps (among the M available), as well as by PM kernels from one of the N filters. Taking into account that each TS operates on different ifmaps and kernels, the parameter PM is constrained by the input / output bandwidth that the architecture can sustain. The subsequent adder tree 130 produces a 2 x B + K +[log2(K)]+[log2(PM)]-bit result (TCoutput), which may be stored in a TC output register 132. To meet timing requirements, the internal stages of the adder tree 130 may be interleaved with registers to facilitate the correct dataflow.

[0099] The TrIM Engine

[0100] A TrIM Engine (TE) architecture may constitute a top level of the hierarchy, where multiple TCs work in parallel and without any data dependency, as shown in Figure 14. Specifically, PN TCS are responsible for generating as many output feature maps (ofmaps), among the N ofmaps that the convolutional layer should furnish for further processing according to Figure 1.

[0101] Considering that PN and PM are the parallelism parameters of the TrIM Engine (TE) and TrIM Core (TC), respectively, up to [N / PN]X[M / PM] computational steps are required to finalize the computations of the target convolutional layer. In the generic case, each TC executes a three-dimensional convolution between one filter of PM K X K kernels and PM HI x Wi ifmaps. As a result, the TC0UtPut generated by the TC is not final, but needs to be accumulated over other [M / PM]- 1 iterations. To achieve this, the TE associates an extra adder 140 and a Psums Buffer 142 to each TC. The Psums Buffer 142 may be conceived as a static random access memory (SRAM), where the necessary capacity is dictated by HoMxWoM rows and 2xB +K +[log2(K)]+[log2(M)] columns, where HOM and WOM represent the spatial sizes of the largest ofmaps generated by the target CNN. For each convolutional layer, the actual portion of SRAM may comprise from the address 0 to the address HoxWo - 1 , where Ho < HOM and Wo < WOM are the spatial sizes of the current ofmaps to be generated. Meanwhile, the remaining PN -1 TCs process different three-dimensional filters, but using the same set of PM ifmaps. Notably, the extra adder 140 and Psums Buffer 142 may only be required when PN < N.

[0102] In order to automatize the entire process over the time, a Control Logic 150 supervises the TE. In particular, the Control Logic 150 (a) provides run-time reconfigurability to targeted modules (i.e., RSRBs); (b) manages clock enables and addressing logic of registers and Psums Buffers; (c) schedules the inputs and weights to each TS over the different cycles, and manages the multiplexers inside each PE to enable the correct triangular input movement.

[0103] The proposed TrIM Engine consists of multiple TrIM Cores, each accommodating several TrIM Slices that directly deal with on-chip input reuse. In order to guarantee that the SA is not affected by the different ifmap sizes, the shift-register buffers that finalize the diagonal movement of inputs can be made reconfigurable at run-time.

[0104] To demonstrate the capabilities of the architecture, the TrIM architecture was implemented onto a Field Programmable Gate Array (FPGA) to accelerate a VGG-16 CNN at a 150 MHz clock frequency. The TE consists of 7 TCs, each having 24 TSs. The proposed architecture reached a peak throughput of 453.6 Giga Operations per Second (GOPs / s), whilst dissipating ~4.2 W.

[0105] The disclosure introduces TrIM, an innovative dataflow for SAs that is compatible with CNN computations. While each PE is required to preliminary store the weights, inputs are reused through a novel triangular movement: this data is initially fetched from the main memory and delivered to the PEs; subsequently, the inputs are moved from right to left; eventually, they are propagated through diagonal connections to other PEs. A complete architecture based on the TrIM dataflow is also presented, and organized in a hierarchical way to cope with convolutional layers in CNNs.

[0106] Embodiments of the present disclosure can be employed in many different applications including, for example, image recognition and natural language processing. The memory referred to herein can be based on conventional CMOS SRAMs, Flash memories, dynamic random access memories (DRAMs), resistive-based memories (like memristor-based memory arrays) or any other type of memories, including those using emerging technologies (e.g. RRAM, spin transfer torque (STT), spin orbit torque (SOT), spin orbit torque field effect transistor (SOTFET), ferroelectric field effect transistor (FeFET), .. etc.).

[0107] The skilled person will understand that in the preceding description and appended claims, positional terms such as ‘above’, ‘along’, ‘side’, etc. are made with reference to conceptual illustrations, such as those shown in the appended drawings. These terms are used for ease of reference but are not intended to be of limiting nature. These terms are therefore to be understood as referring to an object when in an orientation as shown in the accompanying drawings.

[0108] Although the disclosure has been described in terms of preferred embodiments as set forth above, it should be understood that these embodiments are illustrative only and that the claims are not limited to those embodiments. Those skilled in the art will be able to make modifications and alternatives in view of the disclosure which are contemplated as falling within the scope of the appended claims. Each feature disclosed or illustrated in the present specification may be incorporated in any embodiments, whether alone or in any appropriate combination with any other feature disclosed or illustrated herein.

[0109] References

[0110] [1] V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer, “Efficient processing of deep neural networks: A tutorial and survey,” Proc. IEEE, vol. 105, no. 12, pp. 2295- 2329, 2017.

[0111] [2] Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A survey of convolutional neural networks: Analysis, applications, and prospects,” IEEE Trans. Neural Netw. Learn. Syst., vol. 33, no. 12, pp. 6999-7019, 2022.

[0112] [3] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Commun. ACM, vol. 60, no. 6, p. 84-90, may 2017. [Online], Available: https: / / doi.Org / 10.1145 / 3065386 [4] S. Minaee, Y. Boykov, F. Porikli, A. Plaza, N. Kehtarnavaz, and D. Terzopoulos, “Image segmentation using deep learning: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 7, pp. 3523-3542, 2022.

[0113] [5] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), June 2016.

[0114] [6] A. B. Nassif, I. Shahin, I. Attili, M. Azzeh, and K. Shaalan, “Speech recognition using deep neural networks: A systematic review,” IEEE Access, vol. 7, pp. 19143— 19165, 2019. p] X. Zou, S. Xu, X. Chen, L. Yan, and Y. Han, “Breaking the von neumann bottleneck: architecture-level processing-in-memory technology,” Sci. China Inf. Sci., vol. 64, no. 6, p. 160404:1-160404:10, 2021.

[0115] [8] Y.-H. Chen, J. Emer, and V. Sze, “Eyeriss: a spatial architecture for energy-efficient dataflow for convolutional neural networks,” in Proc. 43rd Int. Symp. Comput. Arch. (ISCA), 2016, p. 367-379.

[0116] [9] Y.-J. Lin and T. S. Chang, “Data and hardware efficient design for convolutional neural network,” IEEE Trans. Circuits Syst. I, vol. 65, no. 5, pp. 1642-1651 , 2018.

[0117]

[0010] R. Xu, S. Ma, Y. Guo, and D. Li, “A survey of design and optimization for systolic array-based dnn accelerators,” ACM Comput. Surv., vol. 56, no. 1 , aug 2023. [Online], Available: https: / / doi.org / 10.1145 / 3604802

[0118]

[0011] H.-T. Kung, “Why systolic architectures?” Computer, vol. 15, no. 1 , pp. 37-46, 1982.

[0119]

[0012] X. Wei et aL, “Automated systolic array architecture synthesis for high throughput cnn inference on fpgas,” in Proc. 54th Annu. Design Autom. Conf. (DAC) 2017, New York, NY, USA, 2017.

[0120]

[0013] A. Samajdar, J. M. Joseph, Y. Zhu, P. Whatmough, M. Mattina, and T. Krishna, “A systematic methodology for characterizing scalability of dnn accelerators using scale-sim,” in 2020 IEEE Int. Symp. Perform. Anal. Syst. Softw. (ISPASS), 2020, pp. 58-68.

[0121]

[0014] J. Fornt et aL, “An energy-efficient gemm-based convolution accelerator with on- the-fly im2col,” IEEE Trans. VLSI Syst., vol. 31 , no. 11 , pp. 1874-1878, 2023.

[0122] [is] S. Chetlur et aL, “cudnn: Efficient primitives for deep learning,” arXiv:1410.0759, 2014.

[0016] N. P. Jouppi et aL, “In-datacenter performance analysis of a tensor processing unit,” in Proc. 44th Int. Symp. Comput. Arch. (ISCA), New York, NY, USA, 2017, p. 1-12.

[0123]

[0017] Y.-H. Chen, T. Krishna, J. S. Emer, and V. Sze, “Eyeriss: An energyefficient reconfigurable accelerator for deep convolutional neural networks,” IEEE J. Solid- State Circuits, vol. 52, no. 1 , pp. 127-138, 2017.

[0124] [is] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521 , no. 7553, pp. 436-444, 2015.

[0125]

[0019] S. S. Basha, S. R. Dubey, V. Pulabaigari, and S. Mukherjee, “Impact of fully connected layers on performance of convolutional neural networks for image classification,” Neurocomputing, vol. 378, pp. 112-119, 2020.

[0126]

[0020] K. Hara, D. Saito, and H. Shouno, “Analysis of function of rectified linear unit used in deep learning,” in 2015 Int. Joint Conf. Neural Netw. (IJCNN), 2015, pp. 1-8.

[0127]

[0021] Z. Ma et aL, “Fine-grained vehicle classification with channel max pooling modified cnns,” IEEE Trans. Veh. TechnoL, vol. 68, no. 4, pp. 3224-3233, 2019.

[0128]

[0022] S. Targ, D. Almeida, and K. Lyman, “Resnet in resnet: Generalizing residual architectures,” arXiv: 1603.08029, 2016.

[0129]

[0023] C. Szegedy et aL, “Going deeper with convolutions,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2015, pp. 1-9.

[0130]

[0024] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv:1409.1556, 2014.

[0131]

[0025] F. Spagnolo, P. Corsonello, F. Frustaci, and S. Perri, “Design of a lowpower superresolution architecture for virtual reality wearable devices,” IEEE Sensors J., vol. 23, no. 8, pp. 9009-9016, 2023.

[0132]

[0026] H. Emami, M. M. Aliabadi, M. Dong, and R. B. Chinnam, “Spagan: Spatial attention gan for image-to-image translation,” IEEE Trans. Multimedia, vol. 23, pp. 391-401 , 2021.

[0133]

[0027] S. Kriman et aL, “Quartznet: Deep automatic speech recognition with 1d timechannel separable convolutions,” in 2020 IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), 2020, pp. 6124-6128.

[0134]

[0028] Mustaqeem, M. Sajjad, and S. Kwon, “Clustering-based speech emotion recognition by incorporating learned features and deep bilstm,” IEEE Access, vol. 8, pp. 79861-79875, 2020.

[0135]

[0029] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv:1409.1556, 2014.

[0030] M. Horowitz, “1.1 computing’s energy problem (and what we can do about it),” in 2014 IEEE Int. Solid-State Circuits Conf. Dig. Tech. Papers (ISSCC), 2014, pp. IQ- 14.

[0136]

[0031] A. C. Yuz" ug" uler, C. S" onmez, M. Drumond, Y. Oh, B. Falsafi, and" P. Frossard, “Scale-out systolic arrays,” ACM Trans. Archit. Code Optim., vol. 20, no. 2, Mar 2023.

[0137]

[0032] B. Wang, S. Ma, G. Zhu, X. Yi, and R. Xu, “A novel systolic array processor with dynamic dataflows,” Integration, vol. 85, pp. 42-47, 2022.

[0138]

[0033] R. Xu, S. Ma, Y. Wang, and Y. Guo, “Cmsa: Configurable multidirectional systolic array for convolutional neural networks,” in 2020 IEEE 38th Int. Conf. Comput. Design (ICCD), 2020, pp. 494-497.

[0139]

[0034] R. Xu, S. Ma, Y. Wang, Y. Guo, D. Li, and Y. Qiao, “Heterogeneous systolic array architecture for compact cnns hardware accelerators,” IEEE Trans. Parallel Distrib. Syst., vol. 33, no. 11 , pp. 2860-2871 , 2022.

[0140]

[0035] C. Xin et aL, “Cosy: An energy-efficient hardware architecture for deep convolutional neural networks based on systolic array,” in 2017 IEEE 23rd Int. Conf. Parallel Distrib. Syst. (ICPADS), 2017, pp. 180-189.

[0141]

[0036] S. Ortega-Cisneros, “Design and implementation of an noc-based convolution architecture with gemm and systolic arrays,” IEEE Embedded Syst. Lett., vol. 16, no. 1 , pp. 49-52, 2024.

[0142]

[0037] B. Wu, T. Yu, K. Chen, and W. Liu, “Edge-side fine-grained sparse cnn accelerator with efficient dynamic pruning scheme,” IEEE Trans. Circuits Syst. I, vol. 71 , no. 3, pp. 1285-1298, 2024.

[0143]

[0038] S. K. Nandigama, H. C. Prashanth, and M. Rao, “Design and evaluation of inexact computation based systolic array for convolution,” in 2023 IEEE 14th Latin America Symp. Circ. Syst. (LASCAS), 2023, pp. 1-4.

[0144]

[0039] G. Zhang, R. Zhang, R. Wang, and S. Zhu, “A systolic array-based scheduling strategy for sparse cnn accelerators,” IEEE Trans. Circuits Syst. II, vol. 71 , no. 3, pp. 1436-1440, 2024.

[0145]

[0040] W. Sun, D. Liu, Z. Zou, W. Sun, S. Chen, and Y. Kang, “Sense: Modelhardware codesign for accelerating sparse cnns on systolic arrays,” IEEE Trans. VLSI Syst., vol. 31 , no. 4, pp. 470-483, 2023.

[0146]

[0041] W. Liu, J. Lin, and Z. Wang, “Usca: A unified systolic convolution array architecture for accelerating sparse neural network,” in 2019 IEEE Int. Symp. Circ. Syst. (ISCAS), 2019, pp. 1-5.

[0042] S. Chetlur et aL, “cudnn: Efficient primitives for deep learning,” arXiv:1410.0759, 2014.

Claims

CLAIMS:1 . A computer architecture for a convolutional neural network (CNN) comprising: a systolic array of processing elements, the processing elements being configured to: load at least one weight and a plurality of inputs; selectively multiply the at least one weight with each of the plurality of inputs to produce a product; accumulate the product with a partial sum input from another processing element, when available; and output a partial sum output resulting from the accumulation; wherein the processing elements are configured for triangular movement of inputs between processing elements in the systolic array, which results in the plurality of inputs; at least one storage register configured to temporarily store an input during said triangular movement of inputs; and an adder configured to accumulate the partial sum outputs from the systolic array and to output a result of the accumulation.

2. The computer architecture of claim 1 wherein the plurality of inputs comprise a first input received from a first direction; a second input received from a second direction, orthogonal to the first direction; and a third input received from a processing element or one of the at least one storage registers disposed diagonally with respect to a given processing element, when available.

3. The computer architecture of claim 1 or 2 wherein the at least one storage register is configured to temporarily store an input prior to providing the input diagonally to a processing element in an adjacent row of the systolic array.

4. The computer architecture of claim 2 wherein the processing elements are configured for one or more of: loading of weights in the first direction, optionally wherein the first direction is a vertical column direction; loading of partial sums in the first direction, optionally wherein the first direction is a vertical column direction; loading of the first input in the first direction, wherein the first direction is a vertical column direction; either loading of the second input from a right processing element to a left processing element or storage register in a same row or loading of the second input from a left processing element to a right processing element or storage register in a same row; and either loading of the third input from a processing element or storage register disposed in an adjacent row and left column with respect to a given processing element; or loading of the third input from a processing element or storage register disposed in an adjacent row and right column with respect to a given processing element, wherein the adjacent row is a row above or below the given processing element.

5. The computer architecture of any preceding claim wherein the at least one storage register comprises K-1 storage registers, wherein K is an integer of 2 or more and the systolic array comprises K x K processing elements.

6. The computer architecture of any preceding claim wherein the at least one storage register comprises a shift register buffer.

7. The computer architecture of any preceding claim wherein the at least one storage register comprises a plurality of storage registers provided adjacent to each row of the systolic array, except for an uppermost row.

8. The computer architecture of claim 7 wherein the plurality of registers in each row are configured to be loaded successively from a same row until each of the plurality of registers is loaded with an input, prior to transferring the respectiveinputs from each of the plurality of registers, diagonally to a respective one of the processing elements in an adjacent row.

9. The computer architecture of claim 7 further comprising control circuitry configured for selecting one or more of the plurality of registers for use in each row depending on a size of an input matrix.

10. The computer architecture of claim 9 wherein the plurality of registers in each row comprises a maximum number of registers required in relation to a maximum width of an input matrix and the plurality of registers are split into a plurality of sub-buffers from which inputs are selected for diagonal transfer to respective ones of the processing elements in the row above.

11. The computer architecture of any preceding claim wherein the processing elements comprise at least one multiplexer configured for the selection of the plurality of inputs.

12. The computer architecture of any preceding claim wherein the processing elements are configured to store and reuse the at least one weight once loaded into a desired processing element.

13. A processing element for use in a systolic array of processing elements in a convolutional neural network (CNN), the processing element being configured to: load at least one weight and a plurality of inputs; selectively multiply the at least one weight with each of the plurality of inputs to produce a product; accumulate the product with a partial sum input from another processing element, when available; and output a partial sum output resulting from the accumulation;wherein the processing element is configured for triangular movement of inputs between processing elements in the systolic array, which results in the plurality of inputs.

14. A computer architecture for a convolutional neural network (CNN) comprising: a plurality of the computer architectures of any of claims 1 to 12, wherein each of said computer architectures is configured to process a specific input matrix and weight matrix; and an adder configured to accumulate the outputs from each of said computer architectures and to output a result of the accumulation.

15. A computer architecture for a convolutional neural network (CNN) comprising: a plurality of the computer architectures of claim 14; and a controller configured to control operation of the plurality of computer architectures of claim 14 and to output a plurality of output matrices.

16. The computer architecture of claim 15 further comprising an adder and a partial sum buffer for each of the computer architectures of claim 14, wherein the adder and the partial sum buffer are configured to accumulate the output matrices from the respective computer architectures of claim 14 for a volumetric convolution.

Images