Hardware architectures for efficient vector-matrix multiplication

Abstract

A method of processing data in a neural network using a processor comprises receiving, at a vector-matrix hardware multiplication unit of the processor, a first input vector of a stream of input vectors; computing a first output vector as a matrix multiplication of the first input vector with a weight matrix of the neural network read directly from a region of data memory directly coupled to the vector-matrix hardware multiplication unit; subsequent to computing the first output vector, receiving, at the vector-matrix hardware multiplication unit, a second input vector of the stream of input vectors; and computing a second output vector as a matrix multiplication of the second input vector with the weight matrix read directly from the region of data memory, the weight matrix persisting in the region of data memory between computing the first output vector and computing the second output vector.

Description

HARDWARE ARCHITECTURES FOR EFFICIENT VECTOR-MATRIX MULTIPLICATIONTechnical field

[0001] The present disclosure pertains to hardware architectures for efficient vector-matrix multiplication.Background

[0002] In the field of artificial intelligence (Al), recent years and months have seen rapid advances in so-called large machine learning (ML) models, such as large language models (LLMs) and other foundational models. Such models typically take the form of artificial neural networks (neural networks or neural nets for short) having billions of parameters (weights) or more, requiring vast investments in computational resources. However, advances in the technology have relied on very similar hardware. Existing chips and highly developed tools and libraries are well-optimised for training large language models (LLMs), but they are unsuited to inference, which is the process of running live data (input tokens) through a specific model with learned parameters, to produce results (in LLMs, a series of output tokens). A token may take the form of a vector of numerical values.

[0003] As a consequence, Al models are very expensive to provision and run at scale. Issues such as time taken on conventional hardware to move model parameters from memory to processors mean that very expensive hardware is often used at a small fraction of its theoretical capability.

[0004] Moreover, Al performance is inhibited, as ever faster compute cannot make up for the performance lag caused in inference by moving model weights from memory to the processor units, limiting real-time performance and user experience.

[0005] Potential Al performance in the future is also restricted. Continual advancement of conventional computing is limited by the heat generated by these chips. There is a limit to how fast silicon chips can be cooled, and this has become a new constraint on continuing to scale conventional digital processors (the end of Dennard Scaling). With enough data, bigger Al models are predictably better, but without breakthroughs in compute systems, it may not be possible to continue to scale Al models to be orders of magnitude larger with sufficiently low latency (time per output token, for instance) to be usable.

[0006] Moreover, with Al model developers all building on similar infrastructure and the balance of its use tilting heavily to inference, without novel hardware the opportunity to create long-term differentiation and competitive advantage from faster, cheaper and higher quality token generation in inference will be limited.

[0007] For a better understanding of the present disclosure, and to show how embodiments of the same may be carried into effect, reference is made by way of example only to the following figures in which:Brief description of figures

[0008] Figure 1 illustrates exemplar input and output token vectors.

[0009] Figure 2 depicts a core vector-matrix multiplication concept.

[0010] Figures 3A to 3C show operation of a conventional systolic array over successive cycles.

[0011] Figure 4 visualises accumulation of partial sums within a systolic array.

[0012] Figure 5 highlights fixed-batch limitations of the conventional approach.

[0013] Figures 6A and 6B contrast batch processing with vector-by-vector softmax processing.

[0014] Figures 7A to 7G present an example streaming pipeline with interleaved vector operations.

[0015] Figure 8 compares batch-size requirements across deployment scenarios.Detailed Description

[0016] There are broadly two development paths available when building improved hardware for Al inference. The first is specialisation: honing in on very specific workloads and building chips that are uniquely suited to those specific requirements. Because model architectures evolve rapidly in the world of Al, while designing, verifying, fabricating and testing chips takes considerable time, companies pursuing this approach face the problem of shooting for a moving target whose exact direction is uncertain.

[0017] The second path is to change the way that computational operations themselves are performed, create different chips from novel building blocks, and build scalable systems on top of these.

[0018] A key mathematical operation underpinning foundational models is vector-matrix multiplication, typically involving multiplication of incoming vectors with large matrices of weights (typically containing hundreds, thousands or more weights in a given model layer). A novel hardware architecture for implementing this core operation is disclosed.

[0019] A first aspect of the present disclosure provides a processor, comprising: a data memory; and a vector-matrix hardware multiplication unit directly coupled to the data memory, and configured to: receive a stream of input vectors, and sequentially process the stream of input vectors, so as to compute an output for each input vector before processing the next input vector in the stream, wherein the output is computed as a matrix multiplication of the input vector with a matrix read directly from the data memory.

[0020] Note the terms “vector-matrix multiplication” and “matrix-vector multiplication” (MVM) are used interchangeably herein.

[0021] In embodiments, the processor may be an in-memory processor. In-memory processor design, also known as Processing-in-Memory (PIM), integrates processing capabilities directly within memory. This contrasts with a more conventional approach, in which data is repeatedly moved between a memory and a processor (typically being copied from the memory to one or more registers of the processors, and vice versa). In such embodiments, the vector-matrix hardware multiplication unit takes the form of hardware logic / circuitry (e.g. integrated circuitry) integrated within the data memory.

[0022] A second aspect of the present disclosure provides a die comprising: a data memory; and a vector-matrix hardware multiplication unit directly coupled to the data memory, and configured to: receive a stream of input vectors, and sequentially process the stream of input vectors, so as to compute an output for each input vector before processing the next input vector in the stream, wherein the output is computed as a matrix multiplication of the input vector with a matrix read directly from the data memory.

[0023] A third aspect provides a chip packaging the die of the second aspect. The chip may for example be used as an accelerator processor under the control of a central processing unit (CPU).

[0024] The matrix, in its entirety, is directly accessible to the vector-matrix hardware multiplication unit, and is read directly by the vector-matrix hardware multiplication unit from a region of the data memory in which it is held. The matrix persists in the same regionof memory as the stream of input vectors is processed, and is directly read from that same region each time a new vector of the stream of input vectors is processed.

[0025] In embodiments, the data memory is large enough to store matrices with hundreds, thousands or more components (values), all of which are directly accessible to the vectormatrix hardware multiplication unit.

[0026] In some embodiments, the data memory is co-located on the die with the vectormatrix hardware multiplication unit.

[0027] In certain “in-memory processing” embodiments, the vector-matrix hardware multiplication unit may take the form of hardware logic / circuitry (e.g. integrated circuitry) integrated within memory cells of the data memory itself. In such embodiments, the memory cells essentially operate as multipliers. In such embodiments, the above processor may be described as an in-memory processor.

[0028] This contrasts with a more conventional processor architecture with a hardware multiplication unit and a memory that is not directly accessible to the hardware multiplication unit. Instead, smaller registers would be directly accessible to the hardware multiplication unit, and the matrix (or parts of it) would need to be repeatedly copied to and purged from the registers as a program is executed.

[0029] In other embodiments of the second or third aspect, the data memory may take the form of a set of data registers closely coupled to the vector-matrix hardware multiplication unit (which itself may, for example, take the form of a Multiply- Accumulate (MAC) unit). For example, the data memory and the vector-matrix hardware multiplication unit may be implemented in a systolic array. In such embodiments, the stream of matrix-multiplication operations may be implemented as a “matrix-stationary” operation with the matrix retained in the closely coupled memory throughout. In contrast to conventional approaches, the matrix (in its entirety) is stored statically in the data registers from which it is accessed as the stream of input vectors is processed.

[0030] In accordance with the second aspect, the data memory where the matrix is held is located on the same die as the vector-matrix hardware multiplication unit. A die is a single piece of silicon (typically cut from a wafer) embodying an integrated circuit (comprising the data memory and the vector-matrix hardware multiplication unit), which can subsequently be packaged into a chip.

[0031] A natural unit of compute for frontier models is the vector. Matrices of model weights multiply incoming vectors of data. However, almost all current hardware architectures - and software frameworks - are based around the idea of matrix-matrix multiplications instead, where a fixed-size batch of input vectors has to be gathered up and processed in parallel in order to extract maximum performance from the hardware. All frontier models follow some form of token-based processing, in which every word, image patch, or video frame, for instance, becomes a vector in a sequence. For generative architectures, vectors are also output, often in a sequential, autoregressive manner (one vector at a time). Much of the software complexity in current LLM inference solutions is because of attempts to reconcile this requirement for ideally having very fine-grained compute granularity with the hardware architecture’s need to operate over fixed size batches and matrix-matrix multiplications. Often, a significant performance remains unrealised by the fundamental incompatibility of these two paradigms. Mature ecosystems have developed around prevalent hardware (such as GPUs) of patches, wrappers and third-party companies to try to overcome this.

[0032] By contrast, the architecture described herein natively operates on vectors at the hardware level, and so is highly efficient at the granularity of compute required for modem Al inference workloads.

[0033] Note, unless otherwise indicated, the term “vector” herein refers to a one-dimensional array of values and a matrix refers to a two-dimensional array of values (such as model weights). The output of a matrix multiplication of such a vector with such a matrix is a vector (output vector).

[0034] The architecture is particularly suited to models involving relatively large matrix multiplications (where the size of a matrix is large in relation to the output vector). A significant efficiency gain is achieved in holding the weight matrices at static memory locations directly coupled to their respective vector- matrix multiplication units, and moving the (smaller) output vectors between different processing units.

[0035] The architecture may be extended with interleaving of multiple such vector-matrix multiplication units with other hardware processing units, such as other vector mathematics operations, transport units (transport of output vector from one place to another, e.g. one region of the data memory to another region of the data memory, to enable processing by another vector-matrix multiplication unit or other hardware processing unit).

[0036] When running a model with multiple vector- matrix multiplication units, durations of the matrix-vector multiplications and other vector operations (implemented by other form(s) of vector hardware processing unit) can be matched such that a pipelined operation over multiple vectors does not have any overwhelming bottlenecks.

[0037] Embodiments provide a system that aggressively pipelines a flow of vectors across multiple units, e.g. multiple MVM units and other compute units (such as vector operation units).

[0038] The data memory may take the form of random access memory (RAM), in which case the matrix is read directly from RAM by the vector-matrix hardware multiplication unit directly coupled to the RAM, in which the matrix is held statically as the input stream is processed. In such embodiments of the second or third aspect, the RAM in which the matrix is statically held is integrated in the same die or chip as the vector-matrix hardware multiplication unit directly coupled thereto.

[0039] A fourth aspect provides a method of processing data in a neural network using a processor, the method comprising: receiving at a vector-matrix hardware multiplication unit of the processor a first input vector of a stream of input vectors; computing a first output vector as a matrix multiplication of the first input vector with a weight matrix of the neural network read directly from a region of data memory directly coupled to the vector-matrix hardware multiplication unit; subsequent to computing the first output vector, receiving at the vector-matrix hardware multiplication unit a second input vector of the stream of input vectors; and computing a second output vector as a matrix multiplication of the second input vector with the weight matrix read directly from the region of data memory, the weight matrix persisting in the region of data memory between computing the first output vector and computing the second output vector.

[0040] The processor may comprise multiple such vector-matrix hardware multiplication units and / or multiple processors comprising one or more such units, with different matrix multiplications distributed between multiple such units on the same processor (e.g. same die or chip ) / diff erent processors, for example on different dies or chips (e.g. different dies / chips).

[0041] The method may comprise transferring the first and / or second output vector to a different vector hardware processing unit on the same processor or another processor.

[0042] For example, the first and / or second output vector may be inputted to a second vectormatrix hardware multiplication unit as respective input vectors in a second stream of input vectors for matrix multiplication with a second weight matrix of the neural network held in a second region of data memory directly coupled to the second vector-matrix hardware multiplication unit.

[0043] As another example, the first and / or second output vectors may be processed in one or more other hardware vector processing units (e.g. to apply non-linear processing), resulting in a second stream of input vectors (e.g. to which non-linear processing has been applied) that is transferred back to a second vector-matrix hardware multiplication unit for multiplication with a second weight matrix of the neural network held in a second region of data memory directly coupled to the second vector-matrix hardware multiplication unit.

[0044] A fifth aspect herein provides a computer comprising a central processing unit (CPU) and an accelerator processor embodying the above-described hardware architecture, wherein the CPU is configured to execute a computer program held in program memory coupled to the CPU, which, when executed, causes the CPU to allocate to the accelerator processor a series of vector-matrix multiplication operations performed on a series of input vectors.

[0045] In embodiments, the CPU may be configured to allocate vector-matrix multiplication operations and other vector processing operations to be interleaved with the vector- matrix multiplication operations.

[0046] In embodiments, the CPU may be configured to cause the accelerator processor to perform the method of the fourth aspect.

[0047] Further embodiments will now be described by way of example only. Additional relevant context is also described.

[0048] An example matrix multiplication operation is depicted below. This involves multiplication of a weight matrix (w) with an input matrix (a), resulting in an output matrix (y)-

[0049] Operations of this nature are performance-critical in state of the art neural networks such as transformers. The input matrix may for example be an input to a neural network, or a matrix of activations generated in an earlier one or more earlier layers of the neural network.

[0050] Each column of the input matrix a can be regarded as an input vector. Each column of the output matrix y can be regarded as an output vector, computed as a matrix multiplication of the weight matrix w with the input vector in the corresponding column of the input matrix a.

[0051] The representation of the above operation as a multiplication between two 2D matrices reflects prevailing hardware implementations. In most accelerator hardware currently used, processing of inputs is batched in this manner.

[0052] However, it is observed that all weights of the weight matrix w are used to compute each single output vector (i.e. each column of y is computed from only the corresponding column of the input matrix a but using all of the weights w). Hence, while the weights are a ‘true’ 2D matrix, the inputs can be characterized as a batch of vectors, which could be processed separately.

[0053] FIG 1 schematically introduces notation and conventions adopted throughout the description and figures. On the right-hand side a batch of B input tokens is shown, each token being represented as a column vector of N features denoted an...ani, ai2...an2 and so on. The vectors are arranged side-by-side to form an activation tensor of dimension N x B. A vertical arrow labelled “Feature size, N” marks the length of every vector, while a horizontal doubleheaded arrow labelled “Batch size, B” identifies the number of concurrent tokens. Beneath the columns, the vectors are labelled “Input Token 1”, “Input Token 2”, “Input Token 3” and “Input Token 4” respectively.

[0054] On the left-hand side, corresponding output vectors yn.. ,y4i, yi2...y42, yi3...y43 and yi4...y44 are depicted in the same column- vector format, with legends “Output Token 1” to “Output Token 4”. Matching hatching patterns are applied to each logical input-output pair so that the visual relationship between a given activation vector and its resulting output vector is immediately apparent. Note, each output vector on the right depends on only one input vector on the left.

[0055] No processing hardware is illustrated in Figure 1; the drawing serves solely to establish dimensionality and naming conventions used in subsequent figures.

[0056] Figure 2 formally illustrates vector-matrix multiplication for a batch of four input tokens. In an upper panel the drawing shows: a column vector of N features an.. ,a4irepresenting a first input token; an N x N weight matrix wn.. . W44 rendered with diagonal cross-hatching; and a corresponding output vector yn.. ,y4i.

[0057] The identical hatching applied to the input and output vectors underlines their one-to- one relationship, while the common hatching of all weight cells signifies that every element of the weight matrix participates in the computation of every element of the output vector.

[0058] A lower panel extends the depiction across four successive clock cycles. Four mutually exclusive hatching styles are assigned to the remaining input vectors ai2...a42, ai3...a43 and ai4...a44 and to their respective output vectors yi2...y42, yi3...y43 and yu...y44. The weight matrix retains a dense diagonal shading, emphasising that each output vector depends on the full set of weights in every cycle.

[0059] A reason for forming input batches rather than processing inputs separately is data reuse. While the weights are a true 2D matrix, the inputs are really a batch of vectors, and could be processed separately.

[0060] As illustrated in Figure 2, the same NxN weights are used for each of B (batch size) matrix-vector products. Current accelerator processors such as Graphical Processing Units, Tensor Processing Units etc. have matrix-matrix multiply engines called systolic arrays that seek to implement weight matrix memory loads as efficiently as possible, sharing one load of weights across the whole set of corresponding input activation vectors (the B*N activation matrix). ‘Systolic arrays’ are one approach to matrix-matrix multiplication, and are how TPUs and GPU tensor cores enable weight re-use, as illustrated in the subsequent figures.

[0061] The shading convention adopted in Figure 2 conveys dependency information. In the upper panel the common cross-hatching shows that the complete weight matrix contributes to the calculation of the single active input-output pair. In the lower panel four distinct shading styles differentiate four independent input-output pairs, while the densely hatched weight matrix indicates that each pair continues to draw on every weight element.

[0062] FIG. 3A-C show a conventional systolic array, in which weights and activations / inputs are flowed into the array over multiple time cycles.

[0063] The weights and activations are multiplied together when they hit a processing element (PE), and that PE accumulates multiple elementwise products over multiple cycles. Activations flow right, weights flow down.

[0064] Figures 3A to 3C depict temporal operation of a conventional systolic array while it performs a 4- vector by 4 x 4 matrix multiplication. Figure 3A shows the first cycle in which the top row of the weight matrix has just been loaded into a first column of processing elements and a first input vector an.. ,a4i begins to enter the array. Figure 3B illustrates an intermediate cycle: the weights have shifted down one row, partial products such as a4i wi4 are accumulating inside the processing elements, and subsequent activation rows are progressing horizontally. Because these rows are simply B independent input vectors grouped into a batch for hardware convenience, the array must still wait N - 1 additional cycles before any complete result emerges, adding latency that is unrelated to the actual arithmetic. Figure 3C presents a later cycle where multiple partial sums have accumulated (for example a4i wi4 + asrwis) and the earliest output vector components yn-y4i are ready to be written out of the array.

[0065] Eventually, the values of the output matrix Y are accumulated in the PEs (note, the matrix index notation is transposed relative to earlier figures).

[0066] Figure 4 visualises the moment at which all partial sums have completed their journeys through the processing elements: every column now contains a fully computed output vector. The horizontal arrows indicate that those vectors are then read out row by row so that the array can be reused immediately for the next batch, while the weight matrix remains stationary. Dashed vertical bars inside the central cells mark the pipeline depth and highlight idle cycles that arise because the array cannot begin a new vector until the previous results are drained.

[0067] Systolic arrays have several drawbacks.

[0068] As illustrated in FIG. 5, the batch size is typically hard-coded.

[0069] This lack of flexibility is a problem, as LLMs and other frontier generative Al models have intrinsically varying “batch sizes” — varying input lengths, varying numbers of users, varying output lengths. To accommodate this variation with a fixed batch size, the batch size needs to be fixed in a way that involves some trade off between utilisation and performance.

[0070] Secondly, processing whole tensors, rather than individual vectors, creates a bottleneck, meaning the underlying accelerator hardware is underutilized in performance terms (to put it another way, processing whole tensors, rather than vectors, ‘leaves a lot of performance on the table’).

[0071] Figure 6A illustrates this behaviour with a hardware pipeline based on matrix-matrix units: only after the final vector of the batch has passed through the systolic array does the first output vector become available, meaning the downstream pipeline stages sit idle for B - 1 vector times.

[0072] In ML models, such matrix-multiplication operations (matmul ops) is typically followed by some expensive (typically non-linear) vector operations (vec ops), such as softmax, elementwise sigmoid etc., at which point the processing pipeline is reduced to a vector-by-vector paradigm.

[0073] Figure 6B highlights that, once the batch has been processed, the outputs must typically pass through an expensive non-linear function such as a softmax; this stage itself expects inputs one vector at a time. Consequently the system must buffer the entire batch between the matmul unit and the non-linearity, adding latency and memory traffic.

[0074] As a consequence, it becomes necessary to cache a whole batch of outputs pre- and post- activation. It also becomes necessary to wait for the whole (B*N)*(N*N) matrix-matrix multiply to complete on the systolic array before the vector operation starts running.

[0075] By contrast, running with in-memory compute (IMC), matrix-vector multiplies, allows matrix multiplication and nonlinear vector operations to be interleaved, meaning a materially shorter pipeline per vector, and significantly reduced latency.

[0076] FIGS. 7A-7E show one embodiment of a processing architecture based on streamed matrix- vector multiplication interleaved with nonlinear processing such as softmax. The aforementioned figures show a first MVM 702, which is a vector- matrix hardware multiplication unit, coupled to a vector processing unit 706, which is configured to apply a softmax in this example. The vector processing unit 706 is, in turn, shown coupled to a second MVM 712, which is also a vector-matrix hardware multiplication unit. The first MVM 702 is shown directly coupled to a first data memory 704 and the second MVM 712 is shown directly coupled to a second data memory 714.

[0077] The second MVM 712 may be on the same processor or a different processor than the first MVM 702. Likewise, the vector processing unit 706 may be on the same or different processor as either. Whether in the same processor or distributed across multiple processors, the aforementioned components (indicated by reference numerals 702, 706 and 712) areembodied in a computer with interconnect logic (not shown) configured to route the input and output vectors accordingly.

[0078] In contrast to a conventional systolic array, vectors are processed as streams, with vector- matrix multiplication operations interleaved with other vector operations, such as nonlinear processing (e.g. softmax) and additional MVM operations.

[0079] An example of such vector streaming processing is illustrated in FIGS. 7A-7G, over a series of time steps (defined by processor clock cycles). FIG. 7A shows a stream of input vectors formed of a first input vector 720A, a second input vector 720B, a third input vector 720C and a fourth input vector 720D, which are received at the first MVM 702 in that order.

[0080] Figures 7A-7E illustrate one embodiment vector-streaming pipeline over successive clock cycles. In Figure 7A, a first input vector 720A is supplied to a first MVM 702. The first MVM 702 reads, directly from a co-located data memory 704 to which it is directly coupled, a weight matrix resident in that memory and multiplies the matrix by the input vector so as to generate a first output vector 722A, which is forwarded immediately to a softmax unit 706. In the next cycles a second input vector 720B, a third input vector 720C and a fourth input vector 720D are processed in the same way, producing respective second, third and fourth output vectors 722B, 722C and 722D. Consequently, the three functional blocks — MVM 702, softmax 706 and a second MVM 712 downstream of the first MVM 702 and the softmax unit 706 — operate concurrently on different vectors of the same stream. When the pipeline is full (Figures 7D-7E) no module waits for a complete batch and each vector exits the pipeline after a constant number of cycles, independent of batch size B. For simplicity, a stream of four vectors is shown, but in practice further vectors (not shown in the drawing) can be streamed into the pipeline immediately after vector 720D so that every functional block remains continuously occupied and maximum throughput is maintained. The first MVM 720A and the second MVM 720B have corresponding architectures, with each operating on its respective inputs in the same way as the other.

[0081] The throughput benefit of the streaming approach can be quantified by the comparison captured in the table extracted from Figure 6. For a batch size B = 256, a conventional systolic array incurs O(B) cycles of latency for both the matrix multiplication and the subsequent non-linear operation, so the first answer is available only after roughly 2B cycles and the array is utilised at half of its potential. In contrast, the vector flow of FIGS. 7A-E delivers each vector result after a fixed 0(1) latency for both stages, effectivelydoubling steady-state throughput under the plausible assumption that the matmul and the combined softmax operations each require similar compute time.

[0082] FIG. 8 visualises batch size requirements of modern inference systems.

[0083] Figure 8 illustrates three deployment scenarios which further motivate the design: (i) edge devices that handle a single user at a time and therefore see a batch size of one; (ii) on- device generative models whose pre-fill stages can accept a large batch while the decode stage reverts to single-token operation; and (iii) multi-tenant datacentre inference services where the effective batch size fluctuates widely as differently sized user requests interleave. Because the stream-based architecture imposes no fixed batch size, it is able to sustain high utilisation across all three regimes without any software padding, artificial batching or wasted cycles. The vector streaming architecture described herein is particularly well suited to the latter, given its ability to handle variable batch sizes.

[0084] Previous accelerator hardware designs have all assumed that Al compute paradigms (both training and inference) could be expressed as fixed batch size inputs. This is often true for training, but rarely true in real- world inference.

[0085] According to a first aspect, there is provided a method of processing data in a neural network using a processor, the method comprising: receiving, at a vector-matrix hardware multiplication unit of the processor, a first input vector of a stream of input vectors; computing a first output vector as a matrix multiplication of the first input vector with a weight matrix of the neural network read directly from a region of data memory directly coupled to the vector-matrix hardware multiplication unit; subsequent to computing the first output vector, receiving, at the vector-matrix hardware multiplication unit, a second input vector of the stream of input vectors; and computing a second output vector as a matrix multiplication of the second input vector with the weight matrix read directly from the region of data memory, the weight matrix persisting in the region of data memory between computing the first output vector and computing the second output vector.

[0086] In embodiments, the method further comprises: while the vector-matrix hardware multiplication unit computes the second output vector from the second input vector in the stream, applying at a vector processing unit non-linear processing to the first output vector.

[0087] In embodiments, the method further comprises, while the vector processing unit applies the non-linear processing operation to the second output vector and the vector- matrixhardware multiplication unit processes a third input vector of the stream, computing, at a second vector-matrix hardware multiplication unit, a further output vector by multiplying the first output vector, after the non-linear processing operation, with a second weight matrix read directly from a second region of data memory to which the second vector-matrix hardware multiplication unit is directly coupled.

[0088] In embodiments, the second vector-matrix hardware multiplication unit is on the same or another processor.

[0089] According to a second aspect, there is provided a computer comprising: a data memory; a first vector-matrix hardware multiplication unit directly coupled to the data memory and configured to: receive a stream of input vectors; and sequentially process the stream of input vectors so as to compute, for each input vector before processing a next input vector in the stream, an output vector computed as a matrix multiplication of the input vector with a matrix read directly from the data memory; a hardware vector processing unit; and wherein the computer is configured to transfer the output vector from the first vector-matrix hardware multiplication unit to the hardware vector processing unit for non-linear processing whilst the stream of input vectors is still being processed at the first vector-matrix hardware multiplication unit.

[0090] In embodiments, the computer comprises: a second vector- matrix hardware multiplication unit, wherein the computer is configured to transfer the output vector, after non-linear processing, from the hardware vector processing unit to the second vector- matrix hardware multiplication unit whilst the stream of input vectors is still being processed at the first vector-matrix hardware multiplication unit.

[0091] In embodiments, a processor comprises: a data memory; and a vector-matrix hardware multiplication unit directly coupled to the data memory, the vector-matrix hardware multiplication unit configured to: receive a stream of input vectors, and sequentially process the stream of input vectors, so as to compute an output for each input vector before processing the next input vector in the stream, wherein the output is computed as a matrix multiplication of the input vector with a matrix read directly from the data memory.

[0092] In embodiments, the processor is an in-memory processor that integrates processing capabilities of the vector-matrix hardware multiplication unit directly within memory.

[0093] In embodiments, the vector-matrix hardware multiplication unit is in the form of hardware logic or hardware circuitry integrated within memory cells of the data memory.

[0094] In embodiments, the data memory is a random access memory (RAM), and the matrix is held statically as the stream of input vectors is processed.

[0095] In embodiments, the processor further comprises multiple vector-matrix hardware multiplication units, the processor being configured to distribute different matrix multiplications between multiple vector- matrix hardware multiplication units.

[0096] In embodiments, a die comprises: a data memory; and a vector-matrix hardware multiplication unit directly coupled to the data memory, and configured to: receive a stream of input vectors, and sequentially process the stream of input vectors, so as to compute an output for each input vector before processing the next input vector in the stream, wherein the output is computed as a matrix multiplication of the input vector with a matrix read directly from the data memory.

[0097] In embodiments, a chip packages the die.

[0098] In embodiments, a computer comprises a central processing unit (CPU) and an accelerator processor, wherein the CPU is configured to execute a computer program held in a data memory coupled to the CPU, which, when executed, causes the CPU to allocate to the accelerator processor a series of vector- matrix multiplication operations performed on a series of input vectors.

[0099] In embodiments, the CPU is configured to allocate vector-matrix multiplication operations and other vector processing operations to be interleaved with the vector- matrix multiplication operations using the method.

[0100] A further aspect provides a computer comprising: a data memory; a first vector-matrix hardware multiplication unit directly coupled to the data memory and configured to: receive a stream of input vectors; and sequentially process the stream of input vectors so as to compute, for each input vector before processing a next input vector in the stream, an output vector computed as a matrix multiplication of the input vector with a matrix read directly from the data memory; a hardware vector processing unit; and a second vector-matrix hardware multiplication unit, wherein the computer is configured to transfer the output vector from the hardware vector processing unit to the second vector-matrix hardware multiplication unitwhilst the stream of input vectors is still being processed at the first vector-matrix hardware multiplication unit.

Claims

Claims1. A method of processing data in a neural network using a processor, the method comprising: receiving, at a vector-matrix hardware multiplication unit of the processor, a first input vector of a stream of input vectors; computing a first output vector as a matrix multiplication of the first input vector with a weight matrix of the neural network read directly from a region of data memory directly coupled to the vector-matrix hardware multiplication unit; subsequent to computing the first output vector, receiving, at the vector-matrix hardware multiplication unit, a second input vector of the stream of input vectors; and computing a second output vector as a matrix multiplication of the second input vector with the weight matrix read directly from the region of data memory, the weight matrix persisting in the region of data memory between computing the first output vector and computing the second output vector.

2. The method of claim 1, further comprising: while the vector-matrix hardware multiplication unit computes the second output vector from the second input vector in the stream, applying at a vector processing unit non-linear processing to the first output vector.

3. The method of claim 2, further comprising, while the vector processing unit applies the non-linear processing operation to the second output vector and the vector-matrix hardware multiplication unit processes a third input vector of the stream, computing, at a second vector-matrix hardware multiplication unit, a further output vector by multiplying the first output vector, after the non-linear processing operation, with a second weight matrix read directly from a second region of data memory to which the second vector-matrix hardware multiplication unit is directly coupled.

4. The method of claim 3, wherein the second vector-matrix hardware multiplication unit is on the same or another processor.

5. A computer comprising: a data memory;a first vector-matrix hardware multiplication unit directly coupled to the data memory and configured to: receive a stream of input vectors; and sequentially process the stream of input vectors so as to compute, for each input vector before processing a next input vector in the stream, an output vector computed as a matrix multiplication of the input vector with a matrix read directly from the data memory; a hardware vector processing unit; and wherein the computer is configured to transfer the output vector from the first vectormatrix hardware multiplication unit to the hardware vector processing unit for non-linear processing whilst the stream of input vectors is still being processed at the first vector- matrix hardware multiplication unit.

6. The computer of claim 5, comprising: a second vector-matrix hardware multiplication unit, wherein the computer is configured to transfer the output vector, after non-linear processing, from the hardware vector processing unit to the second vector- matrix hardware multiplication unit whilst the stream of input vectors is still being processed at the first vector-matrix hardware multiplication unit.

7. A processor comprising: a data memory; and a vector-matrix hardware multiplication unit directly coupled to the data memory, the vector- matrix hardware multiplication unit configured to: receive a stream of input vectors, and sequentially process the stream of input vectors, so as to compute an output for each input vector before processing the next input vector in the stream, wherein the output is computed as a matrix multiplication of the input vector with a matrix read directly from the data memory.

8. The processor according to claim 7, wherein the processor is an in-memory processor that integrates processing capabilities of the vector-matrix hardware multiplication unit directly within memory.

9. The processor according to claim 8, wherein the vector-matrix hardware multiplication unit is in the form of hardware logic or hardware circuitry integrated within memory cells of the data memory.

10. The processor according to any of claims 7 to 9, wherein the data memory is a random access memory (RAM), and the matrix is held statically as the stream of input vectors is processed.

11. The processor according to claim 7, further comprising multiple vector-matrix hardware multiplication units, the processor being configured to distribute different matrix multiplications between multiple vector- matrix hardware multiplication units.

12. A die comprising: a data memory; and a vector-matrix hardware multiplication unit directly coupled to the data memory, and configured to: receive a stream of input vectors, and sequentially process the stream of input vectors, so as to compute an output for each input vector before processing the next input vector in the stream, wherein the output is computed as a matrix multiplication of the input vector with a matrix read directly from the data memory.

13. A chip packaging the die according to claim 12.

14. A computer comprising a central processing unit (CPU) and an accelerator processor configured in accordance with any of claims 7 to 11, wherein the CPU is configured to execute a computer program held in a data memory coupled to the CPU, which, when executed, causes the CPU to allocate to the accelerator processor a series of vector-matrix multiplication operations performed on a series of input vectors.

15. The computer of claim 14, wherein the CPU is configured to allocate vector- matrix multiplication operations and other vector processing operations to be interleaved with the vector-matrix multiplication operations using the method of any of claims 1 to 4.

Images