Neural networks including matrix multiplication

Abstract

Neural networks including matrix multiplication are disclosed. A method and a data processing system for implementing a neural network comprising at least one matrix multiplication operation are disclosed. The matrix multiplication operation is mapped to a neural network operation graph comprising at least one transformation and at least one convolution. The at least one convolution is implemented in fixed function hardware of a neural network accelerator.

Description

Technical Field

[0001] This invention relates to neural networks including matrix multiplication. Background Technology

[0002] Matrix multiplication appears in many contexts of neural networks. A single matrix multiplication has the following form:

[0003] Z = XY

[0004] In this equation, X is a matrix of dimension [P,Q], and Y is a matrix of dimension [Q,R]. The output Z is a matrix of dimension [P,R]. Treating the matrices as 4-D tensors, X has dimensions [1,1,P,Q], Y has dimensions [1,1,Q,R], and Z has dimensions [1,1,P,R].

[0005] Throughout the document, data tensors are notated as “NCHW”, where N is the batch size, C is the number of channels, H is the height, and W is the width. Similarly, weights / coefficients are notated as “OIHW”, where O is the number of output channels, I is the number of input channels, H is the kernel height, and W is the kernel width.

[0006] The task of matrix multiplication can be generalized to batch matrix multiplication, which involves performing multiple matrix multiplications in the same operation. For batch matrix multiplication, it is permissible for the two leading dimensions of the 4-D tensor defined above to be greater than 1. Therefore, a more general form is a tensor X of dimension [M,N,P,Q] and a tensor Y of dimension [M',N',Q,R]. This represents the corresponding multiplication between matrices of size [P,Q] and [Q,R] performed in the same operation by (max(M,M′)max(N,N′)).

[0007] If both M and M' are greater than 1, they are equal to each other. Alternatively, one or both of M and M' can be equal to 1. Similarly, if both N and N' are greater than 1, they are equal to each other. Alternatively, one or both of N and N' can be equal to 1. When M = M' and N = N', the matrices being multiplied are all different. When any (or two or three) of M, N, M', and N' is equal to 1, broadcasting is performed on the relevant dimensions (or multiple dimensions), that is, the relevant operands are reused on the broadcast dimensions to form the desired (max(M,M')max(N,N')) matrix multiplication.

[0008] It should be understood that the above definition of batch matrix multiplication applies to any number of dimensions greater than two. In other words, the choice of 4-D tensors is not particularly important, but merely for ease of explanation. Generally speaking, any number of dimensions can exist. Matrix multiplication is performed on two dimensions, and the remaining dimensions (often referred to as the "batch" dimensions) have matching sizes (or have a size equal to 1, in which case it implies broadcasting). The notations [...],P,...,Q,...] and [...],R,...] will be used to indicate any number of additional dimensions that match M, N, M', and N' in the manner described above, where two dimensions require matrix multiplication. In this notation, the additional dimensions can be located in one or more of the following positions: before, between, and after the two dimensions where matrix multiplication is performed. The dimension of size Q in the two input tensors can be referred to as the "inner" dimension. Here, for the sake of simplicity of the example and without loss of generality, it will be assumed that matrix multiplication is performed on the last two dimensions. Therefore, the tensors in the example will have dimensions [...],P,Q] and [...],Q,R. However, it should be understood that this is not intended to impose restrictions.

[0009] Batch matrix multiplication is useful in a variety of situations. During the training phase of a neural network, it appears in the backpropagation of certain layers. This is true, for example, in fully connected layers. Fully connected layers are widely used to generate classification outputs in various neural networks, including but not limited to those for computer vision tasks and natural language processing (NLP). It should be understood that fully connected layers have many other uses, including but not limited to gating functions in multilayer perceptron (MLP) and long short-term memory (LSTM) networks. Batch matrix multiplication also occurs in the forward propagation of certain layers during the training and inference phases. This applies to attention-based neural networks, such as transformer networks. Attention-based neural networks have traditionally been used primarily for NLP; however, recently they have also been used to excel in computer vision tasks.

[0010] The outer product of two vectors can be viewed as a special case of matrix multiplication, where Q = 1. This can be generalized by analogy to batch matrix multiplication to batch outer products, where a tensor X of shape [M, N, P, 1] is multiplied by a tensor Y of shape [M', N', 1, R], and the terms M, N, M', and N' are the same as defined above in the case of batch matrix multiplication. Here, both "single" outer products and batch outer products will be collectively referred to as outer products. To implement an outer product via matrix multiplication, the first input (X) should be a column vector or tensor with a width of 1. The second input (Y) should be a row vector or tensor with a height of 1. If the input vectors for the outer product operation are not provided in these forms, transpose or permutation operations can be applied to one or both of them as needed to arrange them into the desired form. (As used in this paper, the term permutation should be understood as equivalent to the common usage of the term "transpose" in the field of neural networks, meaning to reorder the dimensions of a tensor.)

[0011] Existing neural network accelerator (NNA) hardware is typically designed for evaluating convolutional layers and may not natively support matrix multiplication. In particular, exemplary NNAs are designed to multiply the same set of weights simultaneously by multiple sets of input data elements at multiple processing elements. Much of the computation in a convolutional neural network (CNN) is dedicated to operations such as convolution, which require applying the same coefficients (weights) to multiple sets of input data. For this reason, some neural network accelerators are specifically designed for such operations. In some cases, such NNAs may not natively support matrix multiplication; in others, due to the way the hardware is designed, matrix multiplication may be supported, but at a relatively low efficiency. Summary of the Invention

[0012] This summary is provided to introduce, in a simplified form, a series of concepts further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0013] The goal is to efficiently implement outer products, matrix multiplication, and batch matrix multiplication on existing neural network hardware. Where possible, this will be done by leveraging existing, specially optimized hardware. This avoids the need to design additional, specially optimized hardware to add to the NNA to support matrix multiplication. It also avoids the need to use a separate general-purpose processor to perform matrix multiplication. Data transfers with such external general-purpose processors during neural network evaluation by the NNA can result in significant overhead in terms of memory access bandwidth; therefore, it is desirable to avoid the need for such transfers if possible.

[0014] This invention is defined by the claims.

[0015] This invention discloses a method and data processing system for implementing a neural network that includes at least one matrix multiplication operation. The matrix multiplication operation is mapped to a neural network operation graph that includes at least one transformation and at least one convolution. The at least one convolution is implemented in the fixed-function hardware of a neural network accelerator.

[0016] In this context, "fixed-function" contrasts with general-purpose processor hardware, which is fully programmable to implement any (arbitrary) function or algorithm. The behavior of a fixed-function circuit system can be configured to a limited extent. Modules of a fixed-function circuit system can, for example, be configured to apply different operations or process different tensor sizes, or be provided with a finite set of parameters that can be used to control the behavior of the function it implements. However, it is not fully programmable in the sense that it can execute arbitrary algorithms.

[0017] According to one aspect, a method is provided for implementing a neural network comprising multiple layers using a neural network accelerator including fixed-function hardware, wherein at least one layer includes a matrix multiplication operation defined in two or more dimensions between a first tensor X of dimension [...,P,...,Q,...] and a second tensor Y of dimension [...,Q,...,R,...], the method comprising: mapping the matrix multiplication operation to a neural network operation graph including at least one transformation and at least one convolution operation; and evaluating the neural network operation graph to evaluate the matrix multiplication operation, wherein at least one convolution operation is evaluated in the fixed-function hardware.

[0018] A first tensor X (or a tensor derived from it) can be considered as input data for at least one convolution operation, and a second tensor Y (or a tensor derived from it) can be considered as coefficient data for at least one convolution operation. Tensors can be derived from the first tensor X and / or the second tensor Y through at least one transformation.

[0019] At least one transformation can reconfigure the second tensor Y to arrange a dimension of size R in the output channel dimension before evaluating at least one convolution operation.

[0020] At least one transformation can reconfigure two tensors to arrange dimensions of size Q in the input channel dimension before evaluating at least one convolution operation.

[0021] At least one transformation can reconfigure the first tensor X to arrange dimensions of size P in the dimensions traversed by at least one convolution operation. The traversed dimensions are typically height H and width W. Therefore, in some cases, they may also be referred to as "spatial" dimensions.

[0022] The fixed-function hardware optionally includes: an input buffer configured to temporarily store input data elements; a coefficient buffer configured to temporarily store weights; and a plurality of processing elements, each processing element configured to multiply one or more input data elements by one or more corresponding weights, wherein, in each of the plurality of hardware cycles: the coefficient buffer is configured to simultaneously provide a set of one or more weights to all processing elements, and the input buffer is configured to provide each of the processing elements with a corresponding distinct set of one or more input data elements.

[0023] The hardware can be configured to evaluate at least one convolution operation by processing in parallel several groups of one or more input data elements selected along a first dimension traversed by the convolution operation, wherein at least one transformation optionally reconfigures the first tensor X to arrange a dimension of size P in the first dimension.

[0024] In some examples: the first tensor X has dimensions [1,1,P,Q], and the second tensor Y has dimensions [1,1,Q,R]; at least one transformation reconfigures the first tensor X to form a reconfigured first tensor with dimensions [1,Q,1,P]; at least one transformation reconfigures the second tensor Y to form a reconfigured second tensor with dimensions [R,Q,1,1]; and the reconfigured first and second tensors are input into at least one convolution. This achieves single matrix multiplication.

[0025] In some examples: the first tensor X has dimensions [M, N, P, Q], and the second tensor Y has dimensions [M', N', Q, R], where B = (max(M, M')max(N, N')) > 1; at least one transformation splits and / or copies and reconfigures the first tensor X to form a first tensor reconfigured by B, each tensor having dimensions [1, Q, 1, P], where if M' > M = 1 or N' > N = 1, then at least one transformation includes copying the first tensor in the corresponding dimension, and if M' = M > 1 or N' = N > 1, then... One of the transformations involves splitting the first tensor along the corresponding dimension; at least one transformation splits and / or copies and reconfigures the second tensor Y to form a B-reconfigured second tensor with dimensions [R,Q,1,1], wherein if M>M'=1 or N>N'=1, at least one transformation involves copying the second tensor along the corresponding dimension, and if M'=M>1 or N'=N>1, at least one transformation involves splitting the second tensor along the corresponding dimension; and at least one convolution involves a B-convolution applied to the first reconfigured tensor and the second reconfigured tensor of the corresponding pair. In this way, batch matrix multiplication can be implemented in a manner similar to multiple single matrix multiplications.

[0026] If (i) M' = 1 and M > 1, or (ii) N' = 1 and N > 1, then broadcasting is performed so that the second tensor Y is reused in several convolutions. If (i) M' = 1 and M > 1, or (ii) N' = 1 and N > 1, then broadcasting is performed so that the first tensor X is reused in several convolutions.

[0027] In some examples: the first tensor X has dimensions [M, N, P, Q], and the second tensor Y has dimensions [M', N', Q, R]; at least one transformation reconfigures the first tensor X to form a reconfigured first tensor with dimensions [1, BQ, 1, P]; at least one transformation reconfigures the second tensor Y to form a reconfigured second tensor with dimensions [BR, Q, 1, 1]; and at least one convolution includes grouped convolutions, where each of the B groups has a Q input channel and an R output channel, applied to the reconfigured first tensor and the reconfigured second tensor. Here, B = (max(M, M')max(N, N')). If broadcasting is required (i.e., if M > M' = 1, if M' > M = 1, if N > N' = 1, or if N' > N = 1), then this can be implemented by appropriately repeating the relevant dimensions in the relevant permutation and reshaping tensors. Specifically, if M' > M = 1 and / or N' > N = 1, the reconfiguration of the first tensor involves copying the first tensor M' times and / or N' times in the corresponding dimension; and if M > M' = 1 and / or N > N' = 1, the reconfiguration of the second tensor involves copying the second tensor M times and / or N times in the corresponding dimension. In this way, batch matrix multiplication can be implemented via (single) grouped convolution operations. At least one transformation may also include reconfiguring the results of the grouped convolutions to form an output tensor with dimensions [M, N, P, R].

[0028] The first tensor X may have dimensions [M, N, P, 1] and the second tensor Y may have dimensions [M', N', 1, R]. This performs an outer product. The first tensor is a tensor with a width of 1; and the second tensor is a tensor with a height of 1. Optionally, in some embodiments, the tensor input for the outer product operation may be provided as a pair of tensors with a height of 1 (therefore the first tensor X has dimensions [M, N, 1, P] and the second tensor Y has dimensions [M', N', 1, R]) or a pair of tensors with a width of 1 (therefore the first tensor X has dimensions [M, N, P, 1] and the second tensor Y has dimensions [M', N', R, 1]). In this case, at least one transformation may include a permutation operation performed on at least one tensor to produce a tensor with a width of 1 and a tensor with a height of 1, which are suitable for processing as defined above. The permutation operation may be performed alone or in combination with any other part of at least one transformation.

[0029] Optionally, the first tensor X has dimensions [1,1,P,1] and the second tensor Y has dimensions [1,1,1,R]. At least one transformation can reconfigure (reshape) the first tensor X to form a reconfigured (reshaped) first tensor with dimensions [1,P,1,1]; and at least one transformation can reconfigure (reshape) the second tensor Y to form a reconfigured (reshaped) second tensor with dimensions [R,1,1,1]. At least one convolution can include grouped convolution applied to the reconfigured first tensor and the reconfigured second tensor to produce an output with dimensions [1,PR,1,1]. At least one transformation can reshape this output to have dimensions [1,1,P,R].

[0030] At least one transformation may include one or more first transformations performed on a first tensor X and / or a second tensor Y prior to at least one convolution, and optionally a second transformation performed on the result of at least one convolution. In particular, the second transformation may reverse the effects of one or more first transformations. For example, this may involve permuting the dimensions of the output data to match the dimensions of the input data prior to the first transformation. Optionally, if the neural network comprises consecutive layers containing matrix multiplication operations, the second transformation may be performed only after the last layer of said consecutive layers.

[0031] At least one transformation can be performed, at least in part, using a memory manipulation module configured to manipulate data stored in memory. The MMM can be part of the NNA, particularly part of the NNA's fixed-function hardware. Alternatively, the MMM can be part of a data processing system that includes the NNA.

[0032] The memory manipulation module may include: an internal buffer; a memory read block configured to read data from memory and write data to the internal buffer; a memory write block configured to read data from the internal buffer and write data to memory; and a control channel between the memory read block and the memory write block, wherein the memory read block and the memory write block are configured to communicate via the control channel to maintain synchronization between them when writing data to the internal buffer and reading data from the internal buffer, respectively.

[0033] The method may optionally include analyzing the matrix multiplication operation before mapping it to a neural network operation graph, and determining how to perform the matrix multiplication operation based on the analysis results. This includes determining that at least one transformation and at least one convolution operation should be used to perform the matrix multiplication operation, and rejecting at least one alternative method for performing the matrix multiplication operation. For example, the rejected alternative method may perform the matrix multiplication using a neural network operation graph that includes one or more element-wise operations.

[0034] The steps to determine how to perform matrix multiplication can be based on one or more of the following: the size of the first tensor in one or more dimensions; the size of the second tensor in one or more dimensions; the memory access bandwidth required to perform matrix multiplication using the chosen method; the memory size required to perform matrix multiplication using the chosen method; the number of hardware traversals required to perform matrix multiplication using the chosen method; the execution time on the fixed-function hardware required to perform matrix multiplication using the chosen method; the power consumption required to perform matrix multiplication using the chosen method; and the capabilities of the fixed-function hardware. These capabilities may include the ability to instantiate weights (coefficients) for convolution operations based on variable input data. Some existing NNAs may only be able to instantiate weights based on constants rather than variables.

[0035] A data processing system for implementing a neural network comprising multiple layers is also provided, wherein at least one layer includes matrix multiplication operations defined in two or more dimensions between a first tensor X of dimension [...,P,...,Q,...] and a second tensor Y of dimension [...,Q,...,R,...]. The data processing system includes: a mapping unit configured to map matrix multiplication operations to a neural network computation graph including at least one transformation and at least one convolution operation; and a neural network accelerator including fixed-function hardware, wherein the neural network accelerator is configured to evaluate the neural network computation graph to evaluate the matrix multiplication operations, wherein at least one convolution operation is evaluated in the fixed-function hardware.

[0036] Fixed-function hardware may include: an input buffer configured to temporarily store input data elements; a coefficient buffer configured to temporarily store weights; and a plurality of processing elements, each processing element configured to multiply one or more input data elements by one or more corresponding weights, wherein, in each of the plurality of hardware cycles: the coefficient buffer is configured to simultaneously provide a set of one or more weights to all processing elements, and the input buffer is configured to provide each of the processing elements with a corresponding distinct set of one or more input data elements.

[0037] The data processing system may include a memory manipulation module for manipulating data stored in memory, wherein the memory manipulation module is used to perform at least one transformation.

[0038] The memory manipulation module may include: an internal buffer; a memory read block configured to read data from memory and write data to the internal buffer; a memory write block configured to read data from the internal buffer and write data to memory; and a control channel between the memory read block and the memory write block, wherein the memory read block and the memory write block are configured to communicate via the control channel to maintain synchronization between them when writing data to the internal buffer and reading data from the internal buffer, respectively.

[0039] In some examples, the neural network layer that includes matrix multiplication operations can be a classification layer, used to classify the neural network's input into one of many categories.

[0040] Neural networks can be configured for use in one of the following: natural language processing applications; and image processing applications. NLP applications include, but are not limited to, speech recognition; text-to-speech; and machine translation. The input to a neural network may include text data, audio data, or multimodal data, such as text data and image data (such as captions and associated images). Image processing applications include, but are not limited to, image segmentation; image classification; and optical character recognition. The input to a neural network may include image data or video data.

[0041] Neural networks can include attention-based neural networks.

[0042] A data processing system is also provided, which is configured to perform the methods outlined above. The data processing system can be implemented in hardware on an integrated circuit.

[0043] A method for manufacturing the data processing system as described above using an integrated circuit manufacturing system is also provided.

[0044] A method for manufacturing the data processing system as described above using an integrated circuit manufacturing system is also provided, the method comprising: processing a computer-readable description of the data processing system using a layout processing system to generate a circuit layout description of an integrated circuit embodying the data processing system; and manufacturing the data processing system based on the circuit layout description using an integrated circuit manufacturing system.

[0045] A computer-readable code is also provided, configured to cause the methods outlined above to be executed when the code is run. A computer-readable storage medium (optionally non-transitory) is also provided on which the computer-readable code is encoded.

[0046] It also provides an integrated circuit definition dataset, which, when processed in an integrated circuit manufacturing system, configures the integrated circuit manufacturing system to manufacture the data processing system as outlined above.

[0047] A computer-readable storage medium is also provided, on which a computer-readable description of a data processing system as outlined above is stored, which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the data processing system.

[0048] A computer-readable storage medium is also provided, on which a computer-readable description of a data processing system as described above is stored. When processed in an integrated circuit manufacturing system, the computer-readable description causes the integrated circuit manufacturing system to perform the following operations: process the computer-readable description of the data processing system using a layout processing system to generate a circuit layout description of an integrated circuit embodying the data processing system; and manufacture the data processing system using an integrated circuit generation system based on the circuit layout description.

[0049] An integrated circuit manufacturing system is also provided, which is configured to manufacture the data processing system as described above.

[0050] An integrated circuit manufacturing system is also provided, comprising: a computer-readable storage medium storing a computer-readable description of a data processing system as outlined above; a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit embodying the data processing system; and an integrated circuit manufacturing system configured to manufacture the data processing system based on the circuit layout description. The layout processing system may be configured to determine location information of logic components of a circuit derived from the integrated circuit description to generate a circuit layout description of an integrated circuit embodying the data processing system.

[0051] As will be apparent to those skilled in the art, the above features can be appropriately combined, and can be combined with any aspect of the examples described herein. Attached Figure Description

[0052] The example will now be described in detail with reference to the accompanying drawings, in which:

[0053] Figure 1A The matrix multiplication operation is shown;

[0054] Figure 1B The outer product is shown as a special case of matrix multiplication;

[0055] Figure 1C Batch matrix multiplication is shown;

[0056] Figure 1D This demonstrates batch matrix multiplication using broadcasting;

[0057] Figure 2This is a block diagram of a hardware accelerator including fixed-function hardware, according to an example of this disclosure;

[0058] Figure 3 yes Figure 2 A block diagram of the convolutional engine used;

[0059] Figure 4 It is a block diagram based on an example data processing system;

[0060] Figure 5 yes Figure 4 Block diagram of the memory manipulation module;

[0061] Figure 6A This is a flowchart illustrating a method for implementing a neural network that incorporates matrix multiplication, based on an example.

[0062] Figure 6B This is a flowchart illustrating a method for implementing a neural network that incorporates matrix multiplication, according to another example;

[0063] Figure 7 A computational graph consisting of basic neural network operations for performing matrix multiplication using convolution operations is shown according to an example;

[0064] Figure 8 A computational graph is shown, based on an example, consisting of basic neural network operations for performing batch matrix multiplication using convolution;

[0065] Figure 9 It shows the basis Figure 8 A practical example of batch matrix multiplication in a computation graph;

[0066] Figure 10 A computational graph is shown, based on an example, consisting of basic neural network operations for performing batch matrix multiplication using grouped convolution;

[0067] Figure 11 It shows the basis Figure 10 A practical example of batch matrix multiplication in a computation graph;

[0068] Figure 12 The diagram illustrates a computational graph based on an example, consisting of basic neural network operations for implementing outer products using grouped convolutions.

[0069] Figure 13 A computational graph is shown, based on an example, consisting of basic neural network operations for performing batch matrix multiplication using element-wise operations;

[0070] Figure 14 It shows the basis Figure 13 A practical example of matrix multiplication in a computation graph;

[0071] Figure 15 A computational graph is shown, based on an example, consisting of basic neural network operations for implementing outer products using element-wise operations;

[0072] Figure 16 A computer system in which a data processing system is implemented is shown; and

[0073] Figure 17 A manufacturing system for generating integrated circuits that embody a data processing system is shown.

[0074] The accompanying drawings illustrate various examples. Those skilled in the art will understand that the element boundaries (e.g., boxes, groups of boxes, or other shapes) shown in the drawings represent one example of a boundary. In some examples, it may be that one element can be designed as multiple elements, or multiple elements can be designed as one element. Where appropriate, common reference numerals are used throughout the drawings to indicate similar features. Detailed Implementation

[0075] The following description is given by way of example to enable those skilled in the art to make and use the invention. The invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be apparent to those skilled in the art.

[0076] The implementation scheme will now be described using examples only.

[0077] To meet the need to implement matrix multiplication in systems using neural network accelerators (NNAs), one possibility is to design a dedicated fixed-function hardware module specifically for evaluating matrix multiplication. This hardware module can then be included in the NNA, where it will be responsible for evaluating any layers involved in matrix multiplication as needed.

[0078] Another alternative approach is to evaluate matrix multiplication in general-purpose hardware outside of the NNA, such as a general-purpose CPU or DSP.

[0079] Providing dedicated fixed-function hardware modules in NNAs enables optimized and rapid evaluation. However, a drawback is that these dedicated fixed-function hardware modules occupy additional area within the integrated circuit. This area will be inactive unless matrix multiplication is required to be evaluated.

[0080] Meanwhile, evaluating matrix multiplication in general-purpose hardware offers flexibility and avoids underutilizing the large area of ​​NNAs; however, its efficiency is generally lower due to the lower degree of hardware specialization. The configurability of general-purpose hardware leads to power and area costs because: (i) additional logic is required to flexibly route data; (ii) computational elements are less specialized, meaning computational density is generally lower than that of fixed-function hardware; and (iii) it is difficult to balance hardware bandwidth and computational requirements. Specialized hardware can be more efficient because its design excludes features beyond those strictly necessary for the relevant task.

[0081] Additionally, when using general-purpose hardware external to the NNA, transferring necessary data from the NNA to the general-purpose hardware (e.g., the CPU) incurs overhead. This typically involves the NNA writing data to memory before evaluating matrix multiplications, and the CPU reading data from memory. This can potentially slow down matrix multiplication calculations, especially when memory access speed is (typically) dominant. Furthermore, CPU time is often very valuable due to the demands of the operating system and other running processes. Spending CPU time evaluating matrix multiplications can slow down these other processes and lead to inefficient resource utilization. The same applies to GPUs and DSPs.

[0082] Based on examples of this disclosure, methods for implementing matrix multiplication (including batch matrix multiplication and outer product operations) in hardware are provided, using basic neural network operations available on exemplary NNAs. These basic neural network operations include element-wise operations, transformations, and convolutions.

[0083] Figure 1A The diagram illustrates matrix multiplication between a 4x3 matrix X and a 3x5 matrix Y. Here, the "inner" dimension of the summation has a size Q = 3. The result of the matrix multiplication is a 4x5 matrix Z. Figure 1B The outer product operation is illustrated. The outer product between column vector X and row vector Y represents a special case of matrix multiplication, where the size of the inner dimension is Q = 1.

[0084] Figure 1C The diagram illustrates batch matrix multiplication between a tensor X of dimension [2,4,3] and a tensor Y of dimension [2,3,5]. This involves having Figure 1A The above describes two matrix multiplications of the type shown, producing an output tensor Z of dimensions [2, 4, 5]. Each channel of the first tensor X is multiplied by the corresponding channel of the second tensor Y to produce the corresponding channel of the output tensor Z.

[0085] Figure 1D Another example of batch matrix multiplication is shown. This is similar to... Figure 1CThe example differs in that the second tensor Y has dimensions [1,3,5]. Therefore, broadcasting is performed along the channel dimension of Y such that each channel of the first tensor X is multiplied by a single channel of Y to produce the corresponding channel of the output tensor Z.

[0086] Based on the examples in this disclosure, Figures 1A to 1D All the operations shown can be implemented in the fixed-function hardware of the hardware accelerator (neural network accelerator).

[0087] like Figure 2 As shown, the exemplary hardware accelerator 200 (also referred to herein as a neural network accelerator or NNA) includes the following fixed-function hardware units:

[0088] • A set of 240 convolution engines, dedicated to convolution operations;

[0089] • Element-wise operation unit 285 is specifically designed to perform the same operation on each pair of corresponding elements of two tensors of corresponding sizes;

[0090] • Activation unit 255 is specifically designed to apply an activation function (which may be optional, configurable, or fully programmable) to each element of a tensor;

[0091] • Local Response Normalization (LRN) unit 265 (or simply normalization unit) is specifically designed to perform neighborhood-based normalization operations; and

[0092] • Pooling unit 275 is dedicated to performing pooling operations, such as max pooling and min pooling.

[0093] More specifically, the hardware accelerator 200 includes a digital logic circuit system configured to receive data (including weights and input tensors) and commands for processing the data. The hardware accelerator 200 includes a memory interface 210, an input buffer controller 215, a command decoder 220, a coefficient buffer controller 225, a coefficient buffer 230, an n-input buffer 235, an n-convolution engine 240, an n-accumulator 245, an accumulation buffer 250, an activation unit 255, a local response normalization (LRN) unit 265, a shared buffer 270, a pooling unit 275, and an element-wise operation unit 285. The hardware accelerator 200 can be used to evaluate basic neural network operations to perform matrix multiplication operations.

[0094] Memory interface 210 is configured to provide hardware accelerator 200 and external memory 25 (not in...) Figure 2 As shown in, but in, for example Figure 4The interface between (shown in the diagram) and the external memory 25 can be considered as a separate module from the hardware accelerator 200. Command or configuration information may include, for example, information about weights and data size and format, as well as their location in the external memory.

[0095] Memory interface 210 is configured to receive from external memory 25 weights and data to be used in computations within the neural network, as well as command information for controlling the operations of hardware accelerator 200. The received weights (also referred to herein as coefficients) are passed to coefficient buffer controller 225, and the received data is passed to input buffer controller 215. The received commands are passed to command decoder 220, which is then configured to decode the commands and subsequently publish control information to elements of the hardware accelerator, including coefficient buffer controller 225 and input buffer controller 215, to control how weights and input data are stored in buffers.

[0096] Weights and input data received from external memory via memory interface 210 during a read from external memory may form only a portion of the weights and input data for a single layer, all of which may be used to process that single layer, or may include weights and input data for processing multiple layers. For example, weights received from external memory may form the weights for a single layer, and the received input data may form only a portion of the input data for a single layer (or vice versa). Any combination of data and weights across one or more layers may be received from external memory 25 in a single read from memory (e.g., using a burst read).

[0097] In practice, the amount of weights and data received in a single read from external memory 25 will depend on the size of coefficient buffer 230 and input buffer 235. Weights are passed from coefficient buffer controller 225 to coefficient buffer 230, and received data is passed from input buffer controller 215 to multiple input buffers 235a to 235n. The number of input buffers will depend on the specific implementation of accelerator 200, but can take any value. Input data is shared across all input buffers 235a to 235n. Each input buffer forms an effective bank, allowing the number of input buffers to be increased or decreased depending on the application.

[0098] Input buffers 235a to 235n are connected to each of the multiplexers because each convolution engine 240a to 240n needs access to all valid "repositories" of the input data. Each multiplexer is configured to select an output from one of the input buffers 235 and pass the value output from the selected input buffer 235 to the corresponding convolution engine 240a to 240n. Additionally, weights from coefficient buffer 230 are provided as a second input to each convolution engine 240a to 240n. Convolution engine 240 is configured to perform convolution computation on the received input data using the weights received from coefficient buffer 230. The resulting outputs of each convolution engine 240a to 240n are provided as inputs to the corresponding accumulators of the multiple accumulators 245a to 245n.

[0099] Each accumulator 245a to 245n is connected to an accumulation buffer 250. The accumulation buffer 250 is configured to store the accumulated result received from each accumulator 245a to 245n. The accumulation buffer 250 is connected to a memory interface 210. Therefore, the accumulation buffer 250 is configured to send data to and receive data from an external memory 25 via the memory interface 210. Specifically, the accumulation buffer 250 is configured to store and retrieve its value from the external memory 25 via the memory interface 210, as will be described in more detail below. The accumulation buffer 250 is connected to the inputs of the accumulators 245a to 245n and is configured to feed values ​​back to the accumulators 245a to 245n to enable accumulation calculations.

[0100] The accumulation buffer 250 is configured to pass the accumulated value to the activation unit 255 and / or the element-wise operation unit 285. The activation unit 255 is configured to execute at least one of a plurality of different activation functions.

[0101] The value calculated by activation unit 255 can be passed to LRN unit 265 and / or pooling unit 275 for processing via shared buffer 270. LRN unit 265 is configured to perform local response normalization. This can be performed within a single plane of the input data. Alternatively or additionally, LRN operations can also be performed across planes.

[0102] The results stored in the shared buffer 270 are passed to the memory interface 210, which can store the results in the external memory 25 or pass the results back to the input buffer for further processing, without having to pass them to the external memory first.

[0103] The shared buffer 270 is configured to buffer values ​​from one or more of the activation unit 255, LRN unit 265, pooling unit 275, and element-wise operation unit 285 until all values ​​required for the next operation are available. In this way, the shared buffer 270 is used to improve storage efficiency because it can hold values ​​needed for later operations without having to use external memory 25.

[0104] The element-wise operation unit 285 includes a circuit system configured to perform element-wise operations on tensors received from the accumulation buffer 250 and / or the activation unit 255. Supported element-wise operations may include element-wise addition, subtraction, multiplication, division, and maximum (or minimum) values ​​of the corresponding elements of the tensor.

[0105] Element-wise operations are repeated operations on multiple elements of at least one tensor. Typically, these operations are repeated for all elements of the tensor. Two types of element-wise operations can be considered: unary operations with a single operand and binary operations with two operands. Element-wise operation unit 285 handles binary element-wise operations. Other components of the hardware accelerator can also perform element-wise operations. For example, activation unit 255 can perform unary element-wise operations by applying a function to each element of the tensor.

[0106] Although Figure 2 The hardware accelerator illustrates a specific order of cell arrangement and data processing in the hardware implementation, but it should be understood that the order of specific computations and cross-layer data processing required can vary.

[0107] In some examples of evaluating neural network layers, all functions performed by the activation 255, LRN 265, pooling 275, and element-wise 285 units can be executed. In other examples, only some of these functions can be executed, and not necessarily in the order listed in hardware accelerator 200. To achieve a configurable order of processing these functions, each of the activation 255, LRN 265, pooling 275, and element-wise 285 units can be configured to receive control signaling to configure the unit in bypass mode, in which no function is executed and the input value simply passes through the unit without change.

[0108] In some examples, data from a specific layer may need to be processed first by convolutional engines 240a-n, and then by activation, LRN, pooling, and element-wise units 255, 265, 275, and 285. In these examples, the output from convolutional engines 240a-n is passed to accumulation buffer 250 via accumulators 245a-n, and then to activation, LRN, pooling, and element-wise units 255, 265, 275, and 285 for further processing. In other examples, data may need to be processed differently. For example, data may need to be processed first by activation, LRN, pooling, and element-wise units 255, 265, 275, and 285, and then by convolutional engines 240a-n.

[0109] In these arrangements, data can be directly passed to activation unit 255 via accumulation buffer 250, which has already received input data directly from memory interface 210, which has received data from external memory. In this way, the processing performed by convolution engines 240a-n and accumulators 245a-n is effectively skipped, and data can be directly passed to activation 255, LRN 265, pooling 275, and element-wise 285 units. Then, once the processing using activation, LRN, pooling, and element-wise units 255, 265, 275, and 285 is complete, the resulting value can be passed to input buffer controller 215 via memory interface 210. In some arrangements, the resulting value can first be passed to external memory 25 via memory interface 210 and then retrieved from external memory 25 before use.

[0110] In other arrangements, memory interface 210 can pass the obtained value to input buffer controller 215 instead of passing the value to external memory 25. By avoiding the need to pass the calculated values ​​using activation, LRN, pooling, and element-wise units 255, 265, 275, 285 to external memory 25, memory bandwidth is reduced, and thus the latency of data processing is also reduced.

[0111] Advantageously, because the activation, LRN, pooling, and element-wise units 255, 265, 275, 285 are linearly placed, these operations can be performed sequentially without having to retrieve data from external memory 25. In some embodiments, the connection order of the activation, LRN, pooling, and element-wise units 255, 265, 275, 285 can be varied. For example, the activation, LRN, and pooling units 255, 265, 275 can be connected in reverse, such that the pooling unit is connected to the accumulation buffer 250, and the activation unit is connected to the memory interface 210.

[0112] Figure 3 It shows Figure 2The structure of each element in the convolutional engine 240. The convolutional engine 240 includes multiple elements of multiplication logic 242, each element being configured to multiply weights by input data elements, and multiple elements of addition logic 244, configured in a tree structure to sum the outputs of the elements of multiplication logic 242.

[0113] The exemplary hardware accelerators discussed above are not suitable for the direct evaluation of matrix multiplication, batch matrix multiplication, and outer products. According to various aspects of the invention, these operations are instead mapped to other operations directly supported on the exemplary hardware accelerators. Different approaches may be employed depending on the architecture and capabilities of the hardware accelerator. For example, in some hardware accelerators, the software or hardware may only support convolutions using constant weights (coefficients) and may not support dynamic weights. This would limit the use of convolution-based methods when it is desired to multiply two dynamic data matrices (see below).

[0114] Figure 4 This is a block diagram of a data processing system 10 for implementing a neural network in a hardware accelerator 200 (NNA), based on an example. The neural network includes at least one layer, which includes matrix multiplication operations. The data processing system includes a hardware accelerator 200; a mapping unit 12; a memory 25; and a memory manipulation module (MMM) 40. At least the hardware accelerator 200, the memory 25, and the MMM 40 are connected via a data bus 30. The mapping unit 12 is configured to receive the definition of the neural network and map the layers containing matrix multiplication operations to a basic neural network operation graph that can be executed locally by the hardware accelerator 200. The mapping unit 12 is further configured to control the hardware accelerator 200 (and, if necessary, the MMM 40) to evaluate the matrix multiplications through these basic operations.

[0115] Hardware accelerator 200 is configured to evaluate a basic neural network computation graph. MMM 40 is configured to manipulate multidimensional data in memory in various ways, including permutation operations that modify the order of data dimensions. In some examples, MMM 40 can be configured to transform data by rearranging the channel dimensions of the data in one or both of the width or height dimensions, or by exchanging the channel dimensions with one or both of these spatial dimensions. In alternative examples, MMM can permutate any other combination of the dimensions of the input data, including batch dimensions. A transformation from dimension [B,C,H,W] to dimension [B,1,HC,W] is an example of rearranging the channel dimensions in the spatial dimensions. A transformation from dimension [B,C,H,W] to dimension [B,C / KL,HK,WL] is another such example (where K and L are integers). A transformation from dimension [B,C,H,W] to dimension [B,H,C,W] is an example of exchanging the channel dimensions with one of the spatial dimensions.

[0116] Figure 5 yes Figure 4 A block diagram of the MMM 40 used is shown below. As already mentioned, the MMM 40 is coupled to the memory 25 via bus 30. The MMM 40 includes a memory read block 420; an internal buffer 410; and a memory write block 430. A control channel 440 is used to coordinate the operations performed by the memory read block 420 and the memory write block 430. Both the memory read block 420 and the memory write block 430 are coupled to bus 30. The output of the memory read block 420 is coupled to the input of the internal buffer 410. The input of the memory write block 430 is coupled to the output of the internal buffer 410.

[0117] Memory read block 420 reads data from memory 25. Memory read block 420 writes the data (read from memory 25) to internal buffer 410. Memory write block 430 reads data from internal buffer 410 and writes the data (read from internal buffer 410) back to external memory 25. The combination of operations performed by memory read block 420 and memory write block 430 transforms the data in the aforementioned manner. The transformation can occur when data is moved from memory 25 to internal buffer 410, or when data is moved from internal buffer 410 to memory 25. In some cases, the transformation can occur partially between memory 25 and internal buffer 410, and partially between internal buffer 410 and memory 25.

[0118] The memory read block 420 and the memory write block 430 are provided as separate hardware blocks (e.g., ...). Figure 4 In the example shown, they may be able to operate in parallel. Control channel 240 provides communication between memory read block 220 and memory write block 230 to maintain synchronization between the two blocks. For example, this synchronization can ensure that memory write block 430 does not attempt to read data from internal buffer 410 before data is written by memory read block 420. Similarly, it can be ensured that data in internal buffer 410 is not overwritten by memory read block 420 before data is read from internal buffer 410 by memory write block 430.

[0119] Figure 6AThis is a flowchart illustrating a method performed by a data processing system 10 according to an example of this disclosure. In step 310, mapping unit 12 receives a definition of a neural network including matrix multiplication operations. In step 320, mapping unit 12 maps the matrix multiplication operations to a graph of basic neural network operations. This graph may include (a) at least one convolution operation or (b) at least one element-wise operation. In step 330, hardware accelerator 200 evaluates the neural network operation graph, thereby evaluating the matrix multiplication operations. At least one convolution operation or at least one element-wise operation is evaluated in fixed-function hardware within hardware accelerator 200.

[0120] In some examples, the data processing system 10 can be configured to select how to evaluate matrix multiplication operations. Figure 6B This is a flowchart illustrating such an example. Step 310 and... Figure 6A Same. With Figure 6A Compared to the methods, Figure 6B The method includes two additional steps. In step 312, mapping unit 12 analyzes the matrix multiplication operation. In step 314, mapping unit 12 determines how to implement the matrix multiplication operation based on the analysis results. Specifically, based on the analysis results, mapping unit 12 determines that the matrix multiplication operation should be implemented using a basic neural network operation graph that includes (a) at least one convolution operation and / or (b) at least one element-wise operation, and rejects an alternative. In the subsequent step 321, mapping unit 12 maps the matrix multiplication operation to an appropriate neural network operation graph according to the determination result in step 314. That is, mapping unit 12 maps the matrix multiplication operation to a basic neural network operation graph that includes (a) at least one convolution operation or (b) at least one element-wise operation, as appropriate. In step 330, hardware accelerator 200 evaluates the matrix multiplication through the basic neural network operation graph.

[0121] Mapping unit 12 can perform the analysis in step 312 and the determination in step 314 in various ways. According to one example, mapping unit 12 can determine how to perform matrix multiplication based on the hardware capabilities of the fixed-function hardware, the capabilities of the software or firmware running on the fixed-function hardware, and / or the type of matrix multiplication operation. In some cases, hardware accelerator 200 can distinguish between data tensors and weight tensors. For example, weight tensors can be stored in different formats and / or with different numerical precisions. In such hardware accelerators, it may not be possible to apply data tensors as weight tensors in convolution operations. Therefore, when matrix multiplication requires the multiplication of two data tensors, the hardware accelerator may not have the capability to perform matrix multiplication via convolution operations. In such examples, mapping unit 12 can check whether matrix multiplication involves the multiplication of two data tensors. If so, mapping unit 12 can determine in step 314 that the matrix multiplication operation should be performed via at least one element-wise operation.

[0122] Other criteria can also be used to determine how matrix multiplication should be performed. For example, if the first or second tensor is smaller than a predetermined size (in one or more dimensions), the mapping unit 12 may choose to perform matrix multiplication by at least one element-wise operation. Generally, it is believed that implementation by element-wise operation is less efficient than implementation by convolution. However, if the matrices involved are small, the performance difference may be small, and it is best to keep the convolution engine available for other convolution operations (e.g., to evaluate other previous or subsequent layers of the neural network).

[0123] In some examples, mapping unit 12 can compute the expected cost of performing matrix multiplication via each different possible graph of the neural network operation. The expected cost can take into account execution time, memory size, memory access bandwidth, the number of hardware traversals required to perform matrix multiplication via a given neural network operation graph, and / or power consumption. This can include not only the expected cost of key computational operations (e.g., convolution or element-wise multiplication) but also the expected cost of any necessary transformations of the tensor to place it into the form required for these computational operations.

[0124] Various examples of how matrix multiplication is implemented will now be described in more detail. Without loss of generality, in these examples, it is assumed that the batch size B = (max(M,M′)max(N,N′)) = MN, such that M >= M' and N >= N'. However, it should be understood that this does not limit the scope. If the batch size is different (e.g., if M' > M and / or N' > N), the algorithm will only need to copy and / or permutate and / or reshape the corresponding tensors. Figures 7 to 11 The first set of examples is shown. These examples rely on convolution operations to evaluate matrix multiplication operations.

[0125] exist Figure 7 In the computational graph shown, the neural network operation 700 used to perform matrix multiplication includes convolution 730 and transformations 710, 720, and 740 applied to the input and output of the convolution. A first tensor X provides the data tensor for the convolution; a second tensor Y provides the weights for the convolution. A first permutation operation 710 is applied to the first tensor X. This permutation operation rearranges the dimensions of X from [1, 1, P, Q] to [1, Q, 1, P]. The width dimension of size Q is placed in the channel dimension; and the height dimension of size P is placed in the width dimension.

[0126] In embodiments of the invention, a width dimension greater than 1 is preferably used for the data tensor X because the hardware accelerator 200 is configured to perform convolutions in parallel along the width dimension, for example by processing adjacent windows across the convolution engine 240. In another embodiment, the hardware accelerator may be configured to perform convolutions in parallel along the height dimension. In this case, conversely, it may be advantageous to leave the height dimension in place so that the reconfigured first tensor will have dimensions [1, Q, P, 1].

[0127] The second permutation operation 720 is applied to the second tensor Y. This operation rearranges the dimensions of Y from [1,1,Q,R] to [R,Q,1,1]. Therefore, the height dimension of size Q is placed in the input channel dimension. The width dimension of size R is placed in the output channel dimension. The first and second permutation operations are examples of the first transformation used to prepare the input data before performing the multiplication computation. Note that if one of the inputs to the matrix multiplication is a constant, the corresponding first transformation (permutation operation 710 or 720) can be pre-computed so that the constant matrix is ​​stored ready in its permuted form.

[0128] Matrix multiplication can then be performed using a 1x1 convolution 730, using a reconfigured first tensor X as data and a reconfigured second tensor Y as weights. Convolution 730 has an R filter, Q input channels, a stride of 1, and no padding. The output of this convolution will have dimensions [1, R, 1, P]. A third permutation operation 740 is applied to restore the dimensions to the configuration corresponding to the original inputs X and Y. After permutation 740, the result of the matrix multiplication Z has dimensions [1, 1, P, R]. The third permutation operation is an example of the second transformation applied to the result of the multiplication computation to restore the data so that the dimensions are in the desired order. Typically, this will match the order of the dimensions of the input data. It should be understood that the second transformation (third permutation operation 740) can be eliminated in some cases. In particular, if the neural network includes two consecutive matrix multiplications, the output of convolution 730 in the first matrix multiplication can be directly passed to the input of convolution 730 in the second matrix multiplication (also eliminating the second permutation operation 710 in the second matrix multiplication). Essentially, the third permutation operation 740 in the first matrix multiplication and the first permutation operation 710 in the second matrix multiplication cancel each other out. In this way, the first permutation operation 710 can be performed only at the beginning of a continuous matrix multiplication sequence, and the third permutation operation can be performed only at the end of the sequence.

[0129] Figure 8 Expanded Figure 7 The method is to construct a neural network computation graph 800 for implementing batch matrix multiplication. Figure 8 The method used in this paper is to break down batch matrix multiplication into individual matrix multiplications, and use... Figure 7 The method is used to perform each individual matrix multiplication. The first tensor X, with dimensions [M, N, P, Q], is split into MN component matrices, each with dimensions [1, 1, P, Q]. For the second tensor Y, if broadcasting is required (i.e., if M' = 1 and / or N' = 1), the relevant dimensions of Y are repeated to give it the shape [M, N, Q, R]. The second tensor Y is split into MN component matrices, each with dimensions [1, 1, Q, R]. Using... Figure 7 The neural network operation 700 in the computation graph shown multiplies the corresponding component matrices respectively. Each of these performs the matrix multiplication as a convolution 730 and produces a corresponding tensor of dimension [1, 1, P, R]. If broadcasting is involved, this can be handled by feeding the same component matrices to multiple instances of operation 700. The results of the MN operation are then concatenated 840 to generate the results of a batch matrix multiplication with dimensions [M, N, P, R].

[0130] Figure 9 It shows dependence Figures 7 to 8A concrete example of the method. Batch matrix multiplication is performed between a first tensor X of dimensions [MN=2, P=4, Q=3] and a second tensor Y of dimensions [MN=2, Q=3, R=5]. Here, without loss of generality, only the combined size MN of a single dimension M and N is considered. The method works in the same way, regardless of whether M=1 and N=2 or M=2 and N=1. The first tensor X is split into two tensors, each with dimensions [1, 1, 4, 3]. Each of these is replaced by a first reconfigured tensor of dimensions [1, 3, 1, 4]. The second tensor Y is split into two tensors of dimensions [1, 1, 3, 5], each of which is replaced by a second reconfigured tensor of dimensions [5, 3, 1, 1]. Each first reconfigured tensor (acting as data) is convolved with the corresponding second reconfigured tensor (acting as weights) 730 to produce an output of dimension [1, 5, 1, 4]. These are permuted 740 to have dimensions [1, 1, 4, 5] and concatenated 840 to produce an output tensor Z of dimension [MN = 2, P = 4, R = 5].

[0131] Figure 10 The diagram shows a computation graph that includes a neural network operation graph, which provides an alternative way to implement batch matrix multiplication using grouped convolution. Figure 11 A specific example of applying this method is shown. The first input tensor X (with dimensions [M, N, P, Q]) is permuted 910 to a tensor of dimension [M, N, Q, P]. It is then reshaped 912 to a tensor of dimension [1, MNQ, 1, P]. The second input tensor Y (with dimensions [M', N', Q, R]) is permuted 920 to a tensor of dimension [M', N', R, Q], and then reshaped 922 to a tensor of dimension [M'N'R, Q, 1, 1]. If broadcasting is required (i.e., if M' = 1 and / or N' = 1), it can be implemented by appropriately repeating the relevant dimensions in Y to give it a shape [MNR, Q, 1, 1]. Grouped convolution 930 is then performed using the first reconfigured tensor [1, MNQ, 1, P] as data and the second reconfigured tensor [MNR, Q, 1, 1] as weights. There are MN groups, each with Q channels. This produces a tensor of dimension [1, MNR, 1, P], which is first reshaped by 940 to have dimensions [M, N, R, P], and then permuted by 950 to produce an output tensor Z of dimension [M, N, P, R]. For ease of comparison, Figure 11 The specific instances in the text use dimensions and Figure 9 The input tensors X and Y have the same dimensions. If the batch matrix multiplication is part of a continuous multiplication sequence, the final permutation operation may not be required. (This is in contrast to the previous section on...) Figure 7In a similar manner, the initial permutation 910 of the next batch of multiplications in the sequence effectively cancels out the final permutation 950 of the previous batch of multiplications.

[0132] External products (single external product or batch external product) can be implemented as a special case, used as appropriate. Figure 7 , Figure 8 or Figure 10 The method of any of them. Figure 12 Another alternative shown can be used for a single outer product. The first tensor X is permuted 1010 or reshaped to have dimensions [1,P,1,1]. The second tensor Y is copied 1015P times and permuted 1020 or reshaped. In the example of the invention, this produces a weight tensor with dimensions [RP,1,1,1]. More generally, the permutation / reshaping operation produces a weight tensor whose dimensions are adapted to be grouped convolved with the first tensor (as data) with a reconfigured shape of [1,P,1,1]. Then, using a grouped convolution 1030 with P groups, each group having one input channel and R output channels, the permuted / reshaped weight tensor is convolved with the reconfigured first tensor (as data) with a shape of [1,P,1,1] to produce an output tensor with dimensions [1,PR,1,1]. This is then reshaped 1040 to produce the desired output tensor Z with dimensions [1,1,P,R]. It should be understood that this example is based on the assumption that the data is in NCHW format and the weights are in OIHW format, as mentioned at the beginning above. If different formats are used, the permutation / reshaping operation should be adjusted accordingly to reflect an appropriate grouped convolution with a suitable weight kernel, producing an output of size [1,1,P,R].

[0133] This can also be extended to batch outer products of batch size B, where B = MN. If broadcasting is required (i.e., if M' = 1 and / or N' = 1), this can be implemented by appropriately repeating the product in the relevant dimensions of Y. The first tensor X is permuted 1010 or reshaped to have dimensions [1,BP,1,1], and the second tensor Y is copied 1015P times and permuted 1020 or reshaped to produce a weight tensor of dimension [RBP,1,1 1]. This weight tensor is then convolved with a tensor of shape [1,BP,1,1] using a grouped convolution 1030 with BP groups, each group having one input channel and R output channel, to produce an output tensor of dimension [1,BPR,1,1]. This output tensor is then reshaped 1040 to produce the desired output tensor Z of dimension [B,1,P,R].

[0134] There are various alternative methods to perform matrix multiplication. Figures 13 to 15 A second set of examples is shown. These examples rely on element-wise operations to evaluate matrix multiplication.

[0135] Figure 13 A computational graph for performing batch matrix multiplication using element-wise multiplication is shown according to an example. For a second tensor Y, if broadcasting is required (i.e., if M' = 1 and / or N' = 1), the relevant dimensions of Y are repeated to give it the shape [M, N, Q, R]. A permutation operation 1110 is applied to the second tensor Y to reconfigure it into a third tensor with dimensions [M, N, R, Q]. The third tensor is then split 1120 along the height dimension into R constituent tensors, each with dimensions [M, N, 1, Q]. Element-wise multiplication 1130 is performed between each of these R constituent tensors and the first tensor X. Recall that the first tensor X has dimensions [M, N, P, Q]; therefore, this element-wise multiplication involves broadcasting at least along the height dimension. Alternatively, the constituent tensors can be explicitly copied P times along the height dimension before the element-wise multiplication. The output of the element-wise multiplication 1130 consists of R tensors, each with dimensions [M, N, P, Q]. These tensors are concatenated along the width dimension 1140, resulting in a tensor of dimension [M, N, P, RQ]. Next, the concatenated tensor is permuted 1150 to reconfigure it into a tensor of dimension [M, RQ, N, P]. Essentially, this operation arranges the results of element-wise multiplication into R groups of size Q along the channel axis. The next task is to sum the Q channels of each group. In this example, this is accomplished by means of a grouped convolution 1160, using a weight tensor consisting entirely of 1s and having dimension [R, Q, 1, 1]. The output of this grouped convolution 1160 has dimension [M, R, N, P]. In the final step, this tensor is permuted 1170 to reconfigure it into an output Z of dimension [M, N, P, R].

[0136] Figure 14 It shows the basis Figure 13 A practical example of matrix multiplication in a computational graph. Note that, compared to... Figure 13 On the contrary, Figure 14In the diagram, arrows represent operations, and tensors are represented as blocks. In this example, the first tensor X has dimensions [1,1,4,3] and the second tensor Y has dimensions [1,1,3,5]. In step 1110, the second tensor is permuted to produce a third tensor with dimensions [1,1,5,3]. In step 1120, this third tensor is split into 5 component tensors, each of size [1,1,1,3]. In step 1130, each of the 5 component tensors is element-wise multiplied by the first tensor X. This produces 5 corresponding tensors, each with dimensions [1,1,4,3]. These are concatenated along the horizontal (width) dimension in step 1140 to produce a connected tensor with dimensions [1,1,4,15]. The width dimension now contains 5 groups, each with 4 rows and 3 elements per row. The connected tensor is permuted 1150 to reconfigure it with dimensions [1,15,1,4]. (Note that this step was not included in...) Figure 14 As shown in the diagram, a grouped convolution 1160 is performed on this tensor using a weight tensor of dimension [5,3,1,1] to sum the groups of 3. The result of this summation / grouping convolution has dimensions [1,5,1,4]. This is then permuted 1170 to produce the final output Z, which has dimensions [1,1,4,5].

[0137] It should be noted that the grouped convolution used in the previous example is just one way to sum elements in groups as needed. This step can be implemented in other ways. For example, starting with a connected tensor of dimension [1,1,4,15], it can be changed to convolution with a row of 1 [1,1,1,3], using a stride of 3 in the horizontal direction, and applying it to perform the summation.

[0138] although Figure 14 The diagram shows a neural network operation applied to single matrix multiplication (M=N=1), but the method is general and also applicable to batch matrix multiplication (MN>1) without modification.

[0139] As mentioned earlier, the outer product can be implemented as a special case of matrix multiplication, where Q = 1. Therefore, it can be achieved through... Figure 13 The method shown is used to perform the outer product operation. Figure 15 An alternative method for implementing the outer product using element-wise operations is shown. Starting with X as a tensor [M,N,P,1] of width 1 and Y as a tensor [M',N',1,R] of height 1, X is repeated horizontally R times in step 1210, and Y is repeated vertically P times in step 1220. If broadcasting is required (i.e., if M' = 1 and / or N' = 1), the relevant dimensions of Y are also repeated to give it the shape [M,N,P,R). The outer product can then be computed via element-wise multiplication 1230.

[0140] The following example illustrates the repetition of 1210 and 1220 for the outer product of vectors. It starts with column vector X and row vector Y:

[0141]

[0142] The outer product (matrix multiplication) can be implemented by repeating X three times horizontally and Y four times vertically, and then performing element-wise multiplication:

[0143]

[0144] In some hardware accelerators, element-wise arithmetic units can be configured to broadcast one of the operands X and Y across one or more relevant dimensions. In this case, explicit repetition may not be necessary. Where explicit repetition is required, it can be performed, for example, using MMM 25. Generally, repetition can be performed in several ways. For example, repetition can include concatenating a tensor with itself multiple times along a specified dimension. (This can be done, in particular, with MMM 25.) Another way to implement repetition is by applying a convolution with the weight tensor, where all elements are equal to 1. For example, if replication is desired along the channel dimension, a kernel with 1 input channel and N output channels can be used. If replication is required along the spatial axis, a convolution transpose can be used, where the kernel size in the relevant dimension should be N, with a stride of N in that dimension. These methods can be combined to replicate simultaneously across multiple channels.

[0145] MMM can be used to implement at least one other aspect of a transformation in a neural network computation graph. Specifically, MMM can be used to implement various permutation, reshaping, splitting, and connection operations as needed. Element-wise operations can be implemented using element-wise operation unit 285. Convolutions (including grouped convolutions) can be performed using convolution engine 240. In this way, all neural network operations in the neural network computation graph can be implemented in fixed-function hardware.

[0146] In the foregoing examples, mapping unit 12 and MMM 40 are described as separate components of hardware accelerator 200. This should not be construed as limiting. In some examples, mapping unit 12 (or at least a portion thereof) and MMM 40 may be integrated as part of hardware accelerator 200.

[0147] The methods and data processing systems described herein can be used to process various types of input data in a variety of applications. Of particular interest are image processing applications, where the input to the neural network includes image or video data. The neural network can be configured to process image or video data and produce additional image or video data, for example, image or video data with resolution enhancement, artifact removal, or otherwise, image or video data whose visual content has been modified in some way. Alternatively, the neural network can be configured to analyze the visual content of an image or video, for example, to detect faces, segment and / or classify objects, etc. Also of particular interest are audio processing applications, including tasks such as denoising, speaker detection / recognition. Also of particular interest (and in some cases overlapping with audio processing applications) are NLP applications. These include, but are not limited to, speech recognition; text-to-speech; and machine translation. In some examples, the neural network can be configured to process multimodal data, i.e., data of two or more different types / media. One example is using image data and text (caption) data for classification or validation tasks.

[0148] Figure 16 A computer system is shown in which the data processing system described herein can be implemented. The computer system includes a CPU 902, an NNA 904, a memory 906, and other devices 914, such as a display 916, a speaker 918, and a camera 919. A processing block 910 (corresponding to mapping unit 12, MMM 40, and hardware accelerator 200) is implemented on the NNA 904. The components of the computer system can communicate with each other via a communication bus 905. A temporary register 912 (corresponding to memory 25) is implemented as part of memory 906.

[0149] Figure 4 The data processing system is shown as comprising numerous functional blocks. This is merely illustrative and not intended to define a strict division between the different logical elements of such an entity. Each functional block may be provided in any suitable manner. It should be understood that the intermediate values ​​described herein formed by the data processing system do not need to be physically generated by the data processing system at any point in time, and may merely represent logical values ​​that conveniently describe the processing performed by the data processing system between its inputs and outputs.

[0150] The data processing system described herein may be contained in hardware on an integrated circuit. The data processing system described herein may be configured to perform any of the methods described herein. Generally, any of the functions, methods, techniques, or components described above may be implemented in software, firmware, hardware (e.g., a fixed logic circuit system), or any combination thereof. The terms “module,” “function,” “component,” “element,” “cell,” “block,” and “logic” may be used herein to generally denote software, firmware, hardware, or any combination thereof. In the case of a software implementation, a module, function, component, element, cell, block, or logic represents program code that, when executed on a processor, performs a specified task. The algorithms and methods described herein may be executed by one or more processors that execute code that causes the processor to perform the algorithm / method. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disk, flash memory, hard disk storage, and other memory devices that may use magnetic, optical, and other techniques to store instructions or other data and that may be machine-accessible.

[0151] As used herein, the terms computer program code and computer-readable instructions refer to any kind of executable code for processor execution, including code expressed in machine language, interpreted language, or scripting language. Executable code includes binary code, machine code, bytecode, code defining integrated circuits (such as hardware description languages ​​or netlists), and code expressed in, for example, C++. Executable code can be expressed in programming languages ​​such as OpenCL. Executable code can be, for example, any kind of software, firmware, script, module, or library that, when properly executed, processed, interpreted, compiled, or run in a virtual machine or other software environment, causes the processor of a computer system that supports the executable code to perform the tasks specified by that code.

[0152] A processor, computer, or computer system can be any kind of device, machine, or special-purpose circuit, or a collection or part thereof, having processing capabilities that enable it to execute instructions. A processor can be any kind of general-purpose or special-purpose processor, such as a CPU, GPU, NNA, system-on-a-chip, state machine, media processor, application-specific integrated circuit (ASIC), programmable logic array, field-programmable gate array (FPGA), etc. A computer or computer system may include one or more processors.

[0153] This invention is also intended to cover software defining the configuration of hardware as described herein, such as hardware description language (HDL) software, for designing integrated circuits or for configuring programmable chips to perform desired functions. That is, a computer-readable storage medium may be provided on which computer-readable program code in the form of an integrated circuit definition dataset is encoded, which, when processed (i.e., run) in an integrated circuit manufacturing system, configures the system to manufacture a data processing system configured to perform any of the methods described herein, or to manufacture a data processing system including any of the means described herein. The integrated circuit definition dataset may, for example, be an integrated circuit description.

[0154] Therefore, a method for manufacturing a data processing system as described herein can be provided at an integrated circuit manufacturing system. Furthermore, an integrated circuit definition dataset can be provided, which, when processed in the integrated circuit manufacturing system, enables the method for manufacturing the data processing system to be executed.

[0155] Integrated circuit definition datasets can be in the form of computer code, such as netlists, code for configuring programmable chips, or hardware description languages ​​suitable for manufacturing at any level in integrated circuits, including register-transfer level (RTL) code, high-level circuit representations such as Verilog or VHDL, and low-level circuit representations such as OASIS (RTM) and GDSII. Higher-level representations (such as RTL) that logically define hardware suitable for manufacturing in integrated circuits can be processed on a computer system configured to generate manufacturing definitions of integrated circuits within the context of a software environment that includes definitions of circuit elements and rules for combining these elements to generate the manufacturing definition of the integrated circuit defined by said representation. As is typically the case where software executes at a computer system to define a machine, one or more intermediate user steps (e.g., providing commands, variables, etc.) may be required to configure the computer system to generate the manufacturing definition of the integrated circuit, executing code that defines the integrated circuit to generate the manufacturing definition of that integrated circuit.

[0156] Now refer to Figure 17 Describe an example of processing integrated circuit definition datasets at an integrated circuit manufacturing system in order to configure the system as a manufacturing data processing system.

[0157] Figure 17An example of an integrated circuit (IC) manufacturing system 1002 is shown, configured to manufacture a data processing system as described in any of the examples herein. Specifically, the IC manufacturing system 1002 includes a layout processing system 1004 and an integrated circuit generation system 1006. The IC manufacturing system 1002 is configured to receive an IC definition dataset (e.g., defining a data processing system as described in any of the examples herein), process the IC definition dataset, and generate an IC (e.g., embodying the data processing system as described in any of the examples herein) based on the IC definition dataset. Through the processing of the IC definition dataset, the IC manufacturing system 1002 is configured to manufacture integrated circuits embodying the data processing system as described in any of the examples herein.

[0158] The layout processing system 1004 is configured to receive and process an IC definition dataset to determine a circuit layout. Methods for determining a circuit layout based on an IC definition dataset are known in the art and may involve, for example, synthesizing RTL code to determine the gate-level representation of the circuit to be generated, for example, in relation to logic components (e.g., NAND, NOR, AND, OR, MUX, and FLIP-FLOP components). By determining the location information of the logic components, the circuit layout can be determined based on the gate-level representation of the circuit. This can be done automatically or with user intervention to optimize the circuit layout. Once the layout processing system 1004 has determined the circuit layout, it can output the circuit layout definition to the IC generation system 1006. The circuit layout definition may be, for example, a circuit layout description.

[0159] As is known in the art, IC generation system 1006 generates ICs according to a circuit layout definition. For example, IC generation system 1006 may implement a semiconductor device manufacturing process for generating ICs, which may involve a multi-step sequence of photolithography and chemical processing steps, during which electronic circuits are gradually formed on a wafer made of semiconductor material. The circuit layout definition may be in the form of a mask, which can be used in the photolithography process to generate ICs according to the circuit definition. Alternatively, the circuit layout definition provided to IC generation system 1006 may be in the form of computer-readable code, which IC generation system 1006 can use to form a suitable mask for generating ICs.

[0160] The various processes performed by the IC manufacturing system 1002 can all be implemented in one location, for example, by one party. Alternatively, the IC manufacturing system 1002 can be a distributed system, allowing some processes to be performed at different locations and by different parties. For example, some of the following stages can be performed at different locations and / or by different parties: (i) synthesizing RTL code representing an IC definition dataset to form a gate-level representation of the circuit to be generated; (ii) generating a circuit layout based on the gate-level representation; (iii) forming a mask based on the circuit layout; and (iv) using the mask to manufacture the integrated circuit.

[0161] In some implementations, when the integrated circuit manufacturing definition dataset is processed in the integrated circuit manufacturing system, the integrated circuit manufacturing system can generate devices as described herein. For example, the integrated circuit manufacturing definition dataset, as referred to above... Figure 17 The configuration of the integrated circuit manufacturing system described herein can produce devices as described in this document.

[0162] In some examples, an integrated circuit definition dataset may contain software running on hardware defined at the dataset, or software running in combination with hardware defined at the dataset. Figure 17 In the example shown, the IC generation system can be further configured by the integrated circuit definition dataset to load firmware onto the integrated circuit according to the program code defined at the integrated circuit definition dataset during the manufacturing of the integrated circuit, or otherwise provide the integrated circuit with program code for use with the integrated circuit.

[0163] Compared to known implementations, the concepts set forth in this application can improve performance in devices, apparatuses, modules, and / or systems (and in the methods implemented herein). Performance improvements may include one or more of increased computational performance, reduced latency, increased throughput, and / or reduced power consumption. During the manufacture of such devices, apparatuses, modules, and systems (e.g., in integrated circuits), trade-offs can be made between performance improvements and physical implementation methods, thereby improving manufacturing methods. For example, a trade-off can be made between performance improvements and layout area, thereby matching the performance of known implementations but using less silicon. This can be accomplished, for example, by reusing functional blocks in a serial manner or sharing functional blocks among elements of a device, apparatus, module, and / or system. Conversely, the concepts set forth in this application that cause improvements in the physical implementation of devices, apparatuses, modules, and systems (such as reduced silicon area) can be traded off for performance improvements. This can be accomplished, for example, by manufacturing multiple instances of a module within a predefined area budget.

[0164] The applicant has independently disclosed each individual feature described herein, as well as any combination of two or more such features, to the extent that such features or combinations can be implemented based on the general knowledge of those skilled in the art, in connection with the entire specification, regardless of whether such features or combinations of features solve any problem disclosed herein. In view of the foregoing description, those skilled in the art will understand that various modifications can be made within the scope of this invention.

Claims

1. A method for implementing a neural network comprising multiple layers using a neural network accelerator including fixed-function hardware, wherein at least one of the layers includes matrix multiplication operations defined in two or more dimensions between a first tensor X of dimension [..., P,..., Q,...] and a second tensor Y of dimension [..., Q,..., R,...], the method comprising: The matrix multiplication operation is mapped to a neural network operation graph that includes at least one transformation and at least one convolution operation; as well as The neural network computation graph is evaluated, thereby evaluating the matrix multiplication operation. The at least one convolution operation is evaluated in the fixed-function hardware, and in: The first tensor X has dimensions [M, N, P, Q], and the second tensor Y has dimensions [M', N', Q, R]. The at least one transformation reconfigures the first tensor X to form a reconfigured first tensor with dimensions [1, BQ, 1, P]. The at least one transformation reconfigures the second tensor Y to form a reconfigured second tensor with dimensions [BR, Q, 1, 1]; and The at least one convolution includes grouped convolution, wherein each of the B groups has a Q input channel and an R output channel, applied to the reconfigured first tensor and the reconfigured second tensor. wherein and wherein: If M'>M=1 and / or N'>N=1, then the reconfiguration of the first tensor includes copying the first tensor M' times and / or N' times in the corresponding dimensions; and If M>M'=1 and / or N>N'=1, then the reconfiguration of the second tensor involves copying the second tensor M times and / or N times in the corresponding dimension.

2. The method of claim 1, wherein the first tensor X or a tensor derived therefrom is regarded as input data for the at least one convolution operation, and the second tensor Y or a tensor derived therefrom is regarded as coefficient data for the at least one convolution operation.

3. The method of claim 1 or 2, wherein the fixed-function hardware is configured to evaluate the at least one convolution operation by parallel processing of several groups of one or more input data elements selected along a first dimension traversed by the convolution operation, and The at least one of the transformations reconfigures the first tensor X to arrange the dimension of size P in the first dimension.

4. The method as described in claim 1 or 2, wherein: The first tensor X has dimensions [M, N, P, 1] and the second tensor Y has dimensions [M', N', 1, R]; and / or The at least one transformation includes one or more first transformations performed on the first tensor X and / or the second tensor Y prior to the at least one convolution, and a second transformation performed on the result of the at least one convolution.

5. The method of claim 1 or 2, further comprising, before mapping the matrix multiplication operation to the neural network computation graph, Analyze the matrix multiplication operation, and Based on the analysis results, determine how to perform the matrix multiplication operation, including determining that the matrix multiplication operation should be performed using the at least one transformation and the at least one convolution operation, and rejecting at least one alternative method for performing the matrix multiplication operation.

6. The method of claim 5, wherein determining how to perform the matrix multiplication operation is based on one or more of the following: The size of the first tensor in one or more dimensions; The size of the second tensor in one or more dimensions; The memory access bandwidth required to perform the matrix multiplication operation using the selected method; The memory size required to perform the matrix multiplication operation using the selected method; The number of hardware traversals through the fixed-function hardware required to perform the matrix multiplication operation using the selected method; The execution time on the fixed-function hardware required to perform the matrix multiplication operation using the selected method; The power consumption required to perform the matrix multiplication operation using the selected method; and The capabilities of the fixed-function hardware.

7. The method of claim 1 or 2, wherein the layer of the matrix multiplication operation is a classification layer for classifying the input of the neural network into one of a plurality of categories.

8. The method as claimed in claim 1 or 2, wherein: (A) The neural network is configured to be used for one of the following: Natural language processing applications; and Image processing applications, and / or (B) The neural network includes an attention-based neural network.

9. A data processing system for implementing a neural network comprising multiple layers, wherein at least one of the layers includes matrix multiplication operations defined in two or more dimensions between a first tensor X of dimension [..., P,..., Q,...] and a second tensor Y of dimension [..., Q,..., R,...], the data processing system comprising: Mapping unit (12), the mapping unit being configured to map the matrix multiplication operation to a neural network computation graph including at least one transformation and at least one convolution operation; and A neural network accelerator (200), the neural network accelerator comprising fixed-function hardware, The neural network accelerator (200) is configured to evaluate the neural network computation graph, thereby evaluating the matrix multiplication operation. The at least one convolution operation is evaluated in the fixed-function hardware, and in: The first tensor X has dimensions [M, N, P, Q], and the second tensor Y has dimensions [M', N', Q, R]. The at least one transformation reconfigures the first tensor X to form a reconfigured first tensor with dimensions [1, BQ, 1, P]. The at least one transformation reconfigures the second tensor Y to form a reconfigured second tensor with dimensions [BR, Q, 1, 1]; and The at least one convolution includes grouped convolution, wherein each of the B groups has a Q input channel and an R output channel, applied to the reconfigured first tensor and the reconfigured second tensor. wherein and wherein: If M'>M=1 and / or N'>N=1, then the reconfiguration of the first tensor includes copying the first tensor M' times and / or N' times in the corresponding dimensions; and If M>M'=1 and / or N>N'=1, then the reconfiguration of the second tensor involves copying the second tensor M times and / or N times in the corresponding dimension.

10. The data processing system of claim 9, wherein the fixed-function hardware includes: An input buffer (235a-n) is configured to temporarily store input data elements; A coefficient buffer (230) is configured to temporarily store weights; as well as Multiple processing elements (240a-n), each configured to multiply one or more input data elements by one or more corresponding weights. In each of the multiple hardware cycles: The coefficient buffer (230) is configured to simultaneously supply a set of one or more weights to all the processing elements (240), and The input buffers (235a-n) are configured to supply each of the processing elements (240a-n) with a corresponding different set of one or more input data elements.

11. The data processing system of claim 9 or claim 10, comprising a memory manipulation module (40) for manipulating data stored in a memory (25), wherein the memory manipulation module is used to perform the at least one transformation.

12. The data processing system of claim 11, wherein the memory manipulation module comprises: Internal buffer (410); A memory read block (420) is configured to read data from the memory (25) and write the data to the internal buffer (410); A memory write block (430) configured to read the data from the internal buffer (410) and write the data to the memory (25); and A control channel (440) between the memory read block (420) and the memory write block (430), wherein the memory read block and the memory write block are configured to communicate via the control channel (440) to maintain synchronization between them when writing the data to the internal buffer and reading the data from the internal buffer, respectively.

13. The data processing system of claim 9 or 10, wherein the layer of the matrix multiplication operation is a classification layer for classifying the input of the neural network into one of a plurality of categories.

14. The data processing system as described in claim 9 or 10, wherein: (A) The neural network is configured to be used for one of the following: Natural language processing applications; and Image processing applications, and / or (B) the neural network comprises an attention-based neural network.

15. A computer-readable storage medium having stored thereon computer-readable code which, when executed by a computer, causes the method of any one of claims 1 to 8 to be performed.

Images