Adaptive Matrix Multiplier System
By designing an adaptive matrix multiplier, using burst memory access mode and buffer circuit to optimize matrix multiplication operations, the problem of time-consuming matrix multiplication in deep neural networks is solved, and the acceleration of machine learning and deep learning is achieved.
Patent Information
- Application Number
- CN202010104592.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-05-08
- Filing Date
- 2020-02-20
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-02-20
AI Technical Summary
In machine learning and deep learning, especially deep neural networks, matrix multiplication operations consume a lot of time, resulting in poor performance.
An adaptive matrix multiplier is designed, including a first multiplication unit, a second multiplication unit, a memory loading circuit and an external buffer circuit, and loading data in a single burst pulse through a burst memory access mode and internally Data transmission and calculation are performed between the buffer and the external buffer, and matrix multiplication is optimized.
It significantly accelerates matrix multiplication operations, improves the performance of deep neural networks, and is suitable for the acceleration of machine learning and deep learning.
Smart Images

Figure CN111597501B_ABST
Abstract
Description
[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS]
[0002] This application claims priority to and the benefit of U.S. Provisional Application No. 62 / 808,729, filed on February 21, 2019, entitled “ADAPTIVE MATRIX MULTIPLICATION ACCELERTOR FOR ML / DL APPLICATIONS,” the entire contents of which are incorporated herein by reference. Technical Field
[0003] One or more aspects of the embodiments according to the present disclosure relate to a processing circuit, and more particularly to a circuit for matrix multiplication. Background Art
[0004] Machine learning applications, especially those using deep neural networks, may involve a significant proportion of matrix multiplication operations. In some applications, matrix multiplication can consume most of the running time of the entire deep neural network. Therefore, a system for efficiently multiplying matrices can help improve the performance of deep neural networks.
[0005] [Technical issues]
[0006] The purpose of the present disclosure is to provide an adaptive multiplier that can be used to accelerate machine learning and deep learning. Summary of the invention
[0007] According to some embodiments, a system of an adaptive matrix multiplier is provided, the system comprising: a first multiplication unit; a second multiplication unit; a memory loading circuit; and an external buffer circuit, the first multiplication unit comprising a first internal buffer circuit and a second internal buffer circuit, the second multiplication unit comprising a first internal buffer circuit and a second internal buffer circuit, the memory loading circuit being configured to load data from a memory in a single burst pulse of a burst pulse memory access mode into: the first internal buffer circuit of the first multiplication unit; and the first internal buffer circuit of the second multiplication unit.
[0008] In some embodiments: the first multiplication unit includes an adder circuit, and the first multiplication unit is configured to: form a first product of the following two: a first element of the first internal buffer circuit of the first multiplication unit and a first element of the second internal buffer circuit of the first multiplication unit; transmit the first product to the adder circuit; form a second product of the following two: a second element of the first internal buffer circuit of the first multiplication unit and a second element of the second internal buffer circuit of the first multiplication unit; and transmit the second product to the adder circuit.
[0009] In some embodiments, the first multiplication unit includes: a first multiplication circuit configured to form the first product; and a second multiplication circuit configured to form the second product.
[0010] In some embodiments, the adder circuit is configured to form a first sum of the first product and the second product and send the first sum to the external buffer circuit.
[0011] In some embodiments: the second multiplication unit includes an adder circuit, and the second multiplication unit is configured to: form a third product of: a first element of the first internal buffer circuit of the second multiplication unit and a first element of the second internal buffer circuit of the second multiplication unit; transmit the third product to the adder circuit of the second multiplication unit; form a fourth product of: a second element of the first internal buffer circuit of the second multiplication unit and a second element of the second internal buffer circuit of the second multiplication unit; and transmit the fourth product to the adder circuit of the second multiplication unit.
[0012] In some embodiments: the adder circuit of the second multiplication unit is configured to: form a second sum of the third product and the fourth product, and send the second sum to the external buffer circuit, and the external buffer circuit is configured to add the first sum and the second sum.
[0013] In some embodiments: the first multiplication unit includes an adder circuit; the first internal buffer circuit of the first multiplication unit includes a first mini buffer and a second mini buffer; the second internal buffer circuit of the first multiplication unit includes a first mini buffer and a second mini buffer; the first multiplication unit is configured to: form a first product of the following two: a first element of the first mini buffer of the first internal buffer circuit of the first multiplication unit, and a first element of the first mini buffer of the second internal buffer circuit of the first multiplication unit; transmit the first product to the adder circuit; form a second product of the following two: a first element of the second mini buffer of the first internal buffer circuit of the first multiplication unit, and a first element of the second mini buffer of the second internal buffer circuit of the first multiplication unit; and transmit the second product to the adder circuit.
[0014] In some embodiments: the first multiplication unit is also configured to: form a third product of: the second element of the first mini-buffer of the first inner buffer circuit of the first multiplication unit and the first element of the first mini-buffer of the second inner buffer circuit of the first multiplication unit; transmit the third product to the adder circuit; form a fourth product of: the second element of the second mini-buffer of the first inner buffer circuit of the first multiplication unit and the first element of the second mini-buffer of the second inner buffer circuit of the first multiplication unit; and transmit the fourth product to the adder circuit.
[0015] In some embodiments: the adder circuit is configured to form a first sum of the first product and the second product and send the first sum to the external buffer circuit; and the external buffer circuit is configured to store the first sum in a first element of the external buffer circuit.
[0016] In some embodiments: the adder circuit is further configured to form a second sum of the third product and the fourth product and send the second sum to the external buffer circuit; and the external buffer circuit is configured to store the second sum in a second element of the external buffer circuit.
[0017] In some embodiments: the first multiplication unit includes an adder circuit; and the first multiplication unit is configured to: form a first product of: a first element of the first internal buffer circuit of the first multiplication unit and a first element of an external buffer circuit of the first multiplication unit; transmit the first product to the adder circuit; form a second product of: a second element of the first internal buffer circuit of the first multiplication unit and a second element of the external buffer circuit of the first multiplication unit; and transmit the second product to the adder circuit.
[0018] In some embodiments, the adder circuit is configured to: form a first sum of the first product and the second product; and store the first sum in a first element of the second internal buffer circuit of the first multiplication unit.
[0019] In some embodiments, the first multiplication unit includes: a first multiplication circuit configured to form the first product; and a second multiplication circuit configured to form the second product.
[0020] In some embodiments, the system includes a first entity and a second entity, the first entity including the first multiplication unit and the second multiplication unit.
[0021] In some embodiments, the memory loading circuit is configured to receive a read request from the first entity and to receive a read request from the second entity, and to arbitrate the read requests.
[0022] In some embodiments, the memory loading circuit is configured to load a portion of the matrix from a memory into the second inner buffer circuit of the first multiplication unit in a manner that transposes elements of the portion of the matrix.
[0023] According to some embodiments, a system of an adaptive matrix multiplier is provided, the system comprising: a first multiplication unit; a second multiplication unit; a memory loading circuit; a memory storage circuit; and an external buffer circuit, wherein the first multiplication unit comprises: a multiplication circuit; a first internal buffer circuit; a second internal buffer circuit; a multiplication circuit input multiplexer having a common port connected to the input of the multiplication circuit; a multiplication circuit output multiplexer having a common port connected to the output of the multiplication circuit; and a second internal buffer circuit output multiplexer having a common port connected to the output of the second internal buffer circuit.
[0024] In some embodiments, in a first state of the first multiplication unit, the multiplication circuit input multiplexer is configured to route data from the second inner buffer circuit output multiplexer to a first input of the multiplication circuit; the multiplication circuit output multiplexer is configured to route data from the output of the multiplication circuit to the external buffer circuit; and the second inner buffer circuit output multiplexer is configured to route data from the second inner buffer circuit to the multiplication circuit input multiplexer.
[0025] In some embodiments, in a second state of the first multiplication unit, the multiplication circuit input multiplexer is configured to route data from the external buffer circuit to the first input of the multiplication circuit; the multiplication circuit output multiplexer is configured to route data from the output of the multiplication circuit to the second internal buffer circuit; and the second internal buffer circuit output multiplexer is configured to route data from the second internal buffer circuit to the memory storage circuit.
[0026] According to some embodiments, a system of an adaptive matrix multiplier is provided, the system comprising: a first multiplication unit; a second multiplication unit; a component for loading data from a memory; and an external buffer circuit, the first multiplication unit comprising a first internal buffer circuit and a second internal buffer circuit, the second multiplication unit comprising a first internal buffer circuit and a second internal buffer circuit, the component for loading data from the memory being configured to load the data from the memory into the following in a single burst pulse of a burst pulse memory access mode: the first internal buffer circuit of the first multiplication unit; and the first internal buffer circuit of the second multiplication unit.
[0027] According to some embodiments, a method for performing multiplication is provided, the method comprising: in response to a first register of a processing circuit storing a first value, calculating a matrix product of a first matrix and a second matrix, the first matrix being a first factor in the matrix product, and the second matrix being a second factor in the matrix product; in response to the first register storing a second value, calculating a matrix product of a transpose of the first matrix and the second matrix, the transpose of the first matrix being a first factor in the matrix product, and the second matrix being a second factor in the matrix product; in response to the first register storing a third value, calculating the transpose of the first matrix and the second matrix the first matrix being a first factor in the matrix product and the transpose of the second matrix being a second factor in the matrix product; in response to the first register storing a fourth value, calculating a matrix product of the transpose of the first matrix and the transpose of the second matrix, the transpose of the first matrix being a first factor in the matrix product and the transpose of the second matrix being a second factor in the matrix product; and in response to the first register storing a fifth value, calculating a matrix product of the first matrix and a vector, the first matrix being a first factor in the matrix product and the vector being a second factor in the matrix product.
[0028] According to some embodiments, a method for performing multiplication is provided, the method comprising: calculating a matrix product of a first matrix stored in a first abstract buffer and a second matrix stored in a second abstract buffer; and storing the matrix product in a third abstract buffer, wherein: in response to a second register having a fourth value: mapping the first abstract buffer to a first physical buffer, mapping the second abstract buffer to a second physical buffer, and mapping the third abstract buffer to a third physical buffer; and in response to the second register having a fifth value: mapping the first abstract buffer to the first physical buffer, mapping the second abstract buffer to the third physical buffer, and mapping the third abstract buffer to the second physical buffer.
[0029] In some embodiments, the method further comprises storing the fourth value or the fifth value in the second register based on a size of a matrix to be multiplied.
[0030] According to some embodiments, a method for performing multiplication is provided, the method comprising: reading a portion of a first matrix from a memory into a first buffer in a first unit in a processing circuit and a first buffer in a second unit in the processing circuit in a burst pulse mode; reading a portion of a second matrix from a memory into a second buffer in the first unit in the processing circuit and a second buffer in the second unit; and calculating a matrix product of elements stored in the first buffer in the first unit and elements stored in the first buffer in the second unit.
[0031] In some embodiments, the first unit is located in a first entity in the processing circuit and the second unit is located in a second entity in the processing circuit.
[0032] [Beneficial Effects]
[0033] According to the present disclosure, an adaptive matrix multiplier is provided, which can be used to accelerate machine learning and deep learning. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] These and other features and advantages of the present disclosure will be appreciated and understood with reference to the specification, claims and drawings, in which:
[0035] Figure 1A is a block diagram of a circuit for matrix multiplication according to an embodiment of the present disclosure.
[0036] Figure 1B is a block diagram of a circuit for matrix multiplication according to an embodiment of the present disclosure.
[0037] Figure 2 is a matrix dimension diagram according to an embodiment of the present disclosure.
[0038] Figure 3A It is a data arrangement diagram according to an embodiment of the present disclosure.
[0039] Figure 3B It is a data arrangement diagram according to an embodiment of the present disclosure.
[0040] Figure 4A is a block diagram of a portion of a circuit for matrix multiplication according to an embodiment of the present disclosure.
[0041] Figure 4B is a block diagram of a portion of a circuit for matrix multiplication according to an embodiment of the present disclosure.
[0042] Figure 4C is a block diagram of a portion of a circuit for matrix multiplication according to an embodiment of the present disclosure.
[0043] Figure 5A It is a data arrangement diagram according to an embodiment of the present disclosure.
[0044] Figure 5B It is a data arrangement diagram according to an embodiment of the present disclosure.
[0045] Fig. 6A It is a data arrangement diagram according to an embodiment of the present disclosure.
[0046] Figure 6B It is a data arrangement diagram according to an embodiment of the present disclosure.
[0047] Figure 7 It is a data arrangement diagram according to an embodiment of the present disclosure.
[0048] Fig. 8A It is a data arrangement diagram according to an embodiment of the present disclosure.
[0049] Figure 8B It is a data arrangement diagram according to an embodiment of the present disclosure.
[0050] Fig. 9A is a block diagram of a circuit for matrix multiplication according to an embodiment of the present disclosure.
[0051] Fig. 9B is a block diagram of a circuit for matrix multiplication according to an embodiment of the present disclosure.
[0052] Fig. 9C is a block diagram of a circuit for matrix multiplication according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0053] The detailed description described below in conjunction with the accompanying drawings is intended to be an explanation of the exemplary embodiment of the adaptive matrix multiplier provided according to the present disclosure, rather than being intended to represent that the only form of the present disclosure can be constructed or utilized therein. The description states the features of the present disclosure in conjunction with the illustrated embodiments. However, it should be understood that the same or equivalent functions and structures can be realized by the different embodiments that are also intended to be encompassed within the scope of the present disclosure. As shown elsewhere herein, the same element numbers are intended to indicate the same elements or features.
[0054] Figure 1A An example of an adaptive matrix multiplier according to some embodiments is shown. Figure 1A The illustrated adaptive matrix multiplier includes an entity 105 that includes four multiplication units 110 (labeled U0 through U3). Entity 105 is discussed in further detail below. Each of the multiplication units 110 includes (e.g., eight) multipliers or "multiplication circuits" 405 ( FIG. 4A to FIG. 4CEach of the multiplication units 110 further includes an array 112 of first and second internal buffer circuits 115, second internal buffer circuits 120, and temporary internal buffer circuits 125. Each of the multiplication units 110 may operate (e.g., process) a computational unit that may be referred to as a "data word." Depending on the application, the data word may be an 8-bit integer (INT8), a 16-bit integer (INT16), a 32-bit integer (INT32), a 16-bit floating point (FP16), or a 32-bit floating point (FP32).
[0055] Some embodiments may be used to perform matrix multiplication or matrix and vector multiplication. In some embodiments, the multiplication process may be controlled at least in part by values stored in one or more registers (e.g., stored in register module 180, discussed in further detail below). It should be understood that matrix multiplication is not commutative, so for example, in general, the matrix product A×B is not equal to the matrix product B×A. Some embodiments are capable of performing various multiplications under the control of the one or more registers, including calculating the product of two matrices, calculating the product of the transpose of one matrix and another matrix, the product of the transpose of two matrices, or the product of a matrix and a vector, as further described below.
[0056] The temporary inner buffer circuit 125 is connected to the outer buffer circuit 130, which includes an accumulator 135 and a buffer (e.g., an array of registers) 140. The adaptive matrix multiplier may include a first loading entity module (or "memory loading circuit") 150 for the first inner buffer circuit 115, a second loading entity module 150 for the second inner buffer circuit 120, and a loading unit (or "loading circuit") 155 for the outer buffer circuit 130. The loading entity module 150 and the loading unit 155 may communicate with one or more memories 162 through a memory bridge 165. In some embodiments, the loading entity module 150 performs a read operation from the memory 162 in a burst pulse mode (i.e., the memory may return multiple data words in response to a single read command sent to the memory). Compared with the operation in a mode in which any one of the memories 162 sends only one data word in response to a read command, the operation in the burst pulse mode can obtain significantly higher data throughput. The memory storage circuit 170 (or memory storage unit) may also communicate with the one or more memories 162 through the memory bridge 165 and may store calculation results in the memory 162 .
[0057] The control circuit 175 may control the operation of other circuits in the adaptive matrix multiplier; the control circuit 175 may receive commands from the host (and return status information to the host) via the register module 180 .
[0058] Each of the first inner buffer circuit 115 and the second inner buffer circuit 120 may include an array of mini-buffers (e.g., one mini-buffer for each multiplication circuit in the array of multiplication circuits 112). Each array of mini-buffers may be conceptually considered to form a matrix of elements, with each column of the matrix corresponding to one of the mini-buffers and each row corresponding to a set of elements (with one element in each mini-buffer), with all elements corresponding to a row being located at the same position in the corresponding mini-buffer. For example, the first row may correspond to the first element of each mini-buffer.
[0059] In operation, the array 112 of multiplication circuits may form a dot product of the rows of the minibuffers. As used herein, the "dot product" of two vectors of equal length (e.g., the dot product of the rows of one matrix and the columns of another matrix) refers to the sum of the element-by-element products of the elements of the vectors. The array 112 of multiplication circuits may form, for example, a dot product of (i) the first row of the array of minibuffers of the first inner buffer circuit 115 and (ii) the first row of the array of minibuffers of the second inner buffer circuit 120, and then form a dot product of (i) the second row of the array of minibuffers of the first inner buffer circuit 115 and (ii) the first row of the array of minibuffers of the second inner buffer circuit 120. The use of such operations to perform matrix multiplication is discussed in more detail below.
[0060] Figure 1BAn example of an adaptive matrix multiplier having two entities 105 (each entity 105 having two multiplication units 110) is shown. The number of entities 105 included in the adaptive matrix multiplier and the number of multiplication units 110 of each entity 105 can be selected based on the intended application of the adaptive matrix multiplier and based on hardware resources. The number of entities 105 included in the adaptive matrix multiplier and the number of multiplication units 110 of each entity 105 can be selected at design time or at system startup, or can be changed in real time during operation. In some embodiments, data from the memory 162 is sent to the multiplication unit 110 on a data bus having a width (e.g., 128 bits). The number of minibuffers of the unit can be selected based on the ratio of the width of the data bus to the width of the data word (e.g., equal to the ratio of the width of the data bus to the width of the data word). The number of cells per entity can be selected based on the ratio of memory bus bandwidth / speed to the cycles required for computation (multiplication) (e.g., equal to the ratio of memory bus bandwidth / speed to the cycles required for computation (multiplication)), and fine-tuned by other microarchitectural factors. The granularity of memory access operations is a rectangular, square, or one-dimensional array that can be called a "tile". A tile can be parameterized as "R_TW×C_TW", where R_TW is a number with a value between 1, 2, 4, ..., 2 r , where r is a positive integer (i.e., the block line width can be 1, 2, ..., or other numbers equal to 2 to the power of r), and C_TW is a value between 1, 2, 4, ..., 2 c The block column width may be a block column width of a value within a range of 1, 2, . . . or other number equal to 2 raised to the power of c. Each block may contain a plurality of words. The size may be selected based on the requirements of a proposed application of the adaptive matrix multiplier. Each block, after being extracted from the memory, may be stored in an on-chip storage device, such as in the first inner buffer circuit 115 or the second inner buffer circuit 120 or in a first-in-first-out (FIFO) memory or register based on the requirements of the proposed application of the adaptive matrix multiplier and based on the implementation requirements.
[0061] Taking matrix multiplication as an example, the product of two blocks (ie, A and B) can be calculated and a third block C can be formed:
[0062] [Formula 1]
[0063] C(i,j)=∑ p A(i,p)*B(p,j); where i=0,...,M; j=0,...,N; p=0,...,K
[0064] Memory blocks A and B may be loaded into first inner buffer circuit 115 and second inner buffer circuit 120. If desired, product C may be stored in temporary inner buffer circuit 125 before being sent back to memory 162 in the form of block C.
[0065] Figure 2 An example is shown where two matrices are fetched into two on-chip buffers (e.g., into a first internal buffer circuit 115 and a second internal buffer circuit 120) and the product of the two matrices is written back to memory. As an example, in some embodiments, the size of the first block ("block A") is R_TW×C_TW=12×8 words, the size of the second block ("block B") is R_TW×C_TW=8×4 words, and the size of the third block ("block C") is R_TW×C_TW=12×4 words.
[0066] The first inner buffer circuit 115 and the second inner buffer circuit 120 may have Figure 2 The storage capacity shown in , which is expressed as "entries x width", where the width is the number of mini-buffers in the buffer circuit, and the entries or depth is the number of entries that each mini-buffer can store. Figure 2 In the example shown, for the first inner buffer circuit 115, the entries × width is 12 entries × 8 mini-buffers, for the second inner buffer circuit 120, the entries × width is 4 entries × 8 mini-buffers, and for the outer buffer circuit 130, the entries × width is 12 entries × 4 mini-buffers. The mini-buffer width is 1 word, and a width of 32 bits is used as an example here. To process the multiplication of Formula 1 in parallel, the second inner buffer circuit 120 can be loaded from the block B area of memory in a "transposed" form and filled before starting the multiplication. In some neural network applications, matrix A may be large and constant, or rarely change, while matrix B may be significantly smaller and may change frequently. The latency with respect to matrix B may be an important performance metric, so a second inner buffer circuit 120 with a large number of entries in each mini-buffer may not be very useful, because the use of a large number of entries may result in high latency. In such a system, the number of entries in each mini-buffer of second inner buffer circuit 120 may be significantly smaller than the number of entries in each mini-buffer of first inner buffer circuit 115 (eg, 2 to 8 times smaller (eg, 3 times smaller)).
[0067] In some embodiments, each on-chip "buffer" may be a buffer, a first-in-first-out memory, or a register depending on the requirements of the intended application and the microarchitecture requirements of the adaptive matrix multiplier.
[0068] Reference Figure 3A and Figure 3B As an example, the load entity module 150 and the load unit 155 can load one matrix block from the memory to the on-chip buffer at a time, and if necessary, perform a matrix transposition, as discussed in further detail below. In some embodiments, the load unit is similar to the basic functionality of the load entity. The load unit can fill a buffer (for example, with 8 mini-buffers); and when the data from the memory is a burst transaction, the load entity can fill multiple buffers at the same time. The memory storage circuit 170 stores the contents of the external buffer circuit 130 to the block C area of the memory. In some embodiments, the results written back to the memory do not need to be transposed. In addition, when the block does not fill the on-chip buffer from which it is read, there is no need to pad the block data, because the load entity module 150 and the load unit 155 can handle partial blocks by disabling the buffer enable (BE) of the unused mini-buffer (for example, setting the BE bit to zero). Similarly, the memory storage circuit 170 may handle partial blocking by setting a "word enable" when storing mini-buffer data into memory. Figure 3A and Figure 3B In the example, the size of block A is 12×8 words, the size of block B is 8×4 words, and the size of block C is 12×4 words (e.g. Figure 2 ). Figure 3A and Figure 3B In the example shown, the size of the block (the meaning of the size M, N and K is as follows Figure 2 ) is M_TW=12, N_TW=4 and K_TW=8, and the data size is M=8, N=3 and K=5.
[0069] For example, a first dot product of the first row of buffer A and the first row of buffer B may be calculated first (the first dot product has a value of 90), and the first dot product may be stored in a first entry of buffer C. A second dot product of the first row of buffer A and the second row of buffer B may be calculated second (the second dot product has a value of 100), and the second dot product may be stored in a second entry of buffer C. In other embodiments, the dot product of the second row of buffer A and the first row of buffer B may be calculated second instead.
[0070] Multiplication C(0,0)=∑ i The example of A(0,i)×B(i,0) is Figure 4AAs described above, each multiplication unit 110 may include a multiplier (or "multiplication circuit" 405) such as an array of FP32 multipliers operating in parallel and an adder (or "adder circuit") 410. The first inner buffer circuit 115 may include an array of mini-buffers (e.g., 8 mini-buffers) (each having 12 entries), and the second inner buffer circuit 120 may also include an array of mini-buffers (e.g., 8 mini-buffers) (each having 4 entries). Figure 4A In the example shown, the outer buffer circuit 130 has four mini-buffers, each of which has 12 entries. Figure 4B and Figure 4C C(0,1)=∑ i A(0,i)×B(i,1) and C(1,0)=∑ i Multiplication of A(1,i)×B(i,0).
[0071] Each of the load entity modules 150 may fetch multiple matrix tiles from memory as a burst transaction, perform matrix transpose operations as appropriate (as discussed in further detail below), and populate associated on-chip buffers of entity 105 .
[0072] Figure 5A and Figure 5B A matrix ("Matrix A") is shown being loaded from memory in blocking bursts into three first inner buffer circuits 115 of three of the multiplication units 110 of the entity 105 having four multiplication units 110. As shown, the matrix data is loaded into the three first inner buffer circuits 115 simultaneously in a striping manner.
[0073] Fig. 6A and Figure 6B shows the transfer of a matrix ("matrix B") from memory ( Fig. 6A ) are loaded into three second internal buffer circuits 120 of three of the multiplication units 110 of the entity 105 having four multiplication units 110. Each of the matrix B blocks is extracted from the memory separately, because in this example, the matrix B is stored in the memory "normally" (i.e., not transposed), where LDB=N (where LDB=N means that the matrix B is loaded into the memory in N dimensions, for example Fig. 6A ). After the entire matrix is loaded, the three second inner buffer circuits 120 may be run in parallel (e.g., fed into the corresponding multiplication circuits 405 in parallel) (the three second inner buffer circuits 120 may be filled in parallel or filled one by one, but run in parallel because the three second inner buffer circuits 120 already have the required data. There is no need to run one by one). Fig. 6Aand Figure 6B The "epoch" marked in is the outer loop number. It should be noted that Figures 1A to 6B is an example of one of the options. In some embodiments, Figures 1A to 6B This is merely an example of how a matrix multiplier works and should not be construed as limiting the present disclosure. For example, the number of elements (e.g., the number of entities 105 and the number of multiplication units 110) is merely an example and may be different in different embodiments as will be appreciated by one skilled in the art. The operation period (outer loop) is also Figure 7 Shown in.
[0074] Figure 7 An example is shown of how a first matrix (A) may be stored in three first inner buffer circuits 115 of three multiplication units 110 in an entity 105 having four multiplication units 110, and how portions of a second matrix (B) may be stored in three second inner buffer circuits 120 of three multiplication units 110 in an entity 105 having four multiplication units 110. The product may be formed in three periods, with a new portion of the second matrix (B) being loaded into the three second inner buffer circuits 120 in each period. Figure 7 In the example shown, there is an entity with four multiplication units 110, eight mini-buffers in each first inner buffer circuit 115, and eight mini-buffers in each second inner buffer circuit 120. The block sizes are: M_TW=12; N_TW=4; and K_TW=8; and the data sizes are M=12; N=10; and K=20. There is an inner loop in the K dimension and a middle loop in the M dimension; these inner loops run in parallel and use three epochs (outer loops) to complete Fig. 6A and Figure 6B as well as Figure 7 The matrix multiplication shown in calculates C(i,j)=ΣpA(i,p)*B(p,j).
[0075] Reference Fig. 8A and Figure 8B , when a matrix is transposed while being fetched from memory, both the load entity module 150 and the load unit 155 can handle the matrix transposition. Fig. 8A is an example of a transposed matrix and shows the arrangement of the matrix in memory, and Figure 8B1 shows the arrangement of matrices in three second inner buffer circuits 120 of three of the multiplication units 110 of the entity 105 having four multiplication units 110. The matrix blocks are extracted from the memory as burst pulses and loaded into the second inner buffer circuits 120. The three second inner buffer circuits 120 are simultaneously Fig. 8A and Figure 8B The data burst pulse is filled as shown in .
[0076] Fig. 9A and Fig. 9B Two different configurations of the adaptive matrix multiplier are shown. Fig. 9A In the configuration shown (which may be referred to as "Option One"), the N dimension is the inner ring and Fig. 9B In the configuration shown (which may be referred to as "option two"), the K dimension is the inner loop. The former (option one) may be more efficient when the size of the computation to be performed is large in the N dimension, while the latter (option two) may be more efficient when the size of the computation to be performed is large in the K dimension. In some embodiments, reference is made to Fig. 9C (It supports both option one and option two), multiplexer 905 is used to select between the two configurations. Each multiplexer may have three ports, including common port 910. Fig. 9C The three multiplexers shown include: a multiplication circuit input multiplexer having a common port 910 connected to the input of the multiplication circuit; a multiplication circuit output multiplexer having a common port 910 connected to the output of the multiplication circuit; and a second inner buffer circuit output multiplexer having a common port 910 connected to the output of the second inner buffer circuit.
[0077] In a first state of implementation option two, the multiplication circuit input multiplexer is configured to route data from the second inner buffer circuit output multiplexer to the first input of the multiplication circuit, the multiplication circuit output multiplexer is configured to route data from the output of the multiplication circuit to the external buffer circuit, and the second inner buffer circuit output multiplexer is configured to route data from the second inner buffer circuit to the multiplication circuit input multiplexer.
[0078] In a second state of implementation option one, the multiplication circuit input multiplexer is configured to route data from the external buffer circuit to the first input of the multiplication circuit, the multiplication circuit output multiplexer is configured to route data from the output of the multiplication circuit to the second internal buffer circuit, and the second internal buffer circuit output multiplexer is configured to route data from the second internal buffer circuit to the memory storage circuit.
[0079] The selection of one of the two configurations may be made at design time (e.g., may be hardwired into an integrated circuit fabricated with an adaptive matrix multiplier), or may be made at run time (using a multiplexer, such as Fig. 9C The multiplexer may be controlled by the host (via register module 180) or may be automatically controlled at run time by dedicated control circuitry within the adaptive matrix multiplier.
[0080] In some embodiments, the adaptive matrix multiplier or some parts thereof are implemented in a processing circuit. The term "processing circuit" is used herein to mean any combination of hardware, firmware, and software for processing data or digital signals. Processing circuit hardware may include, for example, an application specific integrated circuit (ASIC), a general or dedicated central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), and a programmable logic device (such as a field programmable gate array (FPGA)). In the processing circuit used herein, each function is performed by hardware configured (i.e., hardwired) to perform the function, or by more general hardware (such as a CPU) configured to execute instructions stored in a non-temporary storage medium. The processing circuit may be fabricated on a single printed circuit board (PCB) or distributed on several interconnected PCBs. The processing circuit may include other processing circuits; for example, the processing circuit may include two processing circuits interconnected on a PCB: an FPGA and a CPU.
[0081] As used herein, a "portion" of an item refers to all or less than all of the item. Thus, a portion of a matrix may be the entire matrix, or may be a portion of a matrix that does not include all of the elements of the matrix. A rectangular or square portion of a matrix is itself a matrix. As used herein, the "matrix product" of two matrices A and B (which may be written as A×B) is the product of the elements ∑ p A(i,p)*B(p,j); as a special case, "matrix product" includes the product of a matrix and a vector, and the dot product of two vectors. The "matrix multiplication" described in this article refers to the calculation of the matrix product.
[0082] In some embodiments, the general purpose multiplier may be configured to support the following matrix multiplications via register settings:
[0083] (C:=A×B) (where A and B are matrices): matrix-matrix multiplication;
[0084] (C:=TA×B), (C:=A×TB) and (C:=TA×TB) (where A and B are matrices and the “T” in front of the matrix indicates a transpose operation): transposed matrix multiplication; and
[0085] (Y:=A×X): Matrix-vector multiplication which is a special case of matrix-matrix multiplication.
[0086] The general multiplier may also be configured to adaptively select different multiplication options based on the selected multiplication option by abstract buffers X, Y, and O representing different A / B / C buffers. Memory burst transactions may be maximized by multiple units, multiple entities running in parallel.
[0087] It should be understood that although the terms "first", "second", "third", etc. may be used herein to describe various elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or section from another element, component, region, layer or section. Therefore, without departing from the spirit and scope of some embodiments, the first element, first component, first region, first layer or first section discussed herein may be referred to as the second element, second component, second region, second layer or second section.
[0088] The terms used herein are only used to illustrate specific embodiments and are not intended to limit some embodiments. The terms "substantially", "about" and similar terms used herein are used as approximate terms rather than as degree terms, and are intended to take into account the inherent deviations of the measured values or calculated values that will be recognized by ordinary technicians in the art. The term "major component" used herein refers to a component present in the composition or product in an amount greater than the amount of any other single component in the composition, polymer or product. In contrast, the term "primary component" refers to a component that constitutes at least 50% or more than 50% by weight of a composition, polymer or product. The term "major portion" used herein means at least half of these items when applied to multiple items.
[0089] Unless the context clearly indicates otherwise, the singular forms "a and an" used herein are intended to include the plural forms as well. It should be further understood that the terms "comprises and / or comprising" used in this specification indicate the presence of stated features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. The terms "and / or" used herein include any and all combinations of one or more of the associated listed items. When located before a series of elements, expressions such as "at least one of..." modify the entire series of elements rather than modifying individual elements in the series. In addition, "may" used when describing an embodiment refers to "one or more embodiments of the present disclosure." In addition, the term "exemplary" is intended to refer to an example or illustration. As used herein, the terms "use," "using," and "used" may be considered synonymous with the terms "utilize," "utilizing," and "utilized," respectively.
[0090] It should be understood that when an element or layer is referred to as being "on," "connected to," "coupled to," or "adjacent to" another element or layer, the element or layer may be directly on, directly connected to, directly coupled to, or directly adjacent to the other element or layer, or one or more intervening elements or layers may be present. In contrast, when an element or layer is referred to as being "directly on," "directly connected to," "directly coupled to," or "immediately adjacent to" another element or layer, there are no intervening elements or layers.
[0091] Any numerical range described herein is intended to include all sub-ranges of the same numerical precision within the described range. For example, the range of "1.0 to 10.0" is intended to include all sub-ranges between the described minimum value of 1.0 and the described maximum value of 10.0 (and including the described minimum value of 1.0 and the described maximum value of 10.0), that is, the minimum value is equal to or greater than 1.0 and the maximum value is equal to or less than 10.0, such as 2.4 to 7.6. Any maximum numerical limit described herein is intended to include all lower numerical limits contained therein, and any minimum numerical limit described in this specification is intended to include all higher numerical limits contained therein.
[0092] Although exemplary embodiments of adaptive matrix multipliers have been specifically described and illustrated herein, many modifications and variations will be apparent to those skilled in the art. Therefore, it should be understood that adaptive matrix multipliers constructed according to the principles of the present disclosure may be implemented in ways other than those specifically described herein. The present disclosure is also defined in the appended claims and their equivalents.
Claims
1. A system of adaptive matrix multipliers, include: First multiplication unit; Second multiplication unit; memory loading circuit; as well as External buffer circuit, The first multiplication unit includes a first internal buffer circuit and a second internal buffer circuit, The second multiplication unit includes a first internal buffer circuit and a second internal buffer circuit, The memory loading circuit is connected to the first internal buffer circuit of the first multiplication unit and the first internal buffer circuit of the second multiplication unit, The memory loading circuit is configured to load data from the memory in a single burst of a burst memory access mode into: the first internal buffer circuit of the first multiplication unit; and The first internal buffer circuit of the second multiplication unit.
2. The system according to claim 1, in: The first multiplication unit comprises an adder circuit, and The first multiplication unit is configured to: Form the first product of: a first element of the first internal buffer circuit of the first multiplication unit, and a first element of the second inner buffer circuit of the first multiplication unit; transmitting the first product to the adder circuit; Form the second product of: a second element of the first internal buffer circuit of the first multiplication unit, and a second element of the second inner buffer circuit of the first multiplication unit; and The second product is transmitted to the adder circuit.
3. The system according to claim 2, wherein the first multiplication unit include: a first multiplication circuit configured to form the first product; as well as A second multiplication circuit is configured to form the second product.
4. The system of claim 2, wherein the adder circuit is configured to: forming a first sum of the first product and the second product, and The first sum is sent to the external buffer circuit.
5. The system according to claim 4, in: The second multiplication unit includes an adder circuit, and The second multiplication unit is configured to: Form the third product of: The first element of the first internal buffer circuit of the second multiplication unit is a first element of the second inner buffer circuit of the second multiplication unit; transmitting the third product to the adder circuit of the second multiplication unit; Form the fourth product of: a second element of the first internal buffer circuit of the second multiplication unit, and a second element of the second inner buffer circuit of the second multiplication unit; as well as The fourth product is transmitted to the adder circuit of the second multiplication unit.
6. The system according to claim 5, in: The adder circuit of the second multiplication unit is configured to: forming a second sum of the third product and the fourth product, and sending the second sum to the external buffer circuit, and The external buffer circuit is configured to add the first sum to the second sum.
7. The system according to claim 1, in: The first multiplication unit comprises an adder circuit; The first internal buffer circuit of the first multiplication unit includes a first mini-buffer and a second mini-buffer; The second internal buffer circuit of the first multiplication unit includes a first mini-buffer and a second mini-buffer; The first multiplication unit is configured to: Form the first product of: a first element of the first mini-buffer of the first inner buffer circuit of the first multiplication unit, and a first element of the first mini-buffer of the second inner buffer circuit of the first multiplication unit; transmitting the first product to the adder circuit; Form the second product of: a first element of the second mini-buffer of the first inner buffer circuit of the first multiplication unit, and a first element of the second mini-buffer of the second inner buffer circuit of the first multiplication unit; as well as The second product is transmitted to the adder circuit.
8. The system according to claim 7, in: The first multiplication unit is further configured to: Form the third product of: a second element of the first mini-buffer of the first inner buffer circuit of the first multiplication unit, and the first element of the first mini-buffer of the second inner buffer circuit of the first multiplication unit; transmitting the third product to the adder circuit; Form the fourth product of: a second element of the second mini-buffer of the first inner buffer circuit of the first multiplication unit, and the first element of the second mini-buffer of the second inner buffer circuit of the first multiplication unit; as well as The fourth product is transmitted to the adder circuit.
9. The system according to claim 8, in: the adder circuit being configured to form a first sum of the first product and the second product and to send the first sum to the external buffer circuit; as well as The external buffer circuit is configured to store the first sum in a first element of the external buffer circuit.
10. The system according to claim 9, in: The adder circuit is further configured to form a second sum of the third product and the fourth product and send the second sum to the external buffer circuit; and The external buffer circuit is configured to store the second sum in a second element of the external buffer circuit.
11. The system according to claim 1, in: The first multiplication unit comprises an adder circuit; and The first multiplication unit is configured to: Form the first product of: a first element of the first internal buffer circuit of the first multiplication unit, and a first element of an external buffer circuit of the first multiplication unit; transmitting the first product to the adder circuit; Form the second product of: a second element of the first internal buffer circuit of the first multiplication unit, and a second element of the external buffer circuit of the first multiplication unit; and The second product is transmitted to the adder circuit.
12. The system of claim 11, wherein the adder circuit is configured to: forming a first sum of the first product and the second product; and The first sum is stored in a first element of the second inner buffer circuit of the first multiplication unit.
13. The system of claim 11, wherein the first multiplication unit include: a first multiplication circuit configured to form the first product; as well as A second multiplication circuit is configured to form the second product.
14. The system according to claim 1, comprising a first entity and a second entity, The first entity includes the first multiplication unit and the second multiplication unit.
15. The system of claim 14, wherein the memory loading circuit is configured to receive a read request from the first entity and to receive a read request from the second entity and to arbitrate the read requests.
16. The system of claim 1, wherein the memory loading circuit is configured to load the portion of the matrix from the memory into the second internal buffer circuit of the first multiplication unit in a manner that transposes elements of the portion of the matrix.
17. A system of adaptive matrix multipliers, include: First multiplication unit; Second multiplication unit; memory loading circuit; Memory storage circuit; as well as External buffer circuit, The first multiplication unit comprises: Multiplication circuit; a first internal buffer circuit; a second internal buffer circuit; a multiplication circuit input multiplexer having a common port connected to an input of said multiplication circuit; a multiplication circuit output multiplexer having a common port connected to the output of the multiplication circuit; and a second inner buffer circuit output multiplexer having a common port connected to the output of the second inner buffer circuit, The multiplication circuit output multiplexer is connected between the multiplication circuit and the second inner buffer circuit.
18. The system of claim 17, wherein in a first state of the first multiplication unit, the multiplication circuit input multiplexer configured to route data from the second internal buffer circuit output multiplexer to a first input of the multiplication circuit; the multiplication circuit output multiplexer configured to route data from the output of the multiplication circuit to the external buffer circuit; as well as The second inner buffer circuit output multiplexer is configured to route data from the second inner buffer circuit to the multiplication circuit input multiplexer.
19. The system of claim 18, wherein in a second state of the first multiplication unit, the multiplication circuit input multiplexer configured to route data from the external buffer circuit to the first input of the multiplication circuit; the multiplication circuit output multiplexer configured to route data from the output of the multiplication circuit to the second internal buffer circuit; as well as The second inner buffer circuit output multiplexer is configured to route data from the second inner buffer circuit to the memory storage circuit.
20. A system of adaptive matrix multipliers, include: First multiplication unit; Second multiplication unit; means for loading data from memory; as well as External buffer circuit, The first multiplication unit includes a first internal buffer circuit and a second internal buffer circuit, The second multiplication unit includes a first internal buffer circuit and a second internal buffer circuit, said means for loading said data from said memory is connected to said first inner buffer circuit of said first multiplication unit and to said first inner buffer circuit of said second multiplication unit, The means for loading the data from the memory is configured to load the data from the memory in a single burst of a burst memory access mode into: the first internal buffer circuit of the first multiplication unit; and The first internal buffer circuit of the second multiplication unit.
