A multiply-accumulate array with output dwell and related devices

Abstract

This invention proposes an output-resident multiply-accumulate array and related devices. The input buffer reads the input matrix X from memory and sends the target input data to the multiply-accumulate unit at time T. The weight buffer reads the weight matrix W from memory and sends the target weight data to the multiply-accumulate unit at time T. The multiply-accumulate unit multiplies the acquired target input data and target weight data one by one and accumulates the sum of the products at time T. If time T is the last time in the current accumulation interval, it also sends the current accumulated value to the addition unit connected to it. When the addition unit obtains the accumulated value, it writes the sum of the historical value and the accumulated value into the buffer. This reduces the number of times the input matrix and weight matrix are read from memory, reduces the memory read bandwidth occupation, allows the acquired accumulated value to reside, thereby obtaining the last accumulated value of the current accumulation interval, reuses the time dimension to improve parallelism, and improves operating efficiency.

Description

Technical Field

[0001] This invention relates to the field of chips, and more specifically, to an output-resident multiply-accumulate array and related devices. Background Technology

[0002] Multiply-accumulate arrays are the core computing units of neural network processors, used to accelerate various convolution and matrix multiplication operators. With the development of deep learning models, especially large models, multiply-accumulate arrays have evolved from early simple single-instruction multiple-data-flow structures to highly parallel, dataflow-optimized complex array architectures.

[0003] In terms of parallelism, convolution operations commonly employ a strategy of parallel computation across multiple output channels, multiple input channels, and multiple output points. For matrix multiplication, a strategy of parallel computation across multiple left matrix rows, multiple right matrix rows, and multiple result matrix columns can be used. In fact, convolution can be transformed into matrix multiplication. The output point dimension, input channel dimension, and output channel dimension of a convolution operation correspond to the left matrix row dimension, right matrix row dimension, and result matrix column dimension of a matrix multiplication operation, respectively. The right matrix in a matrix multiplication operation is typically the weights, and the left matrix is ​​typically the input. This correspondence also applies to the weights and inputs in a convolution operation. Therefore, multiply-accumulate arrays can uniformly utilize the parallelism of each dimension to accelerate computation.

[0004] Current multiply-accumulate arrays often require multiple reads of input data and / or weights from off-chip memory. Large-scale system-on-a-chip (SoC) chips contain numerous intellectual property (IP) cores, and the read bandwidth of dynamic random access memory (DDR) is already limited. Furthermore, the amount of weight and input data in the model increases dramatically with the growth of large models. Multiple reads of weights or inputs from DDR further exacerbate the bandwidth shortage, leading to increased power consumption and reduced overall system performance. Overcoming these problems has become a significant challenge for those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide an output-resident multiply-accumulate array and related devices to improve the above-mentioned problems.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows:

[0007] In a first aspect, embodiments of the present invention provide an output-resident multiply-accumulate array, comprising: an input buffer, a weight buffer, a multiply-accumulate matrix, and an intermediate result buffer, wherein the multiply-accumulate matrix comprises I×J multiply-accumulate units and I×J addition units, and the intermediate result buffer is provided with at least I×J buffers;

[0008] The input buffer is used to read the input matrix X from the memory;

[0009] The weight cache is used to read the weight matrix W from the memory;

[0010] The input buffer is used to send the corresponding target input data to each multiply-accumulate unit at time T, and the weight buffer is used to send the corresponding target weight data to each multiply-accumulate unit at time T, wherein the target input data belongs to the input matrix X, and the target weight data belongs to the weight matrix;

[0011] The multiply-accumulate unit is used to multiply the target input data and target weight data obtained at time T one by one, and accumulate the sum of the products at time T. If time T is the last time in the current accumulation interval, it will also send the current accumulated value to the addition unit connected to it.

[0012] The addition unit is used to read the historical value from its corresponding buffer when the accumulated value is obtained, and write the sum of the historical value and the accumulated value into the buffer.

[0013] Optionally, the multiply-accumulate unit includes P multipliers, adders, and accumulators. One input of each multiplier is connected to the input buffer, and the other input of each multiplier is connected to the weight buffer. The output of each multiplier is connected to the input of each adder, and the output of each adder is connected to the input of the accumulator. The output of the accumulator serves as the output of the multiply-accumulate unit.

[0014] Optionally, the input matrix X includes M rows and N columns of input data, and the weight matrix W includes N rows and K columns, where M = I × m, N = P × n, K = J × k, and m, n, and k are integers greater than or equal to 1.

[0015] Time T = tm×k×n + tk×n + tn, tm = 0, 1, ..., m-1, tn = 0, 1, ..., n-1, tk = 0, 1, ..., k-1;

[0016] MAC ij The target input data at time T is the input matrix X(I×tm+i,P×tn+0) to X(I×tm+i,P×tn+P-1).

[0017] MAC ij The target weight data at time T is from weight matrix W(P×tn+0, J×tk+j) to weight matrix W(P×tn+P-1, J×tk+j).

[0018] Among them, MAC ij This represents the multiplication and accumulation unit in row i and column j, where i = 0, 1, ..., I-1 and j = 0, 1, ..., J-1.

[0019] Optionally, MAC ij-p The input data acquired at time T is the input matrix X (I×tm+i,P×tn+p), and the weight data acquired at time T is the weight matrix W (P×tn+p,J×tk+j).

[0020] MAC ij-p This represents the p-th multiplier in the multiply-accumulate unit of row i and column j, where p = 0, 1, ..., P-1.

[0021] Optionally, the accumulator includes an adder and a register, with one input of the adder connected to the output of the adder and the other input of the adder connected to the register;

[0022] The addition device is used to add the obtained product sum to the accumulated value in the register, and write the resulting sum into the register;

[0023] The accumulator is used to clear the register after sending the current accumulated value to the addition unit connected to it.

[0024] Optionally, at the initial time after obtaining the input matrix X and the weight matrix W, the values ​​of tn, tk, and tm are all 0. Then, at each time change, tn = tn + 1 is set until tn = n - 1. At this time, tk = tk + 1 is set, and tn = 0 is set. This process is repeated at each time change until tk = k - 1 is set. At this time, tm = tm + 1 is set, and tn = 0 and tk = 0 are set. This process is repeated at each time change until tm = m - 1 is set.

[0025] Secondly, embodiments of the present invention provide a neural network processor, the neural network processor including the above-described output-resident multiply-accumulate array.

[0026] Thirdly, embodiments of the present invention provide a system-on-a-chip, the system-on-a-chip including the neural network processor described above.

[0027] Fourthly, embodiments of the present invention provide an electronic device, the electronic device including the above-described system-on-a-chip.

[0028] Compared to existing technologies, the present invention provides an output-resident multiply-accumulate array and related devices. The input buffer reads the input matrix X from memory and sends the corresponding target input data to each multiply-accumulate unit at time T. The weight buffer reads the weight matrix W from memory and sends the corresponding target weight data to each multiply-accumulate unit at time T. The multiply-accumulate unit multiplies the target input data and target weight data obtained at time T one by one and accumulates the sum of the products at time T. If time T is the last time in the current accumulation interval, it also sends the current accumulated value to the addition unit connected to it. When the addition unit obtains the accumulated value, it reads the historical value from its corresponding buffer and writes the sum of the historical value and the accumulated value into the buffer. This reduces the number of times the input matrix and weight matrix are read from memory, reduces the memory read bandwidth usage, allows the obtained accumulated value to reside, thereby obtaining the last accumulated value in the current accumulation interval, reuses the time dimension to improve parallelism, and improves operating efficiency.

[0029] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of the output-resident multiply-accumulate array provided in an embodiment of the present invention.

[0032] Figure 2 This is a schematic diagram of the connection relationship of the multiply-accumulate unit provided in an embodiment of the present invention.

[0033] Figure 3 This is a connection diagram of the accumulator provided in an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0035] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0036] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0037] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0038] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of this invention is usually placed when in use. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0039] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0040] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0041] To alleviate the problem of limited read bandwidth resources in Dynamic Random Access Memory (DDR), this invention provides an output-resident multiply-accumulate array. By using output-resident access, the number of times data is read from DDR is reduced, thereby decreasing the consumption of DDR read bandwidth and ensuring device performance. For details, please refer to... Figure 1 , Figure 1 This is a schematic diagram of the output-resident multiply-accumulate array provided in an embodiment of the present invention.

[0042] The output-resident multiply-accumulate array includes: an input buffer, a weight buffer, a multiply-accumulate matrix, and an intermediate result buffer. The multiply-accumulate matrix includes I×J multiply-accumulate units and I×J add units, where I represents the number of rows of the multiply-accumulate units in the multiply-accumulate matrix, and J represents the number of columns of the multiply-accumulate units in the multiply-accumulate matrix. The values ​​of I and J can be, but are not limited to, 32, 16, and 64, etc. The intermediate result buffer is equipped with at least I×J buffers; the buffers can be RAM or can be constructed from registers.

[0043] Each multiply-accumulate unit is connected to the input buffer and the weight buffer. The multiply-accumulate unit is also connected to the corresponding addition unit, and each addition unit is connected to the intermediate result buffer.

[0044] The input buffer is used to read the input matrix X from the memory. The memory can be, but is not limited to, dynamic random access memory (DDR). The input matrix X consists of M rows and N columns of input data, where M = I × m and N = P × n, where m and n are integers greater than or equal to 1, and P is the number of input data that can be processed in a single multiply-accumulate unit, i.e., the number of multipliers, and its value can be, but is not limited to, 4, 8, 16, 32, etc.

[0045] The weight cache is used to read the weight matrix W from memory. The weight matrix W consists of N rows and K columns, where K = J × k, and k is an integer greater than or equal to 1.

[0046] The input buffer is used to send the corresponding target input data to each multiply-accumulate unit at time T, and the weight buffer is used to send the corresponding target weight data to each multiply-accumulate unit at time T. The target input data belongs to a segment or all of a row in the input matrix X, and the target weight data belongs to a segment or all of a column in the weight matrix X.

[0047] It should be understood that the input data in the input matrix X and the weight data in the weight matrix W are sent to each multiply-accumulate unit at multiple time steps. It is not necessary to read the input data or weight data from the memory at every time step, which can reduce the number of times the input matrix and weight matrix are read from the memory and reduce the memory read bandwidth usage.

[0048] The multiply-accumulate unit is used to multiply the target input data and target weight data obtained at time T one by one, and accumulate the sum of the products at time T (i.e. the sum of the products obtained by the multiply-accumulate unit at time T). If time T is the last time in the current accumulation interval, it will also send the current accumulated value to the addition unit connected to it.

[0049] The length of the current accumulation interval is n, meaning that every n time intervals correspond to one accumulation interval. Whether time T is the last time of the current accumulation interval can be determined by counting or by configuring an external register.

[0050] It should be understood that the multiply-accumulate unit can retain the accumulated value obtained at the middle time of the current accumulation interval, thereby obtaining the last accumulated value of the current accumulation interval, realizing parallel processing of data corresponding to n time points, and improving running efficiency.

[0051] After sending the current accumulated value to the addition unit connected to it, the accumulated value in the multiply-accumulate unit can be cleared to zero.

[0052] The addition unit is used to read the historical value from its corresponding buffer when the accumulated value is obtained, and write the sum of the historical value and the accumulated value into the buffer as the new historical value.

[0053] It should be noted that after the calculation is completed, the values ​​in each register in the intermediate result cache are the final result of multiplying the input matrix X and the weight matrix W.

[0054] In the output-resident multiply-accumulate array provided in this embodiment of the invention, the number of times the input matrix and weight matrix are read from the memory is reduced, the memory read bandwidth is reduced, the acquired accumulated value can be resident, thereby obtaining the last accumulated value of the current accumulation interval, reusing the time dimension to improve parallelism and improve operating efficiency.

[0055] In one alternative implementation, the values ​​of m and n can be determined by combining the current remaining space of the input cache and the current remaining space of the weight cache, so as to maximize the reading of input data and weight data from memory, thereby reducing the number of data reads and reducing the occupation of off-chip memory access bandwidth.

[0056] Please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating the connection relationship of a multiply-accumulate unit provided in an embodiment of the present invention. The multiply-accumulate unit includes P multipliers, adders, and accumulators. One input of each multiplier is connected to an input buffer, and the other input is connected to a weight buffer. The output of each multiplier is connected to the input of an adder, and the output of each adder is connected to the input of an accumulator. The output of the accumulator serves as the output of the multiply-accumulate unit and is connected to the corresponding adder unit.

[0057] Where P is the number of input data that can be processed in a single multiply-accumulate unit. Figure 2 For ease of demonstration, only 4 multipliers are set, but this is not a limitation. More or fewer multipliers can be set in the multiply-accumulate unit.

[0058] The accumulator in the multiply-accumulate unit can accumulate the sum of products obtained in the current accumulation interval, and the output dwell is achieved in the accumulator.

[0059] Based on the foregoing, regarding the data received by the multiply-accumulate unit at various times, this embodiment of the invention also provides an optional implementation method, please refer to the following text.

[0060] The input matrix X consists of M rows and N columns of input data, and the weight matrix W consists of N rows and K columns, where M = I × m, N = P × n, and K = J × k, where m, n, and k are integers greater than or equal to 1. When the input buffer reads less than M rows and N columns of input data from memory, or when the above integer multiple relationship does not hold, the data can be padded with zeros to obtain M rows and N columns of data, thus making the above integer expression true.

[0061] Time T = tm×k×n + tk×n + tn, tm = 0, 1, ..., m-1, tn = 0, 1, ..., n-1, tk = 0, 1, ..., k-1.

[0062] MAC ij The target input data at time T is the input matrix X(I×tm+i,P×tn+0) to X(I×tm+i,P×tn+P-1).

[0063] MAC ij The target weight data at time T is from weight matrix W(P×tn+0,J×tk+j) to weight matrix W(P×tn+P-1,J×tk+j).

[0064] Among them, MAC ijThe input matrix X(I×tm+i, P×tn+0) represents the input data in the I×tm+i row and P×tn column of the input matrix X, and X(I×tm+i, P×tn+P-1) represents the input data in the I×tm+i row and P×tn+P-1 column of the input matrix X. The weight matrix W(P×tn+0, J×tk+j) represents the weight data in the P×tn row and J×tk+j column of the weight matrix W, and W(P×tn+P-1, J×tk+j) represents the weight data in the P×tn+P-1 row and J×tk+j column of the weight matrix W.

[0065] Building upon the preceding text, this embodiment of the invention also provides an optional implementation method for the data received by the multiplier in the multiply-accumulate unit at various times, as detailed below.

[0066] MAC ij-p The input data acquired at time T is the input matrix X (I×tm+i,P×tn+p), and the weight data acquired at time T is the weight matrix W (P×tn+p,J×tk+j).

[0067] Wherein, the input matrix X (I×tm+i, P×tn+p) represents the input data in the I×tm+i row and P×tn+p column of the input matrix X, and the weight matrix W (P×tn+p, J×tk+j) represents the weight data in the P×tn+p row and J×tk+j column of the weight matrix W.

[0068] From the above formula, we can see that MAC ij-p The input data acquired at time T is affected by tm and tn, but is independent of tk. That is, when the value of tk changes, the data corresponding to the input matrix X (I×tm+i, P×tn+p) will be reused. Similarly, MAC ij-p The weight data acquired at time T is affected by tk and tn, but is independent of tm. That is, when the value of tm changes, the data corresponding to the weight matrix W (P×tn+p, J×tk+j) will be reused.

[0069] MAC ij-p This represents the p-th multiplier in the multiply-accumulate unit of row i and column j, where p = 0, 1, ..., P-1.

[0070] MAC ij-p The adder is used to obtain the product of the input data and the weight data acquired at time T. ij The sum of the products obtained by the multiplier at time T, i.e., MAC ij The sum of products at time T.

[0071] Based on the foregoing, regarding the connection relationship of the accumulators, this embodiment of the invention provides an optional implementation method, please refer to... Figure 3 , Figure 3 This is a connection diagram of an accumulator provided in an embodiment of the present invention. The accumulator includes an adding device and a register. One input terminal of the adding device is connected to the output terminal of the adder, and the other input terminal of the adding device is connected to the register. The register is also connected to a corresponding adding unit.

[0072] The addition device is used to add the obtained product sum to the accumulated value in the register, and write the resulting sum into the register as the new accumulated value.

[0073] The accumulator is used to clear the register after sending the current accumulated value to the adder unit connected to it.

[0074] Optionally, at the initial time after obtaining the input matrix X and the weight matrix W, the values ​​of tn, tk, and tm are all 0. Then, at each time step, tn = tn + 1 (incrementing sequentially) until tn = n - 1. At this point, tk = tk + 1, and tn = 0. This process of setting tn = tn + 1 at each time step is repeated until tk = k - 1. At this point, tm = tm + 1, and tn = 0, tk = 0. This process of setting tn = tn + 1 at each time step is repeated until tm = m - 1, thus ending the loop and completing the multiplication of the M x N left matrix (input matrix) and the N x K right matrix (weight matrix). tn, tk, and tm adjust the parallelism of the left matrix columns, right matrix columns, and left matrix rows respectively from the time dimension, and the loop unfolds sequentially from the time dimension.

[0075] In this embodiment of the invention, each multiply-accumulate unit contains P (e.g., 16) multipliers. These P multipliers perform multiply-accumulate operations in parallel along the input channel dimension. Furthermore, configuration registers can control how many times intermediate results are accumulated before being sent from the multiply-accumulate unit, indirectly adjusting the parallelism along the input channel dimension in the time dimension. Additionally, configuration registers can control the order in which weights and inputs are sent to the multiply-accumulate unit from the weight buffer and input buffer, thereby controlling the number of output points calculated each time. This also indirectly adjusts the parallelism along the output point dimension in the time dimension.

[0076] The following analysis uses matrix operations as an example to illustrate how this structure reduces the number of accesses to off-chip memory through register configuration. Assume the register controls the intermediate result to be accumulated n times before being sent from the multiply-accumulate unit. Then, the parallelism of the row dimension of the right matrix (weight matrix) indirectly becomes P*n, denoted as N. Assume the register controls the multiplication of the M-row, N-column left matrix (input matrix) element with the corresponding right matrix element N-row, K-column before multiplying the next M-row, N-column left matrix element with its corresponding right matrix element. This controls the parallelism of the left matrix's row dimension to be M in terms of time. Let the number of rows in the left matrix be XR, the number of columns be XC, the number of rows in the right matrix be XC, and the number of columns be WC.

[0077] Taking a scenario where XC is much larger than XR and WC is large as an example, by setting a smaller N value and a larger M value, such as M being XR, the data of the first N columns of all rows of the left matrix can be placed in the input buffer. Then, the right matrix is ​​retrieved from off-chip storage in granularity of N rows and K columns at a time and placed in the weight buffer. After the elements of the first N columns of all rows of the left matrix are multiplied and summed with the elements of all N rows and all columns of the right matrix, they are placed in the intermediate buffer. At this point, the elements of the left matrix in the input buffer and the elements of the right matrix in the weight buffer will no longer be used and can be discarded. When using the elements of the first N columns of all rows of the left matrix and the elements of the first N rows and all columns of the right matrix, the elements of the second N columns of all rows of the left matrix and the second N rows and all columns of the right matrix can be read from off-chip storage simultaneously and stored in the input buffer and the activation buffer respectively. This process is repeated until all results are calculated. It can be seen that every N columns of all rows of the left matrix are read from off-chip storage only once, that is, the left matrix is ​​read from off-chip storage only once. Each column of the right matrix is ​​read from off-chip storage only once for every N rows. Thus, compared to traditional intermediate result-resident structures, this structure reads the right matrix (weights) and the left matrix (input) from off-chip storage only once, significantly reducing the number of times weights are read from off-chip storage. Compared to input-resident and activation-resident structures, this structure writes data to the intermediate result cache only once every N multiplications and additions, and reads the intermediate result cache only once every N multiplications and additions during subsequent additions. Therefore, the number of times this structure accesses the intermediate result cache or moves intermediate results is only 1 / 2*N of that of the input-resident or activation-resident structures.

[0078] As can be seen from the above analysis, for scenarios with different model sizes and input sizes, the number of times to access off-chip storage can be reduced by configurable intermediate result accumulation and the number of output points calculated each time, thereby reducing the occupation of off-chip storage access bandwidth resources.

[0079] This invention also provides a neural network processor, which includes the above-described output-resident multiply-accumulate array.

[0080] This invention also provides a system-on-a-chip, which includes the neural network processor described above.

[0081] This invention also provides an electronic device, which includes the above-described system-on-a-chip.

[0082] In summary, the present invention provides an output-resident multiply-accumulate array and related devices. The input buffer reads the input matrix X from memory and sends the corresponding target input data to each multiply-accumulate unit at time T. The weight buffer reads the weight matrix W from memory and sends the corresponding target weight data to each multiply-accumulate unit at time T. The multiply-accumulate unit multiplies the target input data and target weight data obtained at time T one by one and accumulates the sum of the products at time T. If time T is the last time in the current accumulation interval, it also sends the current accumulated value to the addition unit connected to it. When the addition unit obtains the accumulated value, it reads the historical value from its corresponding buffer and writes the sum of the historical value and the accumulated value into the buffer. This reduces the number of times the input matrix and weight matrix are read from memory, reduces the memory read bandwidth usage, allows the obtained accumulated value to reside, thereby obtaining the last accumulated value of the current accumulation interval, reuses the time dimension to improve parallelism, and improves operating efficiency.

[0083] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0084] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. An output-resident multiply-accumulate array, characterized in that, include: The system includes an input cache, a weight cache, a multiply-accumulate matrix, and an intermediate result cache. The multiply-accumulate matrix comprises I×J multiply-accumulate units and I×J addition units. The intermediate result cache is provided with at least I×J buffers. The input buffer is used to read the input matrix X from the memory; The weight cache is used to read the weight matrix W from the memory; The input buffer is used to send the corresponding target input data to each multiply-accumulate unit at time T, and the weight buffer is used to send the corresponding target weight data to each multiply-accumulate unit at time T, wherein the target input data belongs to the input matrix X, and the target weight data belongs to the weight matrix; The multiply-accumulate unit is used to multiply the target input data and target weight data obtained at time T one by one, and accumulate the sum of the products at time T. If time T is the last time in the current accumulation interval, it will also send the current accumulated value to the addition unit connected to it. The addition unit is used to read the historical value from its corresponding buffer when the accumulated value is obtained, and write the sum of the historical value and the accumulated value into the buffer.

2. The output-resident multiply-accumulate array as described in claim 1, characterized in that, The multiply-accumulate unit includes P multipliers, adders, and accumulators. One input of each multiplier is connected to the input buffer, and the other input of each multiplier is connected to the weight buffer. The output of each multiplier is connected to the input of each adder, and the output of each adder is connected to the input of each accumulator. The output of each accumulator serves as the output of the multiply-accumulate unit.

3. The output-resident multiply-accumulate array as described in claim 2, characterized in that, The input matrix X comprises M rows and N columns of input data, and the weight matrix W comprises N rows and K columns, where M = I × m, N = P × n, K = J × k, and m, n, and k are integers greater than or equal to 1. Time T = tm×k×n + tk×n + tn, tm = 0, 1, ..., m-1, tn = 0, 1, ..., n-1, tk = 0, 1, ..., k-1; MAC ij The target input data at time T is the input matrix X(I×tm+i,P×tn+0) to X(I×tm+i,P×tn+P-1). MAC ij The target weight data at time T is from weight matrix W(P×tn+0, J×tk+j) to weight matrix W(P×tn+P-1, J×tk+j). Among them, MAC ij This represents the multiplication and accumulation unit in row i and column j, where i = 0, 1, ..., I-1 and j = 0, 1, ..., J-1.

4. The output-resident multiply-accumulate array as described in claim 3, characterized in that, MAC ij-p The input data acquired at time T is the input matrix X (I×tm+i,P×tn+p), and the weight data acquired at time T is the weight matrix W (P×tn+p,J×tk+j). MAC ij-p This represents the p-th multiplier in the multiply-accumulate unit of row i and column j, where p = 0, 1, ..., P-1.

5. The output-resident multiply-accumulate array as described in claim 2 or 4, characterized in that, The accumulator includes an adder and a register. One input of the adder is connected to the output of the adder, and the other input of the adder is connected to the register. The addition device is used to add the obtained product sum to the accumulated value in the register, and write the resulting sum into the register; The accumulator is used to clear the register after sending the current accumulated value to the addition unit connected to it.

6. The output-resident multiply-accumulate array as described in claim 3, characterized in that, At the initial time after obtaining the input matrix X and the weight matrix W, the values ​​of tn, tk, and tm are all 0. Then, at each time change, tn = tn + 1 is set until tn = n - 1. At this time, tk = tk + 1 is set, and tn = 0 is set. This process is repeated at each time change until tk = k - 1 is set. At this time, tm = tm + 1 is set, and tn = 0 and tk = 0 are set. This process is repeated at each time change until tm = m - 1 is set.

7. A neural network processor, characterized in that, The neural network processor includes a multiply-accumulate array with output residency as described in any one of claims 1-6.

8. A system-on-a-chip, characterized in that, The system-on-a-chip includes the neural network processor of claim 7.

9. An electronic device, characterized in that, The electronic device includes the system-on-a-chip as described in claim 8.

Images