A spatial domain based systolic array

Abstract

The application relates to the technical field of chip design, in particular to a spatial domain-based systolic array, which comprises N*M arithmetic logic units, a top storage unit and a bottom storage unit; each arithmetic logic unit comprises an adder, a multiplier and a pipeline timing unit; the output end of the multiplier is connected to the input end of the adder, and the input end of the multiplier is connected to the input end of the pipeline timing unit; the output end of the pipeline timing unit in the i-th row and j-th arithmetic logic unit is connected to the input end of the multiplier in the i-th row and (j+1)-th arithmetic logic unit; the output end of the adder in the i-th row and j-th arithmetic logic unit is connected to the input end of the adder in the (i+1)-th row and j-th arithmetic logic unit; the row storage unit of the i-th row is connected to the first arithmetic logic unit in the i-th row; the top storage unit is connected to the M arithmetic logic units in the first row; and the bottom storage unit is connected to the M arithmetic logic units in the N-th row, so that the storage space occupation is reduced.

Description

Technical Field

[0001] This invention relates to the field of chip design technology, and in particular to a spatial domain-based pulsating array. Background Technology

[0002] Matrix multiplication in processors is typically implemented in the time domain. This time-domain implementation uses a fixed arithmetic logic unit (ALU) and data path. Each time, a matrix of data A, B, and C is loaded from memory and fed into the arithmetic logic unit (ALU) for the operation D = dot(A, B) + C. After D is calculated, it is written back to memory and used as C for the next operation. This method can be considered as an accumulation of D in the time domain. Theoretically, the computational power expansion of the time-domain architecture is positively correlated with the number of ALUs and the clock frequency; increasing the number of ALUs can achieve stronger computational performance. However, this time-domain architecture requires sufficient storage space to continuously meet the matrix data supply for each cycle.

[0003] However, as the number of integrated arithmetic logic units on a single chip increases, the storage space on that chip does not increase accordingly. Insufficient storage space leads to a mismatch between memory access bandwidth, latency, and energy efficiency and the computing power requirements. In other words, the memory wall has become the core bottleneck limiting the continuous enhancement of single-chip computing power. Therefore, there is an urgent need for a pulsating array that can improve computing power. Summary of the Invention

[0004] To address the aforementioned technical problems, the present invention adopts the following technical solution: a spatial domain-based pulsating array, the array comprising: N×M rows of arithmetic logic units, each arithmetic logic unit including an adder, a multiplier, and a pipelined timing unit; the multiplier having a first input and a first output, the adder having a second input and a second output, and the pipelined timing unit having a data input and a data output; wherein, the first output is connected to a second input, and the first input is connected to the data input of the pipelined timing unit; wherein, N≥1, M≥2; wherein, the j-th arithmetic logic unit in the i-th row... The data output terminal of the element is connected to the first input terminal of the (j+1)th arithmetic logic unit in the i-th row; where 1≤i≤N, 0<j≤M-1; where the second output terminal of the j-th arithmetic logic unit in the i-th row is connected to the second input terminal of the j-th arithmetic logic unit in the (i+1)-th row; where each row also includes a row storage unit, the row storage unit of the i-th row is connected to the first input terminal of the first arithmetic logic unit in the i-th row; the top-level storage unit is connected to the second input terminals of the M arithmetic logic units in the first row array respectively; the bottom-level storage unit is connected to the second output terminals of the M arithmetic logic units in the N-th row array respectively.

[0005] The present invention has at least the following beneficial effects:

[0006] The essence of this invention is to transform the existing time-domain architecture into a spatial-domain architecture. By using a pulsating array, matrix data is piped along in a rhythmic fashion. During this flow, data is reused by all relevant computing units along the way, avoiding centralized access to memory. Simultaneously, input data is consumed as it flows between spatially arranged computing units, eliminating the need for temporary storage of original data; intermediate results are directly passed between adjacent units to participate in subsequent calculations, completely avoiding the need for global caching. This data-flow-driven computing model reduces storage space consumption and ultimately solves the computing power bottleneck caused by storage walls in traditional architectures. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 A schematic diagram of a spatial domain-based pulsating array provided for an embodiment of the present invention;

[0009] Figure 2 This is a schematic diagram of the internal structure of an arithmetic logic unit provided in an embodiment of the present invention;

[0010] Figure 3 A schematic diagram of the internal structure of an arithmetic logic unit excluding a dedicated storage unit, provided for an embodiment of the present invention;

[0011] Figure 4 A schematic diagram of the internal structure of an arithmetic logic unit including a dedicated storage unit, provided for an embodiment of the present invention;

[0012] Figure 5 A schematic diagram of a systolic array where each row of arithmetic logic units is located in a stream processor, as provided in an embodiment of the present invention;

[0013] Figure 6 A schematic diagram of a systolic array in which each arithmetic logic unit is located in a stream processor, as provided in an embodiment of the present invention;

[0014] Figure 7 This is a schematic diagram of an N-row, single-column cluster distribution provided in an embodiment of the present invention;

[0015] Figure 8 This is a schematic diagram of a cluster distribution with N rows and M columns and an arithmetic logic unit configured in the central controller, provided as an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Unless otherwise defined, all technical and scientific terms used in the embodiments of this invention have the same meaning as commonly understood by those skilled in the art.

[0018] Please see Figure 1 The diagram illustrates a spatial domain-based systolic array, which includes N×M rows of arithmetic logic units, a top-level storage unit, and a bottom-level storage unit.

[0019] For further information, please refer to [link / reference]. Figure 2 In an N×M row arithmetic logic unit (ALU), each ALU includes an adder, a multiplier, and a pipelined timing unit. The multiplier has a first input and a first output; the adder has a second input and a second output; and the pipelined timing unit has a data input and a data output. The first output is connected to a second input, and the first input is connected to the data input of the pipelined timing unit. N ≥ 1, M ≥ 2. The data output of the j-th ALU in the i-th row is connected to the first input of the (j+1)-th ALU in the i-th row. 1 ≤ i ≤ N, 0 < j ≤ M-1. The second output of the j-th ALU in the i-th row is connected to the second input of the j-th ALU in the (i+1)-th row. Each row also includes a row storage unit, and the row storage unit of the i-th row is connected to the first input of the first ALU in the i-th row.

[0020] Furthermore, the top-level storage unit is connected to the second input terminal of each of the M arithmetic logic units in the first row array.

[0021] Furthermore, the underlying storage units are connected to the second output terminals of the M arithmetic logic units in the Nth row array.

[0022] In one embodiment, the pipeline timing unit is a register. The pipeline timing unit can also be a shift register or a FIFO buffer, and other components used to implement data movement according to the cycle of the pipeline timing unit also fall within the scope of this invention.

[0023] In one embodiment, the top-level storage unit, the bottom-level storage unit, and the row storage unit are all one of the following storage units: shared memory, cache, and buffer. Other types of storage units used for storing data also fall within the scope of protection of this invention.

[0024] In one embodiment, the top-level storage unit, the bottom-level storage unit, and the row storage unit are all one of shared memory, cache, or buffer. In another embodiment, the top-level storage unit is shared memory, the bottom-level storage unit is cache, and the row storage unit is buffer. Other combinations of the types of top-level storage units, bottom-level storage units, and row storage units fall within the protection scope of this invention.

[0025] The row storage unit is used to store matrix data. The matrix data includes the transformation matrix A and the matrix being transformed B.

[0026] In one implementation, please refer to Figure 3 When the arithmetic logic unit does not include a dedicated storage unit, the row storage unit of the i-th row is connected to the two first input terminals of the first arithmetic logic unit in the i-th row, respectively; the row storage unit is used to store the transformation matrix A and the transformed matrix B, respectively. That is, the first input terminal of the multiplier of the j-th arithmetic logic unit in the i-th row includes two input channels, one for inputting the transformation matrix and the other for inputting the transformed matrix.

[0027] As an example, data for the transformation matrix and the transformed matrix are obtained from the row storage unit, where the transformation matrix includes {A} i,1 A i,2 ,…,A i,j ,…,A i,M}, A i,j This is the matrix data for the transformation matrix of the j-th arithmetic logic unit in the i-th row. The transformed matrix includes {B} i,1 B i,2 ,…,B i,j ,…,B i,M}, B i,j This is the matrix data of the transformed matrix of the j-th arithmetic logic unit in the i-th row. Under the timing of the pipelined sequential unit, when data is passed to the j-th arithmetic logic unit in the i-th row, the transformation matrix includes {A}. i,j A i,j+1 ,…,A i,M The transformed matrix also includes {B} i,j B i,j+1 …,B i,M}, extract A from the transformation matrix i,j Take B from the transformed matrix i,j When the next beat arrives, the transformation matrix {A} will be changed.i,j+1 ,…,A i,M} and the transformed matrix {B i,j+1 ,…,B i,M The data is then passed to the (j+1)th arithmetic logic unit in the i-th row; and so on, distributing all the matrix data to the corresponding arithmetic logic units.

[0028] This system stores the transformation matrix and the transformed matrix separately in row storage units, allowing for real-time data updates via these units without reloading local storage, thus supporting dynamic data updates. Furthermore, its arithmetic logic unit does not require pre-loading of the transformation matrix, resulting in short initialization latency. It is suitable for scenarios requiring flexible handling of dynamic tasks or short-term tasks where minimal initialization latency is necessary.

[0029] In another implementation, please refer to Figure 4 When the arithmetic logic unit includes a dedicated storage unit, the other first input terminal of the multiplier of the j-th arithmetic logic unit in the i-th row is connected to the dedicated storage unit; the dedicated storage unit is used to store the transformation matrix A, and the row storage unit is used to store the transformed matrix B.

[0030] Specifically, for the first arithmetic logic unit in the i-th row, its internal multiplier includes two first input terminals. One first input terminal is connected to the row storage unit, and the other first input terminal is connected to the dedicated storage unit. The multiplier obtains the matrix data of the transformed matrix B from the row storage unit and the matrix data of the transformed matrix A from the dedicated storage unit. For the j-th arithmetic logic unit in the i-th row, its internal multiplier includes two first input terminals. One first input terminal is connected to the data output terminal of the (j-1)-th arithmetic logic unit in the i-th row, and the other first input terminal is connected to the dedicated storage unit. The multiplier obtains the matrix data of the transformed matrix B from the data output terminal and the matrix data of the transformed matrix A from the dedicated storage unit.

[0031] As an example, matrix data retrieved from row storage units includes {B i,1 B i,2 ,…,B i,j ,…,B i,M}, B i,j This is the matrix data for the transformed matrix required by the j-th arithmetic logic unit in the i-th row. Under the cycle time of the pipelined sequential unit, when the data is passed to the j-th arithmetic logic unit in the i-th row, the matrix data also includes {B}. i,j B i,j+1 ,…,B i,M}, from the matrix data, B i,j Retrieve and obtain the transformation matrix A from the dedicated storage unit. i,j When the next beat arrives, the matrix data {B} will be...i,j+1 …B i,M The data is then passed to the (j+1)th arithmetic logic unit in the i-th row; and so on, distributing all the matrix data to the corresponding arithmetic logic units.

[0032] It's important to note that this method achieves the decoupling of data localization and pulse transmission. It stores the transformation matrix data in a dedicated storage unit, while the matrix data transmitted between columns within the same row is the transformed matrix. The transformation matrix data only needs to be loaded once to perform calculations with all the transmitted transformed matrices, achieving complete reuse of the transformation matrix data. Compared to methods where both the transformation matrix data and the transformed matrix data are pulse-transmitted from global memory, this reduces the number of global storage accesses to the transformation matrix data, significantly lowering bandwidth pressure. It is particularly suitable for scenarios where the transformation matrix data is static or has a high reuse rate, such as fixed kernel functions in deep learning inference and scientific computing.

[0033] In one implementation, A i,j B is a submatrix of the source transformation matrix. i,j This is a submatrix of the source matrix being transformed. The submatrix contains at least one element.

[0034] In one implementation, the output result OUT2 of the second output terminal of the j-th arithmetic logic unit in the i-th row is... i,j Satisfy: OUT2 i,j =A i,j ×B i,j +OUT2 i,j-1 A i,j To be allocated to ALU i,j The transformation matrix, B i,j To be allocated to ALU i,j The transformed matrix, OUT2 i,j-1 This is the output of the second output terminal of the (j-1)th arithmetic logic unit in the i-th row.

[0035] In one implementation, when N > 1 and M = 1, the pulsating array is suitable for calculating narrow matrices.

[0036] In one implementation, the pulsating array is suitable for calculating square matrices when N and M are equal.

[0037] It should be noted that when there are N rows, and when M is greater than 1, the throughput of the pulse array of N rows and M columns of arithmetic logic units has the same latency as that of the pulse array of N rows and 1 column, but the throughput of N rows and M columns is M times that of N rows and 1 column, and the computing power of N rows and M columns is also M times that of N rows and 1 column.

[0038] It should be noted that there are various relationships between arithmetic logic units and stream processors in a systolic array, and deployment depends on different situations.

[0039] In one implementation, please refer to Figure 5 Each row of arithmetic logic units resides in a stream processor.

[0040] In one implementation, please refer to Figure 6 Each arithmetic logic unit (ALU) resides in an independent stream processor, and the second output of each of the M ALUs in the Nth row of the array is connected to an underlying memory unit. This scheme is suitable for scenarios where each stream processor contains only one ALU or can only dedicate one ALU for matrix operations.

[0041] In one implementation, all stream processors in the systolic array are located within the same GPU. Schemes where all stream processors in the systolic array are located within different GPUs also fall within the scope of this invention.

[0042] In one embodiment, the distribution type of the pulsating array is either a layered distribution or a clustered distribution.

[0043] Please refer to Figure 1 , Figure 5 and Figure 6 All are stacked arrays, which are bar arrays indexed according to the row order in the pulse array; the location of the central controller is not limited. The stacked array makes the hardware structure more compact, allowing more stream processors to be accommodated in the same chip area, thus improving the chip's resource utilization.

[0044] Please refer to Figure 7 and Figure 8 All are distributed in clusters. Figure 7 It is a clustered distribution of N rows and single columns of arithmetic logic units. Figure 8 A cluster distribution consisting of N rows and M columns of arithmetic logic units (ALUs) with the ALUs configured in a central controller. The cluster distribution includes a central controller, and the distribution of the systolic array is a ring array centered on the central controller, indexed by the row order within the systolic array. The cluster distribution offers strong scalability; the system's computing power can be expanded by increasing the number of clusters. Figure 8 In the middle, ALU i,j Located within the central controller. In another embodiment, the location of the central controller remains unchanged, and the ALU... i,j It can be an arithmetic logic unit in an adjacent stream processor.

[0045] In one embodiment, when the distribution type of the pulsating array is a cluster distribution, if the arithmetic logic unit at the center of the array is the arithmetic logic unit (ALU) in the i-th row and j-th column... i,j Then the ALU i,j It is located in the central controller, or in the stream processor located in the i-th row and j-1-th column, or in the stream processor located in the i-th row and j+1-th column, or in the stream processor located in the i-1-th row and j-th column, or in the stream processor located in the i+1-th row and j-th column.

[0046] In summary, this invention provides a spatial domain-based systolic array comprising N×M rows of arithmetic logic units (ALUs). Each ALU includes an adder, a multiplier, and a pipelined timing unit. Each row of ALUs is configured with a row storage unit connected to the first input of the multiplier, and includes top-level and bottom-level storage units. The matrix data involved in the computation is passed column-by-column through the row storage units, with each column iterating through the final computation result, which is then stored in the bottom-level storage unit. This embodiment of the invention uses a systolic array to transmit matrix data in a pipelined manner, allowing data to be reused by all relevant computation units along the flow, avoiding centralized memory access. Simultaneously, input data is consumed as it flows between spatially arranged computation units, eliminating the need for temporary storage of original data; intermediate results are directly passed between adjacent units to participate in subsequent computations, completely avoiding the need for global caching. This data-flow-driven computation mode simultaneously overcomes the limitations of storage space usage and memory access bandwidth requirements, ultimately solving the computational bottleneck caused by storage walls in traditional architectures.

[0047] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0048] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of this invention is defined by the appended claims.

Claims

1. A spatial domain-based pulsating array, characterized in that, The array includes: An N×M row arithmetic logic unit (ALU) is provided, each ALU including an adder, a multiplier, and a pipelined timing unit. Each multiplier has a first input and a first output, each adder has a second input and a second output, and each pipelined timing unit has a data input and a data output. The first output is connected to a second input, and the first input is connected to the data input of the pipelined timing unit. Wherein, N≥1, M≥2. Wherein, the data output terminal of the j-th arithmetic logic unit in the i-th row is connected to the first input terminal of the (j+1)-th arithmetic logic unit in the i-th row; where 1≤i≤N, 0<j≤M-1; The second output of the j-th arithmetic logic unit in the i-th row is connected to the second input of the j-th arithmetic logic unit in the (i+1)-th row. Each row also includes a row storage unit, and the row storage unit of the i-th row is connected to the first input terminal of the first arithmetic logic unit of the i-th row. The top-level storage unit is connected to the second input terminal of each of the M arithmetic logic units in the first row array; The underlying storage units are connected to the second output terminals of the M arithmetic logic units in the Nth row of the array.

2. The pulsating array according to claim 1, characterized in that, Each row of arithmetic logic units resides in a stream processor.

3. The pulsating array according to claim 1, characterized in that, Each arithmetic logic unit is located in an independent stream processor, and the second output of each of the M arithmetic logic units in the Nth row array is connected to an underlying memory unit.

4. The pulsating array according to claim 1, characterized in that, The distribution type of the pulsating array is either a layered distribution or a clustered distribution.

5. The pulsating array according to claim 4, characterized in that, When the distribution type of the pulsating array is clustered, if the arithmetic logic unit at the center of the array is the arithmetic logic unit (ALU) in the i-th row and j-th column... i,j Then the ALU i,j It is located in the central controller, or in the stream processor located in the i-th row and j-1-th column, or in the stream processor located in the i-th row and j+1-th column, or in the stream processor located in the i-1-th row and j-th column, or in the stream processor located in the i+1-th row and j-th column.

6. The pulsating array according to claim 1, characterized in that, When the arithmetic logic unit includes a dedicated storage unit, the other first input terminal of the multiplier of the j-th arithmetic logic unit in the i-th row is connected to the dedicated storage unit; the dedicated storage unit is used to store the transformation matrix A, and the row storage unit is used to store the transformed matrix B.

7. The pulsating array according to claim 1, characterized in that, When the arithmetic logic unit does not include a dedicated storage unit, the row storage unit of the i-th row is connected to the two first input terminals of the first arithmetic logic unit of the i-th row respectively; the row storage unit is used to store the transformation matrix A and the transformed matrix B respectively.

8. The pulsating array according to claim 1, characterized in that, The top-level storage unit, bottom-level storage unit, and row storage unit are all one of the following storage units: shared memory, cache, and buffer.

9. The pulsating array according to claim 1, characterized in that, The output result OUT2 of the second output terminal of the j-th arithmetic logic unit in the i-th row. i,j Satisfy: OUT2 i,j =A i,j ×B i,j +OUT2 i,j-1 A i,j To be allocated to ALU i,j The transformation matrix, B i,j To be allocated to ALU i,j The transformed matrix, OUT2 i,j-1 This is the output of the second output terminal of the (j-1)th arithmetic logic unit in the i-th row.

10. The pulsating array according to claim 1, characterized in that, The pipeline timing unit is a register.

Images