Method and apparatus for optimizing inter-block early switching of systolic arrays

Abstract

The application discloses a kind of matrix block early switching's systolic array optimization method and device, the present application includes according to the scale of systolic array, input matrix A, input matrix B and result matrix C are blocked, initialize block number i as 0;Two registers are used to load the continuous block of matrix B in sequence, so that in the i th block B i When using input matrix B, the i th block B i+1 When using input matrix B, the i th block B i+2 When using input matrix B, the i th block B i When using input matrix B, the i th block B The present application can reduce the data waiting time when systolic array carries out general matrix multiplication, improve the processing of systolic array, the area overhead of increase is small, with good scalability.

Description

Technical Field

[0001] This invention relates to pulsating arrays, and more specifically to a method and apparatus for optimizing pulsating arrays by pre-switching between matrix blocks. Background Technology

[0002] Over the past decade, deep learning has driven a renaissance in both academia and industry for artificial intelligence. Deep neural networks (DNNs) have significantly advanced applications in computer vision, image processing, and speech recognition, and have been integrated into numerous applications and disciplines. Among these, recurrent neural networks (RNNs), long short-term memory (LSTMs), and convolutional neural networks (CNNs) account for 95% of data center demand, and these three types of neural networks are typically transformed into generalized matrix multiplication (GEMM) operations. Among all proposed DNN accelerator architectures, systolic arrays (SAs) are widely adopted in industry (e.g., Google TPU ASICs, Xilinx FPGA overlays xDNNs) and academia because operand movement occurs only between adjacent processing elements (PEs). This results in a regular structure, simple control, and extremely high computational density (high throughput and low area). When performing matrix multiplication on a systolic array, the on-chip buffer is often insufficient to store the complete input matrix, so it needs to be partitioned off-chip. Most systolic arrays employ an on-chip double-buffered architecture, which hides the data preparation delay for the next set of off-chip matrix blocks within the computation process of the current set of matrix blocks, thus eliminating data preparation bubbles in the systolic array. Furthermore, the matrix loaded into the on-chip buffer needs to be automatically partitioned according to the number of rows / columns of the systolic array's PE (Pressure Point) array, and frequent switching between on-chip blocks is performed during operation. Traditional pulse arrays typically have a single-cycle input data volume of 2. O ( O (This refers to the number of PE rows / columns of the pulsating array), and the output data volume is typically [missing information]. O。After data enters the systolic array, it flows in a specific direction until it exits. Therefore, there are pipeline startup and drainage overheads during the filling and emptying of the SA (positively correlated with MAC latency). For systolic array accelerators that support training, MAC units supporting floating-point multiply-accumulate operations are often used to ensure accuracy. Their computation latency is typically 4 to 6 clock cycles, resulting in significant pipeline startup and drainage overhead. Furthermore, existing systolic arrays always switch to the next on-chip block matrix only after completing the computation of the current on-chip block matrix, leading to frequent idle PEs and low PE utilization during operation. Therefore, for high-performance systolic arrays that support floating-point operations, hiding the startup and drainage overhead is particularly important.

[0003] A pulsating array is a type of... O OK, O A two-dimensional array composed of columns of physical objects (PEs), with adjacent PEs interconnected for data transfer, such as... Figure 1 As shown in the medium gray shading. Each PE contains a register for storing the static matrix and a MAC unit with a computation delay of R for multiplication and addition operations. Due to its simplified control of pure data flow, extremely high computational density, and data reuse, it is widely used in GEMM-related operations. Performing general matrix multiplication Y = AB + C When doing this, it is necessary to first determine the static matrix. A , B or C The static matrix is ​​preloaded into the systolic array, and the remaining two matrices, acting as dynamic matrices, flow into the systolic array from left to right and from top to bottom, respectively, and flow to the right and down between the rows and columns of PE per beat. Therefore, depending on the choice of the static matrix, there are three data flow structures: IS (Input Stationary, ...). A (Static matrix), WS (Weight Stationary), B (Static matrix) and OS (Output Stationary) C (This is a static matrix). Furthermore, the performance and energy efficiency of a systolic array are jointly determined by the data flow structure, the parameters of the systolic array (e.g., number of PEs, MAC delay), and the size of the input matrix. For systolic arrays with a MAC delay greater than 1, the OS design is complex and inefficient, and will not be discussed in this paper; WS, because of its... B The efficient reuse of matrices is widely used in modern processors, so this paper will take WS as an example for analysis, but its conclusions are also applicable to IS.

[0004] When performing matrix multiplication, due to limited on-chip storage resources, the input matrix needs to be pre-divided into blocks off-chip (to distinguish it from on-chip partitioning, we call this off-chip partitioning). Then, the partitioned matrices are moved to the on-chip location for computation. Figure 1 As shown. The original matrix A (A-offchip) is composed of... M OK K The columns are divided into blocks, and the original matrix B (B-offchip) is composed of... K OK N The columns are divided into blocks, and the original matrix C (C-offchip) is formed by... M OK N The columns are divided into blocks. A-onchip, B-onchip, and C-onchip represent the first group of off-chip block matrices moved to the on-chip buffer for computation, which will be referred to as [missing information] in the following text. A , B , C This indicates that the off-chip block matrix is ​​moved to the on-chip buffer. A , B , C It also needs to be determined based on the size of the pulsating array SA ( O × O On-chip automatic segmentation (hereinafter referred to as segmentation) is performed. A, C, Y The block size is M × O , B The block size is O × O In particular, when K Not for O When it is an integer multiple of, B The last row of the matrix is ​​the block of rows (i.e. A The last column of the matrix (the column that is divided into blocks) is less than O ;when N Not for O When it is an integer multiple of, B The last column of the matrix is ​​the block of columns (i.e. C The last column of the matrix (the column that is divided into blocks) is less than O .

[0005] by A i 、B i 、C i They represent A, B, C Matrix number i The next block to participate in the calculation M i × Ki 、K i × N i 、M i × N i Representing the block matrix respectively A i 、B i 、C i scale Y i Intermediate results updated to C i middle, C i The final calculation result is Y i The final result, and the calculation process, are shown in the following formula: , (1).

[0006] A systolic array (SA) is a typical data flow-driven architecture. The utilization rate of processing units (PEs) reaches its maximum when data fills the SA. However, due to input / output bandwidth limitations, a large number of PEs become idle when the input matrix flows into the SA and the result matrix flows out, leading to a decrease in PE utilization. Meanwhile, as shown in equation (1), on-chip partitioning... Y = AB + C During the process, frequent switching between inter-block matrices will occur, resulting in a large amount of pulsating array SA filling and emptying overhead, causing PE resource waste. Summary of the Invention The technical problem to be solved by the present invention is to provide a systolic array optimization method and apparatus for early switching between matrix blocks, which can reduce the data waiting time when the systolic array performs general matrix multiplication operations, improve the processing performance of the systolic array, and the increased area overhead is almost negligible, and has good scalability.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for optimizing a pulsating array with pre-switching between matrix blocks includes: S101, based on the size of the pulsation array, divide the input matrix A, input matrix B and result matrix C into blocks; S102, two registers are used to load consecutive blocks of matrix B sequentially, so that at any given time in the input matrix B... i Each block B iAll usage completed, now starting to use the input matrix B. i+ 1 block B i+1 The input matrix B is pre-loaded with the first... i+ 2 blocks B i+2 Loaded into two registers; and in the first... i Each block B i At that time, the corresponding blocks in the input matrix A and the result matrix C are fed into the pulsating array to participate in the general matrix multiplication operation.

[0008] Optionally, step S102, which uses two registers to sequentially load consecutive blocks of matrix B, includes: S201, initialize block number i to 0; S202, determine whether the result of modulo operation on block number i with respect to 2 is 0. If it is true, then input the first block number into matrix B. i Each block B i Load into register B-reg0, otherwise jump to S206; S203, determine the first... i Each block B i Check if all operations have been completed. If not, jump to S203 to continue the judgment; otherwise, calculate the first step of the input matrix B. i+ 2 blocks B i+2 Start loading time T B(i+2) Redirecting to S204; S204, determine the first... i+ 2 blocks B i+2 Start loading time T B(i+2) Is it equal to 0? If not equal to 0, then input the first [value] of matrix B. i+ 2 blocks B i+2 Start loading time T B(i+2) If the value is decremented, proceed to S204 for further evaluation; otherwise, proceed to S205. S205, increment the value of block number i by 2; if block number i is less than the number of blocks in the input matrix B, then jump to S202; otherwise, end and exit. S206, the first line of the input matrix B i Each block B i Loaded into register B-reg1; S207, determine the first... i Each block B i Check if all operations have been completed. If not, jump to S207 to continue the judgment; otherwise, calculate the first step of the input matrix B. i+ 2 blocks B i+2 Start loading time T B(i+2) Redirect to S208; S208, determine the first... i+ 2 blocks B i+2 Start loading time T B(i+2) Is it equal to 0? If not equal to 0, then input the first [value] of matrix B. i+ 2 blocks B i+2 Start loading time T B(i+2) If the value is decremented, proceed to S208 to continue the judgment; otherwise, proceed to S209. S209, increment the value of block number i by 2; if block number i is less than the number of blocks in the input matrix B, then jump to S202; otherwise, end and exit.

[0009] Optionally, the first i+ 2 blocks B i+2 Start loading time T B(i+2) The expression for the computation function is: , Or the first i+ 2 blocks B i+2 Start loading time T B(i+2) The expression for the computation function is: , in, For the first i+ 2 blocks B i+2 The start loading time, R For the delay of the general matrix multiplication calculation unit, For the first i Each block B i The number of columns.

[0010] Optionally, the function expression involved in the general matrix multiplication operation in step S102 is:

[0011] In the above formula, C i For the first i The blocks in the resulting matrix C from the matrix multiplication and addition calculation. A i For the partitioning of the input matrix A, B i The input matrix B for matrix multiplication and addition is divided into blocks, and the blocks are divided into blocks. A i The size is M i × K i Segmentation B i The size is K i × N i Segmentation C i The size is M i × N i .

[0012] Optionally, step S102, which involves dividing the input matrix A and the result matrix C into blocks and feeding them into the pulsating array, includes: S301, divide the first block of the input matrix A. A 0 and the first block of the result matrix C C 0 is input into the pulsating array to participate in the general matrix multiplication operation of the first block, initializing the block number. j =0; S302, determine the first... j Each block A j Check if all operations have been completed. If not, jump to S302 to continue the judgment; otherwise, calculate the first step of the input matrix A. j+ 1 block A j+1 With the j Each block A j Loading gap T gap Redirecting to S303; S303, Determine loading gap T gap Is it equal to 0? If not equal to 0, then load the gap. T gap If the value is decremented, proceed to S303 for further evaluation; otherwise, proceed to S304. S304, number the blocks jAdd 2 to the value of the block number; j If the number of blocks is less than or equal to the number of blocks in the input matrix A, then the th block of the input matrix A will be... j Each block A j and the result matrix C's first j Each block C j Flow into the pulsating array to participate in the first j Perform a general matrix multiplication operation on each block, then jump to S303; otherwise, terminate and exit.

[0013] Optionally, the calculation of the input matrix A's first... j+ 1 block A j+1 With the j Each block A j Loading gap T gap The function expression is:

[0014] In the above formula, K j+1 For the input matrix A, the first... j+ 1 block A j+1 Column size, K j For the first j Each block A j Column size, R This is the delay for the general matrix multiplication calculation unit.

[0015] Furthermore, the present invention also provides a systolic array device for applying the systolic array optimization method of early switching between matrix blocks described above, comprising a systolic array consisting of multiple processing units arranged in an array and sequentially connected between the same row and column. The processing unit is provided with a block buffer unit with dual registers. The block buffer unit with dual registers includes a multiplexer DMUX, a data selector MUX, and two registers for buffering blocks of matrix B. The multiplexer DMUX takes blocks of externally input matrix B as input and, under the control of the control signal ctrl, puts the input blocks into the two registers in turn. The data selector MUX is used to select blocks of matrix B from the two registers under the control of the selection signal sel and send them to the general matrix multiplication calculation unit in the processing unit.

[0016] Optionally, the general matrix multiplication calculation component is a multiply-add calculation unit (MAC). The multiply-add calculation unit (MAC) includes three inputs and one output. The three inputs are used to input blocks of matrix A, blocks of matrix B, and blocks of matrix C, respectively. The one output is used to output blocks of the result matrix Y. The block buffer unit with dual registers is located on the data input channel of the block of matrix B.

[0017] Furthermore, the present invention also provides a systolic array optimization device for early switching between matrix blocks, comprising a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the systolic array optimization method for early switching between matrix blocks.

[0018] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program for being programmed or configured by a microprocessor to execute the systolic array optimization method for early switching between matrix blocks. Compared with the prior art, the present invention has the following main advantages: 1. This invention uses two registers to sequentially load consecutive blocks of matrix B, so that in the input matrix B, the... i Each block B i All usage completed, now starting to use the input matrix B. i+ 1 block B i+1 The input matrix B is pre-loaded with the first... i+ 2 blocks B i+2 By loading data into two registers, the waiting time for data can be reduced, thereby improving the processing performance of the systolic array. Moreover, as the size of the systolic array increases, the static matrix loading cycle of the systolic array becomes longer, resulting in a greater delay in the general multiply-accumulate operation of the systolic array, as well as a longer delay in pipeline filling and emptying. Therefore, the performance improvement of this invention will be more significant for systolic arrays with longer delays in general multiply-accumulate operations.

[0019] 2. This invention only requires adding an extra register and its switching logic (traditionally a single register) to the input channels of the input matrix B of the processing unit. It is simple to implement, requires fewer parameters than the original design, and the implementation area is slightly reduced. Therefore, the area overhead of this part is almost negligible. Although the hardware overhead is linearly positively correlated with the number of processing units, the increased area for a single processing unit is very small. Therefore, even for large-scale systolic arrays, the hardware overhead will not increase drastically. Thus, the method of this invention has good scalability. Attached Figure Description

[0020] Figure 1A schematic diagram illustrating the block-based principle of performing matrix multiplication using existing technologies.

[0021] Figure 2 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0022] Figure 3 This is a detailed flowchart illustrating the method of an embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram comparing the internal structure of the processing unit PE(b) in this embodiment and the prior art (a).

[0024] Figure 5 As described in the embodiments of the present invention K i+2 =K i Time-static matrix B i+2 The loading timing.

[0025] Figure 6 As described in the embodiments of the present invention K i+2 >K i Time-static matrix B i+2 The loading timing.

[0026] Figure 7 As described in the embodiments of the present invention K i+2 <K i Time-static matrix B i+2 The loading timing.

[0027] Figure 8 As described in the embodiments of the present invention K i+1 = K i Time i+ The timing of the inflow of a dynamic matrix.

[0028] Figure 9 As described in the embodiments of the present invention K i+1 >K i Time i+ The timing of the inflow of a dynamic matrix.

[0029] Figure 10 As described in the embodiments of the present invention K i+1 <K i Time i+The timing of the inflow of a dynamic matrix can cause errors in the results.

[0030] Figure 11 As described in the embodiments of the present invention K i+1 <K i Time i+ The inflow timing of a dynamic matrix i+1 The block dynamic matrix needs to be in the first... i The case where the block dynamic matrix enters SA after several cycles.

[0031] Figure 12 This is a graph showing the relationship between PE utilization and M before and after SA optimization when K=113 in an embodiment of the present invention.

[0032] Figure 13 This is a graph showing the relationship between PE utilization and M before and after SA optimization when K=127 in an embodiment of the present invention.

[0033] Figure 14 This is a graph showing the relationship between PE utilization and M before and after SA optimization when K=128 in an embodiment of the present invention. Detailed Implementation

[0034] like Figure 2 As shown, the pulsating array optimization method for early switching between matrix blocks in this embodiment includes: S101, according to the size of the pulsation array (SA), divide the input matrix A, input matrix B and result matrix C into blocks; S102, two registers are used to load consecutive blocks of matrix B sequentially, so that at any given time in the input matrix B... i Each block B i All usage completed, now starting to use the input matrix B. i+ 1 block B i+1 The input matrix B is pre-loaded with the first... i+ 2 blocks B i+2 Loaded into two registers; and in the first... i Each block B i At that time, the corresponding blocks in the input matrix A and the result matrix C are fed into the pulsating array to participate in the general matrix multiplication operation.

[0035] It should be noted that, Figure 2The input matrices A, B, and C in this context refer to the matrices that have been moved to the on-chip buffer. To distinguish them from off-chip matrices, they can be referred to as on-chip matrices A-on-chip, B-on-chip, and C-on-chip. The off-chip states of the input matrices A, B, and C are referred to as off-chip matrices A-off-chip, B-off-chip, and C-off-chip. When moving A-off-chip, B-off-chip, and C-off-chip to the on-chip buffer, the off-chip matrices A-off-chip, B-off-chip, and C-off-chip need to be divided into blocks and then moved sequentially to the on-chip buffer to become the on-chip matrices A-on-chip, B-on-chip, and C-on-chip.

[0036] like Figure 3 As shown, in this embodiment, step S102 uses two registers to sequentially load consecutive blocks of matrix B, including: S201, initialize block number i to 0; S202, determine whether the result of modulo operation on block number i with respect to 2 is 0. If it is true, then input the first block number into matrix B. i Each block B i Load into register B-reg0, otherwise jump to S206; S203, determine the first... i Each block B i Check if all operations have been completed. If not, jump to S203 to continue the judgment; otherwise, calculate the first step of the input matrix B. i+ 2 blocks B i+2 Start loading time T B(i+2) Redirecting to S204; S204, determine the first... i+ 2 blocks B i+2 Start loading time T B(i+2) Is it equal to 0? If not equal to 0, then input the first [value] of matrix B. i+ 2 blocks B i+2 Start loading time T B(i+2) If the value is decremented, proceed to S204 for further evaluation; otherwise, proceed to S205. S205, increment the value of block number i by 2; if block number i is less than the number of blocks in the input matrix B, then jump to S202; otherwise, end and exit. S206, the first line of the input matrix B i Each block B i Loaded into register B-reg1; S207, determine the first... i Each block B i Check if all operations have been completed. If not, jump to S207 to continue the judgment; otherwise, calculate the first step of the input matrix B. i+ 2 blocks B i+2 Start loading time T B(i+2) Redirect to S208; S208, determine the first... i+ 2 blocks B i+2 Start loading time T B(i+2) Is it equal to 0? If not equal to 0, then input the first [value] of matrix B. i+ 2 blocks B i+2 Start loading time T B(i+2) If the value is decremented, proceed to S208 to continue the judgment; otherwise, proceed to S209. S209, increment the value of block number i by 2; if block number i is less than the number of blocks in the input matrix B, then jump to S202; otherwise, end and exit.

[0037] Because we configured two storage systems in the PE environment. B Matrix registers, blocks of two adjacent static matrices B i , B i+1 They will be loaded into two separate buffers, with no dependencies, so you only need to focus on... B i and B i+2 The timing of the switch between them makes B i+2 Loading into SA will not affect the current matrix. Y i The calculation, that is, to ensure B i+2 Loading SA B i It has been used up, and can therefore be safely used. B i+2 Replace without affecting Y i The correctness of the statement.

[0038] according to K i and K i+2 The size relationship can be discussed in three cases, namely... K i =K i+2 , K i <K i+2 , K i >K i+2 ,like Figures 5-7 As shown in the figure, the horizontal axis represents the period, the vertical axis represents the PE row, and the starting time is... A The elements in the last row of the matrix begin to enter SA. T A for A The number of periods that a row of the matrix enters in SA. T N For matrix A The number of cycles the last element flows through in SA. B i At point P ( T p =T A +T N After use, T Bi+2 for B i+2 You can begin loading the timeline.

[0039] K i = K i+2 :for K i = K i+2 In cases such as Figure 5 As shown, B i+2 exist T P After loading is complete, it will not have any impact. Y i The calculation, therefore B i+2 The loading start time is T P - K i+2 (equal T P - K i ).

[0040] K i+2 >K i :for K i+2 >K i In cases such as Figure 6 As shown, B i+2 Also in T P Loading can then be completed, so B i+2 The loading start time is also T P - K i+2 But smaller than T P - K i .

[0041] K i+2 <K i For K i+2 <K i In cases such as Figure 7 As shown, B i+2 It is possible B i The former K i+2 Loading is completed when the line is no longer in use, P' is B i The Middle K i+2 The moment when the line is finished being used, at this time B i+2 The loading time is T P` - K i+2 Less than T P - K i Therefore, in this embodiment, the first... i+ 2 blocks B i+2 Start loading time T B(i+2) The expression for the computation function is:

[0042] In the above formula, For the first i+2 blocks B i+2 The start loading time, R The delay (value greater than) for general matrix multiplication calculation components (e.g., multiply-accumulate unit MAC). ), For the first i Each block B i The number of columns (ranging from 0 to 16).

[0043] According to the principle of automatic on-chip segmentation K i+2 < K i The situation only occurs when B Matrix rows are not O When the number of rows / columns of the systolic array SA is an integer multiple, the block matrix B The transition from the second-to-last line to the last line, such as... Figure 1 When switching from block 4 to block 6, and from block 5 to block 7, it only accounts for a small percentage of all blocks. ,in K For the input matrix A The number of columns, N For the result matrix C The number of columns; and K i+2 > K i This situation can only occur when switching between on-chip matrix blocks, and the static matrix rows of the current block are not... O When it is an integer multiple of . Therefore, in most cases K i+2 All equal to K i At the same time, due to K i+2 >K i and K i+2 <K i hour B i+2 The loading time is earlier than K i+2 = K i So if you use K i+2 = K i Formula calculation K i+2 > K i and K i+2 <Ki hour B i+2 The loading time only affects performance, not correctness. Therefore, to simplify the design, we can use... K i+2 = K i hour B i+2 Start loading time T B(i+2) Calculation formula processing K i+2 >K i and K i+2 <K i The situation at that time is as shown in the following formula: , In this embodiment, the function expression involved in the general matrix multiplication operation in step S102 is:

[0044] In the above formula, C i For the first i The blocks in the resulting matrix C from the matrix multiplication and addition calculation. A i For the partitioning of the input matrix A, B i The input matrix B for matrix multiplication and addition is divided into blocks, and the blocks are divided into blocks. A i The size is M i × K i Segmentation B i The size is K i × N i Segmentation C i The size is M i × N i .

[0045] like Figure 3 As shown, in this embodiment, step S102, which involves dividing the input matrix A and the result matrix C into blocks and feeding them into the pulsating array, includes: S301, divide the first block of the input matrix A. A 0 and the first block of the result matrix C C0 is input into the pulsating array to participate in the general matrix multiplication operation of the first block, initializing the block number. j =0; S302, determine the first... j Each block A j Check if all operations have been completed. If not, jump to S302 to continue the judgment; otherwise, calculate the first step of the input matrix A. j+ 1 block A j+1 With the j Each block A j Loading gap T gap Redirecting to S303; S303, Determine loading gap T gap Is it equal to 0? If not equal to 0, then load the gap. T gap If the value is decremented, proceed to S303 for further evaluation; otherwise, proceed to S304. S304, number the blocks j Add 2 to the value of the block number; j If the number of blocks is less than or equal to the number of blocks in the input matrix A, then the th block of the input matrix A will be... j Each block A j and the result matrix C's first j Each block C j Flow into the pulsating array to participate in the first j Perform a general matrix multiplication operation on each block, then jump to S303; otherwise, terminate and exit.

[0046] Due to the full pipeline mechanism of the systolic array (SA), the next block of the matrix... Y i+1 The calculation does not need to wait until the current block. Yi Completely outflow from SA, but can be done in the dynamic matrix. A i+1 、C i+1 Once the matrix is ​​prepared, it can be entered into SA for computation in advance, provided that the static matrix is ​​guaranteed. B i+1 Loading complete, and Y i+1 When SA is released, it is not related to Y i Conflict. Similarly, according to K i+1 and K i The size relationship can be divided into three cases for discussion.

[0047] K i+1 =K i :for K i+1 =K i In this situation, Y i+1 and Y i The corresponding elements in SA have the same period, so as long as the static matrix is ​​prepared, the period of the first element will be exactly the same. i+1 Block dynamic matrix can follow the first i The block dynamic matrix flows into SA. For example... Figure 8 As shown, the upper and lower parts of the figure respectively represent Y i and Y i+1 The calculation process. Here, line a represents the dynamic matrix. C The process of the first element of the first row flowing through the first column PE of SA is represented by line b. Line c represents the process of the first element of the last row of dynamic matrix C flowing through the first column PE of SA. N The process of listing PEs. (Part 1) i+1 The block dynamic matrix in the first i The next cycle of the block dynamic matrix flows into SA, and its calculation results... Y i+1 Also in Yi The next beat sequentially flows out SA, at this time the first i +1 The last row of the block dynamic matrix and the first row i The first row of the block dynamic matrix flows into the SA interval. T gap =1. It's not hard to see that as long as the dynamic matrix is ​​guaranteed... C i+1 The first element of the first row does not interact with the dynamic matrix as it flows through the first column PE of SA. C i If the first element in the last row conflicts, the entire matrix block will not conflict.

[0048] Therefore, the following discussion will only focus on dynamic matrices. C i The first element of the last row and the dynamic matrix C i+1 The usage of the first element in the first row in the first column PE of SA: K i+1 >K i :forK i+1 >K i In this situation, Y i+1 The computation delay is greater than Y i The calculation delay, so the first i+1 Block dynamic matrix can follow the first i The block dynamic matrix enters SA at this time. T gap =1, such as Figure 9 As shown, line a represents the matrix. C i+1 The first element in the first row is used in the first column PE of SA; line b represents the matrix. C i The usage of the first element in the last row in the first column PE of SA. T gap This is the loading interval.

[0049] K i+1 <K i :for K i+1 <K i In this situation, Y i+1 The computation delay is less than Y i The calculation delay, if the first i+1 Block dynamic matrix follows the first i If the block dynamic matrix enters SA, Y i+1 Will be more Y i Early leakage of SA can lead to incorrect results, such as... Figure 10 As shown, line a represents the matrix. C i+1 The first element in the first row is used in the first column PE of SA; line b represents the matrix. C i The usage of the first element in the last row in the first column PE of SA. T gap This is the loading gap. Therefore, the first... i+1 The block dynamic matrix needs to be in the first... i The block dynamic matrix can only enter SA after several cycles, such as... Figure 11 As shown, line a represents the matrix. C i+1 The first element in the first row is used in the first column PE of SA; line b represents the matrix.C i The usage of the first element in the last row in the first column PE of SA. T gap This is the loading gap. The dynamic matrix is ​​included. C i The delay of the first element of the first row flowing through SA for: , Dynamic matrix C i+1 The delay of the first element of the first row flowing through SA for: , No. i+1 The last row of the block dynamic matrix and the first row i The first row of the block dynamic matrix flows into the loading gap of SA. T gap for:

[0050] in, R The delay (value greater than) for general matrix multiplication calculation components (e.g., multiply-accumulate unit MAC). O represents the row / column size of SA. Therefore, in this embodiment, the first row of the input matrix A is calculated. j+ 1 block A j+1 With the j Each block A j Loading gap T gap The function expression is:

[0051] In the above formula, K j+1 For the input matrix A, the first... j+ 1 block A j+1 Column size, K j For the first j Each block A j Column size, R The delay (value greater than) for general matrix multiplication calculation components (e.g., multiply-accumulate unit MAC). ).

[0052] To evaluate the systolic array optimization method for early switching between matrix blocks in this embodiment. This embodiment also provides a systolic array device for applying the systolic array optimization method for early switching between matrix blocks described above, comprising a systolic array consisting of multiple processing units arranged in an array and sequentially connected between the same row and column, such as... Figure 4 As shown in (b), the processing unit includes a block buffer unit with dual registers (register B-reg0 and register B-reg1, where "0" indicates register B-reg0 and "1" indicates register B-reg1). This block buffer unit with dual registers includes a multiplexer DMUX, a data selector MUX, and two registers for buffering blocks of matrix B. The multiplexer DMUX takes blocks of externally input matrix B as input and, under the control of the control signal ctrl, alternately places the input blocks into the two registers. The data selector MUX, under the control of the selection signal sel, selects blocks of matrix B from the two registers and sends them to the general matrix multiplication calculation unit in the processing unit. Figure 4 As shown in (a), the original processing unit PE of the systolic array SA contains a unit for storing the static block matrix. B i It has a register and a multiply-accumulate unit (MAC) for multiply-accumulate operations. Based on the double-buffered design, this embodiment uses a static matrix inside the processing unit (PE). B Two sets of register resources are provided to store two adjacent static block matrices. B i and B i+1 ,exist B i Preload when participating in calculations B i+1 This completely hides the loading delay of the next static matrix within the computation process of the current matrix, allowing the computation of the next matrix block to begin immediately after the computation of the previous matrix block is completed. The internal structure of the improved post-processing unit PE proposed in this embodiment using the dual-register scheme is as follows: Figure 4 As shown in (b), compared to the original design, only one additional storage unit was added. B i+1 The register, one used for B A matrix-assigned multiplexer DMUX and a multi-way distributor for matrix allocation B The matrix selection data selector (MUX) is used. Because we have configured two sets of storage in the PE environment... B Matrix registers, two adjacent static matrices are divided into blocks. B i , B i+1 They will be loaded into two separate buffers with no dependencies, so you only need to focus on them. Bi and B i+2 The timing of the switch between them makes B i+2 Loading into SA will not affect the current matrix. Y i The calculation, that is, to ensure B i+2 Loading SA B i It has been used up, and can therefore be safely used. B i+2 Replace without affecting Y i The correctness of the calculation is ensured. In this embodiment, the general matrix multiplication calculation component is a multiply-accumulate calculation unit (MAC). The MAC includes three inputs and one output. The three inputs are used to input blocks of matrix A, blocks of matrix B, and blocks of matrix C, respectively. The one output is used to output blocks of the result matrix Y. The block buffer unit with dual registers is located on the data input channel of the block of matrix B. It should be noted that this systolic array device can be in the form of a standalone accelerator chip or an integrated general-purpose processing chip (CPU or GPU). Its external form can be in the form of a chip or a board composed of an accelerator chip and its peripheral circuits. In short, the way it integrates additional circuits and peripheral circuits, as well as its external form, should not constitute a limitation on this systolic array device.

[0053] To evaluate the pre-switching method between matrix blocks based on systolic arrays in this embodiment, we implemented an RTL design and compared it with a conventional systolic array. For implementation complexity considerations, the systolic array (SA) implemented using the pre-switching method between matrix blocks in this embodiment has a processing unit (PE) size of 16×16, with a MAC computation latency of 6 seconds per PE, and the data stream uses WS. A , B , C The on-chip double buffer sizes are 2×3MB, 2×1MB, and 2×256KB, respectively. Meanwhile, to simplify the design, the loading time is... T B(i+2) use Calculation. Under real application loads, M It's not always very big; K , N It is not always possible to satisfy the row / column size requirement of SA. OMultiples of integers. Therefore, to cover as many scenarios as possible and make the evaluation results more convincing, we artificially generate some load scales, including various boundary cases, so that real application loads can be found in these scenarios even if they are not fully simulated. We let M =1, 16, 32, 64, 128, 256, 512. K Take 113, 127, and 128 respectively: when K When it is 113, B The last row of the matrix has only one row, and there are 15 empty rows of PE in SA; K When =127, B The last row of the matrix is ​​divided into 15 blocks, and only the last row PE in SA is empty; K There is no PE idle when =128. N The value is fixed at 64, so that each matrix block can fill every column of SA. Figure 12 , Figure 13 and Figure 14 They are respectively K PE utilization rates before and after SA optimization at values ​​of 113, 127, and 128 are compared with M The relationship diagram. Where the horizontal axis is... M The main vertical axis represents PE utilization rate, and the secondary vertical axis represents the improved PE utilization rate after optimization. (Comparison) Figure 12 , Figure 13 and Figure 14 It can be seen that, in M , N If it remains unchanged, K The closer O The higher the PE utilization rate, the greater the multiple of the PE ratio. K , N With the PE utilization rate remaining constant, M Monotonically increasing, but the improved PE utilization rate after optimization increases with... M First increase, then decrease, and in M It reaches its maximum at =128: K =113 is 36.54%, K =127, which is 43.06%. K=128 yields a performance improvement of 43.54%. It's worth noting that while this paper's experiments only evaluated a specific systolic array (SA size 16×16, MAC calculation delay 6), the proposed method is applicable to systolic arrays with any parameters. Furthermore, based on the data flow characteristics between PEs in the systolic array, the performance improvement is more significant for large-scale systolic arrays with long MAC delays. This is because as the SA size increases, the static matrix loading cycle becomes longer; the larger the MAC delay, the longer the pipeline filling and emptying delays. The hardware modifications involved in this method only involve two aspects: firstly, the calculation of the static matrix loading time and the dynamic matrix inflow time outside the PE array; and secondly, the addition of a register and two data controllers inside the PE (one for...). B A matrix-assigned multiplexer DMUX and a multi-way distributor for matrix allocation B The proposed matrix selection method uses a data selector (MUX). For the former, since the proposed computation method requires fewer parameters than the original design, the implementation area is slightly reduced, making this area overhead almost negligible. As for the latter, although the hardware overhead is linearly positively correlated with the number of physical elements (PEs), the increased area for a single PE is small, so even for large-scale systolic arrays, the hardware overhead will not increase drastically. Therefore, the proposed scheme has excellent scalability.

[0054] Furthermore, this embodiment also provides a systolic array optimization apparatus for early switching between matrix blocks, including a microprocessor and a memory interconnected thereto. The microprocessor is programmed or configured to execute the aforementioned systolic array optimization method for early switching between matrix blocks. Additionally, this embodiment also provides a computer-readable storage medium storing a computer program for being programmed or configured by the microprocessor to execute the aforementioned systolic array optimization method for early switching between matrix blocks.

[0055] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0056] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for optimizing a pulsating array with pre-switching between matrix blocks, characterized in that, include: S101, based on the size of the pulsation array, divide the input matrix A, input matrix B and result matrix C into blocks; S102, two registers are used to load consecutive blocks of matrix B sequentially, so that at any given time in the input matrix B... i Each block B i All usage completed, now starting to use the input matrix B. i+ 1 block B i+1 The input matrix B is pre-loaded with the first... i+ 2 blocks B i+2 Loaded into two registers; and in the first... i Each block B i At that time, the corresponding blocks in the input matrix A and the result matrix C are fed into the pulsating array to participate in the general matrix multiplication operation; Step S102, which uses two registers to sequentially load consecutive blocks of matrix B, includes: S201, initialize block number i to 0; S202, determine whether the result of modulo operation on block number i with respect to 2 is 0. If it is true, then input the first block number into matrix B. i Each block B i Load into register B-reg0, otherwise jump to S206; S203, determine the first... i Each block B i Check if all operations have been completed. If not, jump to S203 to continue the judgment; otherwise, calculate the first step of the input matrix B. i+ 2 blocks B i+2 Start loading time T B(i+2) Redirecting to S204; S204, determine the first... i+ 2 blocks B i+2 Start loading time T B(i+2) Is it equal to 0? If not equal to 0, then input the first [value] of matrix B. i+ 2 blocks B i+2 Start loading time T B(i+2) If the value is decremented, proceed to S204 for further evaluation; otherwise, proceed to S205. S205, increment the value of block number i by 2; if block number i is less than the number of blocks in the input matrix B, then jump to S202; otherwise, end and exit. S206, the first line of the input matrix B i Each block B i Loaded into register B-reg1; S207, determine the first... i Each block B i Check if all operations have been completed. If not, jump to S207 to continue the judgment; otherwise, calculate the first step of the input matrix B. i+ 2 blocks B i+2 Start loading time T B(i+2) Redirect to S208; S208, determine the first... i+ 2 blocks B i+2 Start loading time T B(i+2) Is it equal to 0? If not equal to 0, then input the first [value] of matrix B. i+ 2 blocks B i+2 Start loading time T B(i+2) If the value is decremented, proceed to S208 to continue the judgment; otherwise, proceed to S209. S209, increment the value of block number i by 2; if block number i is less than the number of blocks in the input matrix B, then jump to S202; otherwise, end and exit. The first i+ 2 blocks B i+2 Start loading time T B(i+2) The expression for the computation function is: ， Or the first i+ 2 blocks B i+2 Start loading time T B(i+2) The expression for the computation function is: ， in, For the first i+ 2 blocks B i+2 The start loading time, R For the delay of the general matrix multiplication calculation unit, For the first i Each block B i The number of columns, For block A i Column size, For block A i+2 Column size, A i and A i+2 This refers to the blocks in the input matrix A.

2. The pulsating array optimization method for early switching between matrix blocks according to claim 1, characterized in that, The function expression involved in the general matrix multiplication operation in step S102 is: In the above formula, C i For the first i The blocks in the resulting matrix C from the matrix multiplication and addition calculation. A i For the partitioning of the input matrix A, B i The input matrix B for matrix multiplication and addition is divided into blocks, and the blocks are divided into blocks. A i The size is M i × K i Segmentation B i The size is K i × N i Segmentation C i The size is M i × N i .

3. The pulsating array optimization method for early switching between matrix blocks according to claim 1, characterized in that, Step S102, which involves dividing the input matrix A and the result matrix C into blocks and feeding them into the pulsating array, includes: S301, divide the first block of input matrix A. A 0 and the first block of the result matrix C C 0 is input into the pulsating array to participate in the general matrix multiplication operation of the first block, initializing the block number. j =0; S302, determine the first... j Each block A j Check if all operations have been completed. If not, jump to S302 to continue the judgment; otherwise, calculate the first step of the input matrix A. j+ 1 block A j+1 With the j Each block A j Loading gap T gap Redirecting to S303; S303, Determine loading gap T gap Is it equal to 0? If not equal to 0, then load the gap. T gap If the value is decremented, proceed to S303 for further evaluation; otherwise, proceed to S304. S304, number the blocks j Add 2 to the value of the block number; j If the number of blocks is less than or equal to the number of blocks in the input matrix A, then the th block of the input matrix A will be... j Each block A j and the result matrix C's first j Each block C j Flow into the pulsating array to participate in the first j Perform a general matrix multiplication operation on each block, then jump to S303; otherwise, terminate and exit.

4. The pulsating array optimization method for early switching between matrix blocks according to claim 3, characterized in that, The first calculation input matrix A j+ 1 block A j+1 With the j Each block A j Loading gap T gap The function expression is: In the above formula, K j+1 For the input matrix A, the first... j+ 1 block A j+1 Column size, K j For the first j Each block A j Column size, R This is the delay for the general matrix multiplication calculation unit.

5. A pulsating array device for applying the pulsating array optimization method for early switching between matrix blocks as described in any one of claims 1 to 4, comprising a pulsating array composed of multiple processing units arranged in an array and sequentially connected between the same row and column, characterized in that, The processing unit includes a block buffer unit with dual registers. The block buffer unit with dual registers includes a multiplexer DMUX, a data selector MUX, and two registers for buffering blocks of matrix B. The multiplexer DMUX takes blocks of externally input matrix B as input and, under the control of the control signal ctrl, puts the input blocks into the two registers in turn. The data selector MUX is used to select blocks of matrix B from the two registers under the control of the selection signal sel and send them to the general matrix multiplication calculation unit in the processing unit.

6. The pulsating array device according to claim 5, characterized in that, The general matrix multiplication calculation component is a multiply-add calculation unit (MAC). The multiply-add calculation unit (MAC) includes three inputs and one output. The three inputs are used to input blocks of matrix A, blocks of matrix B, and blocks of matrix C, respectively. The one output is used to output blocks of the result matrix Y. The block buffer unit with dual registers is located on the data input channel of the block of matrix B.

7. A pulse array optimization device for pre-switching between matrix blocks, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to execute the pulsating array optimization method for early switching between matrix blocks as described in any one of claims 1 to 4.

8. A computer-readable storage medium storing a computer program, characterized in that, The computer program is used to be programmed or configured by a microprocessor to execute the pulsating array optimization method for early switching between matrix blocks as described in any one of claims 1 to 4.

Images