A sparse systolic array system supporting adaptive delay

Abstract

This invention discloses a sparse systolic array system supporting adaptive delay. By setting up an input control and mode management module, based on the matrix format of matrices A and B to be multiplied, the system is configured to either a general matrix multiplication mode or a sparse matrix multiplication mode, and matrices A and B are input to the systolic array calculation module to accommodate different computational needs. The systolic array calculation module consists of M rows and N columns of processing units, each of which performs multiplication and accumulation calculations on the matrix elements sequentially input to its corresponding unit. An adaptive delay controller is set up to calculate the upper bound of the output delay based on the statistical information of the non-zero elements in each row of matrix A in sparse matrix multiplication mode. A control signal is generated based on this upper bound, allowing the emptying phase of the systolic array calculation module to be terminated early, effectively reducing the emptying delay in sparse matrix multiplication mode.

Description

Technical Field

[0001] This invention belongs to the field of integrated circuit technology, and more specifically, relates to a sparse systolic array system that supports adaptive delay. Background Technology

[0002] In recent years, deep learning has achieved great success in fields such as image recognition and natural language processing, and sparse matrices are widely used in deep learning models. With the continuous growth of model size and data volume, how to handle the large number of zero elements in sparse matrices has become crucial for improving the training efficiency of deep learning models.

[0003] Existing systolic array architectures possess high parallel computing capabilities and efficient data locality. They achieve pipelined data transfer between computing units through fixed data flow paths, effectively reducing data transport latency. The core characteristic of sparse matrices lies in their high proportion of zero elements and irregular distribution of non-zero elements. However, analysis reveals a conflict between traditional systolic arrays and the characteristics of sparse matrices. Specifically, the fixed data flow path of systolic arrays cannot adapt to the irregular distribution of non-zero elements in sparse matrices, resulting in low computing unit utilization, difficulties in data alignment, and high inefficient power consumption.

[0004] Therefore, designing a systolic array that supports adaptive delay can effectively solve the efficiency and adaptability problems of traditional systolic arrays when processing sparse matrices. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a sparse systolic array system that supports adaptive delay, thereby solving the technical problem that the fixed data flow path of a systolic array cannot adapt to the irregular distribution of non-zero elements in a sparse matrix.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a sparse systolic array system supporting adaptive delay is provided, comprising: Input control and mode management module, pulse array calculation module, adaptive delay controller and result buffer; The input control and mode management module is used to configure the system to a general matrix multiplication mode or a sparse matrix multiplication mode based on the matrix format of matrices A and B to be multiplied, and to input matrices A and B to the systolic array computing module. In the sparse matrix multiplication mode, the input control and mode management module is used to send non-zero elements of matrix A from the left boundary of the systolic array computing module, and pass each row of non-zero elements to the right along the row in each clock cycle between processing units. Matrix B is sent from the top boundary of the systolic array computing module, and each column of data is passed down along the column in each clock cycle between processing units. The pulsating array calculation module consists of M rows and N columns of processing units. Each processing unit is used to perform multiplication and accumulation calculations on the matrix elements that are sequentially input to the corresponding processing unit. The adaptive delay controller operates in sparse matrix multiplication mode, and is used to calculate the upper bound of the output delay based on the statistical information of the non-zero elements in each row of matrix A, and generate a control signal based on the upper bound of the output delay; the control signal is used to terminate the emptying phase of the pulsating array calculation module in advance; the statistical information of the non-zero elements in each row of matrix A includes the total span of the non-zero index of the corresponding row, the total number of non-zero elements, and the maximum value of the difference between adjacent non-zero element indices. The result buffer is used to receive the calculation results of each processing unit in the pulsating array calculation module.

[0007] Based on the aforementioned sparse systolic array system supporting adaptive delay, the upper bound of the output delay is calculated using the following formula:

[0008] in, Let N be the upper bound of the output delay, and N be the number of columns in matrix B. Let be the minimum output delay for the i-th row. , The total span of the non-zero index in the i-th row. Let be the total number of non-zero elements in the i-th row. It represents the maximum difference between the indices of adjacent non-zero elements in the i-th row. This is the default value.

[0009] According to the above-mentioned sparse pulsating array system that supports adaptive delay, the input control and mode management module specifically includes a row compression circuit, a statistical circuit, and a B matrix controller. In sparse matrix multiplication mode, the row compression circuit is used to read the row pointers of matrix A to obtain the total number of non-zero elements nnz in each row. i And the total number of non-zero elements in each row, nnz i Load the data into the counter row_cnt[i] of the corresponding row, and then send the column index and valid value of each non-zero element in each row as a continuous data stream to the processing unit of the corresponding row in the pulsating array calculation module; The statistical circuit calculates the total span of the non-zero indices in each row of matrix A in real time based on the column index and valid value of each non-zero element in each row output by the row compression circuit. i The maximum value of the difference between the indices of adjacent non-zero elements: gap i and the total span of the non-zero indexes in each row. i The maximum value of the difference between the indices of adjacent non-zero elements: gap iIt is latched into a set of output registers and transmitted to the adaptive delay controller; The B matrix controller uses a global counter k_cnt to send the column data of the B matrix into the top boundary of the pulsating array calculation module step by step, in the order of k_cnt from 0 to K-1, where K is the number of columns of matrix B.

[0010] According to the above-described sparse systolic array system that supports adaptive delay, the processing unit has a built-in buffer BufferA; In sparse matrix multiplication mode, within each clock cycle, each processing unit performs an index matching operation on the matrix A elements input to the corresponding processing unit or the head of BufferA and the matrix B elements input to the corresponding processing unit; wherein, when BufferA is not empty, the index matching operation is performed on the matrix A elements at the head of BufferA first, and the matrix A elements currently input to the processing unit are stored in BufferA; If the index match is successful, multiply-accumulate calculation is performed on the corresponding matrix A element and matrix B element, and a pop signal is sent to BufferA when the corresponding matrix A element comes from BufferA to release the corresponding matrix A element. At the same time, a decrementing pulse is sent to the counter row_cnt[i] of the row where the processing unit is located. If the index match is unsuccessful, the corresponding element of matrix A is placed into or held in BufferA.

[0011] Based on the aforementioned sparse systolic array system supporting adaptive delay, the signal value of the control signal is determined according to the following formula:

[0012]

[0013] Wherein, Termination is the control signal; when the BufferA of all processing units in the pulsating array computing module is empty. The value is 0; when all multiplication and accumulation calculations of all processing units in the pulsating array calculation module have been completed, =1; The clock cycles elapsed from the input of the last data of matrix B into the pulsating array calculation module to the current moment are calculated. This is the upper bound of the output delay.

[0014] According to the above-mentioned sparse systolic array system that supports adaptive latency, for any processing unit in the systolic array computing module, if it is detected that any processing unit has completed all calculations, the calculation result of any processing unit is output to the result buffer in advance.

[0015] Based on the aforementioned sparse systolic array system supporting adaptive latency, whether any processing unit has completed all computations is determined using the following formula:

[0016] Where i and j are the row and column corresponding to any of the processing units. A value of 1 indicates that any processing unit has completed all calculations; when the counter row_cnt[i] of the row containing any processing unit is 0, row_done[i] is high. When k_cnt reaches K, a column completion signal propagates down the column from the top of the pulsating array computing module. When any processing unit receives the column completion signal, col_done[j] is latched to a high level.

[0017] Based on the aforementioned sparse systolic array system supporting adaptive delay, in sparse matrix multiplication mode, the enable signal for any processing unit is generated according to the following formula:

[0018] Where i and j are the row and column corresponding to any of the processing units, CE i,j This is the enable signal for any of the processing units; when it is 1, the processing unit is activated. `buffers_nonempty` is a waiting flag, which is high when `BufferA` inside any of the processing units is not empty; `in_flight_match` is a calculation signal, which is high when any of the processing units performs an index matching operation and the index match is successful; `flush_phase` is a global flushing status signal, which is a global control signal. After the last data of matrix B is input to the pulsating array calculation module, the adaptive delay controller sets it to high and broadcasts it to all processing units until the flushing phase ends.

[0019] Based on the aforementioned sparse systolic array system supporting adaptive delay, in the general matrix multiplication mode, the enable signal of the processing unit is generated based on the following formula:

[0020] Where i and j are the row and column corresponding to any of the processing units, CE i,j (t) is the enable signal for any processing unit during the t-th clock cycle; when it is 1, the processing unit is activated. K is the number of columns in matrix A, d pipe To reduce pipeline delays within the processing unit.

[0021] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: The sparse systolic array system with adaptive delay provided by this invention configures the system to either a general matrix multiplication mode or a sparse matrix multiplication mode based on the matrix formats of matrices A and B to be multiplied, and inputs matrices A and B to the systolic array computing module to accommodate different computing needs. The system comprises an M-row, N-column processing unit, where each processing unit performs multiplication and accumulation calculations on the matrix elements sequentially input to the corresponding processing unit. An adaptive delay controller is provided to calculate the upper bound of the output delay based on the statistical information of the non-zero elements in each row of matrix A in sparse matrix multiplication mode, and generates a control signal based on this upper bound to prematurely terminate the emptying phase of the systolic array computing module. This effectively reduces the emptying delay in sparse matrix multiplication mode and improves the adaptability between systolic arrays and sparse matrix characteristics.

[0022] Furthermore, by setting up a buffer and implementing an index matching mechanism within the processing unit, configuring different clock enable logic for the processing unit in different modes, and allowing the processing unit to submit its calculation results out of order, it is possible to ensure synchronous output under different sparsity conditions, and reduce the power consumption of the processing unit and the delay of the entire emptying stage. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the structure of a sparse systolic array system supporting adaptive delay provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the data flow under the general matrix multiplication mode provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of the data flow under the sparse matrix multiplication mode provided in an embodiment of the present invention; Figure 4 This is a circuit diagram illustrating support for floating-point multiplication and accumulation operations provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the operation of the processing unit in the conventional mode provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the operation of the processing unit under clock-enabled logic provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0025] This invention provides a sparse systolic array system that supports adaptive latency, such as... Figure 1 As shown, it includes: Input control and mode management module, pulse array calculation module, adaptive delay controller and result buffer.

[0026] The input control and mode management module configures the system to either a general matrix multiplication mode or a sparse matrix multiplication mode based on the matrix format (including but not limited to dense and row-compressed formats) of matrices A and B to be multiplied, and inputs matrices A and B to the systolic array calculation module. Specifically, when both input matrices A and B are dense matrices, the input control and mode management module configures the system to the general matrix multiplication mode; when matrix A is a sparse matrix, the module configures the system to the sparse matrix multiplication mode.

[0027] In the general matrix multiplication mode, the input control and mode management module uses the traditional pulsating array input method to input matrix A and matrix B into the pulsating array calculation module. The data flow in this mode is as follows: Figure 2 As shown, data for matrix A is fed in from the left boundary of the systolic array computing module, and each row of data is passed between processing units along the row to the right in each clock cycle. Data for matrix B is fed in from the top boundary of the systolic array computing module, and each column of data is passed between processing units along the column to the bottom in each clock cycle.

[0028] In sparse matrix multiplication mode, the data flow is as follows: Figure 3 As shown, the input control and mode management module sends the non-zero elements of matrix A from the left boundary of the systolic array calculation module. The non-zero elements of each row are passed to the right along the row in each clock cycle between processing units. At the same time, matrix B is sent from the top boundary of the systolic array calculation module in the traditional way. The data of each column is passed down along the column in each clock cycle between processing units.

[0029] In some embodiments, the input control and mode management module specifically includes a row compression circuit, a statistical circuit, and a B matrix controller to perform data input and key information statistics in sparse matrix multiplication mode. Specifically, in sparse matrix multiplication mode, the row compression circuit is used to read the row pointers of matrix A to obtain the total number of non-zero elements nnz in each row.i And the total number of non-zero elements in each row, nnz i The data is loaded into the counter row_cnt[i] of the corresponding row, and then the column index and valid value of each non-zero element in each row are sent as a continuous data stream to the processing unit of the corresponding row in the systolic array calculation module; the statistical circuit calculates the total span of the non-zero index of each row in matrix A in real time based on the column index and valid value of each non-zero element in each row output by the row compression circuit. i The maximum value of the difference between the indices of adjacent non-zero elements: gap i and the total span of the non-zero indexes in each row. i The maximum value of the difference between the indices of adjacent non-zero elements: gap i The data is latched into a set of output registers and transmitted to the adaptive delay controller for calculating the upper bound of the output delay (the specific calculation method will be described in detail later). The B matrix controller uses a global counter k_cnt to send the column data of the B matrix into the top boundary of the pulsating array calculation module step by step, according to the order of k_cnt from 0 to K-1. K is the number of columns of matrix B.

[0030] In other embodiments, to ensure high utilization of the systolic array computing module and eliminate computational "bubbles" caused by data loading, the input control and mode management module further integrates a double-buffering (Ping-Pong) mechanism. This double-buffering mechanism includes at least two data buffers (e.g., BankA and BankB). During operation, while the systolic array computing module is consuming matrix data in BankA, the input control and mode management module is configured to prefetch the next batch of matrix data from system memory or the previous level cache in parallel and store it in BankB. When the data in BankA is exhausted, the input control and mode management module immediately switches the data path, allowing the systolic array computing module to seamlessly begin consuming data in BankB. This ping-pong operation hides the time overhead required for data prefetching, thereby ensuring the continuity of data flow to the systolic array and significantly improving the effective utilization of the array.

[0031] The pulsating array computation module consists of M rows and N columns of processing units (PEs). Each processing unit performs multiplication and accumulation calculations on the matrix elements sequentially input to the corresponding processing unit, and outputs the calculation results to a result buffer during the emptying phase. In some embodiments, each processing unit supports multi-precision calculations in INT8, INT16, and INT32 integer formats and FP16 and FP32 floating-point formats. The floating-point calculations use fused multiplication and accumulation instructions. Furthermore, the processing unit can optionally support single-instruction multiple-data (SIMD) packing operations in low-precision mode, or support high-precision accumulation followed by quantization output. Here, since the processing units used in the pulsating array do not support multi-precision calculations in floating-point formats such as FP16 and FP32, this embodiment provides a circuit that supports floating-point multiplication and accumulation operations within the processing unit. Figure 4 As shown, the circuit supporting floating-point multiplication and accumulation operations consists of a preprocessing module, an alignment module, a multiplication module, an accumulation and arithmetic unit, and a normalization module. It is used to implement floating-point multiplication and accumulation operations and output floating-point multiplication and accumulation results and exception flags that conform to the IEEE 754 standard.

[0032] The preprocessing module includes three input terminals for floating-point operands A, B, and C. The preprocessing module decodes the three input floating-point operands, separating the sign bit S from each. A S B S C Exponent E A E B E C and the mantissa M A M B M C Three components.

[0033] The alignment module is used to align the exponent E of A. A and mantissa M A Make adjustments to align it with E B +E C The results are aligned. Specifically, it can be based on E. B and E C The sum of E is calculated. A The offset to be adjusted, and its mantissa M A Perform a shift to achieve the desired result for the exponent E. A Adjustments.

[0034] The multiplication module is used to complete M B and M C The multiplication operation can employ Booth encoding combined with a Wallace tree architecture to achieve efficient partial product generation and compression. Specifically, the multiplication module contains the following sub-modules: Booth encoding: used for MB and M C Booth encoding is used to transform traditional binary multiplication, which involves multiple sets of full additions, into fewer sets of additions / subtractions plus shifts. This reduces the number of partial products and lowers the complexity and latency of multiplication operations. According to M... B The Booth encoding result for M C Perform the corresponding translation operation to generate the partial product sequence to be accumulated. In an embodiment of the present invention, Radix-4 encoding can be used, and its working principle is as follows: For any binary signed number R, it is transformed according to the following:

[0035] Among them, R i Let be the i-th bit of the signed binary number R. After the transformation, the number of summations changes from n-2 to . This reduces the number of summation terms by half, significantly lowering computational resource costs. For multiplication R×S, , , The partial product with the multiplier S under different conditions is given in Table 1.

[0036] Table 1

[0037] Wallace Tree: By compressing the partial products generated by the Booth encoding module through a multi-layer parallel adder, multiple sets of partial products are quickly merged into a two-dimensional vector of sum + carry, which greatly reduces the critical path delay of multiplication operations.

[0038] The accumulation and arithmetic units are used to align the M A With M B and M C The multiplication results are accumulated, and arithmetic adjustments, sign handling, special value detection, and leading zero detection are performed. Internally, it includes a submodule that stores the carry adder, used to add the multiplication result to M. A The low-order mantissa is accumulated in parallel to generate an intermediate sum and a carry vector; the adder module is used to perform carry addition or subtraction operations and process the priority logic of floating-point special values ​​(infinity, zero, NOT digits), and transmit the result to the zero detector; the zero detector is used to calculate the displacement of the final result. In addition, the accumulation and arithmetic unit also includes an incrementer, which is used to incrementally adjust a specific part of the exponent or mantissa, and output the normalized exponent E and mantissa M.

[0039] The normalization module is used to convert the normalization exponent E, normalization mantissa M, sign S, and other information output by the accumulation and arithmetic units into a final floating-point result conforming to the IEEE 754 standard, and to perform special value processing and rounding.

[0040] As can be seen, the above circuit achieves efficient floating-point operation of A+B×C through a pipeline architecture of "preprocessing-alignment-multiplication-accumulation arithmetic-normalization". It optimizes the operation latency and accuracy through Booth encoding, Wallace tree, leading zero detection and other techniques, and is suitable for high-performance floating-point operation scenarios.

[0041] After the input control and mode management module inputs matrices A and B into the systolic array calculation module, each processing unit performs multiplication and accumulation calculations on the matrix elements input to the corresponding processing unit every clock cycle, and outputs the calculation results to the result buffer during the emptying phase. Specifically, when the last data of matrix B is sent to the systolic array calculation module, the entire systolic array calculation module enters the emptying phase, which typically ends after a fixed number of clock cycles (i.e., (M-1) + (N-1) clock cycles). However, in sparse matrix multiplication mode, due to the large number of zero values ​​in the sparse matrix, the entire systolic array calculation module may prematurely end the output of the calculation results, and the aforementioned traditional fixed-cycle emptying phase design would lead to wasted time. To address this, in order to compress the emptying phase, this embodiment of the invention sets up an adaptive delay controller. This adaptive delay controller only operates in sparse matrix multiplication mode and is used to calculate the upper bound of the output delay based on the statistical information of the non-zero elements in each row of matrix A. Based on this upper bound, a control signal is generated, thereby prematurely ending the emptying phase of the systolic array calculation module based on this control signal. The statistical information of non-zero elements in each row of matrix A includes the total span of the non-zero index of the corresponding row, the total number of non-zero elements, and the maximum value of the difference between adjacent non-zero element indices.

[0042] Specifically, the upper bound of the output delay includes a fixed physical delay and a variable data delay. The fixed physical delay is determined by the fixed number of cycles required for the data in matrix B to traverse the systolic array computation module, which is determined by the number of columns in the systolic array computation module, i.e., (N-1) clock cycles. The variable data delay depends on the sparsity pattern of the row with the most valid values ​​(i.e., non-zero elements) in matrix A. To calculate this variable data delay, the adaptive delay controller determines the output delay of each row based on the total span of the non-zero indices, the total number of non-zero elements, and the maximum difference between adjacent non-zero element indices in each row of matrix A, and selects the maximum value (corresponding to the delay contribution of the slowest row) as the variable data delay. The total span of the non-zero indices in each row... iDefined as the difference between the index of the furthest non-zero element and the index of the nearest non-zero element in the row, it represents the effective dimensional range of the row. The maximum value of the total number of non-zero elements in each row and the index difference between adjacent non-zero elements can be used to estimate the most compact output delay for that row. Therefore, the smaller value between the effective dimensional range of the row and the most compact output delay can be selected as the variable data delay. Based on the above upper bound of the output delay, a control signal can be generated to terminate the emptying phase of the systolic array calculation module before the above upper bound of the output delay. This "asynchronous emptying" mechanism is used as the main emptying termination criterion, replacing the traditional fixed-delay emptying.

[0043] In some embodiments, the upper bound of the output delay can be calculated based on the following formula:

[0044] in, The upper bound for the output delay is given, and N is the number of columns in matrix B. Let be the minimum output delay for the i-th row. , The total span of the non-zero index in the i-th row. Let be the total number of non-zero elements in the i-th row. The maximum value of the index difference between adjacent non-zero elements in the i-th row; For example, the default value. ∈{1,2} is used to compensate for pipeline delays within processing units or between modules, ensuring absolute conservatism in the upper bound calculation.

[0045] The superiority of the adaptive delay controller design is illustrated below with a specific numerical example: Taking M=N=8, K=8, and τ0=1 as an example, for a traditional systolic array, its emptying delay (i.e., the duration of the emptying phase) is fixed at (M-1)+(N-1)=7+7=14 clock cycles. Regardless of how sparse the input matrix is, it must consume these 14 clock cycles to obtain the result. Assume the input matrix A has a typical sparse distribution, with its non-zero index distribution as follows:

[0046]

[0047]

[0048] The maximum value of R is calculated. i =3. Therefore, the upper bound of the output delay Δ calculated in this embodiment of the invention is 3. est=(8-1)+3+1=11 clock cycles. Therefore, under this typical sparse distribution, the drain delay can be shortened from 14 clock cycles in the traditional design to 11 clock cycles, achieving a drain stage performance improvement of over 21% without sacrificing any reliability. Furthermore, this upper bound is safe: even in near-worst distributions, such as R... i =7, calculate Δ est =7+7+1=15 beats, which may be slightly higher than the 14 beats of the traditional fixed emptying; this is the redundancy brought about by the conservative upper bound design, used to ensure that the termination criterion is not triggered incorrectly. In actual operation, array_quiet usually triggers termination before this upper bound, so the average emptying delay is still significantly better than fixed emptying.

[0049] In sparse matrix multiplication mode, considering that the input control and mode management module only sends non-zero elements of matrix A to the systolic array calculation module, while for matrix B it sends all data of matrix B to the systolic array calculation module, the elements of matrix A and matrix B arriving at a certain processing unit in the same clock cycle may not match (i.e., the column index of the element of matrix A is different from the row index of the element of matrix B), and therefore cannot be multiplied and accumulated. To address this, in some embodiments, each processing unit in the systolic array calculation module has a built-in buffer BufferA. In sparse matrix multiplication mode, within each clock cycle, such as... Figure 3 As shown, each processing unit performs an index matching operation on the matrix A elements input to the corresponding processing unit or the head of BufferA and the matrix B elements input to the corresponding processing unit. Specifically, when BufferA is not empty, the index matching operation is first performed on the matrix A elements at the head of BufferA (i.e., determining whether the column index of matrix A elements is the same as the row index of matrix B elements), and the matrix A elements currently input to the processing unit are stored in BufferA. If the index matching is successful, a multiplication-accumulation calculation is performed on the corresponding matrix A elements and matrix B elements, and a pop signal is sent to BufferA when the corresponding matrix A element comes from BufferA, releasing the corresponding matrix A element. Simultaneously, a decrementing pulse is sent to the counter row_cnt[i] of the row where the processing unit is located. If the index matching fails, the corresponding matrix A element is placed in or held in BufferA.

[0050] In another embodiment, to address the mismatch issue caused by asynchronous arrival of matrix A elements and matrix B elements at the processing unit (PE) due to factors such as arbitration delays, the processing unit adds a buffer B to the buffer BufferA to temporarily store matrix B elements that arrive before the matching matrix A elements. Since matrix B elements are input into the pulsating array column-by-column, the system configures a corresponding column counter col_cnt[j] for each column to record the number of matrix B elements to be matched in that column.

[0051] In some embodiments, col_cnt[j] is the backlog counter on the BufferB side for the j-th column: it is cleared to zero upon reset or the start of a new round of calculation; col_cnt[j] is incremented when a matrix B element with column index j is written to BufferB; and col_cnt[j] is decremented when a matrix B element with column index j is popped from BufferB and consumed for multiplication and accumulation. Therefore, col_cnt[j] = 0 indicates that there are no unmatched backlog elements in that column on the BufferB side.

[0052] In this scenario, the specific matching logic is as follows: In each clock cycle, the processing unit performs index matching on the elements of the currently input matrices A and B, or the elements of matrices A and B at the beginning of BufferA and BufferB. If BufferA or BufferB is not empty, the system first attempts to match the elements at the beginning of the non-empty buffer and stores the new element currently input into the corresponding buffer. If the match is successful, the processing unit performs a multiply-accumulate operation and performs a pop operation based on the element's source: if the currently matched element of matrix A comes from BufferA, the element is popped from BufferA and the row counter row_cnt[i] of its row is decremented; if the currently matched element of matrix B comes from BufferB, the element is popped from BufferB and the column counter col_cnt[j] of its column is decremented. If the match fails, the current input element (if the element being matched is the current input element) is stored in the corresponding buffer or the buffer state is maintained (if the element being matched comes from the beginning of the buffer), and matching continues in subsequent cycles.

[0053] Based on this, the signal value of the control signal can be determined using the following formula:

[0054]

[0055] Wherein, Termination is the control signal, and is a high-active termination indication signal; when Termination=1, it indicates that the current round of calculation is completed and the termination / end of the emptying phase is triggered; when Termination=0, it indicates that the array continues to perform the matching, multiplication, accumulation, and emptying process; when the BufferA of all processing units in the pulsating array calculation module is empty, The value is 0; when all multiplication and accumulation calculations of all processing units in the pulsating array calculation module have been completed, =1; The clock cycles elapsed from the last data input to the systolic array calculation module in matrix B to the current moment. This is the upper bound of the output delay.

[0056] In some embodiments, `array_quiet` is a global idle criterion for the systolic array computation module, used to characterize that the current round of sparse matrix multiplication no longer generates new valid multiply-accumulate calculations and result write-back requests. Specifically, `array_quiet` can be determined by the following conditions: (1) BufferA of all processing units is empty (representing that there are no more non-zero elements to be matched / consumed on the A side of the matrix). (2) The counter row_cnt[i] of all processing units is 0 (indicating that all non-zero elements of matrix A have been consumed and the matching calculation has been completed); (3) All processing units' col_done[j] are latched to high level (indicating that the data stream in column B of matrix has completed propagation and there is no subsequent valid input); (4) An empty result buffer or write-back arbitration queue indicates that all committable results have been written back.

[0057] When the above conditions are met, array_quiet is considered valid, thus allowing the adaptive delay controller to terminate the emptying phase early.

[0058] It should be noted that in the embodiment where BufferB is enabled, BufferB is used to temporarily store matrix B elements that arrive first but have not yet been matched; when row_cnt[i] are all 0, the remaining elements in BufferB no longer participate in the effective multiplication and accumulation calculation, so whether BufferB is empty is not a necessary condition for array_quiet to be true.

[0059] If the array_quiet signal fails to trigger for an extended period, Timer_flush will exceed the upper bound of the output delay, forcibly terminating the flush. Under normal circumstances, array_quiet always occurs after Timer_flush reaches Δ... est Previously, Termination was triggered, thus minimizing the drain latency. In the example above, although Δ est The calculation is 11 beats, but because the matrix is ​​highly sparse (max(R)... i With a clock cycle count of 3, all processing units are likely to have completed the output of all calculation results by the 5th or 6th clock cycle after the global counter k_cnt reaches K-1. At this point, the adaptive delay controller immediately enters the termination and emptying phase and issues a calculation completion signal. The actual emptying delay is shortened to the 5th or 6th clock cycle after the start of the emptying phase, which is more than double the performance compared to the 14-cycle delay of the traditional design.

[0060] To further reduce output latency, the adaptive delay controller also supports out-of-order submission. That is, for any processing unit in the systolic array computation module, if it is detected that the processing unit has completed all computations, the computation result of that processing unit can be output to the result buffer in advance without waiting for the emptying stage. In some embodiments, the result buffer adopts an addressing storage method based on the coordinates of the output matrix. Specifically, a mapping relationship is established between the result generated by the processing unit PE(i,j) and the address addr(i,j) in the result buffer (e.g., using a linear mapping function addr=i×N+j). This mapping mechanism decouples the computation completion order of the processing units from the write position, allowing each processing unit to write back results out of order without affecting the correctness of the data arrangement of the output matrix.

[0061] When multiple processing units initiate write-back requests within the same clock cycle, the arbitration logic allocates write-back ports based on a polling or fixed priority strategy. For processing units that do not receive write-back authorization, a handshake protocol can be used to apply backpressure to latch their results in the output register, or the results can be temporarily stored in the submission queue and written back in a subsequent idle cycle.

[0062] In some embodiments, whether any processing unit has completed all calculations can be determined based on the following formula:

[0063] Here, i and j are the row and column corresponding to this processing unit. A value of 1 indicates that the processing unit has completed all calculations, and the calculation result of the processing unit can be output to the result buffer in advance. Specifically, when the counter row_cnt[i] of the row where the processing unit is located is 0, row_done[i] is high; when k_cnt reaches K, a column completion signal propagates from the top of the systolic array calculation module down the column, and when the processing unit receives the column completion signal, col_done[j] is latched high.

[0064] In the sparse matrix multiplication processing flow, processing units spend a significant amount of time in an idle state. To avoid power consumption due to idle switching, this embodiment of the invention also provides a clock enable logic in the sparse matrix multiplication mode to control the activation state of the processing units. Specifically, the enable signal for any processing unit is generated based on the following formula:

[0065] Here, i and j are the row and column corresponding to this processing unit, CE i,j This is the enable signal for the processing unit. When it is 1, the processing unit is activated; otherwise, the processing unit remains in a low-power silent state.

[0066] Here, `buffers_nonempty` is a waiting flag, which goes high when `BufferA` inside the processing unit is not empty; `in_flight_match` is a calculation signal, which goes high when the processing unit performs an index matching operation and the index match is successful, ensuring that the high-power arithmetic logic is only activated when performing valid calculations; `flush_phase` is a global flushing status signal, a global control signal that is set high by the adaptive delay controller after the last data input of matrix B is entered into the pulsating array calculation module and broadcast to all processing units until the flushing phase ends. This global flushing status signal ensures that the processing unit remains operational during the flushing phase to ensure the correct processing of remaining data, clearing of buffers, and propagation of the processing unit's idle state to the global `array_quiet` logic, thereby ensuring the reliability of asynchronous flushing. Figure 5 and Figure 6 The diagrams illustrate the operation of the processing unit in the traditional mode and under the clock-enabled logic described above. Taking M=N=3 as an example, black represents the processing unit in an active state, and gray represents the processing unit in a low-power silent state. The comparison shows that the clock-enabled logic provided in this embodiment can set an idle processing unit to a gated silent state, reducing a significant amount of unnecessary computation.

[0067] In other embodiments, under the general matrix multiplication mode, each processing unit is typically kept continuously active. The computation completion time of the systolic array computing module is determined by both its fill latency and computation depth. For an M×N systolic array computing module, the total number of cycles (Total Cycles) for computing an M×K and K×N matrix multiplication can be expressed as: Total Cycles = (M-1) + (N-1) + K For example, in an embodiment where two 8×8 matrices are calculated in an 8×8 pulsating array computing module, the total number of completed cycles is: (8-1)+(8-1)+8=22 cycles.

[0068] Considering that the processing units in the systolic array computing module enter and complete their work sequentially, to achieve the most precise power consumption gating, some embodiments also provide a clock enable control logic for the general matrix multiplication mode to precisely control the enabling of each processing unit, thereby reducing power consumption. Specifically, this clock enable control logic generates a precise enable signal CE for each PE(i,j) (where i,j are the row and column coordinates of PE) at time t. i,j (t), its calculation formula can be expressed as:

[0069] Where i+j is the moment when the processing unit begins its first calculation, K is the number of columns in matrix A, and d pipe This is for pipeline delay within the processing unit. When CE i,j The processing unit is activated when (t) is 1; otherwise, it is not activated.

[0070] The above formula intuitively shows that the working window of PE(i,j) starts from the time it is activated, t=i+j, and lasts exactly K+d. pipe One clock beat.

[0071] Multiplying two 8×8 dense matrices (assuming d) pipe For example, (=0): When i=0 and j=0, the enable window of PE(0,0) is t≥0∧t<8, that is, it is enabled in the 0th to 7th time frame.

[0072] When i=7 and j=7, the enable window for PE(7,7) is t≥14∧t<22, that is, it is enabled from the 14th to the 21st beat.

[0073] In this way, once PE(0,0) completes its K calculations after the 7th cycle, its CE signal will automatically go low on the 8th cycle, causing the PE to immediately enter a low-power silent state without any additional completion status detection logic. This method is consistent with the formula (M-1)+(N-1)+K for the total number of array completion cycles of 22 cycles, achieving the most precise and power-saving control.

[0074] In addition, in the general matrix multiplication mode, different power consumption optimization strategies can be adopted according to the current computational precision: In integer (INT8 / INT16 / INT32) arithmetic mode: in CE i,j During the period when (t) is high, if an element of matrix A or matrix B is detected to be zero, the multiplication module of the processing unit will be further controlled to prevent level flipping. Due to the determinism of integer zero (N×0=0), this operation can be performed safely, significantly reducing dynamic power consumption while maintaining the synchronous transmission of zero values ​​by the processing unit.

[0075] In floating-point (FP16 / FP32) arithmetic mode: Considering that floating-point operations must correctly handle special values, such as Inf × 0.0 or NaN × 0.0, in CE... i,j During the period when (t) is high, if an element of matrix A or matrix B is detected to be zero, the exponent of the non-zero operand will be checked synchronously to determine whether the non-zero operand is a special value, such as infinity, NAN, or an out-of-normal number. If the non-zero operand is a special value, the multiplication module of the control processing unit will not produce a level flip, but will directly output the preset calculation result that conforms to the IEEE 754 arithmetic rules.

[0076] In summary, the sparse systolic array system supporting adaptive delay provided by this embodiment of the invention configures the system to either a general matrix multiplication mode or a sparse matrix multiplication mode based on the matrix formats of matrices A and B to be multiplied, and inputs matrices A and B to the systolic array calculation module to accommodate different computational needs. The system is configured with an M-row, N-column systolic array calculation module, where each processing unit performs multiplication and accumulation calculations on the matrix elements sequentially input to the corresponding processing unit. An adaptive delay controller is provided to calculate the upper bound of the output delay based on the statistical information of the non-zero elements in each row of matrix A in sparse matrix multiplication mode, and generates a control signal based on this upper bound to prematurely terminate the emptying phase of the systolic array calculation module. This effectively reduces the emptying delay in sparse matrix multiplication mode and improves the adaptability between systolic arrays and sparse matrix characteristics. Furthermore, by setting up a buffer and implementing an index matching mechanism within the processing unit, different clock enable logics are configured for the processing unit in different modes. At the same time, the processing unit is allowed to submit its calculation results out of order, which can ensure that the output can be synchronized in step by step under different sparsity, and reduce the power consumption of the processing unit and the delay of the entire emptying stage.

[0077] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A sparse systolic array system supporting adaptive delay, characterized in that, include: Input control and mode management module, pulse array calculation module, adaptive delay controller and result buffer; The input control and mode management module is used to configure the system to a general matrix multiplication mode or a sparse matrix multiplication mode based on the matrix format of matrices A and B to be multiplied, and to input matrices A and B to the systolic array computing module. In the sparse matrix multiplication mode, the input control and mode management module is used to send non-zero elements of matrix A from the left boundary of the systolic array computing module, and pass each row of non-zero elements to the right along the row in each clock cycle between processing units. Matrix B is sent from the top boundary of the systolic array computing module, and each column of data is passed down along the column in each clock cycle between processing units. The pulsating array calculation module consists of M rows and N columns of processing units. Each processing unit is used to perform multiplication and accumulation calculations on the matrix elements that are sequentially input to the corresponding processing unit. The adaptive delay controller operates in sparse matrix multiplication mode, and is used to calculate the upper bound of the output delay based on the statistical information of the non-zero elements in each row of matrix A, and generate a control signal based on the upper bound of the output delay; the control signal is used to terminate the emptying phase of the pulsating array calculation module in advance; the statistical information of the non-zero elements in each row of matrix A includes the total span of the non-zero index of the corresponding row, the total number of non-zero elements, and the maximum value of the difference between adjacent non-zero element indices. The result buffer is used to receive the calculation results of each processing unit in the pulsating array calculation module.

2. The sparse systolic array system supporting adaptive delay as described in claim 1, characterized in that, The upper bound of the output delay is calculated based on the following formula: in, Let N be the upper bound of the output delay, and N be the number of columns in matrix B. Let be the minimum output delay for the i-th row. , The total span of the non-zero index in the i-th row. Let be the total number of non-zero elements in the i-th row. It represents the maximum difference between the indices of adjacent non-zero elements in the i-th row. This is the default value.

3. The sparse systolic array system supporting adaptive delay as described in claim 2, characterized in that, The input control and mode management module specifically includes a line compression circuit, a statistical circuit, and a B-matrix controller; In sparse matrix multiplication mode, the row compression circuit is used to read the row pointers of matrix A to obtain the total number of non-zero elements nnz in each row. i And the total number of non-zero elements in each row, nnz i Load the data into the counter row_cnt[i] of the corresponding row, and then send the column index and valid value of each non-zero element in each row as a continuous data stream to the processing unit of the corresponding row in the pulsating array calculation module; The statistical circuit calculates the total span of the non-zero indices in each row of matrix A in real time based on the column index and valid value of each non-zero element in each row output by the row compression circuit. i The maximum value of the difference between the indices of adjacent non-zero elements: gap i and the total span of the non-zero indexes in each row. i The maximum value of the difference between the indices of adjacent non-zero elements: gap i It is latched into a set of output registers and transmitted to the adaptive delay controller; The B matrix controller uses a global counter k_cnt to send the column data of the B matrix into the top boundary of the pulsating array calculation module step by step, in the order of k_cnt from 0 to K-1, where K is the number of columns of matrix B.

4. The sparse systolic array system supporting adaptive delay as described in claim 3, characterized in that, The processing unit has a built-in buffer BufferA; In sparse matrix multiplication mode, within each clock cycle, each processing unit performs an index matching operation on the matrix A elements input to the corresponding processing unit or the head of BufferA and the matrix B elements input to the corresponding processing unit; wherein, when BufferA is not empty, the index matching operation is performed on the matrix A elements at the head of BufferA first, and the matrix A elements currently input to the processing unit are stored in BufferA; If the index match is successful, multiply-accumulate calculation is performed on the corresponding matrix A element and matrix B element, and a pop signal is sent to BufferA when the corresponding matrix A element comes from BufferA to release the corresponding matrix A element. At the same time, a decrementing pulse is sent to the counter row_cnt[i] of the row where the processing unit is located. If the index match is unsuccessful, the corresponding element of matrix A is placed into or held in BufferA.

5. The sparse systolic array system supporting adaptive delay as described in claim 4, characterized in that, The signal value of the control signal is determined based on the following formula: Wherein, Termination is the control signal; when the BufferA of all processing units in the pulsating array computing module is empty. The value is 0; when all multiplication and accumulation calculations of all processing units in the pulsating array calculation module have been completed, =1; The clock cycles elapsed from the input of the last data of matrix B into the pulsating array calculation module to the current moment are calculated. This is the upper bound of the output delay.

6. The sparse systolic array system supporting adaptive delay as described in claim 3, characterized in that, For any processing unit in the pulsating array calculation module, if it is detected that any processing unit has completed all calculations, the calculation result of any processing unit is output to the result buffer in advance.

7. The sparse systolic array system supporting adaptive delay as described in claim 6, characterized in that, Whether any processing unit has completed all calculations is determined based on the following formula: Where i and j are the row and column corresponding to any of the processing units. A value of 1 indicates that any processing unit has completed all calculations; when the counter row_cnt[i] of the row containing any processing unit is 0, row_done[i] is high. When k_cnt reaches K, a column completion signal propagates down the column from the top of the pulsating array computing module. When any processing unit receives the column completion signal, col_done[j] is latched to a high level.

8. The sparse systolic array system supporting adaptive delay as described in claim 4, characterized in that, In sparse matrix multiplication mode, the enable signal for any processing unit is generated based on the following formula: Where i and j are the row and column corresponding to any of the processing units, CE i,j This is the enable signal for any of the processing units; when it is 1, the processing unit is activated. `buffers_nonempty` is a waiting flag, which is high when `BufferA` inside any of the processing units is not empty; `in_flight_match` is a calculation signal, which is high when any of the processing units performs an index matching operation and the index match is successful; `flush_phase` is a global flushing status signal, which is a global control signal. After the last data of matrix B is input to the pulsating array calculation module, the adaptive delay controller sets it to high and broadcasts it to all processing units until the flushing phase ends.

9. The sparse systolic array system supporting adaptive delay as described in claim 1, characterized in that, In the general matrix multiplication mode, the enable signal of the processing unit is generated based on the following formula: Where i and j are the row and column corresponding to any of the processing units, CE i,j (t) is the enable signal for any processing unit during the t-th clock cycle; when it is 1, the processing unit is activated. K is the number of columns in matrix A, d pipe To reduce pipeline delays within the processing unit.

Images