Fine-grained quantized matrix multiplication apparatus and method based on systolic array

Abstract

The present disclosure relates to the technical field of computer system, and provides a fine-grained quantization matrix multiplication device and method based on systolic array, which comprises: an input scheduling module, configured to receive input matrix data and corresponding quantization factor data, and to attach type identifiers to the input matrix data and the quantization factor data; a systolic array, connected by multiple processing units in a two-dimensional mesh structure, and each processing unit is configured to select to perform a multiply-accumulate operation or a dequantization operation according to the data received from an upstream and the type identifier of the data; wherein the input scheduling module is configured to schedule the data with the attached type identifiers to the systolic array by rows and by columns; and an output module, configured to receive and output the calculation results calculated by the systolic array. The problems of low efficiency of quantization factor insertion, fragmentation of dequantization and calculation pipeline, and large task switching overhead are solved, and the data multiplexing rate, calculation efficiency and overall energy efficiency ratio of the systolic array in the fine-grained quantization scenario are significantly improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer architecture, in particular, to a fine-grained quantization matrix multiplication device and method based on systolic array. BACKGROUND

[0002] As a classic parallel computing architecture, systolic array is widely used in matrix multiplication, digital signal processing and artificial intelligence inference, etc. due to its regular data flow, high parallelism and high efficiency. The basic principle of systolic array is to connect processing units in a regular grid form, make data flow between rows and columns, and complete multiply-accumulate operation in the intersecting processing units, so as to realize efficient calculation of matrix multiplication.

[0003] In related technologies, when performing such fine-grained quantization matrix multiplication, the systolic array has the problems of hardwired split of dequantization calculation and matrix multiplication, discontinuous task scheduling, which leads to difficulty in efficient insertion of quantization factors, low data reuse rate, high task switching overhead, and seriously limits the computing efficiency and energy efficiency ratio when supporting flexible quantization model. SUMMARY

[0004] The purpose of the present disclosure is to provide a fine-grained quantization matrix multiplication device and method based on systolic array to solve the problems in related technologies.

[0005] In order to achieve the above purpose, the present disclosure provides a fine-grained quantization matrix multiplication device based on systolic array, which comprises:

[0006] An input scheduling module is configured to receive input matrix data and corresponding quantization factor data, and to attach type identifiers to the input matrix data and the quantization factor data;

[0007] A systolic array is connected by a plurality of processing units in a two-dimensional mesh structure, each processing unit is configured to select to perform multiply-accumulate operation or dequantization operation according to the data received upstream and the type identifier of the data; wherein the input scheduling module is configured to schedule the data with attached type identifiers to the systolic array by row and by column;

[0008] An output module is configured to receive and output the calculation results calculated by the systolic array.

[0009] Optionally, the input scheduling module comprises:

[0010] A row scheduler is connected to the first processing unit of each row of the systolic array, and is configured to schedule data by row;

[0011] A column scheduler is connected to the first processing unit of each column of the systolic array, and is configured to schedule data by column;

[0012] The row scheduler receives row data of the input matrix through a plurality of row data queues, and receives corresponding quantization factor row data through a plurality of quantization factor row queues;

[0013] The column scheduler receives column data of the input matrix through a plurality of column data queues, and receives corresponding quantization factor column data through a plurality of quantization factor column queues.

[0014] Optionally, the type identifier comprises:

[0015] A first identifier for identifying that the data is from input matrix data;

[0016] A second identifier for identifying that the data is from quantization factor data;

[0017] A third identifier for identifying that the data is from a quantization factor matrix and is a row-column boundary;

[0018] A fourth identifier for identifying that the data is the calculation result.

[0019] Optionally, the processing unit is configured to:

[0020] When data with the first identifier is received from the row and column directions simultaneously, a multiply-accumulate operation is performed;

[0021] When data with the second identifier is received from the row and column directions simultaneously, a dequantization operation is performed, and a locally accumulated multiply-accumulate intermediate result is reset;

[0022] When data with the third identifier is received from the row and column directions simultaneously, a dequantization operation is performed and a calculation result is generated, and then the output of the calculation result is triggered;

[0023] When data with the fourth identifier is received from the row and column directions simultaneously, the calculation result is transmitted.

[0024] Optionally, each processing unit further comprises an output sequence flag bit, which is used to statically configure the processing unit to pass the calculation result to the output module along the row direction or the column direction;

[0025] The output module comprises a plurality of output queues, which are used to cache the calculation result from the systolic array according to the row-major order or the column-major order configured by the output sequence flag bit.

[0026] Optionally, the output module is further configured to:

[0027] The occupancy state of the output queue is monitored, and when the occupancy state meets a first condition, a back pressure signal is sent to the input scheduling module to suspend data scheduling; when the occupancy state meets a second condition, the back pressure signal is withdrawn to resume data scheduling.

[0028] Optionally, the device is configured to support continuous matrix multiplication task pipelining, wherein the input scheduling module starts scheduling input data of a next matrix multiplication task at a predetermined interval after all input data of a previous matrix multiplication task is scheduled.

[0029] Optionally, the predetermined interval is determined based on at least one of the progress of emptying the output data of the previous task from the systolic array, and the occupancy state of the output queue.

[0030] The present disclosure also provides a fine-grained quantized matrix multiplication method based on a systolic array, applied to the fine-grained quantized matrix multiplication device based on a systolic array described above, the method comprising:

[0031] Attaching a type identifier to input matrix data and quantization factor data input to the systolic array;

[0032] In each processing unit of the systolic array, according to the data received upstream and the type identifier of the data, selecting to perform a multiply-accumulate operation or a dequantization operation;

[0033] Outputting the calculation result calculated by the systolic array.

[0034] Optionally, the selecting to perform a multiply-accumulate operation or a dequantization operation according to the data received upstream and the type identifier of the data comprises:

[0035] When the processing unit receives a type identifier indicating that the data comes from input matrix data, performing a multiply-accumulate operation;

[0036] When the processing unit receives a type identifier indicating that the data comes from quantization factor data, performing a dequantization operation and resetting accumulation; wherein if the quantization factor data belongs to a row-column boundary, triggering output of the calculation result after dequantization;

[0037] When the processing unit receives a type identifier indicating that the data is the calculation result, performing a pass-through of the calculation result.

[0038] According to the technical scheme, the input data and the quantization factor are uniformly added with a type identifier, and are accurately inserted into the systolic array through the input scheduling module in the same data stream mode, so that the quantization factor can enter the calculation process at the correct time; the processing unit in the array dynamically switches to perform multiplication and accumulation or inverse quantization operation according to the identifier, so as to integrate the inverse quantization calculation into the hardware pipeline of matrix multiplication in place, realize deep integration and seamless connection of the two, and fundamentally solve the problems of low efficiency of quantization factor insertion, separation of inverse quantization and calculation pipeline, and large task switching overhead, thereby significantly improving the data multiplexing rate, calculation efficiency and overall energy efficiency ratio of the systolic array in the fine-grained quantization scenario.

[0039] Other features and advantages of the present disclosure will be described in detail in the following detailed description section. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings are included to provide a further understanding of the present disclosure and constitute a part of the specification, and are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation on the present disclosure. In the drawings:

[0041] Figure 1 An input and output example diagram of fine-grained quantization matrix multiplication.

[0042] Figure 2 An output fixed systolic array structure workflow schematic diagram.

[0043] Figure 3 A split matrix schematic diagram considering fine-grained quantization.

[0044] Figure 4 A schematic diagram of a fine-grained quantization matrix multiplication device based on a systolic array according to an example embodiment.

[0045] Figure 5 A workflow schematic diagram of a processing unit according to an example embodiment.

[0046] Figure 6 An output timing diagram according to an example embodiment.

[0047] Figure 7 A flowchart of a fine-grained quantization matrix multiplication method based on a systolic array according to an example embodiment.

[0048] Explanation of reference signs

[0049] 10-input scheduling module, 20-systolic array, 30-output module. DETAILED DESCRIPTION

[0050] The specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended for illustration and explanation of the present disclosure and are not intended to limit the present disclosure.

[0051] In the following description, the words "first", "second", etc. are used only for the purpose of distinguishing the described objects, and cannot be understood as indicating or implying relative importance, nor indicating or implying an order.

[0052] First, the systolic array, quantization factor, tensor core, related terms, matrix multiplication based on outer product and inner product, and M / N / K dimensions of matrix multiplication in the present disclosure are explained.

[0053] Systolic array: The systolic array is a parallel computing structure proposed in the 1980s. The core idea is to arrange a large number of simple processing elements (PE) in space, and to achieve high parallelism and high throughput by regular, synchronous, and rhythmic data flow. Data is rhythmically pushed forward in the PE array like blood in the heart, and is processed step by step during the movement.

[0054] Quantization factor: When linear quantization is performed, the original data needs to be scaled according to a certain proportion to the target data precision type. For example, if the range of the original data is [-2047, 2048], it needs to be scaled to int8, i.e. [-255, 256], and the quantization factor is 8. Before calculation, the original data needs to be divided by 8 for quantization, and after calculation, it needs to be multiplied by 8 for dequantization.

[0055] Tensor core: The matrix multiplication acceleration unit used by GPU is a special hardware.

[0056] Matrix multiplication based on outer product and inner product: In mathematics, matrix multiplication has two implementation methods: inner product and outer product. Inner product is based on vector inner product operation, taking one row of matrix A and one column of B, element-wise multiplication and accumulation to get an element in the output matrix; outer product is based on vector outer product operation, taking one column of matrix A and one row of B, calculating the pairwise product of all elements to get a matrix of the same size as the output matrix, and summing all such matrices to get the final result. The systolic array belongs to the inner product method.

[0057] M / N / K dimensions of matrix multiplication: Generally, the shapes of matrices A and B are [M, K] and [K, N] respectively, and K is the accumulation dimension.

[0058] With the increasing scale of large models, using lower-bit quantization such as FP8 / FP6 / FP4 can significantly alleviate the bottleneck of memory capacity and improve computing speed. On the other hand, in order to reduce the precision loss caused by quantization, using finer-grained quantization technology has become a trend. Please refer to Figure 1 , assuming that the shape of the input matrix of the matrix multiplication hardware is A: [M, K] and B: [K, N], the shape of the quantization factor matrix is S_A: [M_scale, K_scale] and S_B: [K_scale, N_scale,], it should satisfy M_scale=M / M_step, N_scale=N / N_step and K_scale=K / K_step (all are integer division), where matrix A is quantized according to [M_step, K_step] granularity, and matrix B is quantized according to [K_step, N_step]. For example, quantize the activation layer parameter matrix A with [1, 128] granularity, and quantize the weight parameter matrix with [128, 128] granularity. The final calculation result should satisfy D=(A S_A)*(B S_B), where the operator * represents standard matrix multiplication, represents element-wise multiplication of matrix blocks, which is defined as multiplying the elements in the [i, j]th block of matrix A by the scalar S_A[i, j] for dequantization.

[0059] Some existing commercial computing architectures do not support matrix multiplication with scaling factors, and must be placed in Tensor Core calculation according to the block size, and the output result is sent to Cuda core for multiplication by the quantization factor for dequantization. Although some architectures support matrix multiplication with quantization factors, they have many limitations: first, S_A: [M, K_scale], S_B: [K_scale, N], and K_scale∈{1, 2, 4}, cannot be quantized with any block; second, there are requirements for the shape of input matrices A and B, especially the K dimension is limited, for example, the maximum value of K is 64 for dense matrix and 128 for sparse matrix under certain architecture, which limits its applicability in larger scale or variable dimension matrix operations.

[0060] On the other hand, although some commercial computing architectures have made some progress in systolic arrays, systolic arrays are typical inner product-based matrix multiplication structures with low communication overhead, high energy efficiency, and flexibility, and natively support unlimited K dimensions (as long as the accumulation precision is sufficient). However, traditional systolic arrays do not consider quantization factors, and without quantization factors, they can only split the original large matrix into multiple small matrix multiplications and then synchronize, significantly reducing the efficiency of computing resources.

[0061] Please refer toFigure 2 , the fixed output systolic array structure is adopted, that is, each row of the A matrix and each column of the B matrix enter the processing units of the systolic array in sequence according to the beats as shown in Figure 2 , each processing unit performs a multiply-add operation to obtain a partial sum, and the final result is saved in the processing unit after the data of the input matrix flows out of the systolic array. It can be deduced that for the input matrices A: [M, K] and B: [K, N], the size [N_syslotic, N_syslotic] of the systolic array needs to satisfy N_syslotic≥max (M, N), then M+K+N-2 cycles are needed to complete the calculation, and N_syslotic cycles are needed to complete the data output. The biggest advantage of the systolic array is high data transmission efficiency and simple processing unit structure, and the disadvantage is long latency, but the larger the K dimension is, the higher the utilization rate of the processing unit is.

[0062] Specifically, for the input matrix using fine-grained quantization, it can only be cut into many small matrices according to the quantization granularity, each sub-block calculation is multiplied by the corresponding quantization factor for dequantization, and then the final output result is multiplied according to the principle of matrix multiplication. Obviously, this will bring huge latency. As shown in Figure 3 , without considering the calculation and data output latency of the quantization factor, the systolic array directly calculates the matrix multiplication of [4, 2]x[2, 4] and needs 4+4+2-2=8 cycles, and the matrix multiplication of 4 [2, 2]x[2, 2] needs 4(2+2+2-2)=16 cycles.

[0063] That is, when the systolic array performs fine-grained quantization matrix multiplication, there are problems of hardware pipeline fragmentation of dequantization calculation and matrix multiplication calculation, discontinuous task scheduling, which leads to difficulty in efficient insertion of quantization factors, low data multiplexing rate, large task switching overhead, and seriously limits the calculation efficiency and energy efficiency ratio in supporting flexible quantization model.

[0064] To solve the above problems, a type identifier is uniformly attached to the input data and the quantization factor, and they are inserted into the systolic array through the input scheduling module in the same data flow mode, so that the quantization factor can enter the calculation process at the right time; the processing unit in the array dynamically switches to perform multiply-add or dequantization operation according to the identifier, so as to integrate the dequantization calculation in situ into the hardware pipeline of matrix multiplication, realize the deep integration and seamless connection of the two, and fundamentally solve the problems of inefficient insertion of quantization factors, fragmentation of dequantization and calculation pipeline, and large task switching overhead, which significantly improves the data multiplexing rate, calculation efficiency and overall energy efficiency ratio of the systolic array in the fine-grained quantization scenario.

[0065] Figure 4Fig. 1 is a schematic diagram of a systolic array based fine-grained quantized matrix multiplication device according to an exemplary embodiment, which can include an input scheduling module 10, a systolic array 20 and an output module 30, which are connected in sequence. Figure 4 The systolic array based fine-grained quantized matrix multiplication device can include an input scheduling module 10, a systolic array 20 and an output module 30, which are connected in sequence.

[0066] The input scheduling module 10 is configured to receive input matrix data and corresponding quantization factor data, and attach type identifiers to the input matrix data and the quantization factor data.

[0067] The input matrix data can refer to elements of two matrices A and B that need to be multiplied. In actual applications, the input matrix data can be quantized, for example, quantized to INT8, to reduce storage and calculation overhead.

[0068] The quantization factor data can refer to scaling factors used to dequantize the quantized data.

[0069] In this embodiment, the input matrices A and B correspond to quantization factor matrices S_A and S_B, respectively. For example, if a certain sub-block of the matrix A uses a quantization factor s_a, the elements of the sub-block need to be multiplied by s_a to restore their original value range during calculation.

[0070] The type identifier can refer to identification information attached to each data element, which is used to indicate whether the data belongs to an input matrix or a quantization factor. For example, "00" in 2 bits can represent data of the input matrix A, "01" can represent data of the input matrix B, "10" can represent data of the quantization factor S_A, and "11" can represent data of the quantization factor S_B. Setting the type identifier can enable the processing units to distinguish between different types of data and perform corresponding operations.

[0071] The systolic array 20 is connected in a two-dimensional mesh structure by a plurality of processing units, each of which is configured to select to perform a multiply-accumulate operation or a dequantization operation according to the data received upstream and the type identifier of the data; and the input scheduling module 10 is configured to schedule the data with the attached type identifiers to the systolic array 20 by rows and by columns.

[0072] The input scheduling module 10 schedules data to the systolic array 20 by rows and by columns: data of the matrix A and the quantization factor S_A are scheduled by rows and fed into the first processing unit of each row of the systolic array 20; data of the matrix B and the quantization factor S_B are scheduled by columns and fed into the first processing unit of each column of the systolic array 20. The data flows in the systolic array 20 in a systolic manner: in each clock cycle, the processing units pass the received data to the adjacent downstream processing units while performing corresponding operations locally.

[0073] A multiply-accumulate operation can refer to a basic operation in a standard matrix multiplication performed by a processing unit, i.e., c = c + a x b, where a and b are from input matrices A and B respectively, and c is the accumulated partial sum.

[0074] A dequantization operation can refer to a quantization factor related operation performed by a processing unit, exemplarily, when a processing unit receives elements from quantization factor matrices S_A and S_B simultaneously, it performs an operation of sum = sum + c x s_a x s_b, where c is the previously accumulated multiply-accumulate result, s_a and s_b are quantization factors, and sum is the calculation result, i.e., the final dequantization result. After the operation, c is reset to 0.

[0075] Exemplarily, for a systolic array 20 of N_syslotic rows and N_syslotic columns, the maximum input matrix data A: [M, K], B: [K, N] it can handle needs to satisfy M < N_systolic and N < N_systolic. Meanwhile, for hardware complexity and resource overhead considerations, a relatively fixed shape of quantization factor matrix data can be set, i.e., S_A: [M, K_scale], S_B: [K_scale, N]. Due to the use of the systolic array 20, the dimensions of K and K_scale can be theoretically infinite, as long as the data loading time can overlap with the calculation time and the accuracy meets the requirements.

[0076] In an embodiment, each processing unit decides which operation to perform according to the type of data it receives:

[0077] Case 1: Multiply-accumulate operation

[0078] When a processing unit receives data identified as “00” from the row direction (from matrix A) and data identified as “01” from the column direction (from matrix B) simultaneously, it performs a multiply-accumulate operation: c = c + a x b.

[0079] Case 2: Dequantization operation

[0080] When a processing unit receives data identified as “10” from the row direction (from quantization factor S_A) and data identified as “11” from the column direction (from quantization factor S_B) simultaneously, it performs a dequantization operation: sum = sum + c x s_a x s_b, and then resets the local accumulator c to 0.

[0081] Case 3: Special processing of boundary quantization factors

[0082] When the quantization factor data identification indicates that it is a row-column boundary, the processing unit will also trigger the operation of passing the final calculation result sum to the output module 30 after performing the dequantization operation.

[0083] An output module 30 is configured to receive and output the calculation result calculated by the systolic array 20.

[0084] The calculation result of the systolic array 20 is output through the output module 30, which comprises a plurality of output queues configured to buffer the final results from the processing units of the systolic array 20 in a row-major order or a column-major order.

[0085] By uniformly adding a type identifier to the input data and the quantization factor and accurately inserting the input data and the quantization factor into the systolic array 20 through the input scheduling module 10 in the same data flow mode, the quantization factor can be inserted into the calculation flow at the correct time; the processing units in the array dynamically switch between performing multiplication and accumulation or dequantization operation according to the identifier, so that the dequantization calculation is integrated into the hardware pipeline of matrix multiplication in situ, and deep integration and seamless connection of the two are achieved; the problems of inefficient insertion of the quantization factor, separation of dequantization and calculation flow, and large switching overhead are fundamentally solved, and the data multiplexing rate, calculation efficiency and overall energy efficiency ratio of the systolic array 20 in the fine-grained quantization scenario are significantly improved.

[0086] In a possible implementation, the input scheduling module 10 can comprise a row scheduler and a column scheduler.

[0087] The row scheduler is connected to the first processing unit of each row of the systolic array 20 and is configured to schedule data in a row direction.

[0088] The column scheduler is connected to the first processing unit of each column of the systolic array 20 and is configured to schedule data in a column direction.

[0089] The first processing unit can refer to the first processing unit of each row or each column of the systolic array 20. It is the entry point of data into the systolic array 20. In the row direction, the first processing unit of the i-th row can be denoted as PEi1; in the column direction, the first processing unit of the j-th column can be denoted as PE1j.

[0090] If the systolic array 20 has N rows, N row schedulers can be correspondingly configured, and the output end of each row scheduler is connected to the first processing unit of the corresponding row through a data path. For example, the first row scheduler is connected to the processing unit PE11, the second row scheduler is connected to the processing unit PE21, and so on. The row scheduler receives all row data of the matrix A and all row data of the quantization factor matrix S_A, and one or more row buffer queues are arranged in each row scheduler to temporarily store the received data.

[0091] When the systolic array 20 is ready to receive new data, the row scheduler transmits the data in the row buffer queue to the corresponding first processing unit in sequence according to a preset timing rule.

[0092] The specific process is as follows:

[0093] (1) Data loading stage, before the start of the calculation task, the i-th row elements of matrix A [a_{i1}, a_{i2}, …, a_{iK}] and the i-th row elements of quantization factor matrix S_A [s_{i1}, s_{i2}, …, s_{iK_scale}] are loaded into the corresponding buffer queue in the i-th row scheduler, respectively.

[0094] Each row scheduler can include a plurality of row data queues and a plurality of row quantization factor queues, the row scheduler receives the row data of the input matrix through the plurality of row data queues, and receives the corresponding quantization factor row data through the plurality of row quantization factor queues, the row data queue is used to store the i-th row elements of matrix A, and the row quantization factor queue is used to store the i-th row elements of quantization factor matrix S_A.

[0095] (2) Transmission start and sequence control, after the calculation start signal, the i-th row scheduler first takes out the first element (such as a_{i1}) from the row data queue, appends the type identifier (such as “00”) that it belongs to the input matrix, and transmits it to the first processing unit PEi1 in the next clock cycle. After that, the scheduler takes out a_{i2}, a_{i3}, …, a_{iK} in turn according to the first-in first-out sequence and transmits them one by one.

[0096] (3) Insertion transmission of quantization factor, according to the preset timing rule, for example, after transmitting X row data elements, the row scheduler automatically switches to the row quantization factor queue to start transmitting a quantization factor element, and after transmitting X row data elements after the quantization factor element, a quantization factor element is transmitted, and so on, until all data transmission is completed.

[0097] In other embodiments, the elements of matrix A and factor matrix S_A can also be arranged in a mixed queue according to a preset rule, for example, every 3 elements of matrix A are followed by an element of factor matrix S_A.

[0098] If the systolic array 20 has M columns, M column schedulers can be correspondingly configured, and the output end of each column scheduler is connected to the first processing unit of the corresponding column through a data path. For example, the first column scheduler is connected to processing unit PE11, the second column scheduler is connected to processing unit PE12, and so on. The column scheduler receives all column data of matrix B and all column data of quantization factor matrix S_B, and each column scheduler is internally provided with a column buffer queue for temporarily storing the received data.

[0099] When the systolic array 20 is ready to receive new data, the column scheduler transmits the data in the column buffer queue to the corresponding first column processing unit in turn according to the preset timing rule.

[0100] The specific process is as follows:

[0101] (1) Data loading stage, before the start of the computation task, the jth column elements of matrix B [b_{1j}, b_{2j}, …, b_{Kj}] and the jth column elements of quantization factor matrix S_B [s_{1j}, s_{2j}, …, s_{K_scale j}] are loaded into the corresponding buffer queues in the jth column scheduler, respectively.

[0102] Each column scheduler can include a plurality of column data queues and a plurality of column quantization factor queues, the column scheduler receives column data of the input matrix through the plurality of column data queues, and receives corresponding quantization factor column data through the plurality of column quantization factor queues, the column data queue is used to store the jth column elements of matrix B, and the column quantization factor queue is used to store the jth column elements of quantization factor matrix S_B.

[0103] (2) Transmission start and sequence control, after the computation start signal, the jth column scheduler first takes out the first element (such as b_{1j}) from the column data queue, appends a type identifier (such as "01") indicating that it belongs to the input matrix, and transmits it to the first processing element PE1j in the next clock cycle. After that, the scheduler takes out b_{2j}, b_{3j}, …, b_{Kj} in the order of first in first out (FIFO) and transmits them one by one.

[0104] (3) Insertion and transmission of quantization factors, according to the preset timing rules, for example, after transmitting Y column data elements, the column scheduler automatically switches to the column quantization factor queue to start transmitting a quantization factor element, and then starts transmitting Y column data elements again, and then transmits a quantization factor element, and so on, until all data transmission is completed.

[0105] In other embodiments, the elements of matrix B and factor matrix S_B can also be arranged in a cross pattern according to a preset rule. For example, after every 2 elements of matrix B, an element of factor matrix S_B is followed to form a mixed queue.

[0106] At the same time, a type identifier is attached to each quantization factor element to indicate that it belongs to quantization factor data and whether it is a boundary element, for non-boundary quantization factor elements (such as the first K_scale-1 elements), a type identifier "10" is attached, and for the last boundary quantization factor element, a type identifier "11" is attached to indicate the end of the row / column quantization factor.

[0107] By setting separate row and column schedulers, row and column data can be independently and in parallel scheduled, fully exerting the two-dimensional parallel computing capability of the systolic array 20, and since each scheduler is only responsible for the data of a specific row or column, data cross access and competition is reduced, and when the size of the systolic array 20 increases, only the number of row / column schedulers needs to be increased accordingly, the architecture expansion is simple, the row and column schedulers can independently control the data flow rate and timing of each other, which is convenient for adapting to different matrix shapes and quantization configurations, the local failure of the row or column scheduler does not affect other parts of the entire system, and the system reliability is also improved.

[0108] In a possible implementation, the type identifier can include:

[0109] A first identifier for identifying that the data comes from the input matrix data;

[0110] A second identifier for identifying that the data comes from the quantization factor data;

[0111] A third identifier for identifying that the data comes from the quantization factor matrix and is a row-column boundary;

[0112] A fourth identifier for identifying that the data is the calculation result sum.

[0113] The third identifier can be used to identify the last element of each row in the quantization factor matrix S_A and the last element of each column in the quantization factor matrix S_B, and in other embodiments, the third identifier can further include a first sub-identifier and a second sub-identifier, the first sub-identifier is used to identify the last element of each row in the quantization factor matrix S_A, and the second sub-identifier is used to identify the last element of each column in the quantization factor matrix S_B.

[0114] By setting a complete data identifier system, the processing units in the systolic array 20 can accurately distinguish data of different sources and properties and perform corresponding calculation operations.

[0115] In a possible implementation, the processing unit is configured to:

[0116] When receiving data with the first identifier from the row and column directions at the same time, performing a multiply-accumulate operation;

[0117] When receiving data with the second identifier from the row and column directions at the same time, performing a dequantization operation and resetting the locally accumulated multiply-accumulate intermediate result;

[0118] When receiving data with the third identifier from the row and column directions at the same time, performing a dequantization operation and generating a final calculation result, and then triggering the output of the final calculation result;

[0119] When receiving data with the fourth identifier from the row and column directions at the same time, performing a pass-through of the calculation result sum.

[0120] The process of the multiply-accumulate operation is as follows:

[0121] When the processing unit receives data carrying the first identification from the row direction and the column direction at the same time, the basic operation c = c + a x b of the row standard matrix multiplication is performed, the row direction data a comes from the input matrix A, the column direction data b comes from the input matrix B, the multiplier inside the processing unit is enabled, the multiplier calculates the result of a x b, the adder adds the result to the current value of the local accumulation register c, and the result is written back to the c register.

[0122] The process of the non-boundary dequantization operation is as follows:

[0123] When the processing unit receives data carrying the second identification from the row direction and the column direction at the same time, the dequantization calculation sum = sum + c x s_a x s_b is performed, and then the local accumulator c is reset to 0, the row direction data s_a comes from the non-boundary element of the quantization factor matrix S_A, the column direction data s_b comes from the non-boundary element of the quantization factor matrix S_B, the multiply-multiply-add logic inside the processing unit is activated, c x s_a is first calculated to obtain an intermediate result, then s_b is multiplied, and finally the sum register is added, after the operation is completed, the reset control logic clears the c register.

[0124] The process of the boundary dequantization operation is as follows:

[0125] When the processing unit receives at least one data carrying the third identification from the row direction and the column direction at the same time, it can be two third identifications, or a combination of one second identification and one third identification, that is, at least one quantization factor is a row or column boundary element, the dequantization calculation sum = sum + c x s_a x s_b is performed, and then c is reset to 0, and then the output of the final calculation result sum is triggered, in addition, the boundary detection logic sets the output trigger flag, and controls the output multiplexer to send the value of the sum register to the output port.

[0126] The pass-through of the calculation result sum can be understood as sequentially passing the calculation result sum to the next processing unit in the row direction in the row-major order, or sequentially passing the calculation result sum to the next processing unit in the column direction in the column-major order.

[0127] Through identification control instead of fixed hardwiring, multiple quantization modes and calculation configurations are supported, and the clear identification driving mechanism makes the data flow and the control flow clearly separated, facilitating verification and debugging.

[0128] In a possible implementation, the processing unit can also be configured to:

[0129] Before passing the data downstream, the step information associated with the data is updated, and it is determined whether to continue passing the data based on the updated step information.

[0130] The step information is a counter value associated with each data element, indicating the remaining step length that the data still needs to be passed in the systolic array 20. When the data enters the systolic array 20 from the input scheduling module 10, an initial step value is assigned, which is usually equal to the column number N (for row direction input matrix data) or the row number M (for column direction input matrix data) for input matrix data, and is set to the column number N (for row direction quantization factor data) or the row number M (for column direction quantization factor data) for quantization factor data.

[0131] When the processing unit is ready to pass the data to the downstream row / column processing unit, the step value carried by the data is decremented by one. Based on the updated step information, it is determined whether to continue passing the data. It can be understood that the processing unit checks the updated step value, and if the updated step value > 0, the data needs to continue to be passed, and the processing unit sends the data, the type identifier, and the updated step to the downstream neighbor processing unit. If the updated step value is 0, the processing unit stops passing the data.

[0132] In other embodiments, the calculation result sum, i.e., the final dequantization result, can also be set with a remaining step length to ensure that the calculation result is transmitted out.

[0133] Through the step control mechanism, the fixed data flow mode of the systolic array 20 is upgraded to a programmable dynamic data flow, which greatly enhances the support capability of the systolic array 20 for diversified computing tasks while maintaining the high efficiency and regularity of the systolic array 20, and provides a solid hardware foundation for implementing high-throughput and low-latency quantization neural network inference.

[0134] In a possible implementation, each processing unit further includes an output sequence flag bit, which is used to statically configure the processing unit to pass the calculation result to the output module 30 in a row direction or a column direction; the output module 30 includes a plurality of output queues, which are used to cache the calculation result from the systolic array 20 according to the row-major order or the column-major order configured by the output sequence flag bit.

[0135] The output sequence flag bit can be a configuration register bit stored in each processing unit, which is used to indicate that the processing unit should pass its final calculation result to the output module 30 in a row direction or a column direction.

[0136] In one embodiment, when the output order flag is 0 or row-major, the processing units pass the calculation results to the right neighboring processing units in the same row, and finally output by the rightmost processing unit in each row; when the output order flag is 1 or column-major, the processing units pass the calculation results to the lower neighboring processing units in the same column, and finally output by the lowermost processing unit in each column.

[0137] The row-major output can be that the calculation results are passed in the row direction inside the systolic array 20, and the rightmost processing unit in each row acts as the output port of the row, and the plurality of output queues can be a plurality of row output queues, each of which is used to receive the calculation results of the rightmost processing unit in a row based on the cache row-major output.

[0138] The column-major output can be that the calculation results are passed in the column direction inside the systolic array 20, and the lowermost processing unit in each column acts as the output port of the column, and the plurality of output queues can be a plurality of column output queues, each of which is used to receive the calculation results of the lowermost processing unit in a column based on the cache column-major output.

[0139] By introducing a configurable output order flag in the processing unit, the same systolic array 20 hardware can directly output the result matrix in row-major or column-major as needed, not only reducing data transfer overhead and delay, but also making the memory layout of the output data directly match the access mode of the subsequent processing unit, significantly improving the overall energy efficiency and throughput of the system, and enhancing the adaptability of the hardware to different computing tasks.

[0140] In a possible implementation, the output module 30 can also be configured to monitor the occupancy state of the output queue, and when the occupancy state meets a first condition, send a backpressure signal to the input scheduling module 10 to suspend data scheduling; when the occupancy state meets a second condition, cancel the backpressure signal to restore data scheduling.

[0141] The occupancy state of the output queue can reflect the degree of data backlog at the output end, and can be the ratio or specific value of the amount of data stored in the output queue but not read out to the total capacity of the queue. It can be represented by absolute quantity (such as "32 data stored"), percentage (such as "occupancy rate 80%") or hierarchical state (such as "idle", "normal", "warning", "full").

[0142] The first condition can be that the occupancy state of the output queue indicates that the queue is about to be full, and continuing to receive new data may cause overflow. The backpressure signal can act on the input scheduling module 10 to send a pause request, requiring it to temporarily stop or slow down the delivery of new data to the systolic array 20.

[0143] The second condition can be that the occupancy state of the output queue indicates that the queue has enough space to receive new data, and can continue to receive new data.

[0144] By setting the flow control mechanism for the output module 30, the input data scheduling is dynamically controlled by monitoring the occupancy status of the output queue, so as to prevent data overflow or idle of computing resources, and ensure stable and efficient operation of the system.

[0145] In a possible implementation, the device is configured to support continuous matrix multiplication task pipeline execution, in which the input scheduling module 10 starts to schedule the input data of the next matrix multiplication task at a predetermined interval after the completion of the scheduling of all input data of the previous matrix multiplication task.

[0146] The completion of the scheduling of all input data of the previous matrix multiplication task (for example, task 1) can be understood as the time when the input scheduling module 10 task 1 has sent the last data element (input matrix element or quantization factor) into the systolic array 20, at which time all the input data of task 1 have entered the device but are still flowing and calculating in the array, the output result of task 1 has not been completely generated and output, and there are still a large amount of intermediate data of task 1 being transmitted and calculated in the systolic array 20, and the output module 30 can also be receiving the results of task 1.

[0147] The predetermined interval can be a configurable number of delay cycles, that is, the number of clock cycles between the completion of the input of task 1 and the start of the input of the next matrix multiplication task (for example, task 2), and it should be understood that the predetermined interval needs to be long enough to ensure that the data of task 2 does not cover or interfere with the data of task 1 still in the array, but cannot be too long to cause hardware idling.

[0148] Through the pipeline execution mechanism, the limitation that the traditional systolic array can only process one computing task at a time is broken, so that the hardware computing resources can be almost continuously and fully utilized, and in particular in the scenario of processing massive matrix operations, significant performance improvement and energy efficiency improvement can be brought.

[0149] In a possible implementation, the predetermined interval is determined based on at least one of the progress of emptying the output data of the previous task from the systolic array 20 and the occupancy status of the output queue.

[0150] The progress of emptying the output data of the previous task from the systolic array 20 can be understood as the degree of completion of the transmission of the computing result data of the previous task from the last processing unit to the output module 30, which can be obtained by the remaining time from the completion of the input of the task to the exit of the last result from the array, the number or proportion of processing units in the systolic array 20 still containing the result data of the previous task, the number of result elements of the previous task that have not been output, and the like.

[0151] The occupancy status of the output queue can be understood as the filling degree of the output queue storing the result of the previous task.

[0152] In one embodiment, N_systolic row FIFO queues can be allocated for receiving each row of elements of matrix A, N_systolic column FIFO queues for receiving each column of elements of matrix B, N_systolic row quantization factor FIFO queues for receiving each row of elements of matrix S_A, and N_systolic column quantization factor FIFO queues for receiving each column of elements of matrix S_B. The depth of each queue is large enough to satisfy the time for a memory controller to load half of the data from shared memory (SRAM) or dedicated tensor memory (TMEM) and for systolic array 20 to complete the computation of the other half of the data, so that pipelining can be achieved. M can be set to be less than or equal to N_systolic, N can be set to be less than or equal to N_systolic, and some of the row and column FIFO queues and quantization factor FIFO queues can be empty, i.e., the number of rows of matrix A is less than the length and width of systolic array 20, and the number of columns of matrix B is less than the length and width of systolic array 20.

[0153] In one embodiment, input scheduling module 10 can be configured with up to N_systolic row data FIFO queues, N_systolic column data FIFO queues, N_systolic row quantization factor FIFO queues, and N_systolic column quantization factor FIFO queues for buffering row data of matrix A, column data of matrix B, row data of quantization factor matrix S_A, and column data of quantization factor matrix S_B, respectively. The size of the input matrices needs to satisfy M < N_systolic and N < N_systolic (M and N are the number of rows of matrix A and the number of columns of matrix B, respectively). In actual execution, only the first M row data FIFO queues and the first M row quantization factor FIFO queues are used for receiving data of matrix A and S_A, and only the first N column data FIFO queues and the first N column quantization factor FIFO queues are used for receiving data of matrix B and S_B. The remaining queues can be kept empty or disabled. The depth of each queue is configured such that the time required for loading a set of data from an external memory, such as a static random-access memory (SRAM) or a dedicated tensor memory, does not exceed the time required for systolic array 20 to compute another set of data that is ready, so that pipelining of data loading and computation is achieved, and the computation units are not idle due to insufficient data supply.

[0154] N_systolic row input schedulers can be provided, the i-th row scheduler is connected with the i-th row FIFO queue and the row quantization factor FIFO queue, if the i-th row is enabled for the current input matrix size, the i-th row scheduler takes the head element from the i-th row element input FIFO queue and gets the quantization factor from the corresponding row quantization factor queue every K_step clock cycles after the i-th clock cycle is started.

[0155] The row / column scheduler in the input scheduling module 10 stops scheduling after receiving the back pressure signal sent by the output module 30, and resumes scheduling after receiving the resume signal sent by the output module 30, i.e. the back pressure signal is cancelled.

[0156] After the row / column scheduler obtains the data, the row / column scheduler adds 2-bit type identification and log2N_syslotic-bit remaining step, i.e. step information, to the data. Four types of identification are defined: 2'b00 represents that the data comes from the input matrix; 2'b01 represents that the data comes from the quantization factor matrix but is not the row / column tail; 2'b10 represents that the data comes from the row / column tail of the quantization factor matrix; and 2'b11 represents the final multiply-accumulate result. The first three types of identification are set after the scheduler obtains the data, and the corresponding remaining steps are initialized to N for the row scheduler and M for the column scheduler. The last one is to set the remaining step left_step to be large enough to ensure that the systolic array 20 can output data, for example, N_syslotic. The left_step is set mainly to enable the continuous execution of several matrix multiplications in a pipeline. After the row / column scheduler schedules the row / column tail element of the quantization factor, the scheduler stops scheduling for several cycles to ensure the correctness of the pipeline.

[0157] In other embodiments, a type identification for invalid data or a specific remaining step value can be added to distinguish the four types of data.

[0158] After the row / column scheduler finishes scheduling all the elements, i.e. the matrix rows and columns have been sent to the systolic array 20, the scheduler can be allowed to schedule the next matrix multiplication after a certain number of clock cycles, i.e. the input of the next matrix multiplication can be executed in a pipeline.

[0159] In the systolic array 20, N_syslotic rows and N_syslotic columns of processing units form a two-dimensional mesh structure, and each processing unit can obtain data from upstream row and column neighbor processing units and transmit the data to downstream row / column neighbor processing units.

[0160] The results of the systolic array 20 computation can be output in row-major or column-major order, and in other embodiments, for simplicity, a static configuration can also be considered, such as each processing element having an output order flag bit to record whether row-major or column-major order is enabled. If the next matrix multiplication is to change the output order, the output order flag bits of all processing elements need to be refreshed after the systolic array 20 is emptied and then the next matrix multiplication task is executed.

[0161] Each processing element can process according to the type of data, the internal processing flow is as shown in Figure 5 .

[0162] a) If the output data of the two upstream processing elements (including the scheduler) are both Ready, and the type identifier of the processing element is 2'b00, that is, the input matrix or element, the standard systolic array 20 processing element multiply-accumulate operation is performed. Specifically, assuming that the inputs of the row and column upstream nodes are and , the locally saved segmented multiply-accumulate sum is (initialized to 0 each time), then is executed. The specific hardware circuit can be optimized according to the data type (integer or floating point) and bit width, which is not limited in this embodiment. After the calculation is completed, left_step is reduced by one, and if it is equal to 0, the Ready signal to the downstream is set to False, otherwise continue to pass and to the downstream row and column processing element neighbors, and set the Ready signal to True.

[0163] b) If the output data of the two upstream processing elements (including the scheduler) are both Ready, and the type identifier of the processing element is 2'b01, that is, the non-row and column tail data of the quantization factor matrix, the inverse quantization multiply operation is performed. Specifically, assuming that the inputs are and , then and reset (i.e. ) operations are performed, where represents the locally saved result for the final output. Similarly, the specific hardware circuit can be optimized according to the data type (integer or floating point) and bit width, which is not limited in this embodiment. For example, and are floating point numbers, but and use unsigned integers to represent the exponents of floating point numbers, for example, UE8M0, then the calculation can be changed to increase the exponent part of c by After the calculation is finished, left_step is decremented by one, and if it equals 0, the Ready signal to the downstream is set to False, otherwise the and are passed to the neighbor of the downstream row / column processing unit and the Ready signal is set to True.

[0164] c) If the output data of both upstream processing units (including the scheduler) are Ready and the type identifier of the processing unit is 2'b10, i.e. the row / column tail data of the quantization factor matrix, after performing the dequantization, the multiplication and the transmission of the quantization factor in the current clock cycle, the local calculation end flag is also set to indicate the transmission of in the next cycle (case e). As in case a), left_step also needs to be updated to decide whether to transmit and and set the corresponding Ready signal.

[0165] d) If only one data of the upstream row or column processing unit (including the scheduler) is Ready and matches the row / column major flag, and the type identifier of the processing unit is 2'b11, the row or column pass-through of the data (containing the upstream and the type identifier of the processing unit is 2'b11) is performed (according to the row / column major flag).

[0166] e) If the local calculation end flag is true and there is no pass-through of the upstream according to the row / column major flag, the local calculation result is sent to the downstream row or column processing unit and the corresponding Ready signal is set, the type identifier of the processing unit is set to 2'b11, is reset.

[0167] f) If both a) and e) are satisfied and the data element does not need to be transmitted to the downstream processing unit (i.e. leaf_step == 0 after updating), the transmission of , the reset of the local and the multiplication-addition update of can be completed in the same cycle, such as T+4 cycles in Figure 6 .

[0168] Output module 30 allocates N_syslotic output queues (FIFOs) to receive the output matrix C in row-major or column-major order. Similar to the input queues, each queue must be sufficiently deep. When configured for row-major order, the tail processing unit of each row in systolic array 20 sequentially sends the accumulated sum (i.e., the calculated result `sum`) to the output queue. The enqueue order is consistent with the address order of the accumulated sum, so the data in the output queue only needs to be written to the output memory address specified by the matrix multiplication function at appropriate times. Similarly, when configured for column-major order, the tail processing unit of each column in systolic array 20 sends the accumulated sum to the output queue. When any output queue is determined to be full (i.e., the remaining buffer size is less than a first set threshold), a backpressure signal is sent to the input module. When the remaining buffer size is greater than a second set threshold, a recovery signal is sent to the input module. The second set threshold is greater than the first set threshold to avoid frequent signaling.

[0169] A separate output queue is allocated primarily to enable the reuse of multiple matrix multiplications. Traditional pulsating array architectures with fixed outputs require flushing the result after each matrix multiplication before executing the next. By creating a pipeline of multiple matrix multiplications, subsequent multiplications do not need to wait for the complete output of the previous one; they can enter the input module after sufficient intervals, improving the fluidity between task operations.

[0170] like Figure 6 As shown, when the number of row elements output by two matrix multiplications is 4 and 2 respectively, the latter can be input into the matrix in the second cycle of the last quantization factor of the former matrix, without waiting for it to be completely emptied. The scheduler records and maintains the type identifier and other information of each data. Based on the number of row or column elements in each matrix operation, whether the output is row-major or column-major, and whether the end of the row or column has been reached, the summation result in each processing unit of the pulsating array also flows in the direction of rows or columns, sequentially arriving at the output queue of the output module.

[0171] exist Figure 6 The diagram shows the first row scheduler and processing unit, with a 4x4 pulsating array. The first matrix multiplication has K=4 and N=4, while the second matrix multiplication has K'=2 and N'=2. The row scheduler can begin scheduling in the second cycle (T+2) after scheduling the first matrix (cycle T in the diagram). Here, ** represents the sum+=c*a*b operation, *+ represents the c+=a*b operation, -> represents the pass-through of the upstream result, =0 represents the local sum being transmitted downstream and then reset to 0, and *+0 represents the local sum being transmitted downstream, sum=0, and c+=a*b operations.

[0172] Based on the same inventive concept, the disclosure also provides a fine-grained quantized matrix multiplication method based on systolic array, which can be applied to a fine-grained quantized matrix multiplication device based on systolic array, please refer to Figure 7 The method can include steps S1-S3.

[0173] In step S1, type identifiers are attached to input matrix data and quantization factor data input to the systolic array.

[0174] In step S2, in each processing unit of the systolic array, according to the data received upstream and the type identifier of the data, it is selected to perform multiply-accumulate operation or dequantization operation.

[0175] In step S3, the calculation result calculated by the systolic array is output.

[0176] In one possible implementation, step S2 can include:

[0177] When the processing unit receives a type identifier indicating that the data comes from the input matrix data, it performs multiply-accumulate operation;

[0178] When the processing unit receives a type identifier indicating that the data comes from the quantization factor data, it performs dequantization operation and resets the accumulation; wherein if the quantization factor data belongs to the row and column boundary, the output of the calculation result is triggered after dequantization;

[0179] When the processing unit receives a type identifier indicating that the data is the calculation result, it performs the pass-through of the calculation result.

[0180] In one embodiment, the fine-grained quantized matrix multiplication method based on systolic array can further include:

[0181] Before passing the data downstream, update the step information associated with the data, and decide whether to continue passing the data based on the updated step information.

[0182] In one embodiment, the fine-grained quantized matrix multiplication method based on systolic array can further include:

[0183] Monitor the occupancy state of the output queue, and send a back pressure signal to the input scheduling module to suspend data scheduling when the occupancy state meets the first condition; and withdraw the back pressure signal to restore data scheduling when the occupancy state meets the second condition.

[0184] As to the fine-grained quantized matrix multiplication method based on systolic array in the above embodiments, the specific way in which each module performs operation has been described in detail in the embodiments of the device, and will not be described in detail here.

[0185] The pulse array-based fine-grained quantization matrix multiplication device and method provided by the present disclosure have at least the following advantages:

[0186] 1) Fine-grained quantization is realized while maintaining the high efficiency and flexibility of the pulse array;

[0187] 2) Continuous multiple matrix multiplication pipelining is supported to realize efficient calculation resource reuse;

[0188] 3) Row-major and column-major output modes are supported without additional matrix transposition.

[0189] The preferred embodiments of the present disclosure are described in detail above with reference to the drawings, but the present disclosure is not limited to the specific details in the above-described embodiments. Within the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all belong to the protection scope of the present disclosure.

[0190] In addition, it should be noted that each specific technical feature described in the above-described specific embodiments can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0191] Furthermore, any combination of the various different embodiments of the present disclosure can also be made, as long as it does not deviate from the idea of the present disclosure, and it should also be considered as disclosed by the present disclosure.

Claims

1. A systolic array based fine-grained quantized matrix multiplication device, characterized in that, The device comprises: an input scheduling module, configured to receive input matrix data and corresponding quantization factor data, and to attach type identifiers to the input matrix data and the quantization factor data; a systolic array, connected by a plurality of processing units in a two-dimensional mesh structure, each of the processing units being configured to select an execution of a multiply-accumulate operation or a dequantization operation according to data received from upstream and type identifiers of the data; wherein the input scheduling module is configured to schedule the data with the attached type identifiers to the systolic array by rows and by columns; an output module, configured to receive and output calculation results calculated by the systolic array. The type identifiers comprise: a first identifier, configured to identify that the data is from the input matrix data; a second identifier, configured to identify that the data is from the quantization factor data; a third identifier, configured to identify that the data is from the quantization factor matrix and is at a row-column boundary. The processing units are configured to: execute the multiply-accumulate operation when data with the first identifier is received from both the row direction and the column direction; execute the dequantization operation and reset a locally accumulated multiply-accumulate intermediate result when data with the second identifier is received from both the row direction and the column direction; execute the dequantization operation and generate a calculation result when data with the third identifier is received from both the row direction and the column direction, and then trigger an output of the calculation result.

2. The apparatus of claim 1, wherein, The input scheduling module comprises: a row scheduler, connected to a first processing unit of each row of the systolic array, configured to schedule data by rows; a column scheduler, connected to a first processing unit of each column of the systolic array, configured to schedule data by columns; The row scheduler receives row data of the input matrix through a plurality of row data queues and receives corresponding quantization factor row data through a plurality of quantization factor row queues. The column scheduler receives column data of the input matrix through a plurality of column data queues and receives corresponding quantization factor column data through a plurality of quantization factor column queues.

3. The apparatus of claim 1 or 2, wherein, The type identifiers further comprise: a fourth identifier, configured to identify that the data is the calculation result.

4. The apparatus of claim 3, wherein, The processing units are configured to: transmit the calculation result when data with the fourth identifier is received from both the row direction and the column direction.

5. The device of claim 1, wherein: each of the processing units further comprises an output sequence flag bit, the output sequence flag bit being configured to statically configure the processing unit to transmit the calculation result to the output module along a row direction or a column direction; the output module comprises a plurality of output queues, the plurality of output queues being configured to buffer the calculation result from the systolic array according to a row-major order or a column-major order configured by the output sequence flag bit.

6. The apparatus of claim 5, wherein, The output module is further configured to: monitor an occupancy state of the output queues, and send a backpressure signal to the input scheduling module to suspend data scheduling when the occupancy state satisfies a first condition, and withdraw the backpressure signal to resume data scheduling when the occupancy state satisfies a second condition.

7. The apparatus of claim 1, wherein, The device is configured to support continuous matrix multiplication task pipeline execution, wherein the input scheduling module starts to schedule input data of a next matrix multiplication task at a predetermined interval after scheduling of all input data of a previous matrix multiplication task is completed.

8. The apparatus of claim 7, wherein, The predetermined interval is determined based on at least one of the progress of emptying the systolic array based on output data of a previous task and the occupancy state of an output queue.

9. A systolic array based fine-grained quantized matrix multiplication method, characterized in that, The method applied to the systolic array based fine-grained quantized matrix multiplication device of any one of claims 1-8, the method comprising: attaching type identifiers to input matrix data and quantization factor data input to the systolic array; in each processing unit of the systolic array, selecting to perform multiply-accumulate operation or dequantization operation according to the data received upstream and the type identifier of the data; outputting the calculation result calculated by the systolic array.

10. The method of claim 9, wherein, The selecting to perform multiply-accumulate operation or dequantization operation according to the data received upstream and the type identifier of the data comprises: when the processing unit receives a type identifier indicating that the data is from the input matrix data, performing multiply-accumulate operation; when the processing unit receives a type identifier indicating that the data is from the quantization factor data, performing dequantization operation and resetting accumulation; wherein if the quantization factor data belongs to a row-column boundary, triggering output of the calculation result after dequantization; when the processing unit receives a type identifier indicating that the data is the calculation result, performing pass-through of the calculation result.

Images