Fine-grained quantization matrix multiplication device and method based on systolic array

Abstract

The invention relates to the technical field of computer systems, and provides a fine-grained quantization matrix multiplication device and method based on a systolic array, and the device comprises an input scheduling module which is used for receiving input matrix data and corresponding quantization factor data, and adding type identifiers to the input matrix data and the quantization factor data; the systolic array is formed by connecting a plurality of processing units in a two-dimensional net structure, and each processing unit is used for selectively executing multiply-accumulate operation or inverse quantization operation according to the data received at the upstream and the type identifier of the data; wherein the input scheduling module is used for scheduling data added with a type identifier to a systolic array according to rows and columns; and the output module is used for receiving and outputting a calculation result calculated by the systolic array. The problems that quantization factor insertion is low in efficiency, inverse quantization is separated from calculation flow and task switching overhead is large are solved, and the data reuse rate, the calculation efficiency and the overall energy efficiency ratio of the systolic array in a fine-grained quantization scene are remarkably improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer architecture technology, and more specifically, to a fine-grained quantitative matrix multiplication device and method based on a pulsating array. Background Technology

[0002] As a classic parallel computing architecture, systolic arrays are widely used in computationally intensive tasks such as matrix multiplication, digital signal processing, and artificial intelligence inference due to their regular data flow, high parallelism, and high efficiency. The basic principle of systolic arrays is to connect processing units in a regular grid, allowing data to systolic between rows and columns, and performing multiplication and accumulation operations in the intersecting processing units, thereby achieving efficient computation of matrix multiplication.

[0003] In related technologies, when performing fine-grained quantization matrix multiplication, systolic arrays suffer from problems such as disconnected hardware pipelines for inverse quantization and matrix multiplication calculations, and discontinuous task scheduling. This results in difficulties in efficiently inserting quantization factors, reduced data reuse, and high task switching overhead, severely limiting their computational efficiency and energy efficiency when supporting flexible quantization models. Summary of the Invention

[0004] The purpose of this disclosure is to provide a fine-grained quantitative matrix multiplication device and method based on a pulsating array to solve problems in related technologies.

[0005] To achieve the above objectives, this disclosure provides a fine-grained quantization matrix multiplication device based on a pulsating array, the device comprising: An input scheduling module is used to receive input matrix data and corresponding quantization factor data, and to attach type identifiers to the input matrix data and the quantization factor data. The pulsating array consists of multiple processing units connected in a two-dimensional mesh structure. Each processing unit is used to select to perform a multiply-accumulate operation or an inverse quantization operation based on the data received from upstream and the data type identifier. The input scheduling module is used to schedule data with attached type identifiers to the pulsating array by row and by column. The output module is used to receive and output the calculation results calculated by the pulsating array.

[0006] Optionally, the input scheduling module includes: A row scheduler, connected to the first processing unit of each row of the pulsating array, is used to schedule data row by row; A column scheduler, connected to the first processing unit of each column of the pulsating array, is used to schedule data by column; The row scheduler receives row data of the input matrix through multiple row data queues and receives corresponding quantization factor row data through multiple row quantization factor queues. The column scheduler receives column data of the input matrix through multiple column data queues and receives corresponding quantization factor column data through multiple column quantization factor queues.

[0007] Optionally, the type identifier includes: The first identifier is used to identify that the data comes from the input matrix data; The second identifier is used to identify that the data comes from quantified factor data; The third identifier is used to identify that the data comes from the quantization factor matrix and is a row and column boundary; The fourth identifier is used to identify the data as the calculation result.

[0008] Optionally, the processing unit is configured to: When data with the first identifier is received simultaneously from both the row and column directions, a multiply-accumulate operation is performed. When data with the second identifier is received simultaneously from both the row and column directions, an inverse quantization operation is performed, and the locally accumulated intermediate multiply-accumulate results are reset. When data with the third identifier is received simultaneously from both the row and column directions, an inverse quantization operation is performed and a calculation result is generated, which is then triggered to output the calculation result. When data with the fourth identifier is received simultaneously from both the row and column directions, the calculation result is passed through.

[0009] Optionally, each of the processing units further includes an output sequence flag, which is used to statically configure the processing unit to pass the calculation result to the output module along the row or column direction; The output module includes multiple output queues, which are used to cache the calculation results from the pulsating array according to the row-major or column-major order configured by the output sequence flag.

[0010] Optionally, the output module is further configured to: The occupancy status of the output queue is monitored, and when the occupancy status meets a first condition, a backpressure signal is sent to the input scheduling module to pause data scheduling; when the occupancy status meets a second condition, the backpressure signal is withdrawn to resume data scheduling.

[0011] Optionally, the apparatus is configured to support continuous pipelined execution of matrix multiplication tasks, wherein, at a predetermined interval after all input data of the previous matrix multiplication task has been scheduled, the input scheduling module begins to schedule the input data of the next matrix multiplication task.

[0012] Optionally, the predetermined interval is determined based on at least one of the progress of emptying the output data of the previous task from the pulsating array and the occupancy status of the output queue.

[0013] This disclosure also provides a fine-grained metric matrix multiplication method based on a systolic array, applied to the aforementioned fine-grained metric matrix multiplication device based on a systolic array, the method comprising: Add type identifiers to the input matrix data and quantization factor data input to the pulsation array; In each processing unit of the pulsating array, based on the data received from upstream and the data type identifier, either a multiply-accumulate operation or an inverse quantization operation is selected. Output the calculation results obtained by the pulsating array.

[0014] Optionally, the step of selecting to perform a multiply-accumulate operation or an inverse quantization operation based on the data received from the upstream and the data type identifier includes: When the processing unit receives a type identifier indicating that the data comes from the input matrix data, it performs a multiply-accumulate operation; When the processing unit receives quantization factor data indicating the data source as a type identifier, it performs dequantization and resets the accumulation; wherein, if the quantization factor data belongs to the row and column boundaries, the calculation result is triggered to be output after dequantization; When the processing unit receives a type identifier indicating that the data is the calculation result, it performs the pass-through of the calculation result.

[0015] By adopting the above technical solution, a unified type identifier is attached to the input data and quantization factor, and they are precisely inserted into the systolic array through the input scheduling module according to the same data flow pattern. This allows the quantization factor to enter the calculation process at the correct time. The processing units in the array dynamically switch to perform multiply-accumulate or dequantization operations based on the identifier, thereby integrating the dequantization calculation in place into the hardware pipeline of matrix multiplication, achieving deep integration and seamless connection between the two. This fundamentally solves the problems of inefficient quantization factor insertion, separation of dequantization and calculation pipeline, and high task switching overhead, significantly improving the data reuse rate, computational efficiency, and overall energy efficiency ratio of the systolic array in fine-grained quantization scenarios.

[0016] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings: Figure 1 This is an example diagram of the input and output of a fine-grained quantitative matrix multiplication.

[0018] Figure 2 This is a schematic diagram of the workflow of a pulse array structure with a fixed output.

[0019] Figure 3 This is a schematic diagram of a split matrix that considers fine-grained quantification.

[0020] Figure 4 This is a schematic diagram of a fine-grained quantization matrix multiplication device based on a pulsating array, according to an exemplary embodiment.

[0021] Figure 5 This is a schematic diagram illustrating the workflow of a processing unit according to an exemplary embodiment.

[0022] Figure 6 This is an output timing diagram illustrated according to an exemplary embodiment.

[0023] Figure 7 This is a flowchart illustrating a fine-grained quantitative matrix multiplication method based on a pulsating array, according to an exemplary embodiment.

[0024] Explanation of reference numerals in the attached figures 10 - Input scheduling module, 20 - Pulse array, 30 - Output module. Detailed Implementation

[0025] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0026] In the following description, the words "first" and "second" are used only to distinguish the purpose of the description and should not be interpreted as indicating or implying relative importance or order.

[0027] First, the pulsating array, quantization factor, tensor core, related terms, matrix multiplication based on outer and inner products, and dimension of matrix multiplication in this disclosure will be explained.

[0028] Pulsating arrays: Pulsating arrays are a parallel computing architecture proposed in the 1980s. The core idea is to arrange a large number of simple processing elements (PEs) in space and achieve high parallelism and high throughput through regular, synchronous, and rhythmic data flow. Data moves rhythmically forward in the PE array like blood pulsating in the heart, and is processed step by step during its movement.

[0029] Quantization factor: During linear quantization, the original data needs to be scaled to the target data precision type according to a certain ratio. For example, if the original data range is [-2047, 2048], and it needs to be scaled to int8, i.e., [-255, 256], the quantization factor is 8. Before calculation, the original data needs to be divided by 8 for quantization. After calculation, when it is necessary to restore the true value, it needs to be multiplied by 8 again for dequantization.

[0030] Tensor Core: A matrix multiplication acceleration unit used in GPUs, it is a type of dedicated hardware.

[0031] Matrix multiplication based on inner and outer products: Mathematically, matrix multiplication has two implementation methods: inner product and outer product. The inner product is based on vector inner product operations, taking a row of matrix A and a column of matrix B, multiplying each element and summing them to obtain an element of the output matrix. The outer product is based on vector outer product operations, taking a column of matrix A and a row of matrix B, calculating the pairwise product of all elements to obtain a matrix of the same size as the output matrix, and summing all such matrices bit by bit to obtain the final result. Pulsating arrays belong to the inner product method.

[0032] M / N / K dimensions of matrix multiplication: Generally, the shapes of matrices A and B are [M, K] and [K, N], respectively, where K is the summation dimension.

[0033] As large models grow in size, using lower-bit quantization such as FP8 / FP6 / FP4 can significantly alleviate memory bottlenecks and improve computation speed. On the other hand, to reduce the accuracy loss caused by quantization, the use of finer-grained quantization techniques has become a trend. (See also...) Figure 1 Assuming the input matrices for matrix multiplication have shapes A: [M, K] and B: [K, N], and the quantization factor matrices have shapes S_A: [M_scale, K_scale] and S_B: [K_scale, N_scale], they should satisfy M_scale = M / M_step, N_scale = N / N_step, and K_scale = K / K_step (all integer division), where matrix A is quantized at a granularity of [M_step, K_step], and matrix B at a granularity of [K_step, N_step]. For example, the activation layer parameter matrix A is quantized at a granularity of [1, 128], and the weight parameter matrix is ​​quantized at a granularity of [128, 128]. The final calculation result should satisfy D = (A... S_A)*(B S_B), where the operator * represents standard matrix multiplication. The element-wise multiplication of a matrix block is defined as multiplying the elements of matrix A that fall into the [i, j]th block by the scalar S_A[i, j] for inverse quantization.

[0034] Some existing commercial computing architectures do not support matrix multiplication with scaling factors. Instead, the matrix must be computed in chunks on Tensor Cores, and the output is then fed into a CUDA core and multiplied by a quantization factor for dequantization. While some architectures support matrix multiplication with quantization factors, they have several limitations: First, S_A: [M, K_scale], S_B: [K_scale, N], and K_scale ∈ {1, 2, 4}, meaning arbitrary chunking is not allowed. Second, there are requirements regarding the shape of the input matrices A and B, especially the K dimension. For example, in some architectures, the maximum K value is 64 for dense matrices and 128 for sparse matrices, limiting its applicability in larger-scale or variable-dimensional matrix operations.

[0035] On the other hand, while some commercial computing architectures have made some progress in systolic arrays, systolic arrays are typical matrix multiplication structures based on inner products, which have advantages such as low communication overhead, high energy efficiency, and flexibility, and natively support infinitely large K dimensions (as long as the accumulation accuracy is sufficient). However, traditional systolic arrays do not consider quantization factors. Without the support for quantization factors, the original large matrix can only be split into multiple small matrix multiplications according to the block size and then synchronized, which significantly reduces the efficiency of computing resources.

[0036] Please see Figure 2 It employs a fixed-output pulsating array structure, where each row of matrix A and each column of matrix B are arranged as follows: Figure 2 The clock cycles shown enter the processing units of the systolic array sequentially. Each processing unit performs multiplication and addition operations to obtain a partial sum. After all the data from the input matrix has flowed out of the systolic array, the processing unit stores the final result. It can be deduced that for input matrices A: [M, K] and B: [K, N], the size of the systolic array [N_syslotic, N_syslotic] ​​must satisfy N_syslotic ≥ max(M, N). Therefore, it takes M + K + N - 2 cycles to complete the calculation and N_syslotic cycles to complete the data output. The biggest advantage of the systolic array is its high data transmission efficiency and simple processing unit structure. The disadvantage is its relatively long latency, but the larger the K dimension, the higher the utilization rate of the processing units.

[0037] Specifically, for input matrices using fine-grained quantization, they can only be divided into many small matrices according to the quantization granularity. After each block is calculated, it is multiplied by the corresponding quantization factor for dequantization, and then multiplied synchronously according to the principle of matrix multiplication to obtain the final output result. This obviously leads to a huge latency. Figure 3As shown, without considering the calculation of the quantization factor and the delay of data output, the pulsating array directly calculates the matrix multiplication of [4, 2] × [2, 4], which requires 4 + 4 + 2 - 2 = 8 cycles. However, splitting it into 4 [2, 2] × [2, 2] matrix multiplications requires 4 (2 + 2 + 2 - 2) = 16 cycles.

[0038] In other words, when performing fine-grained quantization matrix multiplication, systolic arrays suffer from hardware pipeline separation between inverse quantization and matrix multiplication calculations, as well as discontinuous task scheduling. This results in difficulties in efficiently inserting quantization factors, reduced data reuse, and high task switching overhead, severely limiting their computational efficiency and energy efficiency when supporting flexible quantization models.

[0039] To address the aforementioned issues, a unified type identifier is attached to both input data and quantization factors. These are then precisely inserted into the systolic array via the input scheduling module according to the same data flow pattern, ensuring that the quantization factors enter the computation process at the correct time. The processing units within the array dynamically switch between multiply-accumulate or dequantization operations based on this identifier, thereby integrating dequantization computation directly into the matrix multiplication hardware pipeline, achieving deep integration and seamless connection between the two. This fundamentally solves the problems of inefficient quantization factor insertion, the disconnect between dequantization and computation pipelines, and high task switching overhead, significantly improving the data reuse rate, computational efficiency, and overall energy efficiency of the systolic array in fine-grained quantization scenarios.

[0040] Figure 4 This is a schematic diagram illustrating a fine-grained quantization matrix multiplication device based on a pulsating array, according to an exemplary embodiment. Please refer to [link to schematic diagram]. Figure 4 The fine-grained quantization matrix multiplication device based on pulsation array may include an input scheduling module 10, a pulsation array 20, and an output module 30, which are connected in sequence.

[0041] The input scheduling module 10 is used to receive input matrix data and corresponding quantization factor data, and to attach type identifiers to the input matrix data and quantization factor data.

[0042] The input matrix data can refer to the elements of two matrices A and B that need to be multiplied. In practical applications, the input matrix data can be pre-quantized, for example, using INT8 quantization, to reduce storage and computational overhead.

[0043] Quantization factor data can refer to the scaling factor used to dequantize quantized data.

[0044] In this embodiment, input matrices A and B correspond to quantization factor matrices S_A and S_B, respectively. For example, if a certain block of matrix A uses a quantization factor s_a, then during calculation, the elements of that block need to be multiplied by s_a to restore their original numerical range.

[0045] A type identifier can be an identifier attached to each data element to indicate whether the data belongs to the input matrix or the quantization factor. For example, two bits of "00" can represent the data of input matrix A, "01" can represent the data of input matrix B, "10" can represent the data of quantization factor S_A, and "11" can represent the data of quantization factor S_B. Setting a type identifier allows the processing unit to distinguish between different types of data and perform corresponding operations.

[0046] The pulsating array 20 consists of multiple processing units connected in a two-dimensional mesh structure. Each processing unit is used to select to perform a multiply-accumulate operation or an inverse quantization operation based on the data received from the upstream and the data type identifier. The input scheduling module 10 is used to schedule data with attached type identifiers to the pulsating array 20 by row and by column.

[0047] The input scheduling module 10 schedules data to the systolic array 20 in both row and column directions: data for matrix A and quantization factor S_A are scheduled row-wise and sent to the first processing unit of each row in the systolic array 20; data for matrix B and quantization factor S_B are scheduled column-wise and sent to the first processing unit of each column in the systolic array 20. Data flows in a systolic manner within the systolic array 20: each clock cycle, the processing unit passes the received data to the adjacent downstream processing unit while simultaneously performing corresponding operations locally.

[0048] The multiply-accumulate operation can refer to the basic operation in standard matrix multiplication performed by the processing unit, namely c = c + a × b, where a and b come from input matrices A and B respectively, and c is the accumulated partial sum.

[0049] The dequantization operation can refer to the quantization factor-related operation performed by the processing unit. For example, when the processing unit simultaneously receives elements from quantization factor matrices S_A and S_B, it performs the operation sum = sum + c × s_a × s_b, where c is the previously accumulated multiplication and summation result, s_a and s_b are the quantization factors, and sum is the calculated result, i.e., the final dequantization result. After execution, c is reset to 0.

[0050] For example, for a systolic array 20 with N_syslotic rows and N_syslotic columns, the maximum input matrix data A:[M,K] and B[K,N] that it can handle must satisfy M≤N_systolic and N≤N_systolic. Meanwhile, considering hardware complexity and resource overhead, the quantization factor matrix data can be set to a relatively fixed shape, i.e., S_A:[M,K_scale] and S_B:[K_scale,N]. Since a systolic array 20 is used, the dimensions K and K_scale can theoretically be infinitely large, as long as the data loading time can overlap with the computation time and the accuracy requirements are met.

[0051] In one embodiment, each processing unit determines which operation to perform based on the received data type identifier: Case 1: Multiplication and accumulation operation When the processing unit simultaneously receives data marked "00" from the row direction (from matrix A) and data marked "01" from the column direction (from matrix B), it performs a multiply-accumulate operation: c = c + a × b.

[0052] Scenario 2: Inverse Quantization Operation When the processing unit receives data marked "10" (from quantization factor S_A) from the row direction and data marked "11" (from quantization factor S_B) from the column direction simultaneously, it performs an inverse quantization operation: sum = sum + c × s_a × s_b, and then resets the local accumulator c to 0.

[0053] Case 3: Special handling of boundary quantization factors When the quantization factor data identifier indicates that it is a row or column boundary, the processing unit will also trigger the operation of passing the final calculation result sum to the output module 30 after performing the dequantization operation.

[0054] Output module 30 is used to receive and output the calculation results calculated by pulsation array 20.

[0055] The calculation results of the systolic array 20 are output through the output module 30, which contains multiple output queues and caches the final results from each processing unit of the systolic array 20 in row-major or column-major order.

[0056] By uniformly attaching type identifiers to input data and quantization factors, and precisely inserting them into the pulsating array 20 via the input scheduling module 10 according to the same data flow pattern, the quantization factors can enter the calculation process at the correct time. The processing units in the array dynamically switch to perform multiply-accumulate or dequantization operations based on the identifier, thereby integrating the dequantization calculation in place into the hardware pipeline of matrix multiplication, achieving deep integration and seamless connection between the two. This fundamentally solves the problems of inefficient quantization factor insertion, separation of dequantization and calculation pipelines, and high task switching overhead, significantly improving the data reuse rate, computational efficiency, and overall energy efficiency ratio of the pulsating array 20 in fine-grained quantization scenarios.

[0057] In one possible implementation, the input scheduling module 10 may include a row scheduler and a column scheduler.

[0058] The row scheduler, connected to the first processing unit of each row of the pulsating array 20, is used to schedule data row by row; The column scheduler, connected to the first processing unit of each column of the pulsating array 20, is used to schedule data by column.

[0059] The first processing unit can refer to the first processing unit in each row or column of the systolic array 20. It is the entry point for data into the systolic array 20. In the row direction, the first processing unit in the i-th row can be denoted as PEi1; in the column direction, the first processing unit in the j-th column can be denoted as PE1j.

[0060] If the systolic array 20 has N rows, then N row schedulers can be configured accordingly. The output of each row scheduler is connected to the first processing unit of the corresponding row via a data path. For example, the first row scheduler is connected to processing unit PE11, the second row scheduler is connected to processing unit PE21, and so on. The row scheduler receives all row data of matrix A and all row data of quantization factor matrix S_A. Each row scheduler has one or more row buffer queues internally for temporarily storing the data it receives.

[0061] When the pulsating array 20 is ready to receive new data, the row scheduler transmits the data in the row buffer queue to the corresponding row head processing unit in sequence according to the preset timing rules.

[0062] The specific process is as follows: (1) During the data loading phase, before the computation task begins, the elements of the i-th row of matrix A [a_{i1}, a_{i2}, ..., a_{iK}] and the elements of the i-th row of the quantization factor matrix S_A [s_{i1}, s_{i2}, ..., s_{iK_scale}] are loaded into the corresponding buffer queues in the i-th row scheduler.

[0063] Each row scheduler may include multiple row data queues and multiple row quantization factor queues. The row scheduler receives row data of the input matrix through multiple row data queues and receives corresponding quantization factor row data through multiple row quantization factor queues. The row data queues are used to store the i-th row element of matrix A, and the row quantization factor queues are used to store the i-th row element of the quantization factor matrix S_A.

[0064] (2) Transmission start and sequence control: After the start signal is calculated, the i-th row scheduler first takes the first element (e.g., a_{i1}) from the row data queue, adds a type identifier indicating that it belongs to the input matrix (e.g., "00"), and transmits it to the first processing unit PEi1 of the row in the next clock cycle. Thereafter, the scheduler takes out a_{i2}, a_{i3}, ..., a_{iK} in the first-in-first-out order and transmits them one by one.

[0065] (3) The insertion and transmission of quantization factors are carried out according to the preset timing rules. For example, in the next cycle after X row data elements have been transmitted, the row scheduler automatically switches to the row quantization factor queue and starts transmitting a quantization factor element. After that, after X row data elements are transmitted again, a quantization factor element is transmitted, and so on, until all data transmission is completed.

[0066] In other embodiments, the elements of matrix A and factor matrix S_A can be interleaved according to a preset rule. For example, every three elements of matrix A are followed by one element of factor matrix S_A to form a mixed queue.

[0067] If the pulsating array 20 has M columns, then M column schedulers can be configured accordingly. The output of each column scheduler is connected to the first processing unit of the corresponding column via a data path. For example, the first column scheduler is connected to processing unit PE11, the second column scheduler is connected to processing unit PE12, and so on. The column schedulers receive all column data of matrix B and all column data of quantization factor matrix S_B. Each column scheduler has an internal column buffer queue for temporarily storing the data it receives.

[0068] When the pulsating array 20 is ready to receive new data, the column scheduler transmits the data in the column buffer queue to the corresponding column head processing unit in sequence according to the preset timing rules.

[0069] The specific process is as follows: (1) During the data loading phase, before the computation task begins, the elements of the j-th column of matrix B [b_{1j}, b_{2j}, ..., b_{Kj}] and the elements of the j-th column of the quantization factor matrix S_B [s_{1j}, s_{2j}, ..., s_{K_scale j}] are loaded into the corresponding buffer queues in the j-th column scheduler.

[0070] Each column scheduler may include multiple column data queues and multiple column quantization factor queues. The column scheduler receives column data of the input matrix through multiple column data queues and receives corresponding quantization factor column data through multiple column quantization factor queues. The column data queues are used to store the j-th column element of matrix B, and the column quantization factor queues are used to store the j-th column element of quantization factor matrix S_B.

[0071] (2) Transmission start and sequence control: After the start signal is calculated, the j-th column scheduler first takes the first element (e.g., b_{1j}) from the column data queue, adds a type identifier indicating that it belongs to the input matrix (e.g., "01"), and transmits it to the column head processing unit PE1j in the next clock cycle. Thereafter, the scheduler takes out b_{2j}, b_{3j}, ..., b_{Kj} in the order of First In First Out (FIFO) and transmits them one by one.

[0072] (3) The insertion and transmission of quantization factors are carried out according to the preset timing rules. For example, in the next cycle after Y column data elements have been transmitted, the column scheduler automatically switches to the column quantization factor queue and starts to transmit one quantization factor element. After that, it starts to transmit Y column data elements again, then transmits one quantization factor element, and so on, until all data transmission is completed.

[0073] In other embodiments, the elements of matrix B and factor matrix S_B can be interleaved according to a preset pattern. For example, every two elements of matrix B are followed by one element of factor matrix S_B, forming a mixed queue.

[0074] At the same time, each quantization factor element is appended with a type identifier indicating that it belongs to quantization factor data and whether it is a boundary element. For non-boundary quantization factor elements (such as the first K_scale-1 elements), an identifier "10" is appended. For the last boundary quantization factor element, an identifier "11" is appended to indicate the row / column tail quantization factor.

[0075] By setting separate row and column schedulers, row and column data can be scheduled independently and in parallel, giving full play to the two-dimensional parallel computing capabilities of the systolic array 20. Since each scheduler is only responsible for data in a specific row or column, data cross-access and contention are reduced. When the scale of the systolic array 20 increases, only the number of row / column schedulers needs to be increased accordingly, making the architecture easy to expand. The row and column schedulers can independently control their respective data flow rates and timings, making it easy to adapt to different matrix shapes and quantization configurations. Local failures of row or column schedulers do not affect other parts of the entire system, which also improves system reliability.

[0076] In one possible implementation, the type identifier may include: The first identifier is used to identify that the data comes from the input matrix data; The second identifier is used to identify that the data comes from quantified factor data; The third identifier is used to identify that the data comes from the quantization factor matrix and is a row and column boundary; The fourth identifier is used to identify the data as the calculated result, sum.

[0077] The third identifier can be used to identify the last element of each row in the quantization factor matrix S_A and the last element of each column in the quantization factor matrix S_B. In other embodiments, the third identifier may also include a first sub-identifier and a second sub-identifier, wherein the first sub-identifier is used to identify the last element of each row in the quantization factor matrix S_A and the second sub-identifier is used to identify the last element of each column in the quantization factor matrix S_B.

[0078] By defining a complete data identification system, the processing units in the pulsating array 20 can accurately distinguish data from different sources and properties, and perform corresponding calculation operations.

[0079] In one possible implementation, the processing unit is configured as follows: When data with the first identifier is received simultaneously from both the row and column directions, perform a multiply-accumulate operation. When data with the second identifier is received simultaneously from both the row and column directions, perform dequantization and reset the locally accumulated intermediate multiply-accumulate results. When data with a third identifier is received simultaneously from both the row and column directions, an inverse quantization operation is performed and the final calculation result is generated, which is then triggered to output the final calculation result. When data with a fourth identifier is received simultaneously from both the row and column directions, the calculation result sum is passed through.

[0080] The process of multiply-accumulate operation is as follows: When the processing unit receives data carrying the first identifier from both the row and column directions, it performs the basic operation of row standard matrix multiplication, c = c + a × b. The row direction data a comes from the input matrix A, and the column direction data b comes from the input matrix B. The multiply-adder inside the processing unit is enabled, the multiplier calculates the result of a × b, the adder adds the result to the current value of the local accumulator register c, and the result is written back to register c.

[0081] The process of non-boundary dequantization is as follows: When the processing unit receives data carrying the second identifier from both the row and column directions simultaneously, it performs the dequantization calculation sum = sum + c × s_a × s_b, and then resets the local accumulator c to 0. The row direction data s_a comes from the non-boundary elements of the quantization factor matrix S_A, and the column direction data s_b comes from the non-boundary elements of the quantization factor matrix S_B. The multiplication-multiplication-addition logic inside the processing unit is activated. First, c × s_a is calculated to obtain an intermediate result, then multiplied by s_b, and finally added to the sum register. After the operation is completed, the reset control logic clears the c register to zero.

[0082] The boundary dequantization operation process is as follows: When the processing unit receives at least one data carrying a third identifier from both the row and column directions simultaneously (this could be two third identifiers or a combination of a second identifier and a third identifier), meaning at least one quantization factor is a boundary element of the row or column, it performs an inverse quantization calculation sum = sum + c × s_a × s_b, then resets c to 0, and subsequently triggers the output of the final calculation result sum. In addition, the boundary detection logic sets the output trigger flag and controls the output multiplexer to send the value of the sum register to the output port.

[0083] The pass-through of the calculation result sum can be understood as passing the calculation result sum to the next processing unit in the row direction in a row-major order, or passing the calculation result sum to the next processing unit in the column direction in a column-major order.

[0084] By using identifier control instead of fixed hardwired connections, it supports multiple quantization modes and computation configurations, and the explicit identifier-driven mechanism clearly separates the data flow and control flow, making verification and debugging easier.

[0085] In one possible implementation, the processing unit may also be configured as: Before transmitting data downstream, update the step size information associated with the data, and decide whether to continue transmitting the data based on the updated step size information.

[0086] The step size information is a counter value associated with each data element, indicating the remaining step size that the data still needs to be passed in the systolic array 20. When data enters the systolic array 20 from the input scheduling module 10, it is assigned an initial step size value. For input matrix data, the initial step size value is usually equal to the number of columns N (for row-direction input matrix data) or the number of rows M (for column-direction input matrix data). For quantization factor data, the initial step size value is set to the number of columns N (for row-direction quantization factor data) or the number of rows M (for column-direction quantization factor data).

[0087] When a processing unit is ready to pass data to a downstream row / column processing unit, it decrements the step size value carried in the data by one. The decision to continue passing the data is based on the updated step size information. This can be understood as follows: the processing unit checks the updated step size value; if the updated step size value is greater than 0, the data needs to be passed, and the processing unit sends the data, type identifier, and the updated step size to the downstream neighboring processing unit. If the updated step size value is 0, the processing unit stops passing the data.

[0088] In other embodiments, a remaining step size can be set for the calculation result sum, i.e., the final inverse quantization result, to ensure that the calculation result is transmitted.

[0089] By using a step-size control mechanism, the fixed data flow mode of the systolic array 20 is upgraded to a programmable dynamic data flow. While maintaining the high efficiency and regularity of the systolic array 20, its ability to support diverse computing tasks is greatly enhanced, providing a solid hardware foundation for achieving high-throughput, low-latency quantized neural network inference.

[0090] In one possible implementation, each processing unit further includes an output sequence flag, which is used to statically configure the processing unit to pass the calculation results to the output module 30 along the row or column direction; the output module 30 includes multiple output queues, which are used to cache the calculation results from the pulsating array 20 according to the row-major order or column-major order configured by the output sequence flag.

[0091] The output sequence flag can be a configuration register bit stored inside each processing unit, used to indicate that the processing unit should pass its final calculation result to the output module 30 in the row / column direction.

[0092] In one embodiment, when the output sequence flag is 0 or in row mode, the processing unit passes the calculation result to the processing unit adjacent to the right of the same row, and finally outputs it by the rightmost processing unit of each row; when the output sequence flag is 1 or in column mode, the processing unit passes the calculation result to the processing unit adjacent to the bottom of the same column, and finally outputs it by the bottommost processing unit of each column.

[0093] The row-major output can be the calculation result being passed along the row direction inside the pulsating array 20. The rightmost processing unit of each row serves as the output port of that row. Multiple output queues can be multiple row output queues. Each row output queue is used to receive the calculation result of the rightmost processing unit in a row based on the cached row-major output.

[0094] The column-major output can be the calculation result being transmitted along the column direction inside the pulsating array 20. The bottom processing unit of each column serves as the output port of that column. Multiple output queues can be multiple column output queues. Each column output queue is used to receive the calculation result of the bottom processing unit of a column based on the cached column-major output.

[0095] By introducing configurable output sequence flags in the processing unit, the same systolic array 20 hardware can directly output row-major or column-major result matrices as needed. This not only reduces data transfer overhead and latency, but also allows the memory layout of the output data to directly match the access patterns of subsequent processing units, significantly improving the overall system energy efficiency and throughput, and enhancing the hardware's adaptability to different computing tasks.

[0096] In one possible implementation, the output module 30 can also be configured to: monitor the occupancy status of the output queue, and when the occupancy status meets a first condition, send a backpressure signal to the input scheduling module 10 to suspend data scheduling; and when the occupancy status meets a second condition, cancel the backpressure signal to resume data scheduling.

[0097] The occupancy status of the output queue reflects the degree of data backlog at the output end. It can refer to the ratio or a specific value of the amount of data stored but not yet read in the output queue to the total capacity of the queue. It can be expressed as an absolute quantity (e.g., "32 data items stored"), a percentage (e.g., "occupancy rate 80%)", or a graded status (e.g., "idle", "normal", "warning", "full").

[0098] The first condition can be that the occupancy status of the output queue indicates that the queue is about to be full, and continuing to receive new data may cause an overflow. The backpressure signal can be used to send a pause request to the input scheduling module 10, requesting it to temporarily stop or slow down the transmission of new data to the pulse array 20.

[0099] The second condition can be a representation of the occupancy status of the output queue, indicating that the queue has enough space to receive new data and can continue to receive new data.

[0100] By setting a flow control mechanism for the output module 30, the system can dynamically control the scheduling of input data by monitoring the occupancy status of the output queue, thereby preventing data overflow or idle computing resources and ensuring stable and efficient system operation.

[0101] In one possible implementation, the apparatus is configured to support continuous pipelined execution of matrix multiplication tasks, wherein, at a predetermined interval after all input data for the previous matrix multiplication task has been scheduled, the input scheduling module 10 begins to schedule the input data for the next matrix multiplication task.

[0102] When all input data for the previous matrix multiplication task (e.g., task 1) has been scheduled, it can be understood that the last data element (input matrix element or quantization factor) of task 1 has been sent into the pulsation array 20 by the input scheduling module 10. At this time, all input data of task 1 has entered the device, but is still flowing and being calculated in the array. The output results of task 1 have not yet been fully generated and output, and there is still a large amount of intermediate data of task 1 being transmitted and calculated in the pulsation array 20. The output module 30 may also still be receiving the results of task 1.

[0103] The predetermined interval can be a configurable number of delay cycles, i.e., the number of clock cycles between the completion of the input of Task 1 and the start of the input of the next matrix multiplication task (e.g., Task 2). It should be understood that the predetermined interval needs to be long enough to ensure that the data of Task 2 does not prematurely overwrite or interfere with the data of Task 1 that is still in the array; but it cannot be too long to avoid hardware idleness.

[0104] By using a pipelined execution mechanism, the limitation of traditional pulse arrays, which can only process one computing task at a time, is broken. This allows for the near-continuous full utilization of hardware computing resources, especially in scenarios that require processing massive matrix operations, resulting in significant performance improvements and energy efficiency enhancements.

[0105] In one possible implementation, the predetermined interval is determined based on at least one of the progress of emptying the output data of the previous task from the pulsating array 20 and the occupancy status of the output queue.

[0106] The progress of emptying the output data of the previous task from the pulsating array 20 can be understood as the degree to which the calculation result data of the previous task is transferred from the last processing unit to the output module 30. This progress can be obtained through indicators such as the remaining time from the completion of the task input to the last result leaving the array, the number or proportion of processing units in the pulsating array 20 that still contain the result data of the previous task, and the number of result elements of the previous task that have not yet been output.

[0107] The occupancy status of the output queue can be understood as the degree of fill of the output queue that stores the results of the previous task.

[0108] In one embodiment, N_systolic row FIFO queues can be allocated to receive each row element of matrix A, N_systolic column FIFO queues to receive each column element of matrix B, N_systolic row quantization factor FIFO queues to receive each row element of matrix S_A, and N_systolic column quantization factor FIFO queues to receive each column element of matrix S_B. The depth of each queue is sufficiently large to allow the time it takes for the memory controller to load half the data from shared memory (SRAM) or dedicated tensor memory (TMEM) to be sufficient for the systolic array 20 to complete the computation of the other half of the data, thus enabling pipelining. M ≤ N_systolic, N ≤ N_systolic, and some row, column, and quantization factor FIFO queues can be set to be empty, i.e., the number of rows in matrix A is less than the length and width of the systolic array 20, and the number of columns in matrix B is less than the length and width of the systolic array 20.

[0109] In one embodiment, the input scheduling module 10 can be configured with up to N_systolic row data FIFO queues, N_systolic column data FIFO queues, N_systolic row quantization factor FIFO queues, and N_systolic column quantization factor FIFO queues, respectively used to buffer the row data of matrix A, the column data of matrix B, the row data of quantization factor matrix S_A, and the column data of quantization factor matrix S_B. The size of the input matrix must satisfy M ≤ N_systolic and N ≤ N_systolic (M and N are the number of rows in matrix A and the number of columns in matrix B, respectively). In actual execution, only the first M row data FIFO queues and the first M row quantization factor FIFO queues are used to receive data from matrix A and S_A; only the first N column data FIFO queues and the first N column quantization factor FIFO queues are used to receive data from matrix B and S_B. The remaining queues can remain empty or be disabled. The depth of each queue is configured such that the time required to load a set of data from external memory, such as Static Random-Access Memory (SRAM) or dedicated tensor memory, does not exceed the time required for the systolic array 20 to compute another set of ready data. This achieves pipeline overlap between data loading and computation, avoiding idle computing units due to insufficient data supply.

[0110] N_systolic row input schedulers can be configured. The i-th row scheduler is connected to the i-th row FIFO queue and the row quantization factor FIFO queue. If the i-th row is functional for the current input matrix size, the i-th row scheduler starts from the i-th clock cycle, sequentially taking the head element from the i-th row element input FIFO queue, and retrieving the quantization factor from the corresponding row quantization factor queue every K_step clock cycles. The column scheduler works in the same way as the row scheduler and will not be described further here.

[0111] After receiving the back pressure signal sent by the output module 30, the row / column scheduler in the input scheduling module 10 will stop scheduling until it receives the recovery signal sent by the output module 30, that is, cancels the back pressure signal.

[0112] After acquiring data, the row and column schedulers add a 2-bit type identifier and a log2N_syslotic bit remaining step size, i.e., step size information, to the data. Four type identifiers are defined: 2'b00 represents data from the input matrix; 2'b01 represents data from the quantization factor matrix but not from the row or column tail; 2'b10 represents data from the row or column tail of the quantization factor matrix; and 2'b11 represents the final multiplication-accumulation result. The first three type identifiers are set by the scheduler after acquiring data, and the corresponding remaining step size is initialized to N, or M for the column scheduler. The last one is set by the processing unit when generating data, with the remaining step size left_step set large enough to ensure the output of a systolic array 20, such as N_syslotic. Setting left_step is mainly to enable pipelined continuous execution of several matrix multiplications of different shapes. It can also stop scheduling for several cycles after the scheduler schedules the row or column tail elements of the quantization factor to ensure pipeline correctness.

[0113] In other embodiments, a type identifier for invalid data or a specific remaining step value can be added to distinguish the four different types of data.

[0114] Once the row / column scheduler has executed all elements, meaning that the matrix rows and columns have been sent to the systolic array 20, the scheduler can be allowed to schedule the next matrix multiplication at intervals of several clock cycles. In other words, the input of the next matrix multiplication can be pipelined.

[0115] In the systolic array 20, there is a two-dimensional mesh structure consisting of N_syslotic rows and N_syslotic columns of processing units. Each processing unit can obtain data from the upstream row and column neighbor processing units and pass the data to the downstream row / column neighbor processing units.

[0116] The results calculated by the systolic array 20 can be output in either row-major or column-major order. In other embodiments, for simplicity, a static configuration can also be considered, such as each processing unit having an output order flag to record whether row-major or column-major order is enabled. If the next matrix multiplication needs to change the output order, it is necessary to wait until the systolic array 20 is empty, refresh the output order flags of all processing units, and then execute the next matrix multiplication task.

[0117] Each processing unit can process data according to its type, and the internal processing flow is as follows: Figure 5 As shown.

[0118] a) If the output data of both upstream processing units (including the scheduler) is Ready, and the type identifier of the processing unit is 2'b00, i.e., the input matrix... or For elements, a standard systolic array 20-cell multiply-accumulate operation is performed. Specifically, assuming the input of the upstream node in the row and column is... and The locally stored segmented multiplication and summation is (If initialized to 0 each time), then execute. The specific hardware circuit can be optimized according to the data type (integer or floating-point) and bit width; this embodiment does not impose any restrictions. After the calculation is complete, left_step is decremented by one. If it equals 0, the Ready signal downstream is set to False; otherwise, the calculation continues. and The data is transmitted to the downstream row processing unit neighbors, and the Ready signal is set to True for all of them.

[0119] b) If the output data of both upstream processing units (including the scheduler) are both Ready, and the type identifier of the processing unit is 2'b01, i.e., the non-row / column tail data of the quantization factor matrix, then perform the dequantization multiplication operation. Specifically, assume the input is... and Then execute and Reset (i.e.) ) operation, in which This represents the locally stored result used for the final output. Similarly, specific hardware circuitry can be optimized based on data type (integer or floating-point) and bit width; this embodiment does not impose such limitations. For example, and It's a floating-point number, but and Using unsigned integers to represent the exponent of floating-point numbers, such as UE8M0, the calculation can be transformed into increasing the exponent part of c. After the calculation is complete, `left_step` is decremented by one. If it equals 0, the `Ready` signal downstream is set to `False`; otherwise, the calculation continues. and The data is transmitted to the downstream row processing unit neighbors, and the Ready signal is set to True for all of them.

[0120] c) If the output data of both upstream processing units (including the scheduler) is Ready, and the type identifier of the processing unit is 2'b10, i.e., the row and column tail data of the quantization factor matrix, then after performing the dequantization multiplication and quantization factor transfer operations in the current clock cycle, it is also necessary to set the local calculation end flag to indicate that the operation will be performed in subsequent cycles. The transmission (case e). Similar to case a), left_step also needs to be updated before deciding whether to transmit. and And set the corresponding Ready signal.

[0121] d) If only one data item in the upstream row or column processing unit (including the scheduler) is Ready and matches the row / column major order flag, and the processing unit's type identifier is 2'b11, then the execution data (including the upstream data) is executed. The row or column pass-through (based on the row and column primary order flag) is identified by the type identifier of the processing unit as 2'b11.

[0122] e) If the local computation end flag is true, and there is no upstream corresponding to case d) The pass-through, according to the row and column major order flag, transmits the local calculation results. Send the signal to the downstream row or column processing unit, and set the corresponding Ready signal and the type identifier of the processing unit to 2'b11. Reset.

[0123] f) If both a) and e) are satisfied, and the data element no longer needs to be transmitted to the downstream processing unit (i.e., leaf_step == 0 after the update), then the transmission can be completed within the same cycle. ,local Reset, and for Multiply-accumulate update operations, such as Figure 6 The T+4 cycle in the middle.

[0124] Output module 30 allocates N_syslotic output queues (FIFOs) to receive the output matrix C in row-major or column-major order. Similar to the input queues, each queue must be sufficiently deep. When configured for row-major order, the tail processing unit of each row in systolic array 20 sequentially sends the accumulated sum (i.e., the calculated result `sum`) to the output queue. The enqueue order is consistent with the address order of the accumulated sum, so the data in the output queue only needs to be written to the output memory address specified by the matrix multiplication function at appropriate times. Similarly, when configured for column-major order, the tail processing unit of each column in systolic array 20 sends the accumulated sum to the output queue. When any output queue is determined to be full (i.e., the remaining buffer size is less than a first set threshold), a backpressure signal is sent to the input module. When the remaining buffer size is greater than a second set threshold, a recovery signal is sent to the input module. The second set threshold is greater than the first set threshold to avoid frequent signaling.

[0125] A separate output queue is allocated primarily to enable the reuse of multiple matrix multiplications. Traditional pulsating array architectures with fixed outputs require flushing the result after each matrix multiplication before executing the next. By creating a pipeline of multiple matrix multiplications, subsequent multiplications do not need to wait for the complete output of the previous one; they can enter the input module after sufficient intervals, improving the fluidity between task operations.

[0126] like Figure 6 As shown, when the number of row elements output by two matrix multiplications is 4 and 2 respectively, the latter can be input into the matrix in the second cycle of the last quantization factor of the former matrix, without waiting for it to be completely emptied. The scheduler records and maintains the type identifier and other information of each data. Based on the number of row or column elements in each matrix operation, whether the output is row-major or column-major, and whether the end of the row or column has been reached, the summation result in each processing unit of the pulsating array also flows in the direction of rows or columns, sequentially arriving at the output queue of the output module.

[0127] exist Figure 6 The diagram shows the first row scheduler and processing unit, with a 4x4 pulsating array. The first matrix multiplication has K=4 and N=4, while the second matrix multiplication has K'=2 and N'=2. The row scheduler can begin scheduling in the second cycle (T+2) after scheduling the first matrix (cycle T in the diagram). Here, ** represents the sum+=c*a*b operation, *+ represents the c+=a*b operation, -> represents the pass-through of the upstream result, =0 represents the local sum being transmitted downstream and then reset to 0, and *+0 represents the local sum being transmitted downstream, sum=0, and c+=a*b operations.

[0128] Based on the same inventive concept, this disclosure also provides a fine-grained metric matrix multiplication method based on systolic arrays. This fine-grained metric matrix multiplication method based on systolic arrays can be applied to fine-grained metric matrix multiplication devices based on systolic arrays. Please refer to [link to relevant documentation]. Figure 7 The method may include steps S1 to S3.

[0129] Step S1: Add type identifiers to the input matrix data and quantization factor data input to the pulsation array.

[0130] Step S2: In each processing unit of the pulsating array, based on the data received from upstream and the data type identifier, select to perform a multiply-accumulate operation or an inverse quantization operation.

[0131] Step S3: Output the calculation results obtained by the pulsating array.

[0132] In one possible implementation, step S2 may include: When the processing unit receives a type identifier indicating that the data comes from the input matrix data, it performs a multiply-accumulate operation; When the processing unit receives a type identifier indicating that the data source is quantization factor data, it performs dequantization and resets the accumulation; if the quantization factor data belongs to the row and column boundaries, the calculation result is triggered to be output after dequantization. When the processing unit receives a type identifier indicating that the data is a calculation result, it performs the pass-through of the calculation result.

[0133] In one embodiment, the fine-grained quantization matrix multiplication method based on pulsating arrays may further include: Before transmitting data downstream, update the step size information associated with the data, and decide whether to continue transmitting the data based on the updated step size information.

[0134] In one embodiment, the fine-grained quantization matrix multiplication method based on pulsating arrays may further include: The occupancy status of the output queue is monitored, and when the occupancy status meets a first condition, a backpressure signal is sent to the input scheduling module to pause data scheduling; when the occupancy status meets a second condition, the backpressure signal is withdrawn to resume data scheduling.

[0135] Regarding the fine-grained quantization matrix multiplication method based on pulsating arrays in the above embodiments, the specific manner in which each module performs its operations has been described in detail in the embodiments related to the device, and will not be elaborated here.

[0136] The fine-grained quantitative matrix multiplication device and method based on pulsating arrays disclosed herein have at least the following advantages: 1) Achieve fine-grained quantization while maintaining the efficiency and flexibility of the pulsating array; 2) Supports pipelined execution of multiple consecutive matrix multiplications, enabling efficient reuse of computing resources; 3) Supports both row-major and column-major output modes, without the need for additional matrix transpose.

[0137] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0138] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0139] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A fine-grained quantization matrix multiplication device based on a pulsating array, characterized in that, The device includes: An input scheduling module is used to receive input matrix data and corresponding quantization factor data, and to attach type identifiers to the input matrix data and the quantization factor data. The pulsating array consists of multiple processing units connected in a two-dimensional mesh structure. Each processing unit is used to select to perform a multiply-accumulate operation or an inverse quantization operation based on the data received from upstream and the data type identifier. The input scheduling module is used to schedule data with attached type identifiers to the pulsating array by row and by column. The output module is used to receive and output the calculation results calculated by the pulsating array.

2. The apparatus according to claim 1, characterized in that, The input scheduling module includes: A row scheduler, connected to the first processing unit of each row of the pulsating array, is used to schedule data row by row; A column scheduler, connected to the first processing unit of each column of the pulsating array, is used to schedule data by column; The row scheduler receives row data of the input matrix through multiple row data queues and receives corresponding quantization factor row data through multiple row quantization factor queues. The column scheduler receives column data of the input matrix through multiple column data queues and receives corresponding quantization factor column data through multiple column quantization factor queues.

3. The apparatus according to claim 1 or 2, characterized in that, The type identifier includes: The first identifier is used to identify that the data comes from the input matrix data; The second identifier is used to identify that the data comes from quantified factor data; The third identifier is used to identify that the data comes from the quantization factor matrix and is a row and column boundary; The fourth identifier is used to identify the data as the calculation result.

4. The apparatus according to claim 3, characterized in that, The processing unit is configured as follows: When data with the first identifier is received simultaneously from both the row and column directions, a multiply-accumulate operation is performed. When data with the second identifier is received simultaneously from both the row and column directions, an inverse quantization operation is performed, and the locally accumulated intermediate multiply-accumulate results are reset. When data with the third identifier is received simultaneously from both the row and column directions, an inverse quantization operation is performed and a calculation result is generated, which is then triggered to output the calculation result. When data with the fourth identifier is received simultaneously from both the row and column directions, the calculation result is passed through.

5. The apparatus according to claim 1, characterized in that, Each of the processing units further includes an output sequence flag, which is used to statically configure the processing unit to pass the calculation result to the output module along the row or column direction. The output module includes multiple output queues, which are used to cache the calculation results from the pulsating array according to the row-major or column-major order configured by the output sequence flag.

6. The apparatus according to claim 5, characterized in that, The output module is also configured to: The occupancy status of the output queue is monitored, and when the occupancy status meets a first condition, a backpressure signal is sent to the input scheduling module to pause data scheduling; when the occupancy status meets a second condition, the backpressure signal is withdrawn to resume data scheduling.

7. The apparatus according to claim 1, characterized in that, The device is configured to support the pipelined execution of continuous matrix multiplication tasks, wherein, at a predetermined interval after all input data of the previous matrix multiplication task has been scheduled, the input scheduling module begins to schedule the input data of the next matrix multiplication task.

8. The apparatus according to claim 7, characterized in that, The predetermined interval is determined based on at least one of the progress of emptying the output data of the previous task from the pulsating array and the occupancy status of the output queue.

9. A fine-grained quantized matrix multiplication method based on a pulsating array, characterized in that, The method, applied to the fine-grained quantization matrix multiplication apparatus based on a pulsating array as described in any one of claims 1-8, comprises: Add type identifiers to the input matrix data and quantization factor data input to the pulsation array; In each processing unit of the pulsating array, based on the data received from upstream and the data type identifier, either a multiply-accumulate operation or an inverse quantization operation is selected. Output the calculation results obtained by the pulsating array.

10. The method according to claim 9, characterized in that, The step of selecting to perform a multiply-accumulate operation or an inverse quantization operation based on the data received from the upstream and the data type identifier includes: When the processing unit receives a type identifier indicating that the data comes from the input matrix data, it performs a multiply-accumulate operation; When the processing unit receives quantization factor data indicating the data source as a type identifier, it performs dequantization and resets the accumulation; wherein, if the quantization factor data belongs to the row and column boundaries, the calculation result is triggered to be output after dequantization; When the processing unit receives a type identifier indicating that the data is the calculation result, it performs the pass-through of the calculation result.

Images