Fine-grained quantization matrix multiplication device and method based on pulsating array

Abstract

This disclosure relates to the field of computer architecture technology, and provides a fine-grained quantization matrix multiplication device and method based on a systolic array, comprising: a first systolic array for receiving element streams of a first quantization factor matrix and a second quantization factor matrix, and performing quantization factor fusion calculation based on the element streams of the quantization factors to generate a quantization factor product; a second systolic array for receiving element streams of a first data matrix and a second data matrix, and performing multiplication accumulation calculation based on the element streams of the matrices; each first processing unit in the first systolic array is configured to send its calculated quantization factor product to a corresponding anchor processing unit in the second systolic array according to a mapping ratio; the anchor processing unit is configured to receive the quantization factor product and, as a propagation source, propagate the quantization factor product along the row and column directions within the quantization block size range corresponding to the quantization factor product in the second systolic array, thereby improving computational efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of computer architecture technology, and more specifically, to a fine-grained quantitative matrix multiplication device and method based on a pulsating array. Background Technology

[0002] As a classic parallel computing architecture, systolic arrays are widely used in computationally intensive tasks such as matrix multiplication, digital signal processing, and artificial intelligence inference due to their regular data flow, high parallelism, and high efficiency. The basic principle of systolic arrays is to connect processing units in a regular grid, allowing data to systolic between rows and columns, and performing multiplication and accumulation operations in the intersecting processing units, thereby achieving efficient computation of matrix multiplication.

[0003] In related technologies, when performing fine-grained quantization matrix multiplication, systolic arrays suffer from problems such as disconnected hardware pipelines for inverse quantization and matrix multiplication calculations, and discontinuous task scheduling. This results in difficulties in efficiently inserting quantization factors, reduced data reuse, and high task switching overhead, severely limiting their computational efficiency and energy efficiency when supporting flexible quantization models. Summary of the Invention

[0004] The purpose of this disclosure is to provide a fine-grained quantization matrix multiplication device and method based on a pulsating array to solve problems in related technologies.

[0005] To achieve the above objectives, this disclosure provides a fine-grained quantization matrix multiplication device based on a pulsating array, the device comprising:

[0006] The first pulse array, consisting of multiple first processing units connected in a two-dimensional mesh structure, is used to receive the element streams of the first quantization factor matrix and the second quantization factor matrix, and to perform quantization factor fusion calculation based on the element streams of the quantization factors to generate a quantization factor product.

[0007] The second pulsating array, consisting of multiple second processing units connected in a two-dimensional mesh structure, is used to receive the element streams of the first data matrix and the second data matrix, and to perform multiplication and accumulation calculations based on the element streams of the matrices.

[0008] The hardware size of the first pulsating array and the hardware size of the second pulsating array are determined according to a predefined mapping ratio.

[0009] Each first processing unit in the first pulsation array is configured to multiply its calculated quantization factor product according to the mapping ratio and send it to a corresponding anchor processing unit in the second pulsation array.

[0010] The anchor processing unit is configured to receive the quantization factor product and, as a propagation source, propagate the quantization factor product along the row and column directions within the second pulsation array within the quantization block size range corresponding to the quantization factor product.

[0011] Optionally, the first processing unit includes:

[0012] The fusion calculation unit is used to calculate the product of the first quantization factor and the second quantization factor when a first quantization factor is received from the row direction and a second quantization factor is received from the column direction, so as to obtain the quantization factor product.

[0013] The data routing unit is used to pass the quantization factor received from the row direction downstream in the row direction, pass the quantization factor received from the column direction downstream in the column direction, and output the product of the quantization factors generated by the fusion calculation unit to the corresponding anchor processing unit.

[0014] Optionally, the second processing unit includes:

[0015] The multiply-accumulate unit is used to perform multiplication operations on the elements of the first and second data matrices flowing through it, and accumulate the product results to a locally stored partial sum;

[0016] The dequantization fusion unit is used to, when receiving a valid quantization factor product from the first pulsating array, multiply the locally stored partial sum by the quantization factor product, accumulate the result into the final output result, and set the partial sum to zero.

[0017] Optionally, the mapping ratio is determined by the quantization step size basic unit, which includes a row direction basic unit and a column direction basic unit.

[0018] The row direction basic unit represents the smallest granularity of the quantization block supported by the fine-grained quantization matrix multiplication device in the row direction.

[0019] The column-direction basic unit represents the smallest granularity of the quantization blocks supported by the fine-grained quantization matrix multiplication device in the column direction.

[0020] Optionally, the total number of rows in the second pulsating array is the product of the total number of rows in the first pulsating array and the basic unit value in the row direction, and the total number of columns in the second pulsating array is the product of the total number of columns in the first pulsating array and the basic unit value in the column direction.

[0021] Optionally, each of the first processing units in the first pulsation array is configured to be responsible for calculating the product of quantization factors of a rectangular block consisting of row-direction basic units and column-direction basic units. The first processing unit is connected to a second processing unit in the second pulsation array located at the upper left corner of the rectangular block it is responsible for. This second processing unit is the corresponding anchor processing unit.

[0022] Optionally, the range of quantized block size is determined by the row-direction basic unit value and the column-direction basic unit value;

[0023] The anchor processing unit is configured to determine the propagation distance of the quantization factor product in the row direction based on the row direction basic unit value, and to determine the propagation distance of the quantization factor product in the column direction based on the column direction basic unit value.

[0024] The quantization factor product carries a row remaining step count and a column remaining step count during propagation. The initial value of the row remaining step count is the value of the row direction basic unit minus one, and the initial value of the column remaining step count is the value of the column direction basic unit minus one.

[0025] In the second pulsating array, each time the quantization factor product is passed along the row direction, the remaining step count of the row is decremented by one; each time it is passed along the column direction, the remaining step count of the column is decremented by one.

[0026] When the remaining step count of the row or the remaining step count of the column decreases to zero, the further propagation of the quantization factor product along the corresponding direction is stopped.

[0027] Optionally, the row input port and column input port of the first pulsating array receive quantization factors at a first fixed period;

[0028] Within the first pulsating array, the data propagation interval in the row direction is configured as a second fixed period, and the data propagation interval in the column direction is configured as a third fixed period.

[0029] The second fixed period is determined based on the value of the column direction basic unit, and the third fixed period is determined based on the value of the row direction basic unit.

[0030] Optionally, the number of rows in the first quantization factor matrix is ​​an approximation of the number of basic unit values ​​in the column direction; the number of columns in the second quantization factor matrix is ​​an approximation of the number of basic unit values ​​in the row direction.

[0031] This disclosure also provides a fine-grained quantized matrix multiplication method based on a systolic array, applied to the aforementioned fine-grained quantized matrix multiplication device based on a systolic array, the method comprising:

[0032] The first pulsating array receives the element streams of the first quantization factor matrix and the second quantization factor matrix, and performs quantization factor fusion calculation based on the element streams of the quantization factors to generate the quantization factor product.

[0033] The second pulsating array receives the element streams of the first and second data matrices and performs multiplication and addition calculations based on the element streams of the matrices.

[0034] Each first processing unit in the first systolic array sends the quantization factor product generated in the first systolic array to the corresponding anchor processing unit in the second systolic array according to a predefined hardware mapping ratio. The receiving unit then uses the quantization factor product as a propagation source point to propagate the quantization factor product along the row and column directions within the quantization block size range corresponding to the quantization factor product in the second systolic array. In the second systolic array, each second processing unit that receives the quantization factor product performs inverse quantization accumulation on the result of the multiplication calculation using the quantization factor product.

[0035] Through the above technical solution, the first pulsating array is dedicated to quantization factor fusion calculation, and can continuously receive and process quantization factor streams to generate quantization factor products. The second pulsating array focuses on performing quantized matrix multiplication and addition operations. The hardware scale correspondence between the two arrays is determined according to a predefined mapping ratio, ensuring that the calculation results of each first processing unit can be sent to the corresponding anchor processing unit in the second pulsating array. The anchor processing unit, as the propagation source point, synchronously spreads the quantization factor product along the row and column directions within its corresponding quantization block range, so that the inverse quantization factor can be integrated into the multiplication and addition calculation of the corresponding data block in a timely and accurate manner. This achieves deep coupling between quantization factor insertion and matrix multiplication calculation in the hardware pipeline, eliminates task scheduling interruptions, improves data reuse efficiency, and reduces task switching overhead, thereby significantly enhancing the computational efficiency and energy efficiency ratio of the pulsating array when supporting flexible quantization models.

[0036] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0037] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0038] Figure 1 This is an example diagram of the input and output of a fine-grained quantitative matrix multiplication.

[0039] Figure 2 This is a schematic diagram of the workflow of a pulse array structure with a fixed output.

[0040] Figure 3 This is a schematic diagram of a split matrix that considers fine-grained quantification.

[0041] Figure 4 This is a schematic diagram of a fine-grained quantization matrix multiplication device based on a pulsating array, according to an exemplary embodiment.

[0042] Figures 5-18 This is a schematic diagram illustrating the workflow of a pulsating array according to an exemplary embodiment.

[0043] Figure 19 This is a flowchart illustrating a fine-grained quantitative matrix multiplication method based on a pulsating array, according to an exemplary embodiment. Detailed Implementation

[0044] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0045] In the following description, the words "first" and "second" are used only to distinguish the purpose of the description and should not be interpreted as indicating or implying relative importance or order.

[0046] First, the pulsating array, quantization factor, tensor core, related terms, matrix multiplication based on outer and inner products, and dimension of matrix multiplication in this disclosure will be explained.

[0047] Pulsating arrays: Pulsating arrays are a parallel computing architecture proposed in the 1980s. The core idea is to arrange a large number of simple processing elements (PEs) in space and achieve high parallelism and high throughput through regular, synchronous, and rhythmic data flow. Data moves rhythmically forward in the PE array like blood pulsating in the heart, and is processed step by step during its movement.

[0048] Quantization factor: During linear quantization, the original data needs to be scaled to the target data precision type according to a certain ratio. For example, if the original data range is [-2047, 2048], and it needs to be scaled to int8, i.e., [-255, 256], the quantization factor is 8. Before calculation, the original data needs to be divided by 8 for quantization. After calculation, when it is necessary to restore the true value, it needs to be multiplied by 8 again for dequantization.

[0049] Tensor Core: A matrix multiplication acceleration unit used in GPUs, it is a type of dedicated hardware.

[0050] Matrix multiplication based on inner and outer products: Mathematically, matrix multiplication has two implementation methods: inner product and outer product. The inner product is based on vector inner product operations, taking a row of matrix A and a column of matrix B, multiplying each element and summing them to obtain an element of the output matrix. The outer product is based on vector outer product operations, taking a column of matrix A and a row of matrix B, calculating the pairwise product of all elements to obtain a matrix of the same size as the output matrix, and summing all such matrices bit by bit to obtain the final result. Pulsating arrays belong to the inner product method.

[0051] M / N / K dimensions of matrix multiplication: Generally, the shapes of matrices A and B are [M, K] and [K, N], respectively, where K is the summation dimension.

[0052] As large models grow in size, using lower-bit quantization such as FP8 / FP6 / FP4 can significantly alleviate memory bottlenecks and improve computation speed. On the other hand, to reduce the accuracy loss caused by quantization, the use of finer-grained quantization techniques has become a trend. (See also...) Figure 1 Assuming the input matrices for matrix multiplication have shapes A: [M, K] and B: [K, N], and the quantization factor matrices have shapes S_A: [M_scale, K_scale] and S_B: [K_scale, N_scale], they should satisfy M_scale = M / M_step, N_scale = N / N_step, and K_scale = K / K_step (all integer division), where matrix A is quantized at a granularity of [M_step, K_step], and matrix B at a granularity of [K_step, N_step]. For example, the activation layer parameter matrix A is quantized at a granularity of [1, 128], and the weight parameter matrix is ​​quantized at a granularity of [128, 128]. The final calculation result should satisfy C = (A... S_A)*(B S_B), where the operator * represents standard matrix multiplication. The element-wise multiplication of a matrix block is defined as multiplying the elements of matrix A that fall into the [i, j]th block by the scalar S_A[i, j] for inverse quantization.

[0053] Some existing commercial computing architectures do not support matrix multiplication with scaling factors. Instead, the matrix must be computed in chunks on Tensor Cores, and the output is then fed into a CUDA core and multiplied by a quantization factor for dequantization. While some architectures support matrix multiplication with quantization factors, they have several limitations: First, S_A: [M, K_scale], S_B: [K_scale, N], and K_scale ∈ {1, 2, 4}, meaning arbitrary chunking is not allowed. Second, there are requirements regarding the shape of the input matrices A and B, especially the K dimension. For example, in some architectures, the maximum K value is 64 for dense matrices and 128 for sparse matrices, limiting its applicability in larger-scale or variable-dimensional matrix operations.

[0054] On the other hand, while some commercial computing architectures have made some progress in systolic arrays, systolic arrays are typical matrix multiplication structures based on inner products, which have advantages such as low communication overhead, high energy efficiency, and flexibility, and natively support infinitely large K dimensions (as long as the accumulation accuracy is sufficient). However, traditional systolic arrays do not consider quantization factors. Without the support for quantization factors, the original large matrix can only be split into multiple small matrix multiplications according to the block size and then synchronized, which significantly reduces the efficiency of computing resources.

[0055] Please see Figure 2 It employs a fixed-output pulsating array structure, where each row of matrix A and each column of matrix B are arranged as follows: Figure 2 The clock cycles shown enter the processing units of the systolic array sequentially. Each processing unit performs multiplication and addition operations to obtain a partial sum. After all the data from the input matrix has flowed out of the systolic array, the processing unit stores the final result. It can be deduced that for input matrices A: [M, K] and B: [K, N], the size of the systolic array [N_syslotic, N_syslotic] ​​must satisfy N_syslotic ≥ max(M, N). Therefore, it takes M + K + N - 2 cycles to complete the calculation and N_syslotic cycles to complete the data output. The biggest advantage of the systolic array is its high data transmission efficiency and simple processing unit structure. The disadvantage is its relatively long latency, but the larger the K dimension, the higher the utilization rate of the processing units.

[0056] Specifically, for input matrices using fine-grained quantization, they can only be divided into many small matrices according to the quantization granularity. After each block is calculated, it is multiplied by the corresponding quantization factor for dequantization, and then multiplied synchronously according to the principle of matrix multiplication to obtain the final output result. This obviously leads to a huge latency. Figure 3As shown, without considering the calculation of the quantization factor and the delay of data output, the pulsating array directly calculates the matrix multiplication of [4, 2] × [2, 4], which requires 4 + 4 + 2 - 2 = 8 cycles. However, splitting it into 4 [2, 2] × [2, 2] matrix multiplications requires 4 (2 + 2 + 2 - 2) = 16 cycles.

[0057] In other words, when performing fine-grained quantization matrix multiplication, systolic arrays suffer from hardware pipeline separation between inverse quantization and matrix multiplication calculations, as well as discontinuous task scheduling. This results in difficulties in efficiently inserting quantization factors, reduced data reuse, and high task switching overhead, severely limiting their computational efficiency and energy efficiency when supporting flexible quantization models.

[0058] To address the aforementioned issues, a first systolic array is dedicated to quantization factor fusion calculation, continuously receiving and processing quantization factor streams to generate quantization factor products. A second systolic array focuses on performing quantized matrix multiplication and addition operations. The hardware scale correspondence between the two arrays is determined based on a predefined mapping ratio, ensuring that the calculation results of each first processing unit are sent to the corresponding anchor processing unit in the second systolic array in a fixed-point and unique manner. The anchor processing unit, acting as the propagation source, synchronously spreads the quantization factor products along the row and column directions within its corresponding quantization block range. This allows the inverse quantization factors to be integrated into the multiplication and addition calculations of the corresponding data blocks in a timely and accurate manner. This achieves deep coupling between quantization factor insertion and matrix multiplication calculations in the hardware pipeline, eliminating task scheduling interruptions, improving data reuse efficiency, and reducing task switching overhead. Consequently, it significantly enhances the computational efficiency and energy efficiency ratio of the systolic array when supporting flexible quantization models.

[0059] Figure 4 This is a schematic diagram illustrating a fine-grained quantization matrix multiplication device based on a pulsating array, according to an exemplary embodiment. Please refer to [link to schematic diagram]. Figure 4 The fine-grained metric matrix multiplication device based on pulsation arrays may include a first pulsation array and a second pulsation array.

[0060] The first pulsating array, consisting of multiple first processing units connected in a two-dimensional mesh structure, is used to receive the element streams of the first quantization factor matrix and the second quantization factor matrix, and to perform quantization factor fusion calculation based on the element streams of the quantization factors to generate the quantization factor product.

[0061] The second pulsating array, consisting of multiple second processing units connected in a two-dimensional mesh structure, is used to receive the element streams of the first data matrix and the second data matrix, and to perform multiplication and accumulation calculations based on the element streams of the matrices.

[0062] The quantization factor can refer to a scaling factor used to dequantize quantized data.

[0063] The first pulsation array is a dedicated quantization factor array processing unit that receives the first quantization factor matrix and the second quantization factor matrix.

[0064] The second systolic array is the core computing unit, responsible for receiving the element streams of the original or quantized first data matrix (e.g., matrix A) and the second data matrix (e.g., matrix B), and efficiently performing multiplication-accumulation-addition operations. The output of the second systolic array is the unquantized portion and / or intermediate result.

[0065] In one possible implementation, the first processing unit may include a fusion computing unit and a data routing unit:

[0066] The fusion calculation unit is used to calculate the product of the first quantization factor and the second quantization factor when the first quantization factor is received from the row direction and the second quantization factor is received from the column direction at the same time, so as to obtain the quantization factor product.

[0067] The data routing unit is used to pass the quantization factor received from the row direction downstream in the row direction, pass the quantization factor received from the column direction downstream in the column direction, and output the product of the quantization factors generated by the fusion calculation unit to the corresponding anchor processing unit.

[0068] When the fusion computing unit receives the first quantization factor from the row direction and the second quantization factor from the column direction at the same time, it calculates the product of the first quantization factor and the second quantization factor, that is, generates the quantization factor product s=s_a×s_b, where s_a comes from S_A (S_A is the quantization factor of the corresponding matrix A) and s_b comes from the element stream of S_B (S_B is the quantization factor of the corresponding matrix B).

[0069] In one possible implementation, the second processing unit may include a multiply-accumulate unit and an inverse quantization fusion unit:

[0070] The multiply-accumulate unit is used to perform multiplication operations on the elements of the first and second data matrices flowing through it, and accumulate the product results to a locally stored partial sum;

[0071] The dequantization fusion unit is used to multiply the locally stored partial sum with the product of the quantization factor when it receives the effective quantization factor product from the first pulsation array, add the result to the final output result, and set the partial sum to zero.

[0072] The multiply-accumulate-add operation can refer to the basic operation in standard matrix multiplication performed by the second processing unit, namely c = c + a × b, where a and b come from input matrices A and B respectively, and c is the accumulated partial sum.

[0073] The dequantization operation can refer to the quantization factor-related operation performed by the second processing unit. For example, when the second processing unit simultaneously receives the effective quantization factor product s from the first pulsation array, it performs the operation sum = sum + c × s, where c is the previously accumulated multiplication and summation result, s is the quantization factor product, and sum is the final output result. After execution, c is reset to 0.

[0074] Through a micro-architectural loop of accumulation-dequantization fusion-zeroing, dynamic, fine-grained dequantization operations are seamlessly embedded into a static, regular pulsating computation flow, enabling each second processing unit to independently and in parallel complete all computations from raw data to the final quantization result.

[0075] The hardware size of the first pulsating array and the hardware size of the second pulsating array are determined according to a predefined mapping ratio.

[0076] The hardware size between the first and second systolic arrays is not determined independently, but is designed collaboratively based on a predefined mapping ratio. For example, if the mapping ratio is [U, V], it means that each U row and V column of the processing unit area in the second systolic array corresponds to a first processing unit in the first systolic array.

[0077] By setting a preset mapping ratio, it can be ensured that the processing granularity of the first systolic array and the computational block of the second systolic array can be precisely matched in space.

[0078] Each first processing unit in the first pulsation array is configured to send the product of its calculated quantization factors to a corresponding anchor processing unit in the second pulsation array according to the mapping ratio. The anchor processing unit is configured to receive the product of quantization factors and, as a propagation source, propagate the product of quantization factors along the row and column directions within the second pulsation array within the quantization block size range corresponding to the product of quantization factors.

[0079] Each first processing unit in the first pulsation array, after calculating the quantization factor product s, sends it to a corresponding second processing unit in the second pulsation array, namely the anchor processing unit, according to the above mapping ratio.

[0080] For example, if the mapping ratio is [2, 4], the first processing unit with coordinates [i, j] in the first systolic array may be connected to the second processing unit with coordinates [2*i-1, 4*j-1] in the second systolic array as an anchor point. The connection relationship is fixed and one-to-one during hardware design.

[0081] After receiving the quantization factor product s, the anchor processing unit uses it as the propagation source and simultaneously sends the quantization factor product s to its adjacent second processing units to its right (row direction) and below (column direction). This quantization factor product s needs to be used to dequantize all partial sums within a specific region in the second pulsation array. The size of the region is the quantization block size [U, V]. The second processing unit that receives the quantization factor product s also belongs to the scope of this quantization block. It uses the quantization factor product s to dequantize its locally accumulated partial sum and continues to pass the quantization factor product s to its downstream row and column neighbors.

[0082] By attaching row and column remaining step size attributes to the quantization factor product s and decrementing them with each pass until the propagation stops when it reaches zero, the propagation process is strictly limited by the range of quantization block size, ensuring that the quantization factor product s propagates within the set range and does not spread indefinitely to the entire array.

[0083] The first systolic array is dedicated to quantization factor fusion calculation, continuously receiving and processing quantization factor streams to generate quantization factor products. The second systolic array focuses on performing quantized matrix multiplication and addition operations. The hardware scale correspondence between the two arrays is determined according to a predefined mapping ratio, ensuring that the calculation result of each first processing unit can be sent to the corresponding anchor processing unit in the second systolic array in a fixed point and uniquely. The anchor processing unit, as the propagation source, synchronously spreads the quantization factor product along the row and column directions within its corresponding quantization block range, so that the inverse quantization factor can be integrated into the multiplication and addition calculation of the corresponding data block in a timely and accurate manner. This achieves deep coupling between quantization factor insertion and matrix multiplication calculation in the hardware pipeline, eliminates task scheduling interruptions, improves data reuse efficiency, and reduces task switching overhead, thereby significantly enhancing the computational efficiency and energy efficiency ratio of the systolic array when supporting flexible quantization models.

[0084] In one possible implementation, the mapping ratio is determined by the quantization step size basic unit, which includes a row direction basic unit and a column direction basic unit.

[0085] The row-direction basic unit represents the smallest granularity of the quantization block supported by the fine-grained quantization matrix multiplication device in the row direction.

[0086] The column-direction basic unit represents the smallest granularity of the quantization block supported by the fine-grained quantization matrix multiplication device in the column direction.

[0087] The basic unit of quantization step size represents the size of the most basic quantization process that the hardware can support.

[0088] Minimum granularity can refer to the smallest unit size that hardware can recognize and process. In one embodiment, minimum granularity can specifically refer to the size of the smallest data region that hardware can apply the same quantization factor to for dequantization.

[0089] The row direction basic unit can be the component of the quantization step size basic unit in the row direction, and is a positive integer, representing the finest granularity that the hardware device can divide in the row direction when processing the quantization matrix. Any quantization block processed by this hardware has a row number that is an integer multiple of the row direction basic unit.

[0090] The row direction basic unit defines the quantization resolution of the hardware in the vertical direction. For example, if U=2, it means that the hardware requires that the number of rows in any quantization block be at least 2 and must be a multiple of 2 (e.g., 2, 4, 6, 8...), and cannot independently configure a different quantization factor for a single row (i.e., a row with 1 row). The row direction basic unit can represent the minimum span by which the hardware allocates and transmits quantization factors in the row direction.

[0091] The column-direction basic unit can be the component of the quantization step size basic unit in the column direction, and is a positive integer, representing the finest granularity that the hardware device can divide in the column direction when processing the quantization matrix. Any quantization block processed by this hardware has a column number that is an integer multiple of the column-direction basic unit.

[0092] The column-direction basic unit defines the quantization resolution of the hardware in the horizontal direction. For example, if V=8, then the number of columns representing any quantization block of the hardware must be a multiple of 8. The column-direction basic unit can represent the minimum span by which the hardware allocates and propagates quantization factors in the column direction.

[0093] Therefore, UxV constitutes the smallest quantization unit that the hardware can process, and all coarser-grained quantization blocks, such as 2Ux3V, can be regarded as integer multiples of the smallest quantization unit.

[0094] The basic unit of quantization step size [U, V] can be understood as logically dividing the entire computation plane of the second systolic array into several non-overlapping rectangular blocks of size U rows and V columns. Here, the basic unit in the column direction is represented by U, and the basic unit of quantization step size is represented by V. The basic unit of quantization step size determines the mapping ratio. That is, a mapping ratio of [U, V] represents a first processing unit in the first systolic array whose task processing range corresponds exactly to a processing unit region of size U x V in the second systolic array. Therefore, the ratio of the size of the first systolic array (i.e., the number of first processing units) to the size of the second systolic array (i.e., the number of second processing units) is determined by U and V.

[0095] By introducing a basic unit of quantization step size, the abstract mapping ratio is made concrete and quantifiable, and the inherent capabilities and limitations of the hardware are clearly defined, providing a clear boundary for software and hardware co-design.

[0096] In one possible implementation, the total number of rows in the second pulsating array is the product of the total number of rows in the first pulsating array and the basic unit value in the row direction, and the total number of columns in the second pulsating array is the product of the total number of columns in the first pulsating array and the basic unit value in the column direction.

[0097] In one possible implementation, each first processing unit in the first pulsation array is configured to be responsible for calculating the quantization factor product of a rectangular block of size consisting of row-direction basic units and column-direction basic units. The first processing unit is connected to a second processing unit in the second pulsation array located at the upper left corner of the rectangular block it is responsible for, which is the corresponding anchor processing unit.

[0098] The task of calculating a rectangular block of size defined by row and column basic units can be understood as follows: a specific first processing unit in the first systolic array is dedicated to processing the quantization factor associated with a specific rectangular region in the second systolic array. The size of this rectangular region is determined by the quantization step size basic unit, i.e., the height is the row basic unit U and the width is the column basic unit V. The task of this specific first processing unit is to calculate the quantization factor product s that will ultimately be used to dequantize all the calculation results within this rectangular region.

[0099] The second processing unit located at the top left corner of the rectangular block, within the two-dimensional grid of the second systolic array, has a unique, fixed starting point in each rectangular region consisting of U x V second processing units; that is, the topmost and leftmost processing unit in that region. The connection can refer to a physical, direct, point-to-point hardware link between the first processing unit of the first systolic array and the second processing unit at the top left corner of the second systolic array.

[0100] The anchor processing unit is the second processing unit located in the upper left corner of the rectangular block. It serves as the sole entry point for the quantization factor to be injected from the first pulsation array into the second pulsation array and the starting point for subsequent propagation.

[0101] In one embodiment, an anchored interconnection network can be designed. This network differs from the regular grid connection between processing units within the pulsating array. Instead, according to a mapping formula, a dedicated connection is laid for each first processing unit in the first pulsating array, leading directly to its corresponding anchor processing unit in the second pulsating array.

[0102] In one embodiment, the row direction basic unit in the quantization step size basic unit is U=2, the column direction basic unit is 4, the second pulsation array size is 8 rows x 16 columns, and the first pulsation array size, according to the ratio, should be (8 / 2=4) rows x (16 / 4=4) columns.

[0103] The second systolic array is divided into 4x4=16 quantization blocks, each block being 2 rows x 4 columns in size. The first systolic array has 4x4=16 first processing units, each responsible for one of the quantization blocks. For example, the first processing unit located at [2, 3] (row 2, column 3) in the first systolic array is responsible for the 2x4 quantization block in the second systolic array with row range [3, 4] (i.e., (2-1)*2+1=3 to 2*2=4) and column range [9, 12] (i.e., (3-1)*4+1=9 to 3*4=12). The first processing unit [2, 3] in the first systolic array responsible for this quantization block has a corresponding anchor processing unit located at the upper left corner of this quantization block, i.e., the second processing unit at coordinate [3, 9] in the second systolic array. A physical wire can be set from the first processing unit [2, 3] directly to the second processing unit [3, 9].

[0104] When the first processing unit [2, 3] of the first pulsating array completes the fusion calculation of s_a and s_b and obtains the quantization factor product s, it sends the quantization factor product s directly to the anchor processing unit [3, 9] of the second pulsating array through the above-mentioned dedicated connection. The anchor processing unit [3, 9] takes the quantization factor product s as the source and starts the propagation process of the quantization factor product s in the quantization block at the instant it receives it.

[0105] By using a regional responsibility system and anchoring in the upper left corner, the abstract mapping ratio is transformed into a concrete and achievable hardware connection, ensuring that the quantization factor can be delivered to the precise calculation area with minimal overhead and maximum certainty, thereby improving operational efficiency.

[0106] In one possible implementation, the range of quantized block size is determined by the row-direction basic unit value and the column-direction basic unit value.

[0107] The anchor processing unit is configured to determine the propagation distance of the quantization factor product in the row direction based on the row direction basic unit value, and to determine the propagation distance of the quantization factor product in the column direction based on the column direction basic unit value.

[0108] This ensures that within a quantization block defined by the row-direction basic unit values ​​and the column-direction basic unit values, all second processing units perform dequantization operations using the same quantization factor product.

[0109] The propagation of the quantization factor product is controlled by a decrementing counting mechanism. During propagation, the quantization factor product carries the row remaining step count U_left and the column remaining step count V_left. The quantization factor product needs to be propagated downwards from the anchor processing unit until it covers a total of U rows. Since the anchor processing unit itself occupies one row, it needs to be passed to the second processing units of U-1 downstream columns. That is, the initial value of the row remaining step count is the row-direction basic unit value minus one. The quantization factor product also needs to be propagated to the right from the anchor processing unit until it covers a total of V columns. Since the anchor processing unit itself occupies one column, it needs to be passed to the second processing units of V-1 downstream rows. That is, the initial value of the column remaining step count is the column-direction basic unit value minus one.

[0110] In the second pulsating array, each time the quantization factor product is passed along the row direction, the remaining row step count is decremented by one; each time it is passed along the column direction, the remaining column step count is decremented by one.

[0111] When the row remaining step count or column remaining step count decreases to zero, the further propagation of the quantization factor product along the corresponding direction stops.

[0112] The decrementing counting mechanism can be understood as adding an initial count value to the product of quantization factors. Each time data is transmitted, the count value is decremented by one. When the count value reaches zero, transmission stops.

[0113] When the anchor processing unit receives the quantization factor product s from the first pulsation array, it binds the row remaining step count U_left and the column remaining step count V_left. The initial values ​​of the row remaining step count U_left and the column remaining step count V_left are U-1 and V-1, respectively.

[0114] Upon receiving a data packet containing a quantization factor product s and a remaining step count, any second processing unit in the second pulsating array can use the quantization factor product s to dequantize the locally accumulated partial sum, and then make a decision on whether to pass it based on the remaining step count.

[0115] Specifically, the process involves the following steps: If the row remaining step count is greater than 1, it indicates that the quantization factor product 's' needs to continue propagating to the right. The second processing unit generates a new data packet where the quantization factor product 's' value remains unchanged, the row remaining step count is decremented by one, and the column remaining step count remains unchanged (because propagation along the row does not affect the column direction requirement). This data packet is then sent to its right neighbor. If the column remaining step count is greater than 1, it indicates that the quantization factor product 's' needs to continue propagating downwards. The second processing unit generates another new data packet where the quantization factor product 's' value remains unchanged, the column remaining step count is decremented by one, and the row remaining step count remains unchanged. This data packet is then sent to its downward neighbor. If the count in a certain direction decreases to 1, the current second processing unit is the last second processing unit in that direction to require the quantization factor product 's', and propagation in that direction ceases.

[0116] By using a decrementing counting mechanism, the initial count is equal to the step size to be covered, and it is decremented by one with each transmission. Therefore, the quantization factor product s is exactly transmitted in rows U and columns V, ensuring the absolute accuracy of the propagation range.

[0117] In one possible implementation, the row input port and column input port of the first pulsating array receive quantization factors at a first fixed period.

[0118] Within the first pulsating array, the data propagation interval in the row direction is configured as a second fixed period, and the data propagation interval in the column direction is configured as a third fixed period.

[0119] The second fixed period is determined based on the value of the basic unit in the column direction, and the third fixed period is determined based on the value of the basic unit in the row direction.

[0120] The first fixed period can refer to the number of clock cycles required for the row and column input ports of the first pulsating array to receive a new quantization factor data item from external memory or upstream modules.

[0121] The data propagation interval can refer to the number of clock cycles required for a first processing unit (PE) within the first systolic array to transmit the data (quantization factor s_a or s_b) received from upstream to its adjacent downstream first processing unit. The second fixed cycle can refer to the propagation interval in the row direction; the third fixed cycle can refer to the propagation interval in the column direction.

[0122] The second fixed period is determined based on the value of the column-direction basic unit. It can be that the second fixed period is proportional to the value of the column-direction basic unit. The third fixed period is determined based on the value of the row-direction basic unit. It can also be that the third fixed period is proportional to the value of the row-direction basic unit.

[0123] Since the dimensionality of the quantization factor matrix is ​​usually much smaller than the K-dimensionality of the main computation matrix and the calculation is simple, it does not need to be input in every cycle like the data matrix. The data input rhythm of the first pulsation array, i.e. the first fixed cycle, is independent of the second pulsation array.

[0124] The moment when the first processing unit located at [i, j] in the first pulsation array completes the calculation of the quantization factor product s is exactly the moment when the corresponding quantization block of UxV in the second pulsation array, with the anchor processing unit [U*(i-1)+1, V*(j-1)+1] as the upper left corner, completes the current calculation sub-stage (such as a K-dimensional accumulation), the calculated quantization factor product s can be injected instantly through the anchor processing unit and begin to propagate within the quantization block, achieving seamless dequantization.

[0125] In one embodiment, the basic unit of quantization step size is [U=2, V=4]. The first fixed period is set to 5 periods, and the rules for determining the internal propagation interval are: the second fixed period is 4 periods, and the third fixed period is 2 periods.

[0126] At the initial moment Cycle0: the first quantization factor of the first row of S_A enters the leftmost first processing unit of the first row of the first pulsation array; the first quantization factor of the first column of S_B enters the topmost first processing unit of the first column.

[0127] Row direction flow: The data in this row needs to move one jump to the right to the next first processing unit, which requires waiting for 4 cycles. Therefore, it will enter the second column first processing unit in Cycle4.

[0128] Column-wise flow: For the data in this column to move down one jump to the next first processing unit, it needs to wait for 2 cycles, so in Cycle2 it enters the first processing unit of the second row.

[0129] Intersection calculation: The paths of the first quantization factor in the first row of s_a and the first quantization factor in the first column of s_b, due to the fast movement of column data (2 cycles / hop) and the slow movement of row data (4 cycles / hop), will arrive at the first processing unit [1,1] at different times. However, by carefully designing the input start time, it can be ensured that they arrive at the first processing unit [1,1] and trigger the calculation simultaneously in CycleT. In CycleT, the first processing unit [1,1] of the first systolic array calculates the quantization factor product s. According to the quantization block transmission, the quantization block of the second systolic array corresponding to the quantization factor product s is [1:2, 1:4], that is, rows 1-2 and columns 1-4. Meanwhile, as the data from the second pulsating array flows in and is calculated at a rate of one column per cycle, at time CycleT, the quantization blocks [1:2, 1:4] in the second pulsating array have just completed the multiplication and accumulation of the current input elements, generating partial sums. The quantization factor product s is sent to the anchor processing unit [1,1] through a dedicated connection, and immediately begins to propagate. It also dequantizes all the newly generated partial sums in the quantization block, and the calculation and quantization are time-space lockstepped.

[0130] Designed to work precisely in sync with the second pulsating array on a time scale through a fixed-period timing mechanism that is independent and associated with the basic unit, the entire device operates efficiently and without errors.

[0131] In one possible implementation, the number of rows in the first quantization factor matrix is ​​an approximation of the number of basic unit values ​​in the column direction; the number of columns in the second quantization factor matrix is ​​a constraint of the number of basic unit values ​​in the row direction.

[0132] For example, if the row-direction basic unit value is 3 and the row-direction basic unit value is 2, then the number of rows in the first quantization factor matrix can be 2 or 1, and the number of columns in the second quantization factor matrix can be 3 or 1.

[0133] By requiring the dimensions of the quantization factor matrix to be aligned with the basic hardware units by a divisor, an efficient hardware-software contract is established. This data alignment process ensures that the hardware can execute in a native, zero-overhead manner, avoiding the performance loss associated with dynamic format conversion. Simultaneously, it simplifies hardware control logic and data paths, eliminates the overhead of handling irregular boundaries, and keeps computing resources running at full capacity.

[0134] In one embodiment, the second systolic array can be used to process the inputs of matrices A and B, and the first systolic array is used to process the inputs of quantization factor matrices S_A and S_B.

[0135] The second systolic array has a shape of [N_syslotic, N_syslotic]. It can handle maximum matrix sizes A: [M, K] and B[K, N], satisfying M ≤ N_systolic and N ≤ N_systolic. Considering hardware complexity and resource overhead, the quantization factor matrix can be required to have a relatively fixed shape. The quantization step size unit of this hardware can be defined. For any matrix multiplication, the hardware can process it as long as the quantization step size is an integer multiple of the identity element and divisible by the row / column dimension. Therefore, the shape of the first systolic array is... This allows for a relaxation of the requirements on the shape of the quantization factor matrix. For example, if M=64 and N=256, Allow S_A: [x, K_scale], S_B: [K_scale, y], where x takes values ​​of 1, 2, ..., 64, and y takes values ​​of 1, 2, 4. Figure 4 , S_A: [x, K_scale], S_B: [K_scale, y], where x takes values ​​of 1 and 2, and y takes values ​​of 1, 2, and 3. Therefore, a finer-grained quantization step size unit element can support more quantization matrix shapes, but this means a greater number of processing units in the first systolic array. Once the hardware is determined, the unit element cannot be changed; in practice, it is set according to the user's needs.

[0136] Because a pulsating array is used, the dimensions of K and K_scale can be infinitely large in principle, as long as the data loading time can overlap with the computation time and the accuracy requirements are met.

[0137] The second pulsating array can be a two-dimensional dual-flow pulsating array. A special second processing unit connected to the first processing unit of the first pulsating array is defined as an anchor processing unit, for coordinates [...]. The anchor processing unit of the first pulsating array receives the quantization factor product s of the first processing unit [i, j] of the first pulsating array, where , .

[0138] Once the first systolic array obtains a valid quantization factor product s, it broadcasts the quantization factor product s to the other second processing units of the involved second systolic array through the anchor processing unit. Specifically, when the first systolic array sends the quantization factor product s to the second systolic array, it includes row remaining step count U_left and column remaining step count V_left attributes, with initial values ​​of U-1 and V-1 respectively. Before the second systolic array sends the s to the second processing unit of each row or column, the corresponding remaining step count is decremented by one. When it reaches 0, it means that the edge has been reached and no further propagation is needed.

[0139] When a valid quantization factor product *s* is received from the first systolic array or its upstream and downstream components, the second processing unit of the second systolic array performs the following operations in the current cycle: `sum += c × s` and `c = 0`, where `c` is the accumulated sum corresponding to the current quantized block. Each time a new element of matrix A or B is received, `c += a × b` is updated, and `sum` is the total accumulated sum. Note that if valid quantization factor products *s* are received in both rows and columns, the algorithm guarantees that they are equal values ​​and treats them as a single quantization factor product *s* during calculation. In the next cycle, the data `c = c + a` from the upstream second processing unit or the input queue is processed. b. The updates of `sum` and `c` can utilize the same hardware, or the update of `sum` can be specifically optimized. For example, if `s` uses the UE8M0 format, representing only powers of 2, the multiplication operation can be optimized into addition of the floating-point exponent. The second systolic array can distinguish between two data types: the input data of matrices A and B, and the product of the quantization factors transmitted from the first systolic array, requiring only a 1-bit `type` field.

[0140] The first pulsation array can be a two-dimensional dual-flow pulsation array, with each row of the S_A matrix and each column of the S_B matrix as input. The first processing unit in the first pulsation array executes s=s_a s_b, s_a, and s_b are quantization factors transmitted from upstream. The flow rates of the element streams in the first and second systolic arrays are also different. Every K_step+1 cycles, the row and column input queues input one quantization factor to the first systolic array. The starting cycle of the i-th row input queue is... The starting cycle of the input queue for the j-th column is In the first pulse array, the first processing unit is spaced apart at intervals of... Each cycle of data flow occurs once, with each processing unit in the first column processing every [time period]. One cycle per flow. Anchor handling unit per... Each cyce receives a quantization factor product, in order to flexibly support [ ]and[ The quantization granularity allows the i-th row of S_A to enter the input queue of the [(i-1)x+1 : ix]-th row of the first systolic array, and the j-th row of S_B to enter the input queue of the [(j-1)y+1 : jy ]-th column. If the first processing unit finds that only row or column input data is valid, it will only pass through the row or column data, that is, s_a will be passed to the first processing unit downstream of the row or s_b will be passed to the first processing unit downstream of the column.

[0141] If the first processing unit finds that the input data of the rows and columns are valid, in addition to performing multiplication operations and data flow of the first systolic array, it also needs to pass the locally calculated quantization factor product s to the connected anchor processing unit in the second systolic array.

[0142] In one embodiment, using A: [4, 6], B: [6, 4], S_A: [2, 3], S_B: [3, 2], the second pulsation array is [4, 4], and the first pulsation array is [2, 2]. , .

[0143] Cycle1: such as Figure 5 As shown, the second pulsation array begins in the first row and second column of the first cycle matrix A and B, as follows: Figure 6 As shown, the quantization factor has not yet been entered;

[0144] Cycle2: Same as cycle1, starting from the second row and second column;

[0145] Cycle3: such as Figure 7 As shown, S_A and S_B start entering the first pulsation array from the first row and first column, satisfying... , ,like Figure 8 As shown, the quantization factor is input and fusion calculation is performed;

[0146] Cycle4: such as Figure 9 As shown, the anchor processing unit PE11 receives the quantization factor product s111=s_a11×s_b11 from the first pulsation array and performs an inverse quantization operation, as follows. Figure 10 The diagram shows the process of outputting the product of the quantization factors s111.

[0147] Cycle5: such as Figure 11 As shown, the anchor processing unit PE11 broadcasts the quantization factor product s111 to its neighbors and continues updating. The second row and second column of S_A and S_B begin entering the first pulsation array, meeting the requirement that all rows and columns of the first pulsation array flow data once, differing from cycle 3 by 2 cycles, which is equal to... To meet the requirements, such as Figure 12 As shown, the quantization factor is further calculated to obtain the quantization factor product;

[0148] Cycle6: such as Figure 13As shown, anchor processing units PE13 and PE31 receive the data s112=s_a11×s_b12 and s211=s_a21×s_b11 from the first pulse array and update them. Simultaneously, the second processing unit PE22 receives the same s111 from two upstream neighbors and also updates it. It should be understood that the second processing unit PE21 does not transmit s111 to the second processing unit PE31 because the remaining column step size is already 0. Similarly, the second processing unit PE12 does not transmit s111 to the second processing unit PE13. Figure 14 The diagram illustrates the process of outputting the product of quantization factors s211 and s112.

[0149] Cycle7: such as Figure 15 As shown, the second element of the first row and first column of S_A and S_B begins to enter the second pulsation array, satisfying the requirement of cycle3 difference K_step+1. Simultaneously, the data flow in the second pulsation array completes one cycle, differing from cycle5 by 2 cycles. The second processing unit PE22 in the second pulsation array will no longer transmit s111 to its downstream neighbor because the row and column second processing units are already 0. Figure 16 As shown, the quantization factor is further calculated to obtain the quantization factor product;

[0150] Cycle8: such as Figure 17 As shown, anchor processing unit PE11 receives a new quantization factor product s121 again, with an interval of K_step+1 cycles from the previous one, and performs the second stage of dequantization operation. Anchor processing unit PE33 receives a quantization factor product s212 from the first pulsation array, as shown... Figure 18 The diagram illustrates the process of outputting the product of quantization factors s121 and s212.

[0151] The subsequent cycle operations are similar and will not be described in detail.

[0152] Based on the same inventive concept, this disclosure also provides a fine-grained metric matrix multiplication method based on systolic arrays. Embodiments of this disclosure further provide a fine-grained metric matrix multiplication method based on systolic arrays, which can be applied to fine-grained metric matrix multiplication devices based on systolic arrays. Please refer to [link to relevant documentation]. Figure 19 The method may include steps S1 to S3.

[0153] Step S1: The first pulsation array receives the element streams of the first quantization factor matrix and the second quantization factor matrix, and performs quantization factor fusion calculation based on the element streams of the quantization factors to generate the quantization factor product.

[0154] Step S2: The second pulsating array receives the element streams of the first data matrix and the second data matrix, and performs multiplication and accumulation calculations based on the element streams of the matrices.

[0155] Step S3: Each first processing unit in the first systolic array sends the quantization factor product generated in the first systolic array to the corresponding anchor processing unit in the second systolic array according to a predefined hardware mapping ratio. The receiving quantization factor product is used as the propagation source point and propagated along the row and column directions in the second systolic array within the quantization block size range corresponding to the quantization factor product. In the second systolic array, each second processing unit that receives the quantization factor product performs inverse quantization accumulation on the result of the product accumulation calculation.

[0156] Optionally, the method further includes: using a decrementing counting mechanism to control the propagation process of the quantization factor product.

[0157] In the second pulsating array, each time the quantization factor product is passed along the row direction, the remaining row step count is decremented by one; each time it is passed along the column direction, the remaining column step count is decremented by one.

[0158] When the row remaining step count or column remaining step count decreases to zero, the further propagation of the quantization factor product along the corresponding direction stops.

[0159] Optionally, step S1 may include:

[0160] When the first quantization factor is received from the row direction and the second quantization factor is received from the column direction at the same time, the product of the first quantization factor and the second quantization factor is calculated to obtain the quantization factor product.

[0161] The quantization factor received from the row direction is passed downstream in the row direction, and the quantization factor received from the column direction is passed downstream in the column direction. The product of the quantization factors is then output to the corresponding anchor processing unit.

[0162] Optionally, step S2 may include:

[0163] Perform multiplication on the elements of the first and second data matrices, and sum the product to a locally stored partial sum;

[0164] When a valid quantization factor product is received from the first pulsating array, the locally stored partial sum is multiplied by the quantization factor product, the result is added to the final output, and the partial sum is set to zero.

[0165] Regarding the fine-grained quantization matrix multiplication device based on pulsating array in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0166] The fine-grained quantitative matrix multiplication device and method based on pulsating arrays disclosed herein have at least the following advantages:

[0167] 1) Achieve fine-grained quantization while maintaining the efficiency and flexibility of the pulsating array;

[0168] 2) Relaxing the requirements for the shape of the quantization factor matrix can better support quantization methods with weight matrices, such as [128, 128] as blocks, avoiding repeated calculations and additional data handling.

[0169] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0170] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0171] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A fine-grained quantization matrix multiplication device based on a pulsating array, characterized in that, The device includes: The first pulse array, consisting of multiple first processing units connected in a two-dimensional mesh structure, is used to receive the element streams of the first quantization factor matrix and the second quantization factor matrix, and to perform quantization factor fusion calculation based on the element streams of the quantization factors to generate a quantization factor product. The second pulsating array, consisting of multiple second processing units connected in a two-dimensional mesh structure, is used to receive the element streams of the first data matrix and the second data matrix, and to perform multiplication and accumulation calculations based on the element streams of the matrices. The hardware size of the first pulsating array and the hardware size of the second pulsating array are determined according to a predefined mapping ratio. Each first processing unit in the first pulsation array is configured to multiply its calculated quantization factor product according to the mapping ratio and send it to a corresponding anchor processing unit in the second pulsation array. The anchor processing unit is configured to receive the quantization factor product and, as a propagation source, propagate the quantization factor product along the row and column directions within the second pulsation array within the quantization block size range corresponding to the quantization factor product.

2. The apparatus according to claim 1, characterized in that, The first processing unit includes: The fusion calculation unit is used to calculate the product of the first quantization factor and the second quantization factor when a first quantization factor is received from the row direction and a second quantization factor is received from the column direction, so as to obtain the quantization factor product. The data routing unit is used to pass the quantization factor received from the row direction downstream in the row direction, pass the quantization factor received from the column direction downstream in the column direction, and output the product of the quantization factors generated by the fusion calculation unit to the corresponding anchor processing unit.

3. The apparatus according to claim 1, characterized in that, The second processing unit includes: The multiply-accumulate unit is used to perform multiplication operations on the elements of the first and second data matrices flowing through it, and accumulate the product results to a locally stored partial sum; The dequantization fusion unit is used to, when receiving a valid quantization factor product from the first pulsating array, multiply the locally stored partial sum by the quantization factor product, accumulate the result into the final output result, and set the partial sum to zero.

4. The apparatus according to claim 1, characterized in that, The mapping ratio is determined by the quantization step size basic unit, which includes a row direction basic unit and a column direction basic unit. The row direction basic unit represents the smallest granularity of the quantization block supported by the fine-grained quantization matrix multiplication device in the row direction. The column-direction basic unit represents the smallest granularity of the quantization blocks supported by the fine-grained quantization matrix multiplication device in the column direction.

5. The apparatus according to claim 4, characterized in that, The total number of rows in the second pulsating array is the product of the total number of rows in the first pulsating array and the basic unit value in the row direction, and the total number of columns in the second pulsating array is the product of the total number of columns in the first pulsating array and the basic unit value in the column direction.

6. The apparatus according to claim 4, characterized in that, Each of the first processing units in the first pulsation array is configured to calculate the product of quantization factors for a rectangular block of size consisting of row-direction basic units and column-direction basic units. The first processing unit is connected to the second processing unit in the second pulsation array located at the upper left corner of the rectangular block it is responsible for. The second processing unit is the corresponding anchor processing unit.

7. The apparatus according to claim 4, characterized in that, The range of quantized block size is determined by the values ​​of the row-direction basic unit and the column-direction basic unit; The anchor processing unit is configured to determine the propagation distance of the quantization factor product in the row direction based on the row direction basic unit value, and to determine the propagation distance of the quantization factor product in the column direction based on the column direction basic unit value. The quantization factor product carries a row remaining step count and a column remaining step count during propagation. The initial value of the row remaining step count is the value of the row direction basic unit minus one, and the initial value of the column remaining step count is the value of the column direction basic unit minus one. In the second pulsating array, each time the quantization factor product is passed along the row direction, the remaining step count of the row is decremented by one; each time it is passed along the column direction, the remaining step count of the column is decremented by one. When the remaining step count of the row or the remaining step count of the column decreases to zero, the further propagation of the quantization factor product along the corresponding direction is stopped.

8. The apparatus according to claim 4, characterized in that, The row input port and column input port of the first pulsating array receive quantization factors according to a first fixed period; Within the first pulsating array, the data propagation interval in the row direction is configured as a second fixed period, and the data propagation interval in the column direction is configured as a third fixed period. The second fixed period is determined based on the value of the column direction basic unit, and the third fixed period is determined based on the value of the row direction basic unit.

9. The apparatus according to claim 4, characterized in that, The number of rows in the first quantization factor matrix is ​​an approximation of the number of basic unit values ​​in the column direction; the number of columns in the second quantization factor matrix is ​​an approximation of the number of basic unit values ​​in the row direction.

10. A fine-grained quantization matrix multiplication method based on a pulsating array, characterized in that, The method, applied to the fine-grained quantization matrix multiplication apparatus based on a pulsating array as described in any one of claims 1-9, comprises: The first pulsating array receives the element streams of the first quantization factor matrix and the second quantization factor matrix, and performs quantization factor fusion calculation based on the element streams of the quantization factors to generate the quantization factor product. The second pulsating array receives the element streams of the first and second data matrices and performs multiplication and addition calculations based on the element streams of the matrices. Each first processing unit in the first systolic array sends the quantization factor product generated in the first systolic array to the corresponding anchor processing unit in the second systolic array according to a predefined hardware mapping ratio. The receiving unit then uses the quantization factor product as a propagation source point to propagate the quantization factor product along the row and column directions within the quantization block size range corresponding to the quantization factor product in the second systolic array. In the second systolic array, each second processing unit that receives the quantization factor product performs inverse quantization accumulation on the result of the multiplication calculation using the quantization factor product.

Images