A large language model inference accelerator design method based on cross-layer power-law perception quantization

Abstract

The application relates to a large language model inference accelerator design method based on cross-layer power law perception quantization; the large language model inference accelerator comprises a group-level accumulation unit, a dot product operation unit and a dequantization unit. The group-level accumulation unit is used for grouping and accumulating input activation values according to quantization values of weight elements, so as to combine and calculate activation values sharing the same weight elements, generate groups corresponding to respective quantized weights and send the groups to the dot product operation unit; the dot product operation unit is used for multiplying the received group with a quantized weight element corresponding to the group, so as to complete dot product calculation in a manner of reducing redundant multiplication operations, and send a dot product calculation result of a first precision to the dequantization unit; and the dequantization unit is used for performing conversion calculation based on parameters corresponding to a quantization process, converting the dot product calculation result of the first precision into a dot product calculation result of a second precision, wherein the first precision is lower than the second precision.

Description

Technical Field

[0001] This invention relates to the field of accelerator technology, and in particular to a design method for a large language model inference accelerator based on cross-layer power-law perceptual quantization. Background Technology

[0002] Unlike traditional deep learning techniques, large language models based on Transformers follow an autoregressive inference process, generating new words iteratively. In a hierarchical model based on a typical Transformer architecture, each layer uses word embeddings and a weight matrix W. q W k W v The input word embeddings are multiplied and projected into query vectors, key vectors, and value vectors. An attention mechanism is then applied to these three tensors to model word interactions. The attention output is then processed by W... o The multiplication of matrices WF1 and WF2 generates the output of the current layer, and the output of the last layer will be used as the input for the next iteration.

[0003] While traditional large language models retain their full capabilities, quantized large language models are more suitable for practical applications requiring low memory usage, fast inference speed, and low power consumption, such as cloud services. This technique leverages the robustness of large language models in terms of output quality, enabling the approximation of high-precision parameters using low-bit representations, thus reducing memory and computational overhead. For example, quantizing model parameters from 16 bits to 8 bits can simultaneously halve memory capacity and bandwidth consumption with negligible precision loss. Furthermore, the low-bit representation method can be used to design simpler computational units, thereby improving computational efficiency.

[0004] A typical quantization formula is Where W represents the weight tensor, W q W represents the quantization tensor. dq This represents the dequantization tensor. The scaling factor s is determined by the range of W and the target quantization bit width b, and Z0 represents the zero point. When Z0=0, it corresponds to symmetric quantization; otherwise, Z0 acts as an offset term in asymmetric quantization. Furthermore, depending on how s and Z0 are shared among elements, different granularities of quantization can be achieved: matrix-level quantization, channel-level quantization, and group-level quantization.

[0005] In existing technologies, quantizing large language models effectively compresses model size and improves inference speed by converting model weights from high-precision floating-point numbers (such as FP16) to low-precision integers (such as INT8 / INT4). Its core technology lies in optimizing storage and computational efficiency by reducing numerical precision. Due to the precision loss during quantization, the model needs to rely on post-training quantization or quantization-aware training to maintain performance, which introduces additional calibration and fine-tuning overhead. Unlike conventional model inference, while low-precision quantization improves throughput, it increases the uncertainty of numerical representation. Although existing quantization techniques, as described in published literature, have smoothed the weight distribution range, they have not proposed an effective solution to the dynamic range sensitivity problem in large language models, resulting in a significant risk of performance degradation under low-bit quantization.

[0006] CN118569324A discloses a method for accelerating large language models, including: S1, compressing and decomposing the pre-trained weight matrix W of the large language model; S2, performing QR decomposition on the U and VT matrices respectively to obtain Q. U R U Q V R V S3. Perform compressed decomposition of the matrix product using the singular value decomposition algorithm; S4. Merge the matrices separately; S5. Replace the weight matrix W with the obtained matrix for storage; S6. Accelerate inference using the stored matrix; S7. Set the obtained rank r as the low-rank parameterized update matrix ΔW of the corresponding weight matrix W; S8. Accelerate inference of the large language model after fine-tuning using the stored matrix. This technical solution cannot solve the shortcomings of the existing technology mentioned above because: the essence of this technical solution is not a quantization scheme, but rather a matrix compression decomposition technique that decomposes the weight matrix into multiple smaller matrices, rather than achieving model compression by reducing numerical precision. This technical solution focuses on optimizing storage ratio and computational complexity, but does not address the precision loss problem during the quantization process, thus failing to solve the significant performance degradation risk under low-bit quantization. Furthermore, this technical solution does not evaluate the sensitivity of weights to the loss function, nor does it consider the weight importance distribution, lacking an adaptive quantization mechanism, and cannot cope with the dynamic range sensitivity problem in large language models. The core parameter rank r of this technical solution originates from matrix decomposition rather than the quantization bit width. The problem it solves is limited to matrix storage and computation efficiency, rather than quantization precision loss, and therefore fails to solve the aforementioned technical problems.

[0007] Therefore, this invention aims to provide a design method for a large language model inference accelerator based on cross-layer power-law perceptual quantization, which can solve the performance degradation problem of large language models under low bit quantization due to the difference in the sensitivity of weights to the loss function, and achieve optimization of model accuracy and inference efficiency.

[0008] Furthermore, on the one hand, there are differences in understanding among those skilled in the art; on the other hand, the applicant studied a large number of documents and patents when making this invention, but due to space limitations, not all details and contents were listed in detail. However, this does not mean that the present invention does not possess the features of these prior art. On the contrary, the present invention already possesses all the features of the prior art, and the applicant reserves the right to add relevant prior art to the background art. Summary of the Invention

[0009] The shortcomings of existing technologies are as follows: While quantizing large language models can compress the model size and improve inference speed by converting weights from high-precision floating-point numbers to low-precision integers, the quantization process introduces precision loss. This forces the model to rely on post-training quantization or quantization-aware training to maintain performance, thus increasing additional calibration and fine-tuning overhead. Furthermore, while low-precision quantization improves throughput, it exacerbates the uncertainty of numerical representation. Although existing quantization techniques have smoothed the weight distribution range, they have not offered an effective solution to the dynamic range sensitivity problem in large language models, resulting in a significant risk of performance degradation under low-bit quantization. Specifically, existing technologies do not consider the differences in the sensitivity of weights to the loss function and cannot adaptively allocate quantization precision according to weight importance, causing all weights to be treated equally and failing to effectively address the dynamic range sensitivity problem in large language models.

[0010] To address the shortcomings of existing technologies, this invention provides a design method for a large language model inference accelerator based on cross-layer power-law perceptual quantization. By evaluating the importance scores of weighted sub-blocks, adaptively allocating quantization precision, and combining a dot product architecture with addition followed by multiplication and a quantization-aware register compression strategy, this method effectively mitigates performance degradation under low-bit quantization while improving hardware resource utilization.

[0011] This invention provides, from a first aspect, a large language model inference accelerator based on cross-layer power-law perceptual quantization. The large language model inference accelerator includes a group-level accumulation unit, a dot product operation unit, and a dequantization unit. The group-level accumulation unit is used to group and accumulate input activation values ​​according to the quantization values ​​of weight elements, merging activation values ​​sharing the same weight elements to generate a sum corresponding to each quantization weight, which is then sent to the dot product operation unit. The dot product operation unit is used to multiply the received sum by the corresponding quantization weight element to complete the dot product calculation by reducing redundant multiplication operations, and sends the first-precision dot product calculation result to the dequantization unit. The dequantization unit is used to perform a conversion calculation based on parameters corresponding to the quantization process, converting the first-precision dot product calculation result into a second-precision dot product calculation result, where the first precision is lower than the second precision.

[0012] This invention reduces the number of redundant multiplication operations in low-bit computation by first merging and accumulating activation values ​​sharing the same weight elements to generate a group sum, and then performing a multiplication operation. The dequantization unit converts low-precision data into a high-precision data representation through inverse quantization, reducing computational overhead while preserving model accuracy.

[0013] According to a preferred embodiment, the group-level accumulation unit performs group accumulation of input activation values ​​based on the quantized values ​​of weight elements, including: receiving activation values ​​and quantized weight elements; extracting the sign bit of the weight elements and a selection field for selecting a target accumulation register; inputting the corresponding input activation value into the selected target accumulation register based on the selection field, and generating a control signal based on the sign bit to perform addition or subtraction operations on the target accumulation register, thereby completing the accumulation calculation in symmetric quantization mode and / or asymmetric quantization mode, and sending the group sum to the dot product operation unit.

[0014] This invention extracts the high-order bits of the weight encoding as the sign bit to generate a control signal that drives the arithmetic logic unit to perform addition and subtraction operations. Simultaneously, the remaining bits after removing the sign bit are used as a selection field to determine the target accumulator register address for accumulating input activation values. This strategy allows the same physical accumulator register to simultaneously accumulate activation values ​​from both positive and negative weights, thereby reducing the storage overhead and hardware complexity of the accumulator register.

[0015] According to a preferred embodiment, the steps of the symmetric quantization compression mode performed by the group-level accumulation unit include: integrating the sign bits of the weight elements into the operator logic based on the symmetric distribution characteristics of the weight elements; reconstructing the original expression using the remaining bits of the weight elements as indexes; and merging the activation values ​​associated with equal positive and negative amplitude weights into a single accumulation register, thereby halving the number of accumulation registers required for quantization.

[0016] This invention converts the sign bit of the weight elements into an operation control signal, allowing a pair of weights with the same selection field but opposite signs to share the accumulator register resource. For symmetric quantization with a specific width, the activation value is merged into a single accumulator register for calculation through positive addition and negative subtraction logic, reducing the number of accumulator registers required for quantization by half compared to the traditional method, thereby reducing system power consumption and register overhead.

[0017] According to a preferred embodiment, the steps of the asymmetric quantization compression mode performed by the group-level accumulation unit include: defining the center point as the midpoint between the maximum and minimum values ​​based on the asymmetric distribution characteristics of the weight elements; mapping the original asymmetric quantization form to a symmetric quantization form to suit the case of symmetric quantization and reduce register overhead.

[0018] This invention, based on the asymmetric distribution characteristics of the weight elements, maps the original asymmetric quantization form, which lacks zero-centeredness, into a symmetric set of integer values ​​centered at zero through translation and scaling operations. This equivalent transformation allows the asymmetric quantization mode to be directly applied to a quantization-aware register compression strategy optimized for symmetric quantization, reusing efficient data paths and register architectures.

[0019] According to a preferred embodiment, the step of the dot product operation unit multiplying the received group sums with the corresponding quantization weight elements includes: receiving the group sums that have been accumulated from the group-level accumulation unit; matching the pre-stored quantization weight elements according to the quantization weight identifiers corresponding to the group sums; simultaneously multiplying multiple group sums with the quantization weight elements using a multi-path parallel multiplier; inputting the multiplication result into an adder tree structure and accumulating it level by level to obtain a dot product calculation result with first precision; and outputting the dot product calculation result with first precision to the dequantization unit.

[0020] This invention inputs the output of a multi-channel parallel multiplier into an adder tree structure for multi-level, step-by-step accumulation, yielding a dot product calculation result in first-precision format. This approach reduces the number of multiplier calls and improves the throughput and processing efficiency of the dot product calculation.

[0021] According to a preferred embodiment, the step of the dequantization unit performing conversion calculation based on parameters corresponding to the quantization process includes: receiving a dot product calculation result of first precision from the dot product operation unit and loading pre-stored quantization parameters; performing dequantization calculation on the dot product calculation result based on the quantization parameters to obtain an intermediate result; converting the intermediate result into a dot product calculation result of second precision; and outputting the dot product calculation result of second precision as the input activation value for the subsequent inference level.

[0022] The dequantization unit receives the dot product calculation result at the first precision (low precision) level, and performs inverse scaling and offset conversion calculations using the scaling factor and zero-point parameter corresponding to the quantization. This converts the integer data back to a high-precision representation such as floating-point numbers, outputting the high-precision data format required for subsequent model processing. This step compensates for the precision loss and ensures the robustness of the inference output quality of the large language model.

[0023] According to a preferred embodiment, the large language model inference accelerator further includes an evaluation algorithm unit connected thereto for evaluating the importance of weighted sub-blocks; the evaluation algorithm unit is configured to perform the following steps: calculating the weight gradient magnitude to evaluate the importance score of the weighted sub-blocks; determining the quantization granularity of the weighted sub-blocks based on the clustering features of the weight matrix in the row and column directions; searching for a global importance threshold based on the model perplexity, and assigning quantization precision to each weighted sub-block according to the global importance threshold.

[0024] This invention measures parameter sensitivity by calculating the magnitude of the weight gradient, avoiding the shortcomings of the local weight absolute value method, which lacks cross-layer global comparability and is not associated with the loss function. It utilizes the clustering characteristics of weight importance in the matrix row and column directions to determine the sub-block granularity, and combines this with the inflection point of the model perplexity curve to locate the globally optimal threshold. Based on this, quantization precision is adaptively allocated, reducing memory and computational overhead while mitigating the performance degradation caused by low-bit quantization.

[0025] The present invention provides, from a second aspect, a method for accelerating inference of large language models based on cross-layer power-law perceptual quantization. The method includes: grouping and accumulating input activation values ​​according to the quantization values ​​of weight elements to merge activation values ​​sharing the same weight elements and generate a sum corresponding to each quantization weight; multiplying the received sum with the corresponding quantization weight element to complete the dot product calculation by reducing redundant multiplication operations; and performing a conversion calculation based on parameters corresponding to the quantization process to convert the dot product calculation result in low-precision data format into the dot product calculation result in high-precision data format.

[0026] This invention transforms the traditional dot product calculation process from multiplying before adding to grouping and accumulating the results before performing a single multiplication. By merging activation values ​​with shared weights, it reduces the number of multiplication operations and redundant multiplication operations. Combined with a dequantization conversion mechanism, it effectively improves the throughput and inference speed of the computing unit, while simultaneously reducing the storage and computing resource consumption of large language models in cloud service scenarios.

[0027] According to a preferred embodiment, the method further includes: calculating the magnitude of the weight gradient to evaluate the importance score of the weight sub-blocks; determining the quantization granularity of the weight sub-blocks based on the clustering features of the weight matrix in the row and column directions; searching for a global importance threshold based on the model perplexity, and assigning quantization precision to each weight sub-block according to the global importance threshold.

[0028] To address the dynamic range sensitivity issue in large language models, this invention achieves adaptive precision allocation. It avoids the limitation that a unified quantization strategy cannot compensate for the loss of key information and reduces the calibration and fine-tuning costs of post-training quantization.

[0029] According to a preferred embodiment, the step of grouping and accumulating input activation values ​​based on the quantized values ​​of weight elements includes: receiving activation values ​​and quantized weight elements; extracting the sign bit of the weight elements and a selection field for selecting a target accumulation register; inputting the corresponding input activation value into the selected target accumulation register based on the selection field, and generating a control signal based on the sign bit to perform addition or subtraction operations on the target accumulation register, thereby completing the accumulation calculation of symmetric quantization mode and / or asymmetric quantization mode to obtain the group sum.

[0030] This step unifies the accumulation processing mechanism of symmetric quantization and asymmetric quantization, enabling the same physical register to be reused to perform the merging calculation of multiple activation values, reducing the total requirement of the accumulation register, and improving the overall computing efficiency and resource utilization of the accelerator. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the architecture of the large language model inference accelerator provided by the present invention; Figure 2 This is a schematic diagram of the dot product calculation method provided by the present invention, which involves adding first and then multiplying. Figure 3 This is a schematic diagram of the quantization-aware register compression strategy in the GAU during symmetric quantization provided by the present invention; Figure 4 This is a schematic diagram of the architecture of GAU, DPU and DQU provided by the present invention; Figure 5 This is a schematic diagram of the visual weight matrix of the high-level and low-level layers when the appropriate sub-block granularity is defined, as provided by the present invention. Figure 6 This is a schematic diagram showing the distribution of the sub-block importance score provided by the present invention; Figure 7 This is a schematic diagram illustrating the change of perplexity with a threshold provided by the present invention; Figure 8 This is a schematic diagram of the logical principle of the large language model inference accelerator provided by the present invention.

[0032] List of reference numerals 100: Large Language Model Inference Accelerator; 110: Group Accumulation Unit (GAU); 120: Dot Product Unit (DPU); 130: Dequantization Unit (DQU); 140: Weight Buffer; 150: Input Buffer; 200: Memory Controller; 300: Off-chip Memory; 400: Evaluation Algorithm Unit. Detailed Implementation

[0033] The following is a detailed explanation with reference to the accompanying drawings.

[0034] To facilitate understanding, the technical terms used in this invention will be explained first.

[0035] Large language model: refers to a deep learning model trained on a large amount of text data, which enables the large language model to generate natural language text or understand the meaning of language text.

[0036] Quantization of large language models: Large language model quantization is a technique that converts the parameters of large language models (such as GPT-3, BERT, etc.) from high precision (such as 32-bit floating-point numbers) to low precision (such as 8-bit integers) in order to reduce model size and reduce computational resource requirements.

[0037] Power-law distribution: a form of probability distribution whose core characteristic is that variables have a linear relationship on a double logarithmic coordinate system, which is an unbalanced state in which a few elements occupy the majority of the weights.

[0038] Perplexity: In information theory and natural language processing, this is an indicator used to evaluate the predictive ability of a probabilistic model. The lower the value, the more accurate the model's prediction.

[0039] Dequantization is the process of converting continuously changing physical quantities or signals into discrete values.

[0040] Weight elements: These are the numerical units in the weight matrix of a large language model. Weight elements include a sign bit and a selection field. Weight encoding is the binary bitstream representation of the weight elements after quantization; it is a data format. Weight encoding is composed of the sign bit and the selection field.

[0041] Selection field: Consists of the remaining bits after removing the sign bit in the weighted encoding. The selection field serves as the register addressing index for the group-level accumulator (GAU) 110, used to determine the specific register location for the current input activation value group accumulation.

[0042] The sign bit, extracted from the high-order bits of the weighted encoding, is a control signal used to drive the arithmetic logic unit to perform addition or subtraction operations.

[0043] Pulse array: It is an array of processing units consisting of a large number of processing units arranged in a regular manner and communicating and computing through local, synchronous data flow.

[0044] Memory controller 200: This is a hardware component that performs data interaction and addressing access between the large language model inference accelerator 100 and the off-chip memory 300. During the pipeline execution of model inference, memory controller 200 is used to prefetch activation values ​​or feature vectors stored in off-chip memory 300 and load them into the on-chip input buffer 150. This ensures that the group-level accumulation unit (GAU) 110 can continuously read the required data elements at high throughput, thereby meeting the storage bandwidth requirements of the large language model inference accelerator 100 when performing addition-multiplication dot product calculations.

[0045] External memory 300: This is either DRAM or HBM, used to store model weight parameters and input data required for large language model inference, which are prefetched by the memory controller 200 and loaded into the chip.

[0046] Input buffer 150: Used to temporarily store input values ​​of the memory controller 200.

[0047] Weight buffer 140: Used for temporary storage of the weight matrix.

[0048] Group-level accumulation unit (GAU) 110: Used to group multiple partial sums for parallel computation, and then merge the results of each group to improve computation speed.

[0049] Dequantization Unit (DQU) 130: Used to convert low-precision quantized data back to a high-precision representation.

[0050] Dot Product Unit (DPU) 120: Used to calculate the dot product of two vectors.

[0051] In existing technologies, when performing quantization on large language models, the original high-precision floating-point weights (e.g., FP16 format) are converted into low-precision integers (e.g., INT8 or INT4 format). This process significantly reduces the model's footprint and accelerates inference operations. The underlying logic is to optimize both storage resources and computational efficiency by reducing the precision of numerical values. However, quantization inevitably causes some loss of precision. Therefore, large language models need to rely on post-training quantization techniques or incorporate quantization-aware mechanisms during the training phase to maintain their original performance levels. However, both methods increase the cost of model calibration and parameter fine-tuning.

[0052] Unlike the reasoning process of traditional models, while low-precision computation can improve the throughput of data processing, it reduces the stability of numerical representation and significantly increases uncertainty.

[0053] Existing quantization techniques smooth out the distribution range of weights, but these techniques do not offer an effective solution to the problem of sensitivity to the dynamic range of numerical values ​​in large language models. This means that large language models still face the risk of significant performance degradation when using low-bit quantization.

[0054] This invention proposes a design method for a large language model inference accelerator based on cross-layer power-law perceptual quantization. This invention also proposes a large language model inference accelerator and method based on cross-layer power-law perceptual quantization. This invention can also provide an electronic device, including a processor and a storage medium, for implementing the functions of the large language model inference accelerator 100 based on cross-layer power-law perceptual quantization of this invention. This invention can also provide a storage medium for storing the encoding program of the large language model inference acceleration method based on cross-layer power-law perceptual quantization. This invention can also provide a processor for retrieving and running the encoding program of the large language model inference acceleration method based on cross-layer power-law perceptual quantization from the storage medium, as the large language model inference accelerator 100 based on cross-layer power-law perceptual quantization of this invention.

[0055] This invention proposes a method to accelerate inference for large language models based on cross-layer power-law perceptual quantization. The method includes a sub-block-based cross-layer quantization algorithm and an integrated add-then-multiply dot product operation. The sub-block-based cross-layer quantization algorithm obtains a global importance score by evaluating the sensitivity of weighted sub-blocks to the loss function, and adaptively selects the quantization granularity based on the importance distribution characteristics of the weight elements, employing a heuristic strategy to determine the bit width of different layers and sub-blocks. This invention proposes an add-then-multiply dot product calculation method, implemented with a corresponding architecture to reduce the number of multiplications; and adopts a quantization-aware register compression strategy that supports both symmetric and asymmetric quantization to further reduce hardware complexity and improve inference efficiency and resource utilization. This invention effectively alleviates the performance degradation problem caused by low-bit quantization while ensuring model accuracy, achieving a simultaneous reduction in storage and computational overhead.

[0056] The main drawback of existing technologies is that their quantization methods fail to fully consider the non-uniformity and globality of the impact of weight parameters on model performance, resulting in a significant degradation of model performance (perplexity) at low-bit quantization.

[0057] Specifically, existing quantization techniques typically employ a uniform quantization strategy, ignoring the differences in importance across different regions within the weight matrix. Some methods attempt to use local heuristics such as the absolute value of weights to assess importance, but these metrics lack global comparability across layers and are not directly correlated with the model's final loss function, failing to accurately reflect the true impact of quantization on the overall model performance. This results in the inability to effectively compensate for the loss of key weight information under deep, low-bit quantization, leading to a decline in model output quality.

[0058] The technical problem to be solved by this invention is: how to achieve more efficient low-bit quantization while ensuring the inference accuracy of large language models, so as to simultaneously reduce storage overhead and computational complexity, thereby improving the energy efficiency and performance of inference accelerators.

[0059] Therefore, this invention proposes a large language model inference accelerator 100 based on cross-layer power-law perceptual quantization. This invention makes improvements in the following two aspects: At the algorithmic level, a sub-block-based cross-layer power-law-aware quantization algorithm is proposed. First, this method utilizes the magnitude of the weight gradient as a globally comparable importance scoring criterion, surpassing local absolute value methods. Second, visualization analysis shows that the weight importance is distributed in a two-dimensional sub-block cluster within the matrix; therefore, sub-blocks are used instead of entire rows or columns as the quantization granularity, making the quantization strategy more aligned with the importance distribution. Finally, a heuristic algorithm is designed to adaptively allocate bit widths to sub-blocks of different importance, optimizing the balance between accuracy and compression globally.

[0060] At the hardware architecture level: A dedicated accelerator integrating add-then-multiply dot product operations is proposed. Addressing the limited number of weight values ​​after low-bit quantization, the dot product calculation process is restructured, changing the traditional multiply-then-add approach to add-then-multiply. Specifically, activation values ​​sharing the same weight are first accumulated before a single multiplication, significantly reducing redundant multiplication operations. Simultaneously, a quantization-aware register compression technique is proposed, leveraging the numerical distribution characteristics of symmetric / asymmetric quantization to reduce the number of registers by nearly half, further lowering hardware complexity and power consumption.

[0061] This embodiment provides a large language model inference accelerator based on cross-layer power-law perceptual quantization. The large language model inference accelerator 100 communicates with an external evaluation algorithm unit 400 via wired and / or wireless means to transmit information, such as... Figure 8 As shown.

[0062] Preferably, there are several common ways to physically connect the evaluation algorithm unit 400 and the large language model inference accelerator 100. They can be connected via a data bus through an onboard slot, or they can be connected to a high-speed signal line using a dedicated interface chip based on the CCIX cache coherency protocol. In addition, for low latency requirements, they can also be directly connected via a high-density dedicated high-speed connector on the board or backplane.

[0063] Preferably, the evaluation algorithm unit 400 is used to execute the weight importance evaluation algorithm of the present invention. The physical carrier of the evaluation algorithm unit 400 preferably employs a central processing unit (CPU) and / or a graphics processing unit (GPU). The CPU can be a server-grade high-performance processor such as the Intel(R) Xeon(R) Gold 6430, which has multi-core, high-thread characteristics and is suitable for high-concurrency algorithm scheduling; the GPU can be a computing accelerator card such as the NVIDIA H200, which relies on high-bandwidth memory and CUDA cores to perform parallel computing-intensive evaluation tasks.

[0064] The hardware implementation of the large language model inference accelerator 100 is preferably a dedicated hardware accelerator, specifically employing logic circuits such as Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), or Complex Programmable Logic Devices (CPLDs). Preferably, the large language model inference accelerator 100 integrates the Add-Then-Multiply Dot Product (AFDP) method into a pulse array. This pulse array is used to implement the parallel computation process of the AFDP method. Specifically, the connection logic is as follows: the input end of the pulse array is connected to the weight buffer 140 and the input buffer 150 to receive quantized weights and activation values; the output end is directly connected to the dequantization unit (DQU) 130, realizing pipelined execution of group accumulation and dot product operations.

[0065] Specifically, the method for accelerating large language model inference based on cross-layer power-law perceptual quantization in this invention may include the following steps.

[0066] The evaluation algorithm unit 400 executes a weighted importance evaluation algorithm. This weighted importance evaluation algorithm includes three steps: evaluating the importance score of sub-blocks, defining an appropriate sub-block granularity, and determining the global importance threshold and assigning quantization precision.

[0067] The importance score of a sub-block is used to reflect its role in inference, and is measured by calculating the magnitude of the weight gradient. The selection of sub-block granularity is based on the clustering characteristics of the weight matrix in both row and column directions, using sub-blocks as the quantization granularity. The global importance threshold is determined using a heuristic algorithm to approximately obtain the globally optimal threshold, and the bit width corresponding to different sub-blocks is divided accordingly.

[0068] S100: Evaluation of sub-block importance score.

[0069] The core idea of ​​this invention is to achieve low perplexity under low-precision conditions. The evaluation of sub-block importance scores varies based on the different sensitivities of the weight parameters to the loss function.

[0070] Currently, a widely discussed method for importance assessment involves iteratively zeroing out specific weight substructures in a neural network and measuring the impact of this operation on the overall performance of a large language model, typically using the change in perplexity on the validation set as the importance score. This method calculates importance based on the actual impact of removing a specific subset of parameters on the output of a large language model. However, the drawback of this method is its extremely high computational cost.

[0071] To circumvent the aforementioned cost issues, some existing compression tools have resorted to approximate heuristic metrics to estimate parameter importance, the most typical example being the absolute weight method. This method assumes that weights with larger absolute values ​​are more important in large language models. However, this method based on local statistics has fundamental flaws: first, its importance judgment is limited to within a single layer of the network, lacking a unified standard across layers, and the absolute values ​​of weights between different layers lack a unified benchmark for comparison due to differences in distribution; second, it is usually tied to local optimization objectives such as quantization error, failing to establish a direct correlation with the global loss function of the large language model, thus failing to reflect the true impact of weight perturbations on the final prediction accuracy. This leads to limitations in its evaluation results.

[0072] In fact, the performance degradation caused by weight quantization stems from the significant differences in the sensitivity of different weight parameters to the loss function. From a mathematical optimization perspective, the gradient magnitude (i.e., the loss function) L For weight parameters w i The absolute value of the partial derivative, This invention provides a precise and computable measure of importance. The magnitude of the gradient value directly reflects the linear response strength of the loss function to a small perturbation of the parameter: the larger the gradient, the more significant the change in the parameter will be to the loss function value, thus proving that the parameter is more important to the performance of large language models; conversely, the smaller the gradient, the more the parameter is in a relatively flat loss region, and its importance is lower.

[0073] Based on the above principles, this invention will assign each weight element w i The importance rating is defined as: .

[0074] The evaluation algorithm unit 400 receives the following data from the outside world: the weight matrix of the large language model (the set of parameters obtained after model training) and the sample data used for model inference / training (used to calculate the loss function). In the above formula, the meaning of each parameter is as follows: w i : Represents a weight element, i.e., a numerical unit in the weight matrix of a large language model; L: Represents the global loss function of the model; I(w i ): Represents the weight element w i The importance score indicates that the higher the value, the stronger the influence of the weight element on the model's inference results.

[0075] For a sub-block S (e.g., a weight matrix block) consisting of multiple weight elements, its overall importance is calculated by aggregating the importance scores of all its internal elements. This invention uses the L2 norm to aggregate the importance of each element within the sub-block S; the calculation formula is as follows: .

[0076] Suppose a sub-block S contains 4 weighted elements, and the importance scores of each weighted element are I(w1)=2, I(w2)=3, I(w3)=1 and I(w4)=4, then the overall importance score of this sub-block is: .

[0077] Using this calculation formula can avoid the cancellation of positive and negative gradients, fully reflect the sensitivity of sub-blocks, and provide a basis for subsequent selection of sub-block granularity and allocation of quantization precision.

[0078] S200: Define the appropriate sub-block granularity.

[0079] To ensure the quantization process better reflects the actual distribution of weight importance, sub-blocks should be selected as the quantization granularity. The sub-block importance scores obtained in the previous steps will be directly used in this step: by visually analyzing the element-level importance distribution in the weight matrix and combining the clustering degree of sub-block importance under different candidate granularities, the optimal sub-block granularity is finally determined. The more concentrated the clustering, the closer the granularity is to the importance distribution. Candidate granularities are, for example, 64×64 and 128×128. The optimal sub-block granularity is, for example, 128×128.

[0080] Specifically, when determining the granularity of the sub-blocks for weight quantization, a visual analysis of the element-level importance distribution in the weight matrix was first performed. The element-level importance distribution refers to the two-dimensional spatial distribution formed by arranging the importance scores of each weight element in the weight matrix according to its row and column positions. That is, the value at each position is the I(w) of that element. i The element-level importance distribution is first calculated through a model training / validation process, which calculates the importance score of each element in the weight matrix. Then, I(w) of all elements i The result is obtained by filling in the original row and column coordinates of the weight matrix one by one.

[0081] Figure 5 This paper presents a typical example of the weight importance distribution in the first fully connected sublayer of the third decoding layer with high importance scores and the sixth decoding layer with low importance scores in the OPT-1.3B model. Figure 5 The scale shown is for illustrative purposes only, used to demonstrate the clustering effect under larger-scale matrices. Specifically, in this embodiment, the 4096×4096 matrix is ​​divided into 32×32 sub-blocks. Each pixel represents a color corresponding to a 128*128 sub-block. This color represents the importance of that sub-block (i.e., the sub-block importance score I(S) calculated in the previous step), with rows and columns corresponding to the two dimensions of the original weight matrix. Preferably, the original weight matrix is ​​a parameter matrix (e.g., W used for feature mapping in the Transformer layer) obtained through backpropagation optimization during the training of the large language model. q (Query matrix), W FC1 (e.g., the weight matrix of the first layer of the feedforward network), whose row and column dimensions are determined by the model structure.

[0082] exist Figure 5 In this study, the importance distribution exhibits significant non-uniformity and spatial structure. High importance weights are not randomly distributed, but are highly concentrated in certain continuous sub-block regions, forming a clear clustering effect; conversely, the weights in other regions have extremely low importance, even close to zero, and hardly participate in the calculation of important features.

[0083] In this invention, the importance of a sub-block is determined by a quantile threshold of the global importance score. First, the importance score I(S) of all sub-blocks is calculated and sorted in descending order. Then, a threshold is set at the top 10% quantile: sub-blocks with scores greater than or equal to this quantile are defined as high importance; sub-blocks with scores less than the bottom 50% quantile are defined as low importance. Taking the fully connected sub-layer of the OPT-1.3B model as an example, the top 10% quantile of the importance score for all its sub-blocks is 5.2 (i.e., sub-blocks with scores ≥ 5.2 are high importance), and the bottom 50% quantile is 1.1 (i.e., sub-blocks with scores ≤ 1.1 are low importance).

[0084] Figure 5 It contains two subgraphs, left and right, which correspond to the weight importance distribution of the third fully connected sublayer of the decoding layer (left, important layer) and the sixth fully connected sublayer of the decoding layer (right, secondary layer) of the OPT-1.3B model, respectively.

[0085] In the left image, the dark, continuous blocks represent high-importance sub-blocks. These blocks are continuously distributed along the row and column dimensions, forming a clear block clustering effect. The light-colored blocks represent low-importance sub-blocks, located at the edges of the high-importance clusters. In the right image, only a few scattered dark pixels represent high-importance sub-blocks; there are no clearly continuous clusters, and most areas are low-importance sub-blocks. It is noteworthy that this clustering phenomenon does not occur solely along the row or column directions of the matrix, but rather exhibits clear block-like characteristics simultaneously in both row and column dimensions, indicating a more complex two-dimensional locality in importance distribution.

[0086] Choosing a two-dimensional sub-block substructure as the basic unit of quantization allows for higher precision allocation to key parameters during the quantization process, effectively reducing the loss of important information during quantization. A two-dimensional sub-block substructure refers to a rectangular sub-block that is divided along both the row and column dimensions of the weight matrix. It is the same concept as a matrix sub-block, and is different from a one-dimensional substructure that is divided only along the row or column. Figure 5 Each pixel in the model corresponds to a 128×128 region, which is a two-dimensional sub-block substructure. Taking the 4096×4096 fully connected sub-layer weight matrix of the OPT-1.3B model as an example, by dividing it into groups of 128 elements in the row direction and groups of 128 elements in the column direction, we can obtain (4096÷128)×(4096÷128)=32×32 128×128 two-dimensional sub-block substructures.

[0087] Preferably, the sub-block size is specifically set to 128×128 (note that other sub-blocks can also be applied). This sub-block is used as the sub-block granularity. This choice mainly refers to the typical configuration of previous group quantization techniques, which ensures good compatibility with existing hardware and software libraries, and achieves a practically feasible balance between quantization granularity and computational efficiency.

[0088] S300: Determine the global importance threshold.

[0089] First, calculate the importance score I(S) of all two-dimensional sub-blocks in all layers, and sort these scores in descending order to obtain a globally ordered sequence. Each value in the sequence is the global score of the sub-block.

[0090] When determining the global score, the selection of the optimal global importance threshold includes the following steps.

[0091] Specifically, based on the global comparability of the importance scores of sub-blocks, all sub-blocks can be uniformly sorted across layers according to their importance score values, thereby forming a globally ordered sequence of importance.

[0092] like Figure 6 As shown, all sub-blocks in all layers of a large language model are uniformly sorted according to their importance scores, and arranged in descending order to obtain a distribution map of sub-block importance. Since the allocation ratio of high-precision bit width and low-precision bit width (i.e., weight values ​​are represented by lower bit widths) is symmetrical, the original two-boundary optimization problem (i.e., simultaneously determining the high and low importance boundary points) can be simplified to only needing to determine a single threshold. .

[0093] use Quantiles and 1- The quantiles serve as the boundaries for dividing high-precision and low-precision sub-blocks, respectively. They are used to divide important sub-blocks that should be allocated high-precision bit width and unimportant sub-blocks that should be allocated low-precision bit width, while the remaining sub-blocks retain a medium level of precision. Quantiles refer to the score values ​​corresponding to the top proportions after all sub-block importance scores are sorted in descending order (e.g., ...). When =0.1, The quantile (the lowest score of the top 10% of sub-blocks after sorting) is used to delineate important sub-blocks that require allocation of high-precision bit width; 1- Quantiles refer to the score values ​​corresponding to the next proportional position after sorting, used to divide non-important sub-blocks that require low-precision bit width allocation. The remaining sub-blocks retain a medium level of precision. For example, when... When =0.1, 1- =0.9; the quantile is the highest score of the 10% of sub-blocks after sorting.

[0094] Before proposing a globally optimal threshold selection algorithm, the distribution characteristics of sub-block importance scores and different thresholds are analyzed based on the OPT-1.3B model. The impact of value selection on the perplexity of large language models.

[0095] Figure 6 The cumulative distribution of sub-block importance scores is shown. Figure 6 In the graph, the vertical axis represents the standardized importance of sub-blocks, with higher values ​​indicating higher importance scores. The horizontal axis represents the quantiles, indicating the proportion of a sub-block in the overall importance ranking. The curve represents the standardized importance score of the sub-block at the corresponding quantile. When the horizontal quantile increases from 0.0 to 0.2, the vertical importance score rapidly decreases from 1.0 to near 0.0, indicating that the top 20% of sub-blocks concentrate the vast majority of high importance scores, reflecting the power-law distribution characteristic of sub-block importance.

[0096] Figure 7 The perplexity was shown to vary with the threshold. A changing curve. Preferably, a threshold. The value ranges from 0 to 0.5, with a step size of 0.002. Specifically, the threshold value of this invention... The determination requires two key observational pieces of information.

[0097] On the one hand, such as Figure 7 As shown, the perplexity varies with the threshold. The change in importance initially shows a trend of decreasing and then increasing. On the other hand, the globally optimal threshold is usually located near the inflection point of the importance distribution curve (see...). Figure 6 When the threshold When the threshold is small, improving the accuracy of extremely important sub-blocks can significantly improve the performance of large language models, and the benefits far outweigh the losses from reducing the accuracy of less important sub-blocks, thus significantly reducing perplexity. As the threshold... As importance scores increase, the distribution of importance scores tends to flatten out, with some moderately important sub-blocks being included in the low-precision allocation range, leading to diminishing marginal returns and a slower rate of decrease in perplexity. Perplexity reaches its minimum when the gains from high-precision sub-blocks cannot offset the losses caused by low-precision sub-blocks; this corresponds to the threshold value. This is the theoretically optimal solution, i.e., the globally optimal threshold. Therefore, this invention approximates the globally optimal threshold by locating the inflection point of the importance distribution curve, where... Figure 7 The inflection point is Figure 6 The point at which the rate of increase or decrease transitions from a sharp decline at the front end to a slow decline at the back end is the critical point where the rate of increase or decrease changes significantly.

[0098] S400: Since the initial threshold failed to reach the optimal perplexity, this invention further proposes to determine the globally optimal threshold. The algorithm.

[0099] Initial threshold The value is determined by Figure 7 The x-axis of the inflection point of the perplexity curve is determined. Taking the OPT-1.3B model as an example, its initial threshold is... Approximately 0.05; Optimal perplexity refers to the lowest perplexity of a large language model on the test dataset, determined by evaluating the model output at different thresholds on the validation dataset. Figure 7 For example, the optimal perplexity is the value of the ordinate corresponding to the inflection point in the curve.

[0100] Specifically, determine the globally optimal threshold. The algorithm is an efficient search algorithm whose goal is to find a function that minimizes the evaluation function PPL in the interval [0, 0.5]. The threshold for (i.e., perplexity).

[0101] When the calculation begins, the current optimal threshold is initialized. The initial threshold for input At this point, the initial threshold Corresponding to the above Figure 7 The inflection point shown is, that is Figure 7 The critical point at which the decay rate changes significantly is identified, and its corresponding perplexity is calculated as the initial optimal value. Perplexity is a commonly used evaluation metric in language models, used to measure the uncertainty of a large language model for test data. Intuitively, a smaller perplexity indicates a more concentrated probability distribution in the large language model's predictions of the sample sequence; a larger perplexity indicates higher uncertainty in the large language model's predictions of the sample sequence.

[0102] Global optimal threshold The algorithm sets the first trial point on both sides of the current threshold according to an exponential law. Second test point The two trial points are set as follows: ; .

[0103] In the above formula, Indicates the lower bound of the search; Indicates the upper bound of the search.

[0104] First probing point Second test point These represent candidate values ​​for narrowing and expanding the threshold range, respectively. The upper and lower bounds determine the possible range of the globally optimal threshold during the initial iteration. The two will continuously approach each other during the iteration process until the globally optimal threshold is found.

[0105] During the iteration process, the evaluation algorithm unit 400 continuously evaluates these three positions ( , , The perplexity of the data is represented by the position of minimum perplexity (the position of minimum perplexity can be referenced). Figure 7 The search interval is dynamically adjusted at the lowest point in the middle: if the first trial point If the performance is optimal, then the evaluation algorithm unit 400 will narrow the upper bound of the search to the second trial point. And continue to explore to the left in exponential steps; if the second test point If optimal, then similar expansion to the right of the lower bound; if the current initial threshold If the optimal solution remains, the interval is simultaneously narrowed inward from both sides. When the perplexity change is detected to be below a preset threshold for multiple consecutive iterations and the predetermined convergence condition is not met, a random perturbation is introduced to escape the local non-convex region and perform a fine-grained search. This strategy combines the ideas of heuristic expansion and interval halving, avoiding global exhaustive search. Thus, while ensuring search efficiency, it can adaptively explore the solution space and approach the global optimal threshold. Example Figure 7 The lowest point in the middle is the global optimal threshold.

[0106] The evaluation algorithm unit 400 is equipped with a threshold trial control component, which dynamically adjusts the search step size based on the gradient changes of the perplexity curve. When the current perplexity performance is detected to be in an oscillation range, the threshold trial control component controls the trial point to adopt a large step-size jump strategy to cross local non-convex extrema regions (or non-smooth regions); when the perplexity performance enters a smooth decreasing range, the threshold trial control component controls the trial point to switch to an exponential half-convergence state to gradually approach and lock the globally optimal threshold inflection point. In addition, a multi-scale grid partitioning controller can be configured in the weight buffer 140 to dynamically divide the weight matrix into coarse-grained (e.g., 32×32) or fine-grained (e.g., 64×64) grids according to the different decoding layer depths and send them to the group-level accumulation unit (GAU) 110.

[0107] This invention also provides a large language model inference accelerator 100 that integrates the addition-then-multiplication dot product method into a pulse array. The large language model inference accelerator 100 is preferably based on a Quam architecture, such as... Figure 1 As shown. The computational process of this large language model inference accelerator 100 may include the following steps.

[0108] The interaction logic between the evaluation algorithm unit 400 and the large language model inference accelerator 100, as well as the individual components and their execution processes, are as follows.

[0109] The large language model inference accelerator 100 includes a group-level accumulation unit (GAU) 110, a dot product operation unit (DPU) 120, and a dequantization unit (DQU) 130. After the evaluation algorithm unit 400 completes the calculation of sub-block importance scores and the determination of global thresholds, the evaluation algorithm unit 400 transmits two types of data to the large language model inference accelerator 100: ① quantized weight data (allocated with corresponding bit precision according to sub-block importance); ② quantization configuration parameters (including sub-block granularity, quantization precision level of each sub-block, etc.).

[0110] After receiving the above two types of data from the evaluation algorithm unit 400, the large language model inference accelerator 100 executes the quantized matrix operation process.

[0111] The Group Accumulator Unit (GAU) 110 converts the multiple multiplication operations in the Dot Product Unit (DPU) 120 into operations where activation values ​​with the same weight are first accumulated, and then multiplied by the weight in a single operation, thereby reducing the number of multiplication operations and improving overall performance. A quantization-aware register compression strategy is used to further reduce the register storage overhead within the Group Accumulator Unit (GAU) 110, thereby improving hardware performance and resource utilization.

[0112] The dot product unit (DPU) 120 is used to receive the group sum (group accumulation result) output by the group-level accumulation unit (GAU) 110, and multiply the received group sum with the quantization weight element corresponding to the group sum to reduce redundant multiplication operations in low-bit calculations.

[0113] Specifically, such as Figure 1 As shown, input buffer 150 is a high-speed static memory located on the chip, used to temporarily buffer input data to be processed. Figure 1 As shown, during execution, as Figure 2 As shown, the memory controller 200 prefetches input data from off-chip memory 300 into the input buffer 150. Subsequently, the group-level accumulator (GAU) 110 repeatedly reads the required data from the input buffer 150 over multiple computation cycles.

[0114] Specifically, the weight buffer 140 is an on-chip cache similar to the input buffer 150, but it is dedicated to storing model weight parameters.

[0115] The model weight parameters typically remain unchanged during the execution of the large language model, but are repeatedly used by the Group Accumulator Unit (GAU) 110 to process different input data. The execution process begins during the model loading phase, where all weights required for the current task are sent from off-chip memory 300 to the weight buffer 140. In subsequent computation cycles, both the Group Accumulator Unit (GAU) 110 and the Dot Product Unit (DPU) 120 read parameters from the weight buffer 140.

[0116] Group-level accumulator (GAU) 110 implements a quantization-aware register compression strategy that supports both symmetric and asymmetric quantization. For example... Figure 1 As shown, the group-level accumulation unit (GAU) 110 is connected to the dot product operation unit (DPU) 120. The dot product operation unit (DPU) 120 is connected to the dequantization unit (DQU) 130.

[0117] Group-level accumulation unit (GAU) 110 is used to group and accumulate the input activation values ​​according to the quantization value of the weight elements, so as to merge the activation values ​​that share the same weight elements, generate the group sum corresponding to each quantization weight, and send it to the dot product unit (DPU) 120.

[0118] Preferably, when the group-level accumulator unit (GAU) 110 is an FPGA, the accumulator register is an on-chip memory unit within the FPGA. The accumulator register is physically connected to the arithmetic logic unit (ALU) inside the group-level accumulator unit (GAU) 110 via internal signal lines. Each group-level accumulator unit (GAU) 110 incorporates a set of accumulator registers (half the number), a configurable adder / subtractor, and lightweight sign control logic. The accumulator registers are used to temporarily store the accumulated result of the active value.

[0119] Specifically, the group-level accumulation unit (GAU) 110 receives the data elements required for computation from the input buffer 150, dynamically groups and accumulates them according to the weight data to form a group sum for use by subsequent computation units. Data elements refer to activation values ​​in the large language model's inference process, i.e., the output results of the previous layer of the model. For example, the feature values ​​of a token's embedding vector after a linear transformation. , 1, 2. In the model inference process, the input token sequence is processed by pre-processing units such as the embedding layer, multi-head attention layer, and feedforward network to generate corresponding feature vectors (i.e., activation values). These feature vectors are loaded from off-chip memory 300 to input buffer 150 by memory controller 200 and used as input data for group-level accumulation unit (GAU) 110 to participate in subsequent group accumulation calculations.

[0120] Group-level accumulator (GAU) 110 receives a data indexing instruction (the instruction contains the address range of the activation value to be calculated in the input buffer 150) sent by an external control unit, and collects the corresponding activation value data from the input buffer 150 according to the data indexing instruction.

[0121] Preferably, such as Figure 2As shown, the group-level accumulation unit (GAU) 110 receives weight data from the weight buffer 140. The group-level accumulation unit (GAU) 110 groups the activation values ​​corresponding to the same quantized weight value according to the quantized value of the received weight data; then, it sums the activation values ​​within the same group.

[0122] For a certain weight w i The formula for summing the activation values ​​is: .

[0123] In the above formula, b ij Indicates weight w i The j-th activation value in the group.

[0124] For example, if the group-level accumulator (GAU) 110 receives quantization weights of [002, 012, 002, 102], and the corresponding activation values ​​are [2, 3, 4, 5], then the activation values ​​corresponding to quantization weight 002 are 2 and 4, and the group sum is 2 + 4 = 6; the activation value corresponding to quantization value 012 is 3, and the group sum is 3; the activation value corresponding to quantization weight 102 is 5, and the group sum is 5.

[0125] S500: The Group Accumulator Unit (GAU) 110 internally performs a quantization-aware register compression strategy based on the selection field and the sign bit.

[0126] Quantization awareness refers to the system's ability to identify and utilize the numerical distribution characteristics of quantization weights (such as positive and negative symmetry) to send the activation values ​​corresponding to the same weights as the selected fields to the same physical register for multiplexing calculations.

[0127] This invention is based on the ability to reduce register overhead by utilizing the distribution characteristics of quantized values ​​during symmetric quantization. Symmetric quantization is a key data compression technique that reduces computational and storage overhead by mapping high-precision weights to a set of discrete integer values ​​centered at zero.

[0128] Specifically, when the group-level accumulator (GAU) 110 processes the quantization weights, it extracts the sign bit and selection field of the weights. The selection field is used to select the target accumulator register. The group-level accumulator (GAU) 110 selects its internal target accumulator register based on the selection field and generates a control signal based on the sign bit to drive its internal arithmetic logic unit (ALU) to perform addition or subtraction operations on the target accumulator register, thereby directly accumulating the corresponding activation value received from the input buffer 150 into the internal target accumulator register.

[0129] Taking the weight values ​​of +1 (binary 0012) and -1 (binary 1012) as examples, both cases will select the same accumulator register (the corresponding selection field is 1), but the former will add the activation value to the accumulator register, while the latter will subtract the activation value from the accumulator register.

[0130] Specifically, the activation value is received by the GAU from the input buffer 150. The accumulator register is an on-chip storage unit in the FPGA, which is physically connected to the arithmetic logic unit (ALU) inside the GAU through internal signal lines. It is used to temporarily store the accumulated result of the activation value.

[0131] This design allows the hardware accelerator to process all possible values ​​of the 3-bit quantization weight using only four accumulator registers, halving the number of accumulator registers. The solution of this invention is not only applicable to 3-bit quantization but can also be extended to symmetric quantization schemes with other bit widths. For n-bit quantization, traditional methods require 2... n With this optimization, only 2 accumulator registers are needed. n-1 One accumulator register.

[0132] Specifically, in the case of asymmetric quantization, weights are typically represented as unsigned integers (e.g., {0, 1, ..., 7} in 3-bit quantization), whose numerical distribution does not possess zero-centeredness. To extend the quantization-aware register compression strategy previously designed for symmetric quantization to the asymmetric domain, this invention proposes an equivalent mathematical transformation that maps the original quantized values ​​to a zero-centered symmetric form, thereby achieving unified support for hardware optimization strategies. Taking 3-bit quantization as an example, the original set {0,1,...,7}, after translation and scaling, is mapped to a symmetric set of integer values ​​{-7, -5, -3, -1, +1, +3, +5, +7}. Although the numerical interval remains 2, its distribution already possesses strict zero-symmetry.

[0133] Based on this transformation, this invention designs a general-purpose group-level accumulator (GAU) 110 that can simultaneously support symmetric and asymmetric quantization. The GAU 110 executes a quantization-aware register compression strategy, that is, it allocates accumulator registers only based on the weight selection field, while converting the sign bit into an operation control signal. Each GAU 110 incorporates a set of halved accumulator registers (e.g., only 4 are needed for 3-bit quantization), a configurable adder / subtractor, and lightweight sign control logic.

[0134] During operation, the high-order bits of the weight value are extracted as the sign bit, and the remaining bits are used as a selection field to select the target accumulation register within the Group Accumulator Unit (GAU) 110. If the sign bit indicates a negative value, the GAU 110 drives the internal arithmetic logic unit to subtract the corresponding input activation value from the target accumulation register; if the sign bit indicates a positive value, the GAU 110 drives the internal arithmetic logic unit to perform addition and add it to the accumulation register. This mechanism allows the same physical register to simultaneously accumulate activation value contributions from both positive and negative weights, thereby reducing the total number of accumulation registers to half that of traditional schemes.

[0135] Preferably, the group-level accumulation unit (GAU) 110 is internally equipped with lightweight symbol control logic. When the extracted numerical bits of the weight element are non-zero, the lightweight symbol control logic is in the ON position, enabling the arithmetic logic unit and the target accumulation register to perform accumulation calculation normally; when the weight element is zero (such as +0 or -0 encoding), the lightweight symbol control logic is in the OFF state (or OFF state), causing the arithmetic logic unit to skip the accumulation operation of the current clock cycle and remain idle, thereby avoiding invalid multiplication and addition operations on zero-weight elements and improving hardware resource utilization.

[0136] Specifically, Figure 3 The diagram contains two sub-graphs (a) and (b), which respectively show the comparison of the calculation logic of the accumulator register group before and after applying the quantization-aware register compression strategy.

[0137] exist Figure 3 In (a), the set of 3-bit symmetric quantization weight codes is {1112,1102,1012,1002,0112,0102,0012,0002}, corresponding to the set of values ​​{-3,-2,-1,0,+1,+2,+3} (1002 corresponds to the value 0). Each column corresponds to the sum of the activation values ​​of a weight code: the first column is the sum of the activation values ​​corresponding to weight code 0002, i.e. The second column is the sum of the activation values ​​corresponding to weight code 0012, i.e. The eighth column represents the cumulative activation value corresponding to weight code 1112, i.e. ; Figure 3 In (b), utilizing the characteristic of symmetric quantization centered on the value 0, the activation values ​​corresponding to two weights with equal selection fields are merged into the same register, and the calculation is achieved through positive addition and negative subtraction: in the first column, the weight codes 0002(0) and 1002(0) are merged and calculated as follows: In the second column, the weight codes 0012(+1) and 1012(-1) are combined and calculated as follows: In the third column, the weight codes 0102(+2) and 1102(-2) are combined and calculated as follows: In the fourth column, the weight codes 0112(+3) and 1112(-3) are merged and calculated as follows: .

[0138] like Figure 3 As shown, typical 3-bit symmetric quantization maps the original weights to the encoding set {1112, 1102, ..., 0112}, corresponding to the numerical set {-3, -2, -1, 0, +1, +2, +3}. This symmetric distribution not only reduces the number of bits in the data representation but also allows a pair of weights with the same selection field but opposite signs to share the accumulator register resources, providing a structural basis for accumulator register optimization. (Reference) Figure 3 The logic shown is as follows: Originally, the activation value corresponding to weight +1 and the activation value corresponding to weight -1 needed to be stored in two separate registers. After optimization, they can be merged into one register for calculation. The corresponding calculation is as follows: This reduces register overhead by half.

[0139] S600: The accumulated result is transmitted to the dot product unit (DPU) 120.

[0140] Preferably, to address the limitations of traditional pulse array architectures in terms of quantization efficiency, the dot product operation performed by the dot product operation unit (DPU) 120 includes merging calculations based on a finite number of values ​​in the weights.

[0141] Preferably, the operation performed by the dot product arithmetic unit (DPU) 120 includes a method for calculating the dot product by addition followed by multiplication (AFDP).

[0142] like Figure 2 As shown, in the original dot product calculation method, the complexity of dot product multiplication is: O(|W vector |), two vectors (W) vector and a vector In the original text, each pair of elements requires a multiplication-addition operation. However, by using the add-then-multiply dot product method, only the identical w elements need to be multiplied and combined, transforming it into an add-then-multiply dot product. The multiplication complexity of the add-then-multiply dot product is O(2^3). b (b is the number of quantization bits). For example, with a weight vector of 3 quantization weights and a length of 128: the traditional dot product multiplication takes 128 steps (complexity O(128)); in the addition-then-multiplication mode, assuming the weight vector is [00, 01, 10, 11], activation values ​​with the same weight will be merged and accumulated, and finally only 8 unique weight values ​​need to be multiplied (8 times), with a complexity of O(2^3). 3 =O(8), which reduces multiplication operations by 93.75% compared to the traditional method.

[0143] Specifically, Figure 2The diagram contains two subgraphs, (a) and (b), illustrating the complexity comparison between traditional dot product and AFDP. Figure 2 (a) is a schematic diagram of the traditional dot product, with the numbers in the first row representing the weight vector W. vector Its elements 002, 012, 102, and 112 are 0, +1, 0, and -1 respectively, and the numbers in the second row are the activation vector a. vector Its elements , , ... These are the corresponding activation values. The "×+" in the middle indicates the traditional dot product calculation logic, which is to multiply each weight element by its corresponding activation value, and then add all the products together. The formula is: ; Figure 2 (b) is a schematic diagram of the AFDP calculation method. The numbers 00, 01, 10, and 11 in the first row represent all possible values ​​of the 2-bit quantization weight. The "+" sign in the middle indicates the summation of activation values ​​with the same weight. The numbers in the second row represent the sum of the activation values ​​corresponding to the same weight (for example, the sum of the activation values ​​corresponding to 00 is 11). The formula corresponding to the calculation logic is: Where m represents the number of activation values ​​corresponding to a certain weight. This represents the j-th activation value of the i-th weight.

[0144] Specifically, the dot product unit (DPU) 120 is used to handle a small number of addition operations. In each clock cycle, it receives the sum from the group-level accumulator (GAU) 110 and multiplies the sum by the corresponding weight element to complete the overall dot product operation. Since the dot product is performed by adding first and then multiplying, most multiplication operations are reduced. Therefore, in this invention, the number of multiplier calls can be significantly reduced.

[0145] Specifically, the dot product unit (DPU) 120 receives a low-precision accumulation result of the activation values, with corresponding weights being quantized low-bit integers (such as 2-bit / 3-bit integers). The product of these two is still a low-precision integer, therefore the DPU 120 outputs a low-precision final calculation result. The dequantization unit (DQU) 130 is used to restore the final output result to a high-precision format for subsequent processing or storage. The dequantization unit (DQU) 130 receives the low-precision final calculation result from the DPU 120 and, using parameters symmetrical to the quantization unit, performs an inverse scaling and offset mathematical process to convert the integer data back to a high-precision floating-point representation.

[0146] The solution quantization formula for symmetric quantization is: High-precision result = Low-precision result × s.

[0147] The solution quantization formula for asymmetric quantization is: High-precision result = Low-precision result × s + Z0.

[0148] In the above formula, s is the scaling factor used in the quantization process, and Z0 is the zero-point parameter used in the quantization process.

[0149] If asymmetric INT3 quantization (s=0.5, Z0=1.0) is used during quantization, and the low-precision result output by the dot product unit (DPU) 120 is 3, then after dequantization, the corresponding high-precision result is 3×0.5+1.0=2.5.

[0150] The Dot Product Unit (DPU) 120 is integrated as follows: Figure 4 The tree structure of the multi-path parallel multipliers and adders shown can efficiently complete multiplication and final accumulation between different precision domains.

[0151] Figure 4 The hardware logic architecture of the large language model inference accelerator 100, as demonstrated, mainly consists of two parts: a multi-path parallel multiplier and an adder tree structure. The multi-path parallel multiplier corresponds to... Figure 4 The module connecting the input group sums and quantization weights supports simultaneous multiplication operations on multiple group sums from multiple group-level accumulator units (GAUs) 110 with their corresponding quantization weights, adapting to quantization weights of different precisions. The adder tree structure corresponds to... Figure 4 The multi-stage addition module following the multiplier outputs accumulates the outputs of the multiple parallel multipliers stage by stage, ultimately obtaining a single dot product result. For example, four multiplication results are grouped together and added, then the two groups are added together to complete the final accumulation. This structure improves the throughput and efficiency of dot product calculations through parallel multiplication and attribute accumulation.

[0152] Specifically, such as Figure 4 As shown, the hardware logic architecture of the large language model inference accelerator 100 includes a group-level accumulation unit (GAU) 110, a dot product operation unit (DPU) 120, and a dequantization unit (DQU) 130.

[0153] like Figure 4 As shown in (a), the group-level accumulation unit (GAU) 110 includes a weight parsing module, a selection control module, and an accumulation register group. ① is the sign bit (corresponding to...) ); ② is the selection field (corresponding to and DEMUX is a demultiplexer; MUX1 is the first multiplexer, and MUX2 is the second multiplexer. ③ is a feedback adder; This is the accumulator register.

[0154] The input of the group-level accumulator unit (GAU) 110 is connected to the DEMUX, and its output is split into four parallel branches according to the selection field ②. Each branch is connected to the addition or subtraction logic through MUX1 and then enters the accumulator register. The register group outputs converge to the feedback adder ③ via MUX2. The output of the group-level accumulator (GAU) 110 (group sum) The parallel multiplier group ④ is connected to the input terminal of the dot product unit (DPU) 120. The multiplier output of the dot product unit (DPU) 120 is connected to the Adder.

[0155] like Figure 4 As shown in (b), the dot product unit (DPU) 120 includes a multi-path parallel multiplier and an adder tree. ④ is a parallel multiplier group (composed of...) to (Composition); Adder is a cascaded adder. The output of the dot product unit (DPU) 120 is connected to the inverse scaling multiplier ⑤ of the dequantization unit (DQU) 130.

[0156] like Figure 4 As shown in (c), the dequantization unit (DQU) 130 performs precision recovery and compensation. ⑤ ⑥ is an inverse scaling multiplier; ⑦ is a self-adder; ⑧ is an offset-compensated multiplier; ☐ is a shift-adder. Original activation value The outputs of the inverse scaling multiplier ⑤ and the offset compensation multiplier ⑦ are connected together to the displacement adder ⑧.

[0157] Figure 4 The information transmission process of the large language model inference accelerator 100 shown is as follows.

[0158] The group-level accumulator (GAU) 110 outputs a group sum of multiple activation values ​​(e.g., MUX2 and feedback adder ③) through MUX2 and feedback adder ③. The sum of each group is fed to the dot product unit (DPU) 120. The dot product unit (DPU) 120 combines each group with the corresponding quantized weight element ( The products are multiplied in the first-level parallel multiplier ④, and then accumulated level by level through the adder tree to output the single-dot product calculation result with the first precision (low precision).

[0159] The dequantization unit (DQU) 130 receives the single-dot product calculation result of the first precision (low precision) and performs an inverse scaling calculation of the scaling factor s through the inverse scaling multiplier ⑤. Simultaneously, the adder ⑥ performs an inverse scaling calculation on the bypass input. Summation yields The zero-point offset compensation calculation is performed by the offset compensation multiplier ⑦. Finally, the two signals are combined by the displacement adder ⑧ to output the dot product calculation result with second precision (high precision).

[0160] Through the above design, asymmetric quantization can fully reuse the efficient data path and register architecture optimized for symmetric design without significantly increasing hardware overhead, thus significantly improving the hardware efficiency and flexibility of quantization inference.

[0161] Preferably, in asymmetric quantization mode, to optimize the transmission path of compensation data, a data pass-through bypass bus can be set within the large language model inference accelerator 100. This bus is used to directly guide the original input activation value issued by the input buffer 150 to the adder inside the dequantization unit (DQU) 130 when the bypass enable signal is activated. The dot product unit (DPU) 120 internally consists of a heterogeneous multiply-add pipeline branch structure composed of multiple first-stage multipliers, an adder tree structure, and second-stage multipliers, to independently handle the core group and multiplication and offset compensation calculations. In addition, a scaling factor compensation module can be integrated into the dequantization unit (DQU) 130. When a mapping instruction with a specific asymmetric step size (e.g., a step size of 2) is received, this scaling factor compensation module dynamically adjusts the dequantization calculation node, halving the global scaling factor in the dequantization formula, enabling the dequantization unit (DQU) 130 to perform proportional adaptive scaling correction on data with displacement deviations.

[0162] For ease of description, the above-described apparatus is divided into various functional units. Of course, in implementing this invention, the functions of each unit can be implemented in one or more software and / or hardware components.

[0163] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes said element.

[0164] This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0165] It should be noted that the specific embodiments described above are exemplary. Those skilled in the art can devise various solutions inspired by the disclosure of this invention, and these solutions all fall within the scope of this invention and its protection. Those skilled in the art should understand that this specification and its accompanying drawings are illustrative and not intended to limit the scope of the claims. The scope of protection of this invention is defined by the claims and their equivalents. This specification contains multiple inventive concepts; terms such as "preferredly," "according to a preferred embodiment," or "optionally" indicate that the corresponding paragraph discloses an independent concept. The applicant reserves the right to file divisional applications based on each inventive concept.

Claims

1. A large language model inference accelerator based on cross-layer power-law perceptual quantization, characterized in that, The large language model inference accelerator (100) includes: The group-level accumulation unit (110) is used to group and accumulate the input activation values ​​according to the quantization value of the weight element, so as to merge the activation values ​​that share the same weight element, generate the group sum corresponding to each quantization weight, and send it to the dot product operation unit (120). The dot product operation unit (120) is used to multiply the received group sum with the corresponding quantization weight element to complete the dot product calculation by reducing redundant multiplication operations, and send the dot product calculation result with first precision to the dequantization unit (130). The dequantization unit (130) is used to perform a conversion calculation based on parameters corresponding to the quantization process, converting the dot product calculation result of the first precision into a dot product calculation result of the second precision, wherein the first precision is lower than the second precision.

2. The large language model inference accelerator according to claim 1, characterized in that, The group-level accumulation unit (110) performs group accumulation of the input activation values ​​according to the quantization values ​​of the weight elements, including the following steps: Receive activation values ​​and quantized weight elements; Extract the sign bit of the weighted element and the selection field used to select the target accumulator register; Based on the selection field, the corresponding input activation value is input to the selected target accumulator register, and a control signal is generated based on the sign bit to perform addition or subtraction operations on the target accumulator register, thereby completing the accumulation calculation of symmetric quantization mode and / or asymmetric quantization mode, and sending the generated sum to the dot product operation unit (120).

3. The large language model inference accelerator according to claim 1 or 2, characterized in that, The steps of the symmetric quantization compression mode performed by the group-level accumulation unit (110) include: Based on the symmetric distribution characteristics of the weight elements, the sign bit of the weight elements is integrated into the operator logic; Reconstruct the original expression using the remaining bits of the weighted elements as indices; By merging activation values ​​associated with equal positive and negative amplitude weights into a single accumulator register, the number of accumulator registers required for quantization is halved.

4. The large language model inference accelerator according to any one of claims 1 to 3, characterized in that, The steps of the asymmetric quantization compression mode performed by the group-level accumulation unit (110) include: Based on the asymmetric distribution characteristics of the weighted elements, the center point is defined as the midpoint between the maximum and minimum values; The original asymmetric quantization form is mapped to a symmetric quantization form to suit symmetric quantization cases, reducing register overhead.

5. The large language model inference accelerator according to any one of claims 1 to 4, characterized in that, The steps of the dot product operation unit (120) multiplying the received sum of data with the corresponding quantized weight element of the sum of data are as follows: Receive the group sum that has been accumulated from the group-level accumulation unit (110); Based on the group and the corresponding quantization weight identifier, match the pre-stored quantization weight elements; Multiple sets of sums are multiplied by quantized weight elements simultaneously using a multi-path parallel multiplier. The result of the multiplication operation is input into the adder tree structure and accumulated level by level to obtain the dot product calculation result with the first precision; The dot product calculation result of the first precision is output to the dequantization unit (130).

6. The large language model inference accelerator according to any one of claims 1 to 5, characterized in that, The steps by which the dequantization unit (130) performs the conversion calculation based on the parameters corresponding to the quantization process are as follows: Receive the first precision dot product calculation result from the dot product operation unit (120) and load the pre-stored quantization parameters; Based on the quantization parameters, the dot product calculation result is dequantized to obtain an intermediate result; The intermediate results are then converted into second-precision dot product calculation results; The result of the second precision dot product calculation is output as the input activation value for subsequent inference levels.

7. The large language model inference accelerator according to any one of claims 1 to 6, characterized in that, The large language model inference accelerator (100) also includes an evaluation algorithm unit (400) connected thereto, for transmitting quantization weight data and quantization configuration parameters to the large language model inference accelerator (100); The evaluation algorithm unit (400) is configured to perform the following steps: Calculate the magnitude of the weight gradient to evaluate the importance score of the weighted sub-blocks; The quantization granularity of the weight sub-blocks is determined based on the clustering features of the weight matrix in the row and column directions. A global importance threshold is searched based on the model perplexity, and quantization precision is assigned to each weighted sub-block according to the global importance threshold.

8. A method for accelerating reasoning in large language models based on cross-layer power-law perceptual quantization, characterized in that, The method includes: The input activation values ​​are grouped and accumulated according to the quantization value of the weight elements, so as to merge the activation values ​​that share the same weight elements and generate the group sum corresponding to each quantization weight. The received sum of groups is multiplied by the corresponding quantized weight element to complete the dot product calculation by reducing redundant multiplication operations; A conversion calculation is performed based on parameters corresponding to the quantization process to convert the dot product calculation result of first precision into a dot product calculation result of second precision, wherein the first precision is lower than the second precision.

9. The method according to claim 8, characterized in that, The method further includes: Calculate the magnitude of the weight gradient to evaluate the importance score of the weighted sub-blocks; The quantization granularity of the weight sub-blocks is determined based on the clustering features of the weight matrix in the row and column directions. A global importance threshold is searched based on the model perplexity, and quantization precision is assigned to each weighted sub-block according to the global importance threshold.

10. The method according to any one of claims 8 or 9, characterized in that, The step of grouping and accumulating the input activation values ​​based on the quantization values ​​of the weight elements includes: Receive activation values ​​and quantized weight elements; Extract the sign bit of the weighted element and the selection field used to select the target accumulator register; Based on the selection field, the corresponding input activation value is input to the selected target accumulator register, and a control signal is generated based on the sign bit to perform addition or subtraction operations on the target accumulator register, thereby completing the accumulation calculation of symmetric quantization mode and / or asymmetric quantization mode to obtain the sum.

Images