A benchmarking method and system for large models

Abstract

The application discloses a benchmark test method and system for a large model. The method firstly classifies and configures the strategy of a new quantization algorithm, and performs calibration data sensitivity test and cutting semantic alignment based on token distribution consistency on a unified software and hardware test platform; then, the new algorithm is combined with existing modules as a strategy module, and the algorithm and its combination chain are systematically evaluated from three dimensions of calibration data, algorithm strategy and data format; finally, the performance index is rated or filtered according to the preset performance index, and a standardized evaluation report is output. The application solves the technical problems of messy configuration, single dimension and high hardware threshold in the evaluation of large model quantization algorithm by constructing a standardized evaluation system with unified configuration and reproducible process, and can complete the quantization evaluation of a hundred-billion-parameter model with a single 40G GPU, thereby reducing the hardware requirement and providing a reliable and fair benchmark for the screening and deployment of quantization algorithms.

Description

Technical Field

[0001] This invention relates to a benchmarking method for large models, and also to a corresponding benchmarking system, belonging to the field of machine learning technology. Background Technology

[0002] Large language models (or simply large models) typically have hundreds of billions of parameters. Full-precision storage and inference require enormous computing resources and memory, severely limiting their deployment and widespread adoption on edge devices and low-configuration hardware. Model quantization, as a core compression technology, effectively reduces memory usage and improves inference speed by mapping high-precision parameters to a low-bit-width data format. With the continuous emergence of quantization algorithms (such as AWQ, GPTQ, and Smooth Quant), how to scientifically and fairly evaluate and compare the performance of these algorithms has become a crucial prerequisite for algorithm selection and system optimization.

[0003] Currently, there are the following prominent defects in the performance evaluation and benchmarking of quantization algorithms: (1) Inconsistent configuration leads to unfair comparison: The calibration datasets (such as Pile, C4), data preprocessing methods, sequence lengths and other key configurations of different algorithms vary greatly, making it difficult to directly and fairly compare the evaluation results, and users cannot choose the optimal solution based on a unified standard. (2) Single evaluation dimension and lack of systematic analysis: Most existing evaluations only focus on single indicators such as perplexity (PPL) or accuracy of individual downstream tasks, and lack cross-validation and in-depth analysis of multiple dimensions such as the sensitivity of algorithm calibration data, the combined effect of different strategies (transformation, pruning, reconstruction), and adaptability to different data formats (integer / floating point). (3) High resource consumption and poor reproducibility in the testing process: Mainstream evaluation tools often rely on multiple high-end GPUs, which have high hardware thresholds, and the testing process and results are difficult to reproduce stably, which is not conducive to rapid iteration and verification in resource-constrained development environments. Therefore, how to build a benchmarking system with standardized configuration, comprehensive evaluation dimensions, reproducibility and low hardware resource requirements has become a technical problem that needs to be solved to promote the transparency and standardization of large model quantization technology. Summary of the Invention

[0004] The primary technical problem to be solved by this invention is to provide a benchmarking method for large models.

[0005] Another technical problem that this invention aims to solve is to provide a benchmarking system for large models.

[0006] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution: According to a first aspect of the present invention, a benchmarking method for large models is provided, comprising the following steps: The new algorithm performs policy classification and quantifies policy parameter configuration. The policy labels used for policy classification include transformation, pruning, reconstruction, or hybridization. On a unified hardware and software testing platform, consistency testing is performed using unified calibration data. The calibration data is selected as a dataset that satisfies token distribution consistency with the real business inference set of the application scenario of the new algorithm. First, the baseline perplexity PPL0 and the downstream task average accuracy Acc0 are obtained using the calibration data. Then, a dataset with a KL divergence greater than the calibration data is used for calibration set retesting to obtain the baseline perplexity PPL1 and the downstream task average accuracy Acc1. The evaluation is then based on the admission criteria (PPL1 - PPL0) / PPL0 ≤ PPL preset value and (Acc0 - Acc1) / Acc0 ≤ Acc preset value. If the admission criteria are met, the subsequent steps are performed; otherwise, the evaluation is terminated and the algorithm is judged to be calibration sensitive. Based on the quantization symmetry declared by the new algorithm, a forced matching pruning method is used to perform pruning semantic alignment, wherein symmetric quantization matches symmetric pruning and asymmetric quantization matches asymmetric pruning. The new algorithm is used as a single policy module and concatenated with existing policy modules at order 2-3 to obtain a combinatorial chain. The new algorithm and / or the combined chain are subjected to multidimensional benchmarking, which includes testing in at least one of the following dimensions: calibration data dimension, algorithm strategy dimension, and data format dimension. Based on preset performance indicators, the new algorithm and / or the combined chain are rated or filtered for performance.

[0007] Preferably, the performance rating or screening process generates a comprehensive score, wherein the comprehensive score = calibration data sensitivity score + performance rating score; and the new algorithm and / or the combined chain are rated based on the comprehensive score.

[0008] Preferably, the preset performance indicators include: 8-bit weighted activation quantization decoding speedup ≥ 8-bit decoding speedup target value; 4-bit weighted quantization decoding speedup ≥ 4-bit decoding speedup target value; The performance degradation of the int4 KV-cache code generation task is less than or equal to the threshold for point drops. If the new algorithm or the combinatorial chain satisfies all the performance metrics, its comprehensive score is calculated; if any metric is not satisfied, filtering is performed.

[0009] Preferably, the unified hardware and software testing platform includes using a fixed model of GPU, memory capacity and number of computing cores, installing a unified inference backend framework and loading the weights of the quantized model to be tested, and setting a test benchmark dataset; the test benchmark dataset includes a calibration dataset and a downstream task dataset, wherein the calibration data has a uniform data volume and sequence length, and has undergone unified word segmentation and standardization processing.

[0010] Preferably, the cropping matching rules used for the cropping semantic alignment include: Symmetric clipping algorithms are compatible with symmetric quantization, while asymmetric clipping algorithms are compatible with asymmetric quantization. For all transformation strategy algorithms, symmetric quantization is used. Specifically, scaling transformation algorithms use group-based symmetric quantization, while rotation transformation algorithms use channel-based symmetric quantization.

[0011] Preferably, the test of the calibration data dimension is based on different calibration data, comparing the impact of each calibration data on the quantization accuracy of different quantization algorithms or different combinations of algorithms; The algorithm strategy dimension test involves testing the quantization effect of transformation, pruning, reconstruction, and combination strategies respectively, and analyzing the correlation between kurtosis variation and quantization error. The data format dimension test compares the performance differences between integer quantization and floating-point quantization under different bit widths and model types, clarifying the applicable scenarios for each format.

[0012] Preferably, when performing quantization processing on the hardware and software testing platform, the following sub-tensor block partitioning rule is used for resource-constrained quantization: The maximum tensor size for a single processing operation is determined based on the device's memory capacity. Based on the characteristics of the model layer structure, the weight tensor and activation value tensor are divided into non-overlapping sub-tensor blocks according to the channel dimension. The size of each sub-tensor block meets the requirements of device memory capacity and computational parallelism.

[0013] Preferably, the strategy combination includes the step of forming candidate strategies: Select a single algorithm from the transform pool and match the corresponding pruning and reconstruction strategies according to the pruning matching rules to form a quantization strategy; The perplexity and / or accuracy of the model are calculated using quantitative standard data. If the elimination criteria are met, the entire quantitative strategy group is eliminated; otherwise, it proceeds to the next stage. The elimination criteria refer to the increase in model perplexity reaching the PPL preset value and / or the decrease in accuracy reaching the Acc preset value. If all the quantization strategies concatenated by a single algorithm in the transform pool meet the elimination condition, then at least two algorithms are selected from the transform pool and concatenated. Then, the corresponding pruning and reconstruction strategies are matched according to the pruning matching rules to form a new quantization strategy, and it is determined whether the elimination condition is met.

[0014] Preferably, the quantization strategy parameter configuration includes at least one of the following: Quantization bit width: Supports adjustable from 2 to 8 bits; Quantization type: Integer quantization or floating-point quantization; Group size: 64 or 128; Dynamic / static quantization mode: adaptively selected based on the distribution of model activation values; Supports post-training quantization; The output format is compatible with LightLLM, TRT-LLM, or vLLM backends; Configure calibration data selection rules: Based on the token distribution consistency principle with KL divergence less than 1.0, select the data with the highest matching degree from the candidate calibration dataset as the calibration dataset.

[0015] According to a second aspect of the present invention, a benchmarking system for large models is provided, comprising a processor and a memory; wherein the memory is coupled to the processor and is used to store a computer program, which, when executed by the processor, enables the processor to implement the aforementioned benchmarking method for large models.

[0016] Compared with existing technologies, this invention effectively solves the technical problems of chaotic configuration, unfair comparison, single evaluation dimension, and high hardware threshold in the evaluation of large-scale model quantization algorithms by constructing a standardized benchmark testing system with unified configuration and reproducible process. Specifically, this method ensures the fairness of test conditions and the comparability of results by unifying the software and hardware testing platform and calibration dataset; it filters out unstable algorithms from the source and improves the reliability of evaluation by introducing calibration data admission testing based on token distribution consistency and pruning semantic alignment rules; it systematically explores and evaluates the synergistic effects and applicable scenarios of different quantization strategies by designing a modular strategy combination mechanism and multi-dimensional cross-validation covering calibration data, algorithm strategies, and data formats; furthermore, it significantly reduces the dependence on high-end hardware by using resource-constrained quantization techniques such as sub-tensor block partitioning, enabling the quantization evaluation of models with hundreds of billions of parameters to be completed in a resource-limited environment. Finally, this solution provides a complete, efficient, and reproducible technical path for the standardized evaluation, screening, and deployment of large-scale model quantization algorithms. Attached Figure Description

[0017] Figure 1A logical framework diagram of a benchmark testing method for large models provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a benchmark testing system for large models provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical content of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0019] The benchmarking method for large models provided in this invention is based on a quantization compression method for large models. Together, they constitute a complete quantization compression toolchain, aiming to provide a fast, stable, and repeatable evaluation system for systematically comparing and optimizing the performance of different quantization algorithms. Figure 1 As shown, this quantization compression method constructs a closed-loop iterative system from algorithm standardization, execution decision-making, effect evaluation to model deployment through five stages: algorithm integration and standardization, resource-constrained quantization, quantization algorithm optimization and screening, benchmark construction, and multi-backend inference adaptation. Specifically, the first three stages (algorithm integration and standardization, resource-constrained quantization, and quantization algorithm optimization and screening) are used to generate quantization strategies and initially select the optimal ones; while the latter two stages (benchmark construction and multi-backend inference adaptation) are specifically used to conduct systematic effect evaluation and deployment verification of the selected strategies. This invention adopts a modular design to improve system flexibility and, through a data-driven continuous optimization mechanism, achieves full-scene adaptation of various large models—including but not limited to language models (such as GPT series, LLaMA, ChatGLM, PaLM, etc.), multimodal models (such as CLIP, Stable Diffusion's text encoder, Visual Transformer (ViT), BLIP, Flamingo, etc.), and audio models (such as Whisper, AudioLM, MusicGen, etc.)—under different hardware and algorithm configurations.

[0020] First Embodiment As previously mentioned, the quantization compression method for large models consists of five stages: algorithm integration and standardization (first stage), resource-constrained quantization (second stage), quantization algorithm optimization and selection (third stage), multi-dimensional performance testing and selection (fourth stage), and multi-backend inference adaptation (fifth stage), in order to ultimately obtain the optimal quantization configuration.

[0021] In the algorithm integration and standardization phase, a unified multi-model quantization process is achieved by integrating various types of quantization algorithms. Specifically, a quantization algorithm integration framework is constructed, integrating 16 commonly used quantization algorithms, including weight-only quantization, weight-activation quantization, and mixed precision quantization. This framework maps the quantization requirements of Large Language Models (LLM), Visual-Language Models (VLM), and Hybrid Expert Models (MoE) into a standardized, low-resource compression process, and defines quantization parameters, calibration data configuration specifications, and target model output formats.

[0022] The quantization parameters include quantization bit width, quantization type, group size, and dynamic / static quantization mode. In this embodiment, the quantization bit width is adjustable from 2 to 8 bits; the quantization type includes integer quantization and floating-point quantization; the group size supports configurations such as 64 and 128; and the dynamic / static quantization mode can be adaptively selected according to the model type.

[0023] The calibration data configuration specifications include: a uniform set of 128 calibration data entries, a uniform sequence length of 512, text segmentation and standardization as the preprocessing method, and token distribution consistency measured by KL divergence (D_KL), with a lower KL divergence value indicating a higher degree of matching between the distributions of the calibration and test data. The target model output format is set to a quantized weight format compatible with multiple backends.

[0024] To elaborate further, the aforementioned 16 quantization algorithms include AWQ, GPTQ, SmoothQuant, OmniQuant, QuaRot, etc.; the supported large language models include LLaMA2, LLaMA-3 series, etc.; the visual-language models include LLaVA-7B model, etc.; and the hybrid expert models include Mixtral-8x7B model, etc.

[0025] As shown in Table 1, the quantization compression toolchain provided by this invention categorizes the integrated quantization algorithms according to their implementation methods. The quantization bit width supports adjustable 2-8 bits, with 2-3 bits suitable for ultra-low-cost deployment scenarios and 4-8 bits suitable for precision-priority scenarios. Regarding quantization type, integer quantization is suitable for general-purpose hardware, while floating-point quantization is suitable for long-tailed distributed data scenarios. Group size supports 64 and 128 configurations. Dynamic quantization is suitable for models with large fluctuations in activation value ranges, while static quantization is suitable for scenarios prioritizing inference speed. These algorithms can be categorized according to their core strategies into transformation-based algorithms, pruning-based algorithms, and reconstruction-based algorithms.

[0026] Table 1-1 Classification of Quantization Algorithms Based on Transformation Strategies Table 1-2 Classification of Quantization Algorithms Based on Pruning Strategies Table 1-3 Classification of Quantization Algorithms Based on Reconstruction Strategies In the resource-constrained quantization stage, parameter offloading techniques and sub-tensor block partitioning are used to reduce hardware requirements. This embodiment employs offloading techniques for resource-constrained devices, combined with hardware adaptation parameters—including memory capacity, number of computing cores, and computing support types—to divide model weights and activation values ​​into sub-tensor blocks adapted to the device's processing capabilities, thereby completing the quantization calibration and evaluation of models with hundreds of billions of parameters.

[0027] The rules for dividing sub-tensor blocks include: 1) determining the maximum size of the tensor for a single processing based on the device's memory capacity; 2) combining the characteristics of the model layer structure, dividing the weight tensor and activation value tensor into non-overlapping sub-tensor blocks according to the channel dimension, and the size of each sub-tensor block must simultaneously meet the requirements of device memory carrying capacity and computational parallelism.

[0028] Specifically, when dividing the model into sub-tensor blocks, the model weights and activation values ​​are divided into sub-tensor blocks adapted to the processing power of a single 40GB A100 GPU, based on hardware adaptation parameters (device memory capacity, number of computing cores, and computing support type). For example, for the OPT-175B model (weights occupy approximately 350GB), the model layers are split into sub-tensor blocks and loaded sequentially into the GPU memory for quantization calibration using inter-layer offloading technology, thereby completing the quantization parameter calculation and model evaluation.

[0029] This embodiment dynamically determines the granularity of sub-tensor block partitioning, loading order, whether channel-level partitioning is supported, and whether intermediate results need to be cached based on the algorithm category. Specifically, it includes: For transformation strategy algorithms (such as AWQ, SmoothQuant, and QuaRot), since intermediate transformation results (such as scaling factors and rotation matrices) need to be cached, a buffer area should be reserved when dividing sub-tensor blocks, and weight tensors should be loaded first, while activation values ​​can be loaded late.

[0030] For pruning strategy algorithms (such as GPTQ, AWQ, and OmniQuant), the pruning parameters (α, β) need to be determined before the calibration phase. Therefore, when dividing sub-tensor blocks, the entire channel dimension must be loaded at once, and channel-level partitioning is not supported; otherwise, the pruning boundary cannot be calculated.

[0031] For reconstruction strategy algorithms, the K-FAC method (such as GPTQ) requires Hessian matrix calculation, which has high peak memory requirements. Therefore, it is necessary to enable the inter-layer unloading mechanism and divide the tensor into sub-tensor blocks according to the layer granularity, and cross-layer parallelism is not supported.

[0032] The optimization and screening stage of the quantization algorithm includes three steps: 1) Based on the calibration data, the quantization calibration data that meets the token distribution consistency is initially screened out, that is, the dataset with the smallest KL divergence between the model and the actual test scenario dataset is selected (referred to as "selecting the smallest KL"). 2) Select quantization algorithms that meet the cropping matching rules (called cropping alignment filtering); 3) Algorithm combinations are performed among the selected algorithms to form candidate strategies. This embodiment improves quantization accuracy by combining calibration data characteristics (i.e., which type of algorithm it belongs to among transformation, pruning, and reconstruction) with pruning matching rules.

[0033] After completing the screening in step 1, in step 2, a calibration-quantization-inference process is performed on all 16 algorithms using the dataset with the smallest KL divergence (e.g., the WikiText2 dataset), and then the dataset with the larger KL divergence is used for retesting.

[0034] If the perplexity level (PPL) obtained from the retest increases by more than the preset PPL value (e.g., >20%) compared to the previous test, the algorithm is deemed "calibration sensitive" and is discarded. In this embodiment, the baseline PPL is first obtained by calibration and quantization inference using the WikiText2 dataset with a KL divergence <1.0. Then, after switching to a dataset with a larger D_KL (e.g., the C4 or Pile dataset), if the PPL increase reaches the preset PPL value of 20%, the quantization algorithm is discarded. The C4 / Pile dataset is larger in scale, has more noise, and covers a wider range of topics than WikiText2, representing the data diversity that the model may encounter in real-world deployments. Therefore, it is used to test the generalization ability and robustness of the algorithm.

[0035] Here is a brief explanation of the perplexity (PPL) calculation process using the WikiText2 dataset as an example, including the following steps: ① Calibration using WikiText2 to statistically analyze the activation range; ② Post-training quantization; ③ Perplexity inference on the same distribution test set to obtain the PPL value.

[0036] As mentioned earlier, quantization algorithm strategies can be categorized into transformation strategies, pruning strategies, and reconstruction strategies. Transformation strategies include scaling-based transformations (such as AWQ, SmoothQuant, OS+, and OmniQuant) and rotation-based transformations (such as QuaRot). Pruning strategies include symmetric pruning and asymmetric pruning. Reconstruction strategies employ iterative compensation methods based on the Hessian matrix, with GPTQ being a representative example.

[0037] The cropping and matching rules used in this embodiment specifically include: 1) Symmetric clipping algorithms must be compatible with symmetric quantization, and asymmetric clipping algorithms must be compatible with asymmetric quantization; 2) For transformation strategy algorithms (whether scaling or rotation), symmetric quantization is used. Scaling transformation algorithms use group-based symmetric quantization, with group size supporting 64 or 128; rotation transformation algorithms use channel-based symmetric quantization.

[0038] The main operation steps of the reconstruction strategy algorithm GPTQ are as follows: 1) Weighted block partitioning; 2) Cutting; 3) Calculate the Hessian matrix; 4) Quantification; 5) Calculate the quantization error; 6) Calculate Hessian compensation; 7) Repeat the above process until all blocks in this layer have been processed.

[0039] In other words, the GPTQ algorithm must perform a clipping operation before compensating for quantization errors. Therefore, if the clipping step selects symmetric clipping (i.e., α = –β), the subsequent quantization step must match symmetric quantization (i.e., zero point z = 0); if the clipping step selects asymmetric clipping (α ≠ –β), the quantization step must match asymmetric quantization (z ≠ 0).

[0040] Taking the pruning strategy as following the rule of "symmetric pruning matches symmetric quantization, and asymmetric pruning matches asymmetric quantization" as an example, in the 2-bit quantization scenario, the AWQ algorithm using asymmetric pruning can improve the accuracy by 42.47% compared to the AWQ algorithm using symmetric pruning.

[0041] By combining transformation, pruning, and reconstruction strategies, the optimal quantization configuration can be selected. For example, the combination of QuaRot and GPTQ can achieve a 28.94% accuracy improvement in 3-bit quantization.

[0042] After completing the initial strategy screening and pruning alignment screening, the remaining algorithms (e.g., 6-8) are categorized according to "atomic strategies" to form a transformation pool, a pruning pool, and a reconstruction pool: The transform pool includes algorithms such as QuaRot, AWQ-A, and SmoothQuant-S; The cutting pool includes symmetrical cutting and asymmetrical cutting; The reconstruction pool mainly includes GPTQ and its variants.

[0043] The selection process first chooses a single algorithm from the transform pool and matches it with corresponding pruning and reconstruction strategies according to pruning matching rules, thus forming a quantized strategy chain. Then, the perplexity and accuracy of the strategy are evaluated using a quantized standard dataset (such as WikiText2). If the perplexity increases by more than 20% or the accuracy decreases by more than 5%, the strategy is deemed unqualified, the entire group is eliminated, and no further combinations are made.

[0044] If a strategy passes the evaluation, it is retained and proceeds to the next stage. If none of the strategies generated by a single algorithm in the transformation pool meet the requirements, two or more algorithms are selected from the transformation pool, concatenated, and then matched with the corresponding pruning and reconstruction strategies to form a new strategy, which is then re-evaluated for perplexity and accuracy.

[0045] After the above screening, only 1 to 3 "transform-prune-reconstruction" strategy chains can enter the fourth stage—the multi-dimensional performance test. Compared with conventional methods that require testing all 16 algorithms in the fourth stage, this invention only requires tens of minutes on a single 40GB GPU to complete the strategy search, significantly improving deployment efficiency.

[0046] In the fourth stage, a multi-dimensional quantitative benchmark testing system is constructed to systematically screen candidate strategies and output the final optimal quantitative configuration. This system conducts a fair comparison of the quantitative performance of various quantitative configurations from three dimensions: calibration data, algorithm strategy, and data format. The performance indicators used include one or more of the following: perplexity, kurtosis, quantization error, downstream task accuracy, and inference speed.

[0047] Specific testing dimensions include: I. Calibration Data Dimensions: WikiText2, C4, Pile (val), and custom data were selected as calibration sets. By comparing the impact of different calibration data on the accuracy of quantization algorithms or combinations, the effects of token distribution consistency and intra-sentence logic on quantization performance were revealed.

[0048] II. Algorithm Strategy Dimension: The quantization effects of transformation, pruning, reconstruction, and their combinations were tested separately, and the correlation between kurtosis variation and quantization error was analyzed. For example, scaling-based transformations typically increase weight kurtosis, while rotation-based transformations effectively reduce outliers.

[0049] III. Data Format Dimension: Compare the performance differences between integer quantization and floating-point quantization under different bit widths and model types to clarify the applicable scenarios for each format. For example, under w4a4 configuration, floating-point quantization usually has a significant advantage over integer quantization; while in high bit width (≥4 bits) weighted quantization scenarios, integer quantization is more efficient.

[0050] This evaluation system measures language generation fluency through perplexity, assesses model inference accuracy through downstream tasks such as MMLU, ARC-e, and BoolQ, and evaluates deployment efficiency through inference speed testing. The final validation requires that the quantized model accuracy loss not exceed 5%, while the inference speed improvement is no less than 40%.

[0051] In the fifth stage, based on optimal quantization configuration, it is compatible with multiple inference backends such as LightLLM, TRT-LLM, vLLM, and llama.cpp, and employs software pipeline technology to achieve parallel processing of data loading and computation, thereby hiding the scheduling latency from DRAM to the global buffer and improving overall inference efficiency. For example, 8-bit weight-activation quantization can achieve a 20%–30% improvement in pre-filling speed and a 40%–60% improvement in decoding speed on the A100 GPU; 4-bit weight-only quantization can also achieve a 40%–60% improvement in decoding speed. This software pipeline technology preferentially uses a double-buffering mechanism to execute the three stages of data loading, computation processing, and result storage in parallel, and its scheduling latency hiding capability can reach more than 80% of the data transmission time.

[0052] Second Embodiment Based on the first embodiment, the second embodiment of the present invention provides a quantization compression method for large models, which can be used for the deployment of large models, and specifically includes the following steps: S10: Configuring the hardware and software environment required for quantization Install the quantization compression toolchain, prepare the model file to be quantized (e.g., LLaMA, VLM, MoE), determine the target hardware platform (including GPU model, memory capacity, number of computing cores), and set a unified quantization parameter configuration specification. The quantization parameters include: quantization bit width (2-8 bits), quantization type (integer quantization / floating-point quantization), group size (64 / 128), and dynamic / static quantization mode.

[0053] S11: Selecting calibration data based on token distribution consistency Based on the principle of token distribution consistency, calibration data is selected from a pre-set candidate calibration dataset (including WikiText2, C4, Pile-val, and custom data). Specifically, the KL divergence (D_KL) between each candidate set and the actual business inference set is calculated, and the dataset with the smallest KL divergence (ideally less than 1.0, otherwise the smallest) is selected as the calibration data. The calibration data volume is uniformly 128 records, the sequence length is uniformly 512, and uniform word segmentation and standardization processing is performed.

[0054] S12: Algorithm strategy combination for matching clipping rules Based on the model structure and application scenario, a combination of quantization algorithm strategies that conform to the pruning and matching rules is selected and formed.

[0055] S13: Strategy Combinations and Resource-Constrained Quantization Based on the initial screening, transformation, pruning, and reconstruction strategies are combined to form a variety of quantization strategy combinations. Through a quantization calibration process based on selected calibration data and real business inference sets, the feasibility of each strategy in low-resource environments is evaluated. Sub-tensor blocks are then divided in conjunction with hardware adaptation parameters (such as memory capacity and computing core support) to achieve quantization calibration of a model with hundreds of billions of parameters.

[0056] S14: Multi-dimensional Benchmarking Evaluation A multi-dimensional quantitative benchmarking system is used to systematically evaluate each strategy combination. The evaluation includes: 1) Calibration data dimension: Compare the impact of different calibration data on the perplexity (PPL) of the quantized model; 2) Algorithm strategy dimension: Analyze the mechanism by which transformation, pruning, and reconstruction strategies affect kurtosis and quantization error; 3) Data format dimension: Compare the performance differences between integer quantization and floating-point quantization at different bit widths.

[0057] The evaluation metrics used may include one or more of the following: PPL, kurtosis, quantization error, downstream task accuracy (such as MMLU, ARC-e, BoolQ), and inference speed (prefilling speed / decoding speed).

[0058] S15: Result Filtering and Optimal Solution Output The test results are filtered based on the following preset conditions for quantification effect: 1) The increase in PPL after quantification shall not exceed 20%; 2) The average accuracy of downstream tasks decreases by no more than 5%; 3) Reasoning speed is increased by at least 40%.

[0059] The final output is the optimal quantization configuration scheme that meets all conditions, including: the optimal policy chain (e.g., QuaRot→GPTQ), the optimal calibration set (e.g., WikiText2), the optimal bit width / format (e.g., w3a16g128-floating point), the corresponding quantized model file, and the weight format compatible with multiple backends (supporting LightLLM, TRT-LLM, and vLLM loading).

[0060] Compared to existing technologies, this invention upgrades large language model quantization from "manual parameter tuning of single-point algorithms" to an industrial-grade automated optimal quantization process driven by a three-dimensional "strategy-resource-data" architecture. By integrating multiple quantization algorithms (e.g., 16 mainstream quantization algorithms) during the algorithm integration and standardization phase, a unified multi-model quantization process is achieved. In the resource-constrained quantization phase, parameter offloading technology enables the quantization of models with hundreds of billions of parameters to be completed with only a single 40G GPU, significantly reducing the hardware requirements for deployment while maintaining accuracy.

[0061] Third Embodiment Based on the first embodiment, the third embodiment of the present invention provides a benchmarking method for large models, used to perform standardized performance evaluation of a newly proposed quantization algorithm. This benchmarking method includes at least the following steps: S20: Strategy Classification and Parameter Configuration Before the evaluation begins, a unified hardware and software testing platform is configured, including a fixed model of GPU, memory capacity, and number of computing cores. A unified inference backend framework (such as LightLLM, TRT-LLM, vLLM) is installed, and the weights of the quantized model to be tested are loaded. Simultaneously, a benchmark dataset is set, including a calibration dataset and a downstream task dataset: the calibration data is uniformly 128 records, with a sequence length of 512, and uniform word segmentation and standardization processing is performed; the test dataset at least covers WikiText2, C4, Pile (val), etc., and the downstream task dataset includes MMLU, ARC-e, BoolQ, HellaSwag, PIQA, etc.

[0062] The new algorithm to be evaluated is classified into policies, and the policy labels used are one of transformation, pruning, reconstruction or hybridization.

[0063] Configure quantization parameters for this new algorithm, specifically including: 1) Quantization bit width: Supports adjustable from 2 to 8 bits; 2) Quantization type: Integer quantization or floating-point quantization; 3) Group size: 64 or 128; 4) Dynamic / static quantization mode: Adaptively selected based on the distribution of model activation values; 5) Supports post-training quantization (PTQ); 6) The output format must be compatible with LightLLM, TRT-LLM, and vLLM backends; 7) Calibration data selection rules: Based on the token distribution consistency principle that the KL divergence (D_KL) is less than 1.0, the data with the highest matching degree is selected from the candidate calibration dataset as the calibration dataset.

[0064] S21: Calibration Data Consistency Test We uniformly used WikiText2-128 (sequence length 512) as the benchmark calibration dataset, and calculated the benchmark perplexity PPL0 of the model after quantization by the new algorithm on the same distribution test set and the average accuracy Acc0 of downstream tasks.

[0065] Next, a calibration set retest was performed using a dataset with a larger KL divergence (e.g., C4 or Pile(val)) to obtain PPL1 and the downstream task average accuracy Acc1.

[0066] Judgment based on admission criteria: (PPL1-PPL0) / PPL0≤PPL preset value (e.g., 20%), and (Acc0-Acc1) / Acc0≤Acc preset value (e.g., 5%).

[0067] If the admission criteria are met, proceed to the next step; otherwise, it is determined to be "calibration sensitive" and the evaluation is terminated.

[0068] S22: Trimming Semantic Alignment Based on the quantization symmetry declared by the new algorithm, the forced matching pruning method is: symmetric quantization must be matched with symmetric pruning, and asymmetric quantization must be matched with asymmetric pruning.

[0069] If the pruning method used by the new algorithm does not match its declared quantization symmetry, the evaluation will terminate.

[0070] S23: Strategy Combination Screening The new algorithm is used as a single policy module and combined with existing policy modules in a 2-3 order to form a combinatorial chain.

[0071] Quickly scan each combination chain, return the combination PPL and combination Acc measured in step S21, and calculate: PPL change rate = |combined PPL - PPL0| / PPL0, Acc decrease = (combined Acc - Acc0) / Acc0.

[0072] If at least one composite chain satisfies the following: PPL change rate ≤ PPL preset value, and Acc decrease ≤ Acc preset value If so, proceed to the next step; otherwise, terminate the evaluation.

[0073] S24: Multidimensional Benchmarking For the new algorithms and combinatorial chains that passed the aforementioned screening, a comprehensive multi-dimensional evaluation was conducted using downstream task datasets, specifically including: Calibration data dimension: Use N (e.g., 4, but not limited to) calibration sets with increasing KL divergence, and output the PPL decrease curve corresponding to each decrease of 0.5 in KL divergence; Algorithm strategy dimension: Compare the differences between single strategies and combined strategies in terms of average accuracy across multiple tasks, such as kurtosis variation, quantization error, and MMLU. Data format dimension: Analyze the performance applicable range of the same composite chain under different data formats such as integer and floating point, in bit width configurations such as w4a4 and w3a16.

[0074] S25: Performance Rating and Filtration New algorithms or combinatorial chains are rated or filtered based on the following preset performance metrics: 8-bit weighted activation quantization improves decoding speed by ≥8-bit decoding speedup (e.g., 40%). Decoding speed improvement of 4-bit weighted quantization is ≥ 4-bit decoding speed improvement target value (e.g., 40%). The performance degradation of int4 KV-cache on code generation tasks (HumanEval / MBPP) is ≤ the dropout threshold (e.g., 3%).

[0075] If the new algorithm or combinatorial chain satisfies all of the above indicators, then calculate its comprehensive score: Overall score = Calibration data sensitivity score + Performance rating score.

[0076] The calibration data sensitivity score is calculated based on the rate of change of PPL and Acc in step S21. The performance rating score is based on the comprehensive evaluation of the achievement of various preset performance indicators in step S25.

[0077] The algorithm is rated A, B, or C based on a comprehensive score, with A being designated "Recommended Deployment". Alternatively, new algorithms can be ranked against existing strategies based on their comprehensive scores, and "Recommended Deployment" suggestions can be given for new algorithms that score higher than existing ones. If any indicator fails to meet the standard, the algorithm is filtered out.

[0078] S26: Output a standardized evaluation report The report template should include: 1) Algorithm strategy classification, calibration sensitivity conclusions, and clipping alignment conclusions; 2) Baseline raw data and variation curves in three dimensions: calibration data, algorithm strategy, and data format; 3) Optimal combinatorial chain, optimal calibration set, optimal bit width / format; 4) Verification results of multiple backend inference (including screenshots of inference speed and memory usage under LightLLM, TRT-LLM, and vLLM); 5) Overall rating and usage recommendations.

[0079] Through the above process, a standardized performance evaluation of the new quantization algorithm is completed in a reproducible, portable, and comparable manner. This method requires only a single 40G GPU to complete the quantization evaluation of a model with hundreds of billions of parameters, significantly reducing the hardware threshold for large-scale model performance evaluation while maintaining accuracy.

[0080] Fourth embodiment like Figure 2As shown, based on the first embodiment described above, the fourth embodiment of the present invention further provides a benchmarking system for large models. The system includes one or more processors and a memory. The memory is coupled to the processor and is used to store one or more programs. When the programs are executed by the processor, the processor implements the benchmarking method for large models described in the above embodiments.

[0081] The processor controls the overall operation of the system to complete all or part of the steps in the benchmarking method for large models described above. This processor can be a central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), digital signal processing (DSP) chip, etc. The memory stores various types of data to support the operation of the system. This data may include, for example, instructions for any application or method operating on the system, as well as application-related data. The memory can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, etc.

[0082] On the other hand, embodiments of the present invention also provide a computer program product, the computer program product including a computer program, the computer program being stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the above-described benchmark testing method for large models.

[0083] In summary, this invention provides a toolchain for quantization and compression of large language models that supports multiple algorithms, along with corresponding benchmarking methods and systems. This technical solution integrates multiple quantization algorithms (e.g., 16 mainstream quantization algorithms) through the algorithm integration and standardization phase, unifying the quantization process for multiple models. Through the resource-constrained quantization phase, it employs parameter offloading and sub-tensor block partitioning techniques, enabling the quantization of models with hundreds of billions of parameters to be completed with only a single 40G GPU, significantly lowering the hardware threshold. Through the strategy optimization and selection phase, it effectively reduces quantization accuracy loss based on token distribution consistency and pruning matching rules. Through the benchmarking construction phase, it establishes a multi-dimensional fair comparison system, solving the problems of inconsistent configurations and difficulty in direct comparison of existing quantization methods. Through the multi-backend inference adaptation phase, it is compatible with various inference backends and hardware platforms, significantly improving deployment flexibility. Compared to existing quantization solutions, this invention can adapt to various model structures such as LLM, VLM, and MoE, achieving a good balance between accuracy and efficiency, thus providing reliable technical support for the large-scale deployment of large language models.

[0084] This invention upgrades large language model quantization from the traditional "single-point algorithm manual parameter tuning" mode to an industrial-grade, end-to-end optimal quantization process driven by a three-dimensional automatic approach encompassing "strategy-resource-data." It achieves five major technological breakthroughs: quantizing hundreds of billions of parameters on a single 40GB GPU; reproducible benchmark testing based on KL divergence less than 1.0; automatic optimization through strategy combination; automatic format adaptation; and multi-backend latency hiding. This effectively addresses common industry problems such as complex configurations, high hardware barriers, poor scalability, and limited evaluation dimensions. Compared to existing solutions, this invention lowers the hardware threshold for quantization deployment while maintaining model accuracy, and provides a standardized and reproducible evaluation benchmark for quantization strategy selection, thereby significantly improving the practicality and iteration efficiency of large language models on diverse hardware platforms.

[0085] The benchmark testing method and system for large models provided by this invention have been described in detail above. Any obvious modifications made by those skilled in the art without departing from the essence of this invention will constitute an infringement of the patent rights of this invention and will incur corresponding legal liability.

Claims

1. A benchmarking method for large models, characterized in that... Includes the following steps: The new algorithm performs policy classification and quantifies policy parameter configuration. The policy labels used for policy classification include transformation, pruning, reconstruction, or hybridization. On a unified hardware and software testing platform, consistency tests are performed using unified calibration data, wherein the calibration data is selected as a dataset that satisfies token distribution consistency with the real business inference set of the application scenario of the new algorithm. First, the baseline perplexity PPL0 and the downstream task average accuracy Acc0 are obtained using the calibration data. Then, a dataset with a KL divergence greater than the calibration data is used for calibration set retesting to obtain the baseline perplexity PPL1 and the downstream task average accuracy Acc1. The evaluation is then performed based on the admission criteria (PPL1 - PPL0) / PPL0 ≤ PPL preset value and (Acc0 - Acc1) / Acc0 ≤ Acc preset value. If the admission criteria are met, the subsequent steps are performed; otherwise, the evaluation is terminated and the system is determined to be calibration sensitive. Based on the quantization symmetry declared by the new algorithm, a forced matching pruning method is used to perform pruning semantic alignment, wherein symmetric quantization matches symmetric pruning and asymmetric quantization matches asymmetric pruning. The new algorithm is used as a single policy module and concatenated with existing policy modules at order 2-3 to obtain a combinatorial chain. The new algorithm and / or the combined chain are subjected to multidimensional benchmarking, which includes testing in at least one of the following dimensions: calibration data dimension, algorithm strategy dimension, and data format dimension. Based on preset performance indicators, the new algorithm and / or the combined chain are rated or filtered for performance.

2. The benchmarking method for large models as described in claim 1, characterized in that, The performance rating or screening process generates a comprehensive score, which is calculated as: calibration data sensitivity score + performance rating score. The new algorithm and / or the combined chain are rated based on the comprehensive score.

3. The benchmarking method for large models as described in claim 2, characterized in that, The preset performance indicators include: 8-bit weighted activation quantization decoding speedup ≥ 8-bit decoding speedup target value; 4-bit weighted quantization decoding speedup ≥ 4-bit decoding speedup target value; The performance degradation of the int4 KV-cache code generation task is less than or equal to the threshold for point drops. If the new algorithm or the combinatorial chain satisfies all the performance metrics, its comprehensive score is calculated; if any metric is not satisfied, filtering is performed.

4. The benchmarking method for large models as described in claim 1, characterized in that, The unified hardware and software testing platform includes using a fixed model of GPU, memory capacity and number of computing cores, installing a unified inference backend framework and loading the weights of the quantized model to be tested, and setting a test benchmark dataset; the test benchmark dataset includes a calibration dataset and a downstream task dataset, wherein the data volume and sequence length of the calibration data are uniform, and it has undergone unified word segmentation and standardization processing.

5. The benchmarking method for large models as described in claim 1, characterized in that, The cropping matching rules used for the cropping semantic alignment include: Symmetric clipping algorithms are compatible with symmetric quantization, while asymmetric clipping algorithms are compatible with asymmetric quantization. For all transformation strategy algorithms, symmetric quantization is used. Specifically, scaling transformation algorithms use group-based symmetric quantization, while rotation transformation algorithms use channel-based symmetric quantization.

6. The benchmarking method for large models as described in claim 1, characterized in that: The test of the calibration data dimension is based on different calibration data, comparing the impact of each calibration data on the quantization accuracy of different quantization algorithms or different combinations of algorithms; The testing of the algorithm strategy dimension involves testing the quantization effects of transformation, pruning, reconstruction, and combination strategies respectively, and analyzing the correlation between kurtosis variation and quantization error. The data format dimension test compares the performance differences between integer quantization and floating-point quantization under different bit widths and model types, clarifying the applicable scenarios for each format.

7. The benchmarking method for large models as described in claim 1, characterized in that, When performing quantization on the aforementioned hardware and software testing platform, the following sub-tensor block partitioning rules are used for resource-constrained quantization: The maximum tensor size for a single processing operation is determined based on the device's memory capacity. Based on the characteristics of the model layer structure, the weight tensor and activation value tensor are divided into non-overlapping sub-tensor blocks according to the channel dimension. The size of each sub-tensor block meets the requirements of device memory capacity and computational parallelism.

8. The benchmarking method for large models as described in claim 7, characterized in that, The strategy combination includes the step of forming candidate strategies: Select a single algorithm from the transform pool and match the corresponding pruning and reconstruction strategies according to the pruning matching rules to form a quantization strategy; The perplexity and / or accuracy of the model are calculated using quantitative standard data. If the elimination criteria are met, the entire quantitative strategy group is eliminated; otherwise, it proceeds to the next stage. The elimination criteria refer to the increase in model perplexity reaching the PPL preset value and / or the decrease in accuracy reaching the Acc preset value. If all the quantization strategies concatenated by a single algorithm in the transform pool meet the elimination condition, then at least two algorithms are selected from the transform pool and concatenated. Then, the corresponding pruning and reconstruction strategies are matched according to the pruning matching rules to form a new quantization strategy, and it is determined whether the elimination condition is met.

9. The benchmarking method for large models as described in any one of claims 1 to 8, characterized in that, The quantization strategy parameter configuration includes at least one of the following: Quantization bit width: Supports adjustable from 2 to 8 bits; Quantization type: Integer quantization or floating-point quantization; Group size: 64 or 128; Dynamic / static quantization mode: adaptively selected based on the distribution of model activation values; Supports post-training quantization; The output format is compatible with LightLLM, TRT-LLM, or vLLM backends; Configure calibration data selection rules: Based on the token distribution consistency principle with KL divergence less than 1.0, select the data with the highest matching degree from the candidate calibration dataset as the calibration dataset.

10. A benchmarking system for large models, characterized in that... It includes a processor and a memory; wherein the memory is coupled to the processor and is used to store a computer program, which, when executed by the processor, causes the processor to implement the benchmarking method for large models as described in any one of claims 1 to 9.

Images