Quantitative quality evaluation system for large language model

Abstract

The invention discloses a large language model quantitative quality evaluation system, which comprises a resource analysis end, a quantitative execution end and an evaluation analysis end, and is characterized in that the resource analysis end is used for collecting a device data set, determining hardware resource constraints of edge devices, and dividing the sensitivity level of a functional layer to quantification in a large language model to obtain a sensitivity sequence; the quantization execution end is used for establishing a mapping rule and realizing precision grade mapping; balancing quantization precision loss and resource consumption through iterative optimization, determining a final allocation scheme and quantization parameters, constructing a mixed precision quantization operator library, determining a compilation tool chain of edge equipment, and constructing a quantization model of a large language model; the evaluation analysis end tests different tasks so as to evaluate the quality of the quantitative model on the edge device and judge whether the quality reaches the standard or not; and performing precision adjustment iteration and resource cutting optimization of the quantitative model for substandard items in evaluation, monitoring key indexes and storing a system data set.

Description

Technical Field

[0001] This invention relates to the field of large language model evaluation technology, and specifically to a quantitative quality evaluation system for large language models. Background Technology

[0002] With the expansion of Large Language Models (LLMs) applications in edge device scenarios, quantization technology has become a key means to balance model performance and hardware resources. By reducing parameter precision, it reduces memory consumption and improves inference efficiency. Quantization quality assessment, as a core component, can control the precision loss during the quantization process and ensure that the model maintains effective functionality in resource-constrained environments. It is an important support for the edge deployment of large language models and directly affects the reliability of applications and user experience.

[0003] In current large language model quantization technologies, existing evaluation schemes mostly focus on a single dimension and fail to fully consider the hardware characteristics of edge devices and the sensitivity differences of each layer of the model. At the same time, some schemes lack dynamic iterative optimization mechanisms, making it difficult to flexibly adjust the quantization strategy based on the evaluation results. As a result, in edge scenarios with strict resource constraints, the balance between model accuracy and efficiency needs to be further improved, and there is still room for optimization in terms of adaptability and practicality. Summary of the Invention

[0004] The purpose of this invention is to provide a large language model quantitative quality assessment system to solve the problems of existing technologies, such as difficulty in balancing accuracy and efficiency, and the need to improve adaptability and practicality.

[0005] To achieve the above objectives, the present invention employs the following technical solution: A large language model quantitative quality assessment system includes: a resource analysis end, a quantitative execution end, and an assessment analysis end, wherein: The resource analysis end is used to collect device datasets, clarify the hardware resource constraints of edge devices, and classify the sensitivity levels of functional layers in the large language model to quantization to obtain a sensitivity ranking. The quantization execution end establishes mapping rules based on the device dataset and sensitivity ranking to achieve precision level mapping and obtain a preliminary allocation scheme. Based on the preliminary allocation scheme, the final allocation scheme is determined by iteratively optimizing the balance between quantization precision loss and resource consumption. The quantization parameters are determined based on the final allocation scheme. The quantization type and quantization parameters of each functional layer are analyzed, a hybrid precision quantization operator library is constructed, and the compilation toolchain for edge devices is determined, compilation options are configured, and a quantization model for a large language model is constructed. The evaluation and analysis end uses a quantitative model and a standard test set to test different tasks to evaluate the quality of the quantitative model on edge devices and determine whether it meets the standards. For items that do not meet the standards in the evaluation, the accuracy of the quantitative model is adjusted and iterated, and resources are trimmed and optimized. At the same time, key indicators are monitored in real time and the system dataset is stored.

[0006] Furthermore, the resource analysis module includes an edge device demand analysis module; The edge device requirements analysis module includes a hardware resource parameter acquisition unit and a model performance requirement definition unit; wherein: The hardware resource parameter acquisition unit is used to collect device datasets for specified edge devices, including memory data, storage data, and power consumption data; and sends the collected device datasets to the model sensitivity analysis module and the resource optimization iteration module. The model performance requirement definition unit is used to specify the model type of the large language model, and at the same time set the quantization accuracy data loss threshold, edge device inference latency threshold, edge device memory usage threshold, and edge device power consumption threshold, and send them to the quantization quality assessment module of the evaluation and analysis end.

[0007] Furthermore, the resource analysis module also includes a model sensitivity analysis module; The model sensitivity analysis module includes a feature extraction unit and a sensitivity calculation unit, wherein: The feature extraction unit is used to classify the functional layer types of the large language model, extract the parameter scale and computational cost of each functional layer, and mark functional layers with a total layer computational cost greater than a preset proportion as computationally intensive layers. Simultaneously, it collects the data distribution of the input and output tensors of each functional layer and records the data flow direction of each functional layer during inference, generating a large language model layer analysis report, which is then sent to the sensitivity calculation unit. The functional layer types include embedding layers, multi-head attention layers, feedforward network layers, layer normalization layers, and output layers. The functional layer division strictly follows the basic architecture of the large language model and does not disrupt the original inter-layer dependencies of the model. The sensitivity calculation unit, based on the large language model layer analysis report, quantitatively calculates the sensitivity of each functional layer to quantification, classifies sensitivity levels, and obtains a sensitivity ranking.

[0008] Furthermore, the sensitivity calculation unit, based on the large language model layer analysis report, quantitatively calculates the sensitivity of each functional layer to quantization, classifies sensitivity levels, and obtains a sensitivity ranking, including: A1 uses FP32 as the original accuracy benchmark, and the test accuracy set includes INT8, INT4 and FP16; each functional layer is set as the target layer in turn. A2 quantizes the target layer to a specified precision, fixes other functional layers as FP32, and performs isolated quantization tests on the target layer; A3, during isolated quantization testing, input the standard test set and obtain the output feature tensor of the target layer before and after quantization; A4, based on the output feature tensors of the target layer before and after quantization, calculate the sensitivity of each layer and obtain the sensitivity coefficient; Sensitivity coefficient The calculation formula is: ,in, The output feature tensor of the target layer at the original precision FP32. The output feature tensor of the target layer when quantized to a specified precision; A5. Repeat steps A2 to A4 to calculate the sensitivity coefficient of each functional layer at the specified precision. A6 defines a functional layer as follows: if the sensitivity coefficient is greater than or equal to 0.1, it is classified as a high-sensitivity layer; if the sensitivity coefficient is greater than 0.05 and less than 0.1, it is classified as a medium-sensitivity layer; and if the sensitivity coefficient is less than or equal to 0.05, it is classified as a low-sensitivity layer. The recommended accuracy range for each functional layer is also marked to obtain a sensitivity ranking.

[0009] Furthermore, the quantization execution terminal includes a precision allocation module; The hybrid precision allocation module includes a precision level mapping unit and an allocation strategy optimization unit, wherein: The accuracy level mapping unit establishes mapping rules based on the device dataset and sensitivity ranking: FP16 is preferentially allocated to high-sensitivity layers. If the memory usage exceeds the memory usage threshold of the edge device, it is downgraded to INT8 and marked as requiring quality monitoring; INT8 is allocated by default to medium-sensitivity layers. If the computational load exceeds the upper limit of the edge device, it is downgraded to INT4 and the accuracy loss before and after the downgrade is verified; INT4 is allocated to low-sensitivity layers, and parameter sparsity is used to further compress the data to obtain a preliminary allocation scheme. The allocation strategy optimization unit determines the final allocation scheme by iteratively optimizing the balance between quantization accuracy loss and resource consumption based on the initial allocation scheme.

[0010] Furthermore, based on the initial allocation scheme, the allocation strategy optimization unit determines the final allocation scheme by iteratively optimizing the balance between quantization accuracy loss and resource consumption, including: B1, the allocation priority is calculated based on the sensitivity coefficient. Specifically: ; in, The sensitivity coefficient, Use a specified precision for resource consumption rates for the functional layers; B2. For resource overruns in the initial allocation plan, adjust them in descending order of allocation priority, prioritizing the reduction of the precision of the functional layer with the lowest allocation priority; among them, resource overruns include precision data loss exceeding the quantization precision data loss threshold, edge device inference latency exceeding the edge device inference latency threshold, edge device memory usage exceeding the edge device memory usage threshold, and edge device power consumption exceeding the edge device power consumption threshold. B3, for cases where the accuracy loss in the initial allocation scheme exceeds the threshold, the accuracy is improved in ascending order of allocation priority; B4, based on the optimized solutions from B2 and B3, calculate the overall score: ; in, For the overall score, For accuracy loss rate, Resource overspending rate; B5. Repeat steps B2 to B3. After multiple iterations, select the optimized solution with the highest comprehensive score as the final allocation solution.

[0011] Furthermore, the quantization execution terminal also includes a quantization parameter calculation module; The quantization parameter calculation module includes a symmetric quantization parameter calculation unit and an asymmetric quantization parameter calculation unit; The symmetric quantization parameter calculation unit targets the functional layer in the large language model where the data distribution exhibits symmetry. It analyzes the numerical range of the input data, extracts the maximum absolute value of the feature tensor in the input data, and uses this to determine the dynamic range boundary of the input data. Simultaneously, based on the quantization precision, it clarifies the corresponding integer value range. Based on this, it determines the mapping ratio between floating-point data and integers. It randomly selects samples from the input data, and calculates the error between the input data and the recovered data through quantization of floating-point numbers to integers and dequantization of integers to floating-point numbers. If the error is greater than the error threshold, it adjusts the calculation method of the maximum absolute value, including removing extreme outliers, recalculating the ratio, and outputting the quantization precision and scaling factor to obtain the symmetric quantization parameters. The asymmetric quantization parameter calculation unit targets the functional layer where data distribution in a large language model exhibits asymmetry. It analyzes the numerical distribution of the input data, extracts the maximum absolute value of the feature tensor in the input data, and uses this to determine the complete floating-point value range. Based on the quantization precision, it clarifies the corresponding integer value range and determines the total length of the integer range. Simultaneously, it determines the scaling factor and zero point, performs quantization and dequantization operations on the input data, and calculates the average error between the input data and the recovered data. When the error exceeds a preset threshold, it readjusts the value range of the input data, recalculates the scaling factor and zero point, until the preset requirements for the output floating-point number, zero point, and floating-point value range are achieved, thus obtaining the asymmetric quantization parameters.

[0012] Furthermore, the quantization execution terminal also includes a quantization execution and edge adaptation module; The quantization execution and edge adaptation module includes a mixed-precision quantization operator generation unit and an edge device compilation and adaptation unit; The mixed-precision quantization operator generation unit is used to parse the quantization type and quantization parameters of each functional layer based on the final allocation scheme, symmetric quantization parameters, and asymmetric quantization parameters. For the embedding layer in the functional layer, it generates parameter quantization operators, offline quantizing 32-bit floating-point weights into 4-bit integers, and using group compression for storage. For the multi-head attention layer in the functional layer, it generates activation quantization operators, quantizing 32-bit floating-point inputs into 16-bit floating-point inputs in real time during inference. For the feedforward network layer in the functional layer, it generates quantization calculation fusion operators, fusing quantization, matrix multiplication, and dequantization. For the layer normalization layer in the functional layer, it generates numerical range constraint operators. Based on these, a mixed-precision quantization operator library is obtained. The edge device compilation adaptation unit is used to determine the compilation toolchain for edge devices, configure compilation options, perform hardware instruction mapping for the mixed-precision quantization operator library, optimize the memory structure of large language models, link quantization operators and model weights into an executable file, generate the model inference entry function, and reduce inference time through operator scheduling optimization, thereby obtaining the quantization model of the large language model.

[0013] Furthermore, the evaluation and analysis module includes a quantitative quality assessment module; The quantitative quality assessment module includes a quantitative accuracy loss assessment unit and an edge resource consumption assessment unit; The quantization accuracy loss assessment unit is based on a quantization model and a standard test set. The tests include three types of tasks: text generation, semantic understanding, and logical reasoning. The unit tests the benchmark performance of the large language model on a PC and the performance of the quantization model on an edge device. It records the BLEU score, semantic understanding accuracy, and perplexity on the PC and edge devices. The unit also compares the BLEU loss rate, semantic understanding accuracy loss rate, and perplexity increase rate on the edge devices with the PC test to obtain quantization accuracy data. Content with substandard quantization accuracy data is sent to the resource optimization iteration module. The edge resource consumption assessment unit is used to assess the binding force of the quantitative model on resource compliance on edge devices.

[0014] Furthermore, the edge resource consumption assessment unit is used to evaluate the binding force of the quantitative model on resource compliance on edge devices, including: C1, calculate the sum of the memory usage of the quantization model parameters and the memory usage of the inference cache, divide it by the total memory of the edge devices, and obtain the memory usage rate; C2, calculate the total energy consumption of the quantization model inference process, divide it by the total inference time of the quantization model, and obtain the average power consumption; C3, calculate the number of test samples that meet the edge device inference latency threshold, divide by the total number of test samples to obtain the latency compliance rate; C4 compares the memory usage rate, average power consumption, and latency compliance rate with preset edge device memory usage thresholds, edge device power consumption thresholds, and edge device inference latency thresholds, and marks items exceeding the thresholds as non-compliant items, sending them to the resource optimization iteration module.

[0015] Furthermore, the evaluation and analysis module also includes a resource optimization and iteration module; The resource optimization iteration module includes a precision adjustment iteration unit and a resource trimming optimization unit; The precision adjustment iteration unit, based on quantization precision data, examines the original precision allocation of the three functional layers with the highest proportion of quantization precision data loss, improves the precision of the high-sensitivity layers, and recalculates the scaling factor and zero point of the high-sensitivity layers to generate an optimized quantization model. It then retests and calculates the BLEU loss rate, semantic understanding accuracy loss rate, and perplexity increase rate on an edge device. If the quantization precision of the optimized quantization model is still not up to standard when tested on an edge device, a dynamic quantization method is used to adjust the scaling factor in real time until the quantization precision data of the optimized quantization model meets the standard. The resource pruning and optimization unit performs pruning and optimization based on the non-compliant items output by the edge resource consumption assessment unit; when the memory utilization rate is greater than the edge device memory utilization threshold, parameters are pruned for low-sensitivity layers; when the latency compliance rate is less than the preset edge device inference latency threshold, operator fusion is used for computationally intensive layers to reduce the number of data transfers; when the average power consumption is greater than the edge device power consumption threshold, the CPU or GPU frequency is adjusted; thus, the resource optimization scheme is obtained and sent to the quantitative quality assessment module for re-evaluation.

[0016] Furthermore, the evaluation and analysis module also includes a data management module; The data management module is used to monitor key metrics and store system datasets; key metrics include sensitivity coefficient, scaling factor, zero point, and inference latency; system datasets include device datasets, sensitivity ranking, final allocation schemes, and quantization models.

[0017] Compared with the prior art, the present invention has the following technical features: 1. This invention clarifies the actual limitations of equipment in terms of computing, storage, and energy consumption by setting up a resource analysis terminal, as well as the specific requirements for model inference quality and efficiency. At the same time, it analyzes the sensitivity differences of different components within the model to accuracy adjustments, identifies the parts that are significantly sensitive to accuracy changes and the parts that are relatively insensitive, and provides a basis for the subsequent formulation of quantification strategies. This ensures that the strategy can not only adapt to the actual capabilities of the equipment, but also retain the core performance of the model to the maximum extent under resource constraints, thus forming the basic support from requirements to solution design.

[0018] 2. This invention determines a reasonable accuracy standard by setting the sensitivity differences of the quantization execution end to accuracy and the limitations of device resources. It ensures the accuracy of the quantization process by calculating the necessary conversion parameters, and then transforms these settings into an actual model that can run efficiently on edge devices. At the same time, it solves the hardware adaptation problem, ensuring that the quantized model can meet the performance requirements and adapt to the resource constraints of edge devices. It achieves a dynamic balance between accuracy loss and resource consumption, providing a reliable foundation for subsequent quality assessment and optimization.

[0019] 3. This invention examines the actual performance after quantification from multiple dimensions by setting up an evaluation and analysis end. It considers both the degree of data loss in quantification accuracy and the resource consumption to determine whether it meets the preset standards. For cases that do not meet the standards, it iterative improvements are made by adjusting relevant strategies, seeking a balance between maintaining quality and resource consumption, and continuously optimizing until all indicators meet the requirements. This ensures that it can play a stable and efficient role in the target environment, forming a closed loop from evaluation to optimization, and ensuring that the overall effect meets expectations. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall architecture of the large language model quantitative quality assessment system of the present invention; Figure 2 This is a schematic diagram of the resource analysis terminal architecture of the present invention; Figure 3 This is a schematic diagram of the quantitative execution terminal architecture of the present invention; Figure 4 This is a schematic diagram of the evaluation and analysis terminal architecture of the present invention; Figure 5 This is a flowchart illustrating the process of using the present invention for quantitative quality assessment of large language models. Detailed Implementation

[0021] This invention provides a large language model quantization quality assessment system. The server-side device deployed by this assessment system may consist of one or more devices. The assessment system can be implemented as a business instance, a virtual machine, or a hardware device. For example, the assessment system can be implemented as a business instance deployed on one or more devices in a cloud node. Simply put, the assessment system can be understood as software deployed on a cloud node to provide large language model quantization quality assessment for various user terminals; alternatively, the assessment system can also be implemented as a virtual machine deployed on one or more devices in a cloud node, with application software installed to manage various user terminals; or, the assessment system can also be implemented as a server composed of numerous identical or different types of hardware devices, with one or more hardware devices configured to provide large language model quantization quality assessment for various user terminals.

[0022] In terms of implementation, the evaluation system and the user terminal are mutually compatible; that is, if the evaluation system is an application installed on a cloud service platform, then the user terminal is a client that establishes a communication connection with the application; or if the evaluation system is implemented as a website, then the user terminal is implemented as a webpage; or if the evaluation system is implemented as a cloud service platform, then the user terminal is implemented as a mini-program in an instant messaging application.

[0023] See Figure 1 The evaluation system includes: a resource analysis terminal, a quantitative execution terminal, and an evaluation analysis terminal, wherein: The resource analysis end is used to collect device datasets, clarify the hardware resource constraints of edge devices, and classify the sensitivity levels of functional layers in the large language model to quantization to obtain a sensitivity ranking. The quantization execution end establishes mapping rules based on the device dataset and sensitivity ranking to achieve precision level mapping and obtain a preliminary allocation scheme. Based on the preliminary allocation scheme, the final allocation scheme is determined by iteratively optimizing the balance between quantization precision loss and resource consumption. The quantization parameters are determined based on the final allocation scheme. The quantization type and quantization parameters of each functional layer are analyzed, a hybrid precision quantization operator library is constructed, and the compilation toolchain for edge devices is determined, compilation options are configured, and a quantization model for a large language model is constructed. The evaluation and analysis end uses a quantitative model and a standard test set to test different tasks to evaluate the quality of the quantitative model on edge devices and determine whether it meets the standards. For items that do not meet the standards in the evaluation, the accuracy of the quantitative model is adjusted and iterated, and resources are trimmed and optimized. At the same time, key indicators are monitored in real time and the system dataset is stored.

[0024] The components of the present invention will now be described in further detail with reference to the accompanying drawings.

[0025] 1. Resource analysis terminal.

[0026] The resource analysis module includes an edge device demand analysis module and a model sensitivity analysis module.

[0027] (1-1) Edge device requirements analysis module.

[0028] The edge device requirements analysis module includes a hardware resource parameter acquisition unit and a model performance requirement definition unit; wherein: The hardware resource parameter acquisition unit is used to collect device datasets from specified edge devices. These datasets include memory data, storage data, and power consumption data. The acquired datasets are then sent to the model sensitivity analysis module and the resource optimization iteration module. It should be noted that the device datasets are obtained by reading the official documentation from the edge device manufacturers, specifically the memory and power consumption data.

[0029] The model performance requirement definition unit is used to specify the model type of the large language model, and simultaneously set the quantization accuracy data loss threshold, edge device inference latency threshold, edge device memory usage threshold, and edge device power consumption threshold, and send them to the quantization quality assessment module of the evaluation and analysis end. Among these, the quantization accuracy data loss threshold, edge device inference latency threshold, edge device memory usage threshold, and edge device power consumption threshold are manually set and input.

[0030] (1-2) Model sensitivity analysis module.

[0031] The model sensitivity analysis module includes a feature extraction unit and a sensitivity calculation unit, wherein: The feature extraction unit is used to classify the functional layer types of the large language model, extract the parameter scale and computational cost of each functional layer, and mark functional layers with a total computational cost of 60% or more as computationally intensive layers. Simultaneously, it collects the data distribution of the input and output tensors of each functional layer and records the data flow during the inference process, generating a large language model layer analysis report, which is then sent to the sensitivity calculation unit. It should be noted that the functional layer types include embedding layers, multi-head attention layers, feedforward network layers, layer normalization layers, and output layers. The functional layer classification strictly follows the basic architecture of the large language model and does not disrupt the original inter-layer dependencies. The large language model layer analysis report includes the functional layer type of the large language model, the parameter scale and computational cost of each functional layer, and the data distribution of the input and output tensors of each functional layer.

[0032] The sensitivity calculation unit, based on the large language model layer analysis report, quantitatively calculates the sensitivity of each functional layer to quantification, classifies sensitivity levels, and obtains a sensitivity ranking; specifically including: ① Using FP32 as the original accuracy benchmark, the test accuracy set includes INT8, INT4 and FP16; set each functional layer as the target layer in sequence, and then execute the subsequent steps.

[0033] Among them, INT8 uses 8 binary bits to represent integers, and maps the floating-point weights and activation values ​​of LLM to the integer range through a scaling factor, and uses integer operations directly during inference; INT4 uses 4 binary bits to represent integers, which is also an integer quantization, but with fewer bits, requiring more precise numerical range clipping and group quantization; FP16 uses 16 binary bits to represent floating-point numbers, which is a floating-point quantization and retains the ability to represent fractional numbers.

[0034] ② Quantize the target layer to the specified precision, fix other functional layers as FP32, and perform isolated quantization test on the target layer.

[0035] ③ During isolated quantization testing, input the standard test set and obtain the output feature tensors of the target layer before and after quantization.

[0036] The standard test set includes the AGIEval and C-EvaI Chinese subsets; the feature tensors are the sequence length of each layer and the dimension of the hidden layers. ④ Based on the output feature tensors of the target layer before and after quantization, calculate the sensitivity of each layer and obtain the sensitivity coefficient.

[0037] Sensitivity coefficient The calculation formula is: ,in, The output feature tensor of the target layer at the original precision FP32. The output feature tensor of the target layer when quantized to a specified precision (INT8, INT4, and FP16).

[0038] ⑤ Repeat steps ② to ④ to calculate the sensitivity coefficient of each functional layer at the specified precision.

[0039] ⑥ When the sensitivity coefficient of a functional layer is greater than or equal to 0.1, it is determined to be a high-sensitivity layer; when the sensitivity coefficient is greater than 0.05 and less than 0.1, it is determined to be a medium-sensitivity layer; when the sensitivity coefficient is less than or equal to 0.05, it is determined to be a low-sensitivity layer; at the same time, the recommended accuracy range of each functional layer is marked to obtain the sensitivity ranking, and then sent to the mixed accuracy allocation module.

[0040] The recommended accuracy ranges are as follows: FP16 accuracy is preferred for high-sensitivity layers, INT8 accuracy is recommended for medium-sensitivity layers, and INT4 accuracy is recommended for low-sensitivity layers.

[0041] 2. Quantitative execution end.

[0042] The quantization execution module includes a mixed precision allocation module, a quantization parameter calculation module, and a quantization execution and edge adaptation module.

[0043] (2-1) Mixed precision allocation module.

[0044] The hybrid precision allocation module includes a precision level mapping unit and an allocation strategy optimization unit, wherein: The precision level mapping unit establishes mapping rules based on the device dataset and sensitivity ranking. High-sensitivity layers are preferentially allocated FP16; if memory usage exceeds the edge device's memory usage threshold, they are downgraded to INT8 and marked as requiring quality monitoring. Medium-sensitivity layers are allocated INT8 by default; if computational load exceeds the edge device's limit, they are downgraded to INT4, and the precision loss before and after downgrading is verified. Low-sensitivity layers are allocated INT4, and further compression is achieved using parameter sparsity to obtain a preliminary allocation scheme. This allocation refers to converting the original FP32 data to a specified precision while adapting it to the edge device's computational operators, ultimately enabling this layer to complete inference at the target precision on the edge device.

[0045] The allocation strategy optimization unit, based on the initial allocation scheme, determines the final allocation scheme by iteratively optimizing the balance between quantization accuracy loss and resource consumption; specifically, it includes: ① The allocation priority is calculated based on the sensitivity coefficient. Specifically: ; in, The sensitivity coefficient, Use a specified precision for resource consumption rates for the functional layers; The higher the allocation priority, the more the functional layer needs to prioritize accuracy and avoid over-quantization.

[0046] ② For resource overruns in the initial allocation plan, adjust them in descending order of allocation priority, prioritizing the reduction of the precision of the functional layer with the lowest allocation priority.

[0047] Among them, the resource overrun items include precision data loss exceeding the quantization precision data loss threshold, edge device inference latency exceeding the edge device inference latency threshold, edge device memory usage exceeding the edge device memory usage threshold, and edge device power consumption exceeding the edge device power consumption threshold.

[0048] ③ For cases where the accuracy loss exceeds the threshold in the initial allocation scheme, the accuracy is improved in ascending order of allocation priority; for example, from the middle sensitive layer INT4 to INT8.

[0049] ④ Based on the optimized solutions in ② and ③, calculate the comprehensive score: ; in, For the overall score, This refers to the accuracy loss rate (the ratio of loss before and after accuracy reduction). Resource overspending rate (the percentage of items exceeding the resource budget); For example, when the accuracy loss rate is 0.5 and the resource overrun rate is 0.4, The overall score is 0.54.

[0050] ⑤ Repeat steps ② to ③, iterate multiple times, select the optimized scheme with the highest comprehensive score as the final allocation scheme, and send it to the quantization parameter calculation module.

[0051] The final allocation scheme includes the quantization precision and quantization parameters of each functional layer; the quantization parameters include scaling factors and zero points.

[0052] (2-2) Quantization parameter calculation module.

[0053] The quantization parameter calculation module includes a symmetric quantization parameter calculation unit and an asymmetric quantization parameter calculation unit.

[0054] The symmetric quantization parameter calculation unit, targeting the functional layer where the data distribution in the large language model exhibits symmetry, analyzes the numerical range of the input data (the original floating-point data involved in quantization conversion), extracts the maximum absolute value of the feature tensor in the input data, and uses this to determine the dynamic range boundary of the input data. Simultaneously, based on quantization precision, it clarifies the corresponding integer value range to avoid precision loss due to integer overflow. Furthermore, it determines the mapping ratio between floating-point data and integers, randomly selects 30% of the samples from the input data, and calculates the error between the input data and the recovered data through quantization of floating-point numbers to integers and dequantization of integers to floating-point numbers. If the error is greater than 5%, the calculation method for the maximum absolute value is adjusted, including removing extreme outliers, recalculating the ratio, outputting the quantization precision and scaling factor, obtaining the symmetric quantization parameters, and sending them to the quantization execution and edge adaptation module.

[0055] The input data refers to the raw floating-point data in the large language model that has not undergone quantization. The recovered data refers to the approximate data obtained by quantizing (converting to INT8 / INT4, etc.) the raw floating-point data and then restoring it through dequantization. The mapping ratio is determined by the maximum absolute value of the data and the upper limit of the integer range, ensuring that the raw floating-point data can be uniformly mapped to the target integer range. The smaller the ratio, the higher the quantization accuracy.

[0056] The asymmetric quantization parameter calculation unit, targeting the functional layer where data distribution in the large language model exhibits asymmetry, analyzes the numerical distribution of the input data, extracts the maximum absolute value of the feature tensor in the input data, and uses this to determine the complete floating-point value range. Based on the quantization precision, it clarifies the corresponding integer value range and determines the total length of the integer range, while also determining the scaling factor and zero point. It performs quantization and dequantization operations on the input data and calculates the average error between the input data and the recovered data. When the error exceeds a preset threshold, it readjusts the value range of the input data, recalculates the scaling factor and zero point, until the preset requirements for the output floating-point number, zero point, and floating-point value range are achieved, thus obtaining the asymmetric quantization parameters, which are then sent to the quantization execution and edge adaptation module. (2-3) Quantitative execution and edge adaptation module.

[0057] The quantization execution and edge adaptation module includes a mixed-precision quantization operator generation unit and an edge device compilation and adaptation unit.

[0058] The mixed-precision quantization operator generation unit is used to parse the quantization type and quantization parameters (symmetric and asymmetric quantization parameters) of each functional layer based on the final allocation scheme, symmetric quantization parameters, and asymmetric quantization parameters. For the embedding layer in the functional layer, it generates parameter quantization operators, quantizing 32-bit floating-point weights offline into 4-bit integers and using group compression for storage. For the multi-head attention layer in the functional layer, it generates activation quantization operators, quantizing 32-bit floating-point inputs into 16-bit floating-point numbers in real time during inference to avoid precision loss. For the feedforward network layer in the functional layer, it generates quantization calculation fusion operators, fusing quantization, matrix multiplication, and dequantization to reduce data transfer costs. For the layer normalization layer in the functional layer, it generates numerical range constraint operators to ensure that the activation values ​​before quantization fall within the range of the maximum and minimum values ​​of the feature tensor, avoiding data overflow. Based on this, a mixed-precision quantization operator library is obtained and sent to the edge device compilation and adaptation unit.

[0059] The mixed-precision quantization operator library includes quantization operators for embedded layer parameters adapted to different precision quantization, activation quantization operators for multi-head attention layers, quantization calculation fusion operators for feedforward network layers, and numerical range constraint operators for layer normalization layers. All operators are associated with the symmetric or asymmetric quantization parameters of the corresponding layer.

[0060] The edge device compilation adaptation unit is used to determine the compilation toolchain for edge devices and configure compilation options; perform hardware instruction mapping on the mixed-precision quantization operator library, for example, mapping 8-bit integer matrix multiplication to ARM high-level SIMD instructions to improve computational efficiency; optimize the memory structure of large language models, and link quantization operators and model weights into an executable file to generate the model inference entry function, and reduce inference time through operator scheduling optimization, thereby obtaining the quantized model of the large language model.

[0061] Among them, memory optimization of large language model structure includes using block inference, for example: splitting a 1024-length sequence into four 256-length blocks to reduce memory usage per operation; operator scheduling optimization includes parallel execution of the embedding layer and the layer normalization layer.

[0062] 3. Evaluation and analysis end.

[0063] The evaluation and analysis module includes a quantitative quality assessment module, a resource optimization and iteration module, and a data management module.

[0064] (3-1) Quantitative quality assessment module.

[0065] The quantitative quality assessment module includes a quantitative accuracy loss assessment unit and an edge resource consumption assessment unit.

[0066] The quantization accuracy loss assessment unit is based on a quantization model and a standard test set. The tests include three types of tasks: text generation, semantic understanding, and logical reasoning. The unit tests the benchmark performance of the large language model on a PC and the performance of the quantization model on an edge device. It records the BLEU score, semantic understanding accuracy, and perplexity on the PC and edge devices. The unit also compares the BLEU loss rate, semantic understanding accuracy loss rate, and perplexity increase rate on the edge devices with the PC results to obtain quantization accuracy data. Content with substandard quantization accuracy data is sent to the resource optimization iteration module.

[0067] Among them, a BLEU loss rate of less than or equal to 5% is considered satisfactory, a semantic understanding accuracy loss rate of less than or equal to 5% is considered satisfactory, and a perplexity increase rate of less than or equal to 10% is considered satisfactory.

[0068] The edge resource consumption assessment unit is used to evaluate the binding force of the quantitative model on resource compliance on edge devices, specifically including: ① Calculate the sum of the memory usage of the quantization model parameters (scaling factor and zero point) and the memory usage of the inference cache, and divide it by the total memory of the edge device to obtain the memory usage rate.

[0069] ② Calculate the total energy consumption of the quantization model inference process, divide it by the quotient of the total inference time of the quantization model, and obtain the average power consumption.

[0070] ③ Calculate the number of test samples that meet the edge device inference latency threshold, divide by the total number of test samples, and obtain the latency compliance rate. Here, "test sample" refers to the test data selected from the standard test set.

[0071] ④ Based on memory usage, average power consumption, and latency compliance rate, compare them with preset edge device memory usage thresholds, edge device power consumption thresholds, and edge device inference latency thresholds, and mark the items that exceed the thresholds as non-compliant items and send them to the resource optimization iteration module.

[0072] (3-2) Resource optimization and iteration module.

[0073] The resource optimization iteration module includes a precision adjustment iteration unit and a resource pruning optimization unit.

[0074] The precision adjustment iteration unit, based on quantization precision data, examines the original precision allocation of the three functional layers with the highest proportion of quantization precision data loss, improves the precision of the high-sensitivity layers, and recalculates the scaling factor and zero point of the high-sensitivity layers to generate an optimized quantization model. It then retests and calculates the BLEU loss rate, semantic understanding accuracy loss rate, and perplexity increase rate on an edge device. If the quantization precision of the optimized quantization model is still not up to standard when tested on an edge device, a dynamic quantization method is used to adjust the scaling factor in real time until the quantization precision data of the optimized quantization model meets the standard.

[0075] Among them, the optimized quantization model is a quantization model generated after addressing the problems of excessive accuracy loss or excessive resource consumption in quantization quality assessment by improving the quantization accuracy of high-sensitivity layers or pruning the parameters of low-sensitivity layers, recalculating the quantization parameters of the corresponding layers, reconstructing the mixed-precision quantization operator, and adapting it to the edge device compilation environment.

[0076] The resource pruning and optimization unit performs pruning and optimization based on the non-compliant items output by the edge resource consumption assessment unit. When the memory occupancy rate is greater than the edge device memory occupancy threshold, parameters are pruned for low-sensitivity layers, and redundant parameters are removed to compress the quantization model volume. When the latency compliance rate is less than the preset edge device inference latency threshold, operator fusion is used for computationally intensive layers to reduce the number of data transfers. When the average power consumption is greater than the edge device power consumption threshold, the CPU or GPU frequency is adjusted. The resulting resource optimization scheme is then sent to the quantization quality assessment module for re-evaluation.

[0077] Redundant parameters refer to parameters whose absolute value is less than 1e-4; operator fusion methods include fusing convolution and activation into a single operator; adjusting the CPU or GPU frequency, for example, from 1.5GHz to 1.3GHz.

[0078] (3-3) Data Management Module.

[0079] The data management module is used to monitor key metrics and store system datasets; key metrics include sensitivity coefficient, scaling factor, zero point, and inference latency; system datasets include device datasets, sensitivity ranking, final allocation schemes, and quantization models.

[0080] This invention is well-suited to the power-sensitive and low-memory-bandwidth characteristics of edge devices, effectively solving the challenge of balancing resources and accuracy when deploying large language models at the edge. Through precise analysis of model characteristics and reasonable quantization strategy design, it can significantly reduce the model's resource consumption on edge devices while controlling accuracy loss, reducing inference latency and power consumption, and avoiding performance degradation or resource waste caused by over-quantization. At the same time, relying on the full-process quality assessment and dynamic optimization mechanism, it can verify the quantization effect in real time and make timely adjustments, ultimately improving the practicality and reliability of large language models in edge device scenarios, and providing strong support for the implementation of intelligent applications at the edge.

[0081] The evaluation system described in this invention can be set up in a cloud server. In terms of implementation, it can be implemented as one or more service devices, or as an application installed in the cloud (e.g., a mobile service operator's server, server cluster, etc.), or it can be developed as a website. Depending on the functions implemented, the evaluation system may include an edge device demand analysis module, a model sensitivity analysis module, a mixed precision allocation module, a quantization parameter calculation module, a quantization execution and edge adaptation module, a quantization quality evaluation module, a resource optimization iteration module, and a data management module. The modules described in this invention can also be referred to as units, which are a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, stored in the memory of the electronic device.

[0082] In this embodiment of the invention, each of the above modules can be implemented independently and can call other modules. Here, "calling" can be understood as a module connecting to multiple modules of another type and providing corresponding services to those connected modules. For example, the sharing and evaluation module can call the same information collection module to obtain information collected by that module. Based on the above characteristics, in the evaluation system of this invention, without modifying the program code, the applicability of the evaluation system architecture can be adjusted by adding modules and directly calling them, achieving cluster-based horizontal expansion to achieve the goal of quickly and flexibly expanding the evaluation system. In practical applications, the above modules can be set in the same device or different devices, or they can be set in virtual devices, such as service instances in a cloud server.

[0083] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A quantitative quality assessment system for large language models, characterized in that, include: Resource analysis, quantitative execution, and evaluation analysis, among which: The resource analysis end is used to collect device datasets, clarify the hardware resource constraints of edge devices, and classify the sensitivity levels of functional layers in the large language model to quantization to obtain a sensitivity ranking. The quantization execution end establishes mapping rules based on the device dataset and sensitivity ranking to achieve precision level mapping and obtain a preliminary allocation scheme. Based on the preliminary allocation scheme, the final allocation scheme is determined by iteratively optimizing the balance between quantization precision loss and resource consumption. The quantization parameters are determined based on the final allocation scheme. The quantization type and quantization parameters of each functional layer are analyzed, a hybrid precision quantization operator library is constructed, and the compilation toolchain for edge devices is determined, compilation options are configured, and a quantization model for a large language model is constructed. The evaluation and analysis end uses a quantitative model and a standard test set to test different tasks to evaluate the quality of the quantitative model on edge devices and determine whether it meets the standards. For items that do not meet the standards in the evaluation, the accuracy of the quantitative model is adjusted and iterated, and resources are trimmed and optimized. At the same time, key indicators are monitored in real time and the system dataset is stored.

2. The large language model quantitative quality assessment system according to claim 1, characterized in that, The resource analysis module includes an edge device demand analysis module; The edge device requirements analysis module includes a hardware resource parameter acquisition unit and a model performance requirement definition unit; wherein: The hardware resource parameter acquisition unit is used to collect device datasets for specified edge devices, including memory data, storage data, and power consumption data; and sends the collected device datasets to the model sensitivity analysis module and the resource optimization iteration module. The model performance requirement definition unit is used to specify the model type of the large language model, and at the same time set the quantization accuracy data loss threshold, edge device inference latency threshold, edge device memory usage threshold, and edge device power consumption threshold, and send them to the quantization quality assessment module of the evaluation and analysis end.

3. The large language model quantitative quality assessment system according to claim 1, characterized in that, The resource analysis module also includes a model sensitivity analysis module; The model sensitivity analysis module includes a feature extraction unit and a sensitivity calculation unit, wherein: The feature extraction unit is used to classify the functional layer types of the large language model, extract the parameter scale and computational cost of each functional layer, and mark functional layers with a total layer computational cost greater than a preset proportion as computationally intensive layers. Simultaneously, it collects the data distribution of the input and output tensors of each functional layer and records the data flow direction of each functional layer during inference, generating a large language model layer analysis report, which is then sent to the sensitivity calculation unit. The functional layer types include embedding layers, multi-head attention layers, feedforward network layers, layer normalization layers, and output layers. The functional layer division strictly follows the basic architecture of the large language model and does not disrupt the original inter-layer dependencies of the model. The sensitivity calculation unit, based on the large language model layer analysis report, quantitatively calculates the sensitivity of each functional layer to quantification, classifies sensitivity levels, and obtains a sensitivity ranking.

4. The large language model quantitative quality assessment system according to claim 3, characterized in that, The sensitivity calculation unit, based on the large language model layer analysis report, quantitatively calculates the sensitivity of each functional layer to quantification, classifies sensitivity levels, and obtains a sensitivity ranking, including: A1 uses FP32 as the original accuracy benchmark, and the test accuracy set includes INT8, INT4 and FP16; each functional layer is set as the target layer in turn. A2 quantizes the target layer to a specified precision, fixes other functional layers as FP32, and performs isolated quantization tests on the target layer; A3, during isolated quantization testing, input the standard test set and obtain the output feature tensor of the target layer before and after quantization; A4, based on the output feature tensors of the target layer before and after quantization, calculate the sensitivity of each layer and obtain the sensitivity coefficient; Sensitivity coefficient The calculation formula is: ,in, The output feature tensor of the target layer at the original precision FP32. The output feature tensor of the target layer when quantized to a specified precision; A5. Repeat steps A2 to A4 to calculate the sensitivity coefficient of each functional layer at the specified precision. A6 defines a functional layer as follows: if the sensitivity coefficient is greater than or equal to 0.1, it is classified as a high-sensitivity layer; if the sensitivity coefficient is greater than 0.05 and less than 0.1, it is classified as a medium-sensitivity layer; and if the sensitivity coefficient is less than or equal to 0.05, it is classified as a low-sensitivity layer. The recommended accuracy range for each functional layer is also marked to obtain a sensitivity ranking.

5. The large language model quantitative quality assessment system according to claim 1, characterized in that, The quantization execution terminal includes a precision allocation module; The hybrid precision allocation module includes a precision level mapping unit and an allocation strategy optimization unit, wherein: The accuracy level mapping unit establishes mapping rules based on the device dataset and sensitivity ranking: FP16 is preferentially allocated to high-sensitivity layers. If the memory usage exceeds the memory usage threshold of the edge device, it is downgraded to INT8 and marked as requiring quality monitoring; INT8 is allocated by default to medium-sensitivity layers. If the computational load exceeds the upper limit of the edge device, it is downgraded to INT4 and the accuracy loss before and after the downgrade is verified; INT4 is allocated to low-sensitivity layers, and parameter sparsity is used to further compress the data to obtain a preliminary allocation scheme. The allocation strategy optimization unit determines the final allocation scheme by iteratively optimizing the balance between quantization accuracy loss and resource consumption based on the initial allocation scheme.

6. The large language model quantitative quality assessment system according to claim 5, characterized in that, The allocation strategy optimization unit, based on the initial allocation scheme, determines the final allocation scheme by iteratively optimizing the balance between quantization accuracy loss and resource consumption, including: B1, the allocation priority is calculated based on the sensitivity coefficient. Specifically: ； in, The sensitivity coefficient, Use a specified precision for resource consumption rates for the functional layers; B2. For resource overruns in the initial allocation plan, adjust them in descending order of allocation priority, prioritizing the reduction of the precision of the functional layer with the lowest allocation priority; among them, resource overruns include precision data loss exceeding the quantization precision data loss threshold, edge device inference latency exceeding the edge device inference latency threshold, edge device memory usage exceeding the edge device memory usage threshold, and edge device power consumption exceeding the edge device power consumption threshold. B3, for cases where the accuracy loss in the initial allocation scheme exceeds the threshold, the accuracy is improved in ascending order of allocation priority; B4, based on the optimized solutions from B2 and B3, calculate the overall score: ； in, For the overall score, For accuracy loss rate, Resource overspending rate; B5. Repeat steps B2 to B3. After multiple iterations, select the optimized solution with the highest comprehensive score as the final allocation solution.

7. The large language model quantitative quality assessment system according to claim 1, characterized in that, The quantization execution terminal also includes a quantization parameter calculation module; The quantization parameter calculation module includes a symmetric quantization parameter calculation unit and an asymmetric quantization parameter calculation unit; The symmetric quantization parameter calculation unit targets the functional layer in the large language model where the data distribution exhibits symmetry. It analyzes the numerical range of the input data, extracts the maximum absolute value of the feature tensor in the input data, and uses this to determine the dynamic range boundary of the input data. Simultaneously, based on the quantization precision, it clarifies the corresponding integer value range. Based on this, it determines the mapping ratio between floating-point data and integers. It randomly selects samples from the input data, and calculates the error between the input data and the recovered data through quantization of floating-point numbers to integers and dequantization of integers to floating-point numbers. If the error is greater than the error threshold, it adjusts the calculation method of the maximum absolute value, including removing extreme outliers, recalculating the ratio, and outputting the quantization precision and scaling factor to obtain the symmetric quantization parameters. The asymmetric quantization parameter calculation unit is designed for functional layers where the data distribution in a large language model is asymmetric. It analyzes the numerical distribution of the input data and extracts the maximum value of the absolute value of the feature tensor in the input data to determine the complete floating-point value range. Based on quantization accuracy, the corresponding integer value range is defined, and the total length of the integer range is determined. At the same time, the scaling factor and zero point are determined. Quantization and dequantization operations are performed on the input data, and the average error between the input data and the recovered data is calculated. When the error is greater than a preset threshold, the value range of the input data is readjusted, and the scaling factor and zero point are recalculated until the preset requirements for the output floating-point number, zero point, and floating-point number value range are achieved, thus obtaining the asymmetric quantization parameters.

8. The large language model quantitative quality assessment system according to claim 1, characterized in that, The quantization execution terminal also includes a quantization execution and edge adaptation module; The quantization execution and edge adaptation module includes a mixed-precision quantization operator generation unit and an edge device compilation and adaptation unit; The mixed-precision quantization operator generation unit is used to parse the quantization type and quantization parameters of each functional layer based on the final allocation scheme, symmetric quantization parameters, and asymmetric quantization parameters. For the embedding layer in the functional layer, it generates parameter quantization operators, offline quantizing 32-bit floating-point weights into 4-bit integers, and using group compression for storage. For the multi-head attention layer in the functional layer, it generates activation quantization operators, quantizing 32-bit floating-point inputs into 16-bit floating-point inputs in real time during inference. For the feedforward network layer in the functional layer, it generates quantization calculation fusion operators, fusing quantization, matrix multiplication, and dequantization. For the layer normalization layer in the functional layer, it generates numerical range constraint operators. Based on these, a mixed-precision quantization operator library is obtained. The edge device compilation adaptation unit is used to determine the compilation toolchain for edge devices, configure compilation options, perform hardware instruction mapping for the mixed-precision quantization operator library, optimize the memory structure of large language models, link quantization operators and model weights into an executable file, generate the model inference entry function, and reduce inference time through operator scheduling optimization, thereby obtaining the quantization model of the large language model.

9. The large language model quantitative quality assessment system according to claim 1, characterized in that, The evaluation and analysis module includes a quantitative quality assessment module; The quantitative quality assessment module includes a quantitative accuracy loss assessment unit and an edge resource consumption assessment unit; The quantization accuracy loss assessment unit is based on a quantization model and a standard test set. The test includes three types of tasks: text generation, semantic understanding, and logical reasoning. The performance of the large language model on PC and the quantized model on edge devices were tested separately. The BLEU score, semantic understanding accuracy and perplexity were recorded on PC and edge devices. The BLEU loss rate, semantic understanding accuracy loss rate and perplexity increase rate were compared between PC and edge devices to obtain quantization accuracy data. Content with substandard quantization accuracy data was sent to the resource optimization iteration module. The edge resource consumption assessment unit is used to assess the binding force of the quantitative model on resource compliance on edge devices.

10. The large language model quantitative quality assessment system according to claim 1, characterized in that, The evaluation and analysis module also includes a resource optimization and iteration module; The resource optimization iteration module includes a precision adjustment iteration unit and a resource trimming optimization unit; The precision adjustment iteration unit, based on quantization precision data, examines the original precision allocation of the three functional layers with the highest proportion of quantization precision data loss, improves the precision of the high-sensitivity layers, and recalculates the scaling factor and zero point of the high-sensitivity layers to generate an optimized quantization model. It then retests and calculates the BLEU loss rate, semantic understanding accuracy loss rate, and perplexity increase rate on an edge device. If the quantization precision of the optimized quantization model is still not up to standard when tested on an edge device, a dynamic quantization method is used to adjust the scaling factor in real time until the quantization precision data of the optimized quantization model meets the standard. The resource pruning and optimization unit performs pruning and optimization based on the non-compliant items output by the edge resource consumption assessment unit; When the memory usage rate exceeds the edge device memory usage threshold, parameters are pruned for the low-sensitivity layer; when the latency compliance rate is less than the preset edge device inference latency threshold, operator fusion is used for the computationally intensive layer to reduce the number of data transfers; when the average power consumption exceeds the edge device power consumption threshold, the CPU or GPU frequency is adjusted; thus, a resource optimization scheme is obtained and sent to the quantitative quality assessment module for re-evaluation.

Images