A text question and answer method and system based on optimization of W8A8 model quantization
By constructing a professional calibration dataset and employing an optimized W8A8 quantization method with activation smoothing and quantization fallback strategies, the performance efficiency and accuracy issues of large-scale language models on resource-constrained devices are addressed, enabling efficient question-answering tasks on edge devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHICHENG TECH
- Filing Date
- 2026-05-22
- Publication Date
- 2026-07-14
AI Technical Summary
Existing large-scale language models have shortcomings in terms of inference speed, accuracy in maintenance tasks, and hardware resource consumption. They are particularly difficult to execute efficiently on resource-constrained devices, and existing mass-produced solutions lack industry adaptability, resulting in unstable accuracy and insufficient inference speed.
By constructing a professional calibration dataset for the target vertical domain, and employing an optimized W8A8 quantization method with activation smoothing and quantization backoff strategies, including data filtering and quality control, activation smoothing and quantization backoff strategies, an optimized W8A8 quantized model with mixed precision is generated, suitable for edge devices.
It significantly improves the stability and hardware operability of the model at INT8 accuracy, reduces computational complexity and storage consumption, improves question answering accuracy and inference speed, and is suitable for efficient deployment on edge devices.
Smart Images
Figure CN122389864A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the interdisciplinary field of text processing and model quantization, and specifically relates to a text question answering method and system based on optimized W8A8 model quantization. Background Technology
[0002] Large-scale language models (LLMs) have demonstrated outstanding performance in text generation, question answering, reasoning, and multi-domain tasks in recent years. However, as the model size continues to increase, the computational complexity, memory usage, and bandwidth consumption during the inference phase grow rapidly, posing significant challenges to practical deployment. For example, a typical model with tens to hundreds of billions of models requires hundreds of gigabytes of GPU memory for storage and operation at FP16 precision. It must rely on multiple high-end GPUs working together to complete inference computations, resulting in computational costs, energy consumption, and inference latency far exceeding what general applications can tolerate. Therefore, reducing model storage and computational overhead through model quantization has gradually become a widely adopted technical approach in the industry.
[0003] While existing quantization solutions have achieved some success on small-scale models, they still have significant shortcomings in large-scale models. As the model parameter scale expands to the billions or even tens of billions, significant systematic outliers appear in the activation distribution, causing severe quantization noise and obvious performance degradation when directly performing INT8 quantization. Dynamic activation quantization methods, represented by ZeroQuant, can maintain high accuracy in models up to 6 bytes in size, but their accuracy drops significantly on larger-scale models, making them unsatisfactory. LLM.int8 improves quantization error by retaining outlier activations in FP16 precision, but it requires frequent switching between INT8 and FP16, resulting in a complex computation graph structure that is not conducive to efficient execution on hardware accelerators such as NPUs, leading to poor hardware compatibility in practical applications. Furthermore, most existing methods lack adaptability to industry-specific scenarios. When quantized models are applied to professional tasks such as automotive repair question answering, their performance often deteriorates further due to inconsistencies between quantization parameters and industry data distribution, affecting question answering accuracy and semantic consistency. Meanwhile, when deploying quantization models on resource-constrained edge devices, existing quantization solutions often suffer from defects such as unstable accuracy, insufficient inference speed, or high storage consumption due to the limited computing power and bandwidth of the devices. Summary of the Invention
[0004] To address the problems existing in the background technology, this invention provides a text question answering method and system based on optimized W8A8 model quantization, which solves the technical problems of existing large model quantization technology in terms of inference speed, accuracy in maintenance tasks, and hardware resource consumption.
[0005] Existing quantization methods are susceptible to large quantization errors in language models with hundreds of billions of instances, leading to decreased inference accuracy and difficulty in efficient execution on resource-constrained devices. To address this issue, this invention proposes an optimized W8A8 quantization mechanism that includes activation smoothing and quantization backoff strategies. This mechanism improves the model's stability and hardware operability at INT8 accuracy, thereby significantly enhancing overall inference speed.
[0006] The technical solution adopted in this invention is: I. A text question answering method based on optimized W8A8 model quantization: S1. Construct a professional calibration dataset for the target vertical field, and then extract some data from the professional calibration dataset to construct an evaluation dataset.
[0007] S2. Based on the professional calibration dataset and evaluation dataset, optimize the pre-trained large language model to be quantized using W8A8 quantization to obtain the optimized W8A8 quantized model.
[0008] S3. Deploy the optimized W8A8 quantized model to an edge device and run a text question answering task in the target vertical domain on the edge device.
[0009] Step S1 specifically involves: S11. Obtain the original technical documents of the target vertical field, and extract the initial question-and-answer pairs after segmenting the original technical documents to form the basic corpus.
[0010] S12. Using the basic corpus as context, a large language model is used to generate expanded question-answer pairs.
[0011] S13. Filter and perform quality control on the initial question-and-answer pairs and the expanded question-and-answer pairs to obtain a professional calibration dataset.
[0012] S14. Extract several data points from the professional calibration dataset to construct the evaluation dataset.
[0013] The filtration and quality control in step S13 includes: The question-and-answer pairs are length-filtered, removing samples exceeding a preset threshold. Semantic quality is assessed, filtering samples with weak relevance, semantic incoherence, or logical errors. Redundancy is removed, deleting duplicate or highly similar samples.
[0014] The optimized W8A8 quantization process includes sequential activation smoothing and quantization rollback strategy processing.
[0015] The activation smoothing process includes the following steps: D1. Forward propagation professional calibration dataset, collecting activation tensors of each layer in the pre-trained large language model to be quantized. .
[0016] D2. Based on the activation tensor of each layer Obtain the outlier of the corresponding layer ,in , For the activation tensor of the current layer, To activate the tensor The maximum value of the elements in the middle; To activate the tensor The mean of the absolute values of all elements in the set. This represents the outlier degree of the current layer.
[0017] D3, based on outlier degree of each layer The balance factor α in the SmoothQuant method is dynamically determined, and the actual smoothing factor is calculated based on the preset scaling factor s and the determined balance factor α. According to the smoothing factor Perform a joint smoothing transformation on the activation tensor A and the weight tensor W to obtain the smoothed activation tensor. and the corresponding smoothed weight tensor ,in ; .
[0018] D4. Activation tensors after smoothing each layer Outlier suppression is performed, and the tensor is pruned by a threshold T to obtain the pruned activation tensor. .
[0019] D5. Based on the post-clipping activation tensor of each layer And combined with smoothing factor The weight tensor W is used to recalculate the quantization parameters, and the calculated quantization parameters are frozen for use in generating the INT8 deployment model.
[0020] The pruning activation tensor The result is obtained by processing it using the following formula: ; ; in, Activate the tensor after clipping; This is the clipping function; The smoothed activation tensor; This is the dynamic cropping threshold; Represents the calculated smoothed activation tensor of quantiles; Quantities; It is an adjustable constant; This represents the outlier degree of the current layer.
[0021] The quantization rollback strategy processing mechanism includes: F1, Outlier degree obtained according to step D2 Sort each layer in descending order to obtain a descending sorted set for each layer.
[0022] F2. Mark the first N% of layers in the descending sorted set as quantization-sensitive layers.
[0023] F3. Perform quantization sensitivity analysis on each layer between the N% layer and the M% layer in the descending sorted set to obtain the labeled quantization sensitive layers.
[0024] F4. Do not process the last (1-M%) layer in the descending sorted set.
[0025] F5. Perform INT8 operations on the non-quantization sensitive layers in the INT8 deployment model, and perform high-precision backoff processing on the quantization sensitive layers marked in the INT8 deployment model, thereby obtaining the optimized W8A8 quantized model for mixed-precision inference from the INT8 deployment model.
[0026] The quantitative sensitivity analysis specifically refers to: H1. Run the pre-trained large language model to be quantized and the INT8 deployment model on the evaluation dataset respectively, and calculate the error of the current layer in floating-point precision. Error at INT8 precision Thus, the error change is obtained. .
[0027] H2, when the error changes If the threshold is exceeded, the current layer is marked as a quantization-sensitive layer; otherwise, no marking is made.
[0028] The high-precision rollback process specifically involves switching the computational precision of the quantization-sensitive layer from INT8 to FP16 or FP32.
[0029] II. A text question answering system based on optimized W8A8 model quantization: The dataset construction module is used to construct a professional calibration dataset for the target vertical domain; the model quantization module is used to optimize the pre-trained large language model with W8A8 quantization based on the professional calibration dataset to obtain the optimized W8A8 quantized model; the edge deployment module is used to deploy the optimized W8A8 quantized model to edge devices and run the question answering task for the target vertical domain on the edge devices.
[0030] The beneficial effects of this invention are: 1. The quantization rollback strategy proposed in this invention retains floating-point calculations in some quantization-sensitive layers, while most layers still benefit from INT8 acceleration and storage savings, significantly reducing the computational complexity of each layer and decreasing the amount of multiply-accumulate operations, thereby significantly improving the overall inference speed. This design avoids global precision switching, resulting in a concise and efficient computation graph, making it very suitable for deployment on edge AI chips such as NPUs and DSPs with fixed-function INT8 computing units, achieving the best trade-off between precision and speed.
[0031] 2. After optimizing W8A8 quantization, the model's answer accuracy in domain-specific question-answering tasks is significantly improved, with improvements in semantic similarity and BLEU-4 metrics. Optimizing W8A8 quantization enhances question-answering accuracy in the automotive repair domain while reducing the accuracy loss due to model quantization.
[0032] 3. After W8A8 quantization optimization, the weights and most activations in the model are stored and computed in 8-bit format. The model size is reduced by about 50% compared to the FP16 model, memory bandwidth requirements are reduced, and multiply-accumulate operations are reduced. While maintaining high accuracy, the model storage size and memory usage are significantly reduced, which lowers the requirements for hardware computing power and storage bandwidth. It is suitable for high-performance deployment in cloud inference, edge devices, or embedded platforms, thereby significantly reducing the threshold for edge device deployment and system power consumption.
[0033] 4. The method of this invention provides a complete technology chain from data preparation and model quantization to edge deployment. The resulting "domain-specific lightweight large model" can be directly applied to practical scenarios such as vehicle diagnostic assistants, on-site maintenance guidance, and remote technical support, thereby improving the level of industrial intelligence. Attached Figure Description
[0034] Figure 1 This is a flowchart of the method of the present invention.
[0035] Figure 2 This is a block diagram of the system of the present invention. Detailed Implementation
[0036] The present invention will now be described in more detail with reference to the accompanying drawings and embodiments. However, the present invention is not limited thereto. For those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention. Contents not described in detail in this specification are prior art known to those skilled in the art.
[0037] W8A8 stands for 8-bit weights and 8-bit activations. It's a technical term used to describe the quantization precision configuration of a neural network model. "W" represents weights, and the first "8" represents an 8-bit integer, indicating that the original high-precision value (e.g., a 32-bit floating-point number, FP32) is converted to an 8-bit integer (INT8) for representation and storage using quantization techniques. "A" represents activations, which are the intermediate computation results (feature maps) of each layer's input and output during the forward propagation process. The second "8" also represents an 8-bit integer, indicating that the activation value is also quantized to INT8 for calculation.
[0038] like Figure 1 As shown, the text question answering method based on optimized W8A8 model quantization in this embodiment is implemented according to the following steps: S1. Construct a professional calibration dataset for the target vertical field, and then extract some data from the professional calibration dataset to construct an evaluation dataset.
[0039] S11. Data Collection and Preprocessing: Obtain original technical documents in the target vertical field, such as automotive repair manuals and fault diagnosis guides. After segmenting the original technical documents, extract initial question-answer pairs (questions and answers) to form basic corpus.
[0040] In this embodiment, the target vertical domain is the automotive repair field. Original technical documents related to automotive repair are acquired, segmented, and initial question-and-answer pairs are extracted from their content to form a basic corpus. This step aims to obtain data highly matched to the target application scenario for quantitative calibration and effectiveness evaluation.
[0041] In practice, the chunking process employs methods such as fixed-size chunking, semantic chunking, and document structure-based chunking to break down the original technical document into sentences or paragraphs. Then, based on the resulting sentences or paragraphs, rules or models are used to extract initial question-answer pairs.
[0042] S12, High-quality data augmentation: Using the basic corpus as context, a large language model is used to generate a large number of relevant augmented question-answer pairs.
[0043] In practice, the initial question-answer pair (basic corpus) is used as the context input, and the Qwen2.5-32B large model is guided by prompt words to automatically generate questions and their corresponding answers.
[0044] Furthermore, during the generation process, the output of the Qwen2.5-32B large model is automatically formatted to filter out the extended question-answer pairs of the large model that do not conform to the question-answer format specifications, so as to ensure the structured features and parsability of the generated extended question-answer pairs.
[0045] S13. Data Filtering and Quality Control: Filter and control the initial question-answer pairs and expanded question-answer pairs to obtain a professional calibration dataset.
[0046] S14. Extract several data points from the professional calibration dataset to construct the evaluation dataset.
[0047] The filtering and quality control in step S13 include: length filtering: filtering the question-answer pairs by length and removing samples whose length exceeds a preset threshold (too short or too long); semantic quality assessment: assessing the semantic quality of the question-answer pairs and filtering samples with weak question-answer relevance, semantic incoherence, and logical errors; redundancy removal: removing redundancy from the question-answer pairs by semantic similarity calculation (such as calculating cosine similarity after Sentence-BERT encoding) and deleting duplicate or highly similar samples to improve data diversity and overall effectiveness.
[0048] Furthermore, filtering and quality control also include: data balancing and integration: balancing samples from different sources (raw extraction, model generation) and different topics to ensure that the data covers a wide range of knowledge points in the target domain, and ultimately build a moderately sized (e.g., about 10,000 records) and high-quality professional calibration dataset.
[0049] In this embodiment, the evaluation set is divided as follows: a portion of samples (e.g., 1,000) are randomly selected from the final professional calibration dataset as an independent evaluation dataset. The topic distribution and question types are consistent with the calibration data, and it is used for subsequent objective evaluation of the model's performance.
[0050] In practice, semantic quality assessment adopts a combination of automated assessment and manual spot checks.
[0051] Automated evaluation: For each generated question-answer pair (question Q, answer A), the following scores are calculated in parallel: Correlation score The correlation score between Q and A was calculated using a RoBERTa-based cross-encoder model. This step is to filter out samples that are not strongly related to the questions and answers.
[0052] Fluency score The perplexity of answer A is calculated using the GPT-2 model and converted into a fluency score in the range [0,1]. This step is to filter out samples with semantically incoherent meanings.
[0053] Factual consistency score Using answer A as the query, the system retrieves the most similar sentences or paragraphs (calculated using cosine similarity) from all sentences or paragraphs obtained after segmenting the original technical document, thus obtaining the most similar similarity score. This step is to filter out samples with logical errors.
[0054] Set correlation threshold Fluency threshold Consistency threshold with facts Only when and and Only when the current question and answer are correct will they enter the candidate pool.
[0055] Manual sampling inspection: 5% of the samples are randomly selected from the candidate pool and independently reviewed by two automotive repair experts. The review focuses on: the rationality of the operational logic, compliance with safety regulations, and the accuracy of professional terminology. The sample data is considered to have passed the inspection only if both experts deem it qualified. If the pass rate is lower than the preset standard (e.g., 95%), the threshold is reduced and the evaluation is conducted again.
[0056] This dual filtering mechanism ensures that the question-answer pairs in the final calibration dataset are not only grammatically correct and highly relevant, but also possess a high degree of accuracy and logic within their respective professional fields.
[0057] In this embodiment, after filtering and quality control are completed, the distribution of samples from different sources is adjusted to ensure that the data coverage is extensive and the knowledge points are expressed in a variety of ways; finally, a professional calibration dataset of approximately 10,000 entries is obtained.
[0058] S2. Based on the professional calibration dataset and evaluation dataset, optimize the pre-trained large language model to be quantized using W8A8 quantization to obtain the optimized W8A8 quantized model.
[0059] Model preparation and calibration data loading: Select the pre-trained large language model to be quantized (Qwen2.5-7B-Instruct model is used in this embodiment), and load the professional calibration dataset obtained in step S1.
[0060] Optimizing W8A8 quantization processing includes sequential activation smoothing and quantization rollback strategy processing.
[0061] Activating the smoothing process includes the following steps: D1. During the calibration phase, feedforward propagation is used to obtain the activation tensors of each layer in the pre-trained large language model to be quantized from the professional calibration dataset. .
[0062] D2. Based on the activation tensor of each layer Obtain the outlier of the corresponding layer ,in , For the activation tensor of the current layer, To activate the tensor The maximum value of the elements in the middle; To activate the tensor The mean of the absolute values of all elements in the set. This represents the outlier degree of the current layer. A larger δ value indicates a more uneven activation distribution in that layer, and a more prominent impact from outliers.
[0063] D3, based on outlier degree of each layer The balance factor α (0≤α≤1) in the SmoothQuant method is dynamically determined. The actual smoothing factor is calculated based on the preset scaling factor s (usually the maximum absolute value of each channel of the activation layer) and the determined balance factor α. According to the smoothing factor Perform a joint smoothing transformation on the activation tensor A and the weight tensor W to obtain the smoothed activation tensor. and the corresponding smoothed weight tensor ,in ; .
[0064] In practice, based on the outlier degree of each layer The dynamic determination of the balance factor α in the SmoothQuant method is as follows: when In this case, the balance factor α in the SmoothQuant method is set to 0.5, thereby transferring some of the difficulty from activation to weight.
[0065] when In the SmoothQuant method, the balance factor α is set to approach 1. For example, 10 < If the value is ≤50, it is considered that there are moderate outliers in the current layer, and more aggressive smoothing is required. The balance factor α is set to be close to 1 (e.g., α=0.8~0.9) to transfer most of the dynamic range from activation to weight.
[0066] when In this case, the balance factor α in the SmoothQuant method is set to 1.0. For example... When the value is greater than 50, it is considered that there are serious outliers in the current layer. Smoothing alone may not be enough, and subsequent pruning and possible rollback strategies are needed. At this time, full smoothing with α set to 1.0 can be used, and the current layer is marked as a potentially sensitive layer for reference in subsequent quantization rollback strategies.
[0067] D4. Activation tensors after smoothing each layer Outlier suppression is performed, and the tensor is pruned by a threshold T to obtain the pruned activation tensor. .
[0068] D5. Based on the post-clipping activation tensor of each layer And combined with smoothing factor The weight tensor W is used to recalculate the quantization parameters, and the calculated quantization parameters are frozen for use in generating the INT8 deployment model.
[0069] Step D5 specifically involves: activating the tensor based on the clipping of each layer. Calculate the corresponding activation quantization parameters (the scaling factor and zero-point values corresponding to activation), based on the original weight tensor W and smoothing factor. Calculate the corresponding weight quantization parameters (scaling factor and zero point value corresponding to the weight), freeze all the calculated quantization parameters, and quantize the original weight tensor W into INT8 format to generate the INT8 deployment model corresponding to the pre-trained large language model to be quantized.
[0070] Activation Tensor After Clipping The result is obtained by processing it using the following formula: ; ; ; in, Activate the tensor after clipping; This is the clipping function; The smoothed activation tensor; This is the dynamic cropping threshold; Represents the calculated smoothed activation tensor (Right now (take the absolute value of each element) quantiles; This is a dynamic quantile, with a value range of (0,1]. This is an adjustable constant, typically set to 5; This represents the outlier degree of the current layer.
[0071] Specifically, the clipping function For the smoothed activation tensor For each element x in the expression, if x > T, output T; if x < -T, output -T; otherwise, output x.
[0072] This invention's adaptive pruning mechanism dynamically adjusts the pruning intensity based on the distribution characteristics of activations in each layer. When the outlier δ is large, it indicates the presence of significant outliers in the layer's activations. In this case, the formula automatically lowers the quantile q (but not below 0.95), thereby reducing the dynamic pruning threshold T and achieving strict suppression of outliers. When the outlier δ is small, it indicates a more uniform activation distribution. In this case, q is close to 1, and the threshold T increases accordingly, making the pruning operation more lenient to retain more effective information. By setting the lower limit of q to 0.95, it ensures that even with a very large outlier δ, at least 95% of the data is retained without pruning, avoiding information loss that may result from over-pruning. This design of the invention enables the quantization process to be precisely adjusted for different outlier levels in different layers, suppressing outliers while maximizing the model's representational power.
[0073] The quantitative rollback strategy processing mechanism includes: F1, Outlier degree obtained according to step D2 Sort each layer in descending order to obtain a descending sorted set for each layer.
[0074] F2. Mark the first N% of layers in the descending sorted set as quantization-sensitive layers.
[0075] F3. Perform quantization sensitivity analysis on each layer between the N% layer and the M% layer in the descending sorted set to obtain the labeled quantization sensitive layers.
[0076] The quantitative sensitivity analysis specifically includes: H1. Run the pre-trained large language model to be quantized and the INT8 deployment model on the evaluation dataset respectively, and calculate the error of the current layer in floating-point precision. Error at INT8 precision Thus, the error change is obtained. .
[0077] H2, when the error changes If the threshold is exceeded, the current layer is marked as a quantization-sensitive layer; otherwise, no marking is made.
[0078] In practice, layers that are typically labeled as quantization-sensitive layers are often key attention output projection layers or deep feedforward network layers.
[0079] F4. Do not process the last (1-M%) layer in the descending sorted set.
[0080] F5. Perform INT8 operations on the non-quantization sensitive layers in the INT8 deployment model, and perform high-precision backoff processing on the quantization sensitive layers marked in the INT8 deployment model, thereby obtaining the optimized W8A8 quantized model from the mixed-precision inference from the INT8 deployment model.
[0081] The high-precision rollback process specifically involves switching the computation precision of the quantization-sensitive layer from INT8 to FP16 or FP32.
[0082] In practice, both M and N are preset constants. N is generally set between 0 and 5, and M is set between 5 and 15 to ensure that only a small number of layers perform quantization backoff. INT8 operations are still performed on ordinary layers (non-quantization-sensitive layers), while high-precision backoff processing is performed on identified quantization-sensitive layers. This avoids the accuracy loss and numerical instability issues of key linear transformations or attention projection layers, ensuring both inference speed and model accuracy.
[0083] In this embodiment, N is 5 and M is 10. The first 5% of the layers in the descending sorted set are all marked as quantization-sensitive layers. Quantization sensitivity analysis is performed on each layer between the 5th and 10th layers in the descending sorted set. The last 90% of the layers in the descending sorted set are not processed.
[0084] Step F5 specifically involves: Mixed-precision graph compilation: generating a mixed-precision computation graph where non-sensitive layers perform INT8 integer matrix multiplication and addition operations, while sensitive layers perform FP16 or FP32 high-precision floating-point operations. Leveraging the mixed-precision support capabilities of deep learning frameworks (such as TensorRT and ONNX Runtime), the model is exported as the final deployment file.
[0085] Furthermore, to verify the beneficial effects of the present invention, this embodiment conducts a quantitative effect verification experiment: This embodiment optimizes the Qwen2.5-7B-Instruct model with W8A8 quantization, and further improves the quantization effect based on standard W8A8 quantization by introducing activation smoothing and quantization backoff strategies.
[0086] The optimized W8A8 quantized model obtained by this invention was comprehensively tested on the evaluation set, and its performance was compared with that of the original floating-point model and the standard W8A8 quantized model on domain task metrics. Key metrics may include generation quality (BLEU-4, ROUGE series) and semantic similarity (such as BERT-based Sentence Similarity).
[0087] The experimental results are shown in Table 1. The optimized W8A8 quantized model obtained in this invention shows improvement in several evaluation metrics, especially BLEU-4 and ROUGE-1, which improved from 10.9971 to 11.9233 and from 30.3004 to 32.2209 respectively compared to the standard W8A8 quantized model. ROUGE-2 and ROUGE-L also showed varying degrees of improvement. The Qwen2.5-7B-Instruct-optimized W8A8Similarity metric proposed in this invention improved from 72.95 to 75.29 compared to Qwen2.5-7B-Instruct-W8A8 (the standard W8A8 quantized model).
[0088] This indicates that the activation smoothing and quantization rollback strategies proposed in this invention effectively reduce the performance loss caused by quantization and restore some of the generated quality.
[0089] Table 1: This invention addresses the problem that general quantization schemes cannot adapt to industry-specific data distributions. It utilizes a dedicated calibration dataset from the automotive repair field to re-estimate the activation distribution and quantization parameters, making the quantization process more consistent with the characteristic distribution of industry corpora. This achieves targeted quantization optimization for automotive repair question-answering tasks. After optimized W8A8 quantization, the model shows significant improvements in metrics such as BLEU, ROUGE, and domain semantic similarity, effectively enhancing the model's generation quality and semantic consistency in vertical domain question-answering scenarios.
[0090] S3. Deploy the optimized W8A8 quantized model to an edge device and run a text question-answering task targeting the target vertical domain on the edge device. The target vertical domain question-answering task is an automotive repair question-answering task.
[0091] S31. Model Format Conversion: Export the weight file (.onnx or .engine format file) of the optimized W8A8 quantized model, and convert it to the corresponding format file according to the inference engine requirements of the target edge device (such as NVIDIA TensorRT for Jetson, Rockchip RKNN, Huawei Ascend CANN).
[0092] S32. Runtime adaptation: Adapt the model's input and output interfaces, including data type conversion, tensor shape reshaping, and memory layout optimization (such as NHWC to NCHW), to fully utilize hardware acceleration features.
[0093] S33. System Integration and Testing: Integrate the optimized W8A8 quantized model into the application software of the edge device (such as an automotive repair Q&A APP), and conduct end-to-end performance testing in actual or simulated scenarios to ensure that the requirements of real-time performance, accuracy and stability are met.
[0094] like Figure 2 As shown, this embodiment also provides a text question answering system based on optimized W8A8 model quantization, including: a dataset construction module for constructing a professional calibration dataset for the target vertical domain; a model quantization module for performing optimized W8A8 quantization on a pre-trained large language model based on the professional calibration dataset to obtain an optimized W8A8 quantized model; and an edge deployment module for deploying the optimized W8A8 quantized model to an edge device and running a question answering task for the target vertical domain on the edge device.
[0095] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.
Claims
1. A text question answering method based on optimized W8A8 model quantization, characterized in that, Includes the following steps: S1. Construct a professional calibration dataset for the target vertical field, and then extract some data from the professional calibration dataset to construct an evaluation dataset; S2. Based on the professional calibration dataset and evaluation dataset, optimize the pre-trained large language model to be quantized using W8A8 quantization to obtain the optimized W8A8 quantized model. S3. Deploy the optimized W8A8 quantized model to an edge device and run a text question answering task in the target vertical domain on the edge device.
2. The method for quantizing large language models based on optimized W8A8 as described in claim 1, characterized in that, Step S1 specifically involves: S11. Obtain the original technical documents of the target vertical field, and extract the initial question-and-answer pairs after segmenting the original technical documents to form the basic corpus. S12. Using a large language model with basic corpus as context, expandable question-answer pairs are generated. S13. Filter and perform quality control on the initial question-and-answer pairs and the expanded question-and-answer pairs to obtain a professional calibration dataset; S14. Extract several data points from the professional calibration dataset to construct the evaluation dataset.
3. The text question answering method based on optimized W8A8 model quantization according to claim 2, characterized in that, The filtration and quality control in step S13 includes: The length of the question-and-answer pairs is filtered, and samples exceeding a preset threshold are removed. Perform semantic quality assessment on question-answer pairs and filter out samples with weak question-answer relevance, semantic incoherence, and logical errors. Redundancy removal is performed on question-answer pairs, and duplicate or highly similar samples are deleted.
4. The text question answering method based on optimized W8A8 model quantization according to claim 1, characterized in that: The optimized W8A8 quantization process includes sequential activation smoothing and quantization rollback strategy processing.
5. A text question answering method based on optimized W8A8 model quantization according to claim 4, characterized in that, The activation smoothing process includes the following steps: D1. Forward propagation professional calibration dataset, collecting activation tensors of each layer in the pre-trained large language model to be quantized. ; D2. Based on the activation tensor of each layer Obtain the outlier of the corresponding layer ,in , For the activation tensor of the current layer, To activate the tensor The maximum value of the elements in the middle; To activate the tensor The mean of the absolute values of all elements in the set. The outlier degree corresponding to the current layer; D3, based on outlier degree of each layer The balance factor α in the SmoothQuant method is dynamically determined, and the actual smoothing factor is calculated based on the preset scaling factor s and the determined balance factor α. According to the smoothing factor Perform a joint smoothing transformation on the activation tensor A and the weight tensor W to obtain the smoothed activation tensor. and the corresponding smoothed weight tensor ,in ; ; D4. Activation tensors after smoothing each layer Outlier suppression is performed, and the tensor is pruned by a threshold T to obtain the pruned activation tensor. ; D5. Based on the post-clipping activation tensor of each layer And combined with smoothing factor The weight tensor W is used to recalculate the quantization parameters, and the calculated quantization parameters are frozen for use in generating the INT8 deployment model.
6. The text question answering method based on optimized W8A8 model quantization according to claim 5, characterized in that: The pruning activation tensor The result is obtained by processing it using the following formula: ； ； ； in, Activate the tensor after clipping; This is the clipping function; The smoothed activation tensor; This is a dynamic cropping threshold; Represents the calculated smoothed activation tensor of quantiles; Quantities; It is an adjustable constant; This represents the outlier degree of the current layer.
7. A text question answering method based on optimized W8A8 model quantization according to claim 4, characterized in that, The quantization rollback strategy processing mechanism includes: F1, Outlier degree obtained according to step D2 Sort each layer in descending order to obtain a descending sorted set of each layer; F2. Mark the first N% of layers in the descending sorted set as quantization-sensitive layers; F3. Perform quantization sensitivity analysis on each layer between the N% layer and the M% layer in the descending sorted set to obtain the labeled quantization sensitive layers. F4. Do not process the last (1-M%) layer in the descending sorted set; F5. Perform INT8 operations on the non-quantization sensitive layers in the INT8 deployment model, and perform high-precision backoff processing on the quantization sensitive layers marked in the INT8 deployment model, thereby obtaining the optimized W8A8 quantized model for mixed-precision inference from the INT8 deployment model.
8. A text question answering method based on optimized W8A8 model quantization according to claim 6, characterized in that, The quantitative sensitivity analysis specifically refers to: H1. Run the pre-trained large language model to be quantized and the INT8 deployment model on the evaluation dataset respectively, and calculate the error of the current layer in floating-point precision. Error at INT8 precision Thus, the error change is obtained. ; H2, when the error changes If the threshold is exceeded, the current layer will be marked as a quantization-sensitive layer; Otherwise, do not mark it.
9. A text question answering method based on optimized W8A8 model quantization according to claim 6, characterized in that, The high-precision rollback process specifically includes: Switch the computation precision of the quantization-sensitive layer from INT8 to FP16 or FP32.
10. A text question answering system based on optimized W8A8 model quantization using the method described in any one of claims 1-9, comprising: The dataset building module is used to build specialized calibration datasets for a target vertical domain; The model quantization module is used to perform optimized W8A8 quantization on the pre-trained large language model based on the professional calibration dataset to obtain the optimized W8A8 quantized model. The edge deployment module is used to deploy the optimized W8A8 quantized model to edge devices and run question-answering tasks in the target vertical domain on the edge devices.