Method and system for quantizing large-scale voice model after ultra-low training

Abstract

The invention discloses a quantizing method and system after ultra-low training of a large voice model, and relates to the field of voice models. Comprising the steps of weight matrix preprocessing, K-means clustering quantification, abnormal value detection and mixing precision distribution, selective abnormal value retention, model reasoning and performance evaluation. According to the method, dynamic precision distribution is realized, quantization bits are dynamically adjusted according to abnormal value density, and the resource utilization rate is optimized; key information is reserved, abnormal values are selectively reserved, quantization errors are avoided, and the model performance is improved; the generalization ability of the quantitative model is ensured through the cross-domain robustness; and an efficient and flexible solution is provided for ultra-low bit quantization.

Description

Technical Field

[0001] The present invention relates to the field of speech models, and in particular to a method and system for ultra-low-bit training and quantization of large speech models. Background Art

[0002] Existing large speech models (such as Whisper) face storage and deployment challenges due to their large number of parameters (such as 1.5B). Post-training quantization (PTQ) can compress the model without retraining, but existing methods experience a sharp decline in performance when below 8 bits, especially in handling weight outliers commonly present in the Transformer architecture. Traditional linear quantization (such as GPTQ) significantly increases the word error rate (WER) at ultra-low bits (such as 2 bits), leading to model failure. Additionally, existing methods are not optimized for the characteristics of speech models (such as the concentrated distribution of outliers), limiting their practical applications.

[0003] Therefore, those skilled in the art are committed to developing a method and system for ultra-low-bit training and quantization of large speech models. Dynamically adjust the quantization bits according to the outlier density to optimize resource utilization; avoid quantization errors and improve model performance by selectively retaining outliers; ensure the generalization ability of the quantized model. Summary of the Invention

[0004] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is the problem of the decline in the performance and accuracy of large speech models.

[0005] To achieve the above object, the present invention provides a method for ultra-low-bit training and quantization of large speech models, including the following steps:

[0006] Step 1, Weight matrix preprocessing: Extract the weight matrix and group it by column;

[0007] Step 2, K-means clustering quantization: Perform K-means clustering on each column of weights to generate quantization centroids;

[0008] Step 3, Outlier detection and mixed-precision allocation: Dynamically allocate quantization precision according to the outlier density;

[0009] Step 4, Selective outlier retention: Retain key outliers in sparse FP32 format;

[0010] Step 5, Model inference and performance evaluation: Perform inference on the quantized model and evaluate the performance.

[0011] Further, the step 1 includes the following steps:

[0012] Step 1.1, Weight extraction: Extract the weight matrix from the pre-trained large speech model;

[0013] Step 1.2, Column Grouping: Group the weight matrix by column, with each column as an independent quantization group.

[0014] Further, in the K-means clustering, the data set is divided into clusters, maximizing the intra-cluster data similarity and the inter-cluster difference.

[0015] Further, the said Step 2 includes the following steps:

[0016] Step 2.1, Center Point Generation: For each column of weights, use the K-means clustering algorithm to generate the clustering center points;

[0017] Step 2.2, Quantization Mapping: Quantize the original weight values into the clustering center point indices;

[0018] Step 2.3, Error Optimization: Update the unquantized weights.

[0019] Further, the said K-means clustering algorithm includes the following steps:

[0020] Step 2.1.1, Initialization: Randomly select initial cluster centers;

[0021] Step 2.1.2, Assignment: Assign each data point to the nearest cluster center;

[0022] Step 2.1.3, Update: Recalculate the center points of each cluster;

[0023] Step 2.1.4, Iteration: Repeat the assignment and update steps until the cluster centers no longer change significantly or reach the maximum number of iterations.

[0024] Further, in the said Step 2.3, use the GPTQ strategy to update the unquantized weights.

[0025] Further, in the said Step 3, dynamically allocate the quantization bits according to the column-level outlier density: Allocate 4 bits to the outlier-dense columns and 2 bits to the other columns.

[0026] Further, the said Step 4 includes the following steps:

[0027] Step 4.1, Outlier Screening: In the outlier-dense columns, retain the top 5% of the outliers in the sparse FP32 format;

[0028] Step 4.2, Weight Recovery: Superimpose the sparse FP32 outliers and the quantized weights to restore the original weight distribution.

[0029] Further, the said Step 5 includes the following steps:

[0030] Step 5.1, Model Generation: Combine the quantized weight matrix with the sparse FP32 outliers to generate a low-bit model;

[0031] Step 5.2, Performance evaluation.

[0032] The present invention also provides a system for ultra-low-bit training and quantization of a large speech model, including a weight matrix extraction module, a K-means clustering module, a mixed-precision allocation module, an outlier retention module, and a model inference module;

[0033] The weight matrix extraction module extracts a weight matrix from a pre-trained large speech model;

[0034] The K-means clustering module performs K-means clustering on the weight matrix column by column to generate quantization center points;

[0035] The mixed-precision allocation module dynamically allocates quantization precision according to outlier density;

[0036] The outlier retention module retains key outliers in sparse FP32 format;

[0037] The model inference module performs inference on the quantized model to evaluate performance.

[0038] Under existing ultra-low-bit quantization, the performance of the model seriously deteriorates. The present invention uses K-means clustering for non-linear quantization. The present invention groups the weight matrix by column, uses the K-means algorithm to generate clustering center points, dynamically allocates quantization values, and reduces the error caused by distribution mismatch. Under 3-bit quantization of the present invention, compared with the performance of the full-precision 32-bit model, the word error rate (WER) on the LibriSpeech test set only increases by 0.03%, and the model compression rate reaches 90.6%.

[0039] Existing outliers result in loss of quantization precision. The present invention has a mixed-precision allocation strategy. The present invention dynamically allocates quantization bits according to column-level outlier density: high precision (4 bits) is allocated to columns with dense outliers, and low precision (2 bits) is allocated to other columns. The mixed-precision quantization (2.1 bits) of the present invention reduces the word error rate (WER) by 57% compared with pure 2 bits (e.g., from 14.04% to 6.04% on the LibriSpeech test set).

[0040] Existing performance degrades after outlier quantization. The present invention selectively retains key outliers in sparse full-precision 32-bit format. The present invention separates outliers from the quantization process, retains them as a sparse FP32 matrix, and superimposes them on the quantized weights to avoid loss of key information. Under 2.12-bit quantization of the present invention, the error rate (WER) is restored to a level close to the full-precision level (2.19% on the LibriSpeech test set vs. 2.02% for full precision).

[0041] The present invention proposes an ultra-low-bit training and quantization method for a large speech model based on K-means clustering and mixed-precision allocation, and the specific steps are as follows:

[0042] Step 1: Weight Matrix Preprocessing

[0043] Preprocess the weight matrix of the model to prepare for subsequent quantization operations. Ensure that the quantization operation is performed at the column level to reduce the impact of cross-column weight distribution differences on quantization.

[0044] 1. Weight Extraction: Extract the weight matrix from a pre-trained large speech model (such as Whisper-Large-V3).

[0045] 2. Column Grouping: Group the weight matrix by columns, with each column as an independent quantization group.

[0046] In large speech models (such as Whisper, Wav2Vec2, etc.), outliers are one of the key factors affecting quantization effects. Outliers usually refer to those extreme points in the weight matrix that significantly deviate from the majority of the weight distribution. These outliers are particularly common in the Transformer architecture, especially in the attention mechanism, where the weights of some attention heads may contain a large number of outliers.

[0047] 1. Distribution Characteristics of Outliers

[0048] Concentration: Outliers tend to concentrate in certain specific columns (i.e., some columns of the weight matrix), rather than being evenly distributed throughout the matrix. For example, in the Whisper model, 90% of the outliers may only concentrate in 10% of the columns.

[0049] Importance: These outliers are usually crucial for the performance of the model. If these outliers are not properly handled during the quantization process, it will lead to a significant decline in the model's performance.

[0050] 2. Necessity of Column-Level Quantization

[0051] Reduce the impact of cross-column weight distribution differences:

[0052] If global quantization is performed on the entire weight matrix (i.e., without considering the distribution differences between columns), the existence of outliers will lead to uneven distribution of quantization centers. Specifically:

[0053] Disadvantages of Global Quantization: In global quantization, outliers will "skew" the distribution of quantization centers, resulting in insufficient quantization accuracy for most normal weights. For example, the existence of outliers may cause the quantization centers to concentrate near the outliers, while ignoring the distribution of most normal weights.

[0054] Advantages of column-level quantization: By quantizing column by column, quantization center points can be independently generated for the weight distribution of each column. In this way, columns with dense outliers can be assigned more quantization center points (i.e., higher quantization precision), while columns with fewer outliers can be assigned fewer quantization center points (i.e., lower quantization precision). This column-level quantization strategy can better adapt to the distribution characteristics of different columns and reduce the impact of outliers on the overall quantization effect.

[0055] 3. Processing effect of column-level quantization on outliers

[0056] Dynamically adapting to the outlier distribution: Column-level quantization can dynamically adapt to the outlier distribution of each column. For columns with dense outliers, more details can be retained by increasing the quantization bits (e.g., 4 bits); for columns with fewer outliers, a lower quantization bit (e.g., 2 bits) can be used to reduce the storage overhead.

[0057] Reducing quantization error: Since outliers are usually concentrated in certain columns, column-level quantization can avoid the interference of outliers on the quantization precision of other columns. For example, in columns with dense outliers, the quantization center points can better capture the outlier distribution, thus reducing the quantization error; in columns with fewer outliers, the quantization center points can better fit the distribution of normal weights, further improving the quantization effect.

[0058] Step 2: K-means clustering quantization

[0059] Generate quantization center points through K-means clustering to achieve non-linear quantization.

[0060] 1. Center point generation: For each column of weights W :,j , use the K-means algorithm to generate 2 n clustering center points C j .

[0061] 2. Quantization mapping: Quantize the original weight value W i,j to the index q of the nearest clustering center point, i.e.:

[0062] Q i,j = argmin q C j [q] - W i,j

[0063] 3. Error optimization: Use the GPTQ strategy to update the unquantized weights to further reduce the quantization error.

[0064] Dynamic center point generation: Dynamically generate center points according to the weight distribution to avoid the precision loss of linear quantization under non-uniform distribution.

[0065] Resource efficiency: Generate more center points in weight-dense regions, reduce redundancy in sparse regions, and optimize storage utilization.

[0066] In 2-bit quantization, the WER of the quantized Whisper-Large-V3 model on the LibriSpeech test clean test set only increases from 2.02% to 14.04%, significantly outperforming the traditional linear quantization method (2.02% -> 51.13%).

[0067] K-means clustering quantization:

[0068] 1. Overview of K-means clustering quantization

[0069] K-means clustering quantization is a non-linear quantization method based on unsupervised learning. By dynamically generating quantization center points, it significantly improves the model quantization performance. Its core idea is to use the K-means clustering algorithm to divide the weight values into k = 2 n clusters (n is the quantization bit number), and map each weight value to the nearest cluster center point, thus achieving efficient and accurate quantization.

[0070] 1.1 Background of K-means clustering algorithm

[0071] K-means clustering is a classic unsupervised learning algorithm. Its goal is to divide the data set into k clusters, maximizing the similarity within the clusters and the difference between the clusters. The specific steps are as follows:

[0072] 1. Initialization: Randomly select k initial cluster centers.

[0073] 2. Assignment: Assign each data point to the nearest cluster center.

[0074] 3. Update: Recalculate the center point of each cluster (i.e., the mean of all data points within the cluster).

[0075] 4. Iteration: Repeat the assignment and update steps until the cluster centers no longer change significantly or reach the maximum number of iterations.

[0076] In the quantization task, K-means clustering is used to divide the weight values in the weight matrix into k = 2 n clusters, and the center point of each cluster is used as the representative value after quantization.

[0077] 2. Performance advantages of K-means quantization compared to uniform quantization

[0078] 2.1 Adaptability to non-uniform distribution

[0079] Disadvantages of uniform quantization: Assume that the weights are uniformly distributed, but in actual scenarios (such as Transformer models), they often exhibit long-tail characteristics, resulting in quantization errors concentrated in the dense regions.

[0080] Advantages of K-Means: Dynamically adjust the center point density, generate more center points in the weight-dense area, reduce redundancy in the sparse area, and reduce the overall error.

[0081] 2.2 Outlier Robustness

[0082] Problem of uniform quantization: Outliers stretch the quantization range and compress the resolution of normal weights (e.g., the WER of uniform quantization increases by 51.13% in the LibriSpeech experiment).

[0083] K-Means strategy: Outliers are clustered separately to avoid interfering with normal weights. Combined with GPTQ fine-tuning, further correct the quantization error in the outlier area.

[0084] 2.3 Dynamic Precision Allocation

[0085] Mixed precision support: Different quantization bits can be assigned to different weight columns. For example, 4 bits are used for key layers and 2 bits for non-key layers to balance performance and storage.

[0086] Hardware friendliness: The storage of the center point index only requires log2k bits. Combined with the look-up table mechanism, the calculation efficiency is close to that of uniform quantization.

[0087] 2.4 Experimental Effect Verification

[0088] d is the dimension of the weight matrix, i is the index of the weight matrix in the output dimension, and j is the index of the weight matrix in the input dimension

[0089] Quantization accuracy: In the Whisper-Large-V3 model, 2-bit K-Means quantization only increases the WER of LibriSpeech by 0.17%, while uniform quantization causes the model to fail (WER > 50%).

[0090] Generalization ability: On datasets such as Common Voice, K-Means quantization keeps the increase in WER < 1%, which is better than the non-linear distortion of uniform quantization.

[0091] Step 3: Outlier Detection and Mixed Precision Allocation

[0092] Dynamically allocate quantization precision according to the outlier density to improve the quantization effect.

[0093] 1. Outlier detection: Calculate the outlier ratio R :,j of each column of weights W j , and the formula is:

[0094]

[0095] where is Outlier(W i,j ) is defined as:

[0096]

[0097] 2. Mixed precision allocation: According to R j Quantization accuracy:

[0098]

[0099] Dynamic precision adjustment: Dynamically allocate the number of quantization bits according to the outlier density (determined by the pre-set hyperparameter λ) to avoid performance loss caused by the "one-size-fits-all" quantization strategy.

[0100] Resource optimization: Use low precision in outlier sparse columns to significantly reduce storage and computing overhead.

[0101] Under 2.1-bit mixed precision quantization, WER is significantly lower than pure 2-bit quantization. For example, the WER of the quantized Whisper-Large-V3 model on the LibriSpeech test clean test set is reduced from 14.04% to 6.04.

[0102] Step 4: Selective outlier retention

[0103] By retaining key outliers in sparse FP32 format, quantization errors are avoided and model performance is further improved.

[0104] 1. Outlier screening: In the outlier dense column (R j >13%), retaining the top 5% of outliers in sparse FP32 format.

[0105] 2. Weight recovery: superimpose the sparse FP32 outliers with the quantized weights to restore the original weight distribution.

[0106] Key information retention: By selectively retaining outliers, key information loss during the quantification process is avoided.

[0107] Sparse storage optimization: Use sparse matrices to store FP32 outliers to reduce storage overhead.

[0108] Under 2.12-bit quantization, the WER is restored to a level close to full precision. For example, the WER of the quantized Whisper-Large-V3 model on the LibriSpeech test clean test set is reduced from 6.04% to 2.19%, which is almost the same as the 2.02% of the full precision model.

[0109] Step 5: Model Inference and Performance Evaluation

[0110] Perform inference on the quantized model, evaluate its performance, and verify the effectiveness of the quantization strategy.

[0111] 1. Model Generation: Combine the quantized weight matrix with sparse FP32 outliers to generate the final low-bit model.

[0112] 2. Performance Evaluation: Test the model performance on multiple datasets (such as LibriSpeech, Common Voice, etc.) and record the WER (Word Error Rate).

[0113] The WER increase of the quantized model on multiple datasets is less than 1%, maintaining cross-domain robustness. For example:

[0114] · On the LibriSpeech dataset, the WER is 2.19%, only increasing by 0.17% compared to the full-precision model.

[0115] · On the Common Voice dataset, the WER is 11.92%, while traditional quantization methods would cause the model performance to collapse, with a WER as high as 210.94%.

[0116] The three steps of K-means clustering quantization, outlier detection and mixed-precision allocation, and selective outlier retention in the present invention together constitute an efficient quantization strategy. Its core advantages are:

[0117] 1. Dynamic Precision Allocation: Dynamically adjust the quantization bits according to the outlier density to optimize resource utilization.

[0118] 2. Key Information Retention: Avoid quantization errors and improve model performance by selectively retaining outliers.

[0119] 3. Cross-Domain Robustness: The WER increase is less than 1% on multiple datasets, ensuring the generalization ability of the quantized model.

[0120] The present invention provides an efficient and flexible solution for ultra-low-bit quantization, which is particularly suitable for resource-constrained edge computing scenarios.

[0121] An ultra-low-bit post-training quantization system for large speech models based on K-means clustering and mixed-precision allocation in the present invention specifically includes the following necessary components:

[0122] Component 1: Weight Matrix Extraction Module

[0123] Extract the weight matrix from the pre-trained large speech model. Provide the original weight data for quantization operations.

[0124] Connection Relationship: Connected to the K-means clustering module to provide input data.

[0125] Component 2: K-means Clustering Module

[0126] Perform K-means clustering on the weight matrix by column to generate quantization centroids.

[0127] Connection relationship: Connected to the weight matrix extraction module and the mixed-precision allocation module.

[0128] Shape: Supports dynamically generating 2 n cluster center points.

[0129] Principle: Optimize the center point distribution through the K-means algorithm to reduce quantization error.

[0130] Component 3: Mixed-precision allocation module

[0131] Dynamically allocate quantization precision according to the outlier density.

[0132] Connection relationship: Connected to the K-means clustering module and the outlier retention module.

[0133] Shape: Supports dynamically adjusting the quantization bit width (such as 2 bits or 4 bits).

[0134] Principle: Allocate precision according to the outlier ratio R j to avoid the negative impact of outliers on quantization.

[0135] Component 4: Outlier retention module

[0136] Retain key outliers in sparse FP32 format. Under 2.12-bit quantization, the WER is restored to near full-precision level.

[0137] Connection relationship: Connected to the mixed-precision allocation module and the model inference module.

[0138] Shape: Supports storing outliers in sparse FP32 format.

[0139] Principle: Separate outliers from the quantization process to avoid loss of key information.

[0140] Component 5: Model inference module

[0141] Infer the quantized model and evaluate its performance. Verify the actual performance of the quantized model.

[0142] Connection relationship: Connected to the outlier retention module to generate the final low-precision model.

[0143] The technical solution of the present invention is realized through the following steps:

[0144] 1. Weight matrix preprocessing: Extract the weight matrix and group it by column.

[0145] 2. K-means clustering quantization: Perform K-means clustering on each column of weights to generate quantization center points.

[0146] 3. Outlier detection and mixed-precision allocation: Dynamically allocate quantization precision according to the outlier density.

[0147] 4. Selective Outlier Retention: Retain key outliers in sparse FP32 format.

[0148] 5. Model Inference and Performance Evaluation: Perform inference on the quantized model and evaluate its performance.

[0149] The system of the present invention includes a weight matrix extraction module, a K-means clustering module, a mixed-precision allocation module, an outlier retention module, and a model inference module. Each module is connected by a data stream and collaborates to complete the quantization task.

[0150] Among them, WER (Word Error Rate): Word error rate, used to evaluate the performance of a speech recognition model. The lower the value, the better the performance. K-means clustering: An unsupervised learning algorithm used to divide data points into k clusters, and each cluster is represented by its center point. FP32: 32-bit floating-point format, used for high-precision numerical representation. Sparse FP32: An FP32 format that only stores non-zero values, used for efficient storage of outliers.

[0151] Compared with the prior art, the present invention has the following obvious substantial features and remarkable advantages:

[0152] 1. Technical Advantage: Supports ultra-low-bit (2.12-bit) quantization, with a model compression rate exceeding 93%, adapting to the storage limitations of edge devices.

[0153] 2. Performance Metrics: The increase in WER is less than 1% on multiple datasets (such as LibriSpeech, CommonVoice, etc.), maintaining the cross-domain robustness of the model.

[0154] 3. Production Implementation: No need for retraining, and the quantization of a 1B model can be completed within 3 hours (based on RTX4090 GPU), significantly reducing the deployment cost.

[0155] 4. Generalizability: Applicable to various speech models (such as Whisper, Wav2Vec2), with strong scalability.

[0156] The following will further illustrate the concept, specific structure, and technical effects of the present invention with reference to the accompanying drawings to fully understand the purpose, features, and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0157] Figure 1 is a framework diagram of mixed-precision allocation and outlier retention of a preferred embodiment of the present invention;

[0158] Figure 2 is a schematic diagram of the K-means clustering quantization process of a preferred embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0159] The following describes multiple preferred embodiments of the present invention with reference to the accompanying drawings of the specification to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the scope of protection of the present invention is not limited to the embodiments mentioned in the text.

[0160] In the accompanying drawings, components with the same structure are denoted by the same numerical labels, and components with similar structures or functions everywhere are denoted by similar numerical labels. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. To make the illustration clearer, the thickness of some components in the drawings is appropriately exaggerated.

[0161] As Figure 1 shown, the overall quantization process of the present invention includes a mixed-precision allocation module, a sparse FP32 outlier retention module, and quantization weights and outlier superposition. Among them, the weight matrix input, mixed-precision allocation, outlier detection and retention, K-means quantization weights and outlier weights with full precision retained.

[0162] As Figure 2 shown, the clustering process of a single-column weight is shown, including the original distribution, clustering center generation, and quantization index mapping. W is the original weight, Q is the quantized weight, C is the K-means clustering center, is the dequantized weight.

[0163] Embodiment 1: Quantization of the Whisper-Large-V3 Model

[0164] 1. Steps:

[0165] K-means clustering: Group the weight matrix by column, and use the K-means algorithm to generate 2 n center points for each column (n is the quantization bit number).

[0166] Mixed-precision allocation: Calculate the outlier ratio R j for each column. If R j > 13%, allocate 4-bit precision, otherwise allocate 2-bit.

[0167] Outlier retention: In columns with dense outliers (proportion > 10%), retain the top 5% of the outliers in the sparse FP32 format.

[0168] Weight update: Adopt the GPTQ strategy to optimize the unquantized weights and minimize the overall error.

[0169] 2. Experimental results:

[0170] 2.1 2-bit quantization: The WER of the LibriSpeech test set is 2.19% (2.02% in full precision), and the model size is compressed to 1.7% of the original.

[0171] Cross-dataset robustness: The performance loss is less than 1% on datasets such as TED-LIUM (3.67% WER) and VoxPopuli (7.93% WER).

[0172] Example 2: Quantization of the Wav2Vec2 model

[0173] 1. Steps: The same as in Example 1

[0174] 2. Experimental results: In 12-bit quantization, the WER of the LibriSpeech test set is 9.62% (6.11% in full precision), which is a 57% improvement compared to pure 2-bit quantization.

[0175] The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations based on the concept of the present invention without creative labor. Therefore, all technical solutions that can be obtained by those skilled in the art in the technical field based on the concept of the present invention through logical analysis, reasoning, or limited experiments on the basis of the prior art should fall within the protection scope determined by the claims.

Claims

1. A method for quantizing a large speech model after ultra-low bit training, characterized in that: The following steps are involved: Step 1: Weight matrix preprocessing: extract the weight matrix and group it by columns; Step 2: K-means clustering quantization: Perform K-means clustering on each column weight to generate quantized center points; Step 3: Outlier detection and mixed precision allocation: Dynamically allocate quantization precision according to outlier density; Step 4, Selective outlier retention: retain key outliers in sparse FP32 format; Step 5: Model reasoning and performance evaluation: Perform reasoning on the quantized model and evaluate its performance.

2. The large-scale speech model ultra-low-bit post-training quantization method according to claim 1, characterized in that: The step 1 comprises the following steps: Step 1.1, weight extraction: extract the weight matrix from the pre-trained large speech model; Step 1.2, column grouping: Group the weight matrix by columns, with each column as an independent quantization group.

3. The large-scale speech model ultra-low-bit post-training quantization method as claimed in claim 1, characterized in that: The K-means clustering divides the data set into k clusters, maximizing the similarity of data within a cluster and maximizing the difference between clusters.

4. The large-scale speech model ultra-low-bit post-training quantization method as claimed in claim 1, characterized in that: The step 2 comprises the following steps: Step 2.1, center point generation: for each column weight, use K-means clustering algorithm to generate cluster center points; Step 2.2, quantization mapping: quantize the original weight value into the cluster center point index; Step 2.3, Error optimization: Update the unquantized weights.

5. The large-scale speech model ultra-low-bit post-training quantization method as claimed in claim 4, characterized in that: The K-means clustering algorithm includes the following steps: Step 2.1.1, Initialization: Randomly select k initial cluster centers; Step 2.1.2, Assignment: Assign each data point to the nearest cluster center; Step 2.1.3, Update: Recalculate the center point of each cluster; Step 2.1.4, Iteration: Repeat the assignment and update steps until the cluster center no longer changes significantly or the maximum number of iterations is reached.

6. The large-scale speech model ultra-low-bit post-training quantization method as claimed in claim 4, characterized in that: In step 2.3, the unquantized weights are updated using the GPTQ strategy.

7. The large-scale speech model ultra-low-bit post-training quantization method as claimed in claim 1, characterized in that: In step 3, the number of quantization bits is dynamically allocated according to the density of outliers at the column level: 4 bits are allocated to columns with dense outliers, and 2 bits are allocated to other columns.

8. The large-scale speech model ultra-low-bit post-training quantization method as claimed in claim 1, characterized in that: The step 4 comprises the following steps: Step 4.1, outlier screening: in the outlier dense column, retain the top 5% of outliers in sparse FP32 format; Step 4.2, weight recovery: superimpose the sparse FP32 outliers with the quantized weights to restore the original weight distribution.

9. The large-scale speech model ultra-low-bit post-training quantization method as claimed in claim 1, characterized in that: The step 5 comprises the following steps: Step 5.1, model generation: combine the quantized weight matrix with the sparse FP32 outliers to generate a low-bit model; Step 5.2: Performance evaluation.

10. A large-scale speech model ultra-low-bit post-training quantization system, characterized in that: It includes weight matrix extraction module, K-means clustering module, mixed precision allocation module, outlier retention module and model inference module; The weight matrix extraction module extracts the weight matrix from the pre-trained large-scale speech model; The K-means clustering module performs K-means clustering on the weight matrix by column to generate quantized center points; The mixed precision allocation module dynamically allocates quantization precision according to the outlier density; The outlier retention module retains key outliers in sparse FP32 format; The model inference module infers the quantized model and evaluates its performance.

Images