A large language model quantization method based on channel arrangement and activation adaptive smoothing

Abstract

A large language model quantization method based on channel arrangement and activation adaptive smoothing belongs to the technical field of large language model compression and efficient inference. In order to solve the problem that the existing post-training quantization method is difficult to simultaneously consider the difference of different channel activation outliers, the group mode quantization has large dynamic range in the group, and the low bit weight and high bit activation alignment efficiency is low, which leads to the problem of precision decline and inference efficiency limitation, the scheme calibrates the maximum activation amplitude of the input channel by statistical data, divides the significant and non significant activation outlier channels according to the percentile threshold, and respectively uses the logarithmic domain and power function domain smoothing strategy to generate channel smoothing scaling factor, and fuses it into the model parameter to extract the weight multidimensional feature to construct the channel quantization sensitivity index and execute the input channel rearrangement, and further generate the joint quantization scaling factor to realize the integer domain alignment inference. It is suitable for low bit compression deployment and efficient inference scene of large language model in resource limited environment.

Description

Technical Field

[0001] This invention relates to the field of large language model compression technology, and in particular to a large language model quantization scheme based on channel arrangement and activation adaptive smoothing. Background Technology

[0002] As the scale of large language models evolves from billions to tens or even hundreds of billions of parameters, the demand for GPU memory and computing bandwidth during the inference stage increases dramatically. Model deployment is gradually extending from data centers to edge computing and resource-constrained environments, making model compression and efficient inference techniques a current research hotspot. Among numerous compression techniques, post-training quantization has attracted widespread attention because it can achieve model compression without requiring or requiring only a small amount of calibration data and without retraining. Existing research typically employs weight quantization, activation quantization, or a combination of weight and activation quantization to compress floating-point models into low-bit integer models. For example, it uses an 8-bit weight and 8-bit activation quantization scheme, or further compresses it to a 4-bit weight and 8-bit activation W4A8 quantization scheme, to achieve a balance between accuracy and compression ratio.

[0003] To address the issue of large dynamic range in activation value distribution within large language models, existing techniques have proposed activation smoothing or outlier suppression methods. These methods scale activation values ​​before quantization to reduce the amplifying effect of extreme outliers on quantization errors. However, existing smoothing methods typically apply a uniform or near-uniform smoothing intensity to all channels, making it difficult to account for significant differences in outlier amplitude between different layers and channels. The suppression effect is limited in extreme outlier channels, while in non-extreme channels, it may over-scale, altering the original distribution structure. Furthermore, activation smoothing often requires applying inverse scaling to subsequent weights to maintain computational equivalence. This approach changes the original distribution of weight channels, causing the previously relatively uniform weight distribution to locally form new anomalous channels, increasing the error risk in subsequent low-bit weight quantization.

[0004] In terms of weighted quantization, existing technologies mostly employ group quantization with a fixed group size, reducing storage overhead by sharing scaling factors. However, when there are significant differences in channel values ​​within a group, the dynamic range within the group can be dominated by extreme values, thus amplifying the overall quantization error. Some solutions attempt to reduce the overall quantization error by simply sorting or rearranging channels by numerical value, but they lack a system modeling mechanism based on quantization sensitivity, making it difficult to effectively reduce differences in data distribution within groups. Furthermore, in scenarios involving joint quantization of low-bit weights and high-bit activations, existing technologies typically calculate scaling factors separately. This requires multiple format conversions or floating-point calculations during the inference phase, increasing computational complexity and hindering the full utilization of the efficiency advantages of integer matrix multiplication kernels.

[0005] In summary, existing technologies have several drawbacks, including the inability to implement differentiated smoothing for outlier amplitudes in different channels, the tendency of activation smoothing to introduce weight distribution imbalances, the difficulty in reducing the dynamic range within a group through group quantization, and the difficulty in achieving efficient alignment calculations between low-bit weights and high-bit activations. Summary of the Invention

[0006] To address the shortcomings of existing technologies, such as the inability to perform differentiated smoothing for outlier amplitudes in different channels, the tendency of activation smoothing to introduce weight distribution imbalance, the difficulty in reducing the dynamic range within a group due to group quantization, and the difficulty in achieving efficient alignment calculation between low-bit weights and high-bit activations, the technical solution provided by this invention is as follows: A quantization method for large language models based on channel permutation and activation adaptive smoothing includes: The steps involve using calibration data to drive forward inference of a large language model, statistically analyzing the maximum activation amplitude of the input activation values ​​in each linear layer along the input channel dimension, determining a significant activation outlier threshold based on the distribution of the maximum activation amplitude using percentile parameters, dividing the input channels into significant activation outlier channels and non-significant activation outlier channels, generating a smoothing scaling factor using a logarithmic domain smoothing strategy for significant activation outlier channels, and generating a smoothing scaling factor using a power function domain smoothing strategy in combination with the corresponding weight channel amplitude for non-significant activation outlier channels, and then fusing the generated channel smoothing scaling factor into the normalized layer parameters and applying a corresponding inverse scaling to the weights of subsequent linear layers to obtain a smoothed weight matrix. The steps are as follows: extract multidimensional numerical features of the input channels based on the smoothed weight matrix and construct a channel quantization sensitivity index; sort the input channels according to the channel quantization sensitivity index to generate a channel arrangement index; and perform a consistent rearrangement of the weight matrix and the input activation channels in the inference stage according to the channel arrangement index to obtain the structurally rearranged weight matrix. Based on the weight matrix after the structural rearrangement, a quantization scheme is adopted, which uses 4-bit group weight quantization and 8-bit per tag activation quantization. The group quantization scaling factor and the output channel quantization scaling factor are calculated to generate a joint quantization scaling factor. In the inference stage, the joint quantization scaling factor is used to convert the 4-bit weights into integer representations aligned with the 8-bit activations and call the integer matrix multiplication kernel to perform the operation, thus realizing the steps of low-bit high-efficiency inference.

[0007] Furthermore, a preferred embodiment is provided, wherein the process of determining the significant activation outlier threshold according to the percentile parameter includes sorting the maximum activation amplitude of each linear layer input channel by value and selecting the value located at a preset percentage position as the significant activation outlier threshold.

[0008] Furthermore, a preferred embodiment is provided, wherein the process of employing a logarithmic domain smoothing strategy on significantly activated outlier channels includes performing a logarithmic mapping on the activation magnitude of the significantly activated outlier channels and generating a first smoothing scaling factor.

[0009] Furthermore, a preferred implementation is provided, wherein the process of applying a power function domain smoothing strategy to non-significantly activated outlier channels includes determining the power exponent parameter by combining the amplitude information of the corresponding weighted channel and generating a second smoothing scaling factor.

[0010] Furthermore, a preferred implementation method is provided, in which the process of constructing the channel quantization sensitivity index includes extracting the channel mean, channel variance and channel norm of the input channel, normalizing each feature and then weighting and combining them according to preset weights to generate the channel quantization sensitivity index.

[0011] Furthermore, a preferred embodiment is provided, wherein the process of uniformly rearranging the weight matrix and the input activation channels in the inference stage according to the channel arrangement index includes persistently storing the channel arrangement index as a model parameter, and reading the channel arrangement index in the inference stage to perform channel rearrangement in the same order for the input activations.

[0012] Based on the same inventive concept, this invention also provides a large language model quantization device based on channel arrangement and activation adaptive smoothing, comprising: The large language model is driven by calibration data to perform forward inference. The maximum activation amplitude of the input activation value of each linear layer is statistically analyzed in the input channel dimension. Based on the distribution of the maximum activation amplitude, the significant activation outlier threshold is determined by percentile parameter. The input channel is divided into significant activation outlier channel and non-significant activation outlier channel. For significant activation outlier channel, a logarithmic domain smoothing strategy is used to generate a smoothing scaling factor. For non-significant activation outlier channel, a power function domain smoothing strategy is used in combination with the corresponding weight channel amplitude to generate a smoothing scaling factor. The generated channel smoothing scaling factor is fused into the normalized layer parameters and a corresponding inverse scaling is applied to the weights of subsequent linear layers to obtain the module of the smoothed weight matrix. Based on the smoothed weight matrix, multidimensional numerical features of the input channels are extracted and a channel quantization sensitivity index is constructed. The input channels are sorted according to the channel quantization sensitivity index to generate a channel arrangement index. The weight matrix and the input activation channels in the inference stage are uniformly rearranged according to the channel arrangement index to obtain the module of the structurally rearranged weight matrix. Based on the weight matrix after the structural rearrangement, a quantization scheme is adopted, which uses 4-bit group weight quantization and 8-bit per tag activation quantization. The group quantization scaling factor and the output channel quantization scaling factor are calculated to generate a joint quantization scaling factor. During the inference stage, the joint quantization scaling factor is used to convert the 4-bit weights into integer representations aligned with the 8-bit activations and call the integer matrix multiplication kernel to perform the operation, thus realizing a module for low-bit, high-efficiency inference.

[0013] Based on the same inventive concept, the present invention also provides a computer storage medium for storing a computer program, wherein when the computer program is read by a computer, the computer executes the method described thereon.

[0014] Based on the same inventive concept, the present invention also provides a computer, including a processor and a storage medium, wherein when the processor reads a computer program stored in the storage medium, the computer executes the method described thereon.

[0015] Based on the same inventive concept, the present invention also provides a computer program product, which, when executed, implements the method described.

[0016] Compared with the prior art, the advantages of the technical solution provided by the present invention are as follows: The effect of using data sampled from the calibration dataset as model input for forward inference and recording the activation statistics of each linear layer is that it shifts the main source of quantization error from "setting the smoothing intensity based on experience" to "parameter estimation driven by the actual inference activation distribution." This ensures that subsequent smoothing and quantization target the actual activation dynamic range of the model under the target task and the actual input distribution, rather than an abstract assumed distribution. Compared to many studies that only use a few levels of statistics or only observe the mean and variance, this approach uses layer-by-layer and channel-by-channel statistics from calibration inference as a basis, which can more stably capture sparse extreme activation patterns triggered by specific tokens and contexts in the linear layers of large models, reducing the post-quantization accuracy fluctuations caused by "calibration mismatch" from the source.

[0017] The effect of calculating the maximum activation amplitude across samples for each input channel and using it as an outlier is that it locates the outlier risk at the channel granularity rather than the layer granularity or the overall tensor granularity. This allows subsequent processing to accurately address the core contradiction of "a large dynamic range caused by a small number of channels." Compared to some methods that use the overall maximum value or a few statistics to represent the outlier degree of the entire layer, this approach explicitly models the outlier amplitude with the channel maximum value. This can separate the channel that contributes the most to the quantization error from a large number of normal channels, avoiding information compression and accuracy loss caused by scaling the entire layer indiscriminately.

[0018] The effect of dynamically determining the significant outlier threshold based on the distribution of the maximum activation value set of channels using percentile parameters is that it makes the definition of "outlier" adaptively change with the layer distribution, avoiding the failure of fixed thresholds between different layers. Thus, it can still stably identify the real extreme channels even when there are significant differences in activation scales between shallow, medium and deep layers. Compared with common fixed thresholds or fixed-ratio pruning, this scheme makes relative discrimination within each layer using percentiles, which can cover the phenomenon that the outlier intensity changes with the layer and reduce the sensitivity of threshold selection to different model architectures and different datasets.

[0019] The effect of dividing channels into significantly activated outlier channels and non-significantly activated outlier channels based on thresholds is to provide structural conditions for subsequent differentiated processing, allowing the smoothing intensity and smoothing domain to be designed separately for the two types of channels. This addresses the contradiction that a uniform smoothing intensity cannot accommodate both extreme and ordinary outliers. Compared to many studies that uniformly use the same smoothing or pruning strategy for all channels, this approach reduces the side effect of "sacrificing a large amount of ordinary channel resolution to suppress extreme outliers" through significant diversion.

[0020] The effect of using a logarithmic domain smoothing strategy to calculate the smoothing scaling factor for significantly activated outlier channels is that it achieves stronger amplitude compression for extreme outlier channels while maintaining relative monotonicity. This significantly flattens the dynamic range of extreme channels before entering low-bit activation quantization, thereby reducing saturation, truncation, and large-step quantization errors in these channels during 8-bit activation quantization. Compared to linear scaling or gentle pruning, the "strong compression and weak order preservation" of the logarithmic domain is more suitable for handling a small number of huge outliers without excessively distorting the relative changes of the non-outlier intervals in the channel.

[0021] The effect of combining the magnitude information of the corresponding weight channels with the power function domain smoothing strategy for non-significantly activated outlier channels is that it expands the smoothing from simply "looking at activation" to a "coupling of activation and weight" decision. This allows ordinary outlier channels to obtain the necessary suppression while minimizing the impact of reverse scaling on the weight side, reducing the problems of weight distribution being amplified and subtle differences being exaggerated due to smoothing. Compared with the scheme that also uses strong smoothing for non-extreme channels, this scheme controls the smoothing intensity with a gentler and more adjustable power function domain scaling, making activation quantization easier without significantly creating new difficulties in weight quantization.

[0022] The effect of summarizing the smoothing scaling factors of each channel to form a channel smoothing parameter set is that it improves smoothing from a "single scalar" to a "channel vector parameter", so that each input channel has a corresponding scaling factor, thereby achieving fine-grained adjustment within the same linear layer. Compared with some studies that only give one scaling factor per layer or one scaling factor per group, channel-level parameters significantly reduce the problem of channel differences being masked by the same factor, making outlier suppression more accurate and reducing redundant damage.

[0023] The effect of fusing channel smoothing parameters into the parameters of adjacent normalization layers is that it "compiles" the additional smoothing operations into the existing operator parameters, so that no additional operators or additional multiplication paths are added during inference. This achieves outlier suppression while basically not increasing runtime overhead. Compared with the approach that requires explicit insertion of scaling layers or additional preprocessing operators, this solution keeps the inference graph simple through parameter fusion, which is beneficial for deployment in inference engines with limited resources or sensitive to operator fusion.

[0024] The effect of applying inverse scaling to the weights of subsequent linear layers to maintain computational equivalence is that it completes the activation distribution shaping without changing the mathematical function output of the model. This ensures that the smoothed model is consistent with the original model in floating-point semantics, thus limiting the performance change to "reduction of quantization error" rather than "rewriting of model function". At the same time, this inverse scaling will change the weight distribution, which is the premise for the subsequent introduction of channel arrangement and joint scaling factor in this scheme, so that the whole method forms a closed loop before and after, rather than isolated activation smoothing.

[0025] The effect of arranging the input channels based on weight numerical features after completing the activation adaptive smoothing is that the channel arrangement faces the "true weight distribution after inverse scaling". Therefore, the arrangement reflects the weight quantization risk introduced by smoothing rather than the original weight risk, which can more effectively mitigate the side effects of smoothing. Compared with the approach of arranging first and then smoothing or only arranging the original weights, this scheme places the arrangement after smoothing, making it a compensation mechanism for the side effects of smoothing, and the overall quantization error is more controllable.

[0026] The benefit of extracting multidimensional statistics such as channel mean, channel variance, and channel norm for each input channel is that it characterizes the channel distribution from multiple perspectives, avoiding quantization sensitivity misjudgments caused by the inability of a single indicator to reflect complex patterns such as "flat but with peaks" or "small mean but high energy". Compared with simplified solutions that only look at the maximum value or only look at the norm, multidimensional features can more stably approximate the comprehensive sensitivity of the channel to quantization step size, saturation, and rounding errors, thus providing a more reliable ranking basis for subsequent re-ranking.

[0027] The effect of normalizing the features is to eliminate the weight bias caused by the difference in the dimensions of different statistical quantities, so that features such as mean, variance, and norm can be combined on the same scale, avoiding the "dominance" of a sensitivity index due to the large range of values. Compared with the approach of directly weighting without normalization, this scheme reduces the instability caused by the difference in statistical scale between different model layers, making the ranking results more comparable and transferable across layers.

[0028] The effect of using normalized multidimensional feature weighted modeling to obtain the channel quantization sensitivity index is that it upgrades the channel ranking objective from "similar values" to "similar quantization risk," making adjacent channels more likely to have similar dynamic ranges and error sensitivities when entering group quantization, thereby reducing the probability of being dragged down by extreme channels within the same quantization group. Compared with simple rearrangement such as sorting by amplitude or by maximum value, the sensitivity index of this scheme directly addresses quantization error, which can more effectively reduce intra-group dispersion and reduce the amplification of overall error by the group quantization scaling factor.

[0029] The effect of sorting the channels in descending order by the quantization sensitivity index and obtaining the channel arrangement index is that it forms a reusable and persistent static mapping, which enables the optimal channel order obtained in the quantization stage after training to be stably reproduced in the inference stage without relying on runtime statistics or dynamic decision-making, thereby ensuring the consistency of inference results and deployment determinism. Compared with solutions that require online clustering or runtime adaptive grouping, this solution achieves deterministic rearrangement with offline indexing, resulting in lower deployment complexity.

[0030] The effect of storing channel arrangement indexes on model parameters is that it incorporates rearrangement rules into the model file, ensuring that the same quantized model maintains consistent channel alignment when loaded on different inference frameworks and hardware, thus avoiding the engineering risk that channel rearrangement is "only effective in a certain implementation." Compared to hard-coding rearrangement only on the code side, parameterized indexes facilitate version management, model distribution, and the adoption of different indexing strategies at different layers.

[0031] The effect of arranging the input channels of the weight matrix according to the channel arrangement index is that it directly changes the local adjacency relationship of the weight tensor in the input dimension. This makes subsequent group quantization more inclined to group channels with similar sensitivities into the same group under the same group size configuration, thereby reducing the dynamic range within the group and reducing the quantization damage of the group scaling factor to ordinary channels. Compared with the case of "high-risk and low-risk channels mixed in a group" when not rearranged, the consistency within the group is higher after rearrangement, and the W4 weight quantization error is significantly reduced, especially when smooth reverse scaling introduces new weight outliers.

[0032] The effect of reading the stored channel arrangement index and arranging the input activations in the same way during the inference phase is to ensure that the activation input channels and weight input channels maintain a channel-by-channel correspondence after rearrangement, thereby achieving "joint alignment of weights and activations" without changing the network semantics. Compared to the risk of channel mismatch and output errors caused by rearranging only weights without rearranging activations, this solution achieves strict consistency by synchronously rearranging activations during inference, ensuring that rearrangement only affects quantization errors without compromising computational correctness.

[0033] The W4A8 configuration, which uses 4-bit weight quantization and 8-bit activation quantization per tag, significantly reduces weight storage and bandwidth while maintaining high accuracy on the activation side. This aligns with the engineering reality that activations are sensitive to output and weights account for a large portion of storage in large language model inference. Compared to W8A8, this configuration saves more GPU memory and increases throughput potential. Compared to the more aggressive W4A4, it reduces the damage to perplexity and generation quality caused by activation-side quantization. The outlier smoothing and channel rearrangement in this scheme further reduce the accuracy risk of W4A8 to a manageable range.

[0034] The effect of determining the group size for group quantization and quantizing the weights is that it provides an adjustable knob between compression ratio and error. Larger groups are more beneficial for compression and throughput but have a higher risk of error, while smaller groups have lower error but higher overhead. This solution performs sensitivity-driven rearrangement first, which makes the intra-group differences smaller under the same group size. This allows for the selection of a group size that is more beneficial to performance in engineering without significantly sacrificing accuracy. It is more "tolerant of large groups" than unrearranged group quantization.

[0035] The effect of calculating the group quantization scaling factor for each group and the channel quantization scaling factor for each output channel is that it captures two types of information: the shared scale within the group and the overall scale of the output channel. This allows for more refined scaling control in subsequent fusion, avoiding the neglect of scale differences between output channels caused by simple group scaling. Compared to calculating only group scaling, introducing output channel scaling can provide a more accurate scale benchmark when aligning weight dequantization and requantization, reducing error propagation caused by coarse scale estimation.

[0036] The effect of combining the group quantization scaling factor and the channel quantization scaling factor to obtain the joint quantization scaling factor is to minimize the scaling error when migrating the weights from 4-bit representation to 8-bit aligned representation, so that the weights can enter the integer matrix multiplication path at a scale closer to the real floating-point weights. Compared with the scheme of using two scaling factors separately and correcting them multiple times during inference, the joint factor solidifies the scale relationship in advance, reduces runtime calculation and error accumulation, and provides a unified scale interface for "fast alignment of 4-bit weights to 8-bit activation".

[0037] The effect of compressing and storing the weights by quantizing them in groups using 4 bits is that it significantly reduces the size of the model parameters and the memory usage, allowing large models to accommodate larger contexts, larger batches, or more concurrent requests on the same hardware, while reducing memory bandwidth pressure. Compared to only performing 8-bit quantization or mixed precision storage, this solution reduces the main storage overhead to 4 bits and reduces the precision cost of 4 bits through pre-smoothing and rearrangement.

[0038] The effect of calculating the activation quantization scaling factor and performing 8-bit quantization for each token during the inference process is that it improves the activation quantization from "shared scale across the entire sequence" to "adaptive scale per token". This allows for timely tracking of changes in activation amplitude caused by different tokens and different contexts, reducing saturation and insufficient resolution caused by a single scale not being able to cover the dynamic range. Compared to the approach of one activation scale per layer or one activation scale per batch, the per-token scale is more in line with the fact that the activation distribution changes with position during autoregressive generation, which can further stabilize the generation quality.

[0039] The effect of performing fast format conversion on 4-bit quantization weights and using a joint quantization scaling factor to dequantize to floating-point and then requantize to 8-bit integers is that it unifies the weights and activations to the 8-bit integer field, eliminating the format incompatibility problem when low-bit weights directly participate in integer GEMMs, and avoiding prolonged stays in the floating-point field in the main inference path. Compared to solutions that require floating-point matrix multiplication or dedicated mixed-width kernels, this mechanism allows a general-purpose int8 kernel to handle the main computations, resulting in greater engineering usability and wider hardware coverage.

[0040] The advantage of using an 8-bit integer general-purpose matrix multiplication kernel to perform matrix operations and obtain integer output is that it maximizes the use of existing hardware's high-throughput support for int8 GEMM, thereby saving video memory after quantization and achieving considerable acceleration in computation. Compared with solutions that require custom W4-specific kernels or rely on vendor-specific instruction sets, the general-purpose int8 kernel is easier to implement in various inference frameworks and accelerator cards, with lower deployment thresholds and more stable performance.

[0041] The benefit of dequantizing the integer output back to floating-point and continuing to execute subsequent network layers is that it connects the quantization calculation with subsequent modules as a standard floating-point interface, ensuring compatibility with existing model structures and inference graphs. This avoids the introduction of additional operator replacements and precision uncertainties due to full-link integerization. Compared to full-link integerization, which requires rewriting a large number of operators, this solution concentrates the benefits of quantization on the main time-consuming path of matrix multiplication. In engineering, it is easier to combine with attention, normalization, nonlinearity, and other modules to achieve a balance between accuracy and deployability.

[0042] The benefits of these two persistent designs—saving the smoothed model and saving the rearranged model—are that the smoothing parameter fusion results and channel rearrangement indexes are both solidified into the model entity, forming a repeatable and acceptable quantitative process output. This makes it easy to reproduce experimental results when deployed on different hardware and in different batches, and also facilitates subsequent comparisons of ablation with different thresholds, group sizes, and weighted feature combinations. Compared to processes that only temporarily process in the script and do not produce reusable model files, this solution is closer to an engineering mass production path.

[0043] The overall effect of combining the hybrid numerical smoothing strategy with the multi-dimensional feature-aware channel arrangement strategy is to simultaneously reduce the quantization error on the activation side and the quantization error on the weight side. Furthermore, it compensates for the potential side effects of weight distribution introduced by activation smoothing through "smoothing and then rearranging," enabling W4A8 to maintain more stable accuracy and smaller inter-layer variability even under low bit weight conditions. Compared with studies that only focus on activation smoothing, this scheme additionally solves the key contradiction that smoothing is detrimental to weights. Compared with studies that only focus on weight rearrangement or only focus on group quantization optimization, this scheme first shapes the activation outlier root cause, thereby reducing the error pressure of all subsequent quantization steps and forming a complete error control chain from outlier suppression to intra-group consistency to integer kernel alignment.

[0044] It is suitable for working scenarios where large language models are deployed with low bit compression and high efficiency inference in edge devices or memory-constrained environments. Attached Figure Description

[0045] Figure 1 Flowchart of a large language model quantization method based on channel arrangement and activation adaptive smoothing; Figure 2 Flowchart for activating the adaptive smoothing strategy; Figure 3 Flowchart of channel arrangement strategy for multidimensional feature perception; Figure 4 Quantization flowchart for W4A8. Detailed Implementation

[0046] To make the advantages and benefits of the technical solution provided by the present invention clearer, the technical solution provided by the present invention will now be described in further detail with reference to the accompanying drawings, specifically: Implementation Method 1: This implementation method provides a large language model quantization method based on channel arrangement and activation adaptive smoothing, including: The steps involve using calibration data to drive forward inference of a large language model, statistically analyzing the maximum activation amplitude of the input activation values ​​in each linear layer along the input channel dimension, determining a significant activation outlier threshold based on the distribution of the maximum activation amplitude using percentile parameters, dividing the input channels into significant activation outlier channels and non-significant activation outlier channels, generating a smoothing scaling factor using a logarithmic domain smoothing strategy for significant activation outlier channels, and generating a smoothing scaling factor using a power function domain smoothing strategy in combination with the corresponding weight channel amplitude for non-significant activation outlier channels, and then fusing the generated channel smoothing scaling factor into the normalized layer parameters and applying a corresponding inverse scaling to the weights of subsequent linear layers to obtain a smoothed weight matrix. The steps are as follows: extract multidimensional numerical features of the input channels based on the smoothed weight matrix and construct a channel quantization sensitivity index; sort the input channels according to the channel quantization sensitivity index to generate a channel arrangement index; and perform a consistent rearrangement of the weight matrix and the input activation channels in the inference stage according to the channel arrangement index to obtain the structurally rearranged weight matrix. Based on the weight matrix after the structural rearrangement, a quantization scheme is adopted, which uses 4-bit group weight quantization and 8-bit per tag activation quantization. The group quantization scaling factor and the output channel quantization scaling factor are calculated to generate a joint quantization scaling factor. In the inference stage, the joint quantization scaling factor is used to convert the 4-bit weights into integer representations aligned with the 8-bit activations and call the integer matrix multiplication kernel to perform the operation, thus realizing the steps of low-bit high-efficiency inference.

[0047] The process of determining the significant activation outlier threshold based on percentile parameters includes sorting the maximum activation amplitude of each linear layer input channel by value and selecting the value located at a preset percentage position as the significant activation outlier threshold.

[0048] The process of applying a logarithmic domain smoothing strategy to significantly activated outlier channels includes performing a logarithmic mapping on the activation magnitude of the significantly activated outlier channels and generating a first smoothing scaling factor.

[0049] The process of using a power function domain smoothing strategy for non-significantly activated outlier channels includes determining the power exponent parameter by combining the amplitude information of the corresponding weighted channel and generating a second smoothing scaling factor.

[0050] The process of constructing the channel quantization sensitivity index includes extracting the channel mean, channel variance, and channel norm of the input channels, normalizing each feature, and then weighting and combining them according to preset weights to generate the channel quantization sensitivity index.

[0051] The process of consistently rearranging the weight matrix and the input activation channels in the inference phase based on the channel arrangement index includes persistently storing the channel arrangement index as a model parameter, and reading the channel arrangement index in the inference phase to perform channel rearrangement in the same order on the input activations.

[0052] A large language model quantization device based on channel permutation and activation adaptive smoothing is also provided, including: The large language model is driven by calibration data to perform forward inference. The maximum activation amplitude of the input activation value of each linear layer is statistically analyzed in the input channel dimension. Based on the distribution of the maximum activation amplitude, the significant activation outlier threshold is determined by percentile parameter. The input channel is divided into significant activation outlier channel and non-significant activation outlier channel. For significant activation outlier channel, a logarithmic domain smoothing strategy is used to generate a smoothing scaling factor. For non-significant activation outlier channel, a power function domain smoothing strategy is used in combination with the corresponding weight channel amplitude to generate a smoothing scaling factor. The generated channel smoothing scaling factor is fused into the normalized layer parameters and a corresponding inverse scaling is applied to the weights of subsequent linear layers to obtain the module of the smoothed weight matrix. Based on the smoothed weight matrix, multidimensional numerical features of the input channels are extracted and a channel quantization sensitivity index is constructed. The input channels are sorted according to the channel quantization sensitivity index to generate a channel arrangement index. The weight matrix and the input activation channels in the inference stage are uniformly rearranged according to the channel arrangement index to obtain the module of the structurally rearranged weight matrix. Based on the weight matrix after the structural rearrangement, a quantization scheme is adopted, which uses 4-bit group weight quantization and 8-bit per tag activation quantization. The group quantization scaling factor and the output channel quantization scaling factor are calculated to generate a joint quantization scaling factor. During the inference stage, the joint quantization scaling factor is used to convert the 4-bit weights into integer representations aligned with the 8-bit activations and call the integer matrix multiplication kernel to perform the operation, thus realizing a module for low-bit, high-efficiency inference.

[0053] A computer storage medium is also provided for storing a computer program, which, when read by the computer, executes the method.

[0054] A computer is also provided, including a processor and a storage medium, wherein the computer executes the method when the processor reads a computer program stored in the storage medium.

[0055] A computer program product is also provided, which, when executed, implements the method described.

[0056] Implementation Method Two, in conjunction with Appendix Figure 1-4 This embodiment is a further detailed description of the technical solution provided in Embodiment 1. Specifically: This solution is designed for post-training quantization scenarios of large language models. Without relying on retraining or requiring only a small amount of calibration data, it first suppresses activation outliers, then reconstructs the weight input channel structure, and finally completes efficient deployment using an integer domain inference link with four-bit weights and eight-bit activations.

[0057] Combination Figure 1 The process described first involves calibration-driven acquisition of activation outlier information and adaptive smoothing. This requires a calibration dataset, which does not need to be labeled but should represent the target inference distribution. This dataset is typically obtained by random sampling from actual corpora or target task data. The sampling size can be selected based on project resources, consisting of several batches of text sequences, which are then converted into labeled sequences by the model's word segmenter to form the input batch. This input batch is fed into the original large language model for forward inference. During inference, the input activation of each linear layer in the model is statistically recorded, with the basic unit of statistics based on the input channel. For any linear layer, the input activation can be considered as a set of channel activations generated by several samples and several labeled positions. This approach does not store all activations in full; instead, it calculates the maximum activation amplitude across samples and labeled positions for each input channel and combines the maximum activation amplitudes of each channel into a set of maximum channel activation values ​​for that linear layer. This set captures the extreme amplitudes of outlier channels while avoiding statistical noise being diluted by the average, and also makes storage and computational costs more controllable.

[0058] Combination Figure 2 The adaptive smoothing strategy described above, after obtaining the set of maximum activation values ​​for a linear layer's channels, dynamically determines a significant outlier threshold based on the distribution of this set. The threshold is determined using a percentile parameter; that is, the maximum activation values ​​of the channels are sorted in ascending order, and the value at the preset percentile is taken as the threshold for that layer. Subsequently, the channels are divided. If the maximum activation amplitude of a channel is not lower than the threshold, the channel is marked as a significantly activated outlier channel; otherwise, it is marked as a non-significantly activated outlier channel. This division separates extreme outliers from general outliers, allowing for differentiated configuration of subsequent smoothing intensity, thereby avoiding the problems of insufficient or excessive suppression caused by uniform smoothing across all channels.

[0059] For significantly activated outlier channels, a strong logarithmic domain smoothing strategy is employed to calculate the channel smoothing scaling factor. This can be understood as first mapping the maximum activation amplitude of the channel to logarithmic space, then determining the scaling intensity based on the target compression scale. This ensures that extreme outlier amplitudes are compressed more significantly before entering low-bit quantization, while maintaining the monotonicity of the channel's values. For insignificantly activated outlier channels, a power-law domain smoothing strategy is used, incorporating the amplitude information of the corresponding weight channel into the smoothing intensity selection. In implementation, the activation amplitude of the channel can be power-law transformed to achieve gentle compression, with the power parameter set to less than one to achieve the desired compression effect. Simultaneously, considering the amplitude range of the corresponding column vector in the subsequent weight matrix, if the weight channel itself has a large amplitude, the activation smoothing intensity is appropriately reduced to lessen the impact of inverse scaling on the weight distribution; if the weight channel has a small amplitude, a slight enhancement in activation smoothing is allowed to reduce the difficulty of activation quantization. The resulting set of channel smoothing scaling factors consists of channel-by-channel vector parameters, ultimately summing up as the smoothing parameters for the linear layer.

[0060] After obtaining the smoothing parameters, they need to be integrated into the model without increasing the overhead of the inference operators. In implementation, the channel smoothing parameters are fused into the parameters of adjacent normalized layers, allowing them to take effect without the need for additional explicit scaling operators during inference. Since activation-side scaling alters the computation results of subsequent linear layers, to maintain computational equivalence, a corresponding inverse scaling needs to be applied to the weights of subsequent linear layers. This involves adjusting the input channel dimension of the weight matrix inversely at the channel level, ensuring that the overall linear transformation remains consistent in floating-point semantics before and after scaling. After completing the above fusion and inverse scaling, the smoothed model parameter state is obtained and used as the input for the subsequent channel arrangement stage.

[0061] Enter after smoothing Figure 3 The multidimensional feature perception channel arrangement stage shown aims to proactively reconstruct the adjacency relationships of the input channels of the weight matrix to address the introduced weight distribution changes during smoothing, thereby reducing the intra-group dynamic range and differences during subsequent group quantization. In implementation, multidimensional numerical features are extracted channel by channel along the input channel dimension, using the weight matrix of each linear layer as the object. At least the channel mean, channel variance, and channel norm need to be extracted. The channel mean reflects the overall bias level of the channel weights, the channel variance reflects the dispersion of the channel weights, and the channel norm, obtained by taking the square root of the sum of the squares of the elements of the channel weights, reflects the channel energy scale. To avoid the influence of differences in the dimensions of different statistical measures on the combination, each feature needs to be normalized separately. For example, the set of values ​​of a certain feature across all channels of the layer is centered and scaled to fall within a uniform scale range.

[0062] The normalized multidimensional features are then weighted and combined according to preset weights to obtain a single channel quantization sensitivity index. This index comprehensively characterizes the channel's error sensitivity under low-bit quantization. The weight coefficients can be set based on experience or offline evaluation to balance the contributions of mean, variance, and norm to quantization error. After obtaining the quantization sensitivity index for each channel, the channels are sorted from largest to smallest index value to generate a channel permutation index. This index needs to be persistently stored in the model as a static parameter to ensure that the reordering rules can be reproduced during inference deployment. Subsequently, the input channels of the weight matrix are rearranged according to this index, so that channels with similar quantization sensitivities are stored adjacently. This makes them more likely to be assigned to the same quantization group when performing group quantization with a fixed group size, reducing the probability that the dynamic range within a group is dominated by extreme channels.

[0063] Channel rearrangement not only needs to be applied to weights but also requires consistent channel rearrangement of input activations during inference to maintain a one-to-one correspondence. In implementation, the persistent channel arrangement index is read during inference. The input activation tensors entering the linear layer are arranged according to the same index along the channel dimension, and then calculated with the rearranged weight matrix. This ensures that the rearrangement does not change the network semantics, only altering the numerical distribution structure to facilitate quantization. After completing these steps, the rearranged model state is obtained and used as input for the quantization and integer inference stages.

[0064] Finally enter Figure 4 The W4A8 quantization and integer inference stage is shown. During implementation, the quantization configuration is first determined, including whether weights are represented by four bits, activations by eight bits, weight quantization uses group quantization, and the specific value of the group size. Group quantization requires grouping weights according to group size along the input channel dimension, and calculating a group quantization scaling factor for each group. The calculation of the group quantization scaling factor can be expressed as mapping the maximum absolute value of the weights within the group to the upper bound of the range representable by a four-bit integer. That is, the scaling ratio is determined by the ratio between the maximum amplitude and the largest four-bit integer, ensuring that all weights within the group are compressed to the four-bit integer range using the same ratio. Simultaneously, this scheme also needs to pre-calculate the channel quantization scaling factor for each output channel to characterize the overall scale of the output channel. The calculation of the channel quantization scaling factor can be understood as mapping the amplitude range of the output channel weights to the eight-bit integer range.

[0065] After obtaining the group quantization scaling factor and the channel quantization scaling factor, they need to be fused to obtain the joint quantization scaling factor. The role of the joint quantization scaling factor is to support the rapid alignment of four-bit weights to the eight-bit integer domain during the inference stage, so that the weights and eight-bit activations can enter the same integer matrix multiplication kernel. The fusion method can be understood as combining the group scale and the output channel scale into a joint ratio, so that when performing scale restoration and remapping on the weight values ​​of a certain output channel and a certain weight group during inference, alignment can be completed with only one unified ratio processing.

[0066] After scaling factor calculation, the original weights are compressed and stored using 4-bit quantization in groups. During inference, the activations are quantized to 8-bit per tag. Specifically, for each input tag, an activation quantization scaling factor is calculated. This scaling factor can be obtained by the ratio between the maximum absolute value of the tag's activation value and the upper bound of the 8-bit integer representation. This ratio is then used to map the tag's activation to an 8-bit integer. Subsequently, the 4-bit weights are read and quickly converted to their original format. A joint quantization scaling factor is then used to complete the weight alignment process, which involves scaling the 4-bit weights according to the joint ratio and remapping them to an 8-bit integer representation, ensuring the weight bit width matches the activation bit width. Next, the 8-bit integer general matrix multiplication kernel is called to perform matrix multiplication, resulting in an integer output. This integer output is then scaled back according to the scaling ratio of the activations and weights, restoring the result to a floating-point representation, which is then passed to subsequent network layers until the final output is reached. This completes the implementable chain from calibration statistics, blending smoothing, channel rearrangement to W4A8 quantization and integer kernel inference, where the output of each step is explicitly used as the input for the next step.

[0067] Implementation Method 3, in conjunction with Appendix Figure 1-4 This embodiment describes the technical solution provided above in further detail through specific examples. Specifically: Figure 1 This is a flowchart of the large language model quantization method based on channel arrangement and activation adaptive smoothing in this embodiment.

[0068] This large language model quantization method based on channel arrangement and activation adaptive smoothing includes the following steps: Step S1: Use the data sampled from the calibration dataset as input to the large language model and perform forward inference. Record the statistical information of the input activation values ​​of each linear layer, mainly including the maximum activation amplitude of each input channel. Use the maximum activation amplitude of the channel as the outlier and use a hybrid numerical smoothing strategy to suppress the activation outlier. Step S2: Arrange the input channels according to the multiple dimensions of the numerical features of the input channels after smoothing the weight matrix; Step S3: A quantization scheme using 4-bit weight quantization and 8-bit token activation quantization is employed. During the inference phase, a joint quantization scaling factor is used to directly dequantize the weight values ​​to 8-bit representations, and an efficient integer matrix multiplication kernel is used to complete the computation.

[0069] Specifically, the following three embodiments further illustrate the proposed large language model quantization method based on channel arrangement and activation adaptive smoothing.

[0070] Example 1, as Figure 2 As shown, the specific steps to activate the adaptive smoothing strategy are as follows: Step D1: Sample data from the calibration dataset as model input, perform forward inference, and record the input activation values ​​of each linear layer.

[0071] Step D2: Within each linear layer, calculate the maximum activation amplitude across samples for each input channel to form the set of maximum activation values ​​for that layer's channels.

[0072] Step D3: Based on the distribution of the maximum activation set of the channel in this layer, determine the significant outlier threshold using the percentile parameter.

[0073] Step D4: Mark the channels with the maximum activation amplitude not lower than the significant outlier threshold as significant outlier channels, and the rest as non-significant outlier channels.

[0074] Step D5: For significantly activated outlier channels, a strong log-smoothing strategy is adopted.

[0075] Step D6: For non-significantly activated outlier channels, a power function smoothing strategy is adopted by combining the amplitude information of the corresponding weighted channels.

[0076] Step D7: Summarize the smoothing scaling factors of each channel to form a set of channel smoothing scaling factors, which will serve as the smoothing parameters for this layer.

[0077] Step D8: Incorporate the channel smoothing parameters into the parameters of the adjacent normalized layers; to maintain computational equivalence, apply the corresponding inverse scaling to the weights of subsequent linear layers.

[0078] Step D9: Save the smoothed model.

[0079] Example 2, as Figure 3 As shown, the input channels are arranged using a multi-dimensional feature-aware channel arrangement strategy. The specific steps are as follows: Step D1: Obtain the weight matrix after processing and parameter adjustment in Example 1, and perform subsequent analysis with the input channel as the dimension.

[0080] Step D2: Extract multidimensional numerical features for each input channel, including channel mean, channel variance, and channel norm.

[0081] Step D3: Normalize each feature.

[0082] Step D4: Combine the multidimensional features into a single channel quantization sensitivity index.

[0083] Step D5: Sort the channels in descending order according to the channel quantization sensitivity index to obtain the channel arrangement index.

[0084] Step D6: Persistently store the channel arrangement index as a static parameter of this layer.

[0085] Step D7: Arrange the input channels of the weight matrix according to the channel arrangement index.

[0086] Step D8: During the inference phase, read the stored channel arrangement index and arrange the input activations.

[0087] Step D9: Save the arranged model.

[0088] Example 3, as Figure 4 As shown, the specific steps of the W4A8 quantization process are as follows: Step D1: Determine the quantization configuration of the four-bit weight and the eight-bit activation, as well as the group size for group quantization.

[0089] Step D2: Quantize and group the weights after channel rearrangement in Example 2, and calculate the group quantization scaling factor for each group; at the same time, calculate the channel quantization scaling factor for each output channel of the weights.

[0090] Step D3: Combine the group quantization scaling factor and the channel quantization scaling factor to obtain the joint quantization scaling factor.

[0091] Step D4: Compress and store the original weight values ​​using 4-bit group quantization.

[0092] Step D5: Calculate the quantization scaling factor for each tag of the activation value during the inference phase and perform 8-bit quantization.

[0093] Step D6: Perform a fast format conversion on the 4-bit quantization weights, using a joint quantization scaling factor to simultaneously perform dequantization to floating-point operations and quantization to 8-bit integer operations.

[0094] Step D7: Perform matrix operations using an 8-bit general matrix multiplication kernel to obtain integer output results.

[0095] Step D8: Dequantize the integer type result and restore it to a floating-point representation.

[0096] Step D9: Continue executing subsequent network layers until the final result is output.

[0097] According to the large language model quantization method based on channel arrangement and smoothing proposed in this embodiment, W4A8 quantization of the large language model is achieved by using the sampled activation outlier values, the mixed numerical smoothing strategy, and the multi-dimensional feature-aware channel arrangement strategy. This reduces the storage space occupied by the model and the GPU memory required for runtime, thereby improving the accuracy of the model after W4A8 quantization.

[0098] The above description of several specific embodiments further details the technical solution provided by the present invention in order to highlight the advantages and benefits of the technical solution provided by the present invention. However, the above-described specific embodiments are not intended to limit the present invention. Any reasonable modifications and improvements to the present invention, combinations of embodiments, and equivalent substitutions based on the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A quantization method for large language models based on channel arrangement and activation adaptive smoothing, characterized in that, include: The steps involve using calibration data to drive forward inference of a large language model, statistically analyzing the maximum activation amplitude of the input activation values ​​in each linear layer along the input channel dimension, determining a significant activation outlier threshold based on the distribution of the maximum activation amplitude using percentile parameters, dividing the input channels into significant activation outlier channels and non-significant activation outlier channels, generating a smoothing scaling factor using a logarithmic domain smoothing strategy for significant activation outlier channels, and generating a smoothing scaling factor using a power function domain smoothing strategy in combination with the corresponding weight channel amplitude for non-significant activation outlier channels, and then fusing the generated channel smoothing scaling factor into the normalized layer parameters and applying a corresponding inverse scaling to the weights of subsequent linear layers to obtain a smoothed weight matrix. The steps are as follows: extract multidimensional numerical features of the input channels based on the smoothed weight matrix and construct a channel quantization sensitivity index; sort the input channels according to the channel quantization sensitivity index to generate a channel arrangement index; and perform a consistent rearrangement of the weight matrix and the input activation channels in the inference stage according to the channel arrangement index to obtain the structurally rearranged weight matrix. Based on the weight matrix after the structural rearrangement, a quantization scheme is adopted, which uses 4-bit group weight quantization and 8-bit per tag activation quantization. The group quantization scaling factor and the output channel quantization scaling factor are calculated to generate a joint quantization scaling factor. In the inference stage, the joint quantization scaling factor is used to convert the 4-bit weights into integer representations aligned with the 8-bit activations and call the integer matrix multiplication kernel to perform the operation, thus realizing the steps of low-bit high-efficiency inference.

2. The method for quantizing large language models based on channel arrangement and activation adaptive smoothing according to claim 1, characterized in that, The process of determining the significant activation outlier threshold based on percentile parameters includes sorting the maximum activation amplitude of each linear layer input channel by value and selecting the value located at a preset percentage position as the significant activation outlier threshold.

3. The quantization method for large language models based on channel arrangement and activation adaptive smoothing according to claim 1, characterized in that, The process of applying a logarithmic domain smoothing strategy to significantly activated outlier channels includes performing a logarithmic mapping on the activation magnitude of the significantly activated outlier channels and generating a first smoothing scaling factor.

4. The quantization method for a large language model based on channel arrangement and activation adaptive smoothing according to claim 1, characterized in that, The process of using a power function domain smoothing strategy for non-significantly activated outlier channels includes determining the power exponent parameter by combining the amplitude information of the corresponding weighted channel and generating a second smoothing scaling factor.

5. The large language model quantization method based on channel arrangement and activation adaptive smoothing according to claim 1, characterized in that, The process of constructing the channel quantization sensitivity index includes extracting the channel mean, channel variance, and channel norm of the input channels, normalizing each feature, and then weighting and combining them according to preset weights to generate the channel quantization sensitivity index.

6. The quantization method for a large language model based on channel arrangement and activation adaptive smoothing according to claim 1, characterized in that, The process of consistently rearranging the weight matrix and the input activation channels in the inference phase based on the channel arrangement index includes persistently storing the channel arrangement index as a model parameter, and reading the channel arrangement index in the inference phase to perform channel rearrangement in the same order on the input activations.

7. A quantization device for a large language model based on channel arrangement and activation adaptive smoothing, characterized in that, include: The large language model is driven by calibration data to perform forward inference. The maximum activation amplitude of the input activation value of each linear layer is statistically analyzed in the input channel dimension. Based on the distribution of the maximum activation amplitude, the significant activation outlier threshold is determined by percentile parameter. The input channel is divided into significant activation outlier channel and non-significant activation outlier channel. For significant activation outlier channel, a logarithmic domain smoothing strategy is used to generate a smoothing scaling factor. For non-significant activation outlier channel, a power function domain smoothing strategy is used in combination with the corresponding weight channel amplitude to generate a smoothing scaling factor. The generated channel smoothing scaling factor is fused into the normalized layer parameters and a corresponding inverse scaling is applied to the weights of subsequent linear layers to obtain the module of the smoothed weight matrix. Based on the smoothed weight matrix, multidimensional numerical features of the input channels are extracted and a channel quantization sensitivity index is constructed. The input channels are sorted according to the channel quantization sensitivity index to generate a channel arrangement index. The weight matrix and the input activation channels in the inference stage are uniformly rearranged according to the channel arrangement index to obtain the module of the structurally rearranged weight matrix. Based on the weight matrix after the structural rearrangement, a quantization scheme is adopted, which uses 4-bit group weight quantization and 8-bit per tag activation quantization. The group quantization scaling factor and the output channel quantization scaling factor are calculated to generate a joint quantization scaling factor. During the inference stage, the joint quantization scaling factor is used to convert the 4-bit weights into integer representations aligned with the 8-bit activations and call the integer matrix multiplication kernel to perform the operation, thus realizing a module for low-bit, high-efficiency inference.

8. A computer storage medium for storing computer programs, characterized in that, When the computer program is read by the computer, the computer executes the method of claim 1.

9. A computer, comprising a processor and a storage medium, characterized in that, When the processor reads the computer program stored in the storage medium, the computer executes the method of claim 1.

10. A computer program product, as a computer program, is characterized by: When the computer program is executed, it implements the method of claim 1.

Images