Model adjustment method and electronic equipment

Abstract

The invention provides a model adjustment method and electronic equipment, and is applied to the technical field of artificial intelligence. The model adjusting method comprises the steps that first evaluation values of a plurality of model parameters are determined according to calibration data, and the first evaluation values represent the importance degree of the model parameters to a model output result; dividing the plurality of model parameters into a plurality of groups; and determining the calculation precision of the model parameters in the group according to the first evaluation values of the plurality of model parameters in the group, so that the overall calculation precision of the model is smaller than the target calculation precision.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a model adjustment method and an electronic device. Background Technology

[0002] In recent years, large language models have demonstrated outstanding performance in natural language processing tasks, but the rapid increase in their number of parameters and computational requirements has brought serious challenges to their deployment on resource-constrained edge devices. Summary of the Invention

[0003] In view of this, the present disclosure provides a model adjustment method and an electronic device.

[0004] According to a first aspect of this disclosure, a model adjustment method is provided, comprising: determining first evaluation values ​​for multiple model parameters based on calibration data, wherein the first evaluation values ​​characterize the importance of the model parameters to the model output results; dividing the multiple model parameters into multiple groups; and determining the computational accuracy of the model parameters in a group based on the first evaluation values ​​of the multiple model parameters in the group, such that the overall computational accuracy of the model is less than the target computational accuracy.

[0005] A second aspect of this disclosure provides an electronic device comprising: a processor configured to determine first evaluation values ​​for a plurality of model parameters based on calibration data, the first evaluation values ​​characterizing the importance of the model parameters to the model output; to divide the plurality of model parameters into a plurality of groups; and to determine the computational precision of the model parameters in each group based on the first evaluation values ​​of the plurality of model parameters, such that the overall computational precision of the model is less than a target computational precision.

[0006] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0007] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0008] Figure 1 A flowchart illustrating a model adjustment method according to an embodiment of the present disclosure is shown schematically.

[0009] Figure 2 A schematic diagram illustrating the group division according to an embodiment of the present disclosure is shown.

[0010] Figure 3 A schematic diagram illustrating the principle of determining the calculation accuracy of each group according to an embodiment of the present disclosure is shown.

[0011] Figure 4 A schematic diagram illustrating the principle of determining a first evaluation value of a plurality of model parameters according to an embodiment of the present disclosure is shown.

[0012] Figure 5 The test results of the model adjustment method according to an embodiment of the present disclosure are illustrated schematically;

[0013] Figure 6 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0014] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0015] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0016] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0017] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0018] This disclosure provides a model adjustment method and an electronic device. Before introducing the technical solutions provided by this disclosure, the relevant technologies involved in this disclosure will be described first.

[0019] In recent years, large language models have demonstrated outstanding performance in natural language processing tasks, but the rapid increase in their number of parameters and computational requirements has brought serious challenges to their deployment on resource-constrained edge devices.

[0020] Model compression is widely used to reduce model size and inference costs, with quantization being one of the most mature and critical techniques. By converting weights and activation values, originally represented in high-precision floating-point format, into low-bit-width fixed-point or integer representations, storage space consumption and memory bandwidth pressure can be significantly reduced, while accelerating model inference. Within the quantization technology framework, Post-Training Quantization (PTQ) has attracted widespread attention because it achieves precision compression without retraining the original model, offering advantages such as low implementation cost, flexible deployment, and minimal dependence on the original training data.

[0021] In related technologies, post-training quantization methods are typically based on the analysis of features such as weight distribution, gradient information, or layer sensitivity. They allocate bit widths to different network layers or parameters in some form to achieve a trade-off between model accuracy and computational cost. However, with the ever-increasing scale of large language model parameters, quantization strategies relying solely on local statistical features or single-layer sensitivity often struggle to balance accuracy preservation and compression efficiency in extremely low bit width scenarios (e.g., 3 bits and below). On one hand, some methods primarily consider local indicators such as weight magnitude, local gradient changes, and single-layer quantization errors when allocating bits, failing to fully reflect the overall impact of parameters in the complete forward propagation and final output prediction. On the other hand, even schemes that introduce some global indicator mostly remain at a coarse modeling level at the layer level, and their characterization and protection of key weights remain insufficient when facing ultra-low bit width quantization.

[0022] Furthermore, quantization techniques generally lack interpretable descriptions of weights, resulting in a lack of clear theoretical basis and explanation for bit width allocation decisions. This makes quantization strategies prone to empirical and heuristic designs, risking suboptimal quantization configurations, which in turn affects the stability and robustness of the model in real-world deployment scenarios.

[0023] The following will be through Figures 1-5 The model adjustment method of the present disclosure embodiments will be described in detail.

[0024] Figure 1 A flowchart illustrating a model adjustment method according to an embodiment of the present disclosure is shown schematically.

[0025] like Figure 1 As shown, the model adjustment method in this embodiment includes operations S110 to S130.

[0026] In operation S110, based on the calibration data, the first evaluation value of multiple model parameters is determined. The first evaluation value characterizes the importance of the model parameters to the model output results.

[0027] In operation S120, multiple model parameters are divided into multiple groups.

[0028] In operation S130, the calculation accuracy of the model parameters in the group is determined based on the first evaluation value of multiple model parameters in the group, so that the overall calculation accuracy of the model is less than the target calculation accuracy.

[0029] For example, calibration data can be a representative set of input samples selected from the training or validation set to reflect the typical input distribution of the model in real-world application scenarios.

[0030] Model parameters can include weight parameters and / or bias parameters in each layer of the neural network, such as the convolution kernel parameters of the convolutional layer, the weight matrix elements of the fully connected layer, and the projection matrix parameters in the attention layer.

[0031] The first evaluation value can be the degree of influence of each model parameter on the model output. Different model parameters may have the same or different degrees of influence on the model output. For each model parameter, while maintaining full accuracy, calibration data can be input into the model. Based on the differences between the model output and the target output, such as the loss function value, prediction error, and distance between output vectors, a metric (first evaluation value) can be constructed to characterize the degree of influence of that parameter on the model output. For example, the absolute value of the gradient of the target loss function on the model parameter, the absolute value of the product of the gradient and the parameter value can be calculated on the calibration data. Alternatively, the change in model output (such as the norm of the output vector difference, the change in prediction probability, etc.) can be observed after subjecting the parameter to a preset perturbation (such as setting it to zero, scaling, or adding random noise). These values ​​reflecting the degree of influence can be normalized and mapped to a preset interval (such as [0, 1]) to obtain the first evaluation value used to uniformly compare the importance of different parameters. The larger the value, the higher the importance of the model parameter to the model output.

[0032] Grouping can be based on the attributes of model parameters. For example, the attributes of model parameters can be their function, the neural network layer they belong to, or their significance. Here, the significance of a model parameter can be its first evaluation value. For instance, parameters with first evaluation values ​​in the top percentages can be assigned to group 1, those in the middle range to group 2, and those at the bottom to group 3. Alternatively, parameters with similar functions or structural relationships can be grouped together based on the structure of the layer, channel, or tensor they reside in. Furthermore, within each group, subgroups can be further subdivided based on the first evaluation value. Taking the Transformer model as an example, different columns of the query / key / value projection matrix in the self-attention layer can be considered as different parameter units. First, groups can be formed within the same matrix based on the first evaluation value, and then columns with similar importance can be merged across layers to form different groups. Through this grouping, the model parameters in each group achieve relative consistency in importance.

[0033] Computational precision can be defined as the bit width of the data format used to represent and compute numerical values ​​(primarily weights and activation values) in the model. Bit width can be represented in bits; for example, computational precision can be 32-bit, 16-bit, 8-bit, 4-bit, 2-bit, etc. A lower bit width results in less computational power but also lower storage overhead and computational cost. The overall computational precision of the model can be considered as the average of the computational precisions of multiple groups.

[0034] The target computational precision can be the lowest bit value that preserves a certain level of inference performance of the model. The target computational precision can be a pre-defined global constraint on the degree of model compression. Typically, to achieve model compression, the target computational precision is a relatively small value. For example, the target computational precision could be 4 bits, 2 bits, etc. This disclosure does not specifically limit the size of the target computational precision; it can be set according to actual application requirements.

[0035] With a target computational precision of 4 bits as a constraint, the computational precision corresponding to the parameters in each group can be determined by statistically analyzing the distribution of the first evaluation values ​​of the parameters within that group, using bit width as the computational precision index. For example, the first computational precision of group 1 can be mapped to 6 bits (e.g., INT8), the second computational precision of group 2 to 4 bits (e.g., INT4), and the computational precision of group 3 to 2 bits (e.g., INT2). However, after assigning computational precision to each group, the average computational precision of each group is made lower than the target computational precision, i.e., (first computational precision + second computational precision + third computational precision) / 3 = 4.

[0036] Understandably, using calibration data to quantitatively evaluate the impact of each model parameter on the output results from a global perspective, and based on flexible group division, with the target computational accuracy as a constraint, allows for differentiated allocation of computational accuracy for different model parameters. This significantly reduces the model's storage footprint and computational load while ensuring that the overall average computational accuracy is less than the target accuracy. Compared to the traditional approach of applying uniform accuracy to all model parameters, this method can reduce computational complexity and bandwidth requirements during inference while maintaining similar or higher model prediction accuracy, thereby improving inference speed and reducing energy consumption.

[0037] Figure 2 A schematic diagram illustrating the group division according to an embodiment of the present disclosure is shown.

[0038] As described above, in operation S220, multiple model parameters are divided into multiple groups. In one possible implementation, this operation may further include the operation of dividing multiple model parameters into multiple groups according to their function, wherein multiple model parameters in a group belong to at least one neural network layer in the model.

[0039] For example, the function of a model parameter can refer to the computational task or type of role it plays in a neural network. For instance, in a large language model with a Transformer architecture, the query matrix, key matrix, and value matrix parameters of the self-attention layer are responsible for different feature projection functions, the up-projection and down-projection parameters of the feedforward neural network layer are responsible for expanding and compressing feature dimensions, respectively, and the scaling and offset parameters of the layer normalization module are responsible for adjusting the feature distribution. Parameters can be divided into different groups based on these functional differences, so that parameters within the same group have the same or similar functional attributes.

[0040] Multiple model parameters within a group can originate from the same neural network layer or from multiple different neural network layers. For example, when grouping by function, query matrix parameters from all self-attention layers in the model can be grouped into the same group, even if these parameters are distributed across multiple different layers (layer 1, layer 2, up to layer N). Because they perform the same query projection function, they can still be treated as a single group for subsequent computational precision allocation. Similarly, key matrix parameters from all layers can be grouped into another group, and value matrix parameters from all layers into yet another group. By using cross-layer functional grouping, the inter-layer isolation in traditional layer-by-layer quantization methods can be broken, achieving unification based on parameter functional attributes.

[0041] In one example, considering a large language model with a 32-layer Transformer structure, assuming each layer contains a self-attention module and a feedforward neural network module, a preliminary division is made based on functional attributes. All query matrix parameters from the 32 layers are uniformly grouped into the "Query Projection Function Group," which contains 32 query matrices. Similarly, multiple cross-layer function groups are established, such as the Key Projection Function Group (Group 1), Value Projection Function Group (Group 2), Attention Output Projection Function Group (Group 3), Feedforward Up Projection Function Group (Group 4), ..., Feedforward Down Projection Function Group (Group S). Group 1 includes model parameter 1 from the first layer, model parameter 3 from the third layer, ..., model parameter m from the fourth layer; Group 2 includes model parameter 2 from the first layer, model parameter 5 from the third layer, ..., model parameter m-2 from the fourth layer; ... Group S includes model parameter 4 from the first layer, model parameter 8 from the third layer, ..., model parameter m-5 from the fourth layer.

[0042] In this embodiment of the disclosure, when dividing multiple model parameters into multiple groups, corresponding grouping granularities can be adopted according to different quantization methods. Different quantization techniques have different requirements for parameter grouping, and selecting a grouping granularity that matches the quantization method can maximize quantization effect and performance preservation. For example, it can be layer-by-layer quantization, channel-by-channel quantization, structured block-level quantization, etc. Different quantization methods correspond to different computational precision allocations. For example, for layer-by-layer quantization, the computational precision allocation can be 8 bits for high-priority layers, 4 bits for medium-priority layers, and 2 bits for low-priority layers. For structured block-level quantization, 6 bits are allocated for high-priority layers and 2 bits for low-priority layers.

[0043] In other embodiments, in one feasible approach, the model parameters can be functionally divided into fine-grained groups, each group serving as an independent quantization unit. These fine-grained groups may include, but are not limited to, functional units such as attention heads, feedforward fully connected submodules, and input channel groups.

[0044] Fine-grained functional grouping refers to dividing the model at the internal structural level, using sets of parameters with specific computational functions as the smallest grouping units. Fine-grained functional grouping delves into the internal structure of neural network layers, identifying and extracting subsets of parameters that undertake independent computational tasks, and treating each functional subset as an independent quantization unit.

[0045] For example, in one example, suppose the model to be compressed is a 4-layer Transformer language model for text generation. The multi-head attention mechanism contains 5 attention heads. Fine-grained grouping based on attention heads: the 5 attention heads in the multi-head attention mechanism are treated as independent grouping units. Taking the first-layer multi-head attention mechanism as an example, its 5 attention heads are labeled as Head-1 to Head-5, and each attention head is treated as an independent group. The first evaluation value of Head-1 is 0.82, the first evaluation value of Head-2 is 0.76, the first evaluation value of Head-3 is 0.65, the first evaluation value of Head-4 is 0.58, and the first evaluation value of Head-5 is 0.63. After sorting according to the first evaluation value from high to low, those with a first evaluation value above 0.8 are assigned to group 1, those with a first evaluation value between 0.60 and 0.8 are assigned to group 2, and those with a first evaluation value below 0.60 are assigned to group 3. Thus, the 5 attention heads of the first layer are divided into 3 groups. This process is repeated sequentially. The five attention heads in layer 2 are divided into two groups, the five attention heads in layer 3 are divided into two groups, and the five attention heads in layer 4 are divided into three groups. The average of the first evaluation values ​​for each of the ten groups is calculated. The average first evaluation values ​​of the ten groups are then sorted from highest to lowest. Groups with an average first evaluation value greater than 0.8 are allocated 6 bits, groups with an average first evaluation value greater than 0.6 but less than 0.8 are allocated 4 bits, and groups with an average first evaluation value less than 0.6 are allocated 2 bits. The average calculation precision for multiple groups must be less than 4 bits.

[0046] In another example, fine-grained grouping based on feedforward fully connected submodules: The up projection and down projection of the feedforward neural network can be grouped as independent feedforward fully connected submodules. Taking the 18th layer feedforward network as an example: the first evaluation value of the up projection submodule is 0.78, and the first evaluation value of the down projection submodule is 0.71. The up projection submodule is allocated 4 bits of computational precision, and the down projection submodule is allocated 2 bits of computational precision. The average computational precision of the 18th layer feedforward network is calculated to be 3 bits, which is less than 4 bits.

[0047] In another example, fine-grained grouping based on input channel groups: The weight matrix of the linear transformation layer can be divided into multiple channel groups along the input channel dimension, with each group serving as an independent grouping unit. Taking the query projection matrix of layer 20 as an example, it can be divided into 8 input channel groups along the input dimension (column direction). Channel group 1: first evaluation value 0.85, channel group 2: first evaluation value 0.79, channel group 3: first evaluation value 0.92, channel group 4: first evaluation value 0.67, channel group 5: first evaluation value 0.88, channel group 6: first evaluation value 0.73, channel group 7: first evaluation value 0.81, channel group 8: first evaluation value 0.64. Similarly, the weight matrix of the convolutional layer can be divided into multiple channel groups along the input channel dimension, with each group serving as an independent grouping unit. Taking the query projection matrix of layer 10 as an example, it can be divided into 5 input channel groups along the input dimension. Channel group 9: First evaluation value 0.87, Channel group 10: First evaluation value 0.76, Channel group 11: First evaluation value 0.91, Channel group 12: First evaluation value 0.65, Channel group 13: First evaluation value 0.53. The weight matrix of the output layer is divided into multiple channel groups along the input channel dimension, each group serving as an independent unit. It can be divided into two input channel groups along the input dimension. Channel group 14: First evaluation value 0.83, Channel group 15: First evaluation value 0.60. Based on the first evaluation values ​​obtained above, the 8+5+2=15 input channel groups can be sorted from highest to lowest according to their first evaluation values. Then, a first evaluation value greater than 0.80 is allocated 8 bits of computational precision, a first evaluation value greater than 0.60 is allocated 4 bits of computational precision, and a first evaluation value less than 0.60 is allocated 2 bits of computational precision. The average computational precision of multiple input channel groups must be less than 4 bits.

[0048] It should be noted that the present disclosure does not impose specific limitations on the group division, and the group division method can be adjusted according to actual application needs.

[0049] Understandably, by using the aforementioned grouping methods based on function or saliency, especially by allowing a single group to contain parameters from multiple neural network layers, the limitations of traditional layer-by-layer quantization methods in terms of layer granularity can be overcome, enabling finer-grained and more flexible parameter management. This cross-layer grouping strategy allows parameters with similar functional attributes or similar importance to be treated uniformly, avoiding uneven resource allocation caused by inter-layer differences. This optimizes the allocation efficiency of computational precision globally, further improving the overall performance of the compressed model.

[0050] As described above, in operation S230, the computational precision of the model parameters in a group is determined based on the first evaluation values ​​of multiple model parameters in the group. In one possible implementation, this operation may further include the operations of: determining the priority of a group based on the first evaluation values ​​of multiple model parameters in the group; and determining the computational precision of the model parameters in the group based on the priority of the group, wherein the computational precision of the model parameters in the group is positively correlated with the priority of the group.

[0051] For example, the priority of a group can be a ranking index reflecting the importance of that group in the overall performance of the model. The higher the priority value, the more significant the impact of that group on the model output, and the more resources it should receive in the allocation of computational precision. The process of determining group priority is essentially to summarize and analyze the first evaluation values ​​of multiple model parameters within a group to form a comprehensive metric reflecting the overall importance of that group.

[0052] When determining group priorities, various aggregation methods can be used to statistically process the first evaluation values ​​of parameters within each group. For example, the arithmetic mean of the first evaluation values ​​of all parameters within a group can be calculated as the average importance index for that group; alternatively, a weighted average of the first evaluation values ​​of all parameters within a group can be calculated, where the weights can be set based on the number of parameters or the distribution range of parameter values; or, the maximum, median, or cumulative sum of the first evaluation values ​​within a group can be selected as characteristic values ​​representing the importance of the group. Based on the magnitude of these statistical values, all groups are ranked to determine the priority order of each group.

[0053] In one specific implementation, the cumulative importance score for a group can be obtained by summing the first evaluation values ​​of all model parameters within that group. For example, group 1 contains m model parameters, and their first evaluation values ​​are as follows: The cumulative importance score for group 1 is: After calculating the cumulative importance score for all groups in the model, they are sorted from highest to lowest score. The group with the highest score is given the highest priority, the group with the second highest score is given the second highest priority, and so on, thus establishing a priority sequence that covers all groups.

[0054] The positive correlation between computational precision and priority can be seen as follows: the higher the priority of a group, the larger the bit width corresponding to the allocated computational precision. For example, a group with priority 1 (the highest priority) can be allocated 8 bits of computational precision, a group with priority 2 can be allocated 6 bits, a group with priority 3 can be allocated 4 bits, a group with priority 4 can be allocated 3 bits, and groups with priority 5 and below can be allocated 2 bits. Through this positive correlation mapping mechanism, it is ensured that the parameter groups that have the most significant impact on the model output can retain higher numerical representation precision, thereby prioritizing the protection of the expressive power of key parameters during compression.

[0055] It should be noted that when allocating computational precision to model parameters according to their priority, the FP16 format can be retained for the most important parameters to maximize their expressive power. For example, preset parameters in the model can be directly matched with FP16 without participating in grouping. Preset parameters that do not participate in grouping can be divided into one group, while other model parameters can be divided into groups based on function and / or significance. However, it is still necessary to ensure that the average computational precision of the preset parameter group and the other parameter groups is less than the target computational precision (e.g., 4 bits). Parameters in the priority group (1) can also be matched with FP16. However, the average computational precision of all priority groups must still be less than the target computational precision (e.g., 4 bits).

[0056] When determining a specific computational precision allocation scheme, optimization adjustments need to be made while satisfying global constraints. The global constraint states that the average computational precision of all model parameters should be less than or equal to the target computational precision. Assuming the target computational precision is set to 4 bits, the model is divided into s groups, the s-th group contains Ns parameters, and the allocated computational precision is Bs (in bits), then the following constraints need to be satisfied: Under this constraint, higher computational precision can be allocated to high-priority groups first, and then the computational precision of subsequent groups can be gradually reduced until the global average precision requirement is met.

[0057] In one example, using the grouping example of the 32-layer Transformer model mentioned above, assuming that after two levels of grouping by function and significance, the model parameters are divided into 10 groups: Group 1, Group 2, ..., Group 10. For these 10 groups, the priority of each group is first determined. The priority of each group is determined based on the first evaluation values ​​of the multiple model parameters contained in each group. For example, the priority of each group can be determined by sorting them from highest to lowest according to the average significance level of each group, i.e., the average of the first evaluation: Group 1 is priority 1, Group 2 is priority 2, Group 3 is priority 3, and so on, with Group 10 being priority 10. Next, the computational precision of each group is determined based on the priority, while simultaneously satisfying the global constraint of a target computational precision of 4 bits. To meet the constraint of an average of 4 bits, the following allocation scheme can be adopted: Groups 1 to 3: 6 bits of calculation precision; Groups 4 to 6: 4 bits of calculation precision; Groups 7 to 10: 2 bits of calculation precision. Then the average calculation precision of each group is 3.6 < 4.

[0058] It is understandable that the above priority-based computational precision allocation mechanism realizes a systematic mapping from group importance assessment to precision resource allocation, ensuring that, under the premise of meeting the global compression target, limited high-precision computational resources are preferentially allocated to the parameter groups that have the most critical impact on model performance. Thus, even when the overall average computational precision is low, the inference accuracy and generalization ability of the model can still be maintained to the maximum extent.

[0059] Figure 3 A schematic diagram illustrating the principle of determining the calculation accuracy of each group according to an embodiment of the present disclosure is shown.

[0060] As described above, the operation involves determining the priority of a group based on the first evaluation values ​​of multiple model parameters within that group. In one possible implementation, this operation may further include: determining a second evaluation value corresponding to the group based on the first evaluation values ​​of multiple model parameters within that group, where the second evaluation value characterizes the overall importance of the model parameters within the group to the model output; and determining the priority of the group according to the magnitude of the second evaluation value, where the magnitude of the second evaluation value is directly proportional to the priority of the group.

[0061] For example, the second evaluation value can be a comprehensive index obtained by mathematically calculating the first evaluation values ​​of multiple model parameters within a group, used to quantify the overall impact of the group on the model output. Since the first evaluation value of each model parameter reflects the importance of that individual parameter, and a group typically contains multiple or even thousands of parameters, it is necessary to aggregate the first evaluation values ​​of all parameters within the group to form a unified metric that can represent the importance of the entire group, i.e., the second evaluation value.

[0062] When determining the second evaluation value, various mathematical statistical methods can be used to aggregate the first evaluation values ​​of the parameters within the group. For example, the arithmetic mean of the first evaluation values ​​of all model parameters within the group can be calculated, which reflects the average importance level of the parameters within the group; the sum of the first evaluation values ​​of all model parameters within the group can also be calculated, which reflects the overall importance scale of the group; a weighted average of the first evaluation values ​​within the group can also be calculated, where the weights can be set according to the gradient magnitude of the parameter, the absolute value of the parameter value, or the position of the parameter in the computation graph; in addition, statistical measures such as the maximum value, median, geometric mean, or harmonic mean of the first evaluation values ​​within the group can also be selected as the second evaluation value.

[0063] The magnitude of the second assessment value is directly proportional to the priority of the group, meaning that the larger the second assessment value, the higher the corresponding priority.

[0064] In one example, refer to Figure 3Continuing with the example of the grouping and prioritization of the 32-layer Transformer model, let's assume the model parameters are divided into 10 groups after two levels of grouping based on function and significance. For each of these 10 groups, calculate the second evaluation value. Taking group 1 as an example, this group contains 10 model parameters. The first evaluation values ​​for these 10 parameters are 0.91, 0.89, 0.86, 0.82, 0.80, 0.78, 0.77, 0.76, 0.75, and 0.75, respectively. Therefore, the second evaluation value for this group is calculated to be 0.809. Similarly, calculate the second evaluation values ​​for the other 9 groups. Assume the calculation results are as follows: Group 1's second evaluation value 1: 0.809, Group 2's second evaluation value 2: 0.778, Group 3's second evaluation value 3: 0.728, Group 4's second evaluation value 4: 0.612, Group 5's second evaluation value 5: 0.605, Group 6's second evaluation value 6: 0.581, Group 7's second evaluation value 7: 0.573, Group 8's second evaluation value 8: 0.556, Group 9's second evaluation value 9: 0.415, and Group 10's second evaluation value 10: 0.361. Sort the groups according to their second evaluation values ​​from highest to lowest, determining the priority of each group: Group 1 is priority 1, Group 2 is priority 2, Group 3 is priority 3, and so on, with Group 10 being priority 10. Next, determine the calculation precision for each group based on the priority, while simultaneously satisfying the global constraint of a target calculation precision of 4 bits. To meet the constraint of an average of 4 bits, the following allocation scheme can be adopted: Group 1 is allocated 6 bits for the first calculation precision, Group 2 is allocated 6 bits for the second calculation precision, and Group 3 is allocated 6 bits for the third calculation precision; Group 4 is allocated 4 bits for the fourth calculation precision, Group 5 is allocated 4 bits for the fifth calculation precision, and Group 6 is allocated 4 bits for the sixth calculation precision; Group 7 is allocated 2 bits for the seventh calculation precision, Group 8 is allocated 2 bits for the eighth calculation precision, Group 9 is allocated 2 bits for the ninth calculation precision, and Group 10 is allocated 2 bits for the tenth calculation precision. Then, the average calculation precision of each group is 3.6 < 4.

[0065] Understandably, by introducing a second evaluation value as a quantitative indicator of the overall importance of the group and establishing a proportional mapping relationship between the second evaluation value and priority, a transformation from the first evaluation value at the single-parameter level to a comprehensive metric at the group level is achieved, providing a clear basis for subsequent priority-based differentiated calculation precision allocation. This hierarchical evaluation system retains the fine-grained analysis results of the importance of individual model parameters while forming macro-level indicators suitable for resource allocation decisions through group-level aggregation processing, making the entire model compression process theoretically more rigorous.

[0066] As described above, the operation involves determining the computational precision of model parameters within a group based on the group's priority. In one feasible implementation, this operation may further include determining the computational precision of model parameters within a group based on the group's priority and the number of model parameters within that group, wherein the computational precision of the model parameters within a group is negatively correlated with the number of model parameters within that group.

[0067] For example, the number of model parameters in a group can be the total number of parameter elements contained in that group. For a group containing multiple weight matrices or parameter tensors, the number of parameters is equal to the sum of the number of elements in all matrices or tensors. For instance, if a group contains five weight matrices of shape 4096×4096, then the number of model parameters for that group is 5×4096×4096=83,886,080. The number of parameters reflects the size of the group and is a key factor affecting the proportion of model storage space and computational resource usage for that group.

[0068] The negative correlation between computational precision and the number of parameters implies that, among groups with the same or similar priorities, groups with more parameters should be assigned relatively lower computational precision, while groups with fewer parameters can be assigned relatively higher computational precision. This negative correlation design is based on the following technical considerations: during model compression, groups with more parameters occupy more overall storage space, and using lower precision for these groups can achieve more significant compression benefits; while groups with fewer parameters contribute less to overall storage space, and even if they are assigned higher precision, the impact on the overall compression ratio is relatively small, but it can effectively protect the expressive power of these parameters and avoid performance degradation caused by overquantization.

[0069] In one specific implementation, based on the determined group priority, the number of parameters can be further introduced as an adjustment factor to correct the initially allocated computational precision. Assuming the initial computational precision allocated to a certain group based on priority is B_initial (in bits), and the number of model parameters in that group is N, the computational precision can be adjusted downwards based on the number of parameters. The adjustment rule can be set as follows: when the number of parameters exceeds a preset threshold, the initially allocated computational precision is reduced to a certain extent; when the number of parameters is below the preset threshold, the initially allocated computational precision is maintained or increased. For example, tiered thresholds for the number of parameters can be set, such as 100 million as the first threshold, 50 million as the second threshold, and 10 million as the third threshold. For groups with more than 100 million parameters, the initial allocated computational precision is reduced by 1 bit; for groups with 50 million to 100 million parameters, the initial allocated computational precision is reduced by 0.5 bits (if half-bit precision is supported) or remains unchanged; for groups with 10 million to 50 million parameters, the initial allocated computational precision is maintained; and for groups with fewer than 10 million parameters, the initial allocated computational precision is increased by 1 bit. This hierarchical adjustment mechanism achieves a negative correlation between computational precision and the number of parameters.

[0070] In another specific implementation, an optimization algorithm can be used to solve for the optimal computational accuracy configuration for each group, while satisfying the global average computational accuracy constraint. Specifically, the computational accuracy allocation problem can be modeled as a constrained optimization problem: maximizing the model's expected performance metrics (e.g., accuracy or perplexity on the validation set) under the constraint that the weighted average computational accuracy of all groups does not exceed the target computational accuracy. In the objective function, the group priority can be used as a weighting coefficient for performance contribution, and the number of parameters can be used as a weighting coefficient for compression gains. By solving this optimization problem, the optimal allocation scheme that gives higher-priority groups higher accuracy and groups with more parameters lower accuracy can be automatically determined.

[0071] It should be noted that when introducing the number of parameters as a negative correlation factor, the dominant role of priority must be considered. Priority reflects the importance of a group to the model output and should be the primary basis for allocating computational accuracy; the number of parameters is an auxiliary factor for fine-tuning among groups with the same or similar priorities. Therefore, in the actual allocation process, it should be ensured that the computational accuracy of high-priority groups is generally higher than that of low-priority groups, and the negative correlation effect of the number of parameters only plays a moderating role between groups with insignificant priority differences.

[0072] In one example, continuing with the grouping and prioritization of the 32-layer Transformer model mentioned above, assume the model parameters are divided into 10 groups after two levels of grouping based on function and significance. For each of these 10 groups, a second evaluation value is calculated. Taking group 1 as an example, this group contains 10 model parameters. The first evaluation values ​​for the 10 model parameters are 0.91, 0.89, 0.86, 0.82, 0.80, 0.78, 0.77, 0.76, 0.75, and 0.75, respectively. Therefore, the second evaluation value for this group is calculated to be 0.809. Similarly, the second evaluation values ​​for the other 9 groups are calculated. Assume the calculation results are as follows: Group 1's second evaluation value 1: 0.809, Group 2's second evaluation value 2: 0.778, Group 3's second evaluation value 3: 0.728, Group 4's second evaluation value 4: 0.612, Group 5's second evaluation value 5: 0.605, Group 6's second evaluation value 6: 0.581, Group 7's second evaluation value 7: 0.573, Group 8's second evaluation value 8: 0.556, Group 9's second evaluation value 9: 0.415, and Group 10's second evaluation value 10: 0.361. The priority of each group is as follows: Group 1 has priority 1, Group 2 has priority 2, Group 3 has priority 3, and so on, with Group 10 having priority 10. First, determine the baseline calculation precision for each group based on the priority. The baseline calculation precision for high priority groups (priority 1 to 3) can be set to 6 bits, the baseline calculation precision for medium priority groups (priority 4 to 6) can be set to 4 bits, and the baseline calculation precision for low priority groups (priority 7 to 10) can be set to 2 bits.

[0073] Meanwhile, the parameter counts for the 10 groups are as follows: Group 1 has 167,772,160 parameters, Group 2 has 167,772,160 parameters, Group 3 has 450,887,680 parameters, Group 4 has 450,887,680 parameters, Group 5 has 201,326,592 parameters, Group 6 has 541,065,216 parameters, Group 7 has 201,326,592 parameters, Group 8 has 241,065,216 parameters, Group 9 has 167,772,160 parameters, and Group 10 has 250,887,680 parameters. Next, negative correlation adjustments will be made based on the parameter counts. The number of parameters in groups 3, 4, 6, 8, and 10 is approximately 450,887,680 or 541,065,216, significantly higher than the number of parameters in attention layer-related groups (approximately 167,772,160 to 201,326,592). To reflect the negative correlation between computational accuracy and the number of parameters, accuracy can be reduced for groups with more than 200 million and 400 million parameters. Groups 1, 2, and 9: approximately 168 million parameters, less than the 200 million threshold and below the 400 million threshold, maintaining the original computational accuracy; Groups 3, 4, and 6: approximately 451 million parameters, exceeding the 400 million threshold, reduced by 2 bits from the original computational accuracy; Groups 5, 7, 8, and 10: approximately 201 million parameters, greater than the 200 million threshold and below the 400 million threshold, reduced by 1 bit from the original computational accuracy. The final computational precision is 6 bits for group 1, 6 bits for group 2, 4 bits for group 3, 2 bits for group 4, 3 bits for group 5, 2 bits for group 6, 1 bit for group 7, 1 bit for group 8, 2 bits for group 9, and 1 bit for group 10. The recalculated average precision is approximately 2.8 bits, still satisfying the 4-bit target constraint. However, to allocate computational precision more reasonably while meeting the precision requirements, since groups 3 and 4 have higher priority, their computational precision can be further optimized. For example, the computational precision of group 3 can be set to 6 bits, and that of group 4 to 4 bits. The recalculated average precision is approximately 3.2 bits, still satisfying the 4-bit target constraint.

[0074] Understandably, the joint allocation strategy based on priority and parameter quantity achieves refined resource allocation that considers both group importance and compression efficiency. Simultaneously, the negatively correlated adjustment mechanism of parameter quantity ensures that the allocation of computational precision is not only based on parameter importance but also fully considers the impact of parameter size on storage and computational resources, thus achieving a better balance between meeting performance requirements and compression goals.

[0075] In some embodiments, the target calculation precision ranges from 2 bits to 4 bits.

[0076] The target computational precision is used to constrain the overall average quantization bit width of the model, and its value directly affects the balance between model compression and inference performance. In practical applications, to meet the storage and computing resource limitations of edge devices while preserving the model's predictive ability as much as possible, the target computational precision is usually set in a relatively low bit width range. For example, the target computational precision can be set to 2 bits, 2.5 bits, 3 bits, 3.5 bits, or 4 bits. This disclosure does not limit this.

[0077] Understandably, limiting the target computational precision to the range of 2-4 bits can achieve significant model compression to meet the deployment requirements of resource-constrained devices, while also maximizing the preservation of the model's inference performance under this low bit width constraint through a differentiated computational precision allocation mechanism based on the first evaluation value. This achieves the optimal balance between compression ratio and precision preservation in engineering practice.

[0078] In some embodiments, the model adjustment method of this embodiment further includes: responding to a difference between the target calculation accuracy and the calculation accuracy being greater than an adjustment threshold; determining first evaluation values ​​for multiple model parameters based on calibration data, wherein the first evaluation values ​​characterize the importance of the model parameters to the model output results; dividing the multiple model parameters into multiple groups; and determining the calculation accuracy of the model parameters in each group based on the first evaluation values ​​of the multiple model parameters in each group.

[0079] For example, the target computational accuracy refers to the average quantization accuracy of the model parameters that the user or system pre-sets as expected. This accuracy value directly determines the storage space usage and inference computation complexity of the quantized model. Computational accuracy refers to the weighted average or arithmetic mean of the computational accuracy of multiple model parameters actually calculated after a specific accuracy allocation scheme is executed. A difference greater than the adjustment threshold means that the current accuracy allocation scheme is not optimal and a readjustment mechanism needs to be triggered.

[0080] The adjustment threshold is a preset tolerance parameter used to determine whether the deviation between the current computational accuracy and the target computational accuracy is within an acceptable range. Setting this threshold requires a balance between the strictness of accuracy control and the efficiency of computational adjustments. A threshold that is too small will lead to frequent repeated adjustments, while a threshold that is too large may render accuracy constraints ineffective. For example, the adjustment threshold can be set to 15% of the target computational accuracy. For instance, when the target computational accuracy is 3.5 bits, the adjustment threshold can be set to 0.525 bits. That is, the difference between the overall computational accuracy of the final model and the target computational accuracy is less than 0.525.

[0081] In one example, during the re-execution, the initial evaluation values ​​of multiple model parameters are first recalculated based on the calibration data. This can be done using the same algorithm and parameter settings as the initial calculation, or by optimizing the sample size, sample distribution, or parameters of the evaluation algorithm based on feedback from the previous adjustment.

[0082] After obtaining the updated first evaluation value, the grouping operation for multiple model parameters is re-executed. At this point, the grouping strategy can be adjusted, such as changing the number of groups, adjusting the boundary thresholds for each group, or using different clustering algorithms. If the initial division of parameters into 4 groups resulted in insufficient flexibility in precision allocation, the number of groups can be increased to 5 or 6 during re-division, allowing for finer-grained precision configuration for parameters in different importance ranges. Alternatively, if a linear partitioning method with fixed thresholds was used initially (e.g., 0-0.3 for low importance, 0.3-0.7 for medium importance, and 0.7-1.0 for high importance), an adaptive partitioning method based on parameter percentiles can be used during re-division (e.g., the top 30% of parameters are low importance, the middle 45% are medium importance, and the bottom 25% are high importance) to adapt to the actual distribution of the first evaluation value.

[0083] After regrouping, the computational precision of the model parameters in each group is redefined based on the first evaluation values ​​of multiple model parameters in the updated groups. The goal is to search for a precision allocation scheme that minimizes model performance loss while meeting the target computational precision constraint.

[0084] By solving the above optimization problem, we can obtain a group accuracy allocation scheme that minimizes model performance loss while strictly satisfying accuracy constraints. During the solution process, the results of previous adjustments can be used as initial solutions to accelerate the algorithm's convergence speed.

[0085] Understandably, by introducing a monitoring mechanism for the difference between the target and actual computational accuracy, and automatically triggering a readjustment process when the difference exceeds an adjustment threshold, the problem of accuracy constraint failure caused by improper parameter settings, insufficient calibration data, or optimization algorithms getting stuck in local optima in the initial accuracy allocation scheme can be effectively avoided. This mechanism ensures that the final quantization model can strictly meet the constraints of storage and computing resources while preserving the performance of the original model to the greatest extent possible by iteratively optimizing the group division strategy, accuracy allocation scheme, and calibration data quality.

[0086] As described above, in operation S230, the computational accuracy of the model parameters in a group is determined based on the first evaluation values ​​of multiple model parameters in the group. In another possible implementation, this operation may further include: determining the activation values ​​of multiple model parameters based on calibration data; and determining the computational accuracy of the model parameters in the group based on the activation values ​​and the first evaluation values.

[0087] For example, activation values ​​can be the values ​​of intermediate feature tensors output by each layer of the neural network during the forward propagation of the model. Activation values ​​reflect the response strength of the input data after processing by specific layer parameters, and their distribution characteristics directly affect the cumulative propagation of quantization errors. When determining the computational accuracy of model parameters, relying solely on the global importance reflected by the first evaluation value may not fully account for the error amplification effect generated when activation values ​​are multiplied by weight parameters during quantization. Therefore, activation values ​​need to be incorporated as supplementary information into the computational accuracy allocation decision.

[0088] Specifically, during the process of inputting calibration data into the model, the activation value statistics of each model parameter's corresponding layer can be recorded, such as the maximum absolute value, variance, or distribution range of the activation values. For model parameters in a certain group, if the overall amplitude of the activation values ​​in the corresponding layer of that group is large or the distribution dispersion is high, then when performing matrix multiplication with the parameters of that group, even if the first evaluation value of the parameter itself is at a moderate level, the quantization error may still have a significant impact on the final output due to the amplification effect of the activation values. Therefore, when allocating computational precision, the sensitivity of the first evaluation value and the activation value should be considered comprehensively, and the allocated computational precision for parameter groups that are sensitive to activation should be appropriately increased, thereby suppressing the propagation of errors in the network.

[0089] In one example, an activation sensitivity score can be calculated for each group. This score is obtained by calculating the ratio of the maximum absolute value of the activation values ​​in the corresponding layer of that group to the maximum absolute value of the activation values ​​in the entire model. If a group has a first evaluation value of 0.6 and an activation sensitivity score of 0.8, the two can be weighted and fused, for example, by using a combined score of 0.5 × 0.6 + 0.5 × 0.8 = 0.7 to determine the priority of that group and thus allocate corresponding computational precision. In contrast, if another group has a first evaluation value of 0.65 but an activation sensitivity score of only 0.4, its combined score is 0.5 × 0.65 + 0.5 × 0.4 = 0.525. In this case, the computational precision allocation priority of this group is lower than that of the former, and a lower bit width can be allocated.

[0090] In one example, group 4 was originally assigned 4-bit precision. However, analysis of the calibration data revealed that the maximum absolute value of activations in this group was 1.5 times that of the entire model, indicating a significantly high activation sensitivity score. Therefore, this activation-sensitive parameter was separated from the moderately important group to form a separate activation-sensitive subgroup, and its computational precision was increased from 4-bit to 5-bit. Simultaneously, the computational precision of the remaining parameters with lower activation sensitivity in groups 5 and 6 was reduced from 4-bit to 3-bit to maintain the model's average computational precision within the 4-bit constraint.

[0091] Understandably, by introducing activation values ​​as an auxiliary decision-making basis, we can more comprehensively characterize the influence path of model parameters in the actual inference process, avoid the accumulation of quantization errors caused by ignoring activation distribution characteristics, and thus achieve more refined allocation of computational precision while maintaining the target computational precision constraint, significantly improving the performance of the model in extremely low bit-width quantization scenarios.

[0092] Figure 4 A schematic diagram illustrating the principle of determining a first evaluation value of a plurality of model parameters according to an embodiment of the present disclosure is shown.

[0093] As described above, the calibration data includes multiple input samples. In operation S210, based on the calibration data, first evaluation values ​​for multiple model parameters are determined. In one possible implementation, this operation may further include the following steps: for any one of the multiple input samples, obtain the model output result corresponding to the input sample; based on the output result, determine the contribution values ​​of multiple model parameters to the input sample using a layer-by-layer correlation propagation algorithm, where the contribution values ​​characterize the importance of the model parameters to the model output result corresponding to the input sample; for any one of the multiple model parameters, normalize the contribution values ​​of the model parameter to the multiple input samples to obtain the first evaluation value of any one of the multiple model parameters.

[0094] For example, multiple input samples in the calibration data can be representative subsets of data selected from the training or validation sets. For instance, in a natural language processing task, 100 to 1000 text samples covering different topics and language styles can be selected, while in an image classification task, several typical images of different categories can be selected. These input samples need to fully reflect the distribution characteristics of the input data in the actual application scenario.

[0095] The model output can be the prediction result generated by the model in the output layer after the input sample is calculated by each layer of the network during the forward propagation process. For example, the class probability distribution vector in the classification task, the sequence logits in the generation task, and the predicted value in the regression task.

[0096] Layer-wise Relevance Propagation (LRP) is an attribution method that decomposes the contribution of a model's output to the parameters of each layer in reverse order. The basic principle is to propagate the relevance score corresponding to the output layer forward, starting from the output layer, according to a pre-defined relevance conservation rule, to each neuron and parameter, ensuring that the total relevance at each layer remains constant. This ultimately yields a quantitative indicator of the contribution of each model parameter to the output. For example, for a fully connected layer, the relevance score returned from the next layer can be proportionally distributed to the weight parameters of that layer based on the ratio of the product of the parameter value and the activation value to the output of that layer. For a convolutional layer, the contribution of each convolutional kernel parameter to the output feature map can be calculated based on the element-wise product of the kernel parameter and the corresponding receptive field activation value. Through this layer-wise backpropagation mechanism, the contribution value of each model parameter to a specific input sample can be calculated, and the magnitude of this contribution value directly reflects the degree of influence of that parameter on the output of that input sample.

[0097] The contribution value represents the strength of an individual model parameter's influence on the output of a single input sample. Its value can be positive or negative, with a larger absolute value indicating a more significant influence. Since different input samples may activate different paths in the model, the contribution value of the same model parameter may differ for different input samples. Therefore, to obtain an evaluation metric that reflects the overall importance of the model parameter in diverse input scenarios, it is necessary to aggregate the contribution values ​​of the parameter for all input samples.

[0098] Specifically, for a given model parameter, after obtaining its contribution value sequence for all input samples in the calibration data, the statistics of these contribution values ​​can be calculated as the basis for aggregation, such as taking the average, median, or maximum of the absolute values ​​of the contribution values. Then, the aggregated contribution values ​​are normalized, mapping them to a preset numerical range, such as [0, 1], thereby eliminating dimensional differences between different parameters and obtaining a first evaluation value that can be uniformly compared. For example, if the absolute value sequence of a parameter's contribution values ​​for 100 input samples is... First, calculate the average contribution value. Then, using the max-min normalization formula, Mapped to the interval [0, 1], i.e., the first evaluation value ,in and These are the maximum and minimum values ​​of the average contribution of all model parameters, respectively.

[0099] In one example, refer to Figure 4Taking the quantization compression task of a large language model with 7 billion parameters as an example, assume that 500 text samples covering various task types such as question answering, summarizing, and translation are selected from the validation set as calibration data. For one question answer sample, "What is quantum computing?", after inputting it into the model, the model's output layer generates the logits distribution of the corresponding answer text. Subsequently, a layer-by-layer relevance propagation algorithm is applied, starting from the output layer, and propagating the relevance score of the output logits back to each Transformer layer of the model based on the product relationship between the activation values ​​and parameter values ​​of neurons in each layer. For example, for a certain query projection matrix parameter w in the 12th layer self-attention mechanism... q The calculated contribution value for this input sample is 0.025, representing the weight of this parameter's influence on the output result of that sample. This process is repeated for all 500 input samples to obtain the parameter w. q For the contribution value sequence of 500 samples, the average absolute value is calculated to be 0.032. Similarly, after performing the same operation on all parameters in the model, the average contribution value of all parameters is found to be in the range of [0.001, 0.085]. Regarding parameter w... q The average contribution value was normalized to obtain the first evaluation value = (0.032-0.001) / (0.085-0.001)≈0.369.

[0100] Based on the above process, after calculating the first evaluation value for all 7 billion parameters in the model, the rationality of the group division can be further optimized by combining the aforementioned activation-aware computational precision allocation strategy. For example, parameters with a first evaluation value higher than 0.7 are classified as high-importance groups, accounting for approximately 15%; parameters with a first evaluation value between 0.3 and 0.7 are classified as medium-importance groups, accounting for approximately 55%; and parameters with a first evaluation value lower than 0.3 are classified as low-importance groups, accounting for approximately 30%. On this basis, the parameters within each group are further subdivided based on activation sensitivity, ultimately forming a hybrid precision allocation scheme that includes activation-sensitive subgroups. Under the constraint of maintaining a target computational precision of 3.5 bits, high-importance groups are allocated 6 bits of precision, activation-sensitive subgroups are allocated 5 bits of precision, the remaining portion of medium-importance groups are allocated 3 bits of precision, and low-importance groups are allocated 2 bits of precision. The average computational precision of the model is 0.15×6 + 0.1×5 + 0.45×3 + 0.3×2 = 3.45 bits, which satisfies the constraint.

[0101] Understandably, by using the layer-by-layer correlation propagation algorithm, we can start from the model output and accurately quantify the contribution of each parameter to the output in an interpretable way. By normalizing the contribution values ​​of multiple input samples, we can obtain a first evaluation value that reflects the global importance of the parameter. This provides an accurate and reliable decision basis for subsequent differential calculation precision allocation based on the first evaluation value and activation value, thereby maximizing the preservation of the model's inference performance under extremely low bit-width quantization conditions.

[0102] Figure 5 The test results of the model adjustment method according to an embodiment of the present disclosure are illustrated schematically.

[0103] Reference Figure 5 , Figure 5 This section indicates the performance of LLMs under different quantization precipitates, specifically the performance changes of the two large language models under different numerical precision (bit width) settings. "Large Language Models (LLMs)" refers to large-scale pre-trained models used for natural language processing, including Llama3-8B and Llama3.1-8B models; "Llama3-8B" represents the Llama3 version model with approximately 8 billion (8B) parameters, and "Llama3.1-8B" represents the upgraded Llama3.1 version model based on the same 8 billion parameter scale; 2.5-bit, 3-bit, 3.5-bit, and 4-bit represent the effective bit width used after quantization of the model weights. For example, 2.5-bit represents extremely low bit quantization precision, while 3-bit, 3.5-bit, and 4-bit represent progressively higher precision low-bit representations; FP16 represents unquantized or high-precision floating-point representation (16-bit floating-point number, halfprecision), typically used as the baseline precision.

[0104] Numerically, the values ​​in each column (such as 49.97, 67.92, 71.54, etc.) represent the score or accuracy on a certain benchmark evaluation. The larger the value, the better the model's performance. The table shows that as the bit width increases from 2.5-bit to 4-bit, the score of Llama3-8B gradually increases from 49.97 to 71.09, and the score of Llama3.1-8B increases from 50.71 to 71.54. At 4-bit, the scores of both are very close to their respective FP16 baselines (FP16 of 71.00 for Llama3-8B and FP16 of 71.77 for Llama3.1-8B). This indicates that FP16-level performance can be maintained at 4-bit low-bit quantization, while excessively low bit widths (such as 2.5-bit) will lead to a significant performance drop. At the same quantization precision, the score of Llama3.1-8B is generally slightly higher than that of Llama3-8B, indicating that the model version upgrade brings stable performance advantages at all quantization precisions. Therefore, by appropriately selecting medium-to-high bit-width quantization such as 3.5-bit or 4-bit, the reasoning performance of large language models can be maintained well while significantly compressing the model and reducing storage and computation costs.

[0105] The model adjustment method of this disclosure can be implemented based on the GloMix (Global-aware Mixed Precision Quantization) framework through a three-pronged design of "global drive, group optimization, and collaborative deployment". First, global saliency assessment: Weight contribution is inferred from the output. Abandoning traditional local statistical features, layer-wise relevance propagation (LRP) is introduced into the field of large model quantization for the first time to construct a global attribution. The evaluation value not only considers the numerical value of each layer's weights but also reflects their causal influence on the output in the entire network structure, avoiding local misjudgments where large weights are considered important. Second, group-based global bit allocation optimization: Cross-layer collaborative resource scheduling. To balance optimization accuracy and computational feasibility, heuristic search or dynamic programming is used to ensure that the globally most important group receives priority access to high-bit representations. Furthermore, it breaks down inter-layer barriers, enabling dynamic allocation of bit resources across modules, truly achieving "global optimum" rather than "local optimum". Third, activation-aware and hardware-coordinated inference support integrates global saliency evaluation and bit allocation into the activation-aware weighted quantization (AWQ) framework, preserving key weights sensitive to activation during quantization to enhance robustness; it also supports parallel processing of multiple bit operations (such as INT2×FP16 and INT4×FP16) in the same model.

[0106] Based on the above model adjustment method, this disclosure also discloses an electronic device, which will be described below in conjunction with... Figure 6 Provide a detailed description of the electronic device.

[0107] Figure 6 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0108] like Figure 6 As shown, the electronic device 300 of this embodiment includes a processor 310.

[0109] The processor 310 is used to determine the first evaluation value of multiple model parameters based on calibration data, wherein the first evaluation value characterizes the importance of the model parameter to the model output result; divide the multiple model parameters into multiple groups; and determine the calculation accuracy of the model parameters in the group based on the first evaluation value of the multiple model parameters, so that the overall calculation accuracy of the model is less than the target calculation accuracy.

[0110] For details on the specific operations performed by this processor, please refer to the description of the model adjustment method above, which will not be repeated here.

[0111] In some embodiments, the electronic device 300 of this embodiment further includes an image processor 320.

[0112] The graphics processor 320 includes a first computing unit 321 and a second computing unit 322.

[0113] The floating-point calculation precision supported by the first calculation unit 321 matches the first calculation precision of the model parameters in the first group, and the floating-point calculation precision supported by the second calculation unit 322 matches the second calculation precision of the model parameters in the second group, wherein the first calculation precision is higher than the second calculation precision.

[0114] When the electronic device executes the model inference process, the graphics processor is controlled to: use the first computing unit 321 to perform floating-point matrix multiplication on the model parameters in the first group, represented with a first computing precision, and the corresponding model activation values; and use the second computing unit 322 to perform floating-point matrix multiplication on the model parameters in the second group, represented with a second computing precision, and the corresponding model activation values, so that the matrix multiplication operations for different groups are executed in parallel between the first computing unit 321 and the second computing unit 322.

[0115] In one example, the graphics processor 320 can be a standalone graphics processing unit (GPU) chip, a GPU module integrated into a system-on-chip (SoC), or a tensor processing unit (TPU) or neural processing unit (NPU) specifically designed for accelerating neural network inference. The graphics processor 320 typically contains a large number of parallel computing cores, capable of efficiently performing core computational operations in deep learning models such as matrix multiplication and convolution.

[0116] The first computing unit 321 and the second computing unit 322 can be physically isolated or logically partitioned different computing resource regions within the graphics processor 320. In the physically isolated implementation, the first computing unit 321 and the second computing unit 322 can correspond to different Streaming Multiprocessor (SM) clusters or Tensor Core arrays on the GPU chip, respectively. They have independent register files, shared memory, and arithmetic logic units, and can execute different computing tasks in complete parallelism without interference. In the logically partitioned implementation, the first computing unit 321 and the second computing unit 322 can share the same physical computing resources, but through hardware scheduling mechanisms or software driver layer task queue management, computing tasks of different precisions are allocated to different execution streams or CUDA (Compute Unified Device Architecture) streams, achieving pipelined parallel execution.

[0117] The floating-point precision supported by the first computing unit 321 matches the first computing precision of the model parameters in the first group. This means that the hardware circuit design or configurable operation mode of the first computing unit 321 can natively support or efficiently execute floating-point operations expressed in the first computing precision. For example, if the first computing precision of the model parameters in the first group is a 6-bit floating-point format, the first computing unit 321 can have a dedicated 6-bit floating-point multiply-add (FMA) unit built in, or it can achieve optimal throughput and energy efficiency when performing 6-bit floating-point operations by configuring its data path width and exponent mantissa bit allocation. Similarly, the floating-point precision supported by the second computing unit 322 matches the second computing precision of the model parameters in the second group. If the second computing precision is a 2-bit or 3-bit floating-point format, the second computing unit 322 can complete more low-precision floating-point operations in a single clock cycle through a narrower data path and simplified operation logic, thereby improving the overall computing throughput.

[0118] During the inference process of the model executed by the electronic device, the processor 310 first reads the model parameter data with the precision configuration completed from the memory. According to the group identifier of each parameter, the model parameters of the first group are loaded into the video memory area associated with the first computing unit 321 in the graphics processor 320, and the model parameters of the second group are loaded into the video memory area associated with the second computing unit 322. At the same time, the scheduling module of the processor 310 or the graphics processor 320 decomposes the computation graph in the inference task into multiple sub-tasks, and identifies the matrix multiplication operation nodes that need to use the first group of parameters and the matrix multiplication operation nodes that need to use the second group of parameters. For the operation nodes that use the first group of parameters, the scheduling module sends a calculation instruction to the first computing unit 321, instructing it to read the model parameter matrix represented with the first calculation precision and the corresponding model activation value vector from the corresponding video memory area, perform floating-point matrix multiplication operations, and calculate the output features. For the computation node using the second set of parameters, the scheduling module sends a computation instruction to the second computation unit 322, instructing it to read the model parameter matrix and the corresponding model activation value vector, expressed with the second computational precision, from the corresponding video memory area, perform floating-point matrix multiplication, and calculate the output features. Since the first computation unit 321 and the second computation unit 322 are independent at the hardware level or scheduled in parallel at the logical level, they can execute their respective matrix multiplication tasks simultaneously, achieving true parallel computation and thus significantly shortening the total time of the inference process.

[0119] Understandably, by configuring multiple computing units in the graphics processor of electronic devices that match the computational precision of different group model parameters, and using these computing units to perform matrix multiplication operations for different group parameters in parallel during inference, hardware-level acceleration can be achieved by fully utilizing the Tensor Cores and low-precision computing units of modern GPUs. This not only fully leverages the advantages of mixed-precision quantization in storage compression, but also significantly shortens inference latency and reduces inference power consumption through hardware-level parallel acceleration. This provides an efficient and feasible hardware support solution for the real-time deployment of large language models on resource-constrained edge devices, achieving synergistic optimization of model compression and inference acceleration.

[0120] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0121] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A model adjustment method, comprising: Based on the calibration data, first evaluation values ​​are determined for multiple model parameters, where the first evaluation value characterizes the importance of the model parameters to the model output. The multiple model parameters are divided into multiple groups; Based on the first evaluation value of multiple model parameters in the group, the calculation accuracy of the model parameters in the group is determined so that the overall calculation accuracy of the model is less than the target calculation accuracy.

2. The method according to claim 1, wherein the plurality of model parameters are divided into multiple groups, including: Based on the function or significance of the model parameters, the multiple model parameters are divided into multiple groups, and multiple model parameters in one group belong to at least one neural network layer in the model.

3. The method according to claim 1 or 2, wherein determining the calculation accuracy of the model parameters in the group based on the first evaluation value of a plurality of model parameters in the group comprises: The priority of the group is determined based on the first evaluation value of multiple model parameters in the group; The calculation precision of the model parameters in the group is determined according to the priority of the group, wherein the calculation precision of the model parameters in the group is positively correlated with the priority of the group.

4. The method according to claim 3, wherein determining the priority of the group based on a first evaluation value of a plurality of model parameters in the group comprises: Based on the first evaluation values ​​of multiple model parameters in the group, a second evaluation value corresponding to the group is determined, wherein the second evaluation value characterizes the overall importance of the model parameters in the group to the model output result; The priority of the group is determined according to the magnitude of the second evaluation value, and the magnitude of the second evaluation value is proportional to the priority of the group.

5. The method according to claim 3 or 4, wherein determining the calculation accuracy of model parameters in the group based on the priority of the group includes: The calculation accuracy of the model parameters in a group is determined based on the priority of the group and the number of model parameters in the group, wherein the calculation accuracy of the model parameters in a group is negatively correlated with the number of model parameters in the group.

6. The method according to claim 1, wherein the target calculation precision ranges from 2 bits to 4 bits; The method further includes: The difference between the target calculation accuracy and the calculation accuracy is greater than the adjustment threshold. Based on the calibration data, first evaluation values ​​are determined for multiple model parameters, where the first evaluation value characterizes the importance of the model parameters to the model output. The multiple model parameters are divided into multiple groups; The calculation accuracy of the model parameters in the group is determined based on the first evaluation value of multiple model parameters in the group.

7. The method according to claim 1 or 3, wherein determining the calculation accuracy of the model parameters in the group based on the first evaluation value of a plurality of model parameters in the group comprises: Based on the calibration data, determine the activation values ​​for multiple model parameters; The calculation accuracy of the model parameters in the group is determined based on the activation value and the first evaluation value.

8. The method according to claim 1, wherein the calibration data includes multiple input samples, and the step of determining first evaluation values ​​for multiple model parameters based on the calibration data includes: For any one of the plurality of input samples, obtain the model output result corresponding to the input sample; Based on the output results, the contribution values ​​of the multiple model parameters to the input sample are determined by a layer-by-layer correlation propagation algorithm. The contribution values ​​characterize the importance of the model parameters to the model output results corresponding to the input sample. For any one of the multiple model parameters, the contribution value of the model parameter to the multiple input samples is normalized to obtain the first evaluation value of any one of the multiple model parameters.

9. An electronic device, comprising: The processor is configured to determine first evaluation values ​​for multiple model parameters based on calibration data, wherein the first evaluation values ​​characterize the importance of the model parameters to the model output. The multiple model parameters are divided into multiple groups; Based on the first evaluation value of the plurality of model parameters, the calculation accuracy of the model parameters in the group is determined so that the overall calculation accuracy of the model is less than the target calculation accuracy.

10. The electronic device of claim 9, further comprising a graphics processor signal-connected to the processor. The graphics processor includes: First computing unit and second computing unit; The floating-point calculation precision supported by the first calculation unit matches the first calculation precision of the model parameters in the first group, and the floating-point calculation precision supported by the second calculation unit matches the second calculation precision of the model parameters in the second group, wherein the first calculation precision is higher than the second calculation precision; When the electronic device executes the reasoning process of the model, it controls the graphics processor: Using the first computing unit, a floating-point matrix multiplication operation is performed on the model parameters in the first group, expressed with the first computing precision, and the corresponding model activation values; Using the second computing unit, floating-point matrix multiplication is performed on the model parameters and corresponding model activation values ​​in the second group, expressed with the second computing precision, so that matrix multiplication operations for different groups can be performed in parallel between the first computing unit and the second computing unit.

Images